Fairhaven Woodland and Water garden was created by Major Henry Broughton, who later became the 2nd Lord Fairhaven. The hall, woodland, water garden and inner broad were all part of the South Walsham Estate, which he purchased in 1946.

The house and formal gardens had been used as a convalescence home and the woodland and water garden as a training ground for the home guard during World War Two. Pleasure boats were sunk in the inner broad, which was also covered with barbed wire, to prevent flying boat landings. Tanks were hidden in the garden; some of the tank bays can be seen in the garden today. The house had fallen into disrepair and the garden had become a jungle.

Initially efforts were concentrated on restoring the house. The family moved into South Walsham Hall in 1947. The 2nd Lord Fairhaven was an active and enthusiastic gardener and designed the garden himself. He had a team of seven gardeners and two woodmen to assist him in clearing the dense jungle that had grown up in what is now the main garden. He gradually introduced shade and water loving plants, the most spectacular being candelabra primula. Thousands of these colourful plants flower during May and early June. Other plants were imported from around the world, such as the skunk cabbage (Lysichiton americanus) from North America and camellias and rhododendrons from the Himalayas. It took 15 years to create the garden.

Many of the trees were grown in a dedicated tree nursery and huge greenhouses meant that more than 90% of the plants could be grown from seed.

In 1963 Major Broughton became the 2nd Lord Fairhaven, receiving the title as his elder brother had no heirs. Lord Fairhaven died in1973. He had requested that the garden be left in Trust for the public to enjoy. The title of Lord Fairhaven passed onto his son Ailwyn, the 3rd Lord Fairhaven who is Chairman of the Fairhaven Garden Trust. The garden opened to the public on the 18th of April 1975.