Scotney Castle ruins nestle in the beautiful valley of the River Bewl. Although the only substantial remains of the original Scotney Castle is the circular tower in the southern corner (the Ashburnham Tower), building of Scotney Castle with its four corner towers, is believed to have commenced c1378. The castle was roughly rectangular in shape, with the northern perimeter being slightly longer than the others. Prior to the castle , the earliest recorded owner of the manor was Lambert de Scoteni in 1137.

In 1630 the east range was rebuilt in the style of Inigo Jones, and was a substantial three-storied structure dwarfing the remaining Elizabethan and medieval survivals. The ruins of this can still be seen in part today. In 1837, under the ownership of the Hussey family, the first stones of a new manor house were laid. This was completed in 1843, leaving the old estate to be incorporated as a romantic feature of the gardens. Although the Elizabethan wing and tower remained as a dwelling for the Bailiff until 1905, the eastern range was carefully dismantled leaving only the most interesting features.

The Catholic Darrell family owned the estate for some 350 years. From 1591-1598 Scotney Castle was the secret centre of activities of a famous Jesuit, Father Richard Blount. Several well concealed priest holes, which were incorporated by the then owner, Thomas Darrell. The estate was purchased in 1778 by Edward Hussey. It was Edward Hussey's grandson, another Edward, who built the 'new' Scotney Castle on a terrace 25 metres above the Old Castle commissioning Anthony Salvin to design a modern Victorian country house in an Elizabethan style. The mellow sandstone was quarried from the slope below, and the resulting area turned in to a dramatic Quarry Garden. In the Quarry Garden a fossilised impression of a dinosaur's footprint can be seen in the ripples of the 100,000,000 year old Great Wealden Sea.

In 1952 Edward Hussey III's grandson Christopher inherited Scotney, together with his wife Elizabeth, they planted the gardens with year round color. On his death in 1970 the estate was left to the National Trust in apt fulfilment of his family motto: Vix ea nostra voco ('I scarcely call these things our own').