In the 17th century it was the family seat of Mervyn Tuchet, 2nd Earl of Castlehaven.
In 1770 William Beckford inherited the estate from his father. In the 1790s he had a vast house built there, called Fonthill Abbey, in the Gothic style. The house was built quickly and using inferior workmanship, and its enormous tower fell down three times and was rebuilt twice. Almost all of it was subsequently demolished.