Kirtling Towers is all that now remains of the ancient Kirtling Hall, originally built about the reign of Henry VI but the main part of the building was pulled down in 1801. The roof of the tower is flat and leaded, while the house has steeply pitched slate roofs with a gable parapet to the east. The gatehouse is of three storeys with the main south entrance blocked and the ground floor incorporated into the plan of C19 house. The gatehouse has four octagonal corner turrets above an embattled parapet. The two larger turrets are to the south, flanking the original entrance with a four-centred arch infilled with C19 window. Above is a fine two storey limestone oriel window. C19 house has similar detail to the gatehouse and is of two storeys. The house also has a closed, embattled porch. It stands on the site of a C13 castle and hall, which was built in C16 and survived until 1801. The moat which surrounds the site of Kirtling Towers is the largest surviving in Cambridgeshire.
The castle went into decline after 1691 and by 1735 the Victoria County History of the castle describes the property as being “in disorder”. Much of the castle was pulled down in 1748 in order to make the remainder habitable for Lord Elibank, but the property went into decline again after his death in 1762. By the 1770s it was uninhabitable and most of the castle pulled down in 1801. In the 1830s the gatehouse was turned into a residential property and was renamed Kirtling Tower.