The castle was built as a fortress during the early 1400s by the 7th Earl of Desmond. In the early 1600′s the castle was granted to the Southwell family who converted it to a manor house. Sir Thomas Southwell who was a key figure in bringing Palatine refugees to Ireland, lived at Castle Matrix when he settled 100 families on his estate at Rathkeale in 1709. Visitors that came to stay at Castle Matrix were the poet Edmund Spenser and Sir Walter Raleigh. By the 1960s, the castle had fallen into disrepair and was restored by an American, Sean O’Driscoll.