The original two-storey Georgian structure was designed by Mortimer Lewis and featured 13 large and expensive windows in the facade to afford a clear view of shipping activity in Sydney Cove. It was partially dismantled and expanded to three levels under the supervision of the then Colonial Architect, James Barnet, in 1887. Various additions were made over the next century, by his successor Walter L. Vernon, but some significant portions of the original building remain. The building ceased to be the headquarters of the Customs Service in 1990.
The building was taken over City of Sydney Council in 1994, when it became a venue for exhibitions and private functions. After being refurbished in 2003, it has also become the new home of the City of Sydney Library.