The first house on this site was built in c.1698 and was replaced by two houses in 1770. Due to severe structural defects these were in turn demolished and rebuilt in 1963 in the Georgian style. Residents have ranged from an early 18th century chancellor of the exchequer to the Reverend Dr. Dodd, known as the 'Macaroni Parson' owing to his extravagant taste in clothes. After forging a bond for £4,200 the Reverend Dodd was hanged at Tyburn in 1775 - curiously the butler was hanged there almost exactly a year later for burgling the house. A later resident, Sir Richard Worsley, was ambassador to Venice from 1793 until its annexation in 1797. He managed to escape Venice when it was invaded, travelling home on a Royal Navy ship to protect the priceless antiquities he had bought in Italy.
The house and its immediate neighbour became the Irish Office from 1806 until 1922 when British rule ended over much of Ireland.