This square was laid out from 1811 by a successful chimney sweep turned developer and was soon occupied by fashionable and wealthy people. An early tenant of the house was Admiral George Stewart, Earl of Galloway, before it was acquired by the recently widowed Viscountess Dudley and Ward in the 1820s. The enormous wealth of the Ward family derived from vast coal reserves on their land in Staffordshire as well as sugar plantations in Jamaica. Later residents of the house included a director of the East India Company, a Baron of the Exchequer, an MP, Walter Agnew of the famous Bond Street art dealers, and the third Baron Greville.
Notable neighbours included the Duke of Brunswick, and Earls Lathom, Waldegrave, Shaftesbury and Shrewsbury. Another was the pianoforte maker John Broadwood - at whose house Chopin gave a recital in 1837. In the 1920s Somerset Maugham and his interior decorator wife Syrie threw glamorous parties - guests often included Arnold Bennett, H.G. Wells, Virginia Woolf, Ivor Novello, Noel Coward and D.H. Lawrence who mingled with the eccentric Lord Berners, the Sitwells and the Guinnesses.