He had five children: two sons, Mark and Victor, with his first wife Jeanette Haighler; two sons, James (born 1946) and Edward (born 1947), with his second wife Elisabeth Van de Putte; and a daughter, Polly (born 1957), with his secretary Judy Hogg.
Did some training to be a vet in the early nineteen thirties but decided not to pursue this as a career.
Lived with Vogue journalist Barbara Baddeley from 1948-1963.