His step-grandmother Ursula Kuczynski (known as Ruth Werner) fought against the Nazis in in WW2 as a secret agent in the UK. The BBC called her "the most successful female spy in history".
His grandfather Paul was a denazification officer for the Allied Forces after WW2, confronting several leading figures of Nazi Germany.
In 2022, Silber married actress and writer Tracey Graves.
While working at Cicala Filmworks in New York, Silber shared an office with Michael Imperioli.