Sam Azouz was born to parents Rika (later, Rebecca) Zaraya and Isaac Azouz, Sephardic immigrants from Salonica, Greece. He was the fifth of six children (Solomon, Safira [Sophie], Gentille [Jean], Clara [Claire], Simuel [Sam], and Y'uda [Louis]). He was married to Sarah Horowitz, who passed away in 1991.