Jimmy Cavallo was born in Syracuse, New York in 1927 and during high school played alto sax in a swing band, and then formed his own band, playing tenor sax and singing. During his time in the Navy during World War II, he often hung around clubs with his saxophone, and jammed with a lot of the performers.