Berman was born in Chicago, Illinois, the son of Irene and Nathan Berman.
Berman started as a straight actor, receiving his training at the Goodman Theatre in Chicago, honing his acting skills in stock companies in and around Chicago and New York. In the mid-1950s, he became a member of Chicago's Compass Players, which later evolved into The Second City. While performing improvised sketches with Compass, Berman began developing solo pieces, often employing an imaginary telephone to take the place of an onstage partner.
In 1957, Berman landed his first job as a comedian at Mister Kelly's in Chicago, which led to other nightclub bookings, and a recording contract with Verve Records. His comedy albums would earn him three gold records and he'd win the first Grammy Award for a non-musical recording. He was the first stand-up comedian to play Carnegie Hall. Berman would go on to appear on numerous TV specials, and all of the major variety shows of the day.
Berman's success as a comedian enabled him to continue with his first love, acting. He starred on Broadway in A Family Affair and continued to do stage work in productions of The Odd Couple, Damn Yankees, Fiddler on the Roof, I'm Not Rappaport, La Cage aux Folles, and Guys and Dolls, among many others. Comedic and dramatic acting roles on television began to come his way, including appearances on episodes of The Twilight Zone, Bewitched, Peter Gunn, The Mary Tyler Moore Show, Adam-12, Emergency!, Night Court, MacGyver, L.A. Law, Friends, Walker, Texas Ranger, The King of Queens, Grey's Anatomy, Hannah Montana, CSI: NY, and Boston Legal. Since 2002, Berman has appeared as Larry David's father on Curb Your Enthusiasm, a role for which he received a 2008 Emmy Award nomination.
Among Berman's film credits are The Best Man (with Henry Fonda), Divorce American Style (with Dick Van Dyke and Debbie Reynolds), Every Home Should Have One (with Marty Feldman), The Last Producer (with Burt Reynolds), Meet the Fockers (with Robert De Niro and Ben Stiller), The Holiday (with Cameron Diaz), and You Don't Mess with the Zohan (with Adam Sandler).
Berman has been married to Sarah Herman since April 19, 1947. The two met while they were studying acting at Chicago's Goodman Theatre.
In the mid-1960s, Berman and wife Sarah adopted two children, son Joshua and daughter Rachel. The Bermans were planning Joshua's bar mitzvah when he was diagnosed with a brain tumour. Joshua died October 29, 1977, age 12.
Berman has authored three books, two plays, several TV pilot scripts, and numerous poems. For over twenty years, Berman taught humour writing in the Master of Professional Writing programme at University of Southern California, where he is now a lecturer Emeritus. He continues to do film and television work, and make personal appearances across the country year round.