In jazz, he found "the rhythmic drive, the risk, the absolute contrast to the pale, academic approach I had been taught." He also took up playing the baritone saxophone.

　　Gulda wrote a Prelude and Fugue with a theme suggesting swing. Keith Emerson performed it on Emerson, Lake & Palmer's The Return of the Manticore. In addition, Gulda composed "Variations on The Doors' 'Light My Fire'". Another version can be found on Gulda's album As You Like It (1970), an album with standards such as "'Round Midnight" and "What Is This Thing Called Love?". In 1980, he wrote his Concerto for Cello and Wind Orchestra, which has been called "as moving as it is lighthearted", in five movements "involving jazz, a minuet, rock, a smidgen of polka, a march and a cadenza with two spots where a star cellist must improvise."

　　In 1982, Gulda teamed up with jazz pianist Chick Corea, who was between the breakup of Return to Forever and the formation of his Elektric Band. Issued on The Meeting (Philips, 1984), Gulda and Corea communicate in lengthy improvisations mixing jazz ("Some Day My Prince Will Come" and the lesser known Miles Davis song "Put Your Foot Out") and classical music (Brahms' "Wiegenlied" ["Cradle song"]). In the late 1990s, Gulda organised rave parties, where he performed with the support of several DJs and Go-Go dancers.

　　These unorthodox practices along with his refusal to follow clothing conventions (he was notoriously described as resembling, in one South German concert, "a Serbian pimp") or announce the program of his concerts in advance earned him the nickname "terrorist pianist". In 1988, he cancelled a performance after officials of the Salzburg Festival objected to his including jazz musician Joe Zawinul on the program. When the Vienna Music Academy awarded him its Beethoven Ring in recognition of his performances, he accepted it but then reconsidered and returned it. To promote a concert in 1999, he announced his own death in a press release so that the concert at the Vienna Konzerthaus could serve as a resurrection party.

　　Phillips Records included Gulda in its Great Pianists of the 20th Century CD box set, which came out in 1999. His piano students included Martha Argerich, who called Gulda "my most important influence," and the conductor Claudio Abbado.