Harold Budd: the composer with no urge to make music

It's a strange thing for anyone to say, but from a renowned composer it's especially baffling. "I'm not much of a music fan," says Harold Budd towards the end of a warm, engaging if occasionally mystifying conversation. "I just don't listen to music – at all!" Even more surprising, perhaps, is the fact that the 77-year-old doesn't even own his favoured instrument: a piano. "I think they're ugly things," he chuckles. "Architecturally speaking, and in other ways. So to actually live with a piano? Well, that would really insult my aesthetic sense."

There are other ways in which the American composer, over a career spanning several decades, has trodden his own path. His early compositions embraced 1960s minimalism, yet he soon felt so trapped by the movement's conceptualism it caused him to (temporarily) retire. He helped pioneer ambient music with Brian Eno in the early 1980s, yet has little regard for the tag – or tags in general. "I just have utterly no interest in that sort of thing," he sighs. And despite his highbrow credentials (Budd has composed for string quartets, choirs and even penned an extended gong solo), he has devoted much of his career to collaborating with British pop stars, from Cocteau Twins to Jah Wobble.

To confuse things further, Budd cites not music but visual art (in particular, the abstract expressionists Mark Tobey and Jackson Pollock) as his chief inspiration, although the notion that he's somehow immune to the thrill of music isn't true. As a teenager, he fell in love with the electrifying sound of bebop and went on to play drums for saxophonist Albert Ayler's band while serving in the army. "I wanted to be the world's greatest jazz drummer," he says. "And I failed at that!"

Budd hadn't wanted to join the army. "I was drafted. Plucked from life, which is what happens when you're very, very poor." Although the prospect of army life terrified him – "I felt for sure I would die" – it turned out to be a godsend. "I realised that if I didn't go out and get an education I would remain a dumb person all my life," he says. "It was also the first time that I was introduced to new places to live, different aspects of society. At the time, I lived in a black ghetto in LA and black culture was really the only outside culture I had ever confronted. I liked it very much, but I'm not black so I knew I would never fit in."

So Budd studied architecture then music – much to the bafflement of his family. Hearing John Cage deliver his lecture Where Are We Going And What Are We Doing? opened his ears to a new way of thinking and, in 1970, he recorded The Oak Of The Golden Dreams, a minimalist drone piece that arose from his experimentations with a Buchla synthesiser. Yet Budd found himself resenting the "academic pyrotechnics" at the heart of the avant-garde community and gave up composing altogether, before deciding to channel his anger into a reaction against everything he had previously learned.

In 1972, he wrote Madrigals of the Rose Angel which, with its harps, chimes and female choir, favoured surface beauty and emotional pull over theoretical framework. It caught the attention of Eno, who eventually recorded it, as part of The Pavilion of Dreams, Budd's landmark 1978 album. "The Pavilion of Dreams erased my past," says Budd. "I consider that to be the birth of myself as a serious artist. It was like my magna carta."

Budd's inspiration was once again visual art, in particular the work of the Pre-Raphaelites such as Dante Gabriel Rossetti. "I have to admit to you that part of that was political," he says, with a hint of mischief. "Because it was a regard for a kind of art that was completely disengaged from all serious art at the time."

Working with Eno changed Budd's life. "I owe Brian everything," he says, "But the primary thing was attitude. Absolute bravery to go in any direction. I once read an essay by the painter Robert Motherwell and he pointed out a truth that is so obvious and simple that it's overlooked: 'Art without risk is not art.' I agree with that profoundly. Take a flyer – and if it fails don't let it crush you. It's just a failure. Who cares?"

Eno and Budd were soon hailed as the godfathers of ambient thanks to collaborations such as 1980's Ambient 2: The Plateaux of Mirror and 1984's The Pearl. Budd embarked on collaborations with other British artists: John Foxx, Andy Partridge, David Sylvian. "I couldn't get arrested in America," he says. "But as soon as I landed in Britain, I was taken seriously as an artist. What a change from just a few hours earlier!"

His work with the "wonderful" Cocteau Twins was a highlight: together they made the mesmerising 1986 album, The Moon and the Melodies, while his more recent collaborations with the group's Robin Guthrie have produced woozy instrumental magic such as How Distant Your Heart. Budd praises singer-songwriter Liz Fraser for her way with a title, something he's also adept at: take Flowered Knife Shadows, or his new double-CD retrospective, Wind In Lonely Fences, which allows you to enjoy his back catalogue before you've even listened to it.

Budd, though full of admiration for all of his British collaborators, struggles when it comes to describing his own music. "I'm not very good at self-analysis," he says. "I should explore my own feelings more." In fact, throughout the interview, he apologises for not knowing how to answer certain questions. When I wonder if the lack of instruments in his house in South Pasadena, California, is due to an appreciation of silence, he says that it never crossed his mind before. Asked what he does when the sudden urge to play hits him, something you imagine must strike all musicians, he stumbles: "But I ... but I don't get the urge to play!"

He does, however, tell me something that reveals much about his unique approach to music. In 2012, he released Bandits of Stature, an album of cinematic works for string quartet. As he tells me about the recording sessions, his voice starts to waver. "My wife, who was also the mother of my 13-year-old son, passed away a year and a half ago," he says. "I wrote a lot of that music when she was still alive. And so it was extremely painful to revisit that place."

Last year, Budd completed a year-long project writing music for videos made by his friend the artist Jane Maru, in which he'd enter the studio with no preparation – no notes or even ideas – and record whatever he came up with that day. "Whatever I did was mixed and pressed, without it ever having to be revisited," he says, his voice brightening. "I had to use every ounce of everything I had. And I loved the process, oh I loved it! It was a definite reacquaintance with my old self, which I hadn't realised I missed so much. Oh my god, was I grateful!"

Such enthusiasm seems at odds with the idea of Budd not being a music fan, but it does make a sort of sense. I suggest that this may be what sets him apart from the average listener: that he feels music more intensely, so can't put himself through such highs and lows each day. "I never really thought about that," he says, chuckling again, as if baffled by the sheer mystery of himself. "But you know, it's a very good point. I think I'm going to take that as gospel."

Harold Budd's Wind In Lonely Fences 1970-2011 is out now.