Suppose you — or another good jazz pianist — walked into a song they never heard before, just as the first solo began. Would this pianist be able to take a solo, knowing only what the first solo sounded like and the chord changes? Or would it be important to know the original melody?

If you or another good pianist sat in with a house band, and they struck up a tune you knew, but had never played, would your solo be

— entirely invented?

— comprised of riffs you have done many times before, but perhaps combined in innovative ways?

— mainly drawn from other solos of sequences you had practiced extensively?

I guess the heart of this question has to do with how much you are making up a new tune on the spot, and how much you are working with tools and building blocks that are familiar.

The melody consists an extremely simple eight-note phrase which is repeated three times over the 12-bar progression. It’s played one time through by Duke on the piano, then once by the band, after which members of the orchestra take a series of solos over the song’s chord structure. (Actually you might notice that each solo begins with four bars of unaccompanied soloing, before the 12-bar structure of the song repeats). On the fifth solo (Barney Bigard on clarinet), the orchestra begins playing an accompaniment for the solo, which swells into a variation of the main melody for the final cycle through the changes.

The task while listening to this, for someone who’s trying to learn how to improvise, is to figure out what the soloists are doing as they improvise. And the most important thing here, I think, is that they are creating their solos based on the harmonic structure of the song. That’s one of the reasons I’ve chosen this particular piece. In general, the solos you hear here are in keeping with the style and the time. They’re “inside” the chord changes (i.e., they’re not atonal) and they tend toward a more melodic style. Nonetheless, these guys aren’t simply repeating their scales and arpeggios. There are a lot of “blue” notes – notes that aren’t exactly part of the harmonic structure at that moment but that sound wrong in a satisfying way (listen for instance to the note that Rex Stewart lands on and plays twice at 1:04-05). So the question remains: what are they actually doing? What’s the intention that drives the notes they choose to play?

There are several ways to conceptualizes this. One is to think of the soloists creating their own melody over the chord structure. Some musicians are particularly melodic improvisers – Keith Jarrett often plays that way on the piano. (See the three clips below for examples of each of the styles I will describe here.) On the other end of the spectrum is what I would call more “structural” improvising, in which, rather than creating something that sounds melodic, the performer is creating interesting musical patterns using the harmonic structure of the song. That’s the direction Charlie Parker moved in with the development of be-bop. You can often hear the shreds of melodic lines within his solos, but the overall impression those phrases leave (to my ear at least) is not especially melodic – he’s creating extended linear structures, often using little more than scales related to chords of the song (although he does it so fast that it’s hard to take it in). Taking it a step further, John Coltrane (especially in his later years) was extremely “structural” in how he improvised, but in a more abstract sort of way that could depart further and further from the song’s underlying harmonic structure.

So, to make sense of improvisation, you have to try to put yourself into the mind of the great improvisers. A way of listening that I have found that helps me to do this is to listen to a solo as if I were playing it myself. When there’s a pause in the solo, I ask myself what I would play next. Then I listen to what is actually played. I ask myself: what was the state of mind of the performer that led him in that direction rather than in mine?

This is to say that learning to improvise requires a bit of cognitive reverse-engineering. In fact, one of the things that helped me most when I was learning to improvise was analyzing the improvisation of musicians I admired. My high school jazz band teacher made transcribing solos a regular assignment. It’s a fascinating exercise which is not unlike what I experienced later in my professional life as a psychotherapist when I learned to analyze videotapes of parent-child interactions. You take an experience which is the essence of present moment spontaneity and pin it down in time and analyze it. It is always extraordinary to me to realize how much complexity is contained within a single moment of a jazz performance.

So, getting around to my friend’s first question: what would happen if I walked into a song just as the solos began? For me, that question depends largely upon whether or not there was a fake sheet that I could look at. If there were, then I’d be fine. The original melody would actually be completely irrelevant to me. All I’d need to know would be the key changes. But if I didn’t have a chart and the key changes were complex, I’d be in trouble. While I’m sure that other musicians wouldn’t have so much of a problem with this, I learned my lesson the hard way years ago when I tried to sit in at an open jam session in New York. My turn to play came and the group started in on some song that I didn’t know and couldn’t quickly pick up. After a few moments I was tapped on the shoulder and someone who knew the song stepped in.

That was harsh. But the important thing is that it’s a matter of knowing the chord changes. If I had known them, I would have been okay.

My friend’s second question goes to the deeper nature of improvisation itself, which is in many ways a function of the style and personality of the individual musician. When I was younger I spent hours and hours practicing scales and arpeggios so that my fingers would be able to fluidly play those patterns. What that meant was that even if I weren’t feeling particularly inspired, I could play a passable solo on most any song just using the patterns that I had trained my fingers to play.

Not that I was interested in doing this. Looking back, I can see that I was interested in jazz as a sort of means to an end. Although I didn’t always have the concept of “mindfulness” back then, I believe that playing music has always been essentially a form of mindfulness meditation for me. Usually when I play I am simultaneously performing and observed my own attentional state as I play. Because of this, over time I came to understand more deeply the distinction I outlined earlier, between what I call melodic and structural improvisation. There’s a fundamental difference of awareness and intention between the two.

It’s possible to create a solo based on the melodic pattern you hear in your mind…to use a sort of “melodic intention” as your guide. That’s what I hear for instance when I listen to Keith Jarrett improvise (although I have no idea whether he himself would describe it in these terms). On the other hand, It’s also possible to direct your creative intention in a more abstract and structural manner, to focus your mental state on the creation of sonic patterns within the harmonic structure of the song. That becomes more of a mathematical exercise, which can have a beauty all it’s own. And that’s what I hear in Coltrane’s playing.

There are undoubtedly other ways to conceptualize improvisation, but this is how my mind breaks it down. And what’s important here is that I’m talking not so much about musical technique but rather about a musician’s state of mind with regard to improvisation. The fact is, it’s very difficult to improvise without mindfulness. It’s an activity which forces your brain into a heightened state of awareness of the present moment. That’s why it’s so difficult to describe in words: it’s a subjective aesthetic experience. There’s no easy way to measure or quantify or even to describe such a thing.

I believe that this is why there is often an edge of emotional struggle in both the music and the lives of great jazz musicians. Because while this sort of mindfulness creates the potential for deep joy, it also requires a performer to open to the truth of whatever fears and anxiety and grief he or she may hold within them. When you make yourself fully mindful of yourself in the present moment, you don’t get to pick which aspects of your experience you get to be mindful of…it’s all there.

That said, I do think that certain jazz musicians manage to play it safe. I won’t name names. But to my mind, no matter how technically accomplished such performers are, listening to them is an aggravating experience. All I can hear is the missed opportunity. What I love about playing jazz is the sense of infinite possibility I experience every single time I start to play a solo. There is, every time, a vivid and heightened sensation of delight and terror…the awareness that I have the opportunity to create something utterly new. As a listener, therefore, I’m drawn to performers who exhibit novelty and complexity, to the quality of deep and authentic self-revelation in the moment of performance. I love the raw and powerful present-moment energy that band leaders like Charles Mingus and Miles Davis were able to draw from their musicians. I love the weird power of Sun Ra and Carla Bley. And lately I’ve been particularly mesmerized by pianist Vijay Iyer. His compositions and solos are so clearly structured, yet so densely complex that often I can barely figure out what he’s doing.

It’s utterly exhilarating to open yourself to something that is goes beyond your capacity for comprehension.