Gary Koh asked me on another thread where the topic came up to start a thread on (European) classical avantgarde. Here it is (it is not limited to Europe, for example one CD recommendation below is by an American composer).
Background
After WW II, a group of composers, among them Boulez and Stockhausen, wanted to start with composing from scratch, instead of building on established traditions. Initially they started with ideas like 'total serialism' where not just pitch is treated in a series like in 12-tone music, but other parameters as well, such as duration and dynamics. Based on such ideas, certain 'gestural languages' (see below) were developed. Rhythm is often irregular. These gestural languages in turn became the basis for, and were augmented or transformed by, other methods of composition unrelated to serialism (Xenakis for example was never a serialist), or serialism was only loosely applied as an abstract concept and became what could be called 'serial thinking'. As composer Richard Barrett puts it, "it’s not a question of relating everything to a “series” but of relating everything [all musical parameters] to everything else". (Explained in more detail here: http://richardbarrettmusic.com/Stockhausen2012.html) Stockhausen for example has applied such 'serial thinking' to all his compositions including those without any pitch series, but still has based a substantial portion of his output on pitch series nonetheless.
One thing that should be kept in mind is that most great composers of classical avantgarde (past and present) were and are not just wild boys who wanted to be different solely because they did not know enough about how to write 'normal' music. While mostly eschewing 'traditional' forms of musical expression such as themes or melodies (at least in the common sense of 'singing' melody), they were and are well trained in 'traditional' composition techniques and have studied them. For example, Boulez could not be a famous conductor (an occupation that has limited his compositional output in the last few decades) if he were not well versed in classical music theory, and Stockhausen has written scholarly works on Mozart and Bartok and has shown himself to be an able Mozart conductor in the two works he conducted, flute concerto in G and the clarinet concerto. Also, Stockhausen has composed marvelous youth works in a more traditional style (e.g., the a cappella 'Choirs for Doris', with Doris being his later wife) and returned in some of his later music to a more melodic writing, in which he proved himself, at least to my ears, as an incredible melodist (while most of the time, yet not always, his melodies are still more abstract than common ones). Electronic musicians generally admire Stockhausen as a pioneer and a composer of electronic music (he also has been called the grandfather of Techno), but one reason why few electronic musicians can emulate the quality of Stockhausen's electronic music is that they mostly do not have the impressive 'tradtional' compositional skills that Stockhausen has.
What not to listen for in classical avantgarde
As mentioned, there are hardly any themes or melodies, at least not in the common sense. Also, don't try to find a 'series' -- your search will mostly be in vain. If a work is based on a pitch series, it is mostly an abstract relationship that is not readily detected by ear, nor is it composed to be detected as such, but rather is just a compositional template. In fact, since the music as it sounds is not about that, in general it would pervert the composer's intention to search for "the series". And that is, if there even is any; as mentioned, there is also a lot non-serialist avantgarde, and most more recent avantgarde is of that variety as well. And in more melodic works, like some of later Stockhausen, the series mostly just forms the basis of diverse melodic gestures, but is not the melody itself.
What to listen for in classical avantgarde
Listen to gestures, such as characteristic brief pitch and rhythmic motions, or patterns of such motions, which may recur in constantly varied forms. Listen to the motion of the music in general. Mostly there is no typically flowing motion like in music with regular rhythms, but there can be gripping motion nonetheless, irregular and fractured as it may be. That fractured motion often plays out in a constant game of tension between subsequent and/or simultaneous musical elements. In fact, once you get the hang of it, avantgarde music with its abstract, purely gestural language may actually be easier to listen to than 'classical' forms, where it is often desirable to attentively follow the thematic development in order to experience the continuity and logic of musical flow, especially if the listener wants to consciously experience the music as a conceptual whole rather than just a nice collection of beautiful moments and sounds washing over him/her.
Also, listen to just the sound. Unlike 'classical' music where sound is rarely a means to itself, but serves melodic or thematic expression and development, avantgarde music is often composed as mere sound, or gestures in sound. In fact, some of the early avantgarde composers (e.g, Stockhausen, Xenakis) became pioneers in electronic music in part precisely because they became interested in just sculpting sound from the inside out, and in developing new sounds. But sound as such is also important in a lot of avantgarde music for acoustic instruments.
Background
After WW II, a group of composers, among them Boulez and Stockhausen, wanted to start with composing from scratch, instead of building on established traditions. Initially they started with ideas like 'total serialism' where not just pitch is treated in a series like in 12-tone music, but other parameters as well, such as duration and dynamics. Based on such ideas, certain 'gestural languages' (see below) were developed. Rhythm is often irregular. These gestural languages in turn became the basis for, and were augmented or transformed by, other methods of composition unrelated to serialism (Xenakis for example was never a serialist), or serialism was only loosely applied as an abstract concept and became what could be called 'serial thinking'. As composer Richard Barrett puts it, "it’s not a question of relating everything to a “series” but of relating everything [all musical parameters] to everything else". (Explained in more detail here: http://richardbarrettmusic.com/Stockhausen2012.html) Stockhausen for example has applied such 'serial thinking' to all his compositions including those without any pitch series, but still has based a substantial portion of his output on pitch series nonetheless.
One thing that should be kept in mind is that most great composers of classical avantgarde (past and present) were and are not just wild boys who wanted to be different solely because they did not know enough about how to write 'normal' music. While mostly eschewing 'traditional' forms of musical expression such as themes or melodies (at least in the common sense of 'singing' melody), they were and are well trained in 'traditional' composition techniques and have studied them. For example, Boulez could not be a famous conductor (an occupation that has limited his compositional output in the last few decades) if he were not well versed in classical music theory, and Stockhausen has written scholarly works on Mozart and Bartok and has shown himself to be an able Mozart conductor in the two works he conducted, flute concerto in G and the clarinet concerto. Also, Stockhausen has composed marvelous youth works in a more traditional style (e.g., the a cappella 'Choirs for Doris', with Doris being his later wife) and returned in some of his later music to a more melodic writing, in which he proved himself, at least to my ears, as an incredible melodist (while most of the time, yet not always, his melodies are still more abstract than common ones). Electronic musicians generally admire Stockhausen as a pioneer and a composer of electronic music (he also has been called the grandfather of Techno), but one reason why few electronic musicians can emulate the quality of Stockhausen's electronic music is that they mostly do not have the impressive 'tradtional' compositional skills that Stockhausen has.
What not to listen for in classical avantgarde
As mentioned, there are hardly any themes or melodies, at least not in the common sense. Also, don't try to find a 'series' -- your search will mostly be in vain. If a work is based on a pitch series, it is mostly an abstract relationship that is not readily detected by ear, nor is it composed to be detected as such, but rather is just a compositional template. In fact, since the music as it sounds is not about that, in general it would pervert the composer's intention to search for "the series". And that is, if there even is any; as mentioned, there is also a lot non-serialist avantgarde, and most more recent avantgarde is of that variety as well. And in more melodic works, like some of later Stockhausen, the series mostly just forms the basis of diverse melodic gestures, but is not the melody itself.
What to listen for in classical avantgarde
Listen to gestures, such as characteristic brief pitch and rhythmic motions, or patterns of such motions, which may recur in constantly varied forms. Listen to the motion of the music in general. Mostly there is no typically flowing motion like in music with regular rhythms, but there can be gripping motion nonetheless, irregular and fractured as it may be. That fractured motion often plays out in a constant game of tension between subsequent and/or simultaneous musical elements. In fact, once you get the hang of it, avantgarde music with its abstract, purely gestural language may actually be easier to listen to than 'classical' forms, where it is often desirable to attentively follow the thematic development in order to experience the continuity and logic of musical flow, especially if the listener wants to consciously experience the music as a conceptual whole rather than just a nice collection of beautiful moments and sounds washing over him/her.
Also, listen to just the sound. Unlike 'classical' music where sound is rarely a means to itself, but serves melodic or thematic expression and development, avantgarde music is often composed as mere sound, or gestures in sound. In fact, some of the early avantgarde composers (e.g, Stockhausen, Xenakis) became pioneers in electronic music in part precisely because they became interested in just sculpting sound from the inside out, and in developing new sounds. But sound as such is also important in a lot of avantgarde music for acoustic instruments.
