Reddit reviews Generalized Musical Intervals and Transformations

Look into transformational/Neo-Riemannian theory and musical set theory.

David Lewin - Generalized Musical Intervals and Transformations (GMIT)

Richard Cohn - Audacious Euphony

Guerino Mazzola - The Topos of Music

Guerino Mazzola - Cool Math for Hot Music

David Lewin - Klumpenhouwer Networks and Some Isographies that Involve Them (journal article)

Joseph Straus - Introduction to Post-Tonal Theory

Tymoczko is already spoken for.

Probably the most accessible mathematical approach to the basics of music theory that is still really solid scholarship would be Dimitri Tymoczko's A Geometry of Music. About the only downside is that Tymoczko has several bones to pick and he makes sure to pick them! http://www.amazon.com/Geometry-Music-Counterpoint-Extended-Practice/dp/0195336674

Another classic is David Lewin's Generalized Musical Intervals and Transformations. http://www.amazon.com/Generalized-Musical-Intervals-Transformations-David/dp/0199759944

If I were you, I'd start with Tymoczko and then move to Lewin after.

In the mid 20^th century, a fellow named Allen Forte successfully applied notions from mathematical set theory to "atonal" music and subsequently wrote an entire book on it: The Structure of Atonal Music.

This is a good introduction to a set theoretical approach to music theory, but it has been somewhat superseded by David Lewin's Generalized Musical Intervals and Transformations, an altogether more rigorous and detailed exposition of similar ideas, generalized to explain a wider variety of musical thought.

You also may enjoy exploring the writings and music of Iannis Xenakis. He applied ideas from probability and statistics to music theory and came up with several stochastic compositional methods. You can read more about these in his book Formalized Music.

There are probably a dozen other books that have come out in recent years applying all manner of advanced mathematics to music, from algebraic topology to group theory, but I haven't read any of them so I can't tell you if they're bullshit or not. Sometimes contemporary music theory comes off as literary criticism mixed with psychology, and I find it suspect, frankly.

It's like he skimmed David Lewin's book on group theory for musical analysis and misunderstood large chunks of it. Which, to be fair, is relatively easy to do given how Lewin writes out some of his mathematical statements...

That's not to say that Lewin's ideas aren't good or interesting, but his writing style seemed to me to be too 'unclear' for mathematicians and too confusing for musicians (unclear referring to how I recall him notating some mathematical concepts). And I certainly don't remember him drawing such a hamfisted connection between group and music theories.

Try Lewin's Generalized Musical Intervals and Transformations. This is not an easy-reading math book; the notation is dense and there are formal proofs. This is also not an easy-reading music theory book; I think it would be difficult to learn music theory at the appropriate level on-the-fly as you read Lewin, probably going to need to know some theory.