Reddit reviews Sets for Mathematics

Here is a link to John Baez's overview of what Topos theory is.

As /u/ziggurism has already said, you don't need to understand algebraic geometry to understand the theory of elementary toposes but some of the early motivating examples, the category of sheaves on a grothendieck site, are heavily steeped in the language of modern algebraic geometry.

A good, non algebro-geometric introduction to topos theory for those seeking to understand its place in logic is the book 'Topoi: The Categorial Analysis of Logic by Goldblatt. It also serves as a great introduction to the ideas of category theory.

Another fantastic book is Lawvere/Roserugh's Sets for Mathematics. This book seeks to explain the axioms of set theory using the language of category theory. It's not a book on arbitrary topos but it does serve to give you an idea of how topos theory axioms serve to build the logical system that every mathematician is familiar with, the logic in the category of sets. It's a good idea to have this 'concrete' application of topos axioms in Set under your belt before you tackling a book that seeks to explain how an arbitrary topos gives you a more abstract and unfamiliar logical system.

Edit: Also worth looking into is how topos theory can be used in the foundations of physics

If you're in computer science, Awodey is the category theory text for you. There's a lot of computer science and logic examples in there. You can also check out his lecture videos from the Oregon Programming Languages Summer School:
https://www.youtube.com/watch?v=ZKmodCApZwk

For a categorical perspective on set theory, check out Sets for Mathematics by Lawvere and Rosebrugh.

Sets for Mathematics