This book has been suggested a few times so I finally got around to reading it. I think it has some good information in it. I'm only about halfway through it, but I like it so far.

Time Management for System Administrators

Other books would be any of the social books like "How to influence people", "7 healthy habits..." Etc.

I haven't read this one yet, but It has been suggested to me if you plan to go more into management/leadership Start with Why

Other books that have I have ear marked due to being mentioned:

• The Devops Handbook
  
• The Practice of System and Network Administration
  
• The Practice of Cloud System Administration
  
• Cognitive Behavioral Therapy - This one might have been a joke, but as someone who has dealt with depression and burnout on the job. I think this could be beneficial.
  
• How to win friends and influence people
  
• How to win friends and influence people in the Digital Age
  
  Also, do a search for "Books for IT Professionals" to find a lot of other suggestions.

I would be very cautious before you start this. You need to have a much better understanding of why you’re doing this before you start. I think breaking up ops teams may be an answer to a different question. That question should be: “how do I deliver better business value?”

The first step is to understand what you’re trying to achieve. Gene Kim, Patric Debois et al talk about the “three ways”. It’s essentially three steps towards embracing devops culture.

The first is all about increasing flow through the system - that system being your organisation. The idea is to look at how your organisation goes from a business need to realising business value. For instance, how do we go from wanting to provide another payment option on a website, to customers being able to use it?

One way of analysing and visualising your organisation as system is something called “value stream mapping”. This looks at how a piece of work gets requirements, gets developed and how it gets to customers (even if that’s internal customers). You need to understand the process, where the delays are, where teams hand off from one to another, where things go wrong. Ideally you want to optimise this process. One of the issues that many organisations look at just automation and essentially automate a slow and inherently crap process. This will never give the returns that many people are after. Looking at this level, you should be looking at organisation goals. How do you measure this work in a frame that other people are going to understand who are not IT? Is it how fast you can get a feature to market? Increasing individual spend? Increasing reliability of the service you provide to customers? If you’re not framing this initiative in those terms, then it’s doomed to failure. Be specific and measurable.

Once you understand your process, you can look at opportunities to optimise the feedback loops (the second way). It might be that infrastructure is required by dev teams that gets delivered by ops but isn’t what they need. There is a team hand-off here that needs to be addressed. There are many solutions to this problem, but it might be a start to move where the person provisioning that infrastructure sits. Put them in the dev team. You might still keep them as part of the ops team logically to start out with. The point is, you’re looking at the system, understanding the constraints and trying to optimise pain points.

You can achieve a lot without adding any new automation or technology solutions and this shouldn’t be underestimated, but ultimately, handcrafting systems isn’t repeatable or fast enough. This is where reorganising teams might look sensible, but you should know what outcomes you are string to achieve. That might be difficulties provisioning infrastructure fast or flexibly enough, it might be deployment of code to live being too slow, it might be that testing is too slow. Once you know you need improve, you can look at tooling to better achieve that.

/u/quailtop mentions:

>In my (admittedly limited) experience, you can solve needing faster development velocity (the first problem) through staffing a new team whose job it is to help improve the deploy, test and release process for all developer teams. Their job is necessarily cross-cutting across all dev teams. They would develop internal tooling e.g. a standardised build/release process that all teams can employ. This is a great pattern because it avoids encroaching on existing territory and is a very clear contract between engineering.

This is otherwise known as the DevOps “hub and spoke model” and is what my organisation has implemented. It’s worked very well for us and it’s a clever way to start a reorg.

For certain ops teams, you may want to keep them together. For example, you may have a large and complicated network setup and still need a dedicated networks team. My focus then, would be putting an obligation on those teams to allow others to better consume their services. They may need to add other people to this team to make that happen. For example, if you have a complicated network, with lots of steps, look at both automation of those steps but also to allow other teams to more effectively consume them. Amazon spent a lot of effort building the culture that ever system or service is an API that should be able to be consumed easily internally (have a read here: https://apievangelist.com/2012/01/12/the-secret-to-amazons-success-internal-apis/ . So, for this case, you may add a tech lead and some software engineers to build network APIs rather than splitting the team up. This may include some of your more traditional network admins to look at replacing on prem infrastructure to support this. The goal however, should always be about the organisation level goal. Improving the speed at which you can reliably deploy network changes should be in support of one of those strategic objectives.

The third way then focuses more on continual learning and experimentation. You should have embedded a set of objectives that you’re working on achieving but you’ll have lots of legacy systems, legacy processes and behaviours. Focusing on outcomes and consistent asking of “why?” will start to help. This is also where SRE becomes really relevant for me. IMO SRE isn’t something that is particularly useful to start out with. It’s best when you’re looking at elevating and existing DevOps culture to a new level. This will look more at observability of a systems and understanding where the more difficult optimisations can be done.

Let me be clear. DevOps is a long road for any organisation to change to. To really get mature it will take many, many years to properly bed in. My organisation started around 5 years ago and we still have more progress to make. One of which is to move from project based way of organising work to more long lived product teams. This organisational change is probably the biggest thing holding us back right now, and has nothing to do with automation or technical practices.

I wouldn’t start out by reading about SRE, I would start with a book called The Phoenix Project and then read The DevOps Handbook, at least twice. Start with the strategy before you make any changes. I would also look to see if you have a cloud strategy because many of these practices are much harder to implement purely on premise.

​

Edit: Thanks for the silver!

Edit2: One thing that's also worth noting is that for many people, moving from traditional sysadmin to DevOps is a hugely scary change. It means that many of the staff won't have the job security they thought they had and they need crucial skills they don't have. To make this work, understand this point of view and support them. This requires really mature and experienced leadership at all levels. This is a good, short and free ebook to help the more traditional sysadmins understand why they have to change.

They’ve done just that in The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations by the same author as The Phoenix Project.

> give a vague impression that they all do the same thing.

Lots of tooling does overlap but each one has one area it excels at - some excel at the same area.

---

So, you have done a good job so far, it seems like most of your stuff is automated to a good degree and you have identified where your weaknesses are.

You should tackle one thing at a time, identify your largest bottle neck or problem and work to solve that first. In the same vain, only introduce one new tool at a time. Each takes some time to learn and to implement it correctly. Trying to do too much at once will just cause problems.

You have already identified the weaknesses so focus on solving these, starting with what you think is causing the most issues.

> - One server per environment is obviously not super scalable

Look into HA setups. How you do this and how much work it is depends on your application. Typically there are two parts to applications, work and state. Work (such as processing requests) is easy to scale if it contains no state. Just add another server to the environment and load balance between it. For this you need a loadbalancer (HAProxy or Nginx work well, though there are many others to chose from) and to move any state off the node you want to scale.

There are many forms of state, most will be stored in a database but you should also pay attention to session state which is sometimes stored in memory on the node - if you have anything like this you will need to do work to move it into some sort of storage, like a database or storage solution (such as your existing database or redis or memcached etc).

> - No sense of automatic provisioning, we do that "by hand" and write the IPs to a config file per environment

There are loads of tools to help with this.

Terraform for provisioning infrastructure.

Ansible or Chef or Saltstack or Puppet for provisioning nodes (I recommend starting with ansible, though any of them will work).

There is nothing wrong with using bash scripts to glue things together or even do provisioning while you learn to use these tools. I would not shy away from them, but do recognize the benefits each tool provides over just bash scripts. Take your time to learn them and stick with what you know and what works for you while you do. Introduce them a little bit at a time rather than trying to convert your entire infrastructure to use them in one go.

> - Small amounts of downtime per deploy, even if tests pass

This is easiest if you have a HA setup. You can do it without one but it involves just as much work and basically follows the same steps as creating a HA setup. In short, with multiple nodes you can upgrade them one at a time until everything has been upgraded. There are always some nodes running on either the old or new version so everything will continue to work.

You can either update nodes in place, or create new ones (if you have automated their provisioning) and delete the old ones when the new ones are up and working (see immutable infrastructure for this pattern, also canary deploys and blue/green deploys for different strategies).

> - If tests fail, manual intervention required (no rollback or anything) - though we do usually catch problems somewhere before production

Tests should be run before you deploy. These should run on a build server, or ideally a CI system. Ideally these should not only run before all deployments, but also for all commits to your code base. This way you can spot things failing much sooner and thus fix them when they are cheaper to fix. You also likely want to expand on the number of tests you do and what they cover (though this is always true).

Rollbacks should also be as easy as deploying the old version of the code. They should be no more complex than deploying any other version of your code.

> - Bash scripts to do all this get pretty hairy and stay that way

Nothing wrong with some bash scripts, work to keep them in order and replace them with better tooling as you learn/discover it.

---

I have mentioned a few tools here, but there are many more depending on exactly the problems you need to solve. Tackle each problem one at a time and do your research around the areas you have identified. Learn the tools you think will be helpful before you try to put them in production (ie do some small scale trails for them to see if they are fit for purpose). Then slowly roll them out to your infrastructure, using them to control more and more things as you gain confidence in them.

For everything you have said there is no one solution and as long as you incrementally improve things towards the goal you have you will be adding a lot of value to your business.

For now you need to decide on which is the biggest problem you face and focus your efforts on solving that - or at least making it less of a problem for now so you can focus on the next biggest problem. Quite often you will resolve the same problems in different, hopefully better, ways as you learn more and as your overall infrastructure, developmental practices and knowledge improves.

---

Also the 12 factor app is worth a read as is googles SRE book and the devops handbook. The Phenoix Project is also a good read.

Though these are more about the philosophy of DevOps, they are worth a read but wont solve your immediate issues. Reading around different topics is always a good idea, especially about what others have done to solve the problems you are facing. It will give you different perspectives and links to good tools you can use to solve the problems you face.

Excellent book for an introduction to what devops is all about. I worked at a devops consultancy for almost 3 years and we made everyone that joined read that book as well as the follow up, The Dev Ops handbook https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002

The devops handbook goes a little deeper into some of the technical ideas.

Read 'Continuous Delivery', 'The DevOps handbook', and perhaps 'Accelerate'.

I’d also mention a great follow up would be the DevOps Handbook: https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002

> is this a viable career path?

Yes. As systems/products become more complex, people skilled in keeping them reliable are highly coveted.

> What qualities/attributes are necessary for these roles that are different from developers?

Nothing in particular other than a general attitude of "getting shit done" and less "that's someone else's problem". DevOps as a culture strives to eliminate this "throw it over the wall to Dev/Ops, it's their problem" mentality.

> And if I’m interested, how do I learn more?

Google's SRE (not RE) resources are a good place for the fundamentals:
https://landing.google.com/sre/

The DevOps Handbook also includes some bits-and-pieces, not single monolithic chapters, regarding release planning/management.

I don't know of any good books specifically tied to release engineering, though it's a duty very firmly under the DevOps umbrella.

Okay, so my recommendation is similar to many others here, except that I'd say start with(get an audible subscription),

The Goal Audio Book by Eliyahu M. Goldratt

This is a really good book to start with, reasonably easy to listen to in audio format once you get rolling. It all started with this book! It's perfect start before you seque into

The Phoenix Project

It made for a pretty descent audio book as well. I powered through both of them while commuting. And I found it to be good enough.

DevOps HandBook

This however isn't quite as good as an audio book and you are better off with the book itself unless you are tight on time.

From there, this primer is pretty comprehensive to get you rolling,
[O'Reilly Learning Path: Modern DevOps] (https://www.safaribooksonline.com/library/view/learning-path-modern/9781787280236/) which will cover a little bit of every technology you might be interested in. I haven't gone through this myself but it seems to have descent coverage, from the 3 ways, to git, containers, docker, kubernetes, ci(jenkins), swarm, aws, puppet, salt, testing, agile, compliance, etc etc..

As everyone else has mentioned, 12 Factor is a required reading.

And if you want a pretty deep dive on Docker, Docker Mastery: The Complete Toolset From a Docker Captain

Is well maintained. If you want to know AWS better, there are some descent udemy courses as well that you can pickup for $15. Anything from Ryan Kroonenburg is pretty descent. Side note Azure just started offering a managed kubernetes service that is now in preview its worth checking out.

From there, the only other thing I would say is to look at Terraform and every product by hashicorp and some more in depth content for
kubernetes and possibly Powershell if you are a windows person.

I'm going to second that recommendation. DevOps is a really versatile role and you'll want to make sure that you have a solid understanding of the scope involved so that you can confidently set expectations when applying for positions.

It's become a bit of a buzzword in the last year, but for a good reason since it is pretty much essential for agile development and overlaps strongly with architecture / infrastructure development.

There are two books by the same team of authors I strongly recommend reading, including non-referral amazon links below.

• The Phoenix Project - Explains the approach in a narrative form. If you're only going to get one of these and you're new to the concept, I'd go with this.
  
  
• The DevOps Handbook - More abstract, but a really thorough and well organized examination of both DevOps as a discipline and the road to implementing it in an organization.

I'm a software engineer and not in DevOps. However, one of the managers at the company where I work encourages all the developers to read The Phoenix Project, and if/when they finish that she gives them a copy of The DevOps Handbook.

I'm about halfway through the former and haven't started the latter. The Phoenix Project is a novel, but it's kind of like one of those novels with a message, in this case the message is how to be part of a successful IT department at a modern company.

Tell your manager to read this fucking book: https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002

https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002

• Analyzed syslog streams from the SonicWall, planning out further automated reporting.
  
• Patched all servers and workstations to address Patch Tuesday vulnerabilities. Except for the one server we don't ever talk about.
  
• Increased reliability, RPO, and RTO on the Veeam offsite backups, while probably also driving down costs.
  
• Debugged an issue where a wireless Logitech mouse was intermittently controlling two computers randomly, because a staff member was moved to someone who somehow had a wireless Logitech keyboard and a wireless Microsoft mouse and was previously paired to that exact mouse from years earlier.
  
• Began reading The Cuckoo's Egg and The Devops Handbook. For the former, the book was made before I was born, but I find it scary that I still understand everything they're talking about and get nostalgic. On the topic of the latter, I've been really thinking about the whole idea of the System of Record vs. System of Engagement, as that was a bit eye-opening in terms of why doing certain things the proper way is not very enjoyable or streamlined.

The Phoenix Project and The DevOps Handbook by Gene Kim would be the ones I know of.

https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592

https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002

https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002

​

Explained best in this book - The DevOps Handbook.

I highly recommend The DevOps Handbook; I'm in the midst of reading it now, if you appreciated The Phoenix Project then this comes across super practical and to the point.

https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002

Here's your homework:

• https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592
  
  
• https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002/
  
  Then start looking into individual technologies. Docker, Drone, Kubernetes can work together on the same stack, for example. Start there, then figure out what's interchangeable.
  
  You'll need to establish awareness of the business needs and full stack context before DevOps will make proper sense.

This one.

https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002/ref=pd_sbs_14_img_0?_encoding=UTF8&psc=1&refRID=AACV133PGSYZ0D84739R

Have you tried the DevOps Handbook?
The Devops Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations https://www.amazon.co.uk/dp/1942788002/ref=cm_sw_r_cp_tai_WMs4ybAXM9TPX

These look great - I'm buying them asap.

Amazon recommends a third book as well: "Leading the Transformation: Applying Agile and DevOps Principles at Scale".

• https://www.amazon.ca/Phoenix-Project-DevOps-Helping-Business/dp/0988262509
  
• https://www.amazon.ca/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002
  
• https://www.amazon.ca/Leading-Transformation-Applying-DevOps-Principles/dp/1942788010

Senior Level Software Engineer Reading List

Read This First

1. Mastery: The Keys to Success and Long-Term Fulfillment
   
   

   Fundamentals
   

   
   

2. Patterns of Enterprise Application Architecture
   
3. Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions
   
4. Enterprise Patterns and MDA: Building Better Software with Archetype Patterns and UML
   
5. Systemantics: How Systems Work and Especially How They Fail
   
6. Rework
   
7. Writing Secure Code
   
8. Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .NET Libraries
   
   

   Development Theory
   

   
   

9. Growing Object-Oriented Software, Guided by Tests
   
10. Object-Oriented Analysis and Design with Applications
    
11. Introduction to Functional Programming
    
12. Design Concepts in Programming Languages
    
13. Code Reading: The Open Source Perspective
    
14. Modern Operating Systems
    
15. Extreme Programming Explained: Embrace Change
    
16. The Elements of Computing Systems: Building a Modern Computer from First Principles
    
17. Code: The Hidden Language of Computer Hardware and Software
    
    

    Philosophy of Programming
    

    
    

18. Making Software: What Really Works, and Why We Believe It
    
19. Beautiful Code: Leading Programmers Explain How They Think
    
20. The Elements of Programming Style
    
21. A Discipline of Programming
    
22. The Practice of Programming
    
23. Computer Systems: A Programmer's Perspective
    
24. Object Thinking
    
25. How to Solve It by Computer
    
26. 97 Things Every Programmer Should Know: Collective Wisdom from the Experts
    
    

    Mentality
    

    
    

27. Hackers and Painters: Big Ideas from the Computer Age
    
28. The Intentional Stance
    
29. Things That Make Us Smart: Defending Human Attributes In The Age Of The Machine
    
30. The Back of the Napkin: Solving Problems and Selling Ideas with Pictures
    
31. The Timeless Way of Building
    
32. The Soul Of A New Machine
    
33. WIZARDRY COMPILED
    
34. YOUTH
    
35. Understanding Comics: The Invisible Art
    
    

    Software Engineering Skill Sets
    

    
    

36. Software Tools
    
37. UML Distilled: A Brief Guide to the Standard Object Modeling Language
    
38. Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and Iterative Development
    
39. Practical Parallel Programming
    
40. Past, Present, Parallel: A Survey of Available Parallel Computer Systems
    
41. Mastering Regular Expressions
    
42. Compilers: Principles, Techniques, and Tools
    
43. Computer Graphics: Principles and Practice in C
    
44. Michael Abrash's Graphics Programming Black Book
    
45. The Art of Deception: Controlling the Human Element of Security
    
46. SOA in Practice: The Art of Distributed System Design
    
47. Data Mining: Practical Machine Learning Tools and Techniques
    
48. Data Crunching: Solve Everyday Problems Using Java, Python, and more.
    
    

    Design
    

    
    

49. The Psychology Of Everyday Things
    
50. About Face 3: The Essentials of Interaction Design
    
51. Design for Hackers: Reverse Engineering Beauty
    
52. The Non-Designer's Design Book
    
    

    History
    

    
    

53. Micro-ISV: From Vision to Reality
    
54. Death March
    
55. Showstopper! the Breakneck Race to Create Windows NT and the Next Generation at Microsoft
    
56. The PayPal Wars: Battles with eBay, the Media, the Mafia, and the Rest of Planet Earth
    
57. The Business of Software: What Every Manager, Programmer, and Entrepreneur Must Know to Thrive and Survive in Good Times and Bad
    
58. In the Beginning...was the Command Line
    
    

    Specialist Skills
    

    
    

59. The Art of UNIX Programming
    
60. Advanced Programming in the UNIX Environment
    
61. Programming Windows
    
62. Cocoa Programming for Mac OS X
    
63. Starting Forth: An Introduction to the Forth Language and Operating System for Beginners and Professionals
    
64. lex & yacc
    
65. The TCP/IP Guide: A Comprehensive, Illustrated Internet Protocols Reference
    
66. C Programming Language
    
67. No Bugs!: Delivering Error Free Code in C and C++
    
68. Modern C++ Design: Generic Programming and Design Patterns Applied
    
69. Agile Principles, Patterns, and Practices in C#
    
70. Pragmatic Unit Testing in C# with NUnit
    
    

    DevOps Reading List
    

    
    

71. Time Management for System Administrators: Stop Working Late and Start Working Smart
    
72. The Practice of Cloud System Administration: DevOps and SRE Practices for Web Services
    
73. The Practice of System and Network Administration: DevOps and other Best Practices for Enterprise IT
    
74. Effective DevOps: Building a Culture of Collaboration, Affinity, and Tooling at Scale
    
75. DevOps: A Software Architect's Perspective
    
76. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations
    
77. Site Reliability Engineering: How Google Runs Production Systems
    
78. Cloud Native Java: Designing Resilient Systems with Spring Boot, Spring Cloud, and Cloud Foundry
    
79. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
    
80. Migrating Large-Scale Services to the Cloud

The DevOps Handbook, Team Geek and Debugging Teams come to mind.

I don't think there is something specifically for "devops managers" (what is that even?). General leadership books would work for you as a manager I suppose. Just keep the principles of DevOps in mind when you do manage away.

The DevOps Handbook has some good stuff on these topics, in addition to what others here have said.

https://smile.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002/ref=sr_1_1

It seems that you want to cover DevOps to some degree. I'd look at the DevOps handbook https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002/ref=sr_1_10?keywords=DevSecOps+Handbook&qid=1554391709&s=gateway&sr=8-10

, along with covering the technologies that @wsppan touched on.

https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592/

Very good read, also recommend the follow up DevOps Handbook:

https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002/

Haha guess I saw the "tipsy" and checked out after that! My focus is functional programming, so most of my recommendations are around that.

LambdaCast and The REPL are good and worth listening through (full disclosure I was on the REPL).

Other casts that I cherry-pick through:

• Software Engineering Daily
  
• Future of Coding
  
• The Accidental Engineer
  
  Some good books:
  
  
• The Phoenix Project
  
• The DevOps Handbook
  
• Release It
  
  You should also check out hillelogram and his talks/podcasts/writing on TLA+ if you're not familiar.
  
  Do you have any good recommendations?

Read this immediately.
https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002

I'd second all these and add

• https://www.amazon.com/Sprint-Solve-Problems-Test-Ideas/dp/150112174X/ref=sr_1_1?ie=UTF8&qid=1499489674&sr=8-1&keywords=sprint
  
  
• https://www.amazon.com/Lean-Startup-Entrepreneurs-Continuous-Innovation/dp/0307887898/ref=pd_sim_14_8?_encoding=UTF8&pd_rd_i=0307887898&pd_rd_r=T1PBJEZ7FD6W6AT5Z98H&pd_rd_w=Y6d2i&pd_rd_wg=bgZJB&psc=1&refRID=T1PBJEZ7FD6W6AT5Z98H
  
  
• https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002/ref=sr_1_1?s=books&ie=UTF8&qid=1499489786&sr=1-1&keywords=devops
  
  
• https://www.amazon.com/Soft-Skills-Software-Developers-Manual/dp/B0158SJ3EM/ref=sr_1_1?s=digital-text&ie=UTF8&qid=1499490040&sr=1-1&keywords=john+sonmez

This is the Holy Bible of DevOps. Well, it's the Old Testament. The New Testament is https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002

Love the downvotes without comments... the assertion above is taken almost verbatum from a talk I went to where Gene Kim was presenting a couple of years ago.