Containers Will Not Fix Your Broken Culture (and Other Hard Truths)

1. Tech is Not a Panacea

• But let’s be real: we’re unlikely to solve the vast majority of problems in a given organization via the judicious application of kernel features. If you have contention between your ops team and your dev team(s)–and maybe they’re all facing off with some ill-considered DevOps silo inexplicably stuck between them–then cgroups and namespaces won’t have a prayer of solving that.
• Proposal: rename ‘staging’ to ‘theory’. “It works in theory, not on production.” – Najaf Ali
• There is no substitute for experimentation in your real production environment; containers are orthogonal to that, while cross-org communication is crucial to clarity of both purpose and intent. Observability being key is a fundamental tenet of the gospel according to Charity Majors. The conflicts inherent in misaligned incentives continue to manifest no matter where the lines of responsibilities are drawn. Andrew Clay Shafer calls the state of any running system “continuous partial failure;” good tooling is necessary (but not sufficient) to operate a robust fault-tolerant system.

2. Good Team Interactions: Build, Because You Can’t Buy

• Adrian Cockcroft points out that a monolith has as much complexity as microservices; it’s just hidden. Okay so we’re going to deconstruct that dreaded monolith and keep on rockin' in the microservices world! That will solve everything! Clean abstractions and well-defined handoffs sound great until you realize that you’re moving the consequences of decisions (and the conflict inherent in any set of trade-offs) into another part of your stack, which Tim Gross calls “conservation of complexity”.
• Breaking into individuated teams doesn’t change the fact that the teams have to agree where the boundaries lie at any given moment. Writing in the 1960s, Mel Conway could have been talking about today.
• Conway wrote that any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure. This came to be known as Conway’s Law.
• Creative problem solvers have a way of routing around process that we find inconvenient. If your heavyweight change control process applies except in case of emergencies, then (spoiler alert) you’re going to see a surprisingly high rate of sorry-not-sorry “emergencies.”

3. Tech, Like Soylent Green, is Made of People

• Andrew Clay Shafer likes to opine that 90 percent of tech is tribalism and fashion.
• Nobody is doing résumé-driven development with shell scripts; I’m willing to bet that all the janky bash ever written was meant to solve a real problem. When we start getting fancier, there are often motivations less pure than “Let’s do this well,” and even if there aren’t, intention alone doesn’t create maintainable software. The trough of disillusionment is where we all land when dreams meet reality. Whatever slow-burning tire fire results from a given IT project, it’s a sure bet it will burn for a good long while. Software is “done” when it’s decommissioned; until that point, day one is short, while day two lasts until the heat death of the universe.
• A good mental model is Simon Wardley’s “Pioneers, Settlers, Town Planners”. While a proof of concept can be “done” enough to ship it, operationalizing it takes longer, and keeping it running in production is an ongoing project.

4. Good Fences Make Good Neighbors

• Boundary objects and abstractions give needed structure and containers make good boundary objects, but they do not eliminate the liminal space between the metaphorical (or all-too-real) dev and ops. When you implement microservices, how micro is micro? Even if you have a well-defined service that does one thing (somewhat) well, a good rubric is whether the service’s health endpoint can answer unambiguously. If the answer to “Is this working?” is “Welllllll…,” that service isn’t micro enough.
• Deciding what’s yours and what’s theirs is the basis of every sibling-rivalry détente. In Eric Brewer’s CAP theorem you can pick two of consistency, availability, and partition tolerance as long as one of them is partition tolerance, because, as distributed systems expert Caitie McCaffrey puts it, “physics and math.” In a distributed system that contains humans in multiple time zones, you’re inevitably going to have partitions, and waiting 10 hours for headquarters to wake up and make a decision is nobody’s idea of a good time. But decentralized decision making means distributing power to your human edge nodes (sometimes a hard sell).
• Empowering developer choice is facilitated by containers […]. Making thoughtful decisions about tools and architecture can help; well-considered constraints can free us from the decisions that aren’t bringing us distinguishable benefit.
• Being able to reproduce a build allows for separation of concerns. We want this to be effective and yet not introduce unnecessary barriers. The proverbial wall of confusion is all too real, built on the tension between having incentive to ship changes and being rewarded for stability.
• We want to empower people with as much agency as possible within the constraints that work for our organizations. To determine the right constraints for you, you need to talk to your teams.

5. Avoiding Sadness as a Service

• The hipsters among us will […] sell you on serverless (which is just servers you can’t ssh into)
• W. Edwards Deming, a leader in the field of quality control, said, “It’s not necessary to change. Survival is not mandatory.” Change is hard. Not changing is even worse.
• As it turns out, it’s not mandatory to write a Markov to to parse the front page of Hacker News, then yolo absolutely everything out to production instantly.
• If legacy weren’t important, you could just turn it off. But this is where your customers and money live. Glorifying exciting greenfield projects is all well and good, but the reality is that bimodal IT is a lie. It’s ludicrous to tell people that some of them have to stay in “sad mode” indefinitely, while others catapult ahead in “awesome mode.”
• We succeed when we share responsibility and have agency, when we move past learned helplessness to active listening.
• Change for the better is a decision we make together.
• Tools are necessary but not sufficient. To build a future we can all live with, we have to build it together.