Why are Enterprises So Slow?

Background

• Before the enterprise, I worked for a startup that grew from a single room to 700+ people over 15 years.
• When I finally got sick of the startup life I took a job at a huge organisation in financial services over 200 times as large.

Thought Experiment

• imagine an enterprise acted like a startup
• what would happen to perform a relatively routine task, e.g. upgrading a Linux distribution
• [What does a regulated enterprise do to mitigate the risk and consequences of worst-case scenarios]:
  • “Who owns this?”
  • “How is this maintained?”
  • “Who buys it?”
  • “Who’s signed off on the deployment?”

Reducing Risk

‘Who owns this?’ / ‘One Throat to Choke’

• One of the most commonly-asked questions when architecting a solution within an enterprise is: ‘Who is responsible for that component/service/system?’
• This group will be identifiable through some internal system which tracks ownership of tools and technologies.
• Those identified as responsible will be responsible for some or all of the lifecycle management for that technology.
• This ownership results in ‘one throat to choke’ for audit functions.
• A lot of the political footwork in an enterprise revolves around trying to not own technologies.

Enterprises and Vendors

• This also explains enterprises' love of vendor software over pure open source. If you’ve paid someone to maintain and support a technical stack, then they become responsible for that whole stack. That doesn’t solve all your problems […], but from a governance point of view you’ve successfully passed the buck.
• What is governance?
  • IT Governance is a term that covers all the processes and structures that ensures the IT is appropriately managed in a way that satisfies those that govern the organisation.

‘How is this maintained?’

• In our idealised startup above ‘Dev’ and ‘Ops’ were the same thing (ie, one person). Lo and behold you have DevOps!
• Unfortunately, the DevOps slogan ‘you built it, you run it’ doesn’t usually work in an Enterprise context for a few reasons.
• Partly it’s historical […] institutional bias towards not changing this […]. But further bolstering this conservatism is the regulatory framework that governs how software is managed.

Regulations

• Regulations are rules created by regulators [and], effectively, they have the force of law as far as your business is concerned.
• their paradigms are rooted in the experiences of software built in previous decades.
• If your software is regulated, then it’s likely your engineering (dev) and operations teams (ops) will be separate groups of people specialising in those roles, and one of the drivers of this is the regulations, which demand a separation to ensure that changes are under some kind of control and oversight.
• regulations often talk about ‘separation of roles’ between engineering and operations, and don’t explicitly say that these roles need to be fulfilled by different people.
• But if you’re a really big enterprise, that might be technically correct but effectively irrelevant. Why? Because, to ‘simplify’ things, these large enterprises often create a set of rules that cover all the regulations that may ever apply to their business across all jurisdictions. And those rules are generally the strictest you can imagine.
• Added to that, those rules develop a life and culture of their own within the organisation independent of the regulator such that they can’t easily be brought into question.
  • Resistance is futile. Dev and Ops must be separate because that’s what we wrote down years ago.
• So you can end up in a situation where you are forced to work in a way prescribed years ago by your internal regulations, which are in turn based on interpretations of regulations which were written years before that!
• Obviously, this separation slows things down as engineering must make the code more tolerant to mistakes and failure so that another team can pick it up and carry it through to production. Or you just throw it over the wall and hope for the best. Either way, parties become more resistant to change.

Change Control

• In order to ensure that changes to systems can be attributed to responsible individuals, there is usually some kind of system that tracks and audits changes. One person will raise a ‘change record’, which will usually involve filling out an enormous form, and then this change must be ‘signed off’ by one or more other person to ensure that changes don’t happen without due oversight.
• most of the time trust relationships build up between change raiser and change validator which can speed things up. If the change is large and significant, then it is more likely to be closely scrutinised. There might also exist ‘standard changes’ or ‘templated changes’, which codify more routine and lower-risk updates and are pre-authorised.
• While in theory the change can be signed off in minutes, in reality change requests can take months

Security ‘Sign-Off’

• If you’re working on something significant, […] it may become necessary to get what most people informally call ‘security sign-off’.
• essentially, one or more security experts descend at some point on your project and audit it.
• I had imagined such reviews to be a very scientific process, but in reality it’s more like a medieval trial by ordeal.
• The outcome of this is usually some kind of report and a set of risks that have been identified. These risks (depending on their severity–I’ve never heard of there being none) may need to be ‘signed off’ by someone senior so that responsibility lies with them if there is a breach. That process in itself is arduous (especially when the senior doesn’t fully understand the risk) and can be repeated on a regular basis until it is sufficiently ‘mitigated’ through further engineering effort or process controls. After which it’s then re-reviewed. None of this is quick.

Summary: Corporate, not Individual Responsibility

• If there’s a common thread to these factors in reducing risk, it is to shift responsibility and power from the individual to the corporate entity. If you’re a regulated, systemically-significant enterprise, then the last thing you or the public wants is for one person to wield too much power.
• The corollary of this is that it is very hard for one person to make change by themselves.

Example–[Vendor] Sourcing

• To prevent a situation where one person could get too much power it can be the case that technical people have no direct control over the negotiation (or ‘sourcing process’) [for a vendor solution] at all.
• this process can take months or even years. And might need to be repeated if the process takes so long that funding has disappeared or teams have been disbanded.
• sourcing might have its own ‘preferred supplier lists’ of companies that have been vetted and audited in the past. If your preferred supplier isn’t on that list (and hasn’t made a deal with one [of those on the] list), the process could take even longer.

Cumulative Constraints

Dependency Constraints

• One of the joyous things about working in an unregulated startup is that if you see a problem in one of your dependencies you have the option of taking it over and running it yourself.
• So why not do the same in the enterprise? Why not just ‘find your dependencies and eliminate them’?
• Some do indeed take this approach, and it costs them dearly. Either they have to spend great sums of money managing the processes required to maintain and stay ‘within governance’ for the technology they’ve decided to own, or they get hit with an audit sooner or later and get found out.
• taking responsibility and owning a technology or layer of your stack brings with it real costs and risks that you may not be able to bear and stay in business.
• So however great you are as a team, your delivery cadence is constrained to a local maxima based on your external dependencies, which are (effectively) non-negotiable.
• Just as it is significantly harder for you to make that much difference, it is harder for your team to make much difference, for the same structural reasons.

Cultural Constraints

Calcified Paradigms

The ‘machine paradigm’

• Whether you fill out a form and wait for a physical machine or a virtual machine to be provisioned makes little difference–you’re still in the machine paradigm.
• Recently, aPaaSes, Kubernetes, and cloud computing have overthrown the idea that an application needs to sit on a ‘machine’, but the penetration of this novel (or old, if you used mainframes) idea, like the future, is unevenly distributed.

Charging models

• To grossly generalise, IT is moving from a ‘capex’ model to an ‘opex’ model. Instead of buying kit and software and then running it until it wears out (capex), the ‘new’ model is to rent software and services which can be easily scaled up and down as business demand requires.
• moving to these new models can be painful. Trying to cross-charge within an organisation of any size can result in surreal conversations about ‘wooden dollars’ (ie non-existent money exchanged in lieu of real money) or services being charged out to other parts of the business, but never paid for due to conversations that may or may not have been had outside your control.

Learned Helplessness

• After decades of these habits of thoughts, you end up with several consequences:
  • Those who don’t like the way of working leave
  • Those that remain calcify into whole generations of employees
  • Those that remain tend to prize and prefer those that agree with their views
• change is resisted because it won’t work, and it won’t work because it’s resisted. But it’s also quite rational, since the reasons it won’t work are based on the external constraints that exist we have discussed above.

A New Hope?

Senior Leadership Support

• If you’re looking to swim against habits of thought, then stiff resolve is required. If senior management aren’t willing to sacrifice, aren’t united in favour of it, then all sorts of primary decisions and (equally important) second-guessed decisions made by underlings [will] have conflicting aims.
• People don’t like to talk about it, but it helps if people get fired for not constructively working with the changes. That tends to focus the mind. The classic precedent of this is point 6 of Jeff Bezos’s ‘API Mandate’

Reduce Complexity

• you will do yourself favours if you fight tooth and nail to reduce complexity. This may involve taking some risks as you call out that the entire effort may be ruined by compromises that defeat the purpose, or create bureaucratic or technical quicksand that your project will flounder in later.
• Calling those dangers out may get you a reputation, or even cost you your job.

Cross-functional Team

• it’s much easier to achieve change if you have a team of people that span the functions of your organisation working together. The collaboration not only benefits from seeing how things need be designed to fulfil requirements at an earlier stage, but more creative solutions are found by people who understand their function’s needs better, and the requirements of the project. If you want to go the skunkworks route, then the representatives of the other functions can tell you where your MVP shortcuts are going to bite you later on.

Use Your Cynical Old Hands

• The flip side of those that constitute the ‘institutional inertia’ I described above is that many of those people know the organisation inside out. These people often lose heart regarding change not because they no longer care, but because they believe that when push comes to shove the changes won’t get support.
• These people can be your biggest asset. The key is to persuade them that it’s possible, and that you need their help.