Silos are bad. We keep hearing how IT is too tribal and broken up into teams that only care about their swim lanes. The storage team doesn’t care about the network. The server teams don’t care about the storage team. The network team is a bunch of jerks that don’t like anyone. It’s a viscous cycle of mistrust and playground cliques.
Except for DevOps. The savior has finally arrived! DevOps is the silo-busting mentality that will allow us all to get with the program and get everything done right this time. The DevOps mentality doesn’t reinforce teams or silos. It focuses on the only pure thing left in the world – committing code. The way of the CI/CD warrior. But what if I told you that DevOps was just another silo?
Team Players
Before the pitchforks and torches come out, let’s examine why IT has been so tribal for so long. The silo mentality came about when we started getting more specialized with regards to infrastructure. Think about the original compute resources – mainframes. There weren’t any silos with mainframes because everyone pretty much had to know what they were doing with every part of the system. Everything was connected to the mainframe. The mainframe itself was the silo.
When we busted the mainframe apart and started down the road of client/server computing the hardware started becoming more specialized. Instead of one giant machine we had lots of little special machines everywhere. The more we deconstructed the mainframe, the more we needed to focus. The direct-attached storage became NAS and eventually SAN. The computer got bigger and bigger and eventually morphed into a virtualized hypervisor. The network exists to connect everything to the rest of the world, and as technology wore on the network became the transport for the infrastructure to talk to everything else.
Silos exist because you had to have specialized knowledge to operate your specialized infrastructure. Sure, there could be some cross training at lower levels or administration. Buy one you got into really complex topics like disk geometry optimization or route redistribution the ability for a layperson to understand what was going on was shot. Each silo exists to reinforce their own infrastructure. Each silo has their norms and their schedules. The storage team will never lose data. The network always has to be available.
Even as these silos got crammed together and subsumed into new job roles, the ideas behind them stayed consistent. Some of the storage admin’s job roles combined with the virtualization team to be some kind of a hybrid. The networking team has been pushed to adopt more agile development methodologies like automation and orchestration. Through it all, the silos were coming down as people pushed the teams to embrace more software focused on the infrastructure. That is, until DevOps burst onto the scene.
OpSilo
The DevOps tribe has a mantra: “Move Fast. Break Things. Ship. Ship. SHIP!” Maybe not those exact words but something very similar. DevOps didn’t come from mainframes. It didn’t even come from the early days of client/server. DevOps grew out of a time when everything was blown apart and on the verge of being moved into the cloud. These new DevOperators didn’t think about infrastructure as a team or a tribe. Instead, it was an impediment to shipping code.
When you work in software, moving fast and breaking things works. Because you’re pushing the limits of what you can do. You’re focused on features. You want new shiny things. Stability can wait as long as the next code commit is right around the corner. Who cares about what you’ve been doing.
In order to have the best experience with Software X, please turn on Automatic Updates so we can push the code as fast as our commits will allow.
Sound familiar? Who cares about disk geometry or route reflectors. Make my stuff work! Your infrastructure supports all my awesome code. I write the stuff that pays your salary. This place would be out of business if it wasn’t for me!
Granted that’s a little extreme, but the mentality is the same. Infrastructure exists to be consumed. IT is there to support the mission of Moving Fast, Breaking Things, and Shipping. It’s almost like a tribal behavior. Everyone has the same objective – ALL THE COMMITS!
Move fast and break things is the exact opposite of the storage and networking teams. You really don’t want to be screaming along at 800Mph when deploying a new SAN or trying to get iBGP stood up. You want careful. Calm. Collected. You’re working with a whole system that’s built on a house of cards. Unlike DevOps, breaking a thing in a SAN or on the edge of a network could impact the entire system, not just one chat module.
That’s why Networking and storage admins are so methodical. I harken back to some of my days in network engineering. When the network was running slow or the storage array was taxed, it took time to get data back. People were irritated but they got used to the idea of slowness. But if those systems ever went down, it was all-hands-on-deck panic! Contrast that with the mentality of the DevOps tribe. Who cares if it’s kind of broken right now? We need to ship the next feature or patch.
DevOps isn’t a silo buster. It’s just a different kind of tribal silo. The DevOps folks all have similar mentalities and view infrastructure in the same way. Cloud appeals to them because it minimizes infrastructure and gives them the tools they need to focus on developing. Cloud sprawl can easily happen when planning doesn’t occur. When specialized groups get together and talk about what they need, there is a reduction in consumed resources. Storage admins know how to get the most out of what they have. They don’t just spin up another bucket and keep deploying.
Tom’s Take
If you treat DevOps like a siloed tribe you’ll find their behavior is much easier to predict and work with. Don’t look at them as a cross-functional solution to all your problems. Even if you deploy all your assets to the cloud you’re going to need specialized teams to manage them once the infrastructure grows too big to manage by movement. Specialization isn’t the result of bad planning or tribalism. Instead, those specialized teams developed because of the need for deeper understanding. Just like DevOps developed out of a need to understand rapid deployment and fast-moving consumption of infrastructure. In time, the next “solution” to the DevOps problem will come along and we’ll find as well that it’s just another siloed team.
The Networking Nerd 
Tom. An acquaintance mentioned this. I seriously hope not that Airlines join the bandwagon of an agile and immutable model like – we do fast fast fast. If one goes down, will create a new one.
The principles of any process improvement, automation & waste minimization are great but the context is more critical than the coolness. We have been hearing DevOps, DevSecOps, DevNetSecOps, SuperNetSecOps and the list is going on. But I am yet to see a major Telco / ISP changing a BGP config or Mobile Core GW with its multi-million orchestration software or quickly download the beta software without testing it on OLP/TestBed.
The service criticality, SLA and most importantly the Regulatory requirements are there for a reason and I doubt SP can be like facebook or whatsapp where they can simply say ‘sorry’, our service is not available between xxx to xxx because we are bringing cool new features.
“immutable…fast fast fast.”
I’m not clear why immutable change techniques for a critical systems, like airlines, are worse than an alternative. What would be the alternative anyway? Patching in place in prod? Fast CI cycles don’t generally equate to deploying untested, unworkable code into a system, whether critical or not.
“…without testing it on OLP/TestBed”
This seems like a straw man argument. Who is suggesting that software should move to prod without testing that’s consistent with policy? Certainly, even with appropriate, rigorous testing there are plenty of instances where service providers deploy errors into prod.
Tom,
I think the situation is a bit more nuanced.
The way I personally view things, there are infrastructure operations, and there are infrastructure services consumers. And the ways they do things are completely different.
Your examples of network and storage admins fit squarely in the infra space, and I don’t think DevOps are there to take these jobs over. If anything, they would prefer to keep far from infra operations, because it does not help them ship. Their preference is to have access to standadised services that the infrastructure provides like L2/L3 connectivity, firewall rules, LB Virtual Servers, block volumes, object storage endpoints, DB instances (that someone else manages for them). Preferably through an API endpoint, and with a service contract.
On the other side of the fence, you would still have the infrastructure folk, doing what they always did best – design, deploy, and operate highly available infrastructure. The only difference is that today they should be seriously thinking about how they can (a) figure out the set of standardised offerings their infra should provide, with SLAs; (b) how to make these offerings available to the DevOps team(s) in a self-serve manner; and (c) how to design their infra support systems and operational processes to support the (a) and (b).
Think of the infra folks within AWS – I very much doubt they rush into installing freshly released patches to firmware or whatever; but the services their infra provides are created, updated, and deleted millions times a day, or whatever, without them having to care about it at all.
As an aside, I think “move fast and break things” mantra is somewhat misunderstood, because its wording drops important context. That context is “..because the system can take it without affecting our users”. I would agree that this context may have been lost on many very DevOps folks whose responsibility it was to make it so, though. 🙂
DevOps, as the literature usually has it, consists of all the required players coordinating, not separating, say. infra from coders. It makes no sense to accept a business requirement with an expectation of delivery in X weeks, if requisite infra itself requires X+8 weeks.
Usually when the task is pulled into the queue, the line of business that created the requirement will have a coordinated idea of when the result will be in prod. At least there won’t be someone in DB complaining later about a schema that needs to be changed, a networking person requiring another line card for capacity, a storage person with a constraint, or risk/compliance saying the requirement is illegal.
The framing of this article is based on conjecture without any evidence. The origins of DevOps from those meetings in Belgium were not based on the idea that infrastructure was an obstacle to deploying applications.
You would be correct that this is conjecture. It’s based on sponsored reports that I’ve seen from companies like Veriflow, which you can register to see here – https://www.veriflow.net/is-the-public-cloud-hiding-business-risk-in-plain-sight/?utm_source=5ThingsPDF, as well as conversations I’ve had with practitioners that I know through the industry.
Without getting into a massive argument, I want to ask a very simple question: Do you see more organizations implementing pure DevOps as discussed in those Belgian meetings? Or is it more that organizations are relabeling existing organizational structures as “DevOps” in order to check a box and seem to be on the cutting edge?
Tom,
The answer is that, if you want to be consistent with what DevOps offers, you follow DevOps principles.
If you want to just say that you are doing DevOps, then you can retain the same siloed practices that have caused business friction for years and simply label what you’re doing DevOps. Similarly, you can create giant containers with all the contents of a VM and say you’re containerized.
This is a related article. It sounds like you might have experienced one or more of the antipatterns the post mentions.
https://web.devopstopologies.com/
“If you treat DevOps like a siloed tribe you’ll find their behavior is much easier to predict and work with.”
This seems like a misunderstanding of what DevOps is. It’s not a defined group within IT that gets an in progress product and hands off to another group like a car under assembly. It’s the entire process from inception to prod. and follow on through revision cycles. If you consider DevOps in this way, all the issues about silos and tribes go away.
And silos/tribes certainly existed and continue to exist in mainframe land. There’s nothing magical about mainframes that coordinated, say, networking with storage and DB.
Pingback: Silo 2: On-Premise with DevOps | The Networking Nerd
I agree with the general idea. More often than not, the DevOps is it’s own team, which contradicts the initial idea that DevOps should be used to break silos, and it ends up create new ones.
In a way this is only human, a DevOps team still has a leader and subordinates, and thus inevitably becomes tribal.
Well, one could make the argument that this DevOps labeling is not a reorg to produce a different result, but a misunderstanding of what DevOps is.
Generally if an organization has a “DevOps team” among other teams, it’s an indicator that overall DevOps principles aren’t being followed. There are many literature references on this, so I’m not just making things up. 😉