Tag Archives: DevOps

Quality To Be Named Later

Posted on by
1

programming

First off, go watch this excellent video from Ken Duda of Arista at Networking Field Day 28. It’s the second time he’s knocked it out of the park when it comes to talking about code quality:

One of the things that Ken brings up in this video that I thought would be good to cover in a bit more depth is the idea of what happens to the culture of your organization, specifically code quality, when you acquire a new company. Every team goes through stages of development from formation through disagreement and finally to success and performance. One of the factors that can cause a high-performing team to regress back to a state of challenges is adding new team members to the group.

Let’s apply this lesson to your existing code infrastructure. Let’s say you’ve spent a lot of time building the best organization that has figured out and your dev teams are running like a well-oiled machine. You’re pushing out updates left and right and your users are happy. Then, you buy a company to get a new feature or add some new blood to the team. What happens when that new team comes on-board? Are they going to integrate into what you’ve been doing all this time? Do they have their own method for doing development? Are they even using the same tools that you have used? How much upheaval are you in for?

Buying Your Way To Consistency

Over a decade ago, United Airlines bought Continental Airlines. Two years later, the companies finally merged their ticketing systems. Well, merged might be a bit of a stretch. United effectively moved all their reservations to the Continental system and named the whole thing United. There are always challenges with these kinds of integrations but one might think that part of the reason for the acquisition was to move to a more modern reservation system.

United’s system was called Apollo and built in the 1970s. How could they move to a more modern system? Was the reason for the huge purchase of another airline directly related to their desire to adopt a newer, more flexible reservation system? There have certainly been suggestions of that for a number of years. But, more importantly, they also saw the challenges faced by one of their Star Alliance partner US Airways in 2007 when they tried a different approach to merging booking systems. The two radically different code bases clashed and created issues. And that’s for something as simple as an airline reservation system!

In the modern day we have much more control over the way that code is developed. We know the process behind what we do and what we write. When we build something we control it the entire way. However, that is true of everyone that writes code. And even with a large number of “best practices” out there no two developers are going to approach the problem the same way unless they work for the same company. So when you bring someone on board to your team through acquisition you’re also bringing in their processes, procedures, and habits. You have to own what they do because now their development quirks are part of your culture.

Bracing For the Impact

There’s a lot of due diligence that happens when companies are purchased. There’s an army of accountants that pore over the books and find all the potential issues. I’d argue that any successful merger in today’s world also needs to include a complete and thorough code review as well. You need to know how their culture is going to integrate into what you’re doing. Never mind the intellectual property issues. How do they name variables? Do they call the same memory allocation routines? Do they use floats instead of integers because that’s how they were taught? What development tools do they use and can those tools adapt to your workflow?

It may sound like I’m being a bit pedantic when I talk about variable naming being a potential issue but when it comes to code that is not the case. You’re going to have to train someone in procedure and you need to know who that is before they start committing code to your codebase. Those little differences are going to create bugs. Those bugs are going to creep into what you’re working on and create even more problems. Pretty soon something as simple as trying to translate from one IDE to another is going to create a code quality problem. That means your team is going to spend hours solving an issue that could have been addressed up front by figuring out how things were done in two different places. If you think that’s crazy, remember NASA lost a satellite over a unit conversion problem.

Tom’s Take

You may never find yourself in the shoes of someone like Ken Duda. He’s committed to quality software and also in charge of trying to integrate acquisitions with his existing culture. However, you can contribute to a better software culture by paying attention to how things are done. If you do things a certain way you need to document everything. You need to ensure that if someone new comes into the team that they can understand your processes quickly. That way they don’t spend needless hours troubleshooting problems that were lost in translation at the start of the process. Do the hard work up front so you aren’t calling people names later.

The Network Does Too Much

Posted on by
4

I’m at Networking Field Day this week and it’s good to be back in person around other brilliant engineers and companies. One of the other fun things that happens at Networking Field Day is that I get to chat with folks that help me think about things in new ways and come up with awesome ideas for networking blog posts.

One of the ones that was discussed quickly this week really got me thinking again about fragility and complexity. Thanks to Carl Fugate for reminding me about it. Essentially, networks are inherently unstable because they are doing far too much heavy lifting.

Swiss Army Design

Have you heard about the AxeSaw Reddit? It’s a page dedicated to finding silly tools that attempt to combine too many things together into one package that make the overall tool less useful. Like making a combination shovel and axe that isn’t easy to operate because you have to hold on to the shovel scoop as the handle for the axe and so on. It’s a goofy take on a trend of trying to make things too compact at the sake of usability.

Networking has this issue as well. I’ve talked about it before here but nothing has really changed in the intervening time since that post five years ago. The developers that build applications and systems still rely on the network to help solve challenges that shouldn’t be solved in the network. Things like first hop reachability protocols (FHRP) are perfect examples. Back in the dark ages, systems didn’t know how to handle what happened when a gateway disappeared. They knew they needed to find a new one eventually when the MAC address age timed out. However, for applications that couldn’t wait there needed to be a way to pick up the packets and keep them from timing out.

Great idea in theory, right? But what if the application knew how to handle that? Things like Cisco CallManager have been able to designate backup servers for years. Applications built for the cloud know how to fail over properly and work correctly when a peer disappears or a route to a resource fails. What happened? Why did we suddenly move from a model where you have to find a way to plan for failure with stupid switching tricks to a world where software just works as long as Amazon is online?

The answer is that we removed the ability for those stupid tricks to work without paying a high cost. You want to use Layer 2 tricks to fake NHRP? If it’s even available in the cloud you’re going to be paying a fortune for it. AWS wants you to use their tools that they optimize for. If you want to do things the way you’ve always done you can but you need to pay for that privileges.

With cost being the primary driver for all things, increased costs for stupid switching tricks have now given way to better software development. Instead of paying thousands of dollars a month for a layer 2 connection to run something like HSRP you can instead just make the application start searching for a new server when the old one goes away. You can write it to use DNS instead of IP so you can load balance or load share. You can do many, many exciting and wonderful things to provide a better experience that you wouldn’t have even considered before because you just relied on the networking team to keep you from shooting yourself in the foot.

Network Complexity Crunch

If the cloud forces people to use their solutions for reliability and such, that means the network is going to go away, right? It’s the apocalypse for engineer jobs. We’re all going to get replaced by a DevOps script. And the hyperbole continues on.

Reality is that networking engineers will still be as needed as highway engineers are even though cars are a hundred times safer now than in the middle of the 20th century. Just because you’ve accommodated for something that you used to be forced to do doesn’t mean you don’t need to build the pathway. You still need roads and networks to connect things that want to communicate.

What it means for the engineering team is an increased focus on providing optimal reliable communications. If we remove the need to deal with crazy ARP tricks and things like that we can focus on optimizing routing to provide multiple paths and ensure systems have better communications. We could even do crazy things like remove our reliability of legacy IP because the applications will survive a transition when they aren’t beholden to IP address or ARP to prevent failure.

Networking will become a utility. Like electricity or natural gas it won’t be visible unless it’s missing. Likewise, you don’t worry about the utility company solving issues about delivery to your home or business. You don’t ask them to provide backup pipelines or creative hacks to make it work. You are handed a pipeline that has bandwidth and the service is delivered. You don’t feel like the utility companies are outdated or useless because you’re getting what you pay for. And you don’t have to call them every time the heater doesn’t work or you flip a breaker. Because that infrastructure is on your side instead of theirs.

Tom’s Take

I’m ready for a brave new world where the network is featureless and boring. It’s an open highway with no airbags along the shoulder to prevent you from flying off the road. No drones designed to automatically pick you up and put you on the correct path to your destination if you missed your exit. The network is the pathway, not the system that provides all the connection. You need to rely on your systems and your applications to do the heavy lifting. Because we’ve finally found a solution that doesn’t allow the networking team to save the day we can absolutely build a world where the clients are responsible for their own behavior. The network needs to do what it was designed to do and no more. That’s how you solve complexity and fragility. Less features means less to break.

Silo 2: On-Premise with DevOps

Posted on by
Reply

I had a great time stirring up the hornet’s nest with the last post about DevOps, so I figured that I’d write another one with some updated ideas and clarifications. And maybe kick the nest a little harder this time.

Grounding the Rules

First, we need to start out with a couple of clarifications. I stated that the mantra of DevOps was “Move Fast, Break Things.” As has been rightly pointed out, this was a quote from Mark Zuckerberg about Facebook. However, as has been pointed out by quite a few people, “The use of basic principles to enable business requirements to get to production deployments with appropriate coordination among all business players, including line of business, developers, classic operations, security, networking, storage and other functional groups involved in service delivery” is a bit more of definition than motto.

What exactly is DevOps then? Well, as I have been educated, it’s a principle. It’s an idea. A premise, if you will. An ideal to strive for. So, to say that someone is on a DevOps team is wrong. There is no such thing as a classic DevOps team. DevOps is instead something that many other teams do in addition to their other jobs.

That being said, go ask someone what their job is in an organization. I’m willing to be that a lot of people will tell you their on the “DevOps Team”. I know this because some did a report, which I wrote about here and it includes responses from the “DevOps” team. Which, according to the classic definition, is wrong. Right?

Well, almost. See, this is where this tweet of mine comes into play:

“Pure” DevOps is hard to manage. It involves organizational shifts. It pisses people off because it’s hard to track metrics. You can’t track a person that does some traditional stuff and some of that new Dev-Op stuff. Where does that part of their job end up on a report? Putting someone in a team or a silo is almost as much for the purposes of managing that person as it is for them to do their job. If I put you in a silo, I know what you do. Or, at the very least, I can assign you tasks and responsibilities that you should be doing and grade you on those. If your “silo” is a principle and not a team, it’s crazy to grade the effectiveness of how you integrated with the developers to deliver services effectively. It can be tracked, but not as easily as a checkbox.

Likewise, people fear change. So, instead of putting their people into roles that cross functional barriers and reorganize the workflows, they instead just take the young people that are talking about the “new way” of doing things and put them in a team together. They slap a DevOps on the door and it’s done. We do DevOps now. Or, worse yet, they take the old infrastructure teams, move a few people off of them into a new team, and tell them to figure out what to do while they’re repainting the team name on the door. This has rightly been called “DevOps Washing” but a lot of people.

But what happens when that team starts Devving the Ops? Do they look at the enshrined principles of The Holy Book of DevOps and start trying to change organizational culture a little bit at a time to get the happy ending from The Phoenix Project? Do they eliminate the Brents of the world and give the security teams peace of mind?

Or, do they carve out their own little fiefdoms and start behaving like an integrated team with responsibilities and politics? Do they do things like deploy new projects to the cloud with little support from other teams. With the idea that they now “own” that workflow and can control how it’s used and how their team is viewed? If you read the article above with the report from Veriflow, you’ll find that a lot of organizations are seeing this second behavior.

Just as much as people fear proper change, they also get greedy in their new roles and want to be important to the business. And taking ownership of all the new initiatives, like cloud development, is a great way to be important. And, as much as The Phoenix Project preaches that security should be integrated into the DevOps workflow, you still half the 330 respondents to the above survey saying there is an increase in security threats to their new initiatives in public cloud.

Redefining DevOps

In a way, this “definition” of DevOps is like the title of this post. I’m sure more than a few of you bristled at the use of on-premise. Because, in today’s IT landscape we’re fighting a losing battle against a premise. When you refer to something as happening in a location, you say “on-premises”. If you say “on-premise”, you should be referring to an idea or concept. And yet, so many people in Silicon Valley say “on-premise” when referring to “on site” or “on location”. It’s grammatically wrong. But it sounds hip. It’s not the classical definition of the word and yet that word is slowly be redefined to mean what people are using it to mean. It literally happened with “literally”.

For those railing against the DevOps Washing that’s going on, ask yourself this question: Why? If the pure principles of DevOps are so much better and easier, why is everyone just slapping DevOps on existing teams or reforming other people into teams and running with the DevOps idea instead of following the rules as laid down by the sacred DevOps texts?

It could be that all organizations that are doing it this way are wrong. But are their more organizations doing it the proper way? Or is the lazy way more prevalent? I don’t know the answer, but given the number of products I see aimed at “the DevOps team” or the number of people that have given me feedback about how their organization’s DevOps teams display the same behaviors I talked about in my other blog post, I’d say there are more bad apples than purists out there.

So, what does this all mean for DevOps? Are we going to go on pointing and laughing at the DevOps-In-Name-Only crowd? Are we going to silently moan about how Real DevOps doesn’t happen and that we need to stay pure to the ideals? Or are we going to step back and realize that, just like every other technology or organizational shift that has ever occurred, nothing really gets implemented in its purest form? Instead of complaining that those not doing it the “proper” way are wrong, let’s examine why things get done the way they do and figure out how to fix it.

If businesses are implementing DevOps teams to execute the things they need done, find out why it has to be a dedicated team. Maybe they’re doing it wrong, or maybe they’ve stumbled across something that wasn’t included in the strictest definitions of DevOps. If people are giving work to those teams to accomplish and excluding other functional teams at the same time, don’t just wag your finger at them and tell them that’s not the “right way”. Find out what enabled that team to violate the ideas in the first place. Maybe the DevOps Team is responsible for all cloud deployments. Maybe they want some control over things instead of just a nebulous connection to an ideal.

Tom’s Take

DevOps in theory is a great thing. DevOps as presented in The Phoenix Project is a marvelous idea. But we all know that when theory meets reality, what we get is something different than we expected. It’s not unlike von Moltke’s famous quote, “No plan survives first contact with the enemy.” In theory, DevOps is pure and works like it should. But we’re seeing practice differing greatly from reality. The results are usually the same but the paths are radically different. And for the purists out there, if you don’t want DevOps to suffer the same fate as on-premise, you need to start asking yourself the same hard questions we are supposed to ask organizations as they start to deploy these ideas.

DevOps is a Silo

Posted on by
12

Silos are bad. We keep hearing how IT is too tribal and broken up into teams that only care about their swim lanes. The storage team doesn’t care about the network. The server teams don’t care about the storage team. The network team is a bunch of jerks that don’t like anyone. It’s a viscous cycle of mistrust and playground cliques.

Except for DevOps. The savior has finally arrived! DevOps is the silo-busting mentality that will allow us all to get with the program and get everything done right this time. The DevOps mentality doesn’t reinforce teams or silos. It focuses on the only pure thing left in the world – committing code. The way of the CI/CD warrior. But what if I told you that DevOps was just another silo?

Team Players

Before the pitchforks and torches come out, let’s examine why IT has been so tribal for so long. The silo mentality came about when we started getting more specialized with regards to infrastructure. Think about the original compute resources – mainframes. There weren’t any silos with mainframes because everyone pretty much had to know what they were doing with every part of the system. Everything was connected to the mainframe. The mainframe itself was the silo.

When we busted the mainframe apart and started down the road of client/server computing the hardware started becoming more specialized. Instead of one giant machine we had lots of little special machines everywhere. The more we deconstructed the mainframe, the more we needed to focus. The direct-attached storage became NAS and eventually SAN. The computer got bigger and bigger and eventually morphed into a virtualized hypervisor. The network exists to connect everything to the rest of the world, and as technology wore on the network became the transport for the infrastructure to talk to everything else.

Silos exist because you had to have specialized knowledge to operate your specialized infrastructure. Sure, there could be some cross training at lower levels or administration. Buy one you got into really complex topics like disk geometry optimization or route redistribution the ability for a layperson to understand what was going on was shot. Each silo exists to reinforce their own infrastructure. Each silo has their norms and their schedules. The storage team will never lose data. The network always has to be available.

Even as these silos got crammed together and subsumed into new job roles, the ideas behind them stayed consistent. Some of the storage admin’s job roles combined with the virtualization team to be some kind of a hybrid. The networking team has been pushed to adopt more agile development methodologies like automation and orchestration. Through it all, the silos were coming down as people pushed the teams to embrace more software focused on the infrastructure. That is, until DevOps burst onto the scene.

OpSilo

The DevOps tribe has a mantra: “Move Fast. Break Things. Ship. Ship. SHIP!” Maybe not those exact words but something very similar. DevOps didn’t come from mainframes. It didn’t even come from the early days of client/server. DevOps grew out of a time when everything was blown apart and on the verge of being moved into the cloud. These new DevOperators didn’t think about infrastructure as a team or a tribe. Instead, it was an impediment to shipping code.

When you work in software, moving fast and breaking things works. Because you’re pushing the limits of what you can do. You’re focused on features. You want new shiny things. Stability can wait as long as the next code commit is right around the corner. Who cares about what you’ve been doing.

In order to have the best experience with Software X, please turn on Automatic Updates so we can push the code as fast as our commits will allow.

Sound familiar? Who cares about disk geometry or route reflectors. Make my stuff work! Your infrastructure supports all my awesome code. I write the stuff that pays your salary. This place would be out of business if it wasn’t for me!

Granted that’s a little extreme, but the mentality is the same. Infrastructure exists to be consumed. IT is there to support the mission of Moving Fast, Breaking Things, and Shipping. It’s almost like a tribal behavior. Everyone has the same objective – ALL THE COMMITS!

Move fast and break things is the exact opposite of the storage and networking teams. You really don’t want to be screaming along at 800Mph when deploying a new SAN or trying to get iBGP stood up. You want careful. Calm. Collected. You’re working with a whole system that’s built on a house of cards. Unlike DevOps, breaking a thing in a SAN or on the edge of a network could impact the entire system, not just one chat module.

That’s why Networking and storage admins are so methodical. I harken back to some of my days in network engineering. When the network was running slow or the storage array was taxed, it took time to get data back. People were irritated but they got used to the idea of slowness. But if those systems ever went down, it was all-hands-on-deck panic! Contrast that with the mentality of the DevOps tribe. Who cares if it’s kind of broken right now? We need to ship the next feature or patch.

DevOps isn’t a silo buster. It’s just a different kind of tribal silo. The DevOps folks all have similar mentalities and view infrastructure in the same way. Cloud appeals to them because it minimizes infrastructure and gives them the tools they need to focus on developing. Cloud sprawl can easily happen when planning doesn’t occur. When specialized groups get together and talk about what they need, there is a reduction in consumed resources. Storage admins know how to get the most out of what they have. They don’t just spin up another bucket and keep deploying.

Tom’s Take

If you treat DevOps like a siloed tribe you’ll find their behavior is much easier to predict and work with. Don’t look at them as a cross-functional solution to all your problems. Even if you deploy all your assets to the cloud you’re going to need specialized teams to manage them once the infrastructure grows too big to manage by movement. Specialization isn’t the result of bad planning or tribalism. Instead, those specialized teams developed because of the need for deeper understanding. Just like DevOps developed out of a need to understand rapid deployment and fast-moving consumption of infrastructure. In time, the next “solution” to the DevOps problem will come along and we’ll find as well that it’s just another siloed team.

Scotty Isn’t DevOps

Posted on by
2

I was listening to the most recent episode of our Gestalt IT On-Presmise IT Roundtable where Stephen Foskett mentioned one of our first episodes where we discussed whether or not DevOps was a disaster, or as I put it a “dumpster fire”. Take a listen here:

Around 13 minutes in, I have an exchange with Nigel Poulton where I mention that the ultimate operations guy is Chief Engineer Montgomery Scott of the USS Enterprise. Nigel countered that Scotty was the epitome of the DevOps mentality because his crazy ideas are what kept the Enterprise going. In this post, I hope to show that not only was Scott not a DevOps person, he should be considered the antithesis of DevOps.

Engineering As Operations

In the fictional biography of Mr. Scott, all he ever wanted to do was be an engineer. He begrudging took promotions but found ways to get back to the engine room on the Enterprise. He liked working starships. He hated building them. His time working on the transwarp drive of the USS Excelsior proved that in the third Star Trek film.

Scotty wasn’t developing new ideas to implement on the Enterprise. He didn’t spend his time figuring out how to make the warp engines run at increased efficiency. He didn’t experiment with the shields or the phasers. Most of his “miraculous” moments didn’t come from deploying new features to the Enterprise. Instead, they were the fruits of his ability to streamline operations to combat unforeseen circumstances.

In The Apple, Scott was forced to figure out a way to get the antimatter system back online after it was drained by an unseen force. Everything he did in the episode was focused on restoring functions to the Enterprise. This wasn’t the result of a failed upgrade or a continuous deployment scenario. The operation of his ship was impacted. In Is There No Truth In Beauty, Mr. Scott even challenges the designer of the Enterprise’s engines that he can’t handle them as well as Scotty. Mr. Scott was boasting that he was better at operations than a developer. Plain and simple.

In the first Star Trek movie, Admiral Kirk is pushing Scotty to get the Enterprise ready to depart in hours after an eighteen month refit. Scotty keeps pushing back that they need more time to work out the new systems and go on a shakedown cruise. Does that sound like a person that wants to do CI/CD to a starship? Or does it sound more like the caution of an operations person wanting to make sure patches are deployed in a controlled way? Every time someone in the series or movies suggested doing major upgrades or redesigns to the Enteprise, Scotty always warned against doing it in the field unless absolutely necessary.

Montgomery Scott isn’t the King of DevOps. He’s a poster child for simple operations. Keep the systems running. Deal with problems as they arise. Make changes only if necessary. And don’t monkey with the systems! These are the tried-and-true refrains of a person that knows that his expertise isn’t in building things but in making them run.

Engineering as DevOps

That’s not to say that Star Trek doesn’t have DevOps engineers. The Enterprise-D had two of the best examples of DevOps that I’ve ever seen – Geordi LaForge and Data. These two operations officers spent most of their time trying new things with the Enterprise. And more than a few crises arose because of their development aspirations.

LaForge and Data were constantly experimenting on the Enterprise in an attempt to make it run better. Given that the mission of the Enterprise-D did not have the same five-year limit as the original, they were expected to keep the technology on the Enterprise more current in space. However, their experiments often led to problems. Destabilizing the warp core, causing shield harmonics failures, and even infecting the Enterprise’s computer with viruses were somewhat commonplace during Geordi’s tenure as Chief Engineer.

Commander Data was also rather fond of finding out about new technology that was being developed and trying to integrate it into the Enterprise’s systems. Many times, he mentioned finding out about something being developed the the Daystrom Institute and wanting to see if it would work for them. Which leads me to think that the Daystrom Institute is the Star Trek version of Stack Overflow – copy some things you think will make everything better and hope it doesn’t blow up because you didn’t understand it.

Even if it was a plot convenience device, it felt like the Enterprise was often caught in the middle of applying a patch or an upgrade right when the action started. An exploding star or an enemy vessel always waited until just the right moment to put the Enterprise in harm’s way. Even Starfleet seemed to decide the Enterprise was the only vessel that could help after the DevOps team took the warp core offline to make it run 0.1% faster.

Perhaps instead of pushing forward with an aggressive DevOps mentality for the flagship of the Federation, Geordi and Data would have done better to take lessons from Mr. Scott and wait for appropriate windows to make changes and upgrades and quite tinkering with their ship so often that it felt like it was being held together by duct tape and hope.

Tom’s Take

Despite being fictional characters, Scotty, Geordi, and Data all represent different aspects of the technology we look at today. Scotty is the tried-and-true operations person. Geordi and Data are leading the charge to keep the technology fresh. Each of them has their strong points, but it’s hard to overlook Scotty as being a bastion of simple operations mentalities. Even when they all met together in Relics, Scotty was thinking more about making things work and less on making them fast or pretty or efficient. I think the push to the DevOps mentality would do well to take a seat and listen to the venerable chief engineer of the original Enterprise.

SDN Myths Revisited

Posted on by
1

techunplugged-logo

I had a great time at TECHunplugged a couple of weeks ago. I learned a lot about emerging topics in technology, including a great talk about the death of disk from Chris Mellor of the Register. All in all, it was a great event. Even with a presentation from the token (ring) networking guy:

I had a great time talking about SDN myths and truths and doing some investigation behind the scenes. What we see and hear about SDN is only a small part of what people think about it.

SDN Myths

Myths emerge because people can’t understand or won’t understand something. Myths perpetuate because they are larger than life. Lumberjacks and blue oxen clearing forests. Cowboys roping tornadoes. That kind of thing. With technology, those myths exist because people don’t want to believe reality.

SDN is going to take the jobs of people that can’t face the reality that technology changes rapidly. There is a segment of the tech worker populace that just moves from new job to new job doing the same old things. We leave technology behind all the time without a care in the world. But we worry when people can’t work on that technology.

I want you to put your hands on a floppy disk. Go on, I’ll wait. Not so easy, is it? Removable disk technology is on the way out the door. Not just magnetic disk either. I had a hard time finding a CD-ROM drive the other day to read an old disc with some pictures. I’ve taken to downloading digital copies of films because my kids don’t like operating a DVD player any longer. We don’t mourn the passing of disks, we celebrate it.

Look at COBOL. It’s a venerable programming language that still runs a large percentage of insurance agency computer systems. It’s safe to say that the amount of money it would cost to migrate away from COBOL to something relatively modern would be in the millions, if not billions, of dollars. Much easier to take a green programmer and teach them an all-but-dead language and pay them several thousand dollars to maintain this out-of-date system.

It’s like the old story of buggy whip manufacturers. There’s still a market for them out there. Not as big as it was before the introduction of the automobile. But it’s there. You probably can’t break into that market and you had better be very good (or really cheap) at making them if you want to get a job doing it. The job that a new technology replaced is still available for those that need that technology to work. But most of the rest of society has moved on and the old technology fills a niche roll.

SDN Truths

I wasn’t kidding when I said that Gartner not having an SDN quadrant was the smartest thing they ever did (aside from the shot at stretched layer 2 DCI). I say this because it will finally force customers to stop asking for a magic bullet SDN solution and it will force traditional networking vendors to stop packaging a bunch of crap and selling it as a magic bullet.

When SDN becomes a part of the entire solution and not some mystical hammer that fixes all the nails in your environment, then the real transformation can happen. Then people that are obstructing real change can be marginalized and removed. And the technology can be the driver for advancement instead of someone coming down the hall complaining about things not working.

We spend so much time reacting to problems that we forgot how to solve them for good. We’re not being malicious. We just can’t get past the triage. That’s the heart of the fire fighter problem. Ivan wrote a great response to my fire fighter post and his points were spot on. Especially the ones about people standing in the way, whether it be through outright obstruction or by taking power away to affect real change. We can’t hold networking people responsible for the architecture and simultaneously keep them from solving the root issues. That’s the ham-handed kind of organizational roadblock that needs to change to move networking forward.

Tom’s Take

Talks like this don’t happen over night. They take careful planning and thought, followed by panic when you realize your 45-minute talk is actually 20-minutes. So you cut out the boring stuff and get right to the meat of the issue. In this case, that meat is the continued misperception of SDN no matter how much education we throw at the networking community. We’re not going to end up jobless programmers being lied to by silver-tongued marketing wonks. But we are going to have to face the need for organization change and process reevaluation on a scale that will take months, if not years, to implement correctly. And then do it all over again as technology evolves to fit the new mold we created when we broke the old one.

I would rather see the easy money flee to a new startup slot machine and all of the fair weather professionals move on to a new career in whatever is the hot new thing. That means those of us left behind in the newly-transformed traditional networking space will be grizzled veterans willing to learn and implement the changes we need to make to stop being blamed for the problems of IT and be a model for how it should be run. That’s a future to look forward to.