Monthly Archives: November 2018

The Magic of the CCIE

Posted on by
3

I stumbled across a great Reddit thread this week: Is the CCIE as impossible as it seems? There are a lot of great replies on that thread about people passing and the “good old days” of Banyan Vines, Appletalk, and more. It’s also a fascinating look into how the rest of the networking industry sees exams like the CCIE and JNCIE. Because those of us that have the numbers seem to be magicians to some.

Sleight of CLI Hand

Have you ever seen the cups and balls magic trick? Here’s an excellent example of it from the recently departed Ricky Jay:

Impressive, right? It’s amazing to behold a master craftsman at work. Every time I watch that video I’m amazed. I know he’s doing sleight of hand. But I can’t catch it. Now, watch this same video but with annotations turned on. SPOILER ALERT – The annotations will tell you EXACTLY where the tricks are done:

Is it more impressive now that you know how the tricks are done? Check out this demonstration from Penn and Teller that shows you exactly how they do the tricks as well:

Okay, so it’s a little less mystifying now that you’ve seen how all the sleight of hand happens. But it’s still impressive because, as a professional, you can appreciate how the execute their tradecraft. Knowing that it’s not magic doesn’t mean it’s not an impressive feat. It must means you appreciate something different about the performance.

Let’s apply that to the CCIE. When you’re just starting out in networking, every piece of knowledge is new. Everything you learn is something you didn’t know before. Subnet masks, routing tables, and even just addressing an interface are new skills that you acquire and try to understand. It’s like learning how to take a coin from someone’s ear. It’s simple but it provides the building blocks for future tricks.

When you reach the level of studying for the CCIE lab, it does look like a daunting task. If you’ve followed Cisco’s guidelines you probably have your CCNP or equivalent knowledge. However, there is still a lot you don’t know. If you don’t believe that, go pick up Jeff Doyle’s Routing TCP/IP Volume 1 book. That book taught me I still had a lot to learn about networking.

But, as I slogged through the CCIE, I realized that I was acquiring skills. Just like the magicians that practice the cups and balls every day to get it right, I was picking up the ability to address interfaces quickly and see potential routing loops before I made them like I did in my first lab attempt. Each thing I learned and practiced not only made me a better engineer but also made the CCIE seem less like a mountain and more like a hill that could be climbed.

And I truly realized this when I was thumbing through a copy of the CCIE Official Exam guide. Someone had given me a copy to take a look at and I was happy with the depth of knowledge that I found. I wanted to pass it along to another junior engineer because, as I said to myself, “If only I had this book when I started! I could have skipped over all those other books!”

Practice, Practice, Practice!

That’s where I went wrong. Because I jumped right to the end goal instead of realizing the process. Magicians don’t start out making the Statue of Liberty disappear. They start out pulling coins from your ear and finding your card in a deck. They build their basic skills and then move on to harder things. But they most grand tricks in the magician’s top hat all still use the basic skills: sleight of hand, misdirection, and preparation. To neglect those is to court folly on stage.

CCIEs are no different. Every person that asks me about the test asks “How hard is it to pass?” I usually respond with something like “Not hard if you study.” Some of the people I talk to pick up on the “not hard” part and get crushed by the lab their first time out. They even end up with a $1,500 soda for their efforts. The other people, the ones that focus on “study” in my answer, they are the people who pass on the first attempt or the ones that get it right pretty quickly thereafter.

The CCIE isn’t a test. It’s a course in studying. It’s the culmination of teaching yourself the minutia of protocols and how they interact. The exam itself is almost perfunctory. It tests specific combinations of things you might see in the real world. And if you ask any CCIE, the real world is often ten time stranger than the lab. But the lab makes you think about the things you’ve already learned in new ways and apply that knowledge to find ways to solve problems. The lab isn’t hard because it’s easy. The lab becomes easier when you practice enough to not think the knowledge is hard any longer. I think Bruce Lee said it best:

I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times.

Most people would agree that Bruce Lee was one of the best martial artists of all times. And even he practiced until his fingers bled and he body was exhausted. Because he knew that being the best wasn’t about passing an exam for a belt or about showing off for people. It was about knowing what you needed to know and practicing it until it was second nature.

Tom’s Take

The CCIE has a certain magical aura for sure. But it’s not magical in and of itself. It’s a test designed to ensure that the people that pass know their skills at a deep level. It’s a test designed to make you look deeper at a problem and exhaust all your options before throwing in the towel. The CCIE isn’t impossible any more than sawing someone in half is impossible. It’s all about how your practice and prepare for the show that makes the trick seem impressive.

Murphy the Chaos Manager

Posted on by
Reply

I had the opportunity to sit in on a great briefing from Gremlin the other day about chaos engineering. Ken Nalbone (@KenNalbone) has a great review of their software and approach to things here. The more time I spent thinking about chaos engineering and IT, the more I realized that it has more in common with Murphy’s Law that we realize.

Anything That Can Go Wrong

If there’s more than one way to do a job and one of those ways will end in disaster, then somebody will do it that way. – Edward Murphy

 

Anything that can go wrong will go wrong. – Major John Paul Stapp

We live by the adage of Murphy’s Law in IT. Anything that can go wrong will go wrong. And usually it goes wrong at the worst possible time. Database query functions will go wrong when you need them the most. And usually at the height of something like Amazon Prime Day. Data center outages only seem to happen at 4 am on a Sunday during a holiday.

But why do things go wrong like this? Is it because the universe just has it out for IT people? Are we paying off karma from the fall of the Western Roman Empire? Or is it because we can’t anticipate some crazy things? Are we kidding ourselves that we can just manage Murphy and hope for the best?

As it turns out, this is why chaos engineering is so important. Because it doesn’t just make us realize that things are broken. It helps us understand how they will break in unique and different ways each time. A big reason why this is so important is because many large-scale failures aren’t the result of a single problem, but instead a collection of smaller things that build on each other.

One of my favorite stories about this collection of failures comes from a big Amazon Web Services (AWS) outage from last March. People were seeing problems in US-EAST-1 but they couldn’t nail down the issue. Worse yet, every time they logged into the Amazon dashboard they saw green lights for every service. As the minutes dragged on it was eventually discovered that the lights were lying to everyone because Amazon hosted that page on AWS US-EAST-1. They couldn’t log in to reset the lights to show an outage! Coincidentally, many other monitoring services were down as well because they were also hosted in the same region.

What does this teach us about chaos? Well, Murphy was in full effect for sure. Something went wrong and happened at a bad time. But it was also the worst possible time for Amazon to figure out that the status lights and dashboard systems were all hosted out of one region with no backup anywhere else. Perhaps they could have caught that with a system like Gremlin. Perhaps it would have gone under the radar until the worst possible moment like it did in real life. There’s no way to know for sure. Hopefully Amazon has fixed this little problem for now.

People Will Do It Wrong

This also teaches us something about user behavior. One thing we hear frequently about patches or other glaring issues with software is “How was this not caught in testing?!?”

The flip side of that is that most of these corner case issues were never tested in the first place. Testing focuses on testing main functionality of a system. QA testers focus on the big picture stuff first. Does the UI fall apart? Are all the buttons linked to a specific task? What happens when I click HELP on the login screen.

What does QA not test for? Well, lots of things that users actually do. Holding down random keystrokes while clicking buttons. Navigating to random pages and then bookmarking them without realizing that’s a bad idea. Typing the wrong information into a list box that passes validation and screws up the backend. The list of variations is endless.

How does this apply to chaos? Well, as it turns out, engineers and testers are pretty orderly people. We all look at problems and try to figure them out. We try combinations of things until we solve the issue. But everything is based on the idea that we’re trying things in specific combinations until we replicate the issue. We don’t realize that some of the random behavior we see comes from behaviors we can’t control from users.

Another story: I was editing a document the other day in a CMS and I saved the document revisions I’d made as a draft post. When I went to check the post, it had inadvertently published itself. I didn’t want it to publish at that time, so I was perplexed. I knew I had clicked the save function button but I also knew I didn’t click the publish button. I looked through documentation and couldn’t find any issues.

I put it out of my mind until it happened again a couple of weeks later. This time, I went back through every step I had just done. The only thing that was out of the ordinary compared to the last time was the I had saved the document with ⌘+S (CTRL+S for Windows) just like I’d taught myself to do for years. But, in this CMS, that shortcut saves and publishes the current document. Surprise!

Behavior that shouldn’t have triggered a problem did. Because no one ever tested for what might happen if someone used a familiar keystroke in a place where it wasn’t intended. This is what makes chaos engineering so difficult and rewarding. Because you can set up the system to test for those random things without needing to think about them. And when you figure out a new one, like whether or not ⌘+S can crash your system, you can add it to the list to be checked against everything!

Tom’s Take

I love reading and learning about chaos engineering. The idea that we purposely break things to make people thing about building them correctly appeals to me. I find myself trying to figure out how to make better things and always find out that I’m being stymied because I don’t think “outside the box”, which is a clever way of saying that I don’t think like a user. I need something that helps me understand how things will break in new and unique ways every time. Because while we can test for the big stuff, Murphy has a way of showing us what happens when we don’t sweat the small stuff.

Why Is The CCIE Lab Moving?

Posted on by
4

Cisco confirmed big CCIE rumor this week that the RTP lab was going to be moved to Richardson, TX.

The language Cisco used is pretty neutral. San Jose and RTP are being shut down as full time lab locations and everyone is moving to Richardson. We knew about this thanks to the detective work of Jeff Fry, who managed to figure this out over a week ago. Now that we know what is happening, why is it coming to pass?

They Don’t Build Them Like They Used To

Real estate is expensive. Anyone that’s ever bought a house will tell you that. Now, imagine that on a commercial scale. Many companies will get the minimum amount of building that they need to get by. Sometimes they’re bursting at the seams before they upgrade to a new facility.

Other companies are big about having lots of area. These are the companies that have giant campuses. Companies like Cisco, Dell EMC, Intel, and NetApp have multiple buildings spread across a wide area. It makes sense to do this when you’re a large company that needs the room to spread out. In Cisco’s case, each business unit had their own real estate. Wireless was in one building. Firewalls in another. Each part of the company had their own area to play in.

Cisco was a real estate maven for a while. They built out in anticipation of business. There was a story years ago of a buried concrete slab foundation in Richardson that was just waiting for the next big Cisco product to be developed so they could clear away the dirt and start construction. But, why not just build the building and be done with it?

Remember how I said that real estate is expensive? That expense doesn’t come completely from purchases. It comes from operations. You need to have utilities for the building. You need to have services for the building. You need to pay taxes on the building. And those things happen all the time. Even if you never have anyone in the building the electricity is still running. That’s one of the reasons why Cisco shuts down their offices between Christmas and New Year’s every year. And the taxes are still due. Hence the reason why the foundation in Richardson was buried.

Real estate is also not an infinite resource. Anyone that’s been to Silicon Valley knows that. They’re running out of room in the South Bay. And building the new 49ers stadium on the corner of Tasman Drive and Great America Parkway didn’t help either. Sports teams are as hungry for real estate as tech companies. The support structures that cropped up for the stadium ended up buying the Letter Buildings from Cisco, which is why the lab was moved from Building C to Building L years ago.

Home Is Where The Work Is

The other shifting demographic is that more workers are remote in today’s environment. A combination of factors have led people to be just as productive from their home office as their open-plan cubicle. Increased collaboration software coupled with changing job requirements means that people don’t have to go to their desk every day to be productive.

This is especially true now that companies like Cisco are putting more of a focus on software instead of hardware. In the good old days of hardware dominance you needed to go into the office to work on your chipset diagrams. You needed your desktop CAD program to draw the silicon traces on a switch. And you needed to visit the assembly lines and warehouses to see that everything was in order.

Today? It’s all code. Everything is written in an IDE and stored on a powerful laptop. You can work from anywhere. A green space outside your office window. A coffee shop. Your living room. The possibilities are endless. But that also means that you don’t need a permanent office desk. And if you don’t need a desk that means your company doesn’t need to pay for you to have one.

Now, instead of bustling buildings full of people working in their shared offices there are acres of empty open-plan cubicle farms lying fallow. People would rather work from Starbucks than go to the office. People would rather work in their pajamas than toil away in a cube. And so companies like Cisco are paying taxes and utilities for open spaces that don’t have anyone while the offices around the perimeter are filled with managers that are leading people that they don’t see.

CCIE Real Estate

But what does this all mean for the lab? Well, Cisco needs to downsize their big buildings in high-value real estate markets. They’re selling off buildings in San Jose as fast as the NFL will buy them. They are downsizing the workforce in RTP as well. The first hint of the CCIE move was David Blair trying to find a new job. As real estate becomes more and more costly to obtain, Cisco is going to need to expand in less expensive markets. The Dallas/Fort Worth (DFW) area is still one of the cheapest in the country.

DFW is also right in the middle of the country. It’s pretty much the same distance from everything. So people that don’t want to schedule a mobile lab can fly to Richardson and take the test there. RTP and San Jose are being transitioned to mobile lab facilities, which means people that live close to those areas can still take the test, just not on the schedule they may like. This allows Cisco to free up the space in those buildings for other purposes and consolidate their workforce down to areas that require less maintenance. They can also sell off unneeded buildings to other companies and take the profits for reinvestment in other places. Cutting costs and making money is what real estate is all about, even if you aren’t a real estate developer.

Tom’s Take

I’m sad to see the labs moving out of RTP and San Jose. Cisco has said they are going to frame the famous Wall of Pain in RTP as a tribute to the lab takers there. I have some fond memories of San Jose as well, but even those memories are from a building that Cisco doesn’t own any longer. The new reality of a software defined Cisco is that there isn’t as much of a need for real estate any more. People want to work remotely and not live in a cube farm. And when people don’t want an office, you don’t need to keep paying for them to have one. Cisco won’t be shutting everything down any time soon, but the CCIE labs are just the first part of a bigger strategy.

Editor’s Note: An earlier version of this post accidentally referred to David Mallory instead of David Blair. This error has been corrected.

Presenting To The D-Suite

Posted on by
4

Do you present to an audience? Odds are good that most of us have had to do it more than once in our life or career. Some of us do it rather often. And there’s no shortage of advice out there about how to present to an audience. A lot of it is aimed at people that are trying to speak to a general audience. Still more of it is designed as a primer on how to speak to executives, often from a sales pitch perspective. But, how do you present to the people that get stuff done? Instead of honing your skills for the C-Suite, let’s look at what it takes to present to the D-Suite.

1. No Problemo

If you’ve listened to a presentation aimed at execs any time recently, such as on Shark Tank or Dragon’s Den, you know all about The Problem. It’s a required part of every introduction. You need to present a huge problem that needs to be solved. You need to discuss why this problem is so important. Once you’ve got every head nodding, that’s when you jump in with your solution. You highlight why you are the only person that can do this. It’s a home run, right?

Except when it isn’t. Executives love to hear about problems. Because, often, that’s what they see. They don’t hear about technical details. They just see challenges. Or, if they don’t, then they are totally unaware of this particular issue. And problems tend to have lots of nuts and bolts. And the more you’re forced to summarize them the less impact they have:

Now, what happens when you try this approach with the Do-ers? Do they nod their heads? Or do they look bored because it’s a problem they’ve already seen a hundred times? Odds are good if you’re telling me that WANs are complicated or software is hard to write or the cloud is expensive I’m already going to know this. Instead of spending a quarter of your presentation painting the Perfect Problem Picture, just acknowledge there is a problem and get to your solution.

Hi, we’re Widgets Incorporated. We make widgets that fold spacetime. Why? Are you familiar with the massive distance between places? Well, our widget makes travel instantaneous.

With this approach, you tell me what you do and make sure that I know about the problem already. If I don’t, I can stop you and tell you I’m not familiar with it. Cue the exposition. Otherwise, you can get to the real benefits.

2. Why Should I Care?

Execs love to hear about Return on Investment (ROI). When will I make my investment back? How much time will this save me? Why will this pay off down the road? C-Suite presentations are heavy on the monetary aspects of things because that’s how execs think. Every problem is a resource problem. It costs something to make a thing happen. And if that resource is something other than money, it can quickly be quoted in those terms for reference (see also: man hours).

But what about the D-Suite? They don’t care about costs. Managers worry about blowing budgets. People that do the work care about time. They care about complexity. I once told a manager that the motivation to hit my budgeted time for a project was minimal at best. When they finished gasping at my frankness, I hit them with the uppercut: My only motivation for getting a project done quickly was going home. I didn’t care if it took me a day or a week. If I got the installation done and never had to come back I was happy.

Do-ers don’t want to hear about your 12% year-over-year return. They don’t want to hear about recurring investment paying off as people jump on board. Instead, they want to hear about how much time you’re going to save them. How you’re going to end repetitive tasks. Give them control of their lives back. And how you’re going to reduce the complexity of dealing with modern IT. That’s how you get the D-Suite to care.

3. Any Questions? No? Good!

Let me state the obvious here: if no one is asking questions about your topic, you’re not getting through to them. Take any course on active listening and they’ll tell you flat out that you need to rephrase. You need to reference what you’ve heard. Because if you’re just listening passively, you’re not going to get it.

Execs ask pointed questions. If they’re short, they are probably trying to get it. If they’re long winded, they probably stopped caring three slides ago. So most conventional wisdom says you need to leave a little time for questions at the end. And you need to have the answers at your fingertips. You need to anticipate everything that might get asked but not put it into your presentation for fear of boring people to tears.

But what about the Do-ers? You better be ready to get stopped. Practitioners don’t like to wait until the end to summarize. They don’t like to expend effort thinking through things only to find out they were wrong in the first place. They are very active listeners. They’re going to stop you. Reframe conversation. Explore tangent ideas quickly. Pick things apart at a detail level. Because that’s how Do-ers operate. They don’t truly understand something until they take it apart and put it back together again.

But Do-ers hate being lied to more than anything else. Don’t know the answer? Admit it. Can’t think of the right number? Tell them you’ll get it. But don’t make something up on the spot. Odds are good that if a D-Suite person asks you a leading question, they have an idea of the answer. And if your response is outside their parameters they’re going to pin you to the wall about it. That’s not a comfortable place to get grilled for precious minutes.

4. Data, Data, Data

Once you’re finished, how should you proceed? Summarize? Thank you? Go on about your life? If you’re talking to the C-Suite that’s generally the answer. You boil everything down to a memorable set of bullet points and then follow up in a week to make sure they still have it. Execs have data points streamed into the brains on a daily basis. They don’t have time to do much more than remember a few talking points. Why do you think elevator pitches are honed to an art?

Do-ers in the D-Suite are a different animal. They want all the data. They want to see how you came to your conclusions. Send them your deck. Give them reference points. They may even ask who your competitors are. Share that info. Let them figure out how you came to the place where you are.

Remember how I said that Do-ers love to disassemble things? Well, they really understand everything when they’re allow to put them back together again. If they can come to your conclusion independently of you then they know where you’re coming from. Give them that opportunity.

Tom’s Take

I’ve spent a lot of time in my career both presenting and being presented to. And one thing remains the same: You have to know your audience. If I know I’m presenting to executives I file off the rough edges and help them make conclusions. If I know I’m talking to practitioners I know I need to go a little deeper. Leave time for questions. Let them understand the process, not the problem. That’s why I love Tech Field Day. Even before I went to work there I enjoyed being a delegate. Because I got to ask questions and get real answers instead of sales pitches. People there understood my need to examine things from the perspective of a Do-er. And as I’ve grown with Tech Field Day, I’ve tried to help others understand why this approach is so important. Because the C-Suite may make the decisions, but it’s up the D-Suite to make things happen.

Clear Skys for IBM and Red Hat

Posted on by
1

There was a lot of buzz this week when IBM announced they were acquiring Red Hat. A lot has been discussed about this in the past five days, including some coverage that I recorded with the Gestalt IT team on Monday. What I wanted to discuss quickly here is the aspirations that IBM now has for the cloud. Or, more appropriately, what they aren’t going to be doing.

Build You Own Cloud

It’s funny how many cloud providers started springing from the earth as soon as AWS started turning a profit. Microsoft and Google seem to be doing a good job of challenging for the crown. But the next tier down is littered with people trying to make a go of it. VMware with vCloud Air before they sold it. Oracle. Digital Ocean. IBM. And that doesn’t count the number of companies offering a specific function, like storage, and are calling themselves a cloud service provider.

IBM was well positioned to be a contender in the cloud service provider (CSP) market. Except they started the race with a huge disadvantage. IBM was a company that was focused on selling solutions to their customers. Just like Oracle, IBM’s primary customer was external. The people they cared most about wrote them checks and they shipped out eServers and OS/2 Warp boxes.

Compare and contrast that with Amazon. Who is Amazon’s customer? Well, anyone that wants to buy something. But who consumes the products that Amazon builds for IT? Amazon people. Their infrastructure is built to provide a better experience for the people using their site. Amazon is very good at building systems that can handle a high amount of users and load and not buckle.

Now, contrary to the myth, Amazon did not start AWS with spare capacity. While that makes for a good folk tale, Amazon probably didn’t have a lot of spare capacity lying around. Instead, they had a lot of great expertise in building large-scale reliable systems and they parlayed that into a solution that could be used to bring even more people into the Amazon ecosystem. They built AWS with an eye toward selling services, not hardware.

Likewise, Microsoft’s biggest customers are their developers. They are focused on making the best applications and operating systems they can. They don’t sell hardware, aside from the rare occasional foray into phones or laptops. But they wanted their customers to benefit from the years of work they had put in to developing robust systems. That’s where Azure came from.

IBM is focused on customers buying their hardware and their expertise for installing it. AWS and Microsoft want to rent their expertise and software for building platforms. That difference in perspective is why IBM’s cloud aspirations were never going to take off to new heights. They couldn’t challenge for the top three places unless Google suddenly decided to shut down Google Cloud Engine. And no matter how hard they tried, Larry Ellison was always going to be nipping at their heels by pouring money into his cloud offerings to be on top. He may never get there but he is determined to make the best showing he can.

Putting On The Red Hat

Where does that leave IBM after buying Red Hat. Well, Red Hat sells software and services for it. But those services are all focused on integration. Red Hat has never built their own cloud platform. Instead, they work on everyone else’s platform effectively. They can deploy an OS or a container system on Amazon or Azure with no hiccups.

IBM has to realize now that they will never unseat Amazon. The momentum behind this 850-lb gorilla is just too much to challenge. The remaining players are fighting for a small piece of third or fourth place at this point. And yes, while Google has a comfortable hold on third place right now, they do have a tendency to kill projects that aren’t AdWords or the search engine homepage. Anything else lives in a world of uncertainty.

So, how does IBM compete? They need to leverage their expertise. They’ve sold off anything that has blinking lights, save for the mainframe division. They need to embrace their Global Services heritage and shepherd the SMEs that are afraid of the cloud. They need to help enterprises in the mid-range build into AWS and Azure instead of trying to make a huge profit off them and leave them high and dry. The days of making a fortune from Fortune 100 companies with no cloud aspirations are over. Just like the fight for cloud dominance, the battle lines are drawn and the prize isn’t one or two big companies. It’s a bunch of smaller ones.

The irony isn’t lost on me that IBM’s future lies in smaller companies. The days of “No one ever got fired for buying IBM” are long past in the rearview mirror. Instead, companies need the help of smart people to move into the cloud. But they also need to do it natively. They don’t need to keep running their old hybrid infrastructure. They need a trusted advisor that can help them build something that will last. IBM could be that company with the help of Red Hat. They could reinvent themselves all over again and beat the coming collapse of providers of infrastructure. As more companies start to look toward the cloud, IBM can help them along the path. But it’s going to take some realistic looks at what IBM can provide. And the end of IBM’s hope of running their own CSP.

Tom’s Take

I’m an old IBMer. At least, I interned there in 2001. I was around for all the changes that Lou Gerstner was trying to implement. I worked in IBM Global Services where they made the AS/400. As I’m fond of saying over and over again, IBM today is not Tom Watson’s IBM. It’s a different animal that changed with the times at just the right time. IBM is still changing today, but they aren’t as nimble as they were before. Their expertise lies all over the landscape of hot new tech, but people don’t want blockchain-enabled AI for IoT edge computing. They want a trusted partner than can help them with the projects they can’t get done today. That’s how you keep your foot in the door. Red Hat gives IBM that advantage. They key is whether or not IBM can see that the way forward for them isn’t as cloudy as they had first imagined.