Category Archives: Musing

Security Is Bananas

Posted on by
2

I think we’ve reached peak bombshell report discussion at this point. It all started this time around with the big news from Bloomberg that China implanted spy chips into SuperMicro boards in the assembly phase. Then came the denials from Amazon and Apple and event SuperMicro. Then started the armchair quarterbacking from everyone, including TechCrunch. From bad sources to lack of technical details all the way up to the crazy conspiracy theories that someone at Bloomberg was trying to goose their quarterly bonus with a short sale or that the Chinese planted the story to cover up future hacking incidents, I think we’ve covered the entire gamut of everything that the SuperMicro story could and couldn’t be.

So what more could there be to say about this? Well, nothing about SuperMicro specifically. But there’s a lot to say about the fact that we were both oblivious and completely unsurprised about an attack on the supply chain of a manufacturer. While the story moved the stock markets pretty effectively for a few days, none of the security people I’ve talked to were shocked by the idea of someone with the power of a nation state inserting themselves into the supply chain to gain the kind of advantage needed to execute a plan of collection of data. And before you scoff, remember we’re only four years removed from the allegation that the NSA had Cisco put backdoors into IOS.

Why are we not surprised by this idea? Well, for one because security is getting much, much better at what it’s supposed to be doing. You can tell that because the attacks are getting more and more sophisticated. We’ve gone from 419 scam emails being deliberately bad to snare the lowest common denominator to phishing attacks that fool some of the best and brightest out there thanks to a combination of assets and DNS registrations that pass the initial sniff test. Criminals have had to up their game because we’re teaching people how to get better at spotting the fakes.

Likewise, technology is getting better at nabbing things before we even see them. Take the example of Forcepoint. I first found out about them at RSA this year. They have a great data loss prevention (DLP) solution that keeps you from doing silly things like emailing out Social Security Numbers or credit card information that would violate PCI standards. But they also have an AI-powered analysis engine that is constantly watching for behavioral threats. If someone does this on accident once it could just be a mistake. But a repeated pattern of behavior could indicate a serious training issue or even a malicious actor.

Forcepoint is in a category of solutions that are making the infrastructure smarter so we don’t have to be as vigilant. Sure, we’re getting much better at spotting things to don’t look right. But we also have a lot of help from our services. When Google can automatically filter spam and then tag presented messages as potentially phishing (proceed with caution), it helps me start my first read through as a skeptic. I don’t have to exhaust my vigilance for every email that comes across the wire.

The Dark Side Grows Powerful Too

Just because the infrastructure is getting smarter doesn’t mean we’re on the road to recovery. It means the bad actors are now exploring new vectors for their trade. Instead of 419 or phishing emails they’re installing malware on systems to capture keystrokes. iOS 12 now has protection from fake software keyboards that could capture information when something is trying to act as a keyboard on-screen. That’s a pretty impressive low-level hack when you think about it.

Now, let’s extrapolate the idea that the bad actors are getting smarter. They’re also contending with more data being pushed to cloud providers like Amazon and Azure. People aren’t storing data on their local devices. It’s all being pushed around in Virginia and Oregon data centers. So how do you get to that data? You can’t install bad software on a switch or even a class of switches or even a single vendor, since most companies are buying from multiple vendors now or even looking to build their own networking stacks, ala Facebook.

If you can’t compromise the equipment at the point of resale, you have to get to it before it gets into the supply chain. That’s why the SuperMicro story makes sense in most people’s heads, even if it does end up not being 100% true. By getting to the silicon manufacturer you have a entry point into anything they make. Could you imagine if this was Accton or Quanta instead of SuperMicro? If there was a chip inside every whitebox switch made in the last three years? If that chip had been scanning for data or relaying information out-of-band to a nefarious third-party? Now you see why supply chain compromises are so horrible in their potential scope.

This Is Bananas

Can it be fixed? That’s a good question that doesn’t have a clear answer. I look at it like the problem with the Cavendish banana. The Cavendish is the primary variant of the banana in the world right now. But it wasn’t always that way. The Gros Michel used to be the most popular all the way into the 1950s. It stopped because of a disease that infected the Gros Michel and caused entire crops to rot and die. That could happen because bananas are not grown through traditional reproductive methods like other crops. Instead, they are grafted from tree to tree. In a way, that makes almost all bananas clones of each other. And if a disease affects one of them, it affects them all. And there are reports that the Cavendish is starting to show signs of a fungus that could wipe them out.

How does this story about bananas relate to security? Well, if you can’t stop bananas from growing everywhere, you need to take them on at the source. And if you can get into the source, you can infect them without hope of removal. Likewise, if you can get into the supply chain and start stealing or manipulating data a low level, you don’t need to worry about all the crazy protections put in at higher layers. You’ll just bypass them all and get what you want.

Tom’s Take

I’m not sold on the Bloomberg bombshell about SuperMicro. The vehement denials from Apple and Amazon make this a more complex issue than we may be able to solve in the next couple of years. But now that the genie is out the bottle, we’re going to start seeing more and more complicated methods of attacking the merchant manufacturers at the source instead of trying to get at them further down the road. Maybe it’s malware that’s installed out-of-the-box thanks to a staging server getting compromised. Maybe it’s a hard-coded backdoor like the Xiamoi one that allowed webcams to become DDoS vectors. We can keep building bigger and better protections, but eventually we need to realize that we’re only one threat away from extinction, just like the banana.

The Why of Security

Posted on by
1
Security is a field of questions. We find ourselves asking all kinds of them all the time. Who is trying to get into my network? What are they using? How can I stop them? But I feel that the most important question is the one we ask the least. And the answer to that question provides the motivation to really fix problems as well as conserving the effort necessary to do so.

The Why’s Old Sage

If you’re someone with kids, imagine a conversation like this one for a moment:
Your child runs into the kitchen with a lit torch in their hands and asks “Hey, where do we keep the gasoline?”
Now, some of you are probably laughing. And some of you are probably imagining all kinds of crazy going on here. But I’m sure that most of you probably started asking a lot of questions like:
  • – Why does my child have a lit torch in the house?
  • – Why do they want to know where the gasoline is?
  • – Why do they want to put these two things together?
  • – Why am I not stopping this right now?
Usually, the rest of the Five Ws follow soon afterward. But Why is the biggest question. It provides motivation and understanding. If your child had walked in with a lit torch it would have triggered one set of responses. Or if they had asked for the location of combustible materials it might have elicited another set. But Why is so often overlooked in a variety of different places that we often take it for granted. Imagine this scenario:
An application developer comes to you and says, “I need to you open all the ports on the firewall and turn off the AV on all the machines in the building.”
You’d probably react with an immediate “NO”. You’d get cursed at and IT would live another day as the obstruction in “real development” at your company. As security pros, we are always trying to keep things safe. Sometimes that safety means we must prevent people from hurting themselves, as in the above example. But, let’s apply the Why here:
  • – Why do they need all the firewall ports opened?
  • – Why does the AV need to be disabled on every machine?
  • – Why didn’t they tell me about this earlier instead of coming to me right now?
See how each Why question has some relevance to things? If you start asking, I’d bet you would figure some interesting things out very quickly. Such as why the developer doesn’t know what ports their application uses. Or why they don’t understand how AV heuristics are triggered by software that appears to be malicious. Or the value of communicating to the security team ahead of time for things that are going to be big requests!

Digging Deeper

It’s always a question of motivation. More than networking or storage or any other facet of IT, security must understand Why. Other disciplines are easy to figure out. Increased connectivity and availability. Better data retention and faster recall. But security focuses on safety. On restriction. And allowing people to do things against their better nature means figuring out why they want to do them in the first place. Too much time is spent on the How and the What. If you look at the market for products, they all focus on that area. It makes sense at a basic level. Software designed to stop people from stealing your files is necessarily simple and focused on prevention, not intent. It does the job it was designed to do and no more. In other cases, the software could be built into a larger suite that provides other features and still not address the intent. And if you’ve been following along in security in the past few months, you’ve probably seen the land rush of companies talking about artificial intelligence (AI) in their solutions. RSA’s show floor was full of companies that took a product that did something last year and now magically does the same thing this year but with AI added in! Except, it’s not really AI. AI provides the basis for intent. Well, real AI does at least. The current state of machine learning and advanced analytics provides a ton of data (the what and the who) but fails to provide the intent (the why). That’s because Why is difficult to determine. Why requires extrapolation and understanding. It’s not as simple as just producing output and correlating. While machine learning is really good at correlation, it still can’t make the leap beyond analysis. That’s why humans are going to be needed for the foreseeable future in the loop. People provide the Why. They know to ask beyond the data to figure out what’s going on behind it. They want to understand the challenges. Until you have a surefire way of providing that capability, you’re never going to be able to truly automate any kind of security decision making system.

Tom’s Take

I’m a huge fan of Why. I like making people defend their decisions. Why is the one question that triggers deeper insight and understanding. Why concentrates on things that can’t be programmed or automated. Instead, why gives us the data we really need to understand the context of all the other decisions that get made. Concentrating on Why is how we can provide invaluable input into the system and ensure that all the tools we’ve spent thousands of dollars to implement actually do the job correctly.

A Matter of Perspective

Posted on by
3

Have you ever taken the opportunity to think about something from a completely different perspective? Or seen someone experience something you have seen through new eyes? It’s not easy for sure. But it is a very enlightening experience that can help you understand why people sometimes see things entirely differently even when presented with the same information.

Overcast Networking

The first time I saw this in action was with Aviatrix Systems. I first got to see them at Cisco Live 2018. They did a 1-hour presentation about their solution and gave everyone an overview of what it could do. For the networking people in the room it was pretty straightforward. Aviatrix did a lot of the things that networking should do. It was just in the cloud instead of in a data center. It’s not that Aviatrix wasn’t impressive. It’s the networking people have a very clear idea of what a networking platform should do.

Fast forward two months to Cloud Field Day 4. Aviatrix presents again, only this time to a group of cloud professionals. The message was a little more refined from their first presentation. They included some different topics to appeal more to a cloud audience, such as AWS encryption or egress security. The reception from the delegates was the differencue between night and day. Rather than just be satisfied with the message that Aviatrix put forward, the Cloud Field Day delegates were completely blown away! They loved everything that Aviatrix had to say. They loved the way that Aviatrix approached a problem they had seen and couldn’t quite understand. How to extend networking into the cloud and take control of it.

Did Aviatrix do something different? Why was the reaction between the two groups so stark? How did it happen this way? I think it is in part because networking people talk to a networking company and see networking. They find the things they expect to find and don’t look any deeper. But when the same company presents to an audience that doesn’t have networking on the brain for the entirety of their career it’s something entirely different. While a networking audience may understand the technology a cloud audience may understand how to make it work better for their needs because they can see the advantages. Perspective matters in this case because people exposed to new ideas find ways to make them work in ways that can only be seen with fresh eyes.

Letting Go of Wires

The second time I saw an example of perspective at play was at Mobility Field Day 3 with Arista Networks. Arista is a powerhouse in the data center networking space. They have gone up against Cisco and taken them head-to-head in a lot of deals. They have been gaining marketshare from Cisco in a narrow range of products focused on the data center. But they’re also now moving into campus switching as well as wireless with the acquisition of Mojo Networks.

When Arista stepped up to present at Mobility Field Day 3, the audience wasn’t a group of networking people that wanted to hear about CloudVision or 400GbE or even EOS. The audience of wireless and mobility professionals wanted to hear how Arista is going to integrate the Mojo product line into their existing infrastructure. The audience was waiting for a message that everything would work together and the way forward would be clear. I don’t know that they heard that message, but it wasn’t because of anything that Arista did on purpose.

Arista is very much trying to understand how they’re going to integrate Mojo Networks into what they do. They’re also very focused on the management and control plane of the access points. These are solved problems in the wireless world right now. When you talk to a wireless professional about centralized management of the device or a survivable control plane that can keep running if the management system is offline they’ll probably laugh. They’ve been able to experience this for the past several years so far. They know what SDN should look like because it’s the way that CAPWAP controllers have always operated. Wireless pros can tell you the flaws behind backhauling all your traffic through a controller and why there are much better options to keep from overwhelming the device.

Wireless pros have a different perspective from networking people right now. Things that networking pros are just now learning about are the past to wireless people. Wireless pros are focused more on the radio side of the equation than the routing and switching side. That perspective gives the wireless crowd a very narrow focus on solving some very hard problems but it does make them miss the point that their expertise can be invaluable to helping both networking pros and networking companies see how to take the best elements of wireless networking control mechanisms and implement them in such a way as to benefit everyone.

Tom’s Take

For me, the difficulty in seeing things differently doesn’t come from having an open mind. Instead, it comes from the fact that most people don’t have a conception of anything outside their frame of reference. We can’t really comprehend things we can’t conceive of. What you need to do to really understand what it feels like to be in someone else’s shoes is have someone show you what it looks like to be in them. Observe people learning something for the first time. Or see how they react to a topic you know well. Odds are good you might just find that you will know it better because they helped you understand it better.

It’s About Time and Project Management

Posted on by
1

I stumbled across a Reddit thread today from /u/Magician_Hiker that posed a question I’ve always found fascinating. When we work on projects, it always seems like there is a disconnect between the project management team and the engineering team doing the work. The statement posted at the top of this thread is as follows:

Project Managers only plan for when things go right.

Engineers always plan for when things go wrong.

How did we get here? And can anything be done about it?

Projecting Management

I’ve had a turn or two at project management. I got my Project+ many years back, and even more years before that I had to learn all about project management in college. The science behind project management is storied and deep. The idea of having someone assigned to keep things running on task and making sure all the little details get taken care of is a huge boon as the size of projects grow.

As an engineer, can you imagine trying to juggle three different installations across 5 different sites that all need to be coordinated together? Can you think about the effort needed to make sure that everything works together and is done on time? The thought alone probably gives you hives.

Project managers are capable of juggling lots of things in their professional capabilities. That means keeping all the dishes cooking at the same time and making sure that everything is done on time to eat dinner. It also means that people need to know about timelines and how those timelines intersect and can impact the execution of multiple phases of a project. Sure, it’s easy to figure out that we can’t start installing the equipment until it arrives on the dock. But how about coordinating the installers to be on-site on the right day knowing that the company is drop shipping the equipment to three different receiving docks? That’s a bit harder.

Project managers need to know timelines for things because they have to juggle everything together. If you’ve ever had the misfortune to need to use a Gantt chart you’ll know what I’m talking about. These little jewels have all the timeline needs of a project visualized for everyone to figure out how to make things happen. Stable time is key to a project. Estimates need to make sense. You can’t just spitball something and hope it works. If part of your project timeline is off in either direction, you’re going to get messed up further down the line.

Predictability

Project timelines need to be consistent. Most people try to err on the side of caution when trying to make them work. They fudge the numbers and pad things out a bit so that everything will work out in the end. Even if that means that there may be a few hours when someone is sitting around with nothing to do.

I worked with a project manager that jokingly told me that the way he figured out the timing for an installation project was to take the units from his engineers and double it and move to the next time unit. So hours became days, and days became weeks. We chuckled about this at the time, but it also wasn’t surprising when their projects always seemed to talk a LOT longer than most people budgeted for.

The problem with inflated numbers is that no customer is going to want to pay for wasted time. If you think it’s hard to get a customer to buy off on an installation that might take 30 hours try getting them to pay when they are telling you your engineers were sitting around for 10 of those hours. Customers only want to pay for the hours worked, not the hours spent on the phone trying to locate shipments or trying to figure out what this weird error message is.

Likewise, trying to go the other direction and get things done more quickly than the estimate is a recipe for disaster too. There’s even a specific term for it: crashing (sounds great, eh?). Crashing a project means adding resources to a project or removing items from the critical execution path to make a deadline or complete something earlier. If you want a textbook example of why messing with a project timeline is a bad idea, go read or watch The Martian. The first resupply mission is a prime example of this practice in action and why it can go horribly wrong.

These are all great reasons why cloud is so appealing to people. Justin Warren (@JPWarren) did a great presentation a couple of years ago about what happens when projects run late and why cloud fixes that:

Watch that whole video and you’ll understand things from a project manager’s point of view. Cloud is predictable and stable and it always works the same way. The variance on things is tight. You don’t have to worry about projects slipping up or taking too much time. Cloud removes uncertainty and doubt about execution. That’s something that project managers love.

Tom’s Take

I used to get asked to quote my projected installation times to the sales managers for projects. Most of the time, I’d give them an estimate that I felt comfortable with and that would be the end of it. One day, I asked them about why a 10-hour project was quoted as 14 on an order. The sales manager told me that they’d developed “Tom Time”, which was 1.4 times the amount of whatever I quoted. So, 10 hours became 14 and 20 hours became 28, and so on. When I asked why I was told that engineers often run into problems and don’t think to account for it. So project managers need to build in the time somehow. Perhaps that’s one of the reasons why software defined and cloud are more attractive. Because there isn’t any Tom Time involved.

Friday Musings on Network Analytics

Posted on by
2

I’ve been at Networking Field Day this week, and as always the conversations have been great and focused around a variety of networking topics. One that keeps jumping out at me is network analytics. There’s a few things that have come up that were especially interesting to me:

  • Don’t ask yourself if networking monitoring is not worth your time. Odds are good you’re already monitoring stuff in your network and you don’t even realize it. Many networking vendors enable basic analytics for troubleshooting purposes. You need to figure out how to build that into a bigger part of your workflows.
  • Remember that analytics can integrate with platforms you’re already using. If you’re using ServiceNow you can integrate everything into it. No better way to learn how analytics can help you than to setup some kind of ticket generation for down networks. And, if that automation causes you to get overloaded with link flaps you’ll have even more motivation to figure out why your provider can’t keep things running.
  • Don’t discount open source tools. The world has come a long way since MRTG and Cacti. In fact, a lot of the flagship analytics platforms are built with open source tools as a starting point. If you can figure out how to use the “free” versions, you can figure out how to implement the bigger stuff too. The paid versions may look nicer or have deeper integrations, but you can bet that they all work mostly the same under the hood.
  • Finally, remember that you can’t possible deal with all this data yourself. You can collect it but parsing it is like trying to drink from a firehose of pond water. You need to treat the data and then analyze that result. Find tools (probably open source) that help you understand what you’re seeing. If it saves you 10 minutes of looking, it’s worth it.

Tom’s Take

Be sure to say tuned to our Gestalt IT On-Premise IT Roundtable podcast in the coming weeks for more great discussion on the analytics topic. We’ve got an episode that should be out soon that will take the discussion of the “expense” of networking analytics in a new direction.

Independence, Impartiality, and Perspective

Posted on by
1

In case you haven’t noticed recently, there are a lot of people that have been going to work for vendors and manufacturers of computer equipment. Microsoft has scored more than a few of them, along with Cohesity, Rubrik, and many others. This is something that I see frequently from my position at Tech Field Day. We typically hear the rumblings of a person looking to move on to a different position early on because we talk to a great number of companies. We also hear about it because it represents a big shift for those who are potential delegates for our events. Because going to a vendor means loss of their independence. But what does that really mean?

Undeclaring Independence

When people go to work for a manufacturer of a computing product, the necessarily lose their independence. But that’s not the only case where that happens. You can also not be truly independent if you work for reseller. If your company specializes in Cisco and EMC, are you truly independent when discussion Juniper and NetApp? If you make your money by selling one group of products you’re going to be unconsciously biased toward them. If you’ve been burned or had the rug pulled out from under you by a vendor, you may be biased against them.

Likewise, if you work yourself in an independent consulting business, are you honestly and truly independent? That would mean you’re making no money from any vendor as your customer. That means no drafting of whitepapers, no testing, and no interaction with them that involves compensation. This falls into the same line of thinking as the reseller employee. Strictly speaking, you aren’t totally independent.

Independence would really mean not even being involved in the market. A public defender is truly independent of IT. A plumber is independent of IT. They don’t have biases. They don’t care either way what type of computer or network solution they use. But you don’t consider a public defender or a plumber to be an expert on IT. Because in order to know anything about IT you have to understand the companies. You have to talk to them and get what they’re doing. And that means you’re going to start forming opinions about them. But, truthfully, real independence is not what you’re looking for here.

Partial Neutrality

Instead of independence, which is a tricky subject, what you should look for is impartiality. Being impartial means treating everything fairly. You accept that you have bias and you try to overcome it. For example, judges are impartial. They have their own opinions and thoughts about subjects. They rule based on the law. They aren’t independent of the justice system. Instead, they do their best to use logic and rules to decide matters.

Likewise, people in IT should strive to be impartial. Instead of forming an opinion about something without thought we should try to look at all aspects of the ideas before we form our opinions. I used to be a huge fan of IBM Thinkpad laptops. I carried one for many years at my old job. Today, I’m a huge fan of MacBooks. I use one every day. Does that mean that I don’t like Windows laptops any more? It means that I don’t use them enough to have a solid opinion. I may compare some of the features I have on my current laptop against the ones that I see, but I also recognize that I am making that comparison and that I need to take it into account when I arrive at my decision.

Extending this further, when making decisions about how we analyze a solution from a company, we have to take into account how impartial we’re going to be in every direction. When you work for a company that makes a thing, whether it be a switch or a laptop or a car, you’re going to be partial to the thing you make. That’s just how people are. We like things we are involved in making. Does that mean we are incapable of admiring other things? No, but it does mean that we have some things to overcome to truly be impartial.

The hardest part of trying to be impartial is realizing when you aren’t. Unconscious bias creeps in all the time. Thoughts about the one time you used a bad product or ate bad food in a restaurant make you immediately start thinking about buying something else quickly. Even when it’s unfounded we still do it. And we have to recognize when we’re doing it in order to counter it.

Being impartial isn’t easy on purpose. Being able to look at all sides of an issue, both good and bad, takes a lot of extra effort. We don’t always like seeing the good in something we dislike or finding the bad parts of an opinion we agree with. But in order to be good judges of things we need to find the balance.

Tom’s Take

I try my best to be impartial. I know it’s hard to do. I have the fortune to be independent. Sure, I’m still a CCIE so I have a lot of Cisco knowledge. I also have a lot of relationships with people in the industry. A lot of those same people work at companies that are my customers. In the end, I have to realize that I need to work hard to be impartial every day. That’s the only way I can say that I’m independent and capable of evaluating things on their own merits. It’s just a matter of having the right perspective.

Finding Value in Cisco Live 2018

Posted on by
4

The world famous Cisco Live Sign picture, 2018 edition

Another Cisco Live has come and gone. Overall it was a fun time for many. Catching up with friends. Meeting people for the first time. Enjoying the balmy Orlando weather. It was a chance to relive some great times for every one. But does Cisco Live 2018 dictate how the future of the event will go?

Packing The Schedule

Did you get a chance to attend any of the social events at Cisco Live? There were a ton. There were Tweetups and meet ups and special sessions galore. There was every opportunity to visit a lounge or area dedicated to social media presence, Boomerang videos, goofy pictures, or global outreach. Every twenty feet had something for you to do or some way for you to make an impact.

In fact, if you went to all of these things you probably didn’t have time for much else. Definitely not time for the four or five keynote addresses. Or a certification test. Or the classes and sessions. In fact, if you tried to do everything there was to do at Cisco Live, you’d probably not sleep the whole week. There’s almost as much stuff to do outside the conference sessions as there is to do in them.

But is it too much? Are the activities around the learning sessions taking away from the conference itself? Think about something like the Big Ideas theater this year. In theory, it’s a great way to get people to attend sessions that are not specifically related to tech. You can introduce new ideas, especially those that are focused more on changing the world. But you’re also competing for time away from sessions that are focused on new products or building better architectures.

Every booth in the World of Solutions is designed to draw you in and keep you there. For the sponsors of the event it’s important to have conversations about their products and solutions. For Cisco people, it’s almost like they’re competing with the sessions to give you different content or a chance to interview people. Is that how things should be? I can understand the desire of DevNet wanting to change the way people look at programmable networking, for example. But every other little booth like Cisco Advanced Services or the Emergency Response Vehicle? Those feel more like attractions designed to show off rather than educate.

Paying the Piper

And what does all this cost in the long run? Sure, I love having extra features around the conference as much as the next person. But to what end? Things don’t pay for themselves. Every conference has a budget. Every piece of entertainment and every showcase booth costs money in some way or another. And how does that all get paid for? By us, the attendees.

It’s no secret that attending conferences isn’t cheap. A full conference pass for Cisco Live is around $2,000. In the past, there were cheaper options for just attending for the people networking aspect of things. But, with the growth of DevNet and other “included” options at the conference, Cisco needed to find a way to pay for them this year.

I’m not going to spend a lot of time going into the Imagine Pass issue right now because I want to sit down and have an honest discussion with Cisco about the pros and cons of the approach. But it is very important that we examine what we’re getting for the increased cost. There has to be a significant value for people to want to be a part of the event if the costs go up. The way to do that is to create compelling reasons to want to be at Cisco Live.

The way not to do that is to lock the content behind gates. Some of the things at Cisco Live this year were placed in areas that were not easy to access. One of my personal pet peeves is the NetVet lounge. I’m going to start this off by saying that I was a NetVet for many years before I moved to Tech Field Day. I’m no longer a NetVet. However, until 2013 the NetVet lounge was one of the de facto social hangout places. Now, it’s another area where you can get coffee and snacks.

Why does the NetVet lounge bother me? Because of the placement. Front and center across the aisle from the on-site Cisco Store (which took the place of the Social Media hub from 2013). Why does the NetVet lounge get to be outside the World of Solutions? Aside from the historical reasons, I can’t think of a good reason. You need to have a full conference pass to achieve NetVet status. A full conference pass gets you into the World of Solutions. Why not have NetVets meet there?

The obvious reason is that the World of Solutions closes. Yet the NetVet lounge does too. And the hours are pretty similar. Why not move the NetVet lounge into the World of Solutions and give that space to the Social Media folks. There are no restrictions on getting into the Social Media Hub. Why not have them front and center? Again, aside from the “tradition” of having the NetVet lounge outside the World of Solutions I can’t think of a good reason.

Tom’s Take

I love Cisco Live. I realized this year that I’ve been to thirteen of them. Every year since 2006. The conference has changed and grown. The focus has shifted. But the people remain the same. With the changes in the way that the pass structure the people may not be there much longer. We, as IT professionals, need to decide what’s important and give some feedback. We need to make it constructive and honest. Point out what works and what doesn’t. Don’t whine, but offer direct criticism. We can only make the conference we want by telling the people what we need. That’s how you make Cisco Live a place to be for now and for the future.

Conference Impostor Syndrome

Posted on by
1

In IT we’ve all heard of Impostor Syndrome by now. The feeling that you’re not just a lucky person that has no real skills or is skating by on the seat of their pants is a very real thing. I’ve felt it an many of my friends and fellow members of the community have felt it too. It’s easy to deal with when you have time to think or work on your own. However, when you take your show on the road it can creep up before you know it.

Conferences are a great place to meet people and learn about new ideas. It’s also a place where your ideas will be challenged and put on display. It’s not to difficult to imagine meeting a person for the first time at a place like Cisco Live or VMworld and not feeling little awe-inspired. After all, this could be a person whose works you’ve read for a long time. It could be a person you look up to or someone you would like to have mentor you.

For those in the position of being thrust into the limelight, it can be extremely difficult to push aside those feelings of Impostor Syndrome or even just a general level of anxiety. When people are coming up to you and thanking you for the content you create or even taking it to further extremes, like bowing for example, it can feel like you’re famous and admired for nothing at all.

What the members of the community have to realize is that these feelings are totally natural. You’re well within your rights to want to shy away from attention or be modest. This is doubly true for those of us that are introverts, which seems to happen in higher numbers in IT.

How can you fight these feelings?

Realize You Are Enough. I know it sounds silly to say it but you have to realize that you are enough. You are the person that does what they can every day to make the world a better place in every way you can. It might be something simple like tweeting about a problem you fixed. It may be as impressive as publishing your own network automation book. But you still have to stop and realize you are enough to accomplish your goals.

For those out there that want to tell their heroes and mentors in the community how awesome they are, remember that you’re also forcing them to look at themselves in a critical light sometimes. Some reassurances like, “I love the way you write” or “Your ability to keep the podcast going smoothly” are huge compliments that people appreciate. Because they represent skills that are honed and practiced.

Be The Best You That You Can Be. This one sounds harder than it might actually be. Now that you’ve admitted that you’re enough, you need to keep being the best person that you can be. Maybe that’s continuing to write great content. Maybe it’s something as simple as taking a hour out of your day to learn a new topic or interact with some new people on social media. It’s important that you take your skill set and use it to make things better overall for everyone.

For those out there that are amazed at the amount of content that someone can produce or the high technical quality of what they’re working on, remember that we’re all the same. We all have the same 24 hours in the day to do what we do. So the application of the time spent studying or learning about something is what separates leaders from the pack.

Build Up Others Slowly. This one is maybe the hardest of all. When you’re talking to people and building them up from nothing, you need to be sure to take your time in bringing them along. You can’t just swamp them with knowledge and minute details about their life that have gleaned from reading blogs or LinkedIn. You need, instead, to bring people along slowly and build them up from nothing into the greatest person that you know.

This works in reverse as well. Don’t walk up to someone and start listing off their requirements like a resume. Instead, give them some time to discuss it with you. Let the person you’re talking to dictate a portion of the conversation. Even though you may feel the need to overwhelm with information to justify the discussion you should let them come to their place when they are ready. That prevents the feeling of being overwhelmed and makes the conversation much, much easier.

Tom’s Take

It’s very easy to get lost in the world of feeling inadequate about what others think of you. It goes from adulation and excitement to an overwhelming sense of dread that you’re going to let people down. You have to fix that by realizing that you’re enough and doing the best you can with what you have. If you can say that emphatically about yourself then you are well on the way to ensuring that Conference Imposter Syndrome is something you won’t have to worry about.

Avoiding A MacGyvered Network

Posted on by
Reply

Ivan Pepelnjak has an interesting post up today about MacGyver-ing in the network. He and Simon Milhomme are right that most small-to-medium sized networks are pretty much non-reference architectures and really, really difficult to manage and maintain properly on the best of days. On the worst of days, they’re a nightmare that make you want to run screaming into the night. But why?

One Size Never Fits All

Part of the issue is that reference architectures and cookie-cutter designs aren’t made for SMEs. Sure, the large enterprise and cloud providers have their own special snowflakes. But so too do small IT shops that have been handed a pile of parts and told to make it work.

People like Greg Ferro and Peyton Maynard-Koran believe this is due to vendors and VARs pushing hardware and sales cycles like crazy. I have attributed it to the lack of real training and knowledge about networking. But, it also has a lot to do with the way that people see IT as a cost center. We don’t provide value like marketing. We don’t collect checks like accounting. At best, we’re no different than the utility companies. We’re here because we have to be.

Likewise, when IT is forced into making decisions based on some kind of rebate or sale we tend to get stuck with the blame when things don’t work correctly. People that wouldn’t bat an eye at buying a new Rolex or BMW get downright frugal when they’re deciding which switches to buy to run their operation for the next 10 years. So, they compromise. Surely you all can make this switch work? It only has half the throughput as the one you originally specced but it was on sale!

So, stuck with bad decisions and no real documentation or reference points, we IT wizards do what we can with what we have. Just like Angus MacGyver. Sure, those networks are a pain to manage. They’re a huge issue when it’s time to upgrade to the next discount switch. And given the number of fires that we have to fight on a daily basis we never get a chance to go back and fix when we nailed up in the first place. Nothing is more permanent than a temporary fix. And no install lasts longer than one that was prefaced with “let me just get this working for the next week”.

Bespoke Off The Rack

So, how do we fix the MacGyver-ing? It’s not going to be easy. It’s going to be painful and require us to step out of our comfort zones.

  1. Shut Up And Do As I Say. This one is hard. Instead of having management breathing down your neck to get something installed or working on their schedule, you need to push back. You need to tell people that rushing is only going to complicate things. You need to fire back when someone hands you equipment that doesn’t meet your spec. You need to paraphrase the famous Shigeru Miyamoto – “A delayed network is eventually good, but a rushed installation is bad forever.”
  2. Document Like It Will Be Read Aloud In Court. Documentation isn’t just a suggestion. It’s a necessity. We’ve heard about the “hit by a bus” test. It’s more than just that, though. You need to not only be able to replace yourself but also to be able to have references for why you made the decisions you did. MacGyver-ing happens when we can’t remember why we made the decisions we did. It happens when we need to come up with a last minute solution to an impossible problem created by other people (NSFW language). Better to have that solution documented in full so you know what’s going on three years from now.
  3. Plan Like It’s Happening Tomorrow. Want to be ready for a refresh? Plan it today. Come back to your plan. Have a replacement strategy for every piece of equipment in your organization. Refresh the plan every few months. When someone comes to you with a new idea or a new thing, write up a plan. Yes, it’s going to be a lot of spinning your wheels. but it’s also going to be a better solution when someone comes to you and says that it’s time to make a project happen. Then, all you need to do is pull out your plan and make it happen. Imagine it like the opposite of a DR plan – Instead of the worst case scenario this is your chance to make it the best case.

Tom’s Take

There’s a reason the ringtone on my phone has been the theme from MacGyver for the last 15 years. I’m good at building something out of nothing. Making things work when they shouldn’t. And, as bad as it sounds, sometimes that’s what’s needed. Especially in the VAR world. But it shouldn’t be standard operating procedure. Instead, make your plan, execute it when the time comes, and make sure no one pushes you into a situation where MacGyver-ing is the last option you have.

Time To Get Back To Basics?

Posted on by
8

I’ve had some fascinating networking discussions over the past couple of weeks at Dell Technologies World, Interop, and the spring ONUG meeting. But two of them have hit on some things that I think need to be addressed in the industry. Both Russ White and Ignas Bagdonas of the IETF have come to me and talked about how they feel networking professionals have lost sight of the basics.

How Stuff Works

If you walk up to any network engineer and ask them to explain how TCP works, you will probably get a variety of answers. Some will try to explain it to you in basic terms to avoid getting too in depth. Others will swamp you with a technical discussion that would make the protocol inventors proud. But still others will just shrug their shoulders and admit they don’t really understand the protocol.

It’s a common problem when a technology gets to the point of being mature and ubiquitous. One of my favorite examples is the fuel system on an internal combustion engine. On older cars or small engines, the carburetor is responsible for creating the correct fuel and air mixture that is used to power the cylinders. Getting that mix right is half science and half black magic. For the people that know how to do it properly it’s an easy thing that they can do to drive the maxim performance from an engine. For those that have tried it and failed, it’s something best left alone to run at defaults.

The modern engine uses fuel injection. It’s a black box. It can be reprogrammed or tinkered with but it’s something that is tuned in different ways from the traditional carburetor. It’s not something that’s designed to be played around with unless you really know what you’re doing. The technology has reached the point where it’s ubiquitous and easy to use, but very difficult to repair without a specialized skill set.

Most regular car drivers will look under the hood and quickly realize they know nothing about what’s going on. Some technical folks will be able to figure out what certain parts do by observing their behavior. But if you ask those same people how a fuel injection system or carburetor works they’ll probably give you a blank stare.

That’s the issue we find in modern networking. We’ve been creating VLANs and BGP route maps for years. Some people have spent their entire careers tuning multicast or optimizing layer 2 interconnects. But if you corner them and ask them how the protocol works or how best to design an architecture that takes advantage of their life’s work they can’t tell you aside from referencing some old blog post or some vendor’s validated design on their hardware.

Russ and Ignas each touch on something important. In the good old days before there were a hundred certification guides and a thousand SRNDs people had to do real work to find the best solution for a problem. They had to put pencil to paper and sort out the mess before they could make something work. That’s where the engineering side of the network comes from.

Today, it’s more “plug and play”. You drop in pieces of a solution and they should work together. In practice, that usually means all the pieces have to be from the same vendor or from approved partner sources. And anything that goes awry will need a team of experts and many, many consulting hours to figure out.

Imagine if we could only install networks without understanding how they work. Could you see a world where everything we install from a networking perspective is a black box like a fuel injector? That’s the case to a certain degree with cloud networking today. We don’t see what’s going on under the surface. We can only see what the interface exposes to us. That’s fine as long as the applications we are using support the things we’re trying to do with them. But when it comes to being able to fix the network at the level we’re used to seeing it could be difficult if not downright impossible.

Learning The Ropes

But, moreover, are the networking professionals that are configuring these networks even capable of making those changes? Does anyone other than Narbik really understand how EIGRP works? Facebook seems to think that lightweight messaging packets for routing protocols are outdated. So they used ZeroMQ without understanding why that’s a bad idea for slow speed links. They may understand how a routing protocol works in theory, but they don’t completely understand how it’s supposed to work in extreme cases.

Can we teach people the basics and understanding of protocols and design that they need in order to make proper designs outside of vendor reference documents? It’s a tall order for sure. Most blog posts are designed to talk about features or solve problems. Most literature from creators is designed to make their new widget work correctly. Very little documentation exists about integration or design. And a good portion of what does exist is a bit outmoded and needs to be spruced up.

We, as the stewards of networking, need to help this process along. We need to spend more time talking about design and theory. We need to dissect protocols and help people understand how to use the tools they have rather than hoping someone will build the best mousetrap ever to solve each piece of a complicated puzzle. We need to teach people to be thinkers and problem solvers. And, yes, that does mean a bit less complaining about things like vendor code quality and VAR behavior.

Why? Because those people are empowered by a lack of knowledge. Customers aren’t idiots. They have business reasons for the things they do. Our technology needs to support their business needs. Yes, that means we need to think critically about what we’re doing. And yes, they may mean eating our words now and then to avoid a showdown about something that’s ultimately unimportant in the long run.

If we increase the amount of knowledge about the important topics like design and protocols it should make the overall level of understanding go up for everyone. That means better designs. More integrated technology. Less reliance on being force-fed the bare minimum information necessary to make things work. And that means things will run faster and much more smoothly. Which is a win for everyone.

Tom’s Take

I’ll be the first to admit that I don’t know the dirty mechanics of Frame Relay switching or how to tune OSPF Hello timers for non-standard link types. It’s a skill I don’t use every day. But I know where to find them if I need them. And I know that it can help in certain situations where you see odd protocol behavior. Likewise, I know that if I need to go design a network for someone I need to get a lot of information about business needs and build the best network I can with the constraints that I have. Things like budget and time are universal. But one of those constraints shouldn’t be lack of knowledge about protocols and good design. Those are two things that should be ingrained into anyone that wants to have a title of “senior” anything.