About networkingnerd

Tom Hollingsworth, CCIE #29213, is a former network engineer and current organizer for Tech Field Day. Tom has been in the IT industry since 2002, and has been a nerd since he first drew breath.

Knowledge is Powerful and Needs to Be Shared

WritingPen

A tweet this morning from my friend Stephanie stood out in my timeline because she’s talking about something I’ve seen happen over and over again in my lifetime:

How many times have we seen this in our organizations? People want to hoard knowledge because they feel like it’s power. Maybe they’re worried that if anyone knew what they know it would mean they could get fired. Perhaps they enjoy holding the keys to the kingdom and not allowing anyone else to do something or know something they know. It could even be that they like the idea of mystery in the air and not allowing people to know the whole truth keeps things alive, as the founders of Coca-Cola and Colonel Sanders will happily tell you.

Over the years I’ve figured out that hoarding knowledge leads to ruin. I’ve been involved in so many scenarios were a lack of knowledge sharing ended up causing the kinds of problems that were easily avoidable if someone had just committed what they knew to memory. It wasn’t always malicious either. There is a lot of information that people collect incidentally and just flat out forget to write down until it’s too late.

The first time I truly realized how important good documentation was happened on a sad day in my career. My mentor, Wes Williams, tragically passed away during an otherwise uneventful workday during lunch. I wasn’t there but our office got the news shortly thereafter and we were stunned. No one in the office was even thinking about anything other than how tragic it was and how we were going to miss him. Eventually, one of the other engineers stood up and asked, “Does anyone know the password to Wes’s laptop?” We realized that Wes had a ton of knowledge about his clients and other operational things saved on his laptop, which was sitting unlocked on his desk. We rushed to keep it from going to sleep and to make sure that we changed the password to something we all knew. After that, the tedious process of copying all his data to a shared location began. Wes didn’t think to save it all somewhere else because he didn’t know he would need to until it was too late.

Information Is Currency

Since then I’ve tried very hard to share the knowledge that I have so that everyone benefits. The cynics in the organization may see information as a kind of currency to trade power. If you have to come to them to get the answers then they are important in the grand scheme of things. Kind of like a power broker of sorts.

My outlook is different. Having the knowledge written down somewhere means someone can learn what they need and do the job without needing to get me involved. Maybe they have other questions or need further explanation. That’s a great time to engage me. But I don’t need to be a gatekeeper for the basics or feel like I have to hoard it all to myself.

If you think that hoarding knowledge makes you impervious to firing or reassignment, you need to be careful with your hubris. No one is immune from reductions. External factors can limit your involvement with your career or organization before you know it. We use the idea of the “hit by a bus” test to explain how knowledge needs to be shared in the event of a sudden thing like the issue I mentioned above. But what if the situation develops slowly and you get caught up before realizing it’s too late to share what you know?

Power Corrupts

If you’re the person in the company that always calls the ISP to report outages because you know exactly how to navigate the automated trouble ticket system you have value. However, if you never write that down for your team members you’re going to limit your mobility out of your job. No matter how high you climb it will always be your responsibility. Worse yet, if you find yourself in a place where you can’t communicate what you know and someone else has to figure it out on their own your so-called “power” is useless now.

Information is a strange form of power because it loses all it’s inherent value as soon as it’s shared. If I know who the next supervisor is going to be before anyone else then I only have power as long as no one else knows. As soon as someone finds out I no longer control that information and therefore it’s useless. Instead of hoarding knowledge and information to wield as a cudgel to lord over others you should share it freely to ensure it’s not lost. How many advances in human history have been lost because no one wrote them down?

Knowing how something works isn’t a tool you should use to extract further value at the expense of capability. Even if the knowledge is something that shouldn’t be shared for reasons related to secrecy you still need to let people know there is a purpose and they either don’t need to know right now or it’s something that will be available to others. There are many times when I’ve been told that a decision-making process exists and I don’t need to understand it right now. I’m fine with that just so long as it’s written down somewhere. If it’s not then we may all find ourselves recreating a process that was a solved issue simply because someone took the secrets of how it worked with them when they moved on to a different role or out of the organization entirely.


Tom’s Take

There are times when I have to keep things secret. I know something and it’s not time to reveal it. But processes or knowledge about basic things should never fall into that category. It’s one thing to know that your supervisor is being promoted to section leader next week. It’s entirely different to hoard the knowledge of how the factory wireless network is configured because you are afraid if you don’t you’ll get fired. Commit every piece of knowledge you can to paper, physical or virtual, while you can. You never know when someone will need to know what you know. Knowledge is only valuable as currency when it’s shared. And that’s the kind of power that makes you valuable to us all.

Fast Friday Thoughts from the Woods

I’m at camp this week helping put on the second weekend of the Last Frontier Council Wood Badge course which is my idea of a vacation. I’m learning a lot, teaching a lot more, and having fun. But that does’t mean I’m not working too. Lots of fun conversations that make me recall the way people consume information, communicate what they know, and all too often overlook the important things they take for granted.

  • Why is IT one of the few disciplines that expects people to come in fully trained and do the job instead of learning while doing it? Is that because hiring managers don’t want to train people? Or is it because senior people are less likely to impart knowledge to protect their jobs? I don’t have a good answer but I know what the result looks like and it’s not something that’s positive, either for the people doing the job or how it’s perceived outside of IT.
  • There is a ton of value in doing something for real instead of just planning it and calling it good. DR plans need to be tested. Network changes need to be mocked up. No matter what kind of critical thing you’ll be doing you need to try it for real instead of just putting it on paper and hoping that it works. In theory, there’s no difference between theory and practice. In practice, there is.
  • Don’t forget to commit your knowledge to paper at some point in your career. You may know everything under the sun but when you’re not available you will be missed. You know more than you realize and you need to make sure that others benefit from what you’re capable of helping with. Those that hold the knowledge only have the power when they share it. Keeping it to yourself only hurts those that can benefit. Use your currency effectively.

Tom’s Take

These questions don’t have answers. They are designed to make you think about how you do things and what you can do to leave a legacy for the people that come behind you. Don’t be the story that everyone tells about that guy they hated. Be the inspiration for those that want to be like you in the future.

Racing On the Edge of Burnout

Exhibit A:

It’s been a year and more and I think a lot of us are on the ragged edge of burning out completely. Those that think they are superhuman and can just keep grinding away at things without acknowledging what’s going on are kidding themselves. I know I’m feeling it too even though I have a pretty decent handle on what’s going on. Let’s explore some of the ways it’s impacting us and what should be done, if anything can even be done.

Creativity Black Hole

I don’t feel like doing anything remotely creative right now. The cooking will get finished. The dishes will be done. The things in my floor will be picked up and put away. But beyond that? Good. Luck. I’m not feeling any kind of drive to do anything beyond that.

Remember when everyone was picking up quarantine skills? Baking, cooking, knitting, crocheting, home improvement, or even an instrument? Those were fun days filled with massive uncertainty and a need to distract ourselves from what might be coming next. However, those skill pickups are things that need time to work on and refine and continue to master. And now that the world is back in full swing we don’t have any more time than we did before. In fact, we have a lot less.

Now we face a choice of doing what we’ve always done before, albeit in a more restricted fashion, but now with the added pressure of an additional time sink staring us in the face. You can’t improve your cooking skills if you don’t cook. But when you don’t have a mountain of free time to devote to researching recipes or putting together the best shopping list or exploring new places to source ingredients you’re going to feel like it’s back to being a chore and end up churning out chicken nuggets and macaroni and cheese.

That’s what burnout looks like. When something you previously enjoyed becomes a chore like any other because you have no time to devote to it and get enjoyment from it. Whether you want to admit it or not all your creative pursuits feel like this right now. I know I find myself zoning out more often than not when it comes to free time. I don’t want to write or cook or learn to play the harmonica. I just want to spend a few moments not thinking about anything. And that’s what it feels like to be burned out.

How about doing things even remotely adjacent to work? Writing a coverage post from a presentation or recording a new podcast episode or a video? If it feels like actual work you’re probably going to avoid it just as much as you avoid the things you actually like to do. That means the rest of your creative output is going to suffer too.

Escape Velocity

Now that we know we’re burned out and don’t want to admit it, how do we fix it? The short answer is that we can’t. We’re still in the uncertain period of balancing work and creativity and other stuff going on. Our current battle is watching those two things fighting for supremacy. Work commands our attention to get the stuff done that pays the bills. Creative pursuits are clamoring for air because remember that cool time last year when we made all the sourdough bread? How do we make them both work?

Another quote that has been resounding with me recently is “If you prioritize your distractions over your responsibilities then your distractions are your responsibilities.”

We want to get away from the stuff that grinds on us. But when the things we use to get away become a grind then they just fall into the same place. We need to keep those distractions separate and use them when we need to as opposed to just taking a 30-minute break twice a day and working on our harmonica scales. When you associate your distraction with your responsibilities you stop liking it as much.

I use Scouting as one of my distractions. It’s basically my hobby at this point when you consider how much time I’ve invested into it. Yet, I find myself starting to get burned out on it as well. Part of that is my inability to say “no” to doing things. And that lack of time is wearing thin because I can’t be everywhere at once. I need to pull back from all the things that I’m doing because otherwise my hobby will become just another job that gets in the way of me relaxing and letting go.

Understanding each and every part of these battles is key to drawing the lines around what you need to keep burnout at bay. Our brains like to consume all the things around a hobby or topic and then walk away from it when it doesn’t produce the same kind of dopamine response. We have to teach our brains to enjoy a bit of what we like and not eat it all at once and get tired of it. That’s why scheduling time for things is so important. Otherwise you’ll grind yourself away to nothing. Make time for your responsibilities and your distractions and don’t mingle the two or you’re going to end up with some kind of unappetizing oatmeal of things.


Tom’s Take

I’m burned out. And I don’t want to admit it. Things keep slipping out of my head and I can’t seem to keep up like I want. Acknowledging it is the first step. Now that I know I’m burned out I can try and fix it by making those changes. Don’t soldier on and hope that you’re going to pull through it. Admit that you’re more burned out than you realize. You may not be completely gone yet but if you ignore it you soon will be. Instead, take the time to prioritize what you need to take care of and what you want to do to enjoy life. Schedule a hike. Make time to practice your instrument. But make sure you keep it segregated and keep your work life where it belongs. Don’t bake bread at 9am on a Monday and don’t send emails at 9pm on a Friday. And be kind to yourself. Your brain doesn’t like burnout any more than you do. Take a moment, take a breath, and take some time for you.

Real Life Ensues

Hey everyone! You probably noticed that I didn’t post a blog last week. Which means for the first time in over ten years I didn’t post one. The streak is done. Why? Well, real life decided to take over for a bit. I was up to my eyeballs in helping put on our BSA council Wood Badge course. I had a great time and completely lost track of time while I was there. And that means I didn’t get a chance to post something. Which is a perfect excuse to discuss why I set goals the way that I do.

Consistency Is Key

I write a lot. Between my blog here and the writing I do for Gestalt IT I do at least 2-3 posts a week. That’s on top of any briefing notes I type out or tweets I send when I have the energy to try and be funny. For someone that felt they weren’t a prolific writer in the past I can honestly say I spend a lot of time writing out things now. Which means that I have to try and keep a consistent schedule of doing things or else I will get swamped by some other projects.

I set the goal of one post a week because it’s an easy checkpoint for me. If it’s Friday and I haven’t posted anything here I know I need to do something. That’s why a large number of my posts come out on Friday. I keep a running checkpoint in my head to figure out what I want to cover and whether or not I’ve done it. When I can mark it down that I’ve done it then I can rest easy until next week.

With my Gestalt IT writing, I tend to go in batches. I try to find a couple of ideas that work for me and I plow through the posts. If I can get 3-4 done at a time it’s easy to schedule them out. For whatever reason it’s much easier to batch them on that side of the house than it is for me to work ahead on my personal blog.

If I don’t stay consistent I worry that the time I dedicate to blogging is going to be replaced by other things. It’s the same reason I feel like I need to stay on top of exercising or scheduling other meetings. Once the time that I spend taking care of something gets replaced by something else I feel like I never get that time back.

I know that doing things like that doesn’t work the way we would like it to work. Juggling writing without a firm schedule only leads to problems down the road. However, I feel like treating my blog posts like a single juggling ball being tossed up in the air over and over keeps my focus sharp. Unless something major comes along that absolutely steals my focus away I can make it work. I even thought to myself last Thursday that I needed to write something up. Alas, lack of sleep and other distractions get in the way before I could make it happen.

Writing Down the Routine

It’s important that you pencil in your routine to make it stick. Sure, after ten years I know that I need to write something each week. It’s finally ingrained in my head. But with other things, like exercise or harmonica practice or even just remembering to take out the garbage on Thursdays I need to have some way of reminding me or blocking time.

Using a reminders app or a journaling system is a great way to make that happen in your own head. Something you can refer to regularly to make sure that things are getting done. Whatever it is works just fine as long as you’re checking it and updating it regularly. Once you let that slip you’ll find yourself cursing it all because you’re halfway through a month with no updates.

Likewise, you need to make sure to block time on you calendar to take care of important things. My morning routine involves blocking time to go for a walk or a run. I also block time to write down posts and projects that are due. Putting those times on my calendar mean that I not only get notified when it’s time to start working on things but that other people can also see what I’m up to and schedule accordingly. Just be careful that you leave time to do other stuff. Also, while it’s important to use that term wisely don’t just sit there and do nothing if you’ve scheduled writing time. Write something down with the time you have. Even if it’s just a random idea or three. You never know when those half baked ideas can be leveraged to make full-blown magic!


Tom’s Take

I had every intention of writing my makeup post on Monday. Which slipped to Tuesday or Wednesday. And then I realized that real life is never going to stop. You have to make time for the important things. If that means writing something at midnight to post the next day or jogging up and down a muddy dirt road at ten minutes before midnight to ensure you close your last activity rings you have to do what needs to be done. Time isn’t going to magically appear. The gaps in your schedule will fill up. You need to be the one to decide how you’re going to use it. Let your priorities ensue in real life instead of the other way around.

Tom’s Corner and Turning Another Corner

Thanks to everyone that popped in for Tom’s Virtual Corner at Cisco Live Global 2021. It was a great time filled with chats about nothing in particular, crazy stories about unimportant things, and even the occasional funny picture. It was just was Tom’s Corner has always been. A way for the community to come together and be around each other in a relaxing and low-key environment. Maybe we couldn’t meet in person but we got together when we needed it the most.

There was also something else that Tom’s Corner has represented for me for the last year that I didn’t even catch until it was pointed out to me by my wonderful wife Kristin (@MrsNetwrkingNerd). Tom’s Corner was the start of something that made me feel better about everything.

Get On Up and Move

After Tom’s Virtual Corner in 2020, I was energized. I needed to get up and get things done after sitting in a chair for hours talking to all my absent friends and getting the energy I needed to feel after months of being locked away during a pandemic. I felt on top of the world for the first time in quite a while. And I needed to do something with that burst of energy.

So I got up and went for a walk. Exciting, right? I recorded a quick Periscope (RIP) video thanking everyone in the community for turning out and being a part of things even behind the keyboard. Then I kept going. Walking a mile or two and then coming back home. I noticed that after I did I closed my Apple Watch fitness rings for calories burned and exercise.

Rather than just counting steps like a Fitbit, Apple came up with their own measure of activity. I had a love/hate relationship with it. Some days it was easy to get my goal for standing once a minute every hour. Other days was impossible thanks to airplanes and such. Calories could be a breeze on days at events due to all the walking. Or they could be a disaster on days when I barely got out of my chair. Exercise goals were almost impossible. How could I find time to hit those?

After the amazing energy of Tom’s Virtual Corner in 2020, I decided to get serious about fitness. No more excuses. No more giving up and calling it a day because it was too hot outside or raining. I was going to do this because I needed to do it and the people that I care for in the community gave their time to be a part of something special to me. If they could do that I could find the time to do it too.

June 1 became June 30th and I was still going strong. I kept moving. My walking turned into getting back into running with a Couch to 5k beginner program I’ve done several times before. This time I was determined not to give up so easily. I thought back to the year before when I was able to run for a bit but ended up mixing up the walking and running and ultimately going back to being all but sedentary. The key for me this time was moving every day. Not just getting out every other day but walking one day and running the next.

The Running Nerd

Before I knew it, I was running more than I was walking. I was going further with every exercise. I felt better. I also noticed that I was losing weight, which was something that hadn’t happened the last time I was exercising. My health was improving. My belt had to tighten an extra notch. By the time of Mobility Field Day 5 at the end of July someone pointed out that I looked slimmer. I didn’t really notice it that much.

As the months went on more and more people remarked that I looked slimmer and fitter. It took four months before I stepped on a scale to be sure. And yes, I really was losing weight. Not a few pounds but a lot of them. As best as I can guess I was somewhere just over 265 pounds in this picture from February 2020:

Looking back at it now I can’t even believe it. I didn’t change my diet. All that happened was that we started cooking more and more meals at home thanks to the pandemic. I got better at making things I wanted to eat and knowing what was going into them. My every-other-day runs became more frequent. I was only walking twice a week now and running the other five days. My runs went from just barely 3 miles on occasion to at least 5 miles a day. My walks were up to 6 miles a day too.

Here’s a side by side comparison of what I looked like in June of 2020 and what I look like right now (March 2021):

I’m down three pants sizes. I weigh 210 pounds, down 55 lbs from my highest weight just over a year ago. My running is getting faster. I feel better. My doctor has told me everything looks much better than it did when he saw me last year for my checkup. When I got my BSA medical form filled out for my activities right before the pandemic last year I was worried I was going to be in trouble because I was too heavy for my height. When I had it done again last month I was well under the weight limit for the first time in a very long time.

Tom’s Corner is what set me on this path. The positive energy of the community kicked me forward and helped me see that I needed to be healthy. I needed to get back to a place where I didn’t feel self-conscious and worried about weight gain or my fitness to go on hikes or be active. The response of people when they see how much weight I’ve lost or how much I run has boosted my confidence even more and encouraged me to keep going.

I’ve closed my rings for 303 straight days. The light at the end of the year-long tunnel is almost here. I’m going to make it there even if I have to walk at 11:00pm at night to get those last few calories or do an Apple workout in the middle of a blizzard to make sure I didn’t miss my exercise goal. It’s something that drives me to wake up and make plans for the day. When I didn’t get my run in during Tom’s Corner this year I made sure to get it done as soon as I could so I didn’t miss out. I need to stay healthy and happy to ensure there are more Tom’s Corner meetups in the future.


Tom’s Corner

Motivation is hard. I’ve learned that lesson my whole life both the easy way and the hard way. Like a shark, if I stop moving I’m not going to start again. Planning how I’m going to make this all happen takes cycles of my day but results in a happier, healthier, more confident me. Thanks to my friends and fellow community members at Tom’s Corner I’ve transformed myself into something to be proud of. I’m often told that Tom’s Corner gives people something they need. Maybe it’s a place in the community. Or a friendly conversation when they need it the most. For me, Tom’s Corner was the kick in the pants I needed to be a better Tom. Thank you all for helping me turn that corner and get better.

Servant Leadership and Standing Out

LonePawn

My son is fifteen and he’s the typical teenager that either thinks he’s being asked to do way too much or he’s not getting recognized for what he’s accomplished. That last part is hard for him because he’s a bit humble and doesn’t like to tout his own work. I once told him that he didn’t need to do that because he stands out to the people that matter. He did the typical teenager thing where he fought me and said that no one noticed anything he did. I told him that if you do things the way they’re supposed to be done and don’t spend your whole day crowing about what you’re doing that the right people will most assuredly notice.

The worry that your work is going unnoticed isn’t unfounded among teenagers or adults. How many times have we asked ourselves in our daily work roles if we matter? It takes about two weeks worth of meetings in a typical IT department for you to see how things go. There are those that coast by with the knowledge they obtained years ago that have their niche and they intend to fill it. Their entire purpose is to avoid notice. They fear someone will figure out their job is redundant or they don’t want to get found out for having less capability than they really possess.

The other side of the coin is the group we all loathe. The braggarts and proclaimers. The ones that spend all of five minutes turning a wrench or typing in a command and then take the next hour to talk about how great a job they did or how things would have fallen apart without their help and expertise. These are the people that we’re all worried about. They don’t want to avoid attracting attention. In fact, they revel in it. They want the eyes on them. They have no fear of looking inadequate or incapable. They will put any of their skills up against anyone at any time so they can be seen as the best at something.

What about those in the middle? The ones that want to be recognized for what we are without sounding overly confident or even boastful? What about those that are learning but still unsure of their skills and don’t want to do something wrong just in case it’s going to cause problems? Or the people that only want to stand out because they work hard and feel like the braggarts get all the eyeballs? What if I told you there was an easier method to figure out where your stand?

The Pillars of Productivity

You’ve probably already noticed that your leaders come to you when something needs to be done, right? Not the usual kind of busy work or tasks that have little importance in the long run. I’m talking about the big tasks. The important projects. The things that can’t fail and need attention to detail because they have to be done right. You know, the kinds of things you get asked to do all the time?

Why is that? While you may not be able to see it from your vantage point, the best leaders are the ones that have been keeping score the entire time and they know how to get things done. They see those shadowy people in IT that have outdated skill sets trying to keep a low profile. They also have a record of all the times those people have asked to learn something new or stretch their horizon. While those team members may think they’re flying under the radar they’re even more visible to those that know what to look for and how to select them out of important stuff. Those job roles they covet because they’re easy and important? They’re also one step away from being automated and keep unmotivated team members away from critical things.

What about the loud mouths? The ones that don’t miss an opportunity to talk about all the important stuff they’re doing and how management came to them to make it happen. How is it that they get all the good jobs and you get to toil away on these never-ending projects? What makes them so special?

Did you ever stop and ask yourself how important those projects were? Did you ever think that perhaps they’re not as critical as one might believe? Perhaps even that they’re so unimportant that people feel the need to brag about them to make them seem more important? I’ve seen more than a few times in my career that the ones that are spending the most time talking about what they’re doing are actually doing the least amount of work. If the task or project is so critical and important how is it that this person has the time to sit around the water cooler sharing how things are only going smoothly because they’re getting it all done?

They Who Serve Their Fellows

The idea that people should do the work without expecting credit or praise isn’t as foreign as it sounds. It’s a component of the Servant Leadership concept popularized by Robert Greenleaf. Servant leadership is based around the idea that the leader of a group exists to serve their employees and help them grow and be better. One of the important pieces of this the need to do things not for praise but for the greater good.

Leaders that subscribe to the ideas behind servant leadership don’t look for the loud team members crowing about their accomplishments. Instead, they look for those that are quietly getting things accomplished. They measure output, not volume. They seek those that serve others as they do not those that serve themselves. And those are the people that are relied upon over and over again to get things accomplished.

You may be saying to yourself, “That’s not how it works around my office.” And that’s entirely true for a lot of places. If that’s the case, do me a favor and think for moment about the bosses that are always heaping praise onto those that can’t stop talking about their accomplishments. Are they they kinds of people that also brag about themselves and their roles? Do they spend more time announcing their work than they do working? I think you’ll find a strong correlation. Those that can’t see the inherent value in someone’s output often need to be told someone is working hard. And because they feel the same need to boast about what they do they take the discussion at face value instead of understanding it all at a deeper level. I would also venture to guess that they aren’t the kinds of managers that you’d want to be working for.

Since we’re doing thought exercises, let’s do another. Think about a manager that you would want to work for. Maybe it was someone that was concerned with your skill set and encouraged you to learn more. Or that checked in on you frequently to see how you were doing on a project instead of just assuming you weren’t getting the job done because you were super busy with it. Maybe that manager even stood up for you after a mistake or spent time to help you understand where you needed to improve outside of your skills. Sure, they probably gave you lots of projects and you may have even felt overwhelmed at times. But never because you felt your manager was trying to sabotage you or overload you. These are the kinds of people you’d go back to work with in an instant, right?

Unsurprisingly, those characteristics are a hallmark of a servant leader. Those leaders rarely find themselves at the head of a major company. They get the work done instead of bragging about the work they do. They tend to find positions running departments focused on service or deliverables. They’re the rock you count on instead of the face you send out to sell the work. Ironically enough, they don’t thrive in sales positions because they tend to cut through the garbage quickly and judge people on results instead of promises. These are the leaders we should be promoting. Because these are the leaders that recognize talent and nurture it instead of listening to the ones that can only talk about things instead of getting them done.


Tom’s Take

It took me a long time to realize the guiding hand behind servant leadership principles. I knew what it was as soon as someone told me about it because I’d seen it being used my whole life. From my parents to my formative managers to the present day I’ve had the luxury of working for several people that have embodied the ideals of someone that works to better others. Sure, I’ve had to deal with my fair share of braggarts and shadowy figures. But knowing there was someone that knew what I was capable of and that I stood out to them enough to be trusted with the important work has been enough for me so far. As I find myself looking over more and more work and coordinating projects I also see the things that my former managers saw. I see the need to recognize those that are putting in the work and not just telling me what they’re doing. I seek out those that don’t do it for praise or don’t hide from my gaze. And in doing so I hope I can inspire a new generation of servant leaders as well.

Basics First and Basics Last

This week I found my tech life colliding with my normal life in an unintended and somewhat enlightening way. I went to a store to pick up something that was out of stock and while I was there making small talk the person behind the counter asked me what I did for a living. I mentioned technology and he said that he was going to college for a degree in MIS, which just happens to be the thing I have my degree in. We chatted about that for a few more minutes before he asked me something I get asked all the time.

“What is the one thing I need to make sure I pay attention to in my courses?”

It’s simple enough, right? You’ve done this before and you have the benefit of hindsight. What is the one thing that is most important to know and not screw up? The possible answers floating through my head were all about programming or analytical methods or even the dreaded infrastructure class I slept through and then made a career out of. But what I said was the most boring and most critical answer one could give.

“You need to know the basics backwards and forwards.”

Basics Training

Why do we teach the basics? Why do we even call them that? And why are people so keen on skipping over all of them so fast to get to the cool stuff? You have to understand the basics before you even move on and yet so many want to get the “easy” stuff out of the way because memorizing the OSI model or learning how an array works in programming is mind-numbing.

The basics exist because we all need to know how things work at their most atomic level. We memorize the OSI model in networking because it tells us how things should behave. Sure, TCP/IP blows it away. However, if you know how packets are supposed to work with that model it informs you how you need to approach troubleshooting and software design and even data center layouts.

I’ll admit that I really didn’t pay much attention when I took my Infrastructure class twenty years ago. I was hell-bent on being a consultant or a database admin and who needed to know how a CPU register worked? What was this stupid OSI model they wanted me to know? I’ll just memorize it for the test and be done with it. Needless to say that the intervening years have shown me the folly of not paying attention in that class. If I went back today I’d ace that OSI test with my eyes closed.

The basics seem useless because we can’t do much with them right now. They’re just like Lego bricks. We need uniform pieces with predictable characteristics to help us understand how things are supposed to work together. Without that knowledge of how things work you can’t build on it. If you don’t understand the different between RAM and a hard disk you won’t be able to build systems that rely on both. Better yet, when technology changes to incorporate solid state disks and persistent memory storage you need the basics to understand how they are different and where you need to apply that knowledge.

I once picked up a Cisco Press CCIE study guide for the written exam to brush up on my knowledge before retaking the written. The knowledge in the book seemed easy to me. It was all about spanning tree configurations and OSPF area types and what BGP keepalives were. I felt like it was a remedial text that didn’t give me any new knowledge. That’s when I realized that they knowledge in the book wasn’t supposed to be new. It was supposed to be a reminder of what I already learned in my CCNA and CCNP courses. If anything in the text was truly new, was it something I should have already known?

It’s also part of the reason the CCIE is such a fun exam in the lab. You should already know the basics of how things like RIP and OSPF work. So let’s test those basics in new ways. Any of the training lab you can take from companies like INE or Micronics are filled with tricky little scenarios that make you take the basics and apply them outside the box. That’s because the instructors don’t need to spend time teaching you how RIP forms neighbor relationships or adjacencies. They want to see if you remember how that happens so you can apply it to a question designed to stretch your knowledge. You can only do that when you know the basics.

Graduation Day

Basics aren’t just for learning at the beginning. You should also brush up on them when you’re at the top of your game. Why? Because it will answer questions you might not know you had or explain strange things that rely on the architecture we long-ago forgot about because it seemed basic.

A fun example was years ago in the online game City of Heroes. The players can earn in-game currency to buy and sell things. Eventually the game economy got to the point where players were at the maximum amount of currency for a player. What was that number? It was just over two billion. Pretty odd place to stop, right? What made them think that was a good stopping point? Random chance? Desire to keep the amount of currency in circulation low? Or was there a different reason?

That’s when I asked a simple question: How would you store the currency value in the game’s code? The answer for every programmer out there is an integer. And what’s the maximum value for an integer? For a 32-bit value it’s around four billion. But what if you use a signed integer for some reason? The maximum value is just over two billion in each direction. So the developers used a 32-bit signed integer and that’s why the currency value was capped where it was.

Over and over again in my career I find myself turning back to the basics to answer questions about things I need to understand or solve. We really want the solutions to be complex and hard to understand and solve because that shows our critical thinking skills being applied. However, if you start with the basics approach you’ll find that the solutions to problems or the root causes are often defined by something very basic that has far-reaching consequences. And if you forget how those basics work you’re going to spend a lot of time chasing your tail looking for a complex solution to a simple problem.


Tom’s Take

I don’t think my conversation partner was hoping for the answer I gave him. I’m sure he wanted me to say that this high level course was super important because it taught all the secrets you needed to know in order to succeed in life. Everyone wants to hear that the most important things are exciting and advanced. Finding out that the real key to everything is the basics you learn at the beginning of your journey is disappointing. However, for those that master the basics and remember them at every step of their journey, the end of the road is just as advanced and exciting as it was when you stepped on it in the first place. And you get there with a better understanding of how everything works.

Who Pays The Price of Redundancy?

No doubt by now you’ve seen the big fire that took out a portion of the OVHcloud data center earlier this week. These kinds of things are difficult to deal with on a good day. This is why data centers have reductant power feeds, fire suppression systems, and the ability to get back up to full capacity. Modern data centers are getting very good at ensuring they can stay up through most events that could impact an on-premises private data center.

One of the issues I saw that was ancillary to the OVHcloud outage was the small group of people that were frustrated that their systems went down when the fire knocked out the racks where their instances lived. More than a couple of comments mentioned that clouds should not go down like this or asked about credit for time spent being offline or some form of complaints about unavailability. By and large, most of those complaining were running non-critical systems or were using the cheapest possible instances for their hosts.

Aside from the myopia that “cloud shouldn’t go down”, how do we deal with this idea that cloud redundancy doesn’t always translate to single instance availability? I think we need to step back and educate people about their responsibilities to their own systems and who ultimately pays for redundancy.

Backup Plans

I mentioned earlier that big accidents or incidents are the reasons why public cloud data centers have redundant systems. They have separate power feeds, generator power backups, extra cabling to prevent cuts from taking down systems, redundant cooling systems to keep things cold, and even redundant network connectivity across a variety of providers.

What is the purpose of all of this redundancy? Is it for the customers or the provider? The reason why all this exists is because the provider needs to ensure that it will take a massive issue to interrupt service to the customer. In a private data center you can be knocked offline when the primary link goes down or a UPS decides to die on you. In a public data center you could knock out ten customers with a dead rack PDU. So it is in their best interests to ensure that they have the right redundancy built in to keep their customers happy.

Public data centers pay for all of this redundancy to keep their customers coming back each month. If your data center gets knocked offline for some simple issue you can better believe you’re going to be shopping for a new partner. You’re paying for their redundancy with your month billing cycle. Sure, that massive generator may be sitting there just in case they need it. But they’re recouping the cost of having it around by charging you a few extra cents each cycle.

Extending the Backup Bubble

What about providing redundancy for your applications and servers, though? Like the OVHcloud issue above, why doesn’t the provider just back my stuff up or move it to a different server when everything goes down? I mean, vMotion and DRS do it in my private data center. Can’t they just check a box and make it happen?

There are two main reasons why this doesn’t happen in public cloud right now. The first is pretty easy. Having to backup, restore, replicate, and manage customer availability is going to take more than a few extra hands working on the customer infrastructure. Sure, they could configure vMotion (or something similar) to send your VMs to a different rack if one were to go offline. But who keeps tabs on that to make sure it happens? Who tests the failover to keep it consistent? What happens if there is a split brain scenario? Where is the password for that server stored?

You’re probably answering all of these questions off the top of your head because you’re the IT expert, right? So if the cloud provider is doing this for you, what are you going to be doing? Clicking “next” on the installation prompt? If your tasks are being done by some other engineer in the cloud, what are we paying you for again? Just like automation, having a cloud engineer do your job for you means we don’t need to pay you any longer.

The second reason is liability. Right now, if there is an incident that knocks a cloud provider offline they’re liable for the downtime for all their customers. Most of the contracts have a force majeure clause built into them that exempts liability for extraordinary circumstances, such as fire, weather, or even terrorist activity. That way the provider doesn’t need to pay you back for something there was no way to have foreseen. if there is some kind of outage caused by some technical issue then they will owe you for that one.

However, if the cloud provider starts managing your equipment and services for you then they are liable if there is an outage. If they screw up the vMotion settings or DRS starts getting too aggressive and migrates a VM to a bad host who is responsible for the downtime? If it’s you then you get yelled at and the company loses money. If it’s the provider managing for you then the provider gets yelled at, threatened, and possibility litigated to recover the lost income. See now why no provider wants to touch your stuff? We used to have a rule when I worked at a VAR that you better be very careful about which problems you decided to fix out of the project scope. Because as soon as you touch those they become your problems and you’re on the hook to fix them.

Lastly, the provider isn’t responsible for your redundancy for one other simple reason: you’re not paying them for it. If Amazon or Microsoft or Google offered a hosting package that included server replication and monitoring and 99.9999% uptime of your application data do you think it would cost the same as the basic instance pricing? You’d better believe it wouldn’t! These companies would be happy to sell you just what you’re looking for but you aren’t going to want to pay the price for it. It’s easy to build in the cost of a generator spread across hundreds or thousands of customers. But if you want someone backing your data up every day and validating it you’re going to be paying the lion’s share of the cost. And most of the people using low-cost providers for non-critical workloads aren’t going to want to pay extra anyway.


Tom’s Take

I get it. Cloud is nice and it’s always there. Cloud is easier to build out than your on-prem data center. Cloud makes life easy. However, cloud is not magic. Just because AWS doesn’t have outages every month doesn’t mean your servers won’t go down if you’re not backing them up. Just because you have availability zones you can use doesn’t mean data is magically going to show up in Oregon because you want it to. You’re going to have to pay the cost to make your infrastructure redundant on top of the provider infrastructure. That means money invested in software, time invested in deployment, and workers invested in making sure it all checks out when you need it to be there to catch your outages. Don’t assume the cloud is going to solve every one of your challenges. Because if it did we’d be paying a lot more for it and IT workers wouldn’t be getting paid at all.

Tom’s Virtual Corner at Cisco Live Global 2021 – Anniversary

We made it through the year that was March 2020. Here were are on the other side trying to find out whatever this normal is supposed to look like. We’re not out of the woods yet but we do know that things aren’t going to be back to the way they were any time soon. That includes the events that we enjoyed traveling to and hanging out at.

Cisco Live has made the decision to go virtual again this year. One can’t blame them to be honest. Travel uncertainty and the potential liability of having a huge event just didn’t make sense. If you thought the old Conference Crud was bad you really don’t want this new-and-improved version! Cisco has also decided that one global event makes more sense than several events scattered across the calendar. That means that Cisco Live Europe and Cisco Live US are now global and happening at the end of March instead of January or June.

With the announcement that everything will be virtual again this year it also means that the social aspect of the event is going to be virtual as well. As much as we would have liked to hang out at Tom’s Corner in Las Vegas and catch up it just won’t be happening. I am a bit disappointed because this year was going to be the tenth anniversary of Tom’s Corner. It’s been a decade since I plopped down in an uncomfortable chair and set out on a quest to meet people. I was hoping to bring back the stool and the table and the corner live in person but it appears that we’ll have to do it some other time.

In the interim, we’re still going to be getting together in just a few weeks! We’re going to be hosting a Webex call just like we did last year so we can all hang out and chat throughout the day. Here are the details:

Date: March 29, 2021

Time: 7am PT – Whenever we get bored!

Where: Virtually via Webex (link will be in the calendar invite)

If you want to take part in the virtual corner, you need to either send an email to tom@networkingnerd.net or send a DM/PM/RRM/DMVPN to me on social media with the email address you want your calendar invitation sent to. If there is no email address I can’t send out the invitation! We’ll send them out as we can to get everyone signed up. You can pop in and out whenever you want. We’ll try to create breakout rooms for those that just want to chat on specific topics and such. It’ll be fun! Provided, that is, that we all keep to some basic ground rules:

  • Disruptive attendees may be removed at the discretion of the hosts.
  • Wheaton’s Law applies. If you think you’re violating it, you probably are.
  • Be respectful of the community. We want this to be positive for everyone and that means everyone is responsible for keeping it that way.

If you have any questions about Tom’s Corner or things in general just ask! I want to thank Gestalt IT for allowing us to host this meeting and continuing to support community engagement and involvement. A huge part of my job is doing things like this and I still continue to be thrilled to have the support of Stephen Foskett, Claire Chaplais, and the rest of the Gestalt IT team when it comes to taking part of my day to interact with you all, virtual or otherwise.

I wish things could have been different as far as Cisco Live this year but we’re going to make the most of what we have. That means virtual for now but rest assured the the next time we can all meet in person we’re going to have a birthday party for Tom’s Corner that will be sung of in legends to come!

Building Snowflakes On Purpose

We all know that building snowflake networks is bad, right? If it’s not a repeatable process it’s going to end up being a problem down the road. If we can’t refer back to documentation to shows why we did something we’re going to end up causing issues and reducing reliability. But what happens when a snowflake process is required to fix a bigger problem? It’s a fun story that highlights where process can break down sometimes.

Reloaded

I’ve mentioned before that I spent about six months doing telephone tech support for Gateway computers. This was back in 2003 so Windows XP was the hottest operating system out there. The nature of support means that you’re going to be spending more time working on older things. In my case this was Windows 95 and 98. Windows 98 was a pain but it was easy to work on.

One of the most common processes we had for Windows 98 was a system reload. It was the last line of defense to fix massive issues or remove viruses. It was something that was second nature to any of the technicians on the help desk:

  1. Boot from the Gateway tools CD and use GWSCAN to write zeros to the hard drive.
  2. Reboot from the CD and use FDISK to partition the hard disk.
  3. Format the drive.
  4. Insert the Windows 98 OS CD and copy the CAB installation files to a folder on the hard drive.
  5. Run setup from the hard drive and let it complete.
  6. When Windows comes back up, insert the driver CD and let install all the drivers.

The whole process took two or three phone calls to complete. Any time you got to something that would take more than fifteen minutes to complete you logged the steps in the customer trouble ticket and had them call back when it was completed. The process was so standard that it had its own acronym in the documentation – FFR, which stood for “FDISK, Format, Reload”. If you told someone where you were in the process they could finish it no problem.

Me, Me, ME

The whole process was manual with lots of steps and could intimidate customers. At some point in the development process the Gateway folks came up with a solution for Windows ME that they thought worked better. Instead of the manual steps of copying files and drivers and such, Gateway loaded the OS CD with a copy of the image they wanted for their specific model type. The image was installed using ImageCast, an imaging program that just dropped the image down on the drive without the need to do all the other steps. In theory, it was simple and reduced call times for the help desk.

In practice, Windows ME was a disaster to work on. The ImageCast program worked about half the time. If you didn’t pick the right options in the reload process it would partition the hard drive and add a second clean copy of WinME without removing the first one. It would change the MBR to have two installations to choose from with the same identifiers so users would get confused as to which was which. And the image itself seemed to be missing programs and drivers. The fact that there was a Driver CD that shipped with the system made us all wonder what the real idea behind this “improved” process was.

Because Windows ME was such a nightmare to reload, our call center got creative. We had a process that worked for Windows 98. We had all the files we needed on the disks. Why not do it the “right” way? So we did. Informally, the reload process for Windows ME was the same as Windows 98. We would FFR Windows ME boxes and make sure they looked right when they came back. No crazy ImageCasting programs or broken software loads.

The only issue? It was an informal snowflake process. It worked much better but if someone called the main help desk number and got another call center they would be in the middle of an unsupported process. The other call center tech would simply start the regular process and screw up the work we’d done already. To counter that, we would tell the customer to call our special callback voicemail box and not do anything until we called them back. That meant the reload process took many more hours for an already unhappy customer. The end result was better but could lead to frustrations.

Let It Snow

Was our informal snowflake process good or bad? It’s tough to say. It led to happier customers. It meant the likelihood of future support calls was lower because the system was properly reloaded instead of relying on a broken image. However, it was stressful for the tech that worked on the ticket because they had to own the process the whole way through. It also meant that you had to ensure the customer wouldn’t call back and disrupt the process with someone else on the phone.

The process was broken and it needed to be fixed. However, the way to fix the broken process wasn’t easy to figure out. The national line had their process and they were going to stick to it. We came up with an alternative but it wasn’t going to be adopted. Yet, we still kept using our process as often as possible because we felt we were right.

In your enterprise, you need to understand process. Small companies need processes that are repeatable for the ease of their employees. Large companies need processes to keep things consistent. Many companies that face regulatory oversight need processes followed exactly to ensure compliance. The ability to go rogue and just do it the way you want isn’t always desired.

If you have users that are going around the process you need to find out why. We see it all the time. Security rules that get ignored. Documentation requirements that are given the bare minimum effort. Remember the shutdown dialog box in Windows Server 2003? Most people did a token entry before rebooting. And having eighteen entries of “;lkj;kljl;k” doesn’t help figure out what’s going on.

Get your users to tell you why they need a snowflake process. Especially if there’s more than one person relying on it. The odds are very good that they’ve found hiccups that you need to address. Maybe it’s data entry for old systems. Perhaps it’s problem with the requirements or the order of the steps. Whatever it is you have to understand it and fix it. The snowflake may be a better way to do things. You have to investigate and figure it out otherwise your processes will be forever broken.


Tom’s Take

Inbound tech support is full of stories like these. We find a broken process and we go around it. It’s not always the best solution but it’s the one that works for us. However, building a toolbox full of snowflake processes and solutions that no one else knows about is a path to headaches. You need to model your process around what people need to accomplish and how they do it instead of just assuming they’re going to shoehorn their workflow into your checklist. If you process doesn’t work for your people, you’d better get a handle on the situation before you’re buried in a blizzard of snowflakes.