About networkingnerd

Tom Hollingsworth, CCIE #29213, is a former network engineer and current organizer for Tech Field Day. Tom has been in the IT industry since 2002, and has been a nerd since he first drew breath.

The Double-Edged Grindstone

Are you doing okay out there? I hope that you’re well and not running yourself thin with all the craziness still going on. Sometimes it seems like we can’t catch a break and that work and everything keep us going all the time. In fact, that specific feeling and the resulting drive around it is what I wanted to talk about today.

People have drive. We want to be better. We want to learn and grow and change. Whether it’s getting a faster time running a 5K or learning new skills to help our career along. Humans can do amazing things given the right motivation and resource availability. I know because I taught myself a semester of macroeconomics in a Waffle House the night before the final exam. Sure, I was groggy and crashed for a 10-hour nap after the final but I did pass!

It’s that kind of ability to push ourselves past our limits that both defines us and threatens to destroy us. I’m a huge fan of reading and fiction. Growing up I latched on to the Battletech novels, especially those written by Michael A. Stackpole. In his book Lost Destiny there is a great discussion about honing your skills and what it may end up costing you:

Were we to proceed, it would be the battle of the knife against the grindstone. Yes, we would get sharper, we would win great victories, but in the end, we would be ground away to nothing.

 

Running in Place

That quote hit home for me during my CCIE lab attempt training. I spent my free time after work labbing everything I could get my hands on. I would log on about 8pm every night after my eldest went to sleep and lab until midnight. Every night was the same chore. No time for television or reading anything other than a Cisco Press book. Instead, I drilled until I could provision EtherChannels in my sleep and could redistribute routes without a second thought in four different ways. I felt like I was getting so much accomplished!

I also felt tired all the time. I had no outlet to relax. I spent every waking minute of my free time focused on sharpening my skills. As above, you could practically hear the edge of the knife on the grindstone. I was razor-focused on completing this task. As I learned later in life, the joys of hyper focus in ADHD had a lot to do with that.

It wasn’t until I got through my lab that the true measure of this honing process hit home. I spent the weekend after my attempt hanging out with some friends away from technology and the whole time I felt unsettled. I watched horrible movies on the Sci-Fi channel and couldn’t get comfortable. I was antsy and wound up, even though I’d just completed a huge milestone! It wasn’t until about a week later that I was finally able to put a name to this anxiety. I listened to the little voice in my head repeating over and over again in my downtime: “You really should be studying something right now.”

I was in search of my next grindstone. My knife was sharp, but I knew it could be sharper still with the next certification or piece of knowledge. It took me a while before I could quiet that voice and focus on restoring some semblance of my life as I had known it before my lab attempts. Even now there are times when I feel like I should be studying or writing or creating something instead of unwinding with a book or a camping trip to the woods.

The rush of dopamine that we get from learning new things or performing skills to perfection cannot be understated. It makes us feel good. It makes us want to keep doing it to continue that stream of good feelings. We could focus on it to the detriment of our other hobbies or our social lives. And that was in the time before when we could go out whenever we wanted! In the current state of the pandemic it’s easy to get wrapped up in something without the ability to force yourself to walk away from it, as anyone with a half-filled room full of a new hobby can tell you.

Setting Goals with Limits

What’s the solution then? Do we just keep grinding away until there’s nothing left? Do we become the best left-handed underwater basket weaver that has ever existed? Do we keep forcing ourselves to run the same video game level over and over again until we’re perfect, even if that means not getting out of our house for weeks at a time? Can we do something repeatedly until we are destroyed by it?

The key is to set goals but to make them in such a way as to set limits on them. We do this all the time in the opposite direction. When we have a task we don’t like to do we force ourselves to do it for a set amount of time. Maybe it’s an hour of reconciling the checkbook or thirty minutes of exercise. But if it’s something you enjoy you have to set limits on it as well to avoid that burnout.

Find something you really like to do, such as reading a book. However, instead of devouring that book until it’s finished in one sitting and staying up until 4am to finish it, set an alarm to cut yourself off after an hour or two. Be honest with yourself. When the alarm goes off, stop reading and do something else. You could even pair it with a task you like less to use the little dopamine boost to help you through your other activity.

It’s not fun to stop doing something we like doing even when it’s something that is going to make us a better person or better employee. However, you’ll soon see that having that extra time to reinforce what you’ve learned or collect your thoughts does more for you that just binging something until you’re completely exhausted by it.

Remember when TV shows came out weekly? We had to wait until next Thursday for the new episode? Remember how much you looked forward to that day? Or maybe even the return of the new season in the fall? That kind of anticipation helps motivate you. Being able to consume all the content in one sitting for 8-10 hours leaves you feeling great at first. Later, when your dopamine goes back to normal you’re going to feel down. You’ll also realize you can’t get the same hit again because you have consumed your current resource of it. By limiting what you can do at one time you’re going to find that you can keep that great feeling going without burning yourself out.

Yes, this does absolutely apply to studying for things. Your brain needs time to lock in the knowledge. If you’ve ever tried to memorize something you know that you need to spend time thinking of something else and come back to that item before you really learn it. The only thing that transfers knowledge from short-term memory to long-term memory is time. There are no shortcuts. And the more you press the edge of the knife agains the stone the more you lose in the long run because the available resources are just gone.


Tom’s Take

I’m not qualified to delve into the psychoanalysis part of all this stuff. I just know how it works for me because that’s who I am. I can shut out the world and plow through something for hours at a time if I want. I’ve done it many times in my career with both work and personal tasks. But it’s taken a long time for me to finally realize that sprinting like that for extended periods of time will eventually wear you away to nothing. Even the best runners in the world need to rest. Even the smartest people in the industry need to not think about things for a while. You do too. Take some time today or tomorrow or even next week to set goals and limits for yourself. You’ll find that you enjoy the things you do and learn more with those limits in place and you’ll wind up a happier, healthier person. You’ll be sharp and ready instead of a pile of dust under the grindstone.

Planning For The Worst Case You Can’t Think Of

Remember that Slack outage earlier this month? The one that happened when we all got back from vacation and tried to jump on to share cat memes and emojis? We all chalked it up to gremlins and went on going through our pile of email until it came back up. The post-mortem came out yesterday and there were two things that were interesting to me. Both of them have implications on reliability planning and how we handle the worst-case scenarios we come up with.

It’s Out of Our Hands

The first thing that came up in the report was that the specific cause for the outage came from an AWS Transit Gateway not being able to scale fast enough to handle the demand spike that came when we all went back to work on the morning of January 4th. What, the cloud can’t scale?

The cloud is practically limitless when it comes to resources. We can create instances with massive CPU resources or storage allocations or even networking pipelines. However, we can’t create them instantly. No matter how much we need it takes time to do the basic provisioning to get it up and running. It’s the old story of eating an elephant. We do it one bite at a time. Normally we tell the story to talk about breaking a task down into smaller parts. In this case, it’s a reminder that even the biggest thing out there has to be dealt with in small pieces as well.

Slack learned this lesson the hard way. Why? Because they couldn’t foresee a time when their service was so popular that the amount of traffic rushing to their servers crushed the transit gateways. Other companies have had to learn this lesson the hard way too. Disney crawled on launch day because of demand. The release of a new game that requires downloading and patching on Day One also puts stress on servers. Even the lines outside of department stores on Black Friday (in less pandemic-driven years) are examples of what happens when capacity planning doesn’t meet demand.

When you plan for your worst case scenario, you have to think the unthinkable. Instead of asking yourself what might happen if everyone logs on at the same time you also need to ask when happens if they try and something goes wrong. I spent a lot of time in my former job thinking about simple little exercises like VDI boot storms, where office workers can push a storage system to the breaking point by simply turning all their machines on at the same time. It’s the equivalent of being on a shared network resource like a cable modem during the Super Bowl. There aren’t any resources available for you to use.

When we plan for capacity, we have to realize that even our most optimistic projections of usage are going to be conservative if we take off or go viral. Rather than guessing what that might be, take an hour every six months and readjust your projections. See how fast you’re growing. Plan for that crazy scenario where everyone decides to log on at the same time on a day where no one has had their coffee yet. And be ready for what happens when someone throws a wrench into the middle of the process.

What Happens When It Goes Wrong?

The second thing that came up in the Slack post-mortem that is just as worrisome was the behavior of the application when it realized there was a connection timeout. The app started waiting for the pathway to be open again. And guess what happened when AWS was able to scale the transit gateway? Slack clients started hammering the servers with connection requests. The result was something akin to a Distributed Denial-of-Service (DDoS) attack.

Why would the Slack client do this? I think it has something to do with the way the developers coded it and didn’t anticipate every client trying to reconnect all at once. It’s not entirely unsound thinking to be honest. How could we live in a world where every Slack user would be disconnected all at once? Yet, we do and it did and look what happened?

Ethernet figured this part out a long time ago. The CSMA/CD method for detecting collisions on a layer 2 connection has an ingenious solution for what happens when a collision is detected. Once it realizes that there was a problem on the wire it stops what is going on and calculates a random backoff timer based on the number of detected collisions. Once that timer has expired it attempts to transmit again. Because there has to be another station involved in a collision incident both stations do this. The random element of the timer calculation ensures that the likelihood of both stations choosing to transmit again at the same time is very, very low.

If Ethernet behaved like the Slack client did we would never resolve collisions. If every station on a layer 2 network immediately tried to retransmit without a backoff timer the bus would be jammed constantly. The architects of the protocol figured out that every station needs a cool off period to clear the wire before trying again. And it needs to be different for every station so there is no overlap.

Slack really needs to take this idea into account. Rather than pouncing on a connection as soon as it’s available there needs to be a backoff timer that prevents the servers from being swamped. Even a few hundred milliseconds per client could have prevented this outage from getting as big as it did. Slack didn’t plan beyond the worst case scenario because they never conceived of their worst case scenario coming to pass. How could it get worse than something we couldn’t imagine happening?


Tom’s Take

If you design systems or call yourself a reliability engineer, you need to develop a hobby of coming up with disastrous scenarios. Think of the worst possible way for something to fail. Now, imagine it getting worse. Assume that nothing will work properly when there is a recovery attempt. Plan for things to be so bad that you’re in a room on fire trying to put everything out while you’re also on fire. It sounds very dramatic but that’s how bad it can get. If you’re ready for that then nothing will surprise you. You also need to make sure you’re going back and thinking of new things all the time. You never know which piece is going to fail and how it will impact what you’re working on. But thinking through it sometimes gives you an idea of where to go when it all goes wrong.

Managing Leaders, Or Why Pat Gelsinger Is Awesome

In case you missed it, Intel CEO Bob Swan is stepping down from his role effective February 15 and will be replaced by current VMware CEO Pat Gelsinger. Gelsinger was the former CTO at Intel for a number of years before leaving to run EMC and VMware. His return is a bright spot in an otherwise dismal past few months for the chip giant.

Why is Gelsinger’s return such a cause for celebration? The analysts that have been interviewed say that Intel has been in need of a technical leader for a while now. Swan came from the office of the CFO to run Intel on an interim basis after the resignation of Brian Krzanich. The past year has been a rough one for Intel, with delays in their new smaller chip manufacturing process and competition heating up from long-time rival AMD but also from new threats like ARM being potentially sold to NVIDIA. It’s a challenging course for any company captain to sail. However, I think one key thing makes is nigh impossible for Swan.

Management Mentality

Swan is a manager. That’s not meant as a slight inasmuch as an accurate label. Managers are people that have things and look after them. Swan came from the financial side of the house where you have piles of resources and you do your best to account for them and justify their use. It’s Management 101. Managers make good CEOs for a variety of companies. They make sure that the moves are small and logical and will pay off in the future for the investors and eventually the workers as well. They are stewards first and foremost. When their background comes from something with inherent risk they are especially stewardly.

You know who else was a manager? John Sculley, the man who replaced Steve Jobs at Apple back in 1983. Sculley was seen as a moderating force to Jobs’ driving vision and sometimes reckless decision making skills. Sculley piloted the ship into calm waters at first but was ultimately sent packing because his decisions were starting to make less and less sense, such as exploring options to split Apple into separate companies and taking on IBM head-to-head on their turf.

Sculley was ousted and Jobs returned to Apple in 1993. It wasn’t easy at first but eventually the style of Jobs started producing results. Things like the iPod, iMac, and eventually the iPhone came from his vision. He’s a leader in that regard. Leaders are the ones that jump out and take risks to make big results. Leaders are people like John Kennedy that give a vision of going to the moon in a decade without the faintest idea how that might happen. Leadership is what drives companies.

Leaders, however, are a liability without managers. Leaders say “let’s go to the moon!” Managers sit down and figure out how to make that happen without breaking the budgets or losing too many people along the way. Managers are the grounded voices that guide leaders. Without someone telling a leader of the challenges to overcome they won’t see the roadblocks until the drive right into them.

Leaders without brakes on their vision have no reality to shape it. Every iMac has an Apple Lisa. Every iPod has the iPod Hi-Fi. Even the iPhone wasn’t the iPhone until the App Store came around against the original vision of Apple’s driving force. To put it another way, George Lucas is a visionary leader in filmmaking. However, when he was turned loose without management of his process we ended up with the messy prequel trilogy. Why was Empire Strikes Back such a good film? Because it had people like Lawrence Kasdan involved managing the process of Lucas creating art. They helped focus the drive of a leader and make the result something great.

Tech Leadership

Let’s bring this discussion back to Intel and Pat Gelsinger. I know he is the best person to lead Intel right now. I know that because Gelsinger is very much a tech leader. He has visions for how things need to be and he can see how to get there. He knows that reducing costs and reaving product lines at Intel isn’t going to make them a better company down the road no matter what the activist investors have to say on the matter. They may have wanted regime change when they petitioned the board back in December, but they may find the new king a bit harder to deal with.

Gelsinger is also a manager. Going from CTO to being COO at EMC and eventually CEO at VMware has tempered his technical chops. You can’t hope to run a company on crazy ideas and risky bets. Steve Jobs had people like Tim Cook in the background keeping him as grounded in reality as possible. Gelsinger picked up these skills in helming VMware and I think that’s going to pay off for him at Intel. Rather than running out to buy another company to augment capabilities that will never see the light of day, someone like him can see the direction that Intel needs to go and make it happen in a collected manner. No more FPGA acquisitions that never bear fruit. No more embarrassing sales of the mobile chip division because no one could capitalize on it.

Pat Gelsinger is the best kind of technical manager. I saw it in the one conversation I was involved in with him during an event. He stepped in to a talk between myself and a couple of analysts. He listened to them and to me and when he was asked for his opinion, he stopped for a moment to think. He asked a question to clarify and then gave his answer. That’s a tempered leader approach to things. He listened. He thought. He clarified. And then he made a decision. That means there is steel behind the fire. That means the driving factors of the decision-making process aren’t just “cool stuff” or “save as much money as we can”. What will happen is the fusion of the two that the company needs to stay relevant in a world that seems bent on passing it by.


Tom’s Take

I’ve worked for managers and I’ve worked for leaders. I don’t have a preference for one or the other. I’ve seen leaders sell half their assets to save their company. I’ve also seen them buy ridiculous stuff in an effort to build something that no one would buy. I’ve seen managers keep things calm in the middle of a chaotic mess. I’ve also seen them so wracked with indecision that the opportunities they needed to capitalize on sailed off into the sunset. If you want to be the best person to run a company as the CEO, whether it’s a hundred people or a hundred thousand, you should look to someone like Pat Gelsinger. He’s the best combination of a manager and leader that I’ve seen in a long time. In five years we will be talking about how he was the one to bring Intel back to the top of the mountain, both through his leadership and his management skills.

Building Backdoors and Fixing Malfeasance

You might have seen the recent news this week that there is an exploitable backdoor in Zyxel hardware that has been discovered and is being exploited. The backdoor admin account with the clever name ‘zyfwp’ is not something that has been present in the devices forever. The account was put in during firmware version 4.60, which was released in Q4 2020.

Zyxel is rushing to patch the devices and remove the backdoor account. Users are being advised to disable remote administration until the accounts can be deactivated and proven to be removed. However, the bigger question in my mind relates to the addition of the user account in the first place. Why would you knowingly install a backdoor?

Hello, Joshua

Backdoors are nothing new in the computer world. I’d argue the most famous backdoor account in the history of computer hacking belongs to Joshua, the dormant login for the War Operations Programmed Response (WOPR) computer system in the 1983 movie Wargames. Joshua was an old login for the creator to access the system outside of the military chain of command. When the developer was removed from the project the account was forgotten about until a kid discovered it and kicked off the plot of the movie.

Joshua tells us a lot about developers and their desire to have access to the system. I’ll admit I’ve been in the same boat before. I’ve created my own logins to systems with elevated access to get tasks accomplished. I’ve also notified the users and administrators of those systems about my account and let them deal with it as needed. Most were okay with it being there. Some were hesitant and required it to be disabled after my work was done. Either way, I was up front about what was going on.

Joshua and zyfwp are examples of what happens when those systems are installed outside of the knowledge of the operators. What would have happened if the team in the Netherlands hand’t found the account? What if Zyxel devices were getting hacked and networks breached without anyone knowing the vector? I’m sure the account showed up in all the admin dashboards, right?

Easter Egg Hunts

Do you remember the Windows 3.1 Bear? It was a hidden reference in the credits to the development team’s mascot. You had to jump through a hoop to find it by holding down a keystroke combination and clicking a specific square in the Windows logo. People loved finding those little nuggets in the software all the way up to Windows 98.

What changed? Turns out, as part of Microsoft’s Trustworth Computing Initiative in 2002 they removed all undocumented features and code that could cause these kinds of things. It also might have had something to do with the antitrust investigations into Microsoft in the 1990s and how undocumented features in Windows and Office might have given the company a competitive advantage. Whatever the reason, Microsoft has committed to removing undocumented code.

Easter eggs are fun to find but represent the bright side of the dark issue above. What happens when the easter egg in question isn’t a credit roll but an undocumented account? What if the keystroke doesn’t bring up a teddy bear but instead gives the current user account full admin access? You scoff at the possibility but there’s nothing stopping a developer from making that happen.

These issues are part of the reason why all code and features need to be documented. We need to know what’s going on in the program and how it could impact us. This means no backdoors. If there is a way to access the system aside from the controls built in already it needs to be known and be able to be disabled if necessary. If it can’t be disabled then the users need to be aware of that fact and make the choice to not use the software because of security issues.

If you’re following along closely, you should have picked up on the fact that this same logic applies to backdoors that have been mandated by the government too. The current slate of US Senators seem to believe that we need to create keys that allow end-to-end encryption to be weakened and readable by law enforcement. However, as stated by companies like Apple for years, if you create a key for a lock that should only ever be opened under special circumstances you have still created a weakness that can be unlocked. We’ve seen the tools used by intelligence agencies stolen and used to create malware unlike anything we’ve ever seen before. What do you think might happen if they get the backdoor keys to go through encrypted messaging systems?


Tom’s Take

I don’t run Zyxel equipment in my home or anywhere I used to work. But if I did there would be a pile of it in the dumpster after this mess. Having a backdoor is one thing. Purposely making one is another. And having that backdoor discovered and exploited by the Internet is an entirely differently conversation. The only way to be sure that you’ve fixed your backdoor problem is to not have one in the first place. Joshua and zyfwp are what we need to get away from, not what we need to work toward. Malfeasance only stops when you don’t do it in the first place.

Winning in 2021

I’d jump in here and say something about 2020 being a crazy year but we all know it’s nothing we haven’t heard before. I’d also say that we’re going to look back at my big plans for the year however we also know that those got scrapped right after the end of February. I like looking back at a couple of things and then looking forward to what the next year will accomplish. Why? Because retrospectives are boring and putting your planning out there for the world to see is a much more interesting use of your time. The journey you’re taking changes greatly when you change your thinking about the destination.

2020 Good or Bad

2020 wasn’t all bad. I finally justified getting a new office chair! All kidding aside, 2020 was a year that challenged everyone greatly when it came to mental health, professional output, and even personal capability. My biggest focus for 2020 was to start putting blog posts out earlier in the week and focus on continuous improvement. I’d say the first was another miss due to the hectic workload, as a lot of my posts still came out on Fridays.

The second point was a bit more successful. I’ve been more diligent about getting stuff down and in a state when it can be improved. I’ve also added a lot of things to my repertoire over the year that I’m proud of. Here are some specifics:

  • Tomversations: I started a video series this year! I wanted to start coming up with monthly videos around topics that worked better as explorations instead of just simply spouting randomness. We put twelve episodes up last year starting around April. I was very happy with the way they turned out, especially toward the end when my process improved. Video is a great medium for some of the conversations I want to have.
  • The Rundown: Okay, this is a bit of stretch since I’ve been co-hosting the Rundown since it started. But this year my friend Rich Stroffolino headed off to future endeavors and I took over the production part on the back end. It’s been interesting skimming the news and putting it together each week to try and keep the sparkly magic going. It also means I’m much closer to the details behind the tech now.
  • Cooking: This was my big pandemic skill level up. My cooking skill has always been just shy of adequate. This year I pushed myself to get better about learning technique and saving recipes so I have something to pull from when I make food. The tastes have gotten way better and I feel more confident. I’d say the family is happier too since we have something other than Kraft Mac and Cheese all the time.
  • Running: This was my other pandemic level up. I fell off the exercise wagon at the end of 2019 and it showed. I was heavier than I had ever been. I wasn’t thrilled at the idea of getting back in shape either. Once the pandemic set in and I knew I wasn’t going to be on the road for the foreseeable future I jumped back on the road to running. Since June 1, I have run or walked over 900 miles and lost almost 50 pounds. I feel better and I look forward to lacing up my shoes and running every morning.

2021 More Time

That’s where my energy went in 2020. Video and research and cooking meals to eat after I ran. What am I thinking about for 2021?

  • Bullet Journaling: This is an idea I got from my partner in crime Ben Gage. I need a better system for capturing info and logging tasks. I say this every year. And every year I find a way to fail at it somehow. This year I’m going with the less-structured approach. I’m keeping the journal digital in GoodNotes and using these templates from Robert Terekedis (@robterakedis) that I found in a search. I like the hint of organization with the freedom to do more when I want it done. Let’s hope this sticks!
  • More Video Content: Like I don’t spend enough time on camera? I’m going to explore the idea of doing more video content. I’m not going to do a daily log or anything but I’m going to try and figure out if creating more around some of my ideas but putting it on video will help me solidify it a little. I’ve found through Tomversations that my ability to riff on subjects and think through stuff when I’m staring at a camera lens feels much different than facing a computer keyboard. It’s not better or worse. It’s different and I’m curious about where that will lead.
  • Create Content that Resonates: My blog is ten years old now. There are posts from 2011 that don’t apply to anything any longer. Some of the posts that I’ve been putting out recently aren’t as technical and look more at work skills, soft skills, or even just life skills. Many of you have commented that my ideas around time management or organization are things you wanted or needed to hear. I’m going to explore those ideas a bit this year too. Don’t worry – The Networking with a Side of Snark isn’t going away any time soon. And I’m not going to turn into a productivity blogger overnight. Mostly because I don’t have enough productivity to make that happen! But I want people to enjoy reading my content for what it can help them with in the next twelve months of working with the challenges we will face.

Tom’s Take

2020 was a sucky year in general. Too much stress, too much uncertainty, and for those that tend to overanalyze everything it was a year of way too much introspection and questioning. I’m looking forward to the next 52 weeks to sort out what needs to be done and get it finished. I set good habits in 2020 that I want to carry forward. I’m going to keep improving just like last year and use the tools I can to make those changes a part of what I need to do to ensure that 2021 is filled with more winning than anything else. I may not be on a plane at all this year. However, I can win all I can from my house and help you all along with the way too. Let’s enjoy the coming 525,600 minutes and do something that makes us feel like winners.

Making Time For Yourself

I was a recent pop-in guest on the Network Collective Holiday Show with my friends Jordan Martin and Tony Efantis. One of the questions they had been asking their guests was about the big lessons we’ve learned this year. As I thought back on the roller coaster ride that was 2020, I realized that one of the biggest lessons that I’ve learned is that I need to make time for the important things for myself.

Mark It Down

I know it sounds like a given, but we all need to make time for ourselves. I realized that when my usual schedule of running myself in overdrive and jumping from one event or travel opportunity to the next evaporated back in March. I found myself sitting at home and working toward some uncertain future. I never thought that there were going to be huge problems but I also didn’t know how things would end up turning out.

As the days grew into weeks and eventually into months, I quickly figured out that the normal I once knew was going to stay gone for quite a while. In place of that was a situation that I needed to adjust to. And that was going to to take some time. I needed to catch my breath but I also needed to build a skill set that would allow me to continue forward.

Over April and May I got better at cooking. I retaught myself the basics of making all kinds of meals. I gained the confidence of trying new things. It helped me find a bit of stability. It happened because I started doing research and setting aside time every day to practice those skills. Maybe it was something small like making tacos. Or even putting something into a slow cooker. But it was time that I needed to take to do something that I needed to learn.

The second big lesson in taking time for myself came in June. With the move of Cisco Live to a digital event an the likelihood that everything else for the rest of the year was going to go the same way, I took the opportunity to get back into better, healthier shape. I had a hard time exercising on the road with the hotel gym being something that I didn’t appreciate. I started getting up earlier and going for walks and then for short runs. Then I upped my running and my walking distances. I made sure to lace up running shoes every day no matter what. No excuses whether it was raining or blisteringly hot outside.

Taking the time to get into better shape has had a huge impact on my self worth and my health. I’ve dropped 50 pounds since March and my running times keep coming down. My pants size went down significantly and the pictures of myself that I’m taking now barely resemble pre-COVID me. All because I took the time for myself.

Make It Happen

There’s no magic in what I did. There was no special system or secrecy code to get me to where I am right now. The only trick was making the time for myself. It’s like the financial books you can buy that give you tips to put into practice to get rich. One of the first is “pay yourself”. It’s contrite but proves the point that you need to give yourself resources to work with or you’ll never get ahead.

Time is as valuable as resource as anything we have. We can’t save time and use it later. We can’t manufacture time. We can only use the time we have to the best of our abilities. Sometimes that means putting something we want to do on hold because of something we have to do. As someone that prides myself on writing lots of blog posts it meant that I had to put that particular part of my productivity behind more immediate things like getting my morning run in. It meant getting the Gestalt IT Rundown story script done before I could play a game or watch a TV show.

Time is what we make of it. I’ve started to realize that by blocking more and more of my time to do things. Maybe I put down on my calendar that Tuesday evening is a day to draw or practice a new cooking skill. Thursday morning could be my long run of the week and my day to research topics for my Tomversations videos. Whatever it is, I make it stick. I don’t need to schedule my exercise in the mornings because it’s become a habit for me. But I do need to schedule the other things to make sure they’re done. You don’t need to have a mark on every minute of your day to be productive, but you do need to make sure you make time for the pieces that are important.

That means making time for non-work things. It’s easy to fill up our calendar with things for work. It’s harder still to fill up the calendar with non-work tasks and skills. Schedule a hike on a Saturday morning. Make Monday night your night to work on a craft that you want to do like learning leather working or knife making. Maybe you just want to say that Wednesday at lunchtime is the place where you’re going to schedule time to read a few more pages of your new favorite book. There are all things that as valid as the next staff meeting or briefing that you have to do. Because they enrich you and help you become a better person.

More importantly, by scheduling these things in your calendar on a personal level you remind yourself not to let work get out of hand. When you live in the office, which is the same as working from home now, you will find yourself working at midnight some nights because you just couldn’t put the work down. By reminding yourself of what’s important you draw a bright line between work and personal and ensure that you have time to put in effort on both without getting overwhelmed.


Tom’s Take

There’s a bit of irony in me saying that you need to make time for the things that are important to you while I write this post after midnight on Christmas. The fact is that I made time earlier today for my family to open some gifts and help bake cookies. I went for a walk and watched some educational videos. Sure, I eventually found the time to write for myself but it came after I had taken care of other things. My journey through 2020 has taught me that time is the kind of resource you need to pay yourself with as often as you can. But you can’t just mark off your calendar and hope that something magical happens. You need to make the effort to use your time wisely and work on yourself. Every new skill you learn or pound you lose is making you a better, more well-rounded person. And that’s the kind of payoff that you can only get from investing time in yourself.

Setting Boundaries Before You’re Swamped

We’re at the tail end of 2020 and things are hopeful for 2021. People are looking at the way IT has pulled together to enable working from anywhere and moving resources to the cloud and enabling users to get their jobs done. It’s a testament to the resilience of a group of sanitation workers behind the scenes whose job it is to clean up after management and sales and do the jobs no one else wants to do.

The cynic in me is worried about what the future is going to hold now that we’ve managed to transform the way we work. I couldn’t quite put my finger on it until I was checking out this Reddit thread from last week. The top rant had an interesting perspective on the way that 2021 is going to go for workers and I couldn’t agree more. My dread has a name, and it’s Overwork.

Harder, Not Smarter

If anything, 2020 proved that we can do amazing things with the right motivation. The superhero mentality of IT paid off handsomely as we stood up remote access servers and found ways to get access to resources for people that couldn’t come into the office and needed to get things done. We proved invaluable to the company in terms of support and project execution.

However, that superhuman effort also makes us valuable because we can do the impossible. Which means individual workers are great at tasks that are a stretch to get done. Doesn’t sound too bad, right? Now, think back on the number of times over your career that you’ve jumped on something that you’ve seen as hard or challenging and gotten it done. Is that number more than one or two? It probably is.

Now, look at it from the perspective of management and the people that make the hiring decisions. You’ve never failed, or at least you’ve failed so infrequently as to be unimportant. You always get the job done no matter what. If we only had a team of you sitting around we could get so much done! Sound familiar?

Let’s look at it from the finance part of the equation. Revenue is down. Expenses are down too, but we could still trim a bit and save money. Where should we cut? Can’t get rid of sales or marketing because they’re working to bring in money. Can’t get rid of executives (for some reason) because they serve some purpose. IT? Support staff? They’ve never failed us before. If we give them a little extra work and don’t fill that open position or cut someone that’s underperforming they’ll come through for us, right?

It’s a tale that’s as old as any in business. Why pay someone to do the work in a role when we can split it up and pass it out to the rest of the team. An extra job won’t hurt anyone, right? Funny how that never happens to the CFO or the CMO. But the networking/storage/security/wireless/server/cloud teams? They’re rockstars!

Know When to Fold Em

In 2021, you are going to need to set boundaries and stand up for yourself when it comes to work. Every role not filled with a person drawing a salary is directly impacting the bottom line of the company. Which means either you should be getting paid more for doing more work or the executives and stockholders are going to get paid more because the company made higher profits. Who deserves that extra money more? The people doing the work? Or the people parking assets in your company until they recover their position and jump to greener pastures?

You need to draw the line and advocate for yourself. Don’t just blindly accept new responsibilities without some sort of future resolution. Are they going to hire someone to fill that role eventually? If not, what are they going to do with the salary? If you and someone else on your team are doing that role permanently you should both be compensated for it. If the executive team gives you pushback, just remind them that they can forgo their next raise or set of stock options to cover things. You’d never see them doing any more without some form of compensation, right?

Drawing boundaries is going to cause friction because superheroes have never had them before. Superman and Spider Man don’t negotiate about saving people on a train. However, you aren’t a radioactive Kryptonian vigilante either. You’re doing a job that you get paid to do. Adding more to your plate is either worthy of additional money or it’s something that needs to be pushed back. If this was AWS and they wanted to upgrade their server instances to a faster CPU, do you think they could persuade Amazon to give it to them for free? Not on your life.

Take the time in 2021 to set yourself up for success. You may be worried that people are going to fire you because they think you want too much money or because you need to be happy to have a job right now. You also need to realize that no one can do your job better than you right now and they know that too. Have a frank and honest discussion about why you feel it’s unfair to pile more work on you without treating you fairly. Make sure they know you’ll help where you can but if they aren’t going to fill the role you need to get the resources they would dedicate to the person doing it. Chances are they’ll realize they do need to hire someone to take the work or they’ll find a way to do without those responsibilities.


Tom’s Take

You are a valuable resource. But you are not infinite. If you burn yourself out then you are spent. The business will likely look for someone to replace you and get them up to speed on your job. You? You’re going to be a shell of the person you were. 2020 has been a roller coaster that has pushed us all hard. We need to find a balance. And if you don’t let people know that you need to find balance in your work and your compensation you may find yourself with no boundaries, no balance, and knee deep in the swamp.

How Long Should You Practice

A reporter once asked boxing legend Muhammad Ali how many sit-ups he did each day. I’m sure the reporter wasn’t expecting Ali’s answer. Ali replied with:

I don’t know. I don’t start counting them until it hurts. Those are the only ones that count. That’s what makes you a champion.”

Ali knew that counting things is just a numbers game. Five hundred poor sit-ups don’t count as much a fifty done the right way. With any practice that you do the only things that count are the things that teach your something or that push you to be better.

Don’t Practice Until It’s Right

People used to ask me how long I would spend at night studying for the CCIE lab. I told them I usually spent between five and seven hours depending on what I was studying. Sometimes those people would say things like “I’m not talking about setup time. I’m talking about actual lab work.” I always countered by making them explain why the setup isn’t part of the “real” work. That’s usually when they went quiet.

It’s far too easy to fall into the trap of overlooking things that you think are unimportant. A task you’ve done a hundred times is no big deal until you do it wrong the next time. Like Ali above, the things you do that require no effort don’t count. If you’re practicing a skill for a certification or a lab you need to put the same effort into it every time to ensure you’re doing it correctly. Lack of attention means you are doing it without gaining something from it.

I spend a lot of my time teaching things to people all over the place. I teach IT and networking skills to professionals. I teach outdoor skills to scouts of all ages. I teach merit badges and other things to a variety of youth. And I teach my kids life skills they will need. Every one of these lessons comes with instruction in the little details that matter. Every lesson also includes guidance that it needs to be practiced properly until it’s right. And then some.

Until You Can’t Get It Wrong

I tell my students and kids all the time, “Don’t practice until you get it right. Practice until you can’t get it wrong.” The level of involvement that it takes to get past the part where something finally works up to the level where it works every time is as wide as the gap at the lower end of the spectrum.

Too often people are content to work on something until they get it once. Whether it’s tying a knot or programming a router interface or even cooking a grilled cheese sandwich. Once you’ve done it right once you’re done with learning, right? Most of you are already shaking your head because you know that’s not right.

Once you get it right the first time you’ve already made a list of all the wrong ways to do something and you avoid them in the future. However, that list doesn’t include the entirety of all the wrong ways to do a thing. Amateurs make somewhat predictable mistakes because they’re working from the same basic knowledge. It’s when someone says they know what they’re doing that the real crazy stuff starts coming out of the woodwork.

Once you’ve practiced a skill you need to keep going. You need to work a variety of different angles to make sure you’ve covered all the ways you could get it wrong. If you’re tying a knot you need to practice with different kinds of ropes or in different positions. If there are two ways to tie something, practice them both. You don’t want to be an expert at a clove hitch over the end of a pole only to find out you have to tie it around the middle with no way to use the loop method you have memorized.

In IT, we lab things up to make sure we understand them. For these labs, try out the things in wrong ways. Click buttons before you’re supposed to. Put the wrong numbers in the field. See how the system will try to correct your errors. Maybe it doesn’t even bother? It’s easy to figure out you typed something in wrong when you hear a bell and see a message. It’s harder to troubleshoot when you don’t see anything right away and it all falls over later.

The extra practice above and beyond the first success is just like Muhammad Ali’s sit-ups. The hard ones count. The tasks that stretch your mind are the ones that build your skillset. You can’t give up when the answer isn’t right at your fingertips. Going that extra mile is the key to making yourself a better professional in whatever you do.


Tom’s Take

As we wind down 2020 we’re all looking to be better at things. Hobbies, skills, or professional talents are all calling to us to work on in whatever down time we have available to us. Make that practice count. Work hard to get it right every time. If you want to learn to make hollandaise sauce or write a novel or do a forward flip you have to keep practicing even after your success. Get to the point where you have no other choice but to get it right every single time. That’s the perfect amount of practice you need. Anything less counts as much as Muhammad Ali’s sit-ups before they start hurting.

Seeking Knowledge and Willful Ignorance

I had a great time recording a fun episode of Seeking Truth in Networking, an awesome podcast with my friends Derick Winkworth and Brandon Heller. We talked a lot about a variety of different topics, but the one I want to spend a few more minutes on here came in the first five minutes. Brandon asked me what question I liked to be asked and I mentioned that love to be asked about learning. My explanation included the following line:

I feel like the gap between people that don’t understand something and the willfully ignorant is that ability to take a step out and say “I don’t know the answer to this but I’m going to find out.”

I’ve always said that true learners are the ones that don’t accept the unknown. They want to find the answer. They want to be able to understand something as completely as they can. Those that I consider to be willfully ignorant choose not to do that.

Note that there is a difference between incidentally ignorant and willfully ignorant. People who are incidentally ignorant are unaware they don’t know something. They haven’t had the opportunity to learn or change their thought process on something. It would be like going to a random person and asking them about how to launch a rocket to another planet. They’re ignorant of the steps because they’ve never had the opportunity to learn them. They’ve never been exposed to the info or had a need to know it. People who are willfully ignorant choose to not learn something even after they’re exposed to it.

Where There’s a Will

We deal with people who choose not to learn things all the time. Even I choose not to learn everything. I don’t have all the Pokemon memorized. I don’t have the registration number of every Starfleet vessel in my mental Rolodex. There are a variety of other more technical topics that escape me. However, my reasoning for choosing not to learn those things is not because of malice. It’s because of self preservation.

If you exposed to something that you are curious about and choose to learn more you will often find yourself consumed by it. I am always on the lookout for a new laptop bag or hiking backpack. When I search for them I will find myself watching videos and reading reviews that are full of terminology that I don’t understand. I educate myself to the best of my ability but I don’t consider myself to be an expert on messenger bags or ultralight hiking packs. And after a while that knowledge is filed away for another day and I have to relearn something all over again when I’m on the hunt for a new bag.

Let’s contrast the acknowledgment of not being able to know everything with the phenomenon of choosing not to learn something out of spite or malice. This is like a networking engineer saying something along the lines of, “I’m not going to learn OSPF because it sucks and I’ll never use it.” A statement like that should immediately raise flags. In this specific case a working knowledge of OSPF is important for anyone building and maintaining networks. You may not need to know the details for every LSA in the database but you at least need to know how OSPF is different from RIP.

This kind of willful ignorance of information makes IT difficult. Why? Because actively choosing not to learn or understand something creates two hurdles to overcome. The first hurdle is showing people where to learn more about it. That is hard enough in and of itself. Every bookshelf in every office everywhere has the kinds of books that people refer to when they need to teach someone something important about networking or wireless or any other enterprise IT technology. Thanks to the power of search engines today it’s even more accessible to get people on the track to learning something new.

The second huge hurdle with those that are willfully ignorant isn’t access to knowledge. It’s getting past their objections to learning it. People have biases that need to be challenged and overcome. I’m not going to speak on anything aside from technology but we all know that everyone has their viewpoint and their understanding and changing their mind about something has varying degrees of difficulty. If someone is convinced, for example, that SHA-1 is an unbreakable protocol and nothing you can show them to the contrary convinces them that is willful ignorance. Evidence that is contrary to the facts isn’t invalid evidence. The quality of the evidence is always important to understand but choosing to dismiss it entirely out of hand solely because it doesn’t fit your understanding is not the kind of position someone in IT needs to take.

A Changing Landscape

Think about some of the following statements:

  • Switching is cheap, routing is expensive
  • 640K is more than enough memory
  • Unbreakable encryption

These are all statements that have been said in the past. They’ve all been proven over time to be wrong. Could you imagine if there was someone out there today that though programs needed to run in less than 640K of RAM? Or that believed that routing packets was too expensive and everything needs to run at layer 2? Those people would get laughed out of the data center.

Those statements are no longer true, but the attitudes behind them are the real problem. It’s not that something is taken for granted but that we choose not to accept anything other than that fact as the truth. Even today we could have positions like virtual reality will never take off or that quantum computers are too noisy to ever be commercially viable. In five years or a decade those statements may prove to be totally wrong. But if I’m still saying them and purposely choose not to learn why they are incorrect then I’m in the camp of being willfully ignorant of the truth.


Tom’s Take

The point of this post wasn’t to call out anyone specific for anything. Instead, I wanted to highlight that we all believe what we believe and we resist learning things that don’t square with that. How we choose to overcome that friction defines us as well as defining us with our peers. If you want to spend your career believing that a protocol is better and you won’t learn anything else then I hope your career is successful and long. I say “hope” because in the world of IT those that clap their hands over their ears and refuse to update their knowledge and understand are running on that kind of hope. They hope their level of knowledge never needs to change. They hope their skills will be enough for years and years of employment. And, in almost every case, they hope they can learn something new fast enough to get a new job when they realize that the attitude of willful ignorance will leave you high and dry.

What’s Your Work From Home DR Plan?

It’s almost December and the signs are pointing to a continuation of the current state of working from home for a lot of people out there. Whether it’s a surge in cases that is causing businesses to close again or a change in the way your company looks at offices and remote work, you’re likely going to ring in the new year at your home keyboard in your pajamas with a cup of something steaming next to your desk.

We have all spent a lot of time and money investing in better conditions for ourselves at home. Perhaps it was a fancy new mesh chair or a more ergonomic keyboard. It could have been a bigger monitor with a resolution increase or a better webcam for the dozen or so Zoom meetings that have replaced the water cooler. There may even be more equipment in store, such as a better home wireless setup or even a corporate SD-WAN solution to help with network latency. However, have you considered what might happen if it all goes wrong and you need to be online?

In and Outage

Outages happen more often than we realize. That’s never been more evident than the situation we find ourselves in now. There are providers that do maintenance during the day because most of their customers are at work. When that work happened in a building covered on a different grid or service line it was fine to reboot things in the afternoon. When everyone is at home working on video calls or remote classrooms it’s no longer ideal. And those are just the planned outages. What about the ones that happen without warning?

Between the extra usage at home and the increased stress on the system, I’m finding that my Internet connection is becoming much less stable than it has been in the past. When you’re on a consumer-grade line, you pay for the privilege of getting online when it works. When it doesn’t you get lumped in with the same group of people in your neighborhood or on your local loop. The provider response is usually a shrug and a “we’re working on it” response. Business lines cost twice as much for less speed but gain the ability to call and at least file a complaint or a ticket. If you’re lucky enough to have one with a good SLA you might even get a truck roll to your location within a few hours. Otherwise, you need to plan for the worst.

This is nothing new to the enterprise. The best SLA in the world is only a piece of paper with a specific promise. It doesn’t protected from the North American Fiber Seeking Backhoe or an ice storm that knocks out power to a few square miles of your town. We need to have plans in place to deal with the potential for not having what we need when we need it. In the enterprise that was part of the job. We bought firewalls in pairs and had expensive power equipment run into the data center to protect our services. The cloud is armored against outages, so long as the engineers keep their fingers off of things. But our house is neither the enterprise or the cloud. How can we ensure we’re able to work when nothing else looks like it’s going to get the job done?

Planning To Recover

In order to keep working from home in the event of an outage, you need to consider three important situations: Connectivity Outage, Power Outage, and a Location Outage.

Connectivity outages are the most basic situation we’re going to find ourselves in. The Internet is down or severely degraded. We need to get online and figure out how to keep working. That means we need a traffic plan. And we need to figure out how to get it working. The first step is going to require you to figure out how you’re going to get back online. Do you want to use your cell phone as a tethering device? Do you want to “borrow” the neighbor’s wireless (with permission, of course). Do you want to try and work completely from your mobile device. You need to figure this out ahead of time. You also need to test it.

If you’re going to rely on your phone for tethering, make sure you have that option enabled ahead of time. Find out how much data you have available and what happens if you go over. Do you get throttled to a slower speed? Do you need to pay more? You don’t want to find out what happens in the middle of a critical call. You also need to test the speed of the tethering at your house. For example, my LTE coverage at home is pretty terrible, so I need to fail back to 3G in order to have a stable signal. That means no video calls for me until my broadband connection comes back online.

If you plan is use your neighboring connection for a backup, please get permission first. You never want to find out someone is borrowing your network and you don’t want to do that to someone else. You also should verify that the neighboring connection is a diverse circuit. It won’t do you much good to hop on their connection only to find out they’re on the same provider and everything is offline for the entire block. Test it ahead of time and make sure their data plan works for you. And remember that you’re doubling the amount of traffic pouring through their circuit. You’re going to have to decide, as above, what traffic is critical to fail over. And give your neighbor a heads up so they don’t panic when everything gets slower.

Limited Power

Electricity is a bigger issue because it affects everything at a lower level of the stack. You can have bad connectivity and just work offline with your devices. But a power outage changes the game because half your devices may be out of commission. If my power goes out, my 4K monitor goes with it. That means I need to work from my laptop until I can get everything under control again. My time is limited to how long my laptop battery can hold out.

If you want to solve the power outage problem, you need to solve your power issues. The easiest way is a backup battery system, like a personal uninterruptible power supply (UPS) under your desk. Remember that a UPS isn’t designed to keep you running for days. It’s something that is designed to keep you going just long enough to put a plan in place or shut things down gracefully. Those batteries last for 5-7 minutes at best. And the more devices you have connected to them, the less time they can stay up.

What other devices, you might ask? Keeping your computer on a UPS is smart. What about your WAN connectivity device, like your DSL or cable modem? Have to have that if you want to stay online, right? What about your wireless access points? Are they plugged into the wall? Or are you using PoE? Did you plug the PoE switch into the UPS too? That is going to reduce the UPS runtime. Unlike a data center, your house likely doesn’t have the infrastructure to run a huge UPS with enough battery to go for half an hour. You need something that is going to fit under your desk and has a fan small enough to not create a white noise generator.

Speaking of generators, if you have regular power issues or you live in a part of the country that is prone to storms that knock out power for days on end, you need to consider a home generator. These are larger, more expensive, and require some kind of fuel source. The upside is you can run your entire house off the generated electricity until the power is restored. No need to hook things up to batteries. The down side is that your house uses a lot of power for appliances and other things and you still need to prioritize what is going to get first crack at the leftover juice. You also need to have a plan to keep the generator going. If it runs off of combustible fuel, like diesel, you need to have a supply ready to go. You also need to test it regularly to ensure it’s going to work when you need it. Just like in the enterprise data center you need to know things work before everything goes sideways.

Getting Out of Dodge

The last situation can be a combination of the above factors or something totally unrelated. What happens if your workspace isn’t usable? Maybe your power is out and your heat is gone too. Maybe your Internet is down and you need to go somewhere with a better signal in order to have that big call with the CEO. Maybe you have a tree trimming service that has set up camp outside your office window for the rest of the day and the chainsaw symphony is driving you insane. If you need to move, you need to have a plan.

Where are you going to go? A friend’s house works great, provided they are home and there isn’t a quarantine order. You could try a coffee shop, provided they aren’t closed for some reason. Maybe you’re alone in a distant city and you need to get some work done. You need to figure out what’s available and have at least two backup plans in place. Maybe you want to use a local coffee shop. Make sure they’re open and make sure you aren’t violating local laws. If they’re closed, consider somewhere more local to you. Perhaps the parking lot of a store or other place that offers wireless connectivity. There are times when those wireless setups extend a bit outside the store proper and you can borrow it there. Not everyone has a vehicle, so make sure wherever you are going is easy to reach through your preferred transit method. Oh, and if it’s a restaurant or coffee shop, make sure you tip well on your purchases as a form of “rent” for the table.


Tom’s Take

Enterprises have DR plans in place for everything. Natural disasters, security incidents, and even good old fashioned human error all have a place in the Big Binder of Getting Back to Work. Homes don’t have that, even though they should. You need to know what has to happen to get you back to working if something goes wrong. You need to write it all down, test it thoroughly, and keep updating it as you go. Your boss may understand the first time you can’t work because your power went out or they’re doing circuit maintenance in your neighborhood. But, as an IT professional, you need to have a plan in place in case it becomes a regular occurrence. And when you do get everything ready to go for your home DR plan, make sure you update your enterprise DR plans too.