Cleaning Out The Cruft

Posted on by
1

I spent the weekend doing something I really should have done a long time ago. I went through my piles of technology that I was going to get around to using one day and finally got rid of anything I didn’t recognize. Old access points, old networking gear, and even older widgets that went to devices that I don’t even remember owning.

Do you have one of these piles? Boxes? Corners of your office or cave? The odds are good there’s a pile of stuff that you keep thinking you’re eventually going to get around to doing something with some day. Except some day hasn’t come yet. So maybe it’s time to get rid of that pile. Trust me you’re going to feel better for getting rid of that stuff.

What to do with it? It needs to be properly recycled so don’t just toss it in the trash can. Anything with electric circuits needs to be properly disposed of so look for an electronics recycling facility. Yes, there are stories that electronics recycling isn’t all it’s cracked up to be but it’s better than polluting with e-waste everywhere.

Consider donating the devices to a trade school or other maker space. Maybe they won’t work properly as intended but giving students the chance to take them apart is a much better option than just junking it all. Maybe you’ll inspire the next group of scientists and inventors because your old 802.11g access point fascinated them when they pulled it apart.

No matter whether you recycle or donate you should go through it all. Consolidate and be honest with yourself. If you don’t recognize it or haven’t used it in the last few months you’re not going to miss it.

Hedgehog – The Network OS Distro?

Posted on by
Reply

You’ve probably seen by now that there’s a new entrant into the market for network operating systems. Hedgehog came out of stealth mode this week to fanfare from the networking community. If you read through the website you might question why I labeled them as a network operating system. While they aren’t technically the OS I think it’s more important to look at them as an OS distribution.

Cacophony of Choice

Hedgehog starts from a very simple premise. Cloud networking is where we’re all headed. Whether or not you’re running entirely on-premises, fully in the public cloud, or in some kind of super-multi-hybrid cloud offering you’re all chasing the same thing. You want a stable system that acts as a force multiplier for your operations teams to reduce deployment times for users to get their builds done. It’s been said before but the idea of cloud is to get IT out of the way of the business.

Streamlining processes means automating a lot of the things that were formerly done by people. That means building repeatable and consistent tools to make that happen. If anyone has ever worked on AWS or Google Cloud you have lots of access to that tooling. Perhaps it’s not as full-featured as rolling your own toolset but that’s the tradeoff for running in someone else’s cloud. Notice that I left Microsoft Azure off that list.

Azure’s networking stack has been built on SONiC, a LInux-based NOS that has been built to scale in the cloud and solve the challenges that Microsoft has faced in their hyperscale data centers. They’ve poured resources into making SONiC exactly what they needed. However, one of the challenges that is faced when that happens is some of those things don’t scale down to the enterprise. I’m not saying you shouldn’t use SONiC. I’m saying that it’s not easy to adapt SONiC to what you want to do if you’re not Microsoft.

Speedy Adoption

In a way, it’s the same problem that Linux faced 25 years ago. If you really wanted to run it on a system you could download the source code and compile it on your system to get the kernel running. However a kernel doesn’t do much without software running on top of it. How could I write blog posts or check the time or get on the Internet without other applications? That need for additional resources to make the process of using Linux easier and more complete is where we got the rise of the Linux distribution, often shortened to distro.

Distros made adopting Linux easier. You didn’t have to go out and find sources for programs to run them on your system after compiling the kernel. You could just install everything like a big package and get going. It also meant that you could swap out programs and tools much easier than other operating systems. Ever tried to get rid of Notepad on Windows? It’s practically a system tool. On the other hand I think most Linux users can tell me their five favorite text editors off the top of their head. The system is very extensible.

Hedgehog acts like the distro of yore for SONiC. It makes the process of obtaining the OS much easier than it would be otherwise and includes a toolset that amplifies your networking experience. The power of cloud networking comes from optimization and orchestration. Hedgehog gives you that capability. It allows you to run the same kinds of tooling that you would use in the cloud on your enterprise data center networking devices.

If you’re starting to standardize on Kubernetes to make your applications more portable to the cloud then Hedgehog and SONiC can help you. If you’re looking to build more edge computing functionality into the stack Hedgehog has you covered. The Hedgehog team is building the orchestration capabilities that are needed in the enterprise right now to help you leverage SONiC. Because that tooling doesn’t exist outside of Microsoft right now you can believe that the Hedgehog team is addressing the needs of enterprise operations teams. They are the drivers for SONiC adoption. Making sure they can take care of daily tasks is paramount.

The distro launched Linux into the enterprise. Clunky DIY tooling can only scale so far. If you want to be serious about adopting cloud-first mentality in your organization you need to make sure you’re using proven tools that scale and don’t fall apart every time you try to make a minor change. Your data center isn’t Facebook or Google or Azure. However the lessons learned there and the way that we apply them at the enterprise level will go a long way to providing the advantages of the cloud for every day use. Thanks to companies like Hedgehog that are concentrating on the way to bring that market we have a chance to see it sooner than we’d hoped.

To learn more about Hedgehog and how they make SONiC easier for the enterprise, make sure to check out their website at https://githedgehog.com/

Why Do We Accept Bad Wireless Clients?

Posted on by
4

We recorded a fun roundtable discussion last week during Mobility Field Day that talked about the challenges that wireless architects face in their daily lives. It’s about an hour but it’s packed with great discussions about hard things we deal with:

One of the surprises for me is that all the conversations came back to how terrible wireless clients can be. The discussion kept coming back to how hard it is to find quality clients and how we adjust our expectations for the bad ones.

Driven to Madness

Did you know that 70% of Windows crashes are caused by third-party drivers? That’s Microsoft’s own research saying it. That doesn’t mean that Windows is any better or more stable with their OS design compared to Linux or MacOS. However, I’ve fiddled with drivers on Linux and I can tell you how horrible that experience can be1. Windows is quite tolerant of hardware that wouldn’t work anywhere else. As long as the manufacturer provides a driver you’re going to get something that works most of the time.

Apply that logic to a wireless networking card. You can buy just about anything and install it on your system and it will mostly work. Even with reputable companies like Intel you have challenges though. I have heard stories of driver updates working in one release and then breaking horribly in another. I’ve had to do the dance of installing beta software to make a function work at the expense of stability of the networking stack. Anyone that has ever sent out an email cautioning users to not update any drivers on their system knows the pain that can be caused by bad drivers corrupting clients.

That’s just the software we can control. What if it’s an OS we can’t do anything about? More and more users are turning to phones and tablets for their workhorse devices. Just a causal glance at Youtube will reveal a cornucopia of using a tablet as a daily driver machine. Those devices aren’t immune to driver challenges. They just come in a hidden package during system updates. Maybe the developers decided to roll out a new feature. Maybe they wanted to test a new power management algorithm. Maybe they’re just chaotic neutral and wanted to disrupt the world. Whatever the reason you’re stuck with the results. If you can’t test it fast enough you may find your users updated their devices chasing a feature. Most companies stop signing the code for the older version shortly after issuing an update so downgrading is impossible. Then what? You have a shiny brick? Maybe you have to create a special network that disables features for them? There are no solid answers.

Pushing Back

My comment in the roundtable boils down to something simple: Why do we allow this to happen? Why are we letting client manufacturers do this? The answer is probably more elegant than you realize. We do it because users expect every device to work. Just like the Windows driver issues you wouldn’t plug something into a computer and then expect it to not work, right? Wireless is no different to the user. They want to walk in somewhere and connect. Whether it’s a coffee shop or their home office or the corporate network it needs to be seamless and friction-free.

Would you expect the same of an Ethernet cable? or a PATA hard drive? Would you expect to be able to bring a phone from home and plug it into your corporate PBX? Of course not. Part of the issue is a lack of visible incompatibility. If you know the Ethernet cable won’t plug into a device you won’t try to connect it. If the cable for your disk drive isn’t compatible with your motherboard you get a different drive. With wireless we expect the nerds in the back to “make it work”. Wireless is one of the best protocols at making things work poorly just to say it is up and running. If you had an Ethernet network with 15% packet loss you’d claim it was broken. Yet Wi-Fi will connect and drop packets due to bad SNR and other factors because it’s designed to work under adverse conditions.

Why do we tolerate bad clients? Why don’t we push back against the vendors and tell them to do better? The standard argument is that we don’t control the client manufacturing process. How are we supposed to tell vendors to support a function if we can’t make our voices heard? While we may not be able to convince Intel or Apple or Samsung to build in support for specific protocols we can affect that change with consumption. If you work in an enterprise and you need support for something, say 802.11r, you can refuse to purchase a device until it’s supported.

But wait, you say, I don’t control that either. You may not control the devices but you control the network to which they attach. You can tell your users that the device isn’t supported. Just like a PATA hard disk or a floppy drive you can tell users that what they want to do won’t work and you need to do something different. If they want to use their personal iPad for work or their ancient laptop to connect they need to update it or use a different communications method. If your purchasing department wants to save $10 per laptop because they come with inferior wireless cards you can push back and tell them that the specs aren’t compatible with the network setup. Period, full stop, end of sentence.

Tom’s Take

The power to solve bad clients won’t come from companies that make money doing the least amount of work possible. It won’t come from companies that don’t provide feedback in the form of lost sales. It will come when someone puts their foot down and refuses to support any more bad client hardware and software. If the Wi-Fi Alliance won’t enforce good client connectivity it’s time we do it for them.

If you disagree I’d love to hear what you think. Is there a solution I’m not seeing? Or are we just doomed to live with bad client devices forever?

  1. If you say Winmodem around me I will scream. ↩︎

Monday Mobility Quick Thoughts

Posted on by
1

I’m getting ready for Mobility Field Day 8 later this week and there’s been a lot of effort making sure we’re ready to go. That means I’ve spent lots of time thinking about event planning instead of writing. So I wanted to share some quick thoughts with you ahead of this week as well as WLPC Europe next week.

  • I remain convinced than half of the objections that are raised by oversight organizations when it comes to adopting new technology come from the fact they got caught flat-footed and weren’t ready for it to be popular. Whether it’s the Wi-Fi 6E safety issue or the report earlier this year from the FAA about 5G and airports it just seems like organizations spend less time doing actual investigation and more time writing press releases about how they are ready to figure it all out yet.
  • I also remain cautiously optimistic that the new Apple devices rumored to be coming out later this year, namely the iPad Pro and MacBook Pro with M2 chips, will have Wi-Fi 6E support. Yes, the iPhone didn’t. It’s also a smaller device with less room to add new hardware. The iPad and MacBook have historically gotten new chips before the smaller mobile device does. If I’m wrong then I guess we’ll get to see if 6E is enough of a factor to get people to ditch their Apple device for a Google or Samsung one.
  • As we rely more and more on software to expand the capabilities of our hardware I think we’re going to see more and more companies working toward the model of hardware-as-a-service. As in you lease the equipment from them for a monthly payment and, in return, you get to have a base level of features that can be expanded in higher “tiers” of service. Expect some more on this idea in the near future with the launch of solutions like Nile.

Tom’s Take

Make sure you tune in for Mobility Field Day 8 and don’t forget to tell us what you think! Maybe by next year we’ll have lots of Wi-Fi 7 content to discuss.

Intelligence and Wisdom

Posted on by
1

I spent the last week at the Philmont Leadership Challenge in beautiful Cimarron, NM. I had the chance to learn a bit more about servant leadership and work on my outdoor skills a little. I also had some time to reflect on an interesting question posed to me by one of the members of my crew.

He asked me, “You seem wise. How did you get so wise?” This caught me flat-flooted for a moment because I’d never really considered myself to be a very wise person. Experienced perhaps but not wise like Yoda or Gandalf. So I answered him as I thought more about it.

Intelligence is knowing what to do. Wisdom is knowing what not to do.

The more I thought about that quote the more I realized the importance of the distinction.

Basic Botany

There’s another saying that people tweeted back at me when I shared the above quote. It’s used in the context of describing Intelligence and Wisdom for Dungeons and Dragons roleplaying:

Intelligence is knowing that a tomato is a fruit. Wisdom is not putting tomatoes in a fruit salad.

It’s silly and funny but it gets right to the point and is a different version of my other observation. Intelligence is all about the acquisition of knowledge. Think about your certification journey. You spend all your time learning the correct commands for displaying routing tables or how to debug a device and figure out what’s going on. You memorize arguments so you can pass the exam without the use of the question mark.

Intelligence is focused on making sure you have all the knowledge you can ever use. Whether it’s an arcane spell book or Routing TCP/IP Volume 1 you’re working with the kinds of information that you need to ingest in order to get things done. Think of it like a kind of race to amass a fortune in facts.

However, as pointed out above, intelligence is often lacking in the application of that knowledge. Assembling a storehouse full of facts doesn’t do much to help you when it comes to applying that knowledge to produce outcomes. You can be a very intelligent person and still not know what to do with it. You may have heard someone say that a person is “book smart” or is lacking is “common sense”. These are both ways to say that someone is intelligent by maybe not wise.

Applied Science

If intelligence is all about acquisition of knowledge then wisdom is focused on application. Just because you know what commands are used to debug a router doesn’t mean you need to use them all the time. There are apocryphal stories of freshly minted CCIEs walking in to the data center for an ISP and entering debug ip packet detail on the CLI only to watch the switch completely exhaust itself and crash in the middle of the day. The command was correct for what they wanted to accomplish. What was missing was the applied knowledge that a busy switch wouldn’t be able to handle the additional processing load of that much data being streamed to the console.

Wisdom isn’t gained from reading a book. It’s gained from applying knowledge to situations. No application of that knowledge is going to be perfect every time. You’re going to make mistakes. You’re going to do things that cause problems. You’re going to need to fix mistakes and learn as you go. Along the way you’re going to find a lot of things that don’t work for a given situation. That’s where wisdom is gained. You’re not failing. You’re learning what doesn’t work so you don’t apply it incorrectly.

A perfect example of this came just a couple of days ago. The power in my office was out which meant the Internet was down for everyone. A major crisis for sure! I knew I needed to figure out what was going on so I started the troubleshooting process. I knew how electricity worked and what needed to be checked. Along the way I kept working and trying to figure out where the problem was. The wisdom I gained along the way from working with series circuits and receptacles helped me narrow things down to one wall socket that had become worn out and needed to be replaced. More wisdom told me to make sure the power was turned off before I started working on the replacement.

I succeeded not because I knew what to do as much as knowing what not to do when applying the knowledge. I didn’t have to check plugs I knew weren’t working. I knew things could be on different circuits. I knew I didn’t have to mess with working sockets either. All the knowledge of resistance and current would only serve me correctly if I knew where to put it and how to work around the issues I saw in the application of that knowledge.

Not every piece of wisdom comes from unexpected outcomes. It’s often just as important to do something that works and see the result so you can remember it for the next time. The wisdom comes in knowing how to apply that knowledge and why it only works in certain situations. If you’ve every worked with someone that troubleshoots really complex problems with statements like “I tried this crazy thing once and it worked” you know exactly how this can be done.

Tom’s Take

Intelligence has always been my strong point. I read a lot and retain knowledge. I’m at home when I’m recalling trivia or absorbing new facts. However I’ve always worried that I wasn’t very wise. I make simple mistakes and often forget how to use the information I have on hand. However, when I shared the quote above I finally realized that all those mistakes were just me learning how to apply the knowledge I’d gained over time. Wisdom isn’t a passage in a book. It’s not a fact. It’s about knowing when to use it and when not to use it. It’s about learning in a different way that matters just as much as all the libraries in the world.

Redundancy Is Not Resiliency

Posted on by
1

Most people carry a spare tire in their car. It’s there in case you get a flat and need to change the tire before you can be on your way again. In my old VAR job I drove a lot away from home and to the middle of nowhere so I didn’t want to rely on roadside assistance. Instead I just grabbed the extra tire out of the back if I needed it and went on my way. However, the process wasn’t entirely hitless. Even the pit crew for a racing team needs time to change tires. I could probably get it done in 20 minutes with appropriate cursing but those were 20 minutes that I wasn’t doing anything else beyond fixing a tire.

Spare tires are redundant. You have an extra thing to replace something that isn’t working. IT operations teams are familiar with redundant systems. Maybe you have a cold spare on the shelf for a switch that might go down. You might have a cold or warm data center location for a disaster. You could even have redundant devices in your enterprise to help you get back in to your equipment if something causes it to go offline. Well, I say that you do. If you’re Meta/Facebook you didn’t have them this time last year.

Don’t mistake redundancy for resilience though. Like the tire analogy above you’re not going to be able to fix a flat while you’re driving. Yes, I’ve seen the crazy video online of people doing that but aside from stunt driving you’re going to have to take some downtime from your travel to fix the tire. Likewise, a redundant setup that includes cold spares or out-of-band devices that are connected directly to your network could incur downtime if they go offline and lock you out of your management system. Facebook probably thought their out-of-band control system worked just fine. Until it didn’t.

The Right Gear for Resilience

At Networking Field Day 29 last week we were fortunate to see Opengear present again for the second time. I’m familiar with them from all the way back at Networking Field Day 2 in 2011 so their journey through the changes of networking over the past decade has been great to see. They make out-of-band devices and they make them well. They’re one of the companies that immediately spring to mind when you think about solutions for getting access to devices outside the normal network access method.

As a VAR there were times that I needed to make calls to locations in order to reboot devices or get console access to fix an issue. Whether it was driving 3 hours to press F1 to clear a failed power supply message or racing across town to restore phone service after locking myself out of an SSH session there are numerous reasons why having actual physical access to the console is important. Until we perfect quantum teleportation we’re going to have to solve that problem with technology. Here’s a video from the Networking Field Day session that highlights some of the challenges and solutions that Opengear has available:

Ryan Hogg brings up a great argument here for redundancy versus resiliency. Are you managing your devices in-band? Or do you have a dedicated management network? And what’s your backup for that dedicated network if something goes offline? VLAN separation isn’t good enough. In the event of a failure mode, such as a bridging loop or another attack that takes a switch offline you won’t be able to access the management network if you can’t sent packets through it. If the tire goes flat you’re stopped until it’s fixed.

Opengear solves this problem in a number of ways. The first is of course providing a secondary access method to your network. Opengear console devices have a cellular backup function that can allow you to access them in the event of an outage, either from the internal network or from the Internet going down. I can think of a couple of times in my career where I would have loved to have been able to connect to a cellular interface to undo a change that just happened that had unintended consequences. Sometimes reload in 5 doesn’t quite do the job. Having a reliable way to connect to your core equipment makes life easy for network operating systems that don’t keep from making mistakes.

However, as mentioned, redundancy is not resiliency. It’s not enough for us to have access to fix the problem while everything is down and the world is on fire. We may be able to get back in and fix the issue without needing to drive to the site but the users in that location are still down while we’re working. SD-WAN devices have offered us diverse connectivity options for a number of years now. If the main broadband line goes down just fail back to the cellular connection for critical traffic until it comes back up. Easy to do now that we have the proper equipment to create circuit diversity.

As outlined in the video above, Opengear has the same capability as well. If you don’t have a fancy SD-WAN edge device you can still configure Opengear console devices to act as a secondary egress point. It’s as simple as configuring the network with an IP SLA to track the state of the WAN link and installing the cellular route in the routing table if that link goes down. Once configured your users can get out on the backup link while you’re coming in to fix whatever caused the issue. If it’s the ISP having the issue you can log a ticket and confirm things are working on-site without having to jump in a car to see what your users see.

Resilience Really Matters

One of the things that Opengear has always impressed me with is their litany of use cases for their devices. I can already think of a ton of ways that I could implement something like this for customers that need monitoring and resilient connectivity options. Remote offices are an obvious choice but so too are locations with terrible connectivity options.

If you are working in a location with spotty connectivity you can easily deploy an Opengear device to keep an eye on the network and/or servers as well as providing an extra way for the site to get back online in the event of an issue. If the WAN circuit goes down you can just hop over to the cellular link until you get it fixed. Opengear will tell you something happened and you can log into the Lighthouse central management system to go there and collect data. If configured correctly your users may not even realize they’re offline! We’re almost at the point of changing the tire while we’re driving.

Tom’s Take

I am often asked if I miss working on networking equipment since I rarely touch it these days. As soon as I’m compelled to answer that question I remember all the times I had to drive somewhere to fix an issue. Wasted time can never be recovered. Resources cost money whether it’s money for a device or time spent going to fix one. I look at the capabilities that a company like Opengear has today and I wish I had those fifteen years ago and could deploy them to places I know needed them. In my former line of work redundant things were hard to come by. Resilient options were much more appealing because they offered more than just a back plan in case of failure. You need to pick resiliency every time because otherwise you’re going to be losing time replacing that tire when you could be rolling along fixing it instead.

Disclaimer

Opengear was a presenter at Networking Field Day 29 on September 7, 2022. I am an employee of Tech Field Day, which is the company that managed the event. This blog post represents my own personal thoughts about the presentation and is not the opinion of Tech Field Day. Opengear did not provide compensation for this post or ask for editorial approval. This post is my perspective alone.

Brand Protection

Posted on by
Reply

I woke up at 5am this morning to order a new iPhone. I did this because I wanted the new camera upgrades along with some other nice-to-haves. Why did I get an iPhone and not a new Samsung? Why didn’t I look at any of the other phones on the market? It’s because I am a loyal Apple customer at this point. Does that mean I think the iPhone is perfect? Far from it! But I will choose it in spite of the flaws because I know it has room to be better.

That whole story is repeated time and again in technology. People find themselves drawn to particular companies or brands. They pick a new phone or computer or car based on their familiarity with the way they work or the design choices that are made. But does that mean they have to be loyal to that company no matter what?

Agree to Disagree

One of the things that I feel is absolutely paramount to being a trusted advisor in the technology space is the ability to be critical of a product or brand. If you look at a lot of the ambassador or influencer program agreements you’ll see language nestled toward the bottom of the legalese. That language usually states you are not allowed to criticize the brand for their decisions or talk about them in a disparaging way. In theory the idea is important because it prevents people from signing up for the program and then using the platform to harshly and unfairly criticize the company.

However, the dark side of those agreements usually outweigh the benefits. The first issue is that companies will wield the power to silence you to great effect. The worst offenders will have you removed from the program and potentially even sue you. Samsung almost stranded bloggers 10 years ago because of some brand issues. At the time it seemed crazy that a brand would do that. Today it doesn’t seem quite so far-fetched.

The second issue is that those agreements are written in such a way as to be able to cause issues for you even if you didn’t realize you were doing something you weren’t supposed to be doing. Think about celebrities that have tweeted about a new Android phone and the tweet has metadata that says sent with Twitter for iPhone. How about companies that get very upset when you discuss companies that they see as competitors. Even if you don’t see them as competitors or don’t see the issue with it you may find yourself running afoul of the brand when they get mad about you posting a pic of their product next to the supposed competition.

In my career I’ve worked at a value-added reseller (VAR) where I found myself bound by certain agreements to talk positively about brands. I’ve also found myself on the wrong side of the table when that brand went into a bidding process with another VAR and then tried to tell me I could say bad things about them in the process because I was also their partner. The situation was difficult because I was selling against a partner that went with another company but I also needed to do the work to do the bid. Hamstringing me by claiming I had to play by some kind of weird rules ultimately made me very frustrated.

Blind Faith

Do companies really want ambassadors that only say positive things about the brand? Do they want people to regurgitate the marketing points with everyone and never discuss the downsides of the product? Would you trust someone that only ever had glowing things to say about something you were trying to buy?

The reality of our world today is that the way that people discuss products like this influences what we think about them. If the person doing the discussion never has a negative thing to say about a company then it creates issues with how they are perceived. It can create issues for a supposedly neutral or unbiased source if they only ever say positive things, especially if it later comes out they weren’t allowed to say something negative for fear they’d get silenced or sued.

Think about those that never say anything negative toward a brand or product. You probably know them by a familiar epithet: fanboys. Whether it’s Apple or Tesla or Android or Ford there are many people out there that aren’t just bound by agreement to always speak positively about something. They will go out of their way to attack those that speak ill of their favorite product. If you’ve every had an interaction with a fan online that left you shaking your head because you can’t understand why they don’t see the issues you know how difficult that conversation can be.

As a company, you want people discussing the challenges your product could potentially face. You want an honest opinion that it doesn’t fit in a particular vertical, for example. Imagine how upset a customer would be if they bought your product based on a review from a biased influencer only to find that it didn’t fit your need because the influencer couldn’t say anything negative. Would that customer be happy with your product? Would the community trust that influencer in the future?

Tom’s Take

Honesty isn’t negativity. You can be critical of something you enjoy and not insinuate you’re trying to destroy it. I’ll be the first person to point out the shortcomings of a product or company. I’ll be fair but honest. I’ll point out where the improvements need to be made. One of the joys of my day job at Tech Field Day is that I have the freedom to say what I want in my private life and not worry about my work agreements getting me in trouble as has happened with some in the past. I’ll always tell you straight up how I feel. That’s how you protect your brand. Not with glowing reviews but with honest discussion.

When Were You Last a Beginner?

Posted on by
1

In a couple of weeks I’m taking the opportunity to broaden my leadership horizons by attending the BSA leadership course known as Philmont Leadership Challenge. It’s a course that builds on a lot of the things that I’ve been learning and teaching for the past five years. It’s designed to be a sort of capstone for servant leadership and learning how to inspire others. I’m excited to be a part of it in large part because I get to participate for a change.

Being a member of the staff for my local council Wood Badge courses has given me a great opportunity to learn the material inside and out. I love being able to teach and see others grow into leaders. It’s also inspired me to share some of those lessons here to help others in the IT community that might not have the chance to attend a course like that. However the past 3 years have also shown me the value of being a beginner at something from time to time.

Square One

Everyone is new at something. No one is born knowing every piece of information they’ll need to know for their entire lives. We learn language and history and social skills throughout our formative years. When we get to our career we learn skills and trades and figure out how to do complex things easily. For some of us we also learn how to lead and manage others. It’s a process of building layer upon layer to be better at what we do. Those skills give us the chance to show how far we’ve come in a given area by the way we understand how the complex things we do interact.

One of my favorite stories about this process is when I first started studying for my CCIE back in 2008. I knew the first place I should look was the Cisco Press certification guide for the written exam. As I started reading through the copy I caught myself thinking, “This is easy. I already know this.” I even pondered why I bothered with those pesky CCNP routing books because everything I needed to know was right here!

The practitioners in the audience have already spotted the logical fallacy in my thinking. The CCIE certification guide was easy and remedial for me because I’d already spent so much time reading over those CCNP guides. And those CCNP guides only made sense to me because I’d studied for my CCNA beforehand. The advanced topics I was refreshing myself on could be expanded because I understood the rest of the information that was being presented already.

When you’re a beginner everything looks bigger. There’s so much to learn. It’s worrisome to try and figure out what you need to know. You spend your time categorizing things that might be important later. It can be an overwhelming process. But it’s necessary because it introduces you to the areas you have to understand. You can’t start off knowing everything. You need to work you way into it. You need to digest information and work with it before moving on to add more to what you’ve learned. Trying to drink from a firehose makes it impossible to do anything.

However, when you approach things from a perspective of an expert you lose some of the critical nature of being bad at something. You might think to yourself that you don’t need to remember a protocol number or a timer value because “they never worry about that anyway”. I’ve heard more than a few people in my time skip over valuable information at the start of a course because they want to get to the “good stuff” that they just know comes later. Of course, skipping over the early lessons means they’re going to be spending more time reviewing the later information because they missed the important stuff up front.

Those Who Teach

You might think to yourself that teaching something is a harder job. You need to understand the material well enough to instruct others and anticipate questions. You need to prep and practice. It’s not easy. But it also takes away some of the magic of learning.

Everyone has a moment in their journey with some technology or concept where everything just clicks. You can call it a Eureka moment or something similar but we all remember how it felt. Understanding how the pieces fit together and how you grasp that interconnection is one of the keys to how we process complex topics. If you don’t get it you may never remember it. Those moments mean a lot to someone at the start of their journey.

When you teach something you have to grasp it all. You may have had your Eureka moment already. You’re also hoping that you can inspire one in others. If you’re trying to find ways to impart the knowledge to others based on how you grasped it you may very well inspire that moment. But you also don’t have the opportunity to do it for yourself. We’re all familiar with the old adage that familiarity breeds contempt. It’s easy to fall into that trap with a topic you are intimately familiar with.

In your career have you ever asked a question about a technical subject to an expert that started their explanation with “it’s really easy…”? Most of us have. We’ve probably even said that phrase ourselves. But it’s important to remember that not everyone has had the same experiences. Not everyone knows the topic to the level that we know it. And not everyone is going to form the same connections to recall that information when they need it again. It may be simple to you but for a beginner it’s a difficult subject they’re struggling to understand. How they comprehend it relies heavily on how you impart that knowledge.

Wide Eyed Wonder

Lastly, the thing that I think is missing in the expert level of things is the wonder of learning something new for the first time. It’s easy to get jaded when you have to take in a new piece of information and integrate it into your existing view. It can be frustrating in cases where the new knowledge conflicts with old knowledge. We spent a lot of time learning the old way and now we have to change?

Part of the value of being a beginner is looking at things with fresh eyes. No doubt you’ve heard things like “this is the way we’ve always done it” in meetings before. I’ve written about challenging those assumptions in the past and how to go about doing it properly but having a beginner perspective helps. Pretend I’m new to this. Explain to me why we do it that way. Help me understand. By taking an approach of learning you can see the process and help fix the broken pieces or optimize the things that need to be improved.

Even if you know the subject inside and out it can be important to sit back and think through it from the perspective of a beginner. Why is a vanilla spanning tree timer 50 seconds? What can be improved in that process? Why should things not be hurried. What happens when things go wrong? How long does it take for them to get fixed? These are all valid beginner questions that help you understand how others look at something you’re very familiar with. You’ll find that being able to answer them as a beginner would will lead to even more understanding of the process and the way things are supposed to work.

Tom’s Take

There are times when I desperately want to be new at something again. I struggle with finding the time to jump into a new technology or understand a new concept because my tendency is to want to learn everything about it and there are many times when I can’t. But the value of being new at something isn’t just acquiring new knowledge. It’s learning how a beginner thinks and seeing how they process something. It’s about those Eureka moments and integrating things into your process. It’s about chaos and change and eventually understanding. So if you find yourself burned out it’s important to stop and ask when you were last a beginner.

Certification Comfort Food

Posted on by
Reply

I’m a big fan of comfort food. Maybe more than I should be. The idea of something simple and tasty just hits the right spot a lot of time, especially when I’m stressed or don’t have time to do something more involved. I know I really need to be better about cooking but you can’t beat a quick meal that uses something simple and gets the job done, right?

Now, before you ask yourself what I’m on about this week, I want you to think about that analogy in terms of certifications and learning. When we’re starting out in the industry or we’re learning a new skill we have to pick up basic ideas. The more advanced or radical the technology the more we need the kinds of explanations that make the concepts simple to understand. We need the equivalent of learning comfort food. Simple, digestible, and easy to prepare.

Climbing the Ladder

As our skills improve we have the choice to continue on and develop our capabilities to greater depths. Perhaps we want to learn everything there is to know about BGP and policies. We could even parlay that networking knowledge into new adjacencies that build on our skill sets. We also have the option of staying in the basic level and honing those skills. Instead of learning VXLAN we could spend a thousand hours practicing all the ways that you can configure a VLAN.

Which way is right? Is there a need to make a choice? People are going to feel more comfortable doing one thing over the other in almost every case. If you’re like me you want to get to the bottom of every mystery and explore every nuance of something. Once you figure it out you’re going to want to move on to the next hard problem to solve. You become a voracious reader and consumer of knowledge and before you know it you’ve run out of things to consume. It’s partially the reason why I’ve been such a prolific writer for the past twelve years. I’ve been creating the content that I wanted to consume so others can benefit.

The other side of the choice is being content with the skills you have. This is in no way a negative thing. Not everyone that cooks needs to be a four star chef that makes perfect risotto and Beef Wellington every time. There is a place for everyone that learns enough to accomplish their goals and decides that is enough for them. If the above option is the “pull” model where one is trying to pull in new knowledge as fast as possible then this is the “push” version where people must be pushed to learn additional things. Your company might move to cloud and that would facilitate a need to pick up cloud operations skills to complement the ones you have for the network or the virtualization cluster. You’re not actively seeking the knowledge until it’s needed.

Boiling the Mudpuddle

It’s all well and good when you can recognize which type of learner you are. It’s also important to know where your resources are aimed. If your top destinations for content are part of the “push” model and aim at a lower level when you’re someone that wants to grow and investigate new areas you’re going to hit a wall eventually and sour on them.

A personal story for me comes when I was racing through my certification journey in the early part of my career. Once I started with Cisco I was consuming books left and right. Every time I went into the book store I picked up a new tome to teach me more about routing or remote access networks or even firewalls. I would consume that content whenever I could and apply those lessons to my job or my certification process. Eventually I knew I was reaching a limit because there were fewer and fewer books in the bookstore that taught me things I wanted to know. It made me realize there is a target market for these resources.

Things like certification guides are aimed at a wide market. They want to teach skills to the widest possible audience. Not everyone needs to know the ins and outs of EVPN but most everyone in networking needs to know how a switch forwards frames. If you want to sell the most books which would you write about? You’d write the one that covers the most people. It’s a reality of the market. Content for the entry level and the broadest group sells the best. In today’s world the book has been replaced by the blog and the YouTube channel.

As mentioned, I started my blogging career because of the above bookstore issue. Once I started learning things that weren’t in every book I wanted to share those ideas. That got me to Tech Field Day and eventually to different things. It also made me realize that while my content may never have hundreds of thousands of readers for every post it would serve people that needed to find those lessons or understand those topics in a depth that was beyond a paragraph or two in a 400-page encyclopedia of terminology.

To me, the certification comfort food is that entry-level content. It’s always going to be there. It’s simple to write about, especially when you have good analogies to frame new concepts for people. It’s tasty when you’re starving. And you can make a very good living doing it. But if you’re the kind of person that wants to try new tastes and break away from the comfort and ease you’re going to need to figure out your own path. You need to experiment and make mistakes and struggle to conceptualize what you’re talking about. You need to expand your horizons and do new things and then tell the world how you did it. Like a recipe blog or TikTok channel for cooking you’re going to need to put your crazy ideas out there and see how it goes.

Tom’s Take

There are a lot of great creators out there that have made a very good place for themselves teaching newcomers the basics of how things work. I applaud them and wish them nothing but success. I also know that’s not for me. I started writing about my CCIE studies and the challenges I was solving the real world. Now I write about the state of the market or the changing of tech or how to build and lead teams. It’s very representative of my journey as well as the journeys of those in the community that I talk to. My very nature won’t let me stay in a little bubble and create the same things in new ways. I’m going to push the envelope and explore new things. It means I might not land in everyone’s top list but it also means I won’t be bored. Why be mac-n-cheese when I really need to be risotto?

The Puzzle of Peering with Kentik

Posted on by
1

If you’ve worked at an ISP or even just closely with them you’ve probably hearing the term peering quite a bit. Peering is essentially a reciprocal agreement to provide access to networks between two providers. Provider A agrees to allow Provider B to send traffic over and through their network in exchange for the same access in the other direction. Sounds easy, right? On a technical level it is pretty easy. You simply set up a BGP session with the partner provider and make sure all the settings match and you’ve got things rolling.

The technical part isn’t usually where peering gets complicated. Instead it’s almost always related to the business side of things. The policy and negations that have to happen for a good peering agreement take way more time that hammering out some BGP configuration stanzas. The amount of traffic to be sent, the latency requirements, and even the cost of the agreement are all things that have to be figured out before the first hello packet can be exchanged. This agreement is always up for negotiation too, since the traffic patterns can change before you realize it and put you at a disadvantage.

Peerless Data Collection

If you want to get the most out of your peering arrangement you need to know what’s going on. You need to have statistics about the key points of your agreement. You need to know if you’re holding up your end of the bargain as well as the company you’re working with. If you walk into a peering negotiation without the right data you’re going to be working from a disadvantage right away.

For example, did your partner company take all of the traffic they agreed to accept in the peering contract? Or did they have issues that forced you to send the traffic along a different route? Were you forced to send that traffic across a different route that had a higher cost? Did your users complain about network speed because of congestion outside of your control? If you can’t put your fingers on the answers to these questions quickly you’re going to find yourself with lots of angry users and customers not to mention peering partners that want answers from you as well.

Recently I had a chance to listen to a great presentation from Kentik during Networking Field Day: Service Provider. Nina Bargisten laid out some of the challenges that Kentik customers face with peering arrangements and how Kentik is helping to solve them:

One of the points that Nina discusses is that capacity planning is a huge undertaking for ISPs. With the supply chain issues that we’re currently facing in 2022 it’s not easy to order equipment to alleviate congestion problems. Even under somewhat normal circumstances it’s not likely that an ISP is going to go out and order a lot of new hardware just to deal with congestion. They might change some polices to route traffic in different directions but ultimately the decision has to be made about how to get customer packets through and out of their network to the ultimate destination.

Peering agreements can help with congestion. Adding more exit points to your network means some flows can exit through a different provider and either get to their destination faster or prevent a larger connection from being overwhelmed and congested. It’s not unlike having multiple options to use to arrive a destination when driving. Some streets are better for smaller amounts of traffic compared to larger highways and interstates that provide high-speed travel.

As mentioned above, it’s critical that you have data on your traffic and its performance. Are you sending everything through one route? Are you peering with providers that are getting less than half of the traffic load they agreed to take? These are all questions you have to ask to create a capacity plan. If you’re hearing complaints about congestion but you see that only two of your outbound connections are running a full capacity while the rest are sitting idle then you don’t have a congestion issue as much as you have a configuration problem to solve.

Kentik’s solution allows you to see what’s going on and help you make better decisions about the routes that traffic should be taking. As demonstrated above, their dashboard collects data from your network as well as many others and can tell you when you need to be configuring polices to send traffic to low volume peers instead of relying on congested links. It will also help you see trends for when links become congested and allow you to set thresholds to divide your traffic appropriately before it becomes an issue.

There’s a lot more info in the video above to help you with your capacity planning and peering negotiations. It all comes down to a simple maxim: Information is key. You can’t solve these puzzles without knowing what you have and what you need. If you’re just going to keep throwing peering agreements at a problem until it goes away you’re going to fail. You won’t solve your real issues by just adding another connection that never gets used. Instead you can use Kentik’s platform to provide the kinds of insights that will help you create value for your customers and save money at the same time.

Tom’s Take

Service providers think about traffic differently than enterprise admins. They have to worry about it coming into the network and leaving again. Instead of worrying about a couple of links to the wider Internet they have to worry about dozens. If you think it’s hard keeping track of all that data for the enterprise you can just imagine how hard it is when you scale it up to the service provider level. Thankfully companies like Kentik are applying their expertise to provide actionable information to help you make the right choices and maybe even negotiate some better deals.

If you’d like to learn more about this presentation, make sure you check out the full presentation on the Tech Field Day site or go to http://Kentik.com