Sharing Failure as a Learning Model

Earlier this week there was a great tweet from my friends over at Juniper Networks about mistakes we’ve made in networking:

It got some interactions with the community, which is always nice, but it got me to thinking about how we solve problems and learn from our mistakes. I feel that we’ve reached a point where we’re learning from the things we’ve screwed up but we’re not passing it along like we used to.

Write It Down For the Future

Part of the reason why I started my blog was to capture ideas that had been floating in my head for a while. Troubleshooting steps or perhaps even ideas that I wanted to make sure I didn’t forget down the line. All of it was important to capture for the sake of posterity. After all, if you didn’t write it down did it even happen?

Along the way I found that the posts that got significant traction on my site were the ones that involved mistakes. Something I’d done that caused an issue or something I needed to look up through a lot of sources that I distilled down into an easy reference. These kinds of posts are the ones that fly right up to the top of the Google search results. They are how people know you. It could be a terminology post like defining trunks. Or perhaps it’s a question about why your SPF modules are working in a switch.

Once I realized that people loved finding posts that solved problems I made sure to write more of them down. If I found a weird error message I made sure to figure out what it was and then put it up for everyone to find. When I documented weird behaviors of BPDUGuard and BPDUFilter that didn’t match the documentation I wrote it all down, including how I’d made a mistake in the way that I interpreted things. It was just part of the experience for me. Documenting my failures and my learning process could help someone in the future. My hope was that someone in the future would find my post and learn from it like I had.

Chit Chat Channels

It used to be that when you Googled error messages you got lots of results from forum sites or Reddit or other blogs detailing what went wrong and how you fixed it. I assume that is because, just like me, people were doing their research and figuring out what went wrong and then documenting the process. Today I feel like a lot of that type of conversation is missing. I know it can’t have gone away permanently because all networking engineerings make mistakes and solve problems and someone has to know where that went, right?

The answer came to me when I read a Reddit post about networking message boards. The suggestions in the comments weren’t about places to go to learn more. Instead, they linked to Slack channels or Discord servers where people talk about networking. That answer made me realize why the discourse around problem solving and learning from mistakes seems to have vanished.

Slack and Discord are great tools for communication. They’re also very private. I’m not talking about gatekeeping or restrictions on joining. I’m talking about the fact that the conversations that happen there don’t get posted anywhere else. You can join, ask about a problem, get advice, try it, see it fail, try something else, and succeed all without ever documenting a thing. Once you solve the problem you don’t have a paper trail of all the things you tried that didn’t work. You just have the best solution that you did and that’s that.

You know what you can’t do with Slack and Discord? Search them through Google. The logs are private. The free tiers remove messages after a fashion. All that knowledge disappears into thin air. Unlike the Wisdom of the Ancients the issues we solve in Slack are gone as soon as you hit your message limit. No one learns from the mistakes because it looks like no one has made them before.

Going the Extra Mile

I’m not advocating for removing Slack and Discord from our daily conversations. Instead, I’m proposing that when we do solve a hard problem or we make a mistake that others might learn from we say something about it somewhere that people can find it. It could be a blog post or a Reddit thread or some kind of indexable site somewhere.

Even the process of taking what you’ve done and consolidating it down into something that makes sense can be helpful. I saw X, tried Y and Z, and ended up doing B because it worked the best of all. Just the process of how you got to B through the other things that didn’t work will go a long way to help others. Yes, it can be a bit humbling and embarrassing to publish something that admits you that you made a mistake. But It’s also part of the way that we learn as humans. If others can see where we went and understand why that path doesn’t lead to a solution then we’ve effectively taught others too.


Tom’s Take

It may be a bit self-serving for me to say that more people need to be blogging about solutions and problems and such, but I feel that we don’t really learn from it unless we internalize it. That means figuring it out and writing it down. Whether it’s a discussion on a podcast or a back-and-forth conversation in Discord we need to find ways to getting the words out into the world so that others can build on what we’ve accomplished. Google can’t search archives that aren’t on the web. If we want to leave a legacy for the DenverCoder10s of the future that means we do the work now of sharing our failures as well as our successes and letting the next generation learn from us.

How Long Should You Practice

A reporter once asked boxing legend Muhammad Ali how many sit-ups he did each day. I’m sure the reporter wasn’t expecting Ali’s answer. Ali replied with:

I don’t know. I don’t start counting them until it hurts. Those are the only ones that count. That’s what makes you a champion.”

Ali knew that counting things is just a numbers game. Five hundred poor sit-ups don’t count as much a fifty done the right way. With any practice that you do the only things that count are the things that teach your something or that push you to be better.

Don’t Practice Until It’s Right

People used to ask me how long I would spend at night studying for the CCIE lab. I told them I usually spent between five and seven hours depending on what I was studying. Sometimes those people would say things like “I’m not talking about setup time. I’m talking about actual lab work.” I always countered by making them explain why the setup isn’t part of the “real” work. That’s usually when they went quiet.

It’s far too easy to fall into the trap of overlooking things that you think are unimportant. A task you’ve done a hundred times is no big deal until you do it wrong the next time. Like Ali above, the things you do that require no effort don’t count. If you’re practicing a skill for a certification or a lab you need to put the same effort into it every time to ensure you’re doing it correctly. Lack of attention means you are doing it without gaining something from it.

I spend a lot of my time teaching things to people all over the place. I teach IT and networking skills to professionals. I teach outdoor skills to scouts of all ages. I teach merit badges and other things to a variety of youth. And I teach my kids life skills they will need. Every one of these lessons comes with instruction in the little details that matter. Every lesson also includes guidance that it needs to be practiced properly until it’s right. And then some.

Until You Can’t Get It Wrong

I tell my students and kids all the time, “Don’t practice until you get it right. Practice until you can’t get it wrong.” The level of involvement that it takes to get past the part where something finally works up to the level where it works every time is as wide as the gap at the lower end of the spectrum.

Too often people are content to work on something until they get it once. Whether it’s tying a knot or programming a router interface or even cooking a grilled cheese sandwich. Once you’ve done it right once you’re done with learning, right? Most of you are already shaking your head because you know that’s not right.

Once you get it right the first time you’ve already made a list of all the wrong ways to do something and you avoid them in the future. However, that list doesn’t include the entirety of all the wrong ways to do a thing. Amateurs make somewhat predictable mistakes because they’re working from the same basic knowledge. It’s when someone says they know what they’re doing that the real crazy stuff starts coming out of the woodwork.

Once you’ve practiced a skill you need to keep going. You need to work a variety of different angles to make sure you’ve covered all the ways you could get it wrong. If you’re tying a knot you need to practice with different kinds of ropes or in different positions. If there are two ways to tie something, practice them both. You don’t want to be an expert at a clove hitch over the end of a pole only to find out you have to tie it around the middle with no way to use the loop method you have memorized.

In IT, we lab things up to make sure we understand them. For these labs, try out the things in wrong ways. Click buttons before you’re supposed to. Put the wrong numbers in the field. See how the system will try to correct your errors. Maybe it doesn’t even bother? It’s easy to figure out you typed something in wrong when you hear a bell and see a message. It’s harder to troubleshoot when you don’t see anything right away and it all falls over later.

The extra practice above and beyond the first success is just like Muhammad Ali’s sit-ups. The hard ones count. The tasks that stretch your mind are the ones that build your skillset. You can’t give up when the answer isn’t right at your fingertips. Going that extra mile is the key to making yourself a better professional in whatever you do.


Tom’s Take

As we wind down 2020 we’re all looking to be better at things. Hobbies, skills, or professional talents are all calling to us to work on in whatever down time we have available to us. Make that practice count. Work hard to get it right every time. If you want to learn to make hollandaise sauce or write a novel or do a forward flip you have to keep practicing even after your success. Get to the point where you have no other choice but to get it right every single time. That’s the perfect amount of practice you need. Anything less counts as much as Muhammad Ali’s sit-ups before they start hurting.

Seeking Knowledge and Willful Ignorance

I had a great time recording a fun episode of Seeking Truth in Networking, an awesome podcast with my friends Derick Winkworth and Brandon Heller. We talked a lot about a variety of different topics, but the one I want to spend a few more minutes on here came in the first five minutes. Brandon asked me what question I liked to be asked and I mentioned that love to be asked about learning. My explanation included the following line:

I feel like the gap between people that don’t understand something and the willfully ignorant is that ability to take a step out and say “I don’t know the answer to this but I’m going to find out.”

I’ve always said that true learners are the ones that don’t accept the unknown. They want to find the answer. They want to be able to understand something as completely as they can. Those that I consider to be willfully ignorant choose not to do that.

Note that there is a difference between incidentally ignorant and willfully ignorant. People who are incidentally ignorant are unaware they don’t know something. They haven’t had the opportunity to learn or change their thought process on something. It would be like going to a random person and asking them about how to launch a rocket to another planet. They’re ignorant of the steps because they’ve never had the opportunity to learn them. They’ve never been exposed to the info or had a need to know it. People who are willfully ignorant choose to not learn something even after they’re exposed to it.

Where There’s a Will

We deal with people who choose not to learn things all the time. Even I choose not to learn everything. I don’t have all the Pokemon memorized. I don’t have the registration number of every Starfleet vessel in my mental Rolodex. There are a variety of other more technical topics that escape me. However, my reasoning for choosing not to learn those things is not because of malice. It’s because of self preservation.

If you exposed to something that you are curious about and choose to learn more you will often find yourself consumed by it. I am always on the lookout for a new laptop bag or hiking backpack. When I search for them I will find myself watching videos and reading reviews that are full of terminology that I don’t understand. I educate myself to the best of my ability but I don’t consider myself to be an expert on messenger bags or ultralight hiking packs. And after a while that knowledge is filed away for another day and I have to relearn something all over again when I’m on the hunt for a new bag.

Let’s contrast the acknowledgment of not being able to know everything with the phenomenon of choosing not to learn something out of spite or malice. This is like a networking engineer saying something along the lines of, “I’m not going to learn OSPF because it sucks and I’ll never use it.” A statement like that should immediately raise flags. In this specific case a working knowledge of OSPF is important for anyone building and maintaining networks. You may not need to know the details for every LSA in the database but you at least need to know how OSPF is different from RIP.

This kind of willful ignorance of information makes IT difficult. Why? Because actively choosing not to learn or understand something creates two hurdles to overcome. The first hurdle is showing people where to learn more about it. That is hard enough in and of itself. Every bookshelf in every office everywhere has the kinds of books that people refer to when they need to teach someone something important about networking or wireless or any other enterprise IT technology. Thanks to the power of search engines today it’s even more accessible to get people on the track to learning something new.

The second huge hurdle with those that are willfully ignorant isn’t access to knowledge. It’s getting past their objections to learning it. People have biases that need to be challenged and overcome. I’m not going to speak on anything aside from technology but we all know that everyone has their viewpoint and their understanding and changing their mind about something has varying degrees of difficulty. If someone is convinced, for example, that SHA-1 is an unbreakable protocol and nothing you can show them to the contrary convinces them that is willful ignorance. Evidence that is contrary to the facts isn’t invalid evidence. The quality of the evidence is always important to understand but choosing to dismiss it entirely out of hand solely because it doesn’t fit your understanding is not the kind of position someone in IT needs to take.

A Changing Landscape

Think about some of the following statements:

  • Switching is cheap, routing is expensive
  • 640K is more than enough memory
  • Unbreakable encryption

These are all statements that have been said in the past. They’ve all been proven over time to be wrong. Could you imagine if there was someone out there today that though programs needed to run in less than 640K of RAM? Or that believed that routing packets was too expensive and everything needs to run at layer 2? Those people would get laughed out of the data center.

Those statements are no longer true, but the attitudes behind them are the real problem. It’s not that something is taken for granted but that we choose not to accept anything other than that fact as the truth. Even today we could have positions like virtual reality will never take off or that quantum computers are too noisy to ever be commercially viable. In five years or a decade those statements may prove to be totally wrong. But if I’m still saying them and purposely choose not to learn why they are incorrect then I’m in the camp of being willfully ignorant of the truth.


Tom’s Take

The point of this post wasn’t to call out anyone specific for anything. Instead, I wanted to highlight that we all believe what we believe and we resist learning things that don’t square with that. How we choose to overcome that friction defines us as well as defining us with our peers. If you want to spend your career believing that a protocol is better and you won’t learn anything else then I hope your career is successful and long. I say “hope” because in the world of IT those that clap their hands over their ears and refuse to update their knowledge and understand are running on that kind of hope. They hope their level of knowledge never needs to change. They hope their skills will be enough for years and years of employment. And, in almost every case, they hope they can learn something new fast enough to get a new job when they realize that the attitude of willful ignorance will leave you high and dry.