“Experience is a harsh teacher because it gives the test first and the lesson afterwards.” – Vernon Law
When I was in college, I spent a summer working for my father. He works in the construction business as a superintendent. I agreed to help him out in exchange for a year’s tuition. In return, I got exposure to all kinds of fun methods of digging ditches and pouring concrete. One story that sticks out in my mind over and over taught me the value of the object lesson.
One of the carpenters that worked for my father had a really bad habit of breaking sledgehammer handles. When he was driving stakes for concrete forms, he never failed to miss the head of the 2×4 by an inch and catch the top of the handle on it instead. The force of the swing usually caused the head to break off after two or three misses. After the fourth or fifth broken handle, my father finally had enough. He took an old sledgehammer head and welded a steel pipe to it to serve as a handle. When the carpenter brought him his broken hammer yet again, my father handed him the new steel-handle hammer and said, “This is your new tool. I don’t want to see you using any hammer but this one.” Sure enough, the carpenter started driving the 2×4 form stakes again. Only this time when he missed his target, the steel handle didn’t offer the same resistance as the wooden one. The shock of the vibration caused the carpenter to drop the hammer and shake his hand in a combination of frustration and pain. When he picked up the hammer again, he made sure to measure his stance and swing to ensure he didn’t miss a second time. By the end of the summer, he was an expert sledgehammer swinger.
Amusing as it may be, this story does have a purpose. People need to learn from failure. For some, the lesson needs to be a bit more direct. My father’s carpenter had likely been breaking hammer handles his entire life. Only when confronted with a more resilient handle did he learn to adjust his processes and fix the real issue – his aim. In technology, we often find that incorrect methods are as much to blame for problems as bad hardware or buggy software.
Thanks to object lessons, I’ve learned to never bridge the two terminals of an analog 66-block connection with a metal screwdriver lest I get a shocking reward. I’ve watched others try to rack fully populated chassis switches by brute force alone. And we won’t talk about the time I watched a technician rewire a 220 volt UPS receptacle without turning off the breaker (he lived). Each time, I knew I needed to step in at some point to prevent physical harm to the person or prevent destruction of the equipment. But for these folks, the lesson could only be learned after the mistake had been made. I think this recent tweet from Teren Bryson (@SomeClown) sums it up nicely:
Some people don’t listen to advice. That’s a fact born out over years and years of working in the industry. They know that their way is better or more appropriate even against the advice of multiple experts with decades of experience. For those people that can’t be told anything, a lesson in reality usually serves as the best instructor. The key is not to immediately jump to the I Told You So mentality afterward. It is far too easy to watch someone create a bridging loop against your advice and crash a network only to walk up to them and gloat a little about how you knew better. Instead of stroking your own ego against an embarrassed and potentially worried co-worker, instead take the time to discuss with them why things happened the way they did and coach them to not make the same mistakes again. Make them learn from their lesson rather than covering it up and making the same mistake again.
I’ve screwed up before. Whether it was deleting mailboxes or creating a routing loop I think I’ve done my fair share of failing. Object lessons are important because they quickly show the result of failure and give people a chance to learn from it. You naturally feel embarrassed and upset when it happens. So long as you gather your thoughts and channel all that frustration into learning from your mistake then things will work out. It’s only the people that ignore the lesson or assume that the mistake was a one-time occurrence that will continually subject themselves to object lessons. And those lessons will eventually hit home with the force of a sledgehammer.
In some ways, you’ve explained this story in a sort of “Don’t touch the pan on the stove” way. At some point, a child does this and then they never touch the pan again.
In IT, however, we hear lots of advice about what to do or what not to do, and the scale of reference for the person giving the advice (or their competency level in general) might be more indicative of why they are giving the advice rather than some profound understanding of the technology. Even when people are giving you advice, sometimes it’s better to go into the lab and do it anyway and read the docs and sort out the details. I don’t know how many times I’ve heard a chorus of people telling me one thing, but it turns out *not one of them* really understood the problem.
Vendors will frequently sell you magic and unicorns, but when you go to implement, they tell you… “well you know… keep it simple, don’t enable so many complicated features.” I’ve literally seen purchases happen because of feature “x” and then when it’s time to go into production the vendor talks folks out of going down that path. And the customer ended up paying for something they didn’t need or couldn’t use because the vendor didn’t want to admit that the feature wasn’t ready for prime time or was poorly supported.
Even if someone has been, for instance, working LAN and datacenter switching for 20 years… how much do they really know if most of that time was with one vendor, one set of platforms, or at one company in an environment that doesn’t change frequently? One time, 5 years ago, they tried moving to OSPF, but it was too complicated so they went to static routes and EIGRP. Their advice? Don’t use OSPF. They tried that and it failed.
I say.. try stuff. In the lab first. Pick up the book, fiddle with the feature, run debugs, look at output, pull cables, reboot routers and switches and hosts, etc… etc… then go into production. The second thing is, for all the freaking out that happens whenever there is an outage, I haven’t worked too many places where there was real monetary (or other) impact to any business or person as a direct result of the outage. Noone died. No money was lost. The only reason people care is that a customer saw a 1 minute outage and they are escalating to the heavens that they expect 100% uptime and zero packet loss. Obviously there are critically important systems that can’t suffer even a few seconds of downtime, but how many of us work in those environments truly? Ask yourself.. did anyone get hurt? Can some quantify the monetary loss? I worked in the financial vertical for a while, and even there the only real fall out were the angry phone calls. Let’s be honest… a lot of those angry phone calls are about having leverage at the next contract renewal.
I will end up with a piece of advice though. Don’t run hidden commands (or hidden configuration options) in production without testing in a lab first. *ahem*. One of the worst outages I have ever caused was due to this. 🙂