IPv4, My Old Comfy T-Shirt

I have a t-shirt in my closet that I’ve owned since my freshman year of college.  It’s a plain gray shirt, full of holes and practically threadbare.  It’s a size or two too small at this point, not quite having survived the assault waged by my increasing appetite and decreased physical activity.  It’s stained and faded.  It also happens to be the most comfortable shirt that I own despite all of the above statements.  No matter what, when I dig down to the bottom of the vendor t-shirt pile, it’s always there waiting for me.  And every time I see it, I can’t help but put it on and wear it around for the rest of the day.  Because it brings back good memories and it still feels good, small and worn-out though it may be.  And no matter how many times my wife tells me to throw it out or hides it in the back of the closet, I’ll still hold on to it for years to come.

And now you ask yourself, “What was the whole point of that tangent???”  Well, dear readers, IP version 4 is a lot like my old t-shirt.  It’s familiar.  It’s been around forever.  Yes, it’s full of holes and has some ugly dirtiness about it that we’d rather not have to deal with (**cough NAT cough**).  It’s too small, having been outgrown by the very Internet it helped to found.  Our appetites for devices and connectivity have outpaced our activity in keeping current with addresses to consume.  But, it’s still a comfortable old friend to look upon and experience nostalgia.  The first time you “got” subnetting.  The day you read RFC 1918.  The first time you accidentally configured a host with the broadcast address of a subnet.  The day you figured out what those Class E addresses were REALLY for (Shhh…it’s a secret to everybody.)  And it still works for the majority of the world at large.

IP version 6 isn’t just a pie-in-the-sky vision, it’s a cold hard reality of the world we live in.  We’re going to have to move, and we have to do it sooner rather than later.  Pretty soon, all my appliances are going to have IP addresses, and if my toaster can’t talk to the microwave, there’ll be hell to pay.  But in order to facilitate that kind of universal connectivity, we have to make some hard choices.  We have to go back and add on to the years of work we’ve done making our grand Internet.  And it scares the hell out of a lot of us.  Most network, um, rock stars can tell you their IP scheme off the top of their head.  They’ve spent time and effort and reams of paper documenting the management VLAN IPs and loopbacks.  And the thought of wrecking all that for the new, shiny protocol that still a little mysterious frightens and downright pisses them off.  Or, implementing IPv6 on top of all their hard work makes them upset they’re having to reinvent the wheel all over again.  And so, they dig down into the pile of t-shirts and find their old friend, IPv4.  They find ways of making it work for a few more months.  Carrier NAT, class A reallocation, class E allocation, call it what you want.  It’s like telling yourself that if you go to the gym that old t-shirt will fit again, only saying it while you’re eating Doritos.  We all know it’s time for IPv4 to go.  Yes, it will still be around for a long time to come so users can access the Internet.  Similar to good old DOS, it’ll be here in one form or another well past it’s useful life.  And much like my wife, everyone is saying that the old IPv4 has to go.  And much like I do when my wife tells me that, I nod my head in agreement and then promptly ignore her (Shhh…don’t tell her I said that).  Because as long as IPv4 is still around, we’ll still be able to remember the glory days.

So maybe in 2011, it’s time to really take a look at IPv6 and what it’s going to take to pull our networks into a new generation.  Finally enable dual-stack on our devices and stop consuming so many IP addresses.  Drive a stake through the heart of NAT and sweep it under the rug where it belongs.  But when I go through all of that, you can rest assured I’m going to be wearing my comfy old t-shirt.  That is, if my wife doesn’t find it first.

God Help Us, We’re In the Hands of Engineers

A comment over on Jeremy Stretch’s wonderful site touched off a bit of a discussion today about the proper use of the term ‘engineer’.  It appears that the “real” engineers in the world have gotten into a bit of a tiff with us computer nerds about why we aren’t allowed to call ourselves engineers.  A little background:

According to the font of all knowledge, an engineer is:

...a professional practitioner of engineering, concerned with applying scientific knowledge, mathematics and ingenuity to develop solutions for technical problems. Engineers design materials, structures, machines and systems while considering the limitations imposed by practicality, safety and cost. The word engineer is derived from the Latin root ingenium, meaning "cleverness".
Engineers are grounded in applied sciences and are distinguished from scientists who perform research and artists who create with a focus on aesthetics. The work of engineers forms the link between scientific discoveries and the applications that meet the needs of society.

(COPIED FROM HERE)

Hmm, okay.  So, that sounds like it could be a lot of things.

- applying scientific knowledge (check)

- Mathematic knowledge (check)

- Developing solutions and systems (check)

- Considering limitations of cost and practicality (double check)

Okay, so far it looks like myself and my brethren are engineers for sure.  Ah, but wait…here come the scientists:

According to multiple sources, the “real” engineer is distinguished by holding a 4-year Bachelor of Science degree in an engineering discipline, having 6 years experience, and taking one or more tests.  This appears to be the most common set of requirements (more on this later).  So, because I didn’t put down civil engineering as my major in college, I’m out.  Because I don’t have six or more years experience building sewers or roads or bridges, I’m disqualified.  And because I haven’t forked over $1000 or more to the state licensing board, I’m just a lowly technician worthy of engineer nerd scorn.

What I have done, though, is spend my career building complex systems relying on specific scientific and mathematic principles.  Knowledge that not everyone who’s ever hit the power button on their laptop has.  I make the magical packets flow so you can watch Youtube videos and download dirty pictures.  I make your e-mail work so you can get blueprints and site updates.  I keep the QoS flowing so your emergency need to check Google Earth to make sure no one built a subdivision in the way of your highway.  I do things with my knowedge that would make you cringe and look the other way because you don’t understand them.

I took industry-specific tests from Novell, Microsoft, Cisco, HP, and Symantec to prove my knowledge.  Those governing bodies granted me the right to use the titles MCNE, MCSE, ASE, and SCSE (all have the word ‘engineer’).  And, for the record, these are the same tests no matter if you take them in Texas or Thailand.  I know of engineers that took their tests in other states because they were ‘easier’ and used comity to be licensed in their home state. So, in a sense, I am an engineer for all those reasons.  Yet, the professional engineers (P. Eng) get all huffy about it.  For the record, I thought P. Eng was Sean Combs’ name this week…

In the US, you are a P. Eng if you take the classes and tests and have enough experience and get signed off upon by other engineers.  But you can still use the title “engineer” through the use of a industrial exemption.  This means that if I work in industry providing engineering services, I’m exempt from getting licensed.  This exemption is pissing P. Eng’s off left and right.  Why, might you ask?  Because civil, mechnical, and electrical engineers are using it to get past taking the tests.  The IEEE is having kittens because they think anyone involved in engineering services that directly impact public safety should be required to be licensed before they use the term.  I can see that insofar as it impacts the safety of other people.  But yelling at lowly computer nerds because they claim to be “engineers”?  Tilting at windmills, in my opinion.  And heaven help you if you claim to be an engineer in Canada without a license.  I think they dispatch the Mounties to shoot you in the street like a common horse thief.

You wanna complain about me using the title “network engineer”.  Okay, let’s switch spots for a week.  I’ll spend all day staring at blueprints trying to figure out which way the road is supposed to go.  You can head back to my desk and keep the network running and figure out why the boss’s e-mail isn’t lightning fast.  You can unclog the tubes and carry my pager to respond to network outages at 3:30 in the morning on a Sunday.  You can put up with me calling you asking why I can’t get to Youtube today to watch some cat playing the drums.  And if you don’t run screaming back to me within a day loudly proclaiming that I am truly an engineer, I promise I’ll start calling myself something totally different.  Like Network Rock Star.  Since nobody needs any talent to be one of those.

Sending Calls Directly to Voice Mail

The number one feature request I get when I setup a new CallManager system is for direct voice mail transfers.  But you might say, “That’s easy.  Just put the iDivert softkey on the softkey template.” Oh ho, not so fast.  This request is for people to transfer a caller directly to someone else’s voice mail.  It seems that I either deal with some very polite customers that don’t want to interrupt the other person with a phone call or don’t want to the caller to wait the four or so rings before voicemail picks up.  Either that, or direct voice mail transfers are great ways to get rid of pushy salespeople.

At any rate, this is a feature that should probably have been baked into CallManager by now but hasn’t been for some reason.  Don’t ask me why.  But should you ever find the need to set it up for yourself, here’s how you do it:

1.  Figure out a transfer pattern. It goes without saying that  you should figure out what pattern you are going to use to allow calls to do direct voice mail transfer.  Some people are huge fans of using the pound/hash/octothorpe key.  Since I hate explaining exactly what an “octothorpe” is to people, I tend to favor the star/asterisk key.  Mainly because I like saying “asterisk”  You could also use a combination of the octothorpe and asterisk and one or more numbers.  The important thing is that you need to make sure it doesn’t collide something in your current dial plan.

2. Configure the Voice mail Profile.  Ah, the meat of things.  In order to send a call straight to voice mail, you need a new voice mail profile.  Name it something that will make you remember what it does, like “TransferToVM” (my personal favorite).  Just don’t put the word ‘voice mail’ in it.  Unity seems to like to do really screwy things when the name includes the word ‘voice mail’.  Make sure the TransferToVM profile points at your current Voice mail Hunt Pilot.  Also, the voice mail mask needs to match your current extension length.  In my example below, the extension length is 3 digits, so the voice mail mask is “XXX”.

3. Create a CTI Route Point. CTI route points are like the tunnels of the voice world.  They are used to fix a multitude of issues in a pinch.  In this case, we need a device that is going to answer the phone rain or shine, and a CTI route point fits the bill.  I name mine the same as the voice mail profile just to keep everything straight.  Just be sure that it’s in a calling search space that can call your Unity/Unity Connection server.  And mind the naming conventions for CTI route points (they’re kind of picky).

4. Assign a Directory Number. Once you click ‘Save’ on the CTI route point, it’s going to allow you to assign a directory number.  This needs to be a combination of your voicemail transfer access code (from step #1) and your voice mail box mask (from step #2).  Make sure that it is contained in a partition that is dialable from your internal phones.  The important things to note here are that the voice mail profile for the CTI DN needs to be the profile created above (TransferToVM in this case).  Also, the Forward All settings need to be checked to send the call directly to voice mail when dialed.

And there you have it.  If you followed my example above, whenever you want to send a caller directly to Johnny’s voice mail at extension 101, you just need to dial *101.  Works like a charm.

Fast Tracks and Shiny Plaques

HP has announced a new certification program called ExpertONE (http://h10120.www1.hp.com/certification/expert_one-networking.html).  This appears to be the culmination of the acquisition of 3COM/Huawei and the rebranding of Procurve as “HP Networking”.  In this new program, they have consolidated their existing tracks and certifications to fall into the familiar 3-tiered system of associate (Advanced Integration Specialist or AIS), Professional (Advanced System Engineer or ASE) and Expert (Master Advanced Systems Engineer or Master ASE).  The current tracks include networking, wireless, security, and voice.

What is of particular interest is the “Fast Track” program.  This program allows an individual certified in a competitor’s certification system to use these certifications to achieve an equivalent HP certification level.  For instance, if you hold a valid CCNA, you can take the HP2-Z04 Building HP Procurve Campus LANs exam and achieve the HP AIS: Networking certification.  Taking the same test and submitting a valid CCIE: R&S gives you the Master ASE: Networking certification.  While I can say that I like the approach that HP has taken by allowing existing vendor certifications to count towards their certification track, I do have a couple of problems with it.

1.  It’s a major modification from the existing track. My reasoning for this?  In the previous track, you could take one test that covered the convergence aspect of Procurve switches (basically multicast routing and QoS) and you could achieve the ASE: Convergence certification.  In order to become a Master ASE: Convergence all you needed to do was submit a valid CCVP certificate. (http://h10147.www1.hp.com/training/certifications/technical/convergence.htm)  That’s what I did.  And for the next 11 days, I am still a Master ASE: Convergence.  I even have the shirt to prove it.  But as of November 1st, that track will expire and there is no current projected replacement for it.  In an effort to realign their business tracks, HP has expired all previous certifications in favor of the new ExpertONE program.  No option to recertify in a track.  In fact, it appears the ONLY way to become a Master ASE is to hold a CCIE (or perhaps JNCIE) and take this one online test.  No other major vendor has ever expired all of their certification tracks at once, to my knowledge.  When Novell moved from Netware 5 to Netware 6, if you were certified on Netware 5 you could still claim to be a CNE, but Novell would inform those that asked that you were not certified on the current OS.  I’m still a MCNE on Netware 6.  I’m an MCSE on Windows 2000.  All expired tracks, yet the certification is still valid.  But with HP?  Nope.  No ASE for you unless you have the current certification.  But that’s not the most concerning thing about this.

2.  HP seems to be trying to attract Cisco talent out of spite. It’s no real secret that HP and Cisco in the last year have gone from friendly rivals to outright war with each other.  From the Cisco “California” UCS product line to the acquisition of 3COM/Huawei, the pitched battles keep getting fought over and over.  In fact, the announcement of the ExpertONE certification track was released at the same time Cisco announced changes to the CCIE Service Provider, CCNP: Voice, and CCNP: Security tracks.  HP has done everything in its power to pick as many fights with Cisco as it can.  And this new certification track is no different in my mind.  By claiming that anyone with a valid Cisco certification can now hold an equivalent HP Networking certification, HP is telling networking professionals they value the learning that those professionals have accomplished, even if they don’t care much for the logo on the certificate.  One test could certify me in 3 or 4 different tracks for HP due to my Cisco certifications.

This appears to me to be an effort by HP to win over a large portion of the networking professional community by giving them a head start in the HP certification program.  I can say that the idea of being able to gain some nice HP certifications because of my standing with Cisco is a nice idea.  But at the same time, I wonder what is going to happen in the future.  The Fast Track program won’t last forever.  HP is already prepping new tests and tracks for the November – January timeframe.  In my mind, that says that if you want to take advantage of the Fast Track program, you’d best do it now.  It may not be long before HP decides to ‘expire’ the Fast Track option in favor of new, developed coursework.  I’m also curious how long the CCIE will be a prerequisite for the Master ASE.  While you could be very certain that you are getting the cream of the crop by requiring a CCIE as a prerequisite for any certification, given HP’s previous actions of excising any trace of Cisco they can find makes me wonder how long it will last.  Perhaps until HP can implement their own lab program similar to the CCIE or JNCIE.  But those programs take time to develop and properly implement.  Until that time, I think HP is viewing the CCIE as a necessary evil.  And, quite possibly, HP will use the numbers of CCIEs gaining Master ASEs as a marketing tool to justify how advanced their certification program is becoming.

In the end, I think that HP has got the right idea.  While the prospect of losing my Master ASE due to reorganization does chafe somewhat, I think the program realignment was necessary to make the certification program have some prestige and level the playing field.  However, I’m couching my opinion until I see exactly how long the Fast Track program lasts.  And I hope that this isn’t just another example of the networking professional community being dragged into a vendor war.

I Got a +5 Tunnel of SSH!

I had an opportunity this week to record an episode of the PacketPushers Podcast.  It was a great outing that dealt with a lot of listener questions.  One of the questions that we didn’t get time to get to, however, involved online gaming and SSH tunneling.  I figured I’d do a little more research into it and see what exactly it is that makes this service work.

Background

The game in question here is World of Warcraft (WoW), easily the #1 Massively-Multiplayer Online Role Playing Game (MMORPG) in the world.  At any one time, millions of players are logged on to any of the almost 250 servers that comprise the game.  Consequently, these servers are located in datacenters housed all over the world in an effort to provide close support and (hopefully) low latency connections.  In the MMORPG world, the lower your latency to the server, the smoother things appear in game.  When latency increases strange things start happening as the player’s client attempts to update the server as to the location of the player’s character, and the server attempts to update the player’s client as to the location of objects in the realm.  When this starts happening, player’s experience the dreaded lag.  Lag causes objects to appear out of nowhere, or objects to warp around the screen, or in the worst cases the player’s client freezes waiting for an update from the server.  As you can see, having a fast connection is very important for the enjoyment and playability of the game.

SSH Port Forwarding – The Beginning

Originally, the idea of using SSH to forward WoW traffic came about because of firewalls.  WoW communicates with the realm servers on TCP port 3724.  Many firewalls, especially those found in enterprise networks, allow known traffic such as HTTP or DNS while blocking all unknown protocols.  In other cases, firewall admins have specifically blocked traffic known to be associated with peer-to-peer (P2P) protocols, such as Bittorrent or Limewire.  At any rate, players found that being behind these firewalls rendered them unable to play WoW.  Some enterprising players found that if they encapsulated the packets in SSH and forwarded them to an SSH server that had port forwarding enabled, they were able to connect and play.  Essentially, this involves taking the traffic on port 3724 and forcing through an SSH client (like PuTTY) and forwarding it on to an SSH server.  The server would then act as a proxy and forward the traffic on to the WoW datacenter.  Since SSH is a well-known and quite useful protocol, it is very likely to be passed along without a second thought.  Also, as SSH is an encrypted protocol, the firewall isn’t able to break the packet apart and inspect it to determine what kind of traffic it contains.  So, through the use of SSH and a proxy server, users were able to play from just about anywhere

Now, how to get people to pay for it

One of the side effects of using SSH forwarding to circumvent firewalls was that some users were seeing their latency drop as a result.  Especially for players located in more remote areas of the world, tunneling the traffic to a location with a faster connection caused the somewhat-high ping times to drop to more acceptable levels.  A few companies, such as SmoothPING or WoWTunnels have taken this idea to its logical extreme and started charging users for the ability to lower their latency.  For a small fee each month, you pay for the use of a client, which automates the whole process of modifying your system to encapsulate the WoW packets in SSH.  You also get access to a proxy server that then forwards these encapsulated packets on to the WoW datacenters.  The WoWTunnels website claims that the latency is decreased because the packets take a “different path” to your particular WoW server.

This “different path” claim is the reason behind the question to PacketPushers.  The listener wondered if these services were just moving the packets on to a faster connection or if they indeed had a secret backdoor into the WoW datacenters.  The answer to this question is actually quite easy and requires no real magic.  Yes, the packets are taking a different path to the data center.  The packets are being pushed through an SSH tunnel to a server that forwards them on to the WoW servers.  In essence, this forwarding server is acting as a proxy.  If the proxy server has a fast enough connection to the destination it should decrease your latency.  As well, by tunneling the traffic as it exits your network, you avoid having it be scanned by firewalls or packet shapers, thereby avoiding these devices dropping your packet priority or increasing latency.

In the end, tunneling your WoW traffic in SSH can decrease your latency for several reasons related to firewalls and faster connections.  When you pay someone to automate the process for you, you are essentially paying for them to keep upgrading the pipe they have from their servers to the WoW datacenter.  As long as they keep their user traffic segregated and avoid filling up the proxy connection you should see a good connection.  But remember that you don’t necessarily need to pay for this service.  If you have access to a server that can port forward SSH and aren’t afraid to get your hands a little dirty, give this link a try.  But remember your mileage may vary.

Now put that thing back where it came from or so help me…

When troubleshooting problems, we often find ourselves mired in a sea of options.  Google searches, technical documentation, tricks from our magical networking bag, and so on.  And more often that not, it takes more than one solution to actually fix a problem.  Magic bullets are very hard to come by in Information Technology.  So, when Google search option #1 fails, it’s time to move down the list to option #2…

WAIT! STOP RIGHT THERE!!!

Yes, you heard me.  Before you move on to option #2, you’ve got something to do first.  Before you get that big head of steam built up troubleshooting, you’ve got to clean up after yourself.  Yes, it’s time to undo option #1.  Now, I know what you might be asking yourself right now: “Huh?  What?  Undo something?” That’s absolutely correct.  If you try something that doesn’t work, you need to back out that change before you move on.  Why???

1.  If you end up trying 15 things to fix this particular issue, and one of them finally works, which one actually fixed the issue? Your first reaction is to say “Well, duh.  The last thing I did is what fixed it.” Usually that’s a good answer.  Unless Thing #9 needed 5 minutes to fix the problem in the background while you tried #10-#15.  Or, worse, #9 did something that allowed #13 to fix the issue.  The idea is that by backing out the changes if they don’t work, you can pinpoint what works more quickly.  Or, in some cases, narrow down a list of things that need to be done in concert to resolve the issue.  More than once I’ve asked someone how they fixed a problem only to be met with a shrug of the shoulders.  As a consultant or a technical resource for your company it’s vital to remember that if you can’t explain to a customer or your boss what you did to fix the problem, you didn’t actually fix anything.

2.  You don’t want to introduce any extra issues into the mix. If you don’t back out your changes before trying something new, there’s a very good chance that you’ll introduce an unexpected variable into the situation that could make your life miserable later on.  Or, in a worse case scenario, one of your previous ‘fixes’ causes a totally different issue after you’ve finished troubleshooting the original problem.  If you always take the time to back out irrelevant changes as you eliminate them as solutions, you don’t have to worry about them causing unforeseen interactions with your ultimate solution.  You don’t want to end up not being able to fix a routing issue because the routers won’t form neighbor relationships because you configured an access list that drops all multicast packets in a previous attempt.

As long as you remember to clean up as you go and back out any non-useful or non-functional changes, your troubleshooting life will be much easier.  You’ll find that you can more confidently explain solutions to customers and coworkers, as well as not introducing unforeseen consequences into your efforts.  You’ll look like a hero, money will fall from the skies, and the meek will worship the ground that your superior troubleshooting skill occupies.  At least, I think that is what’s supposed to happen if I just change this one other setting…

Misadventures of a Inbound Helpdesk Agent

People are always asking me what I do.  And I lie and say I’m a mortician.  Because invariably the next question involves me figuring out how to cut down on their webmail spam.  Or how to fix their computer.  But there was a time in my life when I didn’t have the option of lying my way out of questions like that.  Yes, in the beginning it wasn’t all the wine and roses of network engineering.  I was an agent on the help desk of a major computer company.  I’ve always said that everyone should spend a month on the help desk just so they can have the same low opinion of humanity that I do.  In the six months I was there, I learned that you can never assume what people will ask.  Ever.  I could spend days talking about all the things I’ve seen and heard.  Instead, I think I’ll share my three favorite stories from my tour of duty.

 


 

Unshiny Happy People

I received a call one night from a nice sounding old lady that can’t seem to get CDs to play on her laptop. I start to troubleshoot the issue, and nothing seems to be working. At about 15 minutes into the call, the following exchange occurs:

Her: Now, I have a question. Does the shiny side of the CD go up or down?
Me: The shiny side goes down, ma’am.
Shuffling goes on in the background…
Her: Oh, your fixed it! You are a miracle worker!

Here’s my vote for the return of the 8-track.

 


 

We, The (Ignorant) People…

A teenager calls me and wants me to help him copy a DVD. Like a movie DVD. Like a illegal-to-copy movie DVD. Informing him of this leads to this informative back-and-forth:

Me: I can’t help you, sir. It’s illegal to copy a DVD, just like it’s illegal to copy a tape or CD.
Him: You can copy a tape! It’s protected under the 3rd Amendment!

Catching me off guard with that one, I rattle off some garbage about the DMCA not allowing you to make copies of copyrighted material.  Yes, it’s the law, but I still think it’s garbage. It got rid of him well enough. But it stuck with me. After the call, we looked up the 3rd Amendment. It protects you against quartering soldiers in your home, and was written in 1789.

Damn British. First they comandeer your house. Then they make copies of Men in Black. Will they never learn?

 


 

To have and to hold…or your money back.

As usually happens on late night calls, I get a drunk customer. So drunk you can almost smell beer through the phone. I ask him to verify his computer serial number. He can’t find it. I ask him for a phone number. He can’t remember. I try to search for his last name. He slurs it so badly I though he sneezed. At this point, I tell him I can’t help him without any info. He then tells me he wants me to help him break his computer.

Me: Sir, I can’t help you do that. Even if I could, I wouldn’t. I fix computers, I don’t break them.
Him: You don’t understand. My wife chats with men on it while I’m at work all day. If I break it, she can’t talk to them.

Even with the impassioned plea of a man whose keyboard it getting more action that he is, I had to decline. He wants to speak to my supervisor. I tell him my supervisor can’t do any more that I have. He insists. My supervisor takes the call while I listen in on another line (and yes, we can hear you). He goes through the same song-and-dance, and gets the same reply. He then reveals this particularly juicy bit of info:

Him: She talks to these men on the computer and then brings them home! She’s already brought three of them home and slept with them while I’m at work!
Rolling laughter from those of us out of earshot
Him: She also downloads pictures off the Internet of naked men and hangs them up in our bedroom!
Hideous laughter from us that interrupts phone calls for other technicians
Him: She even buys dildos! She has one that is this long!
We assumed this to be a length of about 12 inches, and began howling laughter that made the callers think we were watching Clerks or something.

The man then begins to complain about his life in general, as well as his marriage. He even counseled my supervisor to avoid marriage at all costs. At this point comes the greatest line every uttered on a call:

Supervisor: I’m sorry sir, but our company does not warranty your marriage.

Not only did I gain so much respect for my supervisor, but in the multiple retellings of the story over the next few days, we both became heroes. And for the life of us, we couldn’t figure out why he didn’t just hit it with a hammer. Maybe he was too drunk to remember how to use one.