The Dangers of Knowing Everything

Posted on by
Reply

By now I’m sure you’ve heard that the Internet is obsessed with ChatGPT. I’ve been watching from the sidelines as people find more and more uses for our current favorite large language model (LLM) toy. Why a toy and not a full-blown solution to all our ills? Because ChatGPT has one glaring flaw that I can see right now that belies its immaturity. ChatGPT knows everything. Or at least it thinks it does.

Unknown Unknowns

If I asked you the answer to a basic trivia question you could probably recall it quickly. Like “who was the first president of the United States?” These are answers we have memorized over the years to things we are expected to know. History, math, and even written communication has questions and answers like this. Even in an age of access to search engines we’re still expected to know basic things and have near-instant recall.

What if I asked you a trivia question you didn’t know the answer to? Like “what is the name of the metal cap at the end of a pencil?” You’d likely go look it up on a search engine or on some form of encyclopedia. You don’t know the answer so you’re going to find it out. That’s still a form of recall. Once you learn that it’s called a ferrule you’ll file it away in the same place as George Washington, 2+2, and the aglet as “things I just know”.

Now, what if I asked you a question that required you to think a little more than just recalling info? Such as “Who would have been the first president if George Washington refused the office?” Now we’re getting into more murky territory. Instead of being able to instantly recall information you’re going to have analyze what you know about the situation. For most people that aren’t history buffs they might recall who Washington’s vice president was and answer with that. History buffs might take more specialized knowledge about matters would apply additional facts and infer a different answer, such as Jefferson or even Samuel Adams. They’re adding more information to the puzzle to come up with a better answer.

Now, for completeness sake, what if I asked you “Who would have become the Grand Vizier of the Galactic Republic if Washington hadn’t been assassinated by the separatists?” You’d probably look at me like I was crazy and say you couldn’t answer a question like that because I made up most of that information or I’m trying to confuse you. You may not know exactly what I’m talking about but you know, based on your knowledge of elementary school history, that there is no Galactic Republic and George Washington was definitely not assassinated. Hold on to this because we’ll come back to it later.

Spinning AI Yarns

How does this all apply to a LLM? The first thing to realize is that LLMs are not replacements for search engines. I’ve heard of many people asking ChatGPT basic trivia and recall type questions. That’s not what LLMs are best at. We have a multitude of ways to learn trivia and none of them need the power of a cloud-scale computing cluster interpreting inputs. Even asking that trivia question to a smart assistant from Apple or Amazon is a better way to learn.

So what does an LLM excel at doing? Nvidia will tell you that it is “a deep learning algorithm that can recognize, summarize, translate, predict and generate text and other content based on knowledge gained from massive datasets”. In essence it can take a huge amount of input, recognize certain aspects of it, and produce content based on the requirements. That’s why ChatGPT can “write” things in the style of something else. It knows what that style is supposed to look and sound like and can produce an output based on that. It analyzes the database and comes up with the results using predictive analysis to create grammatically correct output. Think of it like Advanced Predictive Autocorrect.

If you think I’m oversimplifying what LLMs like ChatGPT can bring to the table then I challenge you to ask it a question that doesn’t have an answer. If you really want to see it work some magic ask it something oddly specific about something that doesn’t exist, especially if that process involves steps or can be broken down into parts. I’d bet you get an answer at least as many times as you get something back that is an error message.

To me, the problem with ChatGPT is that the model is designed to produce an answer unless it has specifically been programmed not to do so. There are a variety of answers that the developers have overridden in the algorithm, usually something racially or politically sensitive. Otherwise ChatGPT is happy to spit out lots of stuff that looks and sounds correct. Case in point? This gem of a post from Joy Larkin of ZeroTier:

https://mastodon.social/@joy/109859024438664366

Short version: ChatGPT gave a user instructions for a product that didn’t exist and the customer was very frustrated when they couldn’t find the software to download on the ZeroTier site. The LLM just made up a convincing answer to a question that involved creating something that doesn’t exist. Just to satisfy the prompt.

Does that sound like a creative writing exercise to you? “Imagine what a bird would look like with elephant feet.” Or “picture a world where people only communicated with dance.” You’ve probably gone through these exercises before in school. You stretch your imagination to take specific inputs and produce outputs based on your knowledge. It’s like the above mention of applied history. You take inputs and produce a logical outcome based on facts and reality.

ChatGPT is immature enough to not realize that some things shouldn’t be answered. If you use a search engine to find the steps to configure a feature on a product the search algorithm will return a page that has the steps listed. Are the correct? Maybe. Depends on how popular the result is. But the results will include a real product. If you search for nonexistent functionality or a software package that doesn’t exist your search won’t have many results.

ChatGPT doesn’t have a search algorithm to rely on. It’s based on language. It’s designed to approximate writing when given a prompt. That means, aside from things it’s been programmed not to answer, it’s going to give you an answer. Is it correct? You won’t know. You’d have to take the output and send it to a search engine to determine if that even exists.

The danger here is that LLMs aren’t smart enough to realize they are creating fabricated answers. If someone asked me how to do something that I didn’t know I would preface my answer with “I’m not quite sure but this is how I think you would do it…” I’ve created a frame of reference that I’m not familiar with the specific scenario and that I’m drawing from inferred knowledge to complete the task. Or I could just answer “I don’t know” and be done with it. ChatGPT doesn’t understand “I don’t know” and will respond with answers that look right according to the model but may not be correct.

Tom’s Take

What’s funny is that ChatGPT has managed to create an approximation of another human behavior. For anyone that has ever worked in sales you know one of the maxims is “never tell the customer ‘no'”. In a way, ChatGPT is like a salesperson. No matter what you ask it the answer is always yes, even if it has to make something up to answer the question. Sci-fi fans know that in fiction we’ve built guardrails for robots to save our society from being harmed by functions. AI, no matter how advanced, needs protections from approximating bad behaviors. It’s time for ChatGPT and future LLMs to learn that they don’t know everything.

Friction as a Network Security Concept

Posted on by
Reply

I had the recent opportunity to record a podcast with Curtis Preston about security, data protection, and networking. I loved being a guest and we talked about quite a bit in the episode about how networking operates and how to address ransomware issues when they arise. I wanted to talk a bit more about some concepts here to help flesh out my advice as we talked about it.

Compromise is Inevitable

If there’s one thing I could say that would make everything make sense it’s this: you will be compromised. It’s not a question of if. You will have your data stolen or encrypted at some point. The question is really more about how much gets taken or how effectively attackers are able to penetrate your defenses before they get caught.

Defenses are designed to keep people out. But they also need to be designed to contain damage. Think about a ship on the ocean. Those giant bulkheads aren’t just there for looks. They’re designed to act as compartments to seal off areas in case of catastrophic damage. The ship doesn’t assume that it’s never going to have a leak. Instead, the designers created it in such a way as to be sure that when it does you can contain the damage and keep the ship floating. Without those containment systems even the smallest problem can bring the whole ship down.

Likewise, you need to design your network to be able to contain areas that could be impacted. One giant flat network is a disaster waiting to happen. A network with a DMZ for public servers is a step in the right direction. However, you need to take it further than that. You need to isolate critical hosts. You need to put devices on separate networks if they have no need to directly talk to each other. You need to ensure management interfaces are in a separate, air-gapped network that has strict access controls. It may sound like a lot of work but the reality is that failure to provide isolation will lead to disaster. Just like a leak on the ocean.

The key here is that the controls you put in place create friction with your attackers. That’s the entire purpose of defense in depth. The harder it is for attackers to get through your defenses the more likely they are to give up earlier or trigger alarms designed to warn you when it happens. This kind of friction is what you want to see. However, it’s not the only kind of friction you face.

Failing Through Friction

Your enemy in this process isn’t nefarious actors. It’s not technology. Instead, it’s the bad kind of friction. Security is designed by its very nature to create friction with systems. Networks are designed to transmit data. Security controls are designed to prevent the transmission of data. This bad friction comes when these two aspects are interacting with each other. Did you open the right ports? Are the access control lists denying a protocol that should be working? Did you allow the right VLANs on the trunk port?

Friction between controls is maddening but it’s a solvable problem with time. The real source of costly friction comes when you add people into the mix. Systems don’t complain about access times. They don’t call you about error messages. And, worst of all, they don’t have the authority to make you compromise your security controls for the sake of ease-of-use.

Everyone in IT has been asked at some point to remove a control or piece of software for the sake of users. In organizations where the controls are strict or regulatory issues are at stake the requests are usually disregarded. However, when the executives are particularly insistent or the IT environment is more carefree you can find yourself putting in a shortcut to get the CEO’s laptop connected faster or allow their fancy new phone to connect without a captive portal. The results are often happy and have no impact. That is, until someone finds out they can get in through your compromised control and create a lot of additional friction.

How can you reduce friction? One way is to create more friction in the planning stages. Ask lots of questions about ports and protocols and access list requirements before something is implemented. Do your homework ahead of time instead of trying to figure it out on the fly. If you know that a software package needs to communicate to these four addresses on these eight ports then anything outside of that list should be suspect and be examined. Likewise, if someone can’t tell you what ports need to be opened for a package to work you should push back until they can give you that info. Better to spend time up front learning than spend more time later triaging.

The other way to reduced friction in implementation is to shift the friction to policy. If the executives want you to compromise a control for the sake of their own use make them document it. Have them write it down that you have been directed to add a special configuration just for them. Keep that information stored in your DR plan and note it in your configuration repositories as well. Even a comment in the access list can help understand why you had to do something a certain way. Often the request to document the special changes will have the executives questioning the choice. More importantly, if something does go sideways you have evidence of why the change was made. And for executives that don’t like to look like fools this is a great way to have these kinds of one-off policy changes stopped quickly when something goes wrong and they get to answer questions from a reporter.

Tom’s Take

Friction is the real secret of security. When properly applied it prevents problems. When it’s present in too many forms it causes frustration and eventually leads to abandonment of controls or short circuits to get around them. The key isn’t to eliminate it entirely. Instead you need to apply it properly and make sure to educate about why it exists in the first place. Some friction is important, such as verifying IDs before entering a secure facility. The more that people know about the reasons behind your implementation the less likely they are to circumvent it. That’s how you keep the bad actors out and the users happy.

Why Do YOU Have To Do It?

Posted on by
1

One of the things that I’ve seen as a common thread among people in the industry as of late is the subject of burnout. Sure, burnout is a common topic no matter what year we’re in but a lot more of what I’m starting to hear about is self-inflicted burnout. Taking on too many projects, doing more than one job, and even having too many things going on outside of your specific role are all contributors to burnout. How can we keep that from happening?

Atlas and His Burden

For me, one of the biggest reasons why I find myself swimming in frustration is because I am very quick to volunteer to do things. In part it’s because I want to make sure the job is done correctly. In another part it’s because I want to be seen as someone that is always willing to get things done. Add in a dash of people pleasing and you can see how this spirals out of control. I’m sure you’ve even heard that as a career advice at some point. I’ve even railed against it many times on this blog.

How can you overcome the impulse to want to volunteer to do everything? If you’re not in a more senior role it’s going to be hard to tell someone you can’t or won’t do something. As I learned last year from commenters you don’t always have that luxury. If you are in a senior role you also may find yourself quickly volunteering to ensure that the job is done properly. That’s when you need to ask an important question:

Why do I have to do this?

Check your ego at the door and make sure that your Aura of Superiority is suppressed. This isn’t about you being better than the job or task. This is about determining why you are the best person to do this job. Seems easy at first, right? Just explain why this is something you have to do. But when you dig into it things get a little less clear.

Are you the most skilled person at this task in the company? That’s a good reason for you to do it. But could you offer to show someone else how to do the thing instead? Especially if you’re the only one that can do it? Cross training ensures that others know what to do when time is critical. It’s also nice to be able to take a vacation without needing to check your email every ten minutes. Enabling others to do things means you’re not the only phone call every time it needs to be done.

Is this something you’re worried won’t be done correctly if you don’t do it? Why? Is it something very difficult to accomplish? If so, why not have a team work on it? Is this something that you already have an idea of how you want to do it? That’s a recipe for trouble. Because you’ll implement your ideas for the thing and then either get bored or distracted and forget all about it. That leads to others thinking you’ve dropped the ball. It could also lead to people passing you over when you have good ideas because they’re afraid you’re going to take the ball and drop it later. If you think that it won’t be done correctly without your input you should find a way to add your input but not make yourself responsible for the completion of the project.

Are you just taking on the task for the accolades of a job well done? Do you enjoy the feeling of being called out for a successful completion of something? That’s fairly standard. Do you enjoy being chastised when you fail? Does it bother you when you’re called out in front of the team for not delivering something? Again, standard behavior of a normal person. The problem is when the need for the former outweighs the aversion to the latter.

In this excellent Art of Network Engineering episode with Mike Bushong he recounts a story of a manager that pushed back against him when he complained that no one knew how busy he really was. Her response of “everyone just sees you not getting things done” really made him stop and realize that taking the entire world on your shoulders wouldn’t make anything better if you kept failing to deliver.

I could go on and on and belabor the point more but I think you understand why it’s important to ask why the task can’t be reassigned or shared. Rather than just refusing you’re trying to figure out if anyone else should be doing it instead of you. As someone with too many things to do it’s critical you’re able to get those done. Adding more to your plate won’t make anyone’s job any easier.

Tom’s Take

I feel that I will always struggle to keep from taking on too many things at once. It’s not quite a compulsion but it’s also difficult not to want to do something for someone or take on a task that really should be done by someone with more skill or more time. The key for me is to stop and ask myself the question in the title. If I’m not the best person to be doing the job or if there is someone else that I can show so I’m not the only one that knows what to do then I need to do that instead. Sharing knowledge and ensuring others can do the tasks means everyone is involved and you’re not overwhelmed. And that makes for a happier workplace all around.

Why Aren’t There More Technical MBAs?

Posted on by
3

This weekend I was listening to the latest episode of the Art of Network Engineering podcast featuring Russ White. Russ is one of those guests that has a breadth of knowledge outside of technology that colors the way he looks at things in the realm of enterprise IT. Plus he’s a fun interview.

One of the questions he asked was around the idea of technical MBAs. For those that might not know, MBA stands for Masters of Business Administration, which is a post-graduate college business degree. An MBA is widely regarded as a way to put yourself on a track to be a manager for a company in some capacity. An MBA is punching a ticket to be a future CEO. Why does it seem like the number of MBAs coming out of prestigious business schools don’t have a technical background then?

Where the Action Is

I’m going to quote liberally from a book I’ve been reading called The Personal MBA by Josh Kaufman. In it, Kaufman lays out the reasoning behind using his book to study the principles behind the information given in MBA classes instead of spending $200,000 to go to a business school for two years for the same material. The layout of the first few sections gives a great overview of the way that most people think about MBAs as well as how most MBAs view the business world.

The original reason to get an MBA was to apply scientific reasoning to management. The goal was to have managers for factories that looked at the numbers and made sound decisions based on facts and not hunches. In theory that was a great way to get started. However, the numbers started to change in favor of a newer, more exciting way to apply numbers. Instead of crunching them to make people more productive, MBAs started moving toward ways to apply that analysis to make the company rich.

In the past 50 years the number of business school graduates going into finance and consulting has exploded. Per Kaufman, something like 60% of the people getting MBAs have moved toward finance as their preferred area of management. When you think about it that makes all the sense in the world. Finance is where a company can get bigger and more profitable. Value creation can be heavily impacted by workers that are looking to invest profits and reduce costs across the board to increase return to investors. Likewise, investors more heavily reward managers that make them money. This feedback loop means the easiest way to pay off your MBA debt is to go into finance and make the company rich.

On the flip side of the discussion, do you think of value creation when you think of IT? I can remember way back in college when getting my BBA in MIS and being taught that I was going to be spending a lot of my time convincing management that IT departments were not just cost centers. Sales generates revenue. Marketing supports sales. Accounting makes sure we’re actually making money. IT? Well, we ask for stuff and then install it and then fix it when it breaks. We don’t actually do anything to make money.

I know a lot of my readers are probably shouting at their screens right now talking about how IT supports the business and enables the value creation that other parts of the organization do. You’d be absolutely right in your assertion. However, the stigma that IT has is that we take money and don’t give any back to the organization directly. Kind of like how plumbers and electricians are still seen as “dirty” or “low brow” kinds of jobs. Despite evidence the contrary IT is drag on the whole organization. IT doesn’t do the hard work of making money through emails or apps or ensuring databases are up and running to search through for customer leads.

Given that IT is a cost center and the real rock stars all go to finance to keep the shareholders and investors happy is it any wonder that MBAs want to work on spreadsheets instead of Excel? If you had a choice between limiting your career trajectory while also having a lot of business school debt to pay off would you do it knowing tech is a much more exciting role? The question pretty much answers itself.

Promote From Within

Let’s look at the second major reason that MBAs are always technical disciplines. Who is in charge of your IT department now? Is it an MBA? Is it a manager that came from accounting or sales? Or is it someone technical that got promoted a few times and now finds themselves in charge of all the IT teams? Does that manager have a formal background in management or a degree? Or do they learn the ropes as they go and feel a bit rough around the edges?

In many IT organizations that I’ve worked with and worked for the management in those areas tends to be promoted from the bottom up. Instead of a more formalized structure of managers trained to be managers and hired into the organization to manage, IT is more about enabling intelligent and qualified individuals to maintain the department. You see far more people with advanced technical certifications or years and decades of technical experience working to keep the lights on as opposed to the lessons of quantitative analytics from business school being applied to streamline operations.

Admit it. You’d rather work for someone who knew the tech as opposed to someone that had been trained to be a manager right? Is there anything worse than walking into a meeting with the head of the IT department knowing that they won’t understand the challenges of a new hardware installation or implementing a new protocol that could impact operations? The nerve of those people! If they would just get someone technical in that seat that you could talk to things would be different for sure. This place might even run right for once!

The fact is that most technical professionals prefer someone that understands them over someone that is good at managing others. They would rather spend time talking to a promoted tech resource that understands the differences between SSL and TLS instead of someone that was trained in how to go to management and ask for more budget this quarter. We’re more comfortable with people that understand the things we do. It’s the same in accounting and finance and sales. We want to be managed by people that know how to deal with our special areas of expertise.

Extend that to the business school. MBAs go into finance so most finance people are managed by MBAs. IT is managed by people that worked in IT, which is more focused on certification and practical knowledge rather than formal learning in universities. The divide grows each time a new class of business school graduates starts applying for roles in a company. Would you take an interview with a cost center department full of people that automatically don’t like you because they think you don’t know the difference between a switch as an access point? Or would you rather talk to people that “get” you in the finance department?

There have been attempts to bridge this gap. In the above episode AJ Murray mentioned a hybrid degree that offered training in IT and in business. I too looked at the possibility of obtaining an MBA and a MS in MIS from my graduate school. As mentioned by AJ that major was the least popular option at his school and it didn’t last too long at mine either. That’s because the amount of work that you have to do for both degrees is far above what you’d have to do in order to get just the MBA. And there isn’t a lot of crossover either. You could be in a quantitive analysis class for finance on Monday and writing Python code on Tuesday for a program that was due at the end of the semester. It just doesn’t make sense when most of the candidates would prefer one over the other.

Tom’s Take

Russ wondered why there aren’t more people with a technical background getting an MBA. My answer is that the ones with a technical background don’t want it and the ones that want it don’t want to deal with tech. Management doesn’t care if you can manage a team of high-performing nerds to streamline operations for the business. They do care if you can cut costs and use financial wizardry to make the company more profitable. Why do you think so many people who are CEOs are former CFOs and not former CTOs or CIOs? The MBA is a degree to teach people to manage and the roots of it go back much further than enterprise IT as a discipline. As much as we might hate to admit, MBAs are focused on the business, not the departments. And as long as the road to the CEO’s chair involves making money for the shareholders you won’t see many technical people in that chair any time soon.

Multicasting Content in the Twilight of Social Media

Posted on by
1

During Cisco Live EMEA last week I had an interesting conversation with a few people at the show around social media and how the usage of the platforms appears to be changing thanks to decisions made by the smartest people in a given broom closet. With the acceleration of the demise of Twitter as a platform I couldn’t help but comment on the fact that social media is becoming less about conversation and more about broadcast, which seemed to catch some of the people in the conversation off guard.

Back and Forth

Ever since the beginning of my time on Twitter, I’ve seen the platform as conversational instead of content-focused. Perhaps that’s the reason why the idea of a tweet storm has irritated me so much over the past few years. Twitter is about talking to people. It’s about interacting with them and creating a conversation in the noise. Twitter allows us to connect to people and exchange ideas and viewpoints.

Contrast that with other platforms in the social media spectrum. Specifically I’m thinking of Youtube video or TikTok videos. These platforms are designed to create content and send it to a number of people to view. It’s multicasting content to viewers. You don’t interact with people on a one-on-one basis. If Twitter is about the conversation then video platforms are all about the message. Sit here, listen to what I have to say, and don’t talk back.

You may already be thinking to yourself, “But what about the comments!” And you might even be right. However, comments aren’t conversation. They’re a way to add your own viewpoint to a message that isn’t necessarily seen by someone consuming the content. Case in point? Youtube has started hiding the comments behind a widget that you need to click on. TikTok puts the comments behind a button. You can’t see them unless you go looking for them. And most of the time you don’t even want to read them anyway. I mean, Don’t Read the Comments is pretty much a meme at this point, right?

Comments aren’t conversation. Why would you think they are? Because your voice gets added to the mix? Because you get to say what you think? Is anyone even listening? How many times does a conversation or reply chain in the comments on a video involve the creator aside from answering a specific question? How often do the comments turn into a disaster area of people arguing about things not even related to the topic of the content in the first place? Comment hijacking is more common than most people realize. For some platforms it may be the only reason comments exist.

Another important point about comment sections turning into cesspools comes from the lack of interaction with the content creator. A real conversation, like one that happens in person, allows for the expression of viewpoints and discussion of ideas. You don’t have to agree but you do have to listen. And you do have to follow rules for conversation. You don’t get to walk up to someone who expresses an idea and say “You’re wrong and I’m right and here’s why” without also having to listen to the other person’s viewpoint. Well, I mean you probably could but you’ll find yourself excluded from the conversation quickly because you’re not listening. You’re just broadcasting.

Blogs as a Medium

Note that blogs and blogging fall into the same category as the videos above. I write the things I think and you read them. You can think about it or comment about what you believe. I will read it (and I do read them) but I can choose not to reply. The comment section on my NAT66 posts are still going strong something like ten years later and I haven’t left a comment on that video in forever. I’ve said what I wanted to say on the matter and that’s that.

However, blogs differ from videos in one important aspect as far as I’m concerned. The text of the blog post is searchable, as are the comments. You can look for topics or ideas much more easily in written form than you can on video. TikTok especially prefers brevity over conversation. Sure, maybe you highlight and comment and make a new video about what you read but there’s no transaction there. You could spend hours with a conversation that takes about two minutes to have in person. All this happens in a vacuum because you can’t search for the conversation. You can only watch it play out.

Yes, that’s how conversations happen in real life too. Even now I’m recalling parts of what I said to people in the conversation that I had. I’m recalling the discussion that happened and what I said. The key difference here is that I’m writing it down for others to read later and add their own viewpoints. If you choose to share this post with your audience and have your own viewpoint you’re creating conversation with others. That’s a big difference in my mind. How many times have you sent a video to someone to start a conversation? Aside from “this looks pretty cool” or “did you know about this” or “this was funny”?

Tom’s Take

The twilight of Twitter and text-based social media is upon us. The future right now is wrapped up in video content because that’s what people are consuming. Look at how many platforms have added “features” to replicate what the hottest apps are doing right now? There’s still value in the conversation to me. I still feel that we are better off having the discussions and not just blindly consuming content in multicast format. But I’m also old. I love writing. I feel I can get my ideas across in a better way here and on conversational platforms. I relish the idea of back-and-forth over broadcast. But the waning light tells me that my way isn’t long for this world.

Monitoring Other People’s Problems

Posted on by
Reply

It’s Always the Network is a refrain that causes operations teams to shudder. No matter what your flavor of networking might be it’s always your fault. Even if the actual problem is DNS, a global BGP outage, or even some issue with the SaaS provider. Why do we always get blamed? And how can you prevent this from happening to you?

User Utopia

Users don’t know about the world outside of their devices. As soon as they click on something in a browser window they expect it to work. It’s a lot like ordering a package and having it delivered. It’s expected that the package arrives. You don’t concern yourself with the details of how it needs to be shipped, what routes it will take, and how factors that exist half a world away could cause disruptions to your schedule at home.

The network is the same to the users. If something doesn’t work with a website or a remote application it must be the “network” that is at fault. Because your users believe that everything not inside of their computer is the network. Networking is the way that stuff happens everywhere else. As professionals we know the differences between the LAN, the WAN, the cloud, and all points in between. However your users do not.

Have you heard about MTTI? It’s a humorous metric that we like to quote now that stands for “mean time to innocence”. Essentially we’re saying that we’re racing to find out why it’s not our problem. We check the connections, look at the services in our network, and then confidently explain to the user that someone else is at fault for what happened and we can’t fix that. Sounds great in theory, right?

Would you proudly proclaim that it’s someone else’s fault to the CEO? The Board of Directors? How about to the President of the US? Depending on how important the person you are speaking to might be you might change the way you present the information. Regular users might get “sorry, not my fault” but the CEO could get “here’s the issue and we need to open a ticket to fix it”. Why the varying service levels? Is it because regular users aren’t as important as the head of the company? Or is it because we don’t want to put in the extra effort for a knowledge workers that we would gladly do for the person that in ultimately going to approve a raise for us if we do extra work? Do you see the fallacy here?

Keeping Your Eyes Peeled

The answer, of course, is that every user at least deserves an explanation of where the problem is. You might be bent on justifying that it’s not in your network so you don’t have to do any more work but you also did enough work to place the blame outside of your area of control. Why not go one step further? You could check a dashboard for services on a cloud provider to see if they’re degraded. Or do a quick scan of news sites or social media to see if it’s a more widespread issue. Even checking a service to see if the site is down for more than just you is more information than just “not my problem”.

The habit of monitoring other services allows you to do two things very quickly. First is that it lowers the all-important MTTI metric. If you can quickly scan the likely culprits for outages you can then say with certainty that it’s out of your control. However, the biggest benefit to monitoring services outside of your organization that you rely on is that you’ll have forewarning for issues in your own organization. If your users come to you and say that they can’t get to Microsoft 365 and you already know that’s because they messed up a WAN router configuration you’ll head off needless blame. You’ll also know where to look next time to understand challenges.

If you’re already monitoring the infrastructure in your organization it only makes sense to monitor the infrastructure that your users need outside of it. Maybe you can’t get AWS to send you the SNMP strings you need to watch all their internal switches but you can easily pull information from their publicly available sources to help determine where the issues might lie. That helps you in the same way that monitoring switch ports in your organization helps you today. You can respond to issues before they get reported so you have a clear picture of what the impact will be.

Tom’s Take

When I worked for a VAR one of my biggest pet peeves was “not my problem” thinking. Yes, it may not be YOUR problem but it is still A problem for the user. Rather than trying to shift the blame let’s put our heads together to investigate where the problem lies and create a fix or a workaround for it. Because the user doesn’t care who is to blame. They just want the issue resolved. And resolution doesn’t come from passing the buck.

Being the Best at Beginning

Posted on by
Reply

The other day I was listening to an excellent episode of The Art of Network Engineering talking about technical marketing engineers (TME). The discussion was excellent and there was one line from Pete Lumbis in the episode that stuck with me. He said that one of the things that makes you good as a TME is being an “expert beginner”. That phrase resonates at lot with me.

Fresh Eyes on the Problem

I talked a bit about this last year when I talked about being a beginner and how exciting that it was to start over with something. As I compared that post to the AONE episode I realized that what Pete was talking about was a shift in mindset that gives you the energy and focus to pick things up quickly.

You may have heard the phrase “familiarity breeds contempt”. It’s a common phrase used to describe how we feel less impressed with things the more we learn about then. Our brains are wired to enjoy new things. We love new experiences, going to new places, or even meeting new people. The excitement and rush that we get from something unfamiliar causes our brain to devour things. It’s only once we become familiar with them that we feel the contempt set in.

It’s not always possible to avoid the contempt part of things. Think about something as dreary as your morning commute to work, whether it’s walking down the stairs or driving to the office. I used to joke that my car was practically on auto-pilot most mornings because I knew every bump in the road and every turn by heart. When I would go somewhere new I would have to focus more on road signs or directions.

Back to Basics

The beginner aspect of things is easier to deal with. That’s because we can trick ourselves into seeing something with fresh eyes. On a number of occasions I’ve mentioned my friend and mentor John Pross and his assertion that every upgrade or deployment that happened was like the first time doing it. He never took anything for granted. He always took things step-by-step. While this had the affect of him making sure everything was followed to the letter it also gave him the beginner aspect of looking for ways to improve or discovering new solutions to problems along the way.

Once the contempt or apathy sets in you’re going to get very good at just clicking through the steps to get to the end as fast as possible. If you don’t believe me think about how many times you’ve given directions that involve something like “Just click Next, Next, Next, Next and then you’re done”. Trust me, it sounds funnier when you say it out loud. But it speaks to the fact that we know the dialog boxes so well that we know they aren’t important. But what if they are?

If you want to understand what it feels like to be a beginner again and you’re having a hard time getting yourself in that mindset you should find a beginner and coach them through a task like a setup. Don’t just tell them what to do. Let them figure it out. Answer questions as they come up. Make them explain why they’re doing something. I bet you’ll learn a lot more as you have to help them understand why that configuration line is in there or why you always choose twice the amount of RAM in an instance. Once you see the process through the eyes of a beginner you have to learn it more completely in order to help them understand it.

In some roles, like a TME or a VAR engineer, the ability to be an expert beginner is critical to your job. You have to see a technology for the first time and pick up the basics quickly. I used to tell people that the excitement of being an engineer at a VAR was the variety of problems I’d be called on to solve. One day might be wireless clients. The next could be iSCSI storage arrays. Whatever the case may be I could count on finding myself in a new situation pretty regularly. It kept things exciting and made me realize I had to stay on my toes.

For those that work as product managers or on more specialized teams you need to make sure you’re taking time to approach things as a beginner. The “same old, same old” may not actually be the same any more. That kind of contempt and familiarity leads to the phrase “the way we’ve always done it” and doesn’t force you to challenge the process to understand how to improve it. Sometimes you need to step back and remember that you have to see everything for the first time.

Tom’s Take

Beginners shouldn’t feel like they’re a nuisance. In fact they should be celebrated for the energy and focus they bring to a task or project. For roles like a TME it’s important to bring the same kind of energy to new things. You can learn a lot when you allow your brain to soak up knowledge like a fresh sponge. More importantly, the ability to be a beginner helps you refine your knowledge base more and will ensure that you can explain a concept or process to someone with absolute certainty.

Friday Networking Field Day Thoughts

Posted on by
1

I’m wrapping up Networking Field Day 30 this week and as is always the case there was a lot of great discussion from both the presenters and the delegates outside of the presentations. It’s one of the reasons why I love doing this job even after almost ten years. I get to meet fun people and have an impact on so many things in the tech industry.

  • Network-as-a-Service is coming. We recorded a roundtable discussion about it and I think the impact that it’s going to have on mid-sized businesses is massive. It’s going to be like cloud. Not just in operational capability. It’s also going to be a huge driver for what you can do with your network in support of applications. The snowflakes may melt under the weight of the cookies we make from the cookie cutter deployments.
  • It feels like a lot of companies are trying to find what’s next. Part of that is coming from the ways that organizations are changing their outlook for what an office should be after the pandemic shutdowns. But still others are realizing they can’t use the same revenue stream for the next five years and hope to survive. This isn’t simply a game of trying to find an adjacent market to move into to drive growth to keep shareholders and investors happy. This is more about finding something that needs to be done because the alternative is no longer having a company.
  • Sadly, it looks as though third party Twitter clients are gone for good. This is beyond irritating to me because of the way that I choose to consume Twitter. I’ve seen a lot of chatter in various comment threads about third parties making money from the good graces of Twitter but the fact is that those programs drive a LOT of the way that power users interact with the system. If the exodus wasn’t already accelerating I would imagine you’re going to see a lot more coming very, very soon.

Tom’s Take

Stay tuned for all the great info from Networking Field Day on the Tech Field Day YouTube channel and don’t forget to thank your networking team today. You may not need to call them today to tell them something is down but trust me they will appreciate you calling just to say you appreciate them.

Controlling Your View of the World

Posted on by
2

Straw Bales on Hill Landscape, Tuscany, Italy

As I’m writing this it looks like Twitter has made some changes to the way that third-party clients interact with service. My favorite client, Tweetbot, is locked out right now. The situation is still developing but it’s not looking pretty for anyone using anything other than the web interface. While I will definitely miss the way I use Tweetbot I think it’s the kick I needed to move away from Twitter more than before.

A Window on the World

The apps that we use to consume and create content are the way that we view things. Maybe you prefer a webpage over an app or the way that one client displays things over another but your entire view is based on those preferences. If the way you consume your media changes your outlook on it changes too.

I didn’t always use Tweetbot to view Twitter. I tried using the standard app for a long time. It wasn’t until the infamous “Dickbar” incident back in 2011 that I broke away for something that wasn’t so slavishly dependent on ads. The trending topic bar might not have been specifically for ads at the time but the writing on the wall was there. The way I chose to view my content wasn’t compatible with the way that the service wanted to monetize it.

Fast forward to today. Instagram doesn’t show you time sequence posts. Neither does Facebook. Twitter now defaults to an algorithmic timeline. Apps like TikTok are built on their algorithms. It’s all a way to show you things the system thinks you want to see with a sprinkling of ads mixed in whenever they want to show them to you. Could you imagine a TV channel where the ads were just thrown in to the show whenever they wanted you to see one as opposed to more of a standard time? You probably can imagine because that’s what it feels like to watch platforms like Youtube.

Platforms cost money to operate. That fact isn’t lost on me. What is annoying is that trying to change the way I access the platform in order to serve more ads is going to cause more damage in the long run. Given the choice between using the web version of Twitter and just not using it I’m more inclined to the latter. I want a timeline that shows me tweets in chronological order without the need to change to that format every time I open the page. I want to see what I want to see without being forced to view things that don’t interest me. Forcing me to use the web app to see ads is almost as bad as forcing me to follow people that you think are interesting that I have no desire to interact with.

The Only Winning Move

I’m more than a passive participant in services like Twitter. I create content and use them to share it. I participate actively. And I don’t like the way I’m being forced to play.

I started a Mastodon account many years ago the last time Twitter looked like it was making changes. In the past few months I’ve migrated it to a new profile. The Mastodon interface has robust apps and a great way to interact with people. It may not have the numbers that Twitter does right now but it won’t take long for a lot of the best creators to go there instead.

More importantly, choosing an open platform over a monetized nightmare gives me hope that the real value of what we do on the Internet isn’t selling ads. It’s creating valuable things that people enjoy and sharing them with a receptive audience. That may not be the millions of people on Twitter right this minute but those millions of people are about to find out what it feels like when the best reasons to be on a platform are gone.

Tom’s Take

I want to choose how I see the world, not how someone wants me to see it. I want to decide how I share what I’ve made with people and when they see it, not have it placed behind other things a software function decides are more likely to be clicked. If the future of social media is endless ads and trickery designed to make me spend more time fighting through the timeline instead of consuming it then I guess my view of the world is outdated and needs to change. But it will change on my terms.

Problem Replication or Why Do We Need to Break It Again?

Posted on by
Reply

There was a tweet the other day that posited that we don’t “need” to replicate problems to solve them. Ultimately the reason for the tweet was that a helpdesk refused to troubleshoot the problem until they could replicate the issue and the tweeter thought that wasn’t right. It made me start thinking about why troubleshooters are so bent on trying to make something happen again before we actually start trying to fix an issue.

The Definition of Insanity

Everyone by now has heard that the definition of insanity is doing the same thing over and over again and expecting a different result. While funny and a bit oversimplified the reality of troubleshooting is that you are trying to make it do something different with the same inputs. Because if you can make it do the same thing over and over again you’re closer to the root cause of the issue.

Root cause is the key to problem solving. If you don’t fix what’s actually wrong you are only dealing with symptoms and not issues. However, you can’t know what’s actually wrong until you can make it happen more than once. That’s because you have to narrow the actual issue down from all of the symptoms. If you do something and get the same output every time then you’re on the right track. However, if you change the inputs and get the same output you haven’t isolated the issue yet. It’s far too common in troubleshooting to change a bunch of things all at once and not realize what actually fixed the problem.

Without proper isolation you’re just throwing darts. As the above comic illustrates sometimes the victory is causing a different error message. That means you’re manipulating the right variables or inputs. It also means you know what knobs need to be turned in order to get closer to the root cause and the solution. So repeatability in this case is key because it means you’re not trying to fix things that aren’t really broken. You’re also narrowing the scope of the fixes by eliminating things that don’t need to be monitored.

Resource Utilization

How long does it take to write code? A couple of hours? A day? Does it take longer if you’re trying to do tests and make sure your changes don’t break anything else? What about shipping that code as part of the DevOps workflow? Or an out-of-band patch? Do you need to wait until the next maintenance window to implement the changes? These are all extremely valuable questions to ask to figure out the impact of changes on your environment.

Now multiply all of those factors by the number of things you tried that didn’t work. The number of times you thought you had the issue solved only to get the same error output. You can see how wasting hours of a programming team working on things can add up to the company’s bottom line quickly. Nothing is truly free or a sunk cost. You’re still paying people to work on something, whether it’s fixing buggy code or writing a new feature for your application. If they’re spending more of their time tracking down bugs that happen infrequently with no clear root cause are they being used to their highest potential?

What happens when that team can’t fix the issue because it’s too hard to cause it again? Do they bring in another team? Pay for support calls? Do you throw up your hands and just live with it? I’ve been involved in all of these situations in the past. I’ve tried to replicate issues to no avail only to be told that they’ll just live with the bug because they can’t afford to have me work on it any longer. I’ve also spent hours trying to replicate issues only to find out later that the message has only ever appeared once and no one knows what was going on when it happened. I might have been working for a VAR at the time but my time was something the company tracked closely. Knowing up front the issue had only ever happened once might have changed the way I worked on the issue.

I can totally understand why people would be upset that a help desk closed a ticket because they couldn’t replicate an issue. However, I also think that prudence should be involved in any structured troubleshooting practice. Having been on the other side of the Severity One all-hands-on-deck emergencies only to find a simple issue or a non-repeatable problem in the past I can say that having that many resources committed to a non-issue is also maddening as a tech and a manager of technical people.

Tom’s Take

Do you need to replicate a problem to troubleshoot it? No. But you also don’t need to follow a recipe to bake a cake. However, it does help immensely if you do because you’re going to get a consistent output that helps you figure out issues if they arise. The purpose of replicating issues when troubleshooting isn’t nefarious or an attempt to close tickets faster. Instead it’s designed to help speed problem isolation and ensure that resources are tasked with the right fixes to the issues instead of just spraying things and hoping one of them gets the thing taken care of.