Time Is Not On Your Side

Posted on by
Reply

It’s been almost five years since I wrote about the challenges of project management and timing your work as an engineer. While most of that information is still very true even today I’ve recently had my own challenges with my son’s Eagle Scout project. He is of a mind that you can throw together a plan and just do a whole week of work in just a couple of days. I, having worked in the IT industry for years, have assured him that it absolutely doesn’t work like that. Why is there a disconnect between us? And how does that disconnect look to the rest of the world?

Time Taking You

The first problem that I often see when working with people that aren’t familiar with projects is that they vastly underestimate the amount of time it takes to get something done. You may recall from my last post that my project managers at my old VAR job had built in something they called Tom Time to every quote. That provided a way for my estimate to reflect reality once I arrived on site and found the things didn’t go according to plan.

Part of the reason why my estimates didn’t reflect reality was because there are a lot of things that go into a project that can’t quite be explained or calculated into the final estimate. For example, how long does it take for a switch to reboot? Some of them can be ready to pass traffic in a couple of minutes. Larger devices that need to test modules may take up to ten minutes to be ready to go. If you have to reboot that switch multiple times during your project how do you account for that time? Is there a line item for a hour’s worth of switch reboots? What about the project closeout meetings a paperwork? How do you build that into a project timeline?

People that underestimate the timeline of a project are almost always only focused on the work. They see that it should take them about five minutes to copy the config the switch and ten minutes to put it in the rack. Did they think about the time to unbox it? Cable it? Do a final test to ensure all configuration is correct and saved to the startup config? Each of these things sound trivial but they add time. Maybe you don’t do the final config test and hope for the best. But you can’t shave time on unboxing unless you have someone helping you do that. Which, of course, just adds time to the project in a different way.

The Price of Time

Does this mean that you just need to increase the amount of time that you put on a project? No, it doesn’t. One of the connectivity providers I worked with in the past had what they called a “foolproof method” of getting the right time estimate for a circuit. They doubled the number and increased to the next time unit. So two hours became four days. Three days became six weeks. And I became infuriated when I realized how much time something like this would take.

Part of the reasoning behind that thinking was that the project management overhead always took longer than expected. But the other thinking was that quoting much longer timelines gave them more room to cram in too much work for a single team. They could juggle deployments because they had enough hours in the quote that they could be more interrupt driven. Work on something until someone complains then move to that project and work on it until the complaining stops. You can see why providers like that quickly get a reputation for padding their projects.

Time costs money. Either someone is paying you to do the job or you’re paying for that resource to be unavailable for doing the job. You have to learn how to allocate your resources effectively. If you need to help your teams or your contractors understand the additional time that it takes to do a project you need to either package that time as a line item or educate them about what additional tasks you see. Accounting for that extra time is a better way to show value than just adding lots of extra wiggle room to a project so you don’t go over budget. The education aspect is especially important for talent that isn’t familiar with things from the outset. Teaching them how to look for those time sinks and making sure they’re tracked means their estimates will be much more accurate in the future.

Tom’s Take

My son is going to complete his project but he’s going to learn a lot about the way the world works in the process. Paint doesn’t dry overnight. It takes time to load and unload lumber. People need more than 24 hours notice to show up to work on something. These are all lessons I’ve learned over the years that I’m happy to teach. Time is important to us all because we don’t get any more of it. Every minute that goes by is a minute we can’t get back. Make the most of your time by tracking it appropriately and building those hidden things into your project estimates. That’s how you get time to be on your side for once.

Aruba Isn’t A Wireless Company (Any More)

Posted on by
2

Remember when Aruba was a wireless company? I know it sounds like something that happened 40 years ago but the idea that Aruba only really made wireless access points and some campus switches to support them isn’t as old as you think. The company, now known as HPE Aruba Networking (née Aruba, a Hewlett Packard Enterprise Company), makes more than just Wi-Fi gear. Yet the perception of the industry is that they’re still a wireless company looking to compete with the largest parts of the market.

Branching Out of Office

This year’s Aruba Atmopshere showed me that Aruba is trying to do more than just campus wireless. The industry has shifted away from just providing edge connectivity and is now focused on a holistic lineup of products that are user-focused. You don’t need to go much further than the technical keynote on the second day of the conference to see that. Or the Networking Field Day Experience videos linked above.

Do you know what Aruba wanted to showcase?

  • Campus Switches
  • Data Center Switches
  • Private 5G/LTE
  • SASE/SSE
  • IoT
  • Cloud-Enabled Management

You know what wasn’t on that list? Access points. For a “wireless” company that’s a pretty glaring omission, right? I think it’s actually a brilliant way to help people understand that HPE Aruba Networking is a growing part of the wider HPE business dedicated to connectivity.

It’s been discussed over the years that the HPE acquisition of Aruba was a “reverse acquisition”. That basically means that HPE gave Aruba control over their campus (and later data center) networking portfolio and let them run with it. It was successful and really helped highlight the needs that HPE had in that space. No one was talking about the dominance of Procurve switches. HPE was even reselling Arista gear at the time for the high end customers. Aruba not only was able to right the ship but help it grow over time and adopt home-grown offerings.

When you think of companies like Juniper and Cisco, do you see them as single product vendors? Juniper makes more than just service provider routers. Cisco makes more than just switches. They have distinct lines of business that provide offerings across the spectrum. They both sell firewalls and access points. They both have software divisions. Cisco sells servers and unified communications gear on top of everything else they do. There’s more to both of them than meets the eye.

Aruba needed to shed the wireless moniker in order to grow into a more competitive market segment. When you’re known as a single product vendor you tend to be left out of conversations. Would you call Palo Alto for switches or wireless? No, because they’re a firewall or SASE company. Yes, they make more than those products but they have a niche, as opposed to more diverse companies. I’m not saying Palo Alto isn’t diverse, just that they define their market segment pretty effectively. So much so that people don’t even call application firewalls by that name any longer. They’re “Palo Altos”, giving the company the same generic trademark distinction as Kleenex and Velcro.

User Face-to-Face

Aruba needs to develop the product lines that help get users connected. Wireless is an easy layup for them now so where do they expand? Switches are a logical extension so the CX lines were developed and continue to do well. The expansion into private LTE and security also help significantly, which are bolstered by their recent acquisitions.

Security is an easy one to figure out. Aruba has gone from SD-Branch, focused on people working in remote offices, to add on true SD-WAN functionality with the Silver Peak purchase, to now offering SSE with Axis Security being folded in to the mix. SSE is a growing market segment because the services offered are what users consume. SASE works great if you’re working from home all the time. In the middle of the pandemic that was a given. People had home offices and did their work there.

But now that restrictions are relaxed and people aren’t going into the office all the time. This hybrid work model means no hardware to do the inspection. Since SSE is not focused on hardware it’s a great fit for a mobile hybrid workforce. If you remember how much Aruba was touting the BYOD wireless-only office trend back in 2016 and 2017 you can see how SSE would have been a wonderful fit back then if it had existed. Given how the concept of a wireless-only BYOD office was realized through not having an office I’d say SSE is a perfect fit for the modern state of the enterprise.

Private 5G is a bit more complicated. Why would Aruba embrace a technology that effectively competes with its core business? I’d say that’s because they need to understand the impact that private cellular will have on their business. People aren’t dumping Wi-Fi and moving en masse to CBRS. We’ve reached a point where we’re considering what the requirements for private LTE deployments need to look like and where the real value lies for them. If you have a challenging RF environment and have devices capable of taking SIM cards it makes a lot of sense. Aruba having a native way of providing that kind of connectivity for users that are looking to offer it is also a huge win. It’s also important to note that Aruba wants to make sure it has complete control over the process, so what better way than acquiring a mature company that can integrate into their product lines?

Tom’s Take

I can’t take full credit for this idea. Avril Salter pointed it out during a briefing and I thought it was a wonderful point. Aruba isn’t a wireless company now because they’ve grown to become a true networking company. They offer more than just APs and devices that power them. There have a full line of products that address the needs of a modern user. The name change isn’t just a branding exercise. It represents a shift in the way people need to see the company. Growing beyond what you used to be isn’t a bad thing. It’s a sign of maturity.

The Shifting Lens of Mentoring

Posted on by
Reply

The other day I realized that I had become the “old man” at Tech Field Day. Not so much that I’m ready for AARP but more that I’ve been there longer than anyone else but Stephen. The realization was a long time coming but the thing that pushed me to understand it was when someone asked a question about a policy we had and I not only knew the reason why we did it but also a time before we had it.

As I spent time thinking about the way that I’ve graduated from being the new guy to the old mentor I thought about the inflection point when the changeover happened.

Green and Growing

The first part of the demarcation between mentor and mentee in my eyes is where the knowledge lies. When you’re first starting out you’re the one that needs to understand things. You ask lots and lots of questions and try to understand how things are done and why you do them that way. Focusing on that knowledge acquisition is part of the marker of someone in need of mentorship.

For those trying to mentor these eager employees don’t make the mistake of getting frustrated at the constant questioning. As someone that constantly has to understand the what and the why behind things I have been known to overwhelm those that would prefer to just tell me how things are done and move on with it.

When I see that level of curiosity in others I realize that they’re not trying to change things for the sake of change. Unlike others who might just want to make changes as a method of controlling the processes, eager learners are usually asking questions about the process because they need to understand the reasoning behind it. Often they have a unique perspective they can impart to the problem or some other knowledge they can use to streamline things. Even if they don’t you can help them understand why the process or policy is done in a specific way.

Guidance for the Eager

Coming back to that moment of realization from earlier means knowing the answers to the questions being asked are ones you have. Some people are designated as mentors based on their desire to share knowledge with others. In smaller organizations that may not be possible. You may find yourself mentoring others simply because you know what they need to learn and there’s no one else to teach them.

When you realize that you’re the one that knows the answer to the question you should step forward into a mentoring role. That’s what it feels like to be the “old timer” at the office. You’ve been around when the policies were made or perhaps you were the mentee asking all the questions right after that. Either way you have knowledge that needs to be shared with others.

That is the real inflection point. The knowledge transfer. Note that this has nothing to do with seniority or age or even organizational structure. This has everything to do with skills and information. You could be mentoring a younger new employee in the process for contracts today. And that same employee could be offering you guidance and help in a new email program or social media platform tomorrow. The mentoring relationship doesn’t always have to be one-way.

The dynamic nature of the mentoring relationship is one area I feel like we could always strive to do better at. We often see the older, more tenured employees as the default mentors. While that is true it undervalues the knowledge that new employees can have. Maybe this person is just starting out in the accounting department. However, if they were an accountant for the last three years do you think that means they don’t have the skills? Or perhaps it’s just that they need to understand the specifics of their role here. I’d wager that if you asked them for ways you could improve the accounting process they’d have some suggestions for you.

Tom’s Take

I didn’t necessarily see myself as a mentor until it was staring me right in the face. Yes, I had agreed to train people in certain aspects of their roles but the idea that I was doing it more as a form of knowledge transfer hadn’t really occurred to me until I found myself answering questions because I was the only one that had those answers. As you look for ways to cultivate and grow mentoring relationships don’t forget to share what you’ve learned but also seek out things that you want to understand. That knowledge will serve you well and also give you an opportunity to give it back down the road to a new group of people in need of mentorship.

Mastodon Needs More Brand Support

Posted on by
Reply

As much as I want to move over to Mastodon full time, there’s one thing I feel that is massively holding it back. Yes, you can laud the big things about federations and freedom as much as you want. However, one thing I’ve seen hanging out in the fringes of the Fediverse that will ultimately hold Mastodon back is the hostility toward brands.

Welcoming The Crowd

If you’re already up in arms because of that opening, ask yourself why. What is it about a brand that has you upset? Don’t they have the same right to share on the platform as the rest of us? I will admit that not every person on Mastodon has this outward hostility toward companies. However I can also sense this feeling that brands don’t belong.

It reminds me a lot of the thinly veiled distaste for companies that some Linux proponents have. The “get your dirty binary drivers out of my pristine kernel” crowd. The ones that want the brands to bend to their will and only do things the way they want. If you can’t provide us the drivers and software for free with full code support for us to hack as much as we want then we don’t want you around.

Apply that kind of mentality to brands venturing into the Fediverse. Do you want them to share their message? Share links to content or help people join webinars to learn more about the solutions? Or do you only want the interns and social media professionals to be their authentic selves and pretend they aren’t working for a bigger company?

The fact is that in order to get people to come to Mastodon to consume content you’re going to need more than highly motivated people. You’re going to need people that are focused on sharing a message. You’re really going to want those that are focused on outreach instead of just sharing random things. Does that sound a lot like the early days of Twitter to you? Not much broadcast but lots of meaningless status updates.

That’s the biggest part of what’s holding Mastodon back. There’s no content. Yes, there’s a lot of sharing. There’s lots of blog posts or people clipping articles to put them out there for people to read. But it’s scattered and somewhat unsupported. There’s no driving force to get people to click through to sites with deeper information or other things that brands do to support campaigns.

Tom’s Take

You’re going to disagree with me and I don’t blame you one bit. You may not like my idea about getting more brand support on Mastodon but you can’t deny that the platform needs users with experience to grow things. And if you keep up the hostility you’re going to find people choosing to stay on platforms that support them instead of wading into the pool where they feel unwelcome.

Consuming Content the Way You Want

Posted on by
1

One of the true hidden gems of being a part of a big community is the ability to discuss ideas and see different perspectives. It’s one of the reasons why I enjoy working at Tech Field Day and why I’m lamenting the death spiral of Twitter. My move to Mastodon is picking up steam and I’m slowly replicating the way that I consume content and interact there but it’s very much the same way I felt about Twitter thirteen years ago. There’s promise but it needs work.

As I thought about my journey with social media and discussed it with people in the community I realized that a large part of what has me so frustrated is the way in which my experience has been co-opted into a kind of performative mess. Social media is becoming less about idea exchange and more about broadcast.

Give and Take

When I first started out on Twitter I could post things that were interesting to me. I could craft the way I posted those short updates. Did I want to be factual and dry? Or should I be more humorous and snarky? I crafted my own voice as I shared with others. My community grew organically. People that were interested in what I had to say joined up. Others chose to stick with their own circles. The key is that I was allowed to develop what I wanted to present to those around me.

As time went on I realized that I was an aberration in the grand scheme of Twitter. I made content. I offered opinions and analysis. I was a power user. Twitter wasn’t filled with power users. It was filled with passive consumers of content. Twitter wasn’t overly concerned with enabling features that allowed users like me to have an easier time. Instead, it was focused on delivering content to the passive audience. Content that Twitter determined was either interesting enough to keep users coming back to the service or generated enough revenue to keep the lights on.

That shift happens in pretty much every social platform that I’ve been a part of. Facebook moved from reading through other people’s status updates about their dogs or their lunch and into a parade of short form videos about craft projects or memes about Star Wars. Every interaction with those posts just enhanced the algorithm to show me more of them. Facebook only shoves more of what you see into your face. It doesn’t take what you create and build from there.

The algorithms that run these services now don’t care about you. They don’t facilitate the discussions and information exchange that make us all better. Instead, they feed us mindless interaction. They give us 60-word posts about a topic with vapid insights or any one of a number of endless popcorn videos about “life hacks” or people having accidents or, worse yet, very clever advertising that looks like a random person posting about how amazing a t-shirt is.

Does It Ad Up?

If you’re thinking to yourself this is starting to sound a lot like television advertising, you’re not far off the mark. The explosion of content that has been pushed in front of us is all about the advertising. It’s either brands that are looking to have users buy their product or service or it’s services looking to gain tons of users for other reasons.The advertising dollar rules all now.

This isn’t a new thing. Anyone that tries to tell you that invasive advertising is a modern construct has never opened up a copy of Computer Shopper magazine from the 90s or enjoyed hearing the host of a 60s game show shilling for Lucky Strike cigarettes. Advertising has always been a huge part of the content that we consume.

Modern YouTube videos have pre-roll ads and breaks in the middle for more ads. Podcasts have one or two ad reads, either by the hosts or through a slick, produced read. For a society that hates advertising we sure don’t mind taking money from them when they want to place an ad in the content we’re creating. Yet unless we’re willing to bankroll our own platforms completely we’re stuck with the way that those platforms make money.

This all comes together in an insidious way. The algorithms show us things out of order because they want to grab our attention. The system wants to weave in content we might enjoy along with ads that pay for the platform alongside of the content we actually want to see. Unlike broadcast television, which has specific rules about advertising, these systems can flood us with content that is designed to make us stick around or pay for something that someone wanted us to buy.

At no point in that whole process did we see highlighting of blog posts (unless they were boosted with ads) or bringing conversations to the top of the feed because we’ve interacted with those people. Power users and non-sponsored content creators are a drag on the system. Because they’re not interesting enough to draw in the regular users, unless they’re famous, and they don’t pay the platform to prioritize that content.

As the social network matures and relies less and less on users to create the interactions that sustains the user base it flips the model to be more focused on providing for the brands that pay to keep the lights on and the popcorn-style content that keeps the users hanging around. That’s the ultimate reason why the twilight of social media platforms feels so wasteful. What was once a place to grow and expand your horizons becomes the same mindless drivel that we see on TV. A late-stage social network is practically indistinguishable from what The History Channel has turned into.

Tom’s Take

I want Mastodon to succeed. I want the idea exchange to return. There are many on the platform right now that are hostile to brands because they worry about the inevitable slide into the advertising model. That doesn’t happen because of the brands themselves. The move happens when users grow and the platform needs to keep them around. When the costs of running the infrastructure grow past the ability of the users to support it. Here’s hoping the idea exchange and learning continue to be the primary focus for the time being. At least until the next new things comes along.

Perfection Paralysis

Posted on by
Reply

This is a sort of companion piece to my post last week because I saw a very short post here about doing less. It really hit home with me because I’m just as bad as Shawn about wanting everything to be perfect when I write it or create it.

Maximizing Mistakes

One of the things that I’ve noticed in a lot of content that I’ve been consuming recently is the inclusion of mistakes. When you’re writing you have ample access to a backspace key so typos shouldn’t exist (and autocorrect can bugger off). But in video and audio content you can often make a mistake and not even realize it. Flubbing a word or needed to do a retake for something happens quite often, even if you never see or hear them.

What has me curious and a bit interested is that more of those quick errors are making it in. These are things that could easily be fixed in post production and yet they stay. It’s almost like the creators are admitting that mistakes happen and it’s hard to read scripts perfectly every time like some kind of robot. Honest mistakes over things like pronunciation or difficult word combinations help remind us that not everything needs to be exactly perfect every time.

That’s not to say that you can get away with not doing things with the appropriate amount of practice. The difference between a simple mistake in a long passage of text and a haphazard idea just thrown out there without care is very apparent. As I tell the people that I work with for public speaking, the more something sounds off the cuff the more practice went into it to make it sound natural.

Accumulating Assets

My friend Ivan Pepelnjak reached out to me after my last post and reminded me of something he wrote a decade ago that talks about his view of the creative process. One of the big takeaways for me was the section on ideas. It’s important to realize that nothing will spring forward from your mind completely realized.

It’s a lot like baking. The ingredients are easy enough to measure. The trick is mixing them together. You have to add the right ideas in the right amounts and then let them mix together and even settle a little bit before you can make something out of them. However, you also have to be careful about how you go about doing it. Mixing merengue is a very different skill than a pound cake. Some things shouldn’t be mixed too much lest they become ruined by the extra attention. It is entirely possible to do too much to ideas without realizing it.

Tom’s Take

If you find yourself struggling with creativity or need to figure out a way to make something happen don’t be afraid to mix things up a little. Go for a walk. Play some music to force your brain into a new space. Look over your collection of half-formed ideas and see what pops up. Make something happen to change the status quo. You’d be surprised what might happen. But above all don’t get stuck on the idea that it needs to be perfect. The best ideas are very often imperfect.

Content Creation Complications

Posted on by
1

If you’ve noticed my regular blog posts have been a bit irregular as of late you’re not alone. I’m honestly working through a bit of writer’s block as of late. The irony is that I’m not running out of things to talk about. I’m actually running out of time to talk about them the way that I want.

Putting in the Work

By now you, my dear readers, know that I’m not going to put out a post of 200-300 words just to put something out during the week. I’d rather spend some time looking into a topic and creating something that informs or encourages discussion. That means having sources or doing research.

Research takes time. Ironically enough I’ve always had a much easier time writing things so long as I have the info to pull from in my head. One of the side effects of neurodivergence that I’ve learned about recently is that neurodivergent people tend to write their ‘first draft’ in their head throughout the creation process. Rather than writing and rewriting over and over again I pool all the information in my brain and work through it all to put down my final thoughts. That means what comes out is what I want to say.

However, the time it takes to make that content soup isn’t immediate. Sometimes I find myself doing a massive amount of research to learn about something that ultimately becomes two or three sentences. The rest of the information gets discarded or filed away for use later on in something that might be totally unrelated.

Lightning Bolts

Now you probably see the difficulty in the content creation process when it happens like that. When I’m motivated to write something the words are flowing as I create and edit on the fly. I have lots to say about things and I often change course in mid-stream to pivot into an entirely different idea.

However, when I’m not feeling it the content is a bit harder to create. I have starter ideas that need to germinate but just like growing a plant it takes time. Sometimes that happens when I listen to a podcast or get a spark. Other times I’m walking in circles in my backyard hoping for a bolt from the blue to hit me with inspiration. When that doesn’t happen I just find myself struggling to come up with anything that can develop into a few hundred words.

I’ve been told by many friends that this is how the creative process feels for them all the time. They have ideas but no way to gain the inspiration to write them down. I would hope that there are ways to create and inspire that kind of creative process frequently. I can honestly say that it sucks when you can’t create because the information is stuck in there and it wants to come out. I just can’t make it do the work!

Tom’s Take

The behind-the-scenes part of content creation isn’t easy. It’s also much less glamorous than you might imagine. Remember that when you wonder how you’re going to create something. Don’t worry about making it perfect but make sure to get it all down. Keep yourself to your schedules and make something happen. Otherwise you’re going to be wandering in circles until you do.

Assume Disaster

Posted on by
Reply

One of the things that people have mentioned to me in the past regarding my event management skills is my reaction time. They say, “You are always on top of things when they go wrong. How do you do it?”

My response never fails to make them laugh. I offer, “I always assume something is going to go wrong. I may not know what it is but when it does happen I’m ready to fix it.”

That may sound like a cynical take on planning and operations but it’s served me well for many years. Why is it that things we spend so much time working on always seem to go off the rails?

Complexity Fails

Whether it’s an event or a network or even a carpentry project you have to assume that something is going to go wrong. Why? Because the more complex the project the more likely you are to hit a snag. Systems that build on themselves and require input to proceed are notorious for hitting blocks that cause the whole thing to snarl into a mess of missed timelines.

When I was in college studying project management I learned there’s even a term for time saving: crashing a project. Not literally crashing the project into something but instead looking for ways to trim the timeline and work through issues. Why is this a common term? I’d hazard a guess that very few projects actually stick to their timeline. It could be a parts delay. It could be a team taking longer to work through an issue. Mercury could be in retrograde during sunspots. Whatever the case may be, projects are designed to have floating timelines.

This imprecision built into project planning made me realize that the only way to be really sure that something would get done properly was to anticipate the errors and work through them. Part of the way to prevent these issues is to reduce complexity. You may not be able to work through every potential scenario where something is going to go sideways but you can almost always tell where the problems will arise. Any module of work that has lots of moving parts or lots of people with specific deadlines is going to be a trouble spot. The more components that depend on each other means a greater chance that any one of them slipping will cause a delay that requires attention.

If you have a project or are planning something that has complicated steps for a specific goal, try to break those down into more simple things that don’t depend on each other. Have a team that needs to write a report based on the research from another team? Don’t bundle those together. Have the writing team working on things that aren’t dependent upon the research team just in case the data isn’t delivered. If you’re building a house and you are planning on having things done that require a roof being installed you should have a plan for what happens if the roofers are behind or the shingles don’t arrive on schedule. Finding these extra bits of complexity and eliminating them will go a long way toward solving recurring sources of frustration.

Be Prepared for Problems

The motto of the Boy Scouts is “be prepared”. It’s something I constantly remind the youth in the program weekly. Be prepared for what exactly? It doesn’t matter what if you’re properly prepared. You don’t have to be prepared for every possible scenario but you need to have the flexibility to address a wide variety of potential problems.

Take information security, as a prime example. How will your enterprise be breached? There’s almost too many ways to consider. New zero day? Backdoor password installed years ago? Phishing your key employees? Good old fashioned malfeasance? The list of things are endless! But the results are always the same. Attackers look for things of value and either steal them or disable them. Thieves steal and chaotic souls cause chaos. The entry is unknown but the results of entry can be quantified and considered.

You may not know how they’ll get in but you know how to stop them once they do. That’s why you should always assume you’re under attack or already breached. If you construct the system in such a way as to prevent lateral movement or even create policies to keep data safe at rest you’ll go a long way to preventing unauthorized users from accessing it, malicious or otherwise.

Is assuming that you’re always under attack kind of paranoid? Yes, it is. However, if you assume you’ve been breached and you are wrong all you’ve done is ensure that your data is safe and secure. If you assume you’re not and you end up being wrong you get to spend a lot of time cleaning up and sending emails to your boss and your resume to the next place where you get to make all new assumptions.

Tom’s Take

The optimist in me wants to believe that you can plan something so well that there isn’t a chance a problem can happen. The realist in me knows the optimist is crazy. That doesn’t mean I should just stop planning and hope for the best when I need to tap dance my way out of a problem. Instead, it means that I need to consider all the possibilities and try to have an answer for them, event if they’re remote. That way I’m never caught off guard by the wackiest of issues.

The Dangers of Knowing Everything

Posted on by
Reply

By now I’m sure you’ve heard that the Internet is obsessed with ChatGPT. I’ve been watching from the sidelines as people find more and more uses for our current favorite large language model (LLM) toy. Why a toy and not a full-blown solution to all our ills? Because ChatGPT has one glaring flaw that I can see right now that belies its immaturity. ChatGPT knows everything. Or at least it thinks it does.

Unknown Unknowns

If I asked you the answer to a basic trivia question you could probably recall it quickly. Like “who was the first president of the United States?” These are answers we have memorized over the years to things we are expected to know. History, math, and even written communication has questions and answers like this. Even in an age of access to search engines we’re still expected to know basic things and have near-instant recall.

What if I asked you a trivia question you didn’t know the answer to? Like “what is the name of the metal cap at the end of a pencil?” You’d likely go look it up on a search engine or on some form of encyclopedia. You don’t know the answer so you’re going to find it out. That’s still a form of recall. Once you learn that it’s called a ferrule you’ll file it away in the same place as George Washington, 2+2, and the aglet as “things I just know”.

Now, what if I asked you a question that required you to think a little more than just recalling info? Such as “Who would have been the first president if George Washington refused the office?” Now we’re getting into more murky territory. Instead of being able to instantly recall information you’re going to have analyze what you know about the situation. For most people that aren’t history buffs they might recall who Washington’s vice president was and answer with that. History buffs might take more specialized knowledge about matters would apply additional facts and infer a different answer, such as Jefferson or even Samuel Adams. They’re adding more information to the puzzle to come up with a better answer.

Now, for completeness sake, what if I asked you “Who would have become the Grand Vizier of the Galactic Republic if Washington hadn’t been assassinated by the separatists?” You’d probably look at me like I was crazy and say you couldn’t answer a question like that because I made up most of that information or I’m trying to confuse you. You may not know exactly what I’m talking about but you know, based on your knowledge of elementary school history, that there is no Galactic Republic and George Washington was definitely not assassinated. Hold on to this because we’ll come back to it later.

Spinning AI Yarns

How does this all apply to a LLM? The first thing to realize is that LLMs are not replacements for search engines. I’ve heard of many people asking ChatGPT basic trivia and recall type questions. That’s not what LLMs are best at. We have a multitude of ways to learn trivia and none of them need the power of a cloud-scale computing cluster interpreting inputs. Even asking that trivia question to a smart assistant from Apple or Amazon is a better way to learn.

So what does an LLM excel at doing? Nvidia will tell you that it is “a deep learning algorithm that can recognize, summarize, translate, predict and generate text and other content based on knowledge gained from massive datasets”. In essence it can take a huge amount of input, recognize certain aspects of it, and produce content based on the requirements. That’s why ChatGPT can “write” things in the style of something else. It knows what that style is supposed to look and sound like and can produce an output based on that. It analyzes the database and comes up with the results using predictive analysis to create grammatically correct output. Think of it like Advanced Predictive Autocorrect.

If you think I’m oversimplifying what LLMs like ChatGPT can bring to the table then I challenge you to ask it a question that doesn’t have an answer. If you really want to see it work some magic ask it something oddly specific about something that doesn’t exist, especially if that process involves steps or can be broken down into parts. I’d bet you get an answer at least as many times as you get something back that is an error message.

To me, the problem with ChatGPT is that the model is designed to produce an answer unless it has specifically been programmed not to do so. There are a variety of answers that the developers have overridden in the algorithm, usually something racially or politically sensitive. Otherwise ChatGPT is happy to spit out lots of stuff that looks and sounds correct. Case in point? This gem of a post from Joy Larkin of ZeroTier:

https://mastodon.social/@joy/109859024438664366

Short version: ChatGPT gave a user instructions for a product that didn’t exist and the customer was very frustrated when they couldn’t find the software to download on the ZeroTier site. The LLM just made up a convincing answer to a question that involved creating something that doesn’t exist. Just to satisfy the prompt.

Does that sound like a creative writing exercise to you? “Imagine what a bird would look like with elephant feet.” Or “picture a world where people only communicated with dance.” You’ve probably gone through these exercises before in school. You stretch your imagination to take specific inputs and produce outputs based on your knowledge. It’s like the above mention of applied history. You take inputs and produce a logical outcome based on facts and reality.

ChatGPT is immature enough to not realize that some things shouldn’t be answered. If you use a search engine to find the steps to configure a feature on a product the search algorithm will return a page that has the steps listed. Are the correct? Maybe. Depends on how popular the result is. But the results will include a real product. If you search for nonexistent functionality or a software package that doesn’t exist your search won’t have many results.

ChatGPT doesn’t have a search algorithm to rely on. It’s based on language. It’s designed to approximate writing when given a prompt. That means, aside from things it’s been programmed not to answer, it’s going to give you an answer. Is it correct? You won’t know. You’d have to take the output and send it to a search engine to determine if that even exists.

The danger here is that LLMs aren’t smart enough to realize they are creating fabricated answers. If someone asked me how to do something that I didn’t know I would preface my answer with “I’m not quite sure but this is how I think you would do it…” I’ve created a frame of reference that I’m not familiar with the specific scenario and that I’m drawing from inferred knowledge to complete the task. Or I could just answer “I don’t know” and be done with it. ChatGPT doesn’t understand “I don’t know” and will respond with answers that look right according to the model but may not be correct.

Tom’s Take

What’s funny is that ChatGPT has managed to create an approximation of another human behavior. For anyone that has ever worked in sales you know one of the maxims is “never tell the customer ‘no'”. In a way, ChatGPT is like a salesperson. No matter what you ask it the answer is always yes, even if it has to make something up to answer the question. Sci-fi fans know that in fiction we’ve built guardrails for robots to save our society from being harmed by functions. AI, no matter how advanced, needs protections from approximating bad behaviors. It’s time for ChatGPT and future LLMs to learn that they don’t know everything.

Friction as a Network Security Concept

Posted on by
Reply

I had the recent opportunity to record a podcast with Curtis Preston about security, data protection, and networking. I loved being a guest and we talked about quite a bit in the episode about how networking operates and how to address ransomware issues when they arise. I wanted to talk a bit more about some concepts here to help flesh out my advice as we talked about it.

Compromise is Inevitable

If there’s one thing I could say that would make everything make sense it’s this: you will be compromised. It’s not a question of if. You will have your data stolen or encrypted at some point. The question is really more about how much gets taken or how effectively attackers are able to penetrate your defenses before they get caught.

Defenses are designed to keep people out. But they also need to be designed to contain damage. Think about a ship on the ocean. Those giant bulkheads aren’t just there for looks. They’re designed to act as compartments to seal off areas in case of catastrophic damage. The ship doesn’t assume that it’s never going to have a leak. Instead, the designers created it in such a way as to be sure that when it does you can contain the damage and keep the ship floating. Without those containment systems even the smallest problem can bring the whole ship down.

Likewise, you need to design your network to be able to contain areas that could be impacted. One giant flat network is a disaster waiting to happen. A network with a DMZ for public servers is a step in the right direction. However, you need to take it further than that. You need to isolate critical hosts. You need to put devices on separate networks if they have no need to directly talk to each other. You need to ensure management interfaces are in a separate, air-gapped network that has strict access controls. It may sound like a lot of work but the reality is that failure to provide isolation will lead to disaster. Just like a leak on the ocean.

The key here is that the controls you put in place create friction with your attackers. That’s the entire purpose of defense in depth. The harder it is for attackers to get through your defenses the more likely they are to give up earlier or trigger alarms designed to warn you when it happens. This kind of friction is what you want to see. However, it’s not the only kind of friction you face.

Failing Through Friction

Your enemy in this process isn’t nefarious actors. It’s not technology. Instead, it’s the bad kind of friction. Security is designed by its very nature to create friction with systems. Networks are designed to transmit data. Security controls are designed to prevent the transmission of data. This bad friction comes when these two aspects are interacting with each other. Did you open the right ports? Are the access control lists denying a protocol that should be working? Did you allow the right VLANs on the trunk port?

Friction between controls is maddening but it’s a solvable problem with time. The real source of costly friction comes when you add people into the mix. Systems don’t complain about access times. They don’t call you about error messages. And, worst of all, they don’t have the authority to make you compromise your security controls for the sake of ease-of-use.

Everyone in IT has been asked at some point to remove a control or piece of software for the sake of users. In organizations where the controls are strict or regulatory issues are at stake the requests are usually disregarded. However, when the executives are particularly insistent or the IT environment is more carefree you can find yourself putting in a shortcut to get the CEO’s laptop connected faster or allow their fancy new phone to connect without a captive portal. The results are often happy and have no impact. That is, until someone finds out they can get in through your compromised control and create a lot of additional friction.

How can you reduce friction? One way is to create more friction in the planning stages. Ask lots of questions about ports and protocols and access list requirements before something is implemented. Do your homework ahead of time instead of trying to figure it out on the fly. If you know that a software package needs to communicate to these four addresses on these eight ports then anything outside of that list should be suspect and be examined. Likewise, if someone can’t tell you what ports need to be opened for a package to work you should push back until they can give you that info. Better to spend time up front learning than spend more time later triaging.

The other way to reduced friction in implementation is to shift the friction to policy. If the executives want you to compromise a control for the sake of their own use make them document it. Have them write it down that you have been directed to add a special configuration just for them. Keep that information stored in your DR plan and note it in your configuration repositories as well. Even a comment in the access list can help understand why you had to do something a certain way. Often the request to document the special changes will have the executives questioning the choice. More importantly, if something does go sideways you have evidence of why the change was made. And for executives that don’t like to look like fools this is a great way to have these kinds of one-off policy changes stopped quickly when something goes wrong and they get to answer questions from a reporter.

Tom’s Take

Friction is the real secret of security. When properly applied it prevents problems. When it’s present in too many forms it causes frustration and eventually leads to abandonment of controls or short circuits to get around them. The key isn’t to eliminate it entirely. Instead you need to apply it properly and make sure to educate about why it exists in the first place. Some friction is important, such as verifying IDs before entering a secure facility. The more that people know about the reasons behind your implementation the less likely they are to circumvent it. That’s how you keep the bad actors out and the users happy.