What Is Closed-Loop Automation?

During Networking Field Day 22 last week, a lot the questions that were directed at the presenters had to do with their automation systems. One term kept coming up that I was embarrassed to admit that I’d never heard of. Closed-loop automation is the end goal for these systems. But what is closed-loop automation? And why is it so important. I decided to do a little research and find out.

Open Up

To understand closed-loop systems, you have to understand open-loop systems first. Thankfully, those are really simple. Open-loop systems are those where the output isn’t directly affected by the control actions of the system. It’s a system where you’re going to get the output no matter how you control it. The easiest example is a clothes dryer. There are a multitude of settings that you can choose for a clothes dryer, including the timing of the cycle. But no matter what, the dryer will stop at the end of the cycle. There’s no sensor in a basic clothes dryer that senses the moisture level of the clothes and acts accordingly.

Open-loop systems are stable and consistent. Every time you turn on the dryer, it will run until it finishes. There’s no variable in the system that will change that. Aside from system failure, it’s going to run exactly 30 minutes every time it’s set to that cycle. It’s also not going to run unless you set the cycle. As my family will tell you, putting clothes in the dryer and not setting it will not result in magic happening.

Close It Off

In contrast, closed-loop systems have outputs that are dependent upon the control function of the system. If the control function requires something change in the system to achieve the desired output then it will change that thing to get there.

The most classic example of a closed-loop system is the HVAC system in your house. The control function is the thermostat. If you want the temperature in your house to be 70 degrees Fahrenheit (21 degrees C), you set the thermostat and let the system take care of things. If the temperature falls below the required setting, the heating unit will turn on and bring the temperature up to the required level before shutting off. In the summertime, rising above the temperature setting will cause the air conditioning compressor to kick on and cool things down.

Closed loop systems are great because you set them and forget them. Unlike the dryer example above I can set my thermostat and it will run even if I forget to go turn on the heater/AC. But they’re also more complicated to troubleshoot and figure out. As someone with very little practical knowledge of the operation of HVAC it’s rough to figure out if it’s the thermostat or the unit or some other relay somewhere that’s causing your house to be too warm or too cold.

Closed loop systems can also take more inputs given the right control settings. Using the same A/C example, I upgraded my thermostat from a basic model to one from Ecobee. Once I got it installed I had a lot of extra control over what I could do with it. For example, I could now have the settings in the house run based on time-of-day instead of just one basic setting all the time. If I wanted it colder at night I could tell the system to look at the time and change the setting until it was sunrise. I could also tell it to look for me to be home (using geolocation) and raise and lower the temperature if my geotoken, in this case my phone, wasn’t in the area. The possibilities are endless because the system is driven by those inputs.

Automatic for the Non-people

Let’s extend the idea of closed-loop systems to network automation. Now, you can make a system (the network) behave a certain way based on inputs to the control functions. This is a massive change from the steady-state that we’ve worked years to achieve. The system can now react to changes in state or inputs. Massive file transfer activity being done between two branch locations? Closed-loop automation can reprogram the edge SD-WAN gateways to implement QoS policies based on the traffic types to preserve bandwidth for voice calls or critical application traffic. When the transfers are done the system can clean up the policies.

Because closed-loop automation can do a wide variety of actions based on inputs, data becomes super valuable. The information your system is providing as feedback can create more stable results. Open-loop systems are super stable because they are incapable of change. They also run every time someone tells them to run. They require intervention. Closed-loop systems are capable of running without the need for people based solely on the data you get from the system. But they also have issues because bad data or inputs can cause the system to react in strange ways. For example, if the thermostat in a house is placed in direct sunlight or has an error that causes it to think the house is 90 degrees, the A/C compressor may kick on even if the house temperature is far, far below that. Data has to be correct for the system to work as intended.


Tom’s Take

The promise of closed-loop automation is exciting. The ability for the network to run without our help is music to my ears. But it also means we have to be more diligent about keeping the control functions of the system working properly with the correct data inputs. It also means we need to monitor the control system outputs to head off problems before they can impact the reliability of the system. I can’t wait to see how we continue to close the loop and create better, more responsive systems in the future.

Fast Friday – Networking Field Day 22 Thoughts

Since I’m on the road again at Networking Field Day this week, I have had some great conversations with the delegates and presenters. A few stray thoughts that may develop into full blown blog posts at some point, but I figured I could get some of them out here for some quick entertainment.

  • The startup model means flexibility. That also means you can think about problems in a new light. So it would follow that you get to develop some new idea without a mountain of technical debt. Things like archaic platforms and crusty old user interfaces. You’d be surprised the amount of stuff that gets carried forward as technical debt.
  • Integrating products isn’t easy. Even if you think you’ve got the right slot for your newest acquisition you may find it isn’t the best fit overall. Or, even better, you may find a synergy you didn’t know existed because of a forgotten tool. Very rarely does anything just neatly fit into all your plans.
  • The more guest Wi-Fi I have to register for, the more I long for the days of Passport and OpenRoaming. If you already know who I am, why oh why must I continually register. Who wants to create Envoy, but for Wi-Fi?
  • There are days when I miss the CLI and doing stuff. Then I look at how complicated networks are now with the cloud and I realize I’d be in over my head. Also, no one wants to parse thousands of lines of log files. Even when I have insomnia.

Tom’s Take

I’ll have more good stuff soon. Don’t forget to check out the stuff I write for Gestalt IT, which includes posts from previous Field Day events and some briefings I’ve taken.

Agility vs. Flexibility

When you’re looking at moving to a new technology, whether it be SD-WAN or cloud, you’re going to be told all about the capabilities it has and all the shiny new stuff it can do for you. I would almost guarantee that you’re going to hear the words “agile” and “flexible” at some point during the conversation. Now, obviously those two things are different based on the fact there are two different words to describe what they do. But I’ve also heard people use them interchangeably. What does it mean to be agile? And is it better to be flexible too?

Agile Profile

Agility is the ability to move quickly and easily. It’s a quality displayed by athletes and fighters the world over. It’s a combination of reflexes and skill. Agility gives you the ability to react quickly to situations.

What does that mean in a technology sense? Mostly, agile solutions or methodologies are able to react to changing conditions or requirements quickly and adapt to meet those needs. Imagine a platform that can react to the changing needs of users. Or add new functions on the fly on demand. That’s the kind of agility that comes from software functionality or programmability. It’s a development team that can react without technical debt weighing them down.

But agile doesn’t always mean extensible. Just because you can react quickly doesn’t mean that you have the ability to extend the platform in ways that it can’t manage. Agile solutions can be rebuilt quickly but they have limitations. Usually, with technology, those limitations revolve around hardware. Agile solutions have to be built that way from the start. But it often means sacrifices must be made. Perhaps it didn’t ship with an interface that allows hardware to be added. Maybe the form factor is a limitation. A Raspberry Pi is a very agile platform within reason. But you’re not ever going to be able to build them into a GPU farm. Because they are locked into a specific kind of agility.

Flex Specs

Flexibility is the ability to react to new environments or changing requirements. That definition sounds an awful lot like the one above for agility, doesn’t it? They both sort of mean that you can change what you’re capable of. Flexibility is a characteristic that is usually used to describe gymnasts or dancers. Would you confuse a ballerina for a boxing champion? Likely not. Even though they can react to different situations they’re both different in many ways.

First and foremost, flexibility doesn’t require speed. Agility implicitly requires quick reactions. Flexibility can take time to adapt to things. Maybe that means adding new hardware to a server to expand GPU capabilities. Maybe it means adding modules to a software program to add new functions, like financial tracking added to a roster program. It may not be available right away but it is something that can be built in.

Flexibility on a hardware platform can take many directions. I always think of SD-WAN appliances as the ultimately form of flexibility. The more advanced units can run 4G/LTE modems in USB form. Or they can even run in the cloud without any specific hardware. The software platform isn’t tied to one specific hardware configuration or even form factor. It’s truly flexible because it doesn’t have any prerequisites or requirements.

But, as mentioned, flexibility isn’t always equated to agility. You can have a very flexible platform that requires a lot of time to build out. A classic example would be a desktop computer. It’s a very flexible platform but it takes time to install expansion cards and optional hardware. It’s also something that has to be configured and built to be flexible from the start. ATX motherboards have a certain kind of flexibility. Micro-ATX boards trade expansion flexibility for size flexibility. I can’t add two extra graphics cards to them but I can put the board into a case the size of a toaster.


Tom’s Take

What’s better? Agile or flexible? It depends on what kind of solution you need. Do you want to build on something? Or be able to upgrade it quickly? Is speed more important that creativity? There are so many dimensions that need to be considered. Most modern platforms have a few elements of each in their design. SD-WAN is both agile and flexible. Some solutions are more one than the other and that’s fine. Just remember that you need to ask for something very specific to meet criteria because if you’re looking for one you may end up with the other and not realize it until it’s too late.

Meraki Is Almost An Enterprise Solution

You may remember a three or so years ago when I famously declared that Meraki is not a good solution for enterprises. I know the folks at Meraki certainly haven’t. The profile for the hardware and services has slowly been rising inside of Cisco. More than just wireless with the requisite networking components, Meraki has now embraced security, SD-WAN, and even security cameras. They’ve moved into a lot of areas that customers have been asking about while also still trying to maintain the simplicity that Meraki is known for.

Having just finished up a Meraki presentation during Tech Field Day Extra at Cisco Live Europe, I thought it would be a good time to take a look at the progress that Meraki has been making toward embracing their enterprise customer base. I’m not entirely convinced that they’ve made it yet, but the progress is starting to look good.

Playing for Scale

The first area where Meraki is starting to really make strides is in the scalability department. This video from Tech Field Day Extra is all about new security features in the platform, specifically with firewalls. Take a quick look:

Toward the end of the video is one of the big things I got excited about. Meraki has introduced rule groups into their firewall platform. Sounds like a strange thing to get excited about, right? Kind of funny how the little things end up mattering in the long run. The real reason I’m getting excited about it has nothing to do with the firewall or even the interface. It has everything to do with being scalable.

One of the reasons why I’ve always seen Meraki as a solution that is more appropriate for small businesses is the lack of ability to scale. Meraki’s roots as a platform built for small deployments means that the interface has always been focused on making it easy to configure. You may remember from my last post that I wasn’t a fan of the way everything felt like it was driven through deployment wizards. Hand holding me through my first firewall deployment is awesome. Doing it for my 65th deployment is annoying. In enterprise solutions I can easily script or configure this via the command line to avoid the interface. But Meraki makes me use the UI to get it done.

Enterprises don’t run on wizards. They don’t work with assistance turned on. Enterprises need scalability. They need to be able to configure things to run across dozens or hundreds of devices quickly and predictably. They need that to happen quickly, too. Sure, it may only take four minutes to configure something via the firewall. Now, multiply that by 400 devices. Just that one little settings going to take over 26 hours to configure. And that’s assuming you don’t need to take time for a break or to sleep. When you’re working at the magnitude of an enterprise, those little shortcuts matter.

You might be saying right now, “But what about policies and groups for devices?” You would be right that groups can definitely speed up the process. But how many groups do you think the average enterprise would have for devices? I doubt all routers or switches or firewalls would conveniently fit into a single group. Or even ten groups. And there’s always the possibility that a policy change among those groups may get implemented correctly nine times out of those ten. The tenth time it gets an error that could still affect hundreds of devices. You see how this could get out of hand.

That’s why I’m excited about the little things like firewall groups. It means that Meraki is starting to see that these things need to be programmatically done. Building a series of policies in software makes it easy to deploy over and over again through scripting or enhanced device updating. Polices are good for rules. They’re not so good for devices. So the progress means that Meraki needs to keep building toward letting us script these deployments and updates across the entire organization.

Hextuple Option

The other thing that’s neatly buried at the end of the video is courtesy of a question from my friend Jody Lemoine (@GhostInTheNet). He points out that there are IPv6 addresses on the dashboard. The Meraki presenters confirm that they are testing IPv6 support natively and not just in bridged mode. Depending on when you read this post in the future, it may even be out already. You know that I’m an IPv6 fan and I’ve been tough on Meraki in the past about their support for it. So I’m happy to see that it’s in the works.

But more importantly I’m pleased that Meraki has jumped into a complex technical solution with both feet. Enterprises don’t need a basic set of services. They don’t want you to just turn on the twenty most common settings. Enterprises need odd things sometimes. They need longer VPN lifetimes or weird routing LSA support. Sometimes they need to do the really odd things because their thousand-odd devices really have to have this feature turned on to make it work.

Now, I’ve rightfully decried the idea that you should just do whatever your customers want, but the truth is that doing something silly for one customer isn’t the same as doing it for a dozen or more that are asking for a feature. Meraki has always felt shy to me about the way they implement features in their software. It’s almost the opposite of Cisco, in a way. Cisco is happy to include corner-case options on software releases on a whim to satisfy million-dollar customers. Meraki, on the other hand, has seemed to wait until well past critical mass to turn something on. It almost feels like you have to break down their door to get something advanced enabled.

To me, IPv6 is the watershed. It’s something that the general public doesn’t think they need or doesn’t know they really should have. Cisco has had IPv6 support in IOS for years. Meraki has been dragging along until they feel the need to implement it. But implementing it in 2020 makes me feel they will finally start implementing features in a way that makes sense for users. Hopefully that also means they’ll be more responsive to their Make A Wish feature requests and start indexing how many customers really want a certain feature or certain option enabled.

Napoleon Complex

The last thing that I’ll say about the transformation of Meraki is about their drive to embrace complexity. I know that Russ White and I don’t always see eye-to-eye about complexity. But I feel that hiding it is ultimately detrimental to IT staff members. Sure, you don’t want the CEO or the janitor in the wireless system deploying new SSIDs on a daily basis or disabling low data rates on APs. But you also need to have access to those features when the time comes. That was one of my big takeaways in my previous Meraki post.

I know that Meraki prides themselves on having a clean, easy-to-use interface. I know that it’s going to take a while before Meraki starts exposing their interface to power users. But, it also took Microsoft a long time to let people start doing massive modifications via PowerShell. Or Apple letting users go wild under the hood. These platforms finally opened a little and let people do some creative things. Sure, Apple IOS is still about as locked down as Meraki is, but every WWDC brings some new features that can be tinkered with here and there. I’m not expecting a fully complexity-embracing model in the next couple of years from Meraki, but I feel that the right people internally are starting to understand that growth comes in the form of enterprise customers. Enterprises don’t shy away from complexity. They don’t want it hidden. They want to see it and understand it and plan for it. And, ultimately, embrace it.


Tom’s Take

I will freely admit that I’m hard on the Meraki team. I do it because I see potential. I remember seeing them for the first time all the way back at Wireless Field Day 2 in their cramped San Francisco townhome office. In the years since the Cisco acquisition they’ve grown immensely with talent and technology. The road to becoming something more than you start out doing isn’t easy. And sometimes you need someone willing to stop you now and then and tell you where directions make more sense. I don’t believe for one moment that my armchair quarterbacking has really had a significant impact on the drive that Meraki has to get into larger enterprises. But I hope that my perspective has shown them how the practitioners of the world think and how they’re slowly transforming to meet those needs and goals. Hopefully in the next couple of years I can write the final blog post in this trilogy when Meraki embraces the enterprise completely.

Really Late Company Christmas Shopping

I’m headed out to Cisco Live Europe today, so I’m trying to get everything packed before I head to the airport. I also realize I need to go buy a few things for my suitcase. Which must be the same thing that a bunch of companies thought this week as they went on a buying spree! Seriously:

I don’t think we’re quite done yet, either. An oblique tweet from a friend with some inside sources leads me to believe that the reason why this is happening right now is because some of the venture funds are getting antsy and are calling in their markers. Maybe they need the funds to cash out investors? Maybe they’re looking to reduce their exposure to other things? Maybe they’re ready to jump on a plane to an uncharted island somewhere?

This is one of the challenges when you’re beholden to investors. Sure, not all of us are independently wealthy and capable of bootstrapping our own startup. We need some kind of funding to make that happen. But as soon as we do we are going to find ourselves at the mercy of their decisions and be forced to play by their rules.

If it’s time for them to get out of the position they have in a company, you’d better have the money. And if you don’t, they’re going to get it. I don’t know for sure what the situation is in both of those cases, but no one had really been talking publicly about buying Nyansa or Big Switch in the last few months. I had always figured that Nyansa would go to a bigger company, much like Aruba buying Rasa Networks in 2016. VMware is an interesting fit for them and a much better enterprise use of the technology in the long term.

Big Switch is puzzling for sure. From what I’ve heard they were profitable last quarter and bullish on the entire outlook for 2020. Did something change? Did the investors decide they wanted out? Or did some other market force push Big Switch to find a new home? When you look at the list of companies that were interested in buying them it’s not surprising. Dell Technologies would have been my first guess given their close working relationship. VMware would have been the second. Juniper and Extreme were interesting options but I’m not quite sure where the fit would be with them. And Cisco would have purchased as a purely defensive measure. So Arista is an interesting fit. I’m still waiting to hear some more details given how fresh this story is.

We’re into Q1 for most companies now. Or at least the ones that don’t have an odd FY schedule. So they’re realizing they either need to catch up on some R&D or that they have enough cash or equity lying around to go shopping. And if some of the companies on the market are selling at lower prices, it only makes sense to snap them up. Even if the integration pieces are going to take a while. Nyansa has great analytics, but it’s focused on the endpoint side. It’s going to take some work to make it all play nice with the other analytics pieces of VMware. That’s not cheap, but if the price of doing it through acquisition is cheaper than doing it through in-house efforts then buying your way in looks better in the long run. And if some venture fund is looking for cash at the same time, it could be a match made in heaven.


Tom’s Take

I’m a tech person. Even through the stuff I’ve done with Tech Field Day where I’ve had to learn more about financing and such I still consider myself a tech grunt first and foremost. When the talk turns to preferred share options and funding rounds and other such stuff I tend to look back at technology and figure out where that stuff is going. People that work with money for a living have a much different opinion of technology than tech people do. If that weren’t the case, we’d be talking about Betamax and HD-DVD more than we do now. But, money is still the way that tech gets done. And sometimes you need to do a little shopping to get the tech you need to keep building.

Why Do You Need NAT66?

It’s hard to believe that it’s been eight years since I wrote my most controversial post ever. I get all kinds of comments on my NAT66 post even to this day. I’ve been told I’m a moron, an elitist, and someone that doesn’t understand how the Internet works. I’ve also had some good comments that highlight a specific need for tools like NAT66. I wanted to catch up with everything and ask a very important question.

WHY?

Every Tool Has A Purpose

APNIC had a great post about NAT66 back in 2018. You should totally read it. I consider it a fair review of the questions surrounding NAT’s use in 2020. Again, NAT has a purpose and when used properly and sparingly for that purpose it works well. In the case of the article, Marco Cilloni (@MCilloni) lays out the need to use NAT66 to use IPv6 at his house due to ISP insanity and the latency overhead of using tunnels with Hurricane Electric. In this specific case, NAT66 was a good tool for him to use to translate his /128 address to something useable in his network.

If you’re brave, you should delve into the comments. A couple of my favorite passages:

People from your side completely fail to understand that while NAT was not designed for security, it did bring security, in particular for home users.

Either the IPv6 community sobers up and starts actively supporting NAT or you can kiss the IPv6 protocol goodbye. I’ve put many many hours into IPv6 integration and I’m starting to realize it’s a doomed protocol and should be scraped.

It’s obvious to me a decade later that there are two camps of people that discuss NAT66: Those that are using it for a specific purpose. And those that think it has to be enabled because they use it with IPv4 networks. An example of the former:

Pieter knew what he needed to do to make it work and he did it. Not because it was something that he configured on his home router to make the Internet work. But he also knew this wasn’t the optimal solution. When you can’t change the ISP router to accept RAs you need a workaround. There are a ton of stories I get from people just like Pieter that involve a workaround or a specific thing like provider-independent address space not being available. These are the kinds of situations that tools like NAT were designed to solve.

X, Why, Z

Let’s get back to my earlier question: WHY?

For those that believe that NAT66 needs to be used everywhere, why? Is it because you’re used to using RFC1918 address space to conserve addresses? Maybe you don’t like the idea of your MAC address “leaking” onto the Internet? How about providing enhanced security, as the commenters above mentioned? There’s even a comment on my original post from late last year about using NAT to force redirects for DNS entries to avoid having Google overriding DNS on his Android phone. Why?

For me, this always comes back to the same answer I give again and again. NAT, used for a purpose, isn’t inherently evil or wrong. It’s when people start using it for things it wasn’t designed for and breaking every other thing that it becomes a crutch and a problem. And those problematic solutions always cause issues somewhere down the line.

NAT44 broke FTP. It broke end-to-end communications. It created the need for big, hungry middle boxes to track state of connections. It conflated addressing and firewall functions. Anyone that screams at me and tells me that NAT provides security by obscuring addresses usually has an edge firewall doing NAT for them. In a true one-to-one NAT configuration, accessing the public or global IP address of the host in question does nothing to halt the traffic from being passed through. People who talk to be about address obfuscation with NAT44 or NAT66 usually mean PAT, not NAT. They want one address masquerading as all the addresses in their organization to “hide” themselves from the Internet.

Why does NAT need to solve those problems? I get the complexity of using provider independent (PI) space internally and the need to configure BGP to avoid asymmetric routing. Especially if your upstream provider isn’t paying attention to communities or attributes you’re using to avoid creating a transit network or weight traffic to prefer one link over the other. NAT may be a good short-term solution for you in these cases. But do you really want to run NAT66 for the next decade because of a policy issue with your ISP? That, to me, is the ultimate in passive-aggressive configuration. Why not just jump through hoops instead of hammering out a real solution?

I may sound like a 5-year-old, but “WHY” is really the most important question you can ask. Why do you need NAT66? Why do you even need IPv6? Is it a requirement for a contract? Maybe you have to enable it to allow your application to be uploaded to the walled garden store for your mobile provider. Maybe you just want to play around with it and get an Hurricane Electric Sage t-shirt. But if you can’t answer “WHY” then all the other things you want aren’t going to make sense.

I don’t run my HE.net tunnel at home any longer. I didn’t have an advantage in running IPv6 and I had more than a few headaches that had to be addressed. There will come a day when I want to do more with IPv6, but that’s going to require more bandwidth than I have right now. I still listen to IPv6 podcasts all the time, like the excellent IPv6 Buzz featuring my friend Ed Horley. Even the experts are bullish about the adoption of IPv6 but not ignorant of the challenges. And these guys run a business doing IPv6.

For those of you that are already limbering up your fingers to leave me a comment, stop and ask yourself “WHY” first. Why do you need NAT66? Is it because you have a specific problem you can’t solve any other way? Or is it because you need NAT66 to be there just like ISDN dialer maps and reserved VLANs on switches? To me, even past my days in the trenches as an engineer, the days of needing NAT everywhere are long gone. The IPv4 Internet relies on NAT. We are hobbled by that fact. VPNs need NAT traversal. Game consoles and VoIP devices need to be NAT-aware, which increases complexity.

The IPv6 Internet doesn’t need to be like that. Let’s teach the “right” way to do things. You don’t need NAT66 for privacy. RFC 4941 exists for that. You don’t need to think NAT66 provides security. That’s what perimeter devices are for. Anything more complicated than those two “simple” cases is going to be an exercise in frustration. You don’t need to bellow from the rooftops that NAT is a necessary and mandatory piece of all Internet traffic. Instead, come back to “WHY”. Why do two devices need a middle box to translate for them and hold state information? Why can’t my ISP provide me the address space I want or the connectivity options that make this work easily? The more “WHY” questions you ask, the more the answers will come. If you just want to fold your arms together and claim that NAT is needed because “This is the way,” you may find yourself alone on the Island of NAT sooner than you think.


Tom’s Take

My identity as the “I Hate NAT” guy is pretty solid at this point in my life. It’s too late to change now. Sure, I don’t hate NAT completely. It’s like a vulture to me. It serves a specific purpose but having it everywhere is almost always problematic. By now, the only way I can work against the needless expansion of NAT is to ask hard questions. Ironically enough, the hard questions aren’t multi-part essays that require an open-book test to resolve. Often, the hardest questions can just be a single word that forces you to question what you need and why you need it.

The Art of Saying “No”

No.

It’s the shortest sentence in the English language. It requires no other parts of speech. It’s an answer, a statement, and a command all at once. It’s a phrase that some people have zero issues saying over and over again. And yet, some others have an extremely difficult time answering anything in the negative.

I had a fun discussion on twitter yesterday with some friends about the idea behind saying “no” to people. It started with this tweet:

Coincidentally, I tweeted something very similar to what Bob Plankers had tweeted just hours before:

The gist is the same though. Crazy features and other things that have been included in software and hardware because someone couldn’t tell another person “no”. Sadly, it’s something that happens a lot in the IT industry. As a bad as IT’s reputation for being the Department of NO is we often find ourselves backed into a corner when it comes to saying “yes” way too much. I wanted to examine a couple of specific situations when we really should be saying “no” to people instead of just agreeing to keep the conversation moving.

Whatever You Need, We Do

When I worked at a VAR, I did both pre- and post-sales. I would go out to the customer site with the account managers to discuss technologies and try to get the potential customer what they needed. One of the AMs I worked with loved to introduce me and infer my skill level by saying, “Tom is the guy that makes all my lies come true.” It was his favorite icebreaker. We would all chuckle and get the conversation started.

Sadly, that icebreaker was true more often than it should have been. Because he (and some other AMs) would very often tell the customer whatever they wanted to hear to close the sale. Promise we could install the whole system in three hours? Easy. Tell them it will fix all their crazy Internet speed problems? You got it. Even as bad as telling this this will make their applications run so much faster and keep them super secure the whole time. Whatever it takes to make you sign the check.

When I arrived on site with a pile of equipment and a list of things that I needed to configure, I was quite often stricken with frustration because of the way my AMs had fibbed to the customer about the capabilities of the solution. Maybe they sold the wrong licenses to keep the costs down. Or, in some cases, they sold a feature that was much harder to implement than others. I seriously couldn’t count on both hands and feet the number of times I was forced to go to the customer and ask them what they were expecting from the solution based on what was sold to them.

Sometimes, you have to say “no”. That’s a hard phrase to say when you work in sales. You want the customer to get your product or service instead of your competitors. You want to book revenue. You want to keep your boss happy and keep yourself employed. You want to meet your goals. But you also don’t want to burn your bridges when it comes to being a good resource instead of someone just looking to make a buck.

I always tried to position myself as someone that could off impartial advice about a subject. If the customer wanted something that I couldn’t deliver I would say, “That’s not a good idea” or “Have you thought about why you want that?” I wanted to make sure that the customer really did want the thing they were asking for. Anyone that’s ever had a CEO or CIO clamor to implement a thing they say in an airport ad after coming back from a conference trip will attest to the power of wanting cool, shiny things.

Being a truly trusted advisor to your client means you have to be honest. No, that open source project won’t get you what you’re looking for just because it’s free. No, you can’t make your old intercom system work with a new VoIP UC solution. No, you can’t just keep running this server another three years on Windows 2003 Server so you can avoid the upgrade fees for your new clients. Saying “no” isn’t just about making them avoid things they don’t want to do. It’s about helping them understand a strategy and vision for what they need to be doing. Customers don’t always need to be told what they want to hear. They really do need to be told what they need to hear though.

Managing Products, I Think

The other side of the equation comes from the vendor side with product managers. I’ll admit that I have a limited view here, but the people that I’ve talked to seem to back up my thoughts on the matter. As stated above, I’ve always wondered how crazy random features made it into a software product. My supposition is that someone wanted to close a million-dollar deal somewhere and that feature was one of the things that it took to make that happen.

I also know that crazy things like this happen more often than you might realize. For example, ever wonder why wireless access points come configured with 80 MHz channels out-of-the-box when everyone you know, vendors included, tell you to configure them for 20 MHz or even 40 MHz instead? Could it be that when testing companies pull the APs out of the box that they don’t reconfigure the channels? Or perhaps it’s because those APs with 80 MHz defaults seem “faster” on those same tests? It’s a silly default configuration but it wins contests and reports. That’s the kind of decision that gets made by a product manager that wants to win customers or awards.

I would hope that the people that make products understand that people don’t really need insane corner case features to make products work. Worse yet, having those crazy features involved to support a random solution that is likely going to be replaced in a few years anyway cuts into partner revenue. The vendor shouldn’t be the one making their equipment compatible with every piece of hardware under the sun. Microsoft doesn’t write all the drivers for hardware to work with Windows, for example. They just write the specs for interfacing with the OS and leave the driver software writing up to the people that make the webcams or Bluetooth coffee mugs.

Vendors need to let the integration work happen with the integrators. Maybe they get access to some kind of advanced API or toolkit that assists with writing the “glue” that ties systems together. But building in basic support for everything under the sun from the outset creates support nightmares and unforeseen interactions with things that you will own for the next decade. Take the easy way out and tell people “no” and that they need to find someone to help them instead of just begging to have that crazy feature request included in a one-off build. Or, worse yet, included in main release and enabled by default.


Tom’s Take

I will admit that I have a really hard time saying no to things. It increases my workload and makes me so distracted that I can barely see straight most of the time. But there are times that I know I need to respond in the negative to something. It’s usually when I see that the person making the request either doesn’t know what they’re asking for or will end up regretting it later on. The key is to help them understand that you have the experience they lack and the vision to see this isn’t going to work the way they are planning. Hopefully they’ll come around to your way of thinking. But if not, just remember that “No.” is a complete sentence.