What Is Closed-Loop Automation?

During Networking Field Day 22 last week, a lot the questions that were directed at the presenters had to do with their automation systems. One term kept coming up that I was embarrassed to admit that I’d never heard of. Closed-loop automation is the end goal for these systems. But what is closed-loop automation? And why is it so important. I decided to do a little research and find out.

Open Up

To understand closed-loop systems, you have to understand open-loop systems first. Thankfully, those are really simple. Open-loop systems are those where the output isn’t directly affected by the control actions of the system. It’s a system where you’re going to get the output no matter how you control it. The easiest example is a clothes dryer. There are a multitude of settings that you can choose for a clothes dryer, including the timing of the cycle. But no matter what, the dryer will stop at the end of the cycle. There’s no sensor in a basic clothes dryer that senses the moisture level of the clothes and acts accordingly.

Open-loop systems are stable and consistent. Every time you turn on the dryer, it will run until it finishes. There’s no variable in the system that will change that. Aside from system failure, it’s going to run exactly 30 minutes every time it’s set to that cycle. It’s also not going to run unless you set the cycle. As my family will tell you, putting clothes in the dryer and not setting it will not result in magic happening.

Close It Off

In contrast, closed-loop systems have outputs that are dependent upon the control function of the system. If the control function requires something change in the system to achieve the desired output then it will change that thing to get there.

The most classic example of a closed-loop system is the HVAC system in your house. The control function is the thermostat. If you want the temperature in your house to be 70 degrees Fahrenheit (21 degrees C), you set the thermostat and let the system take care of things. If the temperature falls below the required setting, the heating unit will turn on and bring the temperature up to the required level before shutting off. In the summertime, rising above the temperature setting will cause the air conditioning compressor to kick on and cool things down.

Closed loop systems are great because you set them and forget them. Unlike the dryer example above I can set my thermostat and it will run even if I forget to go turn on the heater/AC. But they’re also more complicated to troubleshoot and figure out. As someone with very little practical knowledge of the operation of HVAC it’s rough to figure out if it’s the thermostat or the unit or some other relay somewhere that’s causing your house to be too warm or too cold.

Closed loop systems can also take more inputs given the right control settings. Using the same A/C example, I upgraded my thermostat from a basic model to one from Ecobee. Once I got it installed I had a lot of extra control over what I could do with it. For example, I could now have the settings in the house run based on time-of-day instead of just one basic setting all the time. If I wanted it colder at night I could tell the system to look at the time and change the setting until it was sunrise. I could also tell it to look for me to be home (using geolocation) and raise and lower the temperature if my geotoken, in this case my phone, wasn’t in the area. The possibilities are endless because the system is driven by those inputs.

Automatic for the Non-people

Let’s extend the idea of closed-loop systems to network automation. Now, you can make a system (the network) behave a certain way based on inputs to the control functions. This is a massive change from the steady-state that we’ve worked years to achieve. The system can now react to changes in state or inputs. Massive file transfer activity being done between two branch locations? Closed-loop automation can reprogram the edge SD-WAN gateways to implement QoS policies based on the traffic types to preserve bandwidth for voice calls or critical application traffic. When the transfers are done the system can clean up the policies.

Because closed-loop automation can do a wide variety of actions based on inputs, data becomes super valuable. The information your system is providing as feedback can create more stable results. Open-loop systems are super stable because they are incapable of change. They also run every time someone tells them to run. They require intervention. Closed-loop systems are capable of running without the need for people based solely on the data you get from the system. But they also have issues because bad data or inputs can cause the system to react in strange ways. For example, if the thermostat in a house is placed in direct sunlight or has an error that causes it to think the house is 90 degrees, the A/C compressor may kick on even if the house temperature is far, far below that. Data has to be correct for the system to work as intended.


Tom’s Take

The promise of closed-loop automation is exciting. The ability for the network to run without our help is music to my ears. But it also means we have to be more diligent about keeping the control functions of the system working properly with the correct data inputs. It also means we need to monitor the control system outputs to head off problems before they can impact the reliability of the system. I can’t wait to see how we continue to close the loop and create better, more responsive systems in the future.

Fast Friday – Networking Field Day 22 Thoughts

Since I’m on the road again at Networking Field Day this week, I have had some great conversations with the delegates and presenters. A few stray thoughts that may develop into full blown blog posts at some point, but I figured I could get some of them out here for some quick entertainment.

  • The startup model means flexibility. That also means you can think about problems in a new light. So it would follow that you get to develop some new idea without a mountain of technical debt. Things like archaic platforms and crusty old user interfaces. You’d be surprised the amount of stuff that gets carried forward as technical debt.
  • Integrating products isn’t easy. Even if you think you’ve got the right slot for your newest acquisition you may find it isn’t the best fit overall. Or, even better, you may find a synergy you didn’t know existed because of a forgotten tool. Very rarely does anything just neatly fit into all your plans.
  • The more guest Wi-Fi I have to register for, the more I long for the days of Passport and OpenRoaming. If you already know who I am, why oh why must I continually register. Who wants to create Envoy, but for Wi-Fi?
  • There are days when I miss the CLI and doing stuff. Then I look at how complicated networks are now with the cloud and I realize I’d be in over my head. Also, no one wants to parse thousands of lines of log files. Even when I have insomnia.

Tom’s Take

I’ll have more good stuff soon. Don’t forget to check out the stuff I write for Gestalt IT, which includes posts from previous Field Day events and some briefings I’ve taken.

Agility vs. Flexibility

When you’re looking at moving to a new technology, whether it be SD-WAN or cloud, you’re going to be told all about the capabilities it has and all the shiny new stuff it can do for you. I would almost guarantee that you’re going to hear the words “agile” and “flexible” at some point during the conversation. Now, obviously those two things are different based on the fact there are two different words to describe what they do. But I’ve also heard people use them interchangeably. What does it mean to be agile? And is it better to be flexible too?

Agile Profile

Agility is the ability to move quickly and easily. It’s a quality displayed by athletes and fighters the world over. It’s a combination of reflexes and skill. Agility gives you the ability to react quickly to situations.

What does that mean in a technology sense? Mostly, agile solutions or methodologies are able to react to changing conditions or requirements quickly and adapt to meet those needs. Imagine a platform that can react to the changing needs of users. Or add new functions on the fly on demand. That’s the kind of agility that comes from software functionality or programmability. It’s a development team that can react without technical debt weighing them down.

But agile doesn’t always mean extensible. Just because you can react quickly doesn’t mean that you have the ability to extend the platform in ways that it can’t manage. Agile solutions can be rebuilt quickly but they have limitations. Usually, with technology, those limitations revolve around hardware. Agile solutions have to be built that way from the start. But it often means sacrifices must be made. Perhaps it didn’t ship with an interface that allows hardware to be added. Maybe the form factor is a limitation. A Raspberry Pi is a very agile platform within reason. But you’re not ever going to be able to build them into a GPU farm. Because they are locked into a specific kind of agility.

Flex Specs

Flexibility is the ability to react to new environments or changing requirements. That definition sounds an awful lot like the one above for agility, doesn’t it? They both sort of mean that you can change what you’re capable of. Flexibility is a characteristic that is usually used to describe gymnasts or dancers. Would you confuse a ballerina for a boxing champion? Likely not. Even though they can react to different situations they’re both different in many ways.

First and foremost, flexibility doesn’t require speed. Agility implicitly requires quick reactions. Flexibility can take time to adapt to things. Maybe that means adding new hardware to a server to expand GPU capabilities. Maybe it means adding modules to a software program to add new functions, like financial tracking added to a roster program. It may not be available right away but it is something that can be built in.

Flexibility on a hardware platform can take many directions. I always think of SD-WAN appliances as the ultimately form of flexibility. The more advanced units can run 4G/LTE modems in USB form. Or they can even run in the cloud without any specific hardware. The software platform isn’t tied to one specific hardware configuration or even form factor. It’s truly flexible because it doesn’t have any prerequisites or requirements.

But, as mentioned, flexibility isn’t always equated to agility. You can have a very flexible platform that requires a lot of time to build out. A classic example would be a desktop computer. It’s a very flexible platform but it takes time to install expansion cards and optional hardware. It’s also something that has to be configured and built to be flexible from the start. ATX motherboards have a certain kind of flexibility. Micro-ATX boards trade expansion flexibility for size flexibility. I can’t add two extra graphics cards to them but I can put the board into a case the size of a toaster.


Tom’s Take

What’s better? Agile or flexible? It depends on what kind of solution you need. Do you want to build on something? Or be able to upgrade it quickly? Is speed more important that creativity? There are so many dimensions that need to be considered. Most modern platforms have a few elements of each in their design. SD-WAN is both agile and flexible. Some solutions are more one than the other and that’s fine. Just remember that you need to ask for something very specific to meet criteria because if you’re looking for one you may end up with the other and not realize it until it’s too late.

Really Late Company Christmas Shopping

I’m headed out to Cisco Live Europe today, so I’m trying to get everything packed before I head to the airport. I also realize I need to go buy a few things for my suitcase. Which must be the same thing that a bunch of companies thought this week as they went on a buying spree! Seriously:

I don’t think we’re quite done yet, either. An oblique tweet from a friend with some inside sources leads me to believe that the reason why this is happening right now is because some of the venture funds are getting antsy and are calling in their markers. Maybe they need the funds to cash out investors? Maybe they’re looking to reduce their exposure to other things? Maybe they’re ready to jump on a plane to an uncharted island somewhere?

This is one of the challenges when you’re beholden to investors. Sure, not all of us are independently wealthy and capable of bootstrapping our own startup. We need some kind of funding to make that happen. But as soon as we do we are going to find ourselves at the mercy of their decisions and be forced to play by their rules.

If it’s time for them to get out of the position they have in a company, you’d better have the money. And if you don’t, they’re going to get it. I don’t know for sure what the situation is in both of those cases, but no one had really been talking publicly about buying Nyansa or Big Switch in the last few months. I had always figured that Nyansa would go to a bigger company, much like Aruba buying Rasa Networks in 2016. VMware is an interesting fit for them and a much better enterprise use of the technology in the long term.

Big Switch is puzzling for sure. From what I’ve heard they were profitable last quarter and bullish on the entire outlook for 2020. Did something change? Did the investors decide they wanted out? Or did some other market force push Big Switch to find a new home? When you look at the list of companies that were interested in buying them it’s not surprising. Dell Technologies would have been my first guess given their close working relationship. VMware would have been the second. Juniper and Extreme were interesting options but I’m not quite sure where the fit would be with them. And Cisco would have purchased as a purely defensive measure. So Arista is an interesting fit. I’m still waiting to hear some more details given how fresh this story is.

We’re into Q1 for most companies now. Or at least the ones that don’t have an odd FY schedule. So they’re realizing they either need to catch up on some R&D or that they have enough cash or equity lying around to go shopping. And if some of the companies on the market are selling at lower prices, it only makes sense to snap them up. Even if the integration pieces are going to take a while. Nyansa has great analytics, but it’s focused on the endpoint side. It’s going to take some work to make it all play nice with the other analytics pieces of VMware. That’s not cheap, but if the price of doing it through acquisition is cheaper than doing it through in-house efforts then buying your way in looks better in the long run. And if some venture fund is looking for cash at the same time, it could be a match made in heaven.


Tom’s Take

I’m a tech person. Even through the stuff I’ve done with Tech Field Day where I’ve had to learn more about financing and such I still consider myself a tech grunt first and foremost. When the talk turns to preferred share options and funding rounds and other such stuff I tend to look back at technology and figure out where that stuff is going. People that work with money for a living have a much different opinion of technology than tech people do. If that weren’t the case, we’d be talking about Betamax and HD-DVD more than we do now. But, money is still the way that tech gets done. And sometimes you need to do a little shopping to get the tech you need to keep building.

The Art of Saying “No”

No.

It’s the shortest sentence in the English language. It requires no other parts of speech. It’s an answer, a statement, and a command all at once. It’s a phrase that some people have zero issues saying over and over again. And yet, some others have an extremely difficult time answering anything in the negative.

I had a fun discussion on twitter yesterday with some friends about the idea behind saying “no” to people. It started with this tweet:

Coincidentally, I tweeted something very similar to what Bob Plankers had tweeted just hours before:

The gist is the same though. Crazy features and other things that have been included in software and hardware because someone couldn’t tell another person “no”. Sadly, it’s something that happens a lot in the IT industry. As a bad as IT’s reputation for being the Department of NO is we often find ourselves backed into a corner when it comes to saying “yes” way too much. I wanted to examine a couple of specific situations when we really should be saying “no” to people instead of just agreeing to keep the conversation moving.

Whatever You Need, We Do

When I worked at a VAR, I did both pre- and post-sales. I would go out to the customer site with the account managers to discuss technologies and try to get the potential customer what they needed. One of the AMs I worked with loved to introduce me and infer my skill level by saying, “Tom is the guy that makes all my lies come true.” It was his favorite icebreaker. We would all chuckle and get the conversation started.

Sadly, that icebreaker was true more often than it should have been. Because he (and some other AMs) would very often tell the customer whatever they wanted to hear to close the sale. Promise we could install the whole system in three hours? Easy. Tell them it will fix all their crazy Internet speed problems? You got it. Even as bad as telling this this will make their applications run so much faster and keep them super secure the whole time. Whatever it takes to make you sign the check.

When I arrived on site with a pile of equipment and a list of things that I needed to configure, I was quite often stricken with frustration because of the way my AMs had fibbed to the customer about the capabilities of the solution. Maybe they sold the wrong licenses to keep the costs down. Or, in some cases, they sold a feature that was much harder to implement than others. I seriously couldn’t count on both hands and feet the number of times I was forced to go to the customer and ask them what they were expecting from the solution based on what was sold to them.

Sometimes, you have to say “no”. That’s a hard phrase to say when you work in sales. You want the customer to get your product or service instead of your competitors. You want to book revenue. You want to keep your boss happy and keep yourself employed. You want to meet your goals. But you also don’t want to burn your bridges when it comes to being a good resource instead of someone just looking to make a buck.

I always tried to position myself as someone that could off impartial advice about a subject. If the customer wanted something that I couldn’t deliver I would say, “That’s not a good idea” or “Have you thought about why you want that?” I wanted to make sure that the customer really did want the thing they were asking for. Anyone that’s ever had a CEO or CIO clamor to implement a thing they say in an airport ad after coming back from a conference trip will attest to the power of wanting cool, shiny things.

Being a truly trusted advisor to your client means you have to be honest. No, that open source project won’t get you what you’re looking for just because it’s free. No, you can’t make your old intercom system work with a new VoIP UC solution. No, you can’t just keep running this server another three years on Windows 2003 Server so you can avoid the upgrade fees for your new clients. Saying “no” isn’t just about making them avoid things they don’t want to do. It’s about helping them understand a strategy and vision for what they need to be doing. Customers don’t always need to be told what they want to hear. They really do need to be told what they need to hear though.

Managing Products, I Think

The other side of the equation comes from the vendor side with product managers. I’ll admit that I have a limited view here, but the people that I’ve talked to seem to back up my thoughts on the matter. As stated above, I’ve always wondered how crazy random features made it into a software product. My supposition is that someone wanted to close a million-dollar deal somewhere and that feature was one of the things that it took to make that happen.

I also know that crazy things like this happen more often than you might realize. For example, ever wonder why wireless access points come configured with 80 MHz channels out-of-the-box when everyone you know, vendors included, tell you to configure them for 20 MHz or even 40 MHz instead? Could it be that when testing companies pull the APs out of the box that they don’t reconfigure the channels? Or perhaps it’s because those APs with 80 MHz defaults seem “faster” on those same tests? It’s a silly default configuration but it wins contests and reports. That’s the kind of decision that gets made by a product manager that wants to win customers or awards.

I would hope that the people that make products understand that people don’t really need insane corner case features to make products work. Worse yet, having those crazy features involved to support a random solution that is likely going to be replaced in a few years anyway cuts into partner revenue. The vendor shouldn’t be the one making their equipment compatible with every piece of hardware under the sun. Microsoft doesn’t write all the drivers for hardware to work with Windows, for example. They just write the specs for interfacing with the OS and leave the driver software writing up to the people that make the webcams or Bluetooth coffee mugs.

Vendors need to let the integration work happen with the integrators. Maybe they get access to some kind of advanced API or toolkit that assists with writing the “glue” that ties systems together. But building in basic support for everything under the sun from the outset creates support nightmares and unforeseen interactions with things that you will own for the next decade. Take the easy way out and tell people “no” and that they need to find someone to help them instead of just begging to have that crazy feature request included in a one-off build. Or, worse yet, included in main release and enabled by default.


Tom’s Take

I will admit that I have a really hard time saying no to things. It increases my workload and makes me so distracted that I can barely see straight most of the time. But there are times that I know I need to respond in the negative to something. It’s usually when I see that the person making the request either doesn’t know what they’re asking for or will end up regretting it later on. The key is to help them understand that you have the experience they lack and the vision to see this isn’t going to work the way they are planning. Hopefully they’ll come around to your way of thinking. But if not, just remember that “No.” is a complete sentence.

Time For Improvement

Welcome to 2020! First and foremost, no posts from me involving vision or eyesight or any other optometrist puns for this year. I promise 366 days free of anything having to do with eyeballs. That does mean a whole world of other puns that I’m going to be focusing on!

Now, let’s look back at 2019. The word that I could use to describe it was “hectic”. It felt like everything was in overdrive all year long. There were several times that I got to the end of the week and realized that I didn’t have any kind of post ready to go. I’m the kind of person that likes to write when the inspiration hits me. And instead I found myself scrambling to write up some thoughts. And that was something I told myself that I was going to get away from. So we’re going to call that one a miss and get back to trying to post on a day other than Friday.

That also means that, given all the other content that I’ve been working on with Gestalt IT that I’m going to have to schedule some time actually working on that content instead of hoping that some idea is going to fly out of the blue at 11:30pm the night before I’m supposed to put a post up. The good news is that also means that I’m going to be upping the amount of content that I’m consuming for inspiration. Since I spent a good chunk of they year going on a morning walk it meant that I had a lot more time to consume podcast episodes and wash those ideas around. I’m sure that means that I’m going to find the time and the motivation to keep turning out content.

Part of the reason for that is because of something that Stephen Foskett (@SFoskett) told me during a call this past year. He said that I’ve been consistently turning out content for the last 9 years on a weekly basis. I’m proud of that fact. Sure, there’s been a couple of times in the last year or two when I’ve missed and had to publish something on a Saturday or the Monday after. But overall I’m happy with the amount of content that I’ve been writing here. And because you all keep on reading it I’m going to keep writing it. There’s a lot of value in what I do here and I hope you all continue to value it too.

IA Writing

Last January I switched over to using IA Writer for my posts on my iPad. I wrote primarily on that platform all year long. I can say that It’s very handy to be able to grab your mobile device and hammer out a post. Given that I can do split screen and reference my hand-written notes from briefings it’s a huge advantage to keeping my thoughts organized and ready to put down on paper.

Between IA Writer for writing, Notability for taking notes during briefings, and Things to keep me on track for the posts that I need to cover I’ve gotten my workflow down to something that works for me. I’m going to keep tweaking it for sure but I’m happy that I can get information to a place where I can refer to it later and have reminders about what I need to cover. It makes everything seamless and consistent. There are still some things that I need to use Microsoft Word to write, but those are long-form projects. Overall, I’m going to keep refining my process to make it better and more appropriate for me.

Ultimately, that’s a big goal for me in 2020 and something that I’ve finally realized that I do regularly without conscious thought. If you’ve read any books on process or project management you’ve probably heard of kaizen, the Japanese concept of continuous improvement of processes. It’s something that drives companies like Toyota to get better at everything they do and never accept anything as “complete”.

I’ve read about kaizen before but I never really understood that it could mean any improvement before. I had it in my head that the process was about change all the time. It wasn’t until I sat down this year and analyzed what I was doing to find that I’m always trying to optimize what I do. It’s not about finding shortcuts for the sake of saving time. It’s about optimizing what I do to save effort and the investment of time. For me it’s not about spending 8 hours to write a script that will automate a one-time 30-minute task. It’s about breaking down the task and figuring out how many times I’ll do it and how I need to optimize the process to spend less time on it. If the answer is a script or an automation routine then I’m all for it. But the key is recognizing the kaizen process and putting a name to my behavior.


Tom’s Take

2020 is going to be busy. Tech Field Day is going to be busy. I’m going to be at a lot of events checking out what’s going on and how to make new things happen. I’m also going to be writing a lot. And when you factor in my roles outside of work with Wood Badge and a trip to Philmont, NM with my son for a high adventure trip with his scout troop you can see I’m going to be quite occupied even when I’m not writing. But I’m not going to remove anything from my process. As I said above, I’m going to kaizen everything and fit it all in. That might mean having a couple of posts queued up when I’m in the back country or taking some extra time after dinner to write. But 2020 is going to be a big year of optimizing my workflows and improving in every way.

Fast Friday- Keeping Up With The Times

We’re at the end of the 2010s. It’s almost time to start making posts about 2020 and somehow working vision or eyesight into the theme so you can look just like everyone else. But I want to look back for a moment on how much things have changed for networking in the last ten years.

It’s true that networking wasn’t too exciting for most of the 2000s. Things got faster and more complicated. Nothing really got better except the bottom lines of people pushing bigger hardware. And that’s honestly how we liked it. Because the idea that we were all special people that needed to be at the top of our game to get things done resonated with us. We weren’t just mechanics. We were the automobile designers of the future!

But if there’s something that the mobile revolution of the late 2000s taught us, it was that operators don’t need to be programmers to enjoy using technology. Likewise, enterprise users don’t need to be CCIEs or VCDXs to make things work. That’s the real secret behind all the of the advances in networking technology in the 2010s. We’re not making networking harder any more. We’re not adding complexity for the sake of making our lives more important.

The rapid pace of change that we’ve had over the last ten years is the driver for so many technologies that are changing the role of networking engineers. Automation, orchestration, and software-driven networking aren’t just fads. They’re necessities. That’s because of the focus on new features, not in spite of them. I can remember administering CallManager years ago and not realizing what half of the checkboxes on a line appearance did. That’s not helping things.

Complexity Closet

There are those that would say that what we’re doing is just hiding the complexity behind another layer of abstraction, which is a favorite saying of Russ White. I’d argue that we’re not hiding the complexity as much as we’re putting it back where it belongs – out of sight. We don’t need the added complexity for most operations. This flies in the face of what we want networking engineers to know. If you want to be part of the club you have to know every LSA type and how they interact and what happens during an OSPF DR election. That’s the barrier for entry, right?

Except not everyone needs to know that stuff. They need to know what it looks like when routing is down. More likely, they just need to recognize more basic stuff like DNS being down or something being wrong in the service provider. The odds are way better that something else is causing the problem somewhere outside of your control. And that means your skills are good for very little once you’ve figured out that the problem is somewhere you can’t help.

Hiding complexity behind summary screens or simple policy decisions isn’t bad. In fact, it tends to keep people from diving down rabbit holes when fixing the problems. How many times have we tried to figure out some complicated timer issue when it was really a client with a tenuous connection or a DNS issue? We want the problems to be complicated so we can flex our knowledge to others when in fact we should be applying Occam’s Razor much earlier in the process. Instead of trying to find the most complicated solution to the problem so we can justify our learning, we should instead try to make it as simple as possible to conserve that energy for a time when it’s really needed.


Tom’s Take

We need to leverage the tools that have been developed to make our lives easier in the 2020s instead of railing against them because they’re upsetting our view of how things should be. Maybe networking in 2010 needed complexity and syntax and command lines. But networking in 2022 might need automated scripts and telemetry to figure things out faster since there are ten times the moving parts. It’s like memorizing phone numbers. It works well when you only need to know seven or eight with a few digits. But when you need to know several hundred each with ten or more digits it’s impossible. Yet I still hear people complain about contact lists or phone books because “they used to be good at memorizing numbers”. Instead of us focusing on what we used to be good at, let’s try to keep with the times and be good at what we need to be good at now and in the future.