Visibility In Networking – Quick Thoughts from Networking Field Day

nfd-logo

I’m at Networking Field Day 13 this week. You can imagine how much fun I’m having with my friends! I wanted to drop some quick thoughts on visibility for this week on you all about what we’re hearing and raise some interesting questions.

I Can See Clearly Now

Visibility is a huge issue for companies. Seeing what’s going on is hard for people. Companies like Ixia talk about the need to avoid dropping any packets to make sure we have complete knowledge of the network. But that requires a huge amount of hardware and design. You’re always going to need traditional monitoring even when everything is using telemetry and other data models. Make sure you size things right.

Forward Networks told us that there is an increasing call for finding a way to monitor both the underlay network and the overlay network. Most overlay companies give you a way to tie into their system via API or other telemetry. However, there is no visibility into the underlay because of the event horizon. Likewise, companies like Forward Networks are focusing on the underlay with mapping technologies and modeling software but they can’t pass back through the event horizon to see into the overlay. Whoever ends up finding a way to marry both of these together is going to make a lot of money.

Apstra is taking the track of not caring what the underlay looks like. They’re going to give you the tools to manage it all without hard setup. You can rip and replace switches as needed with multivendor support. That’s a huge win if you run a heterogeneous network or you’re looking to start replacing traditional hardware with white or bright box options. Likewise, their ability to pull configs can help you visualize your device setup more effectively no matter what’s under there.


Tom’s Take

I’ve got some more Networking Field Day thoughts coming soon, but I wanted to get some thoughts out there for you to think about this weekend. Stay tuned for some new ideas coming out of the event!

How To Ask A Question At A Conference

question-mark-706906_1280

The last time you went to a conference, did you ask any questions? Were you curious about a technology and wanted to know more? Was there something that you didn’t quite get and needed an explanation? Congratulations. You’re in a quiet group of people that ask questions for knowledge. More and more, we are seeing questions becoming a vehicle for more than just knowledge acquisition. If you want to learn how to ask a proper question at a conference, read on.

1. Have A Question

I know it goes without saying, but if you’re going to raise your hand at a conference to ask a question, you should actually have a question in mind. Some people grab a microphone without thinking through what they’re going to say. This leads to stammering and broken thoughts that usually culminate in a random question mark here or there. This makes it difficult for the speaker to figure out what you’re trying to ask.

If you’re going to raise your hand, jot some notes down first. Bullet points help as does making a note or two. This is especially true if the speaker is answering questions before yours. If they answer part of your question before you get to ask it, you may have to reframe your thoughts. It never hurts to have an idea of what you’re going to say before you say it.

2. Look For Knowledge, Not To Make A Statement

The other side of the coin from the above recommendation of actually having a question is to make sure that what you’re asking is actually a question and not a statement. A great example of this is a video from Scott Bradner during a recent ONUG meeting:

I’m sure Scott has seen his fair share of statements masquerading as questions during his time. And I can’t disagree with him. Far too often, people seeking questions aren’t really asking to get information. Instead, they are trying to make a point about why they think they are right or why they disagree with the speaker. The point stops becoming a question and more of a soliloquy or soapbox. The most egregious will usually end this rant with an actual question along the lines of, “So, what do you think of my opinion?”

Please, at all costs, avoid this behavior. This is singularly the most annoying thing a speaker has to deal with. It’s enough to be questioned on your material, but it’s something else entirely to have to shift your thinking to someone else’s viewpoint while on stage. If you have a point you’d like to bring up with the speaker that is contrary to their thought process, you should do it after the presentation without people watching. Have a discussion and express opinions there. Don’t grandstand in front of the crowd just to satisfy your ego.

theres-no-question-youre-clever

3. Make Sure Your Question Wasn’t Already Answered.

This one’s a bit tougher. If you’re sitting in a session and you have a question, it’s important to make sure it wasn’t already asked and answered beforehand. This can be tougher if you have to duck out to take a call or you miss a section of the presentation. In these cases, you can ask for clarification or additional information but it would be better to ask after the session. Audiences tend to get a bit irritated if someone asks a question that was previously answered or that was covered earlier.

This one is probably the most forgivable of the question faux pas. People at conferences know that ducking out to deal with things is more common now. But if you are going to ask a question because you missed something, please make sure to address then when you ask. That helps everyone get the frame of reference for why you’re asking it. That will keep the audience on your side and less likely to boo you.


Tom’s Take

I ask lots of questions. I also answer them. And nothing irritates me more than having to deal with someone making a point during Q&A to try and make them look smarter than me. I get it. I have a hatred of keynotes and other speeches with no ability to get feedback. But at the same time, as Scott Bradner says above, the focus of the presentation is about the people presenting. It’s about the people doing the work and sharing the ideas. If you want to use Q&A time to pontificate about your position, then you need to volunteer to be a speaker.

Designer or Architect? It’s A Matter Of Choice

hearthfire_draftingtable

I had a great time at ONUG this past week. I got to hear a lot of great presentations from some great people, and I got a chance to catch up with some friends as well. One of those was Pete Lumbis (@PeteCCDE) who had a great presentation this past spring at Interop. We talked a lot about tech and networking, but one topic he brought up that made me stop and think for a moment was the wide gulf between design and architecture.

Binary Designers

Design is a critical part of an IT project. Things must fit and make sense before the implementors can figure out how to put the pieces together. Design is all about building a list of products and describing how they’ll interact once turned on. Proper design requires you to step away from the keyboard for a moment and think about a bigger picture than just hacking CLI commands or Python code to make some lights start blinking in the right order.

But design is inherently limited. Think about the last design you did, whether it be wireless or networking or even storage. When you start a design, you automatically make assumptions about what’s going on in the scenario. Perhaps they want to expand their near-line storage capacity. That brings a set of products into play that you choose from. But what if the goal is something different? What if they want a fast caching tier? What if the goal is to create a new pod for object storage?

All of these scenarios are broad enough to require a designer to come up with a good mix of products to fulfill the goals of the project. But the designer has already had assumptions put down for them: The scope and the requirements are pre-determined for them before they ever start thinking about the technology that will be involved in the setup.

Design is all about choices. You have to choose the right product to meet the goals. Once you know the product, you have to make the right choices about which set of products to use? The orange ones or the blue ones? The cheap ones or the expensive ones? Design is about making good choices so implementers can focus on making those choices work.

Visionary Architects

Architecture, on the other hand, has very little to do with choice. Architects are idea people. They look at a problem faced by an organization and try to narrow the focus of the issue to make the designer’s choices easier. Architects don’t worry about individual products or even minor solution sets. They focus on technology areas.

Think back to our storage problem from above. How did the designer arrive at the near-line storage decision? Or the object storage idea? It’s because an architect is the one driving those ideas from a higher level. Architects may not know how to build an object storage bill of materials or how to assemble a chassis switch but they do know what those are used for. Architects instead know that you should be using flash storage in lower density, faster reaction systems when cost is sensitive. They know that a rack may only need a 1U ToR switch instead of a chassis if that ToR switch doesn’t have to provide power or advanced features. They won’t know the specific part number, but they know the technology.

Architects have vision. Designers know products. They need each other to make solutions work and designs happen. The same person can fulfill both roles provided they understand how things break down in the end. A designer architect needs to know that the solutions to customer problems should come before any decisions are made about products. Too often, we find ourselves cornered in a mess because the product mix was decided before the solution was determined.

It’s like trying to bake a cake when all you have in the house is flour, eggs, and swiss cheese. Maybe a cake isn’t what you should be making. The architect would realize that the problem is a limited set of ingredients. instead of deciding on a cake, the architect can work with the designer to find a solution to the problem of food with limited ingredients. Perhaps the designer realizes what’s needed is a soufflé instead. The team figures out the problem with the best design instead of deciding on a design before knowing what the problem is.


Tom’s Take

I was a designer in my past life at a VAR. I still had to implement my designs at the end of the day, but I was the one making the decisions about the products that were needed to meet the solutions my customers had to have. Now, at Tech Field Day I understand the technology at an architecture level. I know why you need this solution for that problem. My ability to hack CLI has gone down a bit but my understanding of the bigger picture has increased several times over that. I now think that I have a better idea of what needs to happen to make tech work the right way and be implemented easier when the architect’s vision can solve the problems that allows the designers to make the right choices.

Thoughts on Theft

ShareArrows

It’s been a busy week for me. In fact, it’s been a busy few weeks. I’ve had lots of time to enjoy NetApp Insight, Cloud Field Day, and Storage Field Day. I’ve also been doing my best to post interesting thoughts and ideas. Whether it’s taking on the CCIE program or keynote speakers, I feel like I owe a debt to the community and my readers to talk about topics that are important to them, or at least should be. Which is why I’m irritated right now about those ideas being stolen.

Beg, Borrow, and Steal

A large part of my current job is finding people that are writing great things and shining a spotlight on them. I like reading interesting ideas. And I like sharing those ideas with people. But when I share those ideas with people, I make absolutely sure that everyone knows where those ideas came from originally. And if I use those ideas for writing my own content, I make special care to point out where they came from and try to provide the context for the original statement in the first place.

What annoys me to no end is when people take ideas as their own and try to use them for their own ends. It’s not all that difficult. You can use weasel words like “sources” or “I heard once” or even “I read this article”. Those are usually good signs that content is going to be appropriated for some purpose. It’s also a sign that research isn’t being done or attributed properly. It’s lazy journalism at best.

What really grinds my gears is when my ideas are specifically taken and used elsewhere without attribution. Luckily, I haven’t had to deal with it much so far. I have a fairly liberal policy about sharing my work. I just want people to recognize the original author. But when my words end up in someone else’s mouth, that’s when the problems start.

Credit Where It Is Due

Taking ideas given freely without offering a clue as to where they come from is theft. Plain and simple. It takes the hard work that someone has put in to thinking through an issue and wraps it up in a cloudy mess. Now, who is to say (beyond dates) who was the originator of the idea? It’s just as easy to say that someone else came up with it. That’s what makes the tracing the origin of things so difficult. Proper attribution for ideas is important in a society where knowledge carries so much weight.

I don’t expect to make millions of dollars from my ideas. I have opinions. I have thoughts. Sometimes people agree with them. Just as often, people disagree. The point is not to be right or wrong or rich. The true point is to make sure that the thoughts and ideas of a person are placed where they belong when the threads are all unwound.

Honestly, I don’t even really want a ton of credit. It does me little good to have someone shouting from the rooftops that I was the first person to talk about something. Or that I was right when everyone else was wrong. But when the butcher’s bill comes due, I’d at least like to have my name attached to my thoughts.


Tom’s Take

I’ve luckily been able to have most of my appropriated content taken down. Some have used it as fuel for a link bait scheme to get paid. Others have used it as a way to build a blog for readership for some strange purpose. Thankfully, I’ve never run into anyone that was vocally taking credit for my writing and passing it off as their own. If you are a smart person and willing to writing things down, do the best you can with what you have. You don’t need to take something else that someone has written and attempt to make it you own. That just tarnishes what you’re trying to do and makes all your writing suspect. Be the best you can be and no one will ever question who you are.

Keystone Keynotes

keystonekeynotepatrol

My distaste for keynotes is well known. With the possible exception of Justin Warren (@JPWarren) there may not be a person that dislikes them more than I do. I’ve outlined my reasons for it before, so I won’t go into much depth about it here. But I do want to highlight a few recent developments that are doing a great job of helping me find new things to dislike.

Drop The “Interviews”

When you walk into a keynote ballroom or arena and see two comfy chairs on stage, you know what’s coming. As someone told me recently, “This is when I know the next hour is going to suck.” The mock interview style of keynote speech is not good. It’s a thinly-veiled attempt to push an agenda. Perhaps it’s about innovation. Or transformation. Or some theme of the conference. Realistically, it’s mostly a chance for a keynote host (some form of VP) to provide forced banter with a celebrity that’s being paid to be there.

These “interviews” are rarely memorable. They seem self serving and very plastic. The only ones that even stand out to me in recent memory are the ones that went off the rails. The time when Elon Musk was “interviewed” on stage at Dell World and responded with clipped answers and no elaboration. Or the time Richard Branson was hitting on the host at Cisco Live. Or the Cisco Live when William Shatner started taking shots at Cisco on stage!

It’s time to drop the fake interviews. Let the speakers tell their stories. Kevin Spacey at Cisco Live 2016 was a breath of fresh air. He was compelling. Invigorating. Engaging. People around me said it was the best keynote they’d heard in year. It was easily the best one I’d see since John Cleese in Orlando in 2008. Give the people who spend their time telling stories a chance to shine. Don’t inject yourself into the process. Because actors and celebrity storytellers don’t play. They live their stories.

All By My Selfie

If the keynote involves talking about community or the power of the user base or some other contrite platitude, you can almost guarantee that the host VP is going to pause at some point, usually during the big celebrity interview, to take selfie with their guest and the whole audience in the background. It’s a nod to how hooked in and in the know with the community. Think back to Ellen Degeneres and her infamous Oscars selfie:

Except it’s not. It’s a big steaming pile of patronizing behavior. Hey everyone that paid $1,500 to hear our transformation strategy! Let me take a picture of myself on a stage with blurry, badly lit faces in the audience! Let me post it to Facegram and Instabook and Snapfilter Stories! Let me have my social team repost it and heart/like/favorite it as many times as it takes for me to look like I “get” social. And after the conference is over, my InstaFaceSnapgrambookfilter feed will go back to auto posting the content fed to it by a team of people trying to make me seem human but not be controversial or get us sued.

Don’t take a selfie with 4,000 people in a hall. Meet those users. The ones that paid you. The ones that run your hardware even though your competitor is knocking on the door every week trying to get them to dump you. The users and customers that are supporting your efforts to cut your nose off to spite your face as you transform yourself into a software company. Or a cloud provider. Or an app company. Don’t pretend that the little people matter in a selfie that needs Super Troopers-style ENHANCE to find my shining freckles in the dark. Be a human and take a selfie with one user that has stuck by you through thick and thin. Make their day. Don’t make yours.

Distrupting Disruption

“We’re like the Uber of….”

No. You aren’t. If you are a part of the market, you aren’t disrupting it. You may be shifting your ideas or repositioning your strategies, but that’s not disruption. You still support your old equipment and software. You’re not prepared to jettison your existing business models to move somewhere new. A networking company building networking software isn’t disruption. A server company buying a networking startup isn’t disruption. It’s strategy.

Uber is the Business School case study for disruption. Every keynote in the last two years has mentioned them. Expect their disruption of the transportation market is far from total or completely impressive. Sure, they are farming out taxi services. They’re also cutting rates to drive business to increase profits without helping drivers with new lower rates. They are bullying municipalities to get laws passed to protect them. They’re driving other companies out of business to reduce competition. Does that sound like the Disruptors of Taxis? Or does is sound like the very cab companies that are getting run out of business by the very tactics they themselves have used?

Don’t tell me how you’re disrupting digital or accelerating change. Tired cliches are tired. Tell me what you’re doing. Tell me how you’re going to head off your competitors. Tell me how you’re addressing a shrinking market for hardware or a growing landscape of people doing it faster, cheaper, and better. This is one of the things that I enjoy about being an analyst. These briefings are generally a little more focused on the concrete and less on the cheerleading, which is a very pleasant surprise to me given my distaste for professional analyst firms.

If you’re tempted to say that you’re the Uber of your industry, do us all a favor and request one to drive you off the stage.


Tom’s Take

Does my dislike of keynotes show yet? Are some you sitting in your chairs cheering? Good. Because it’s all a show for you. It’s a hand-holding, happy hugging reinforcement of how awesome we are. Outside of a few dynamic speakers (who are usually made CTO or VP of Technology), we don’t get the good stuff any more.

If you’re sitting in your chair and getting offended that I’m picking on your event, you should know two things. First, I’m not singling anyone out. EVERY keynote I’ve seen in the last two years is guilty of these things. And if you think yours is, you’re probably right. Fix it. Transform and Disrupt your own keynote. Let story tellers talk. Cut down on the attempts to relate to people. Tell your story. Tell people why they should be excited again. Don’t use cliches. Or funny videos. Or cameraphones. Get back to the business of telling people why you’re in business. Ditch the Keystone Keynotes and I promise you’ll have happier audiences. Including me.

DevOps and the Infrastructure Dumpster Fire

dumpsterfire2

We had a rousing discussion about DevOps at Cloud Field Day this week. The delegates talked about how DevOps was totally a thing and it was the way to go. Being the infrastructure guy, I had to take a bit of umbrage at their conclusions and go on a bit of a crusade myself to defend infrastructure from the predations of developers.

Stable, Boy

DevOps folks want to talk about continuous integration and continuous deployment (CI/CD) all the time. They want the freedom to make changes as needed to increase bandwidth, provision ports, and rearrange things to fit development timelines and such. It’s great that they have they thoughts and feelings about how responsive the network should be to their whims, but the truth of infrastructure today is that it’s on the verge of collapse every day of the week.

Networking is often a “best effort” type of configuration. We monkey around with something until it works, then roll it into production and hope it holds. As we keep building more patches on to of patches or try to implement new features that require something to be disabled or bypassed, that creates a house of cards that is only as strong as the first stiff wind. It’s far too easy to cause a network to fall over because of a change in a routing table or a series of bad decisions that aren’t enough to cause chaos unless done together.

Jason Nash (@TheJasonNash) said that DevOps is great because it means communication. Developers are no longer throwing things over the wall for Operations to deal with. The problem is that the boulder they were historically throwing over in the form of monolithic patches that caused downtime was replaced by the storm of arrows blotting out the sun. Each individual change isn’t enough to cause disaster, but three hundred of them together can cause massive issues.

arrows-blot-out-the-sun-800x500

Networks are rarely stable. Yes, routing tables are mostly stabilized so long as no one starts withdrawing routes. Layer 2 networks are stable only up to a certain size. The more complexity you pile on networks, the more fragile they become. The network really only is one fat-fingered VLAN definition or VTP server mode foul up away from coming down around our ears. That’s not a system that can support massive automation and orchestration. Why?

The Utility of Stupid Networking

The network is a source of pain not because of finicky hardware, but because of applications and their developers. When software is written, we have to make it work. If that means reconfiguring the network to suit the application, so be it. Networking pros have been dealing with crap like this for decades. Want proof?

  1. Applications can’t take to multiple gateways at a time on layer 2 networks. So lets create a protocol to make two gateways operate as one with a fake MAC address to answer requests and ensure uptime. That’s how we got HSRP.
  2. Applications can’t survive having the IP address of the server changed. Instead of using so many other good ideas, we create vMotion to allow us to keep a server on the same layer 2 network and change the MAC <-> IP binding. vMotion and the layer 2 DCI issues that it has caused has kept networking in the dark for the last eight years.
  3. Applications that run don’t need to be rewritten to work in the cloud. People want to port them as-is to save money. So cloud networking configurations are a nightmare because we have to support protocols that shouldn’t even be used for the sake of legacy application support.

This list could go on, but all these examples point to one truth: The application developers have relied upon the network to solve their problems for years. So the network is unstable because it’s being extended beyond the use case. Newer applications, like Netflix and Facebook, and thrive in the cloud because they were written from the ground up to avoid using layer 2 DCI or even operate at layer 2 beyond the minimum amount necessary. They solve tricky problems like multi host combinations and failover in the app instead of relying on protocols from the golden age of networking to fix it quietly behind the scenes.

The network needs to evolve past being a science project for protocols that aim to fix stupid application programming decisions. Instead, the network needs to evolve with an eye toward stability and reduced functionality to get there. Take away the ability to even try to do those stupid tricks and what you’re left with is a utility that is a resource for your developers. They can use it for transport without worrying about it crashing every day with some bug in a protocol no one has used in five years yet was still installed just in case someone turned on an old server accidentally.

Nowhere is this more apparent than cloud networking stacks like AWS or Microsoft Azure. There, the networking is as simplistic as possible. The burden for advanced functionality per group of users isn’t pushed into a realm where network admins need to risk outages to fix a busted application. Instead, the app developers can use the networking resources in a basic way to encourage them to think about failover and resiliency in a new way. It’s a brave new world!


Tom’s Take

I’ll admit that DevOps has potential. It gets the teams talking and helps analyze build processes and create more agile applications. But in order for DevOps to work the way it should, it’s going to need a stable platform to launch from. That means networking has to get its act together and remove the unnecessary things that can cause bad interactions. This was caused in part by application developers taking the easy road and pushing against the networking team of wizards. When those wizards push back and offer reduced capabilities countered against more uptime and fewer issues you should start to see app developers coming around to work with the infrastructure teams to get things done. And that is the best way to avoid an embarrassing situation that involves fire.

Cloud Apps And Pathways

jam

Applications are king. Forget all the things you do to ensure proper routing in your data center. Forget the tweaks for OSPF sub-second failover or BGP optimal path selection. None of it matters to your users. If their login to Seibel or Salesforce or Netflix is slow today, you’ve failed. They are very vocal when it comes to telling you how much the network sucks today. How do we fix this?

Pathways Aren’t Perfect

The first problem is the cloud focus of applications. Once our packets leave our border routers it’s a giant game of chance as to how things are going to work next. The routing protocol games that govern the Internet are tried and true and straight out of RFC 1771(Yes, RFC 4271 supersedes it). BGP is a great tool with general purpose abilities. It’s becoming the choice for web scale applications like LinkedIn and Facebook. But it’s problematic for Internet routing. It scales well but doesn’t have the ability to make rapid decisions.

The stability of BGP is also the reason why it doesn’t react well to changes. In the old days, links could go up and down quickly. BGP was designed to avoid issues with link flaps. But today’s links are less likely to flap and more likely to need traffic moved around because of congestion or other factors. The pace that applications need to move traffic flows means that they tend to fight BGP instead of being relieved that it’s not slinging their traffic across different links.

BGP can be a good suggestion of path variables. That’s how Facebook uses it for global routing. But decisions need to be made on top of BGP much faster. That’s why cloud providers don’t rely on it beyond basic connectivity. Things like load balancers and other devices make up for this as best they can, but they are also points of failure in the network and have scalability limitations. So what can we do? How can we build something that can figure out how to make applications run better without the need to replace the entire routing infrastructure of the Internet?

GPS For Routing

One of the things that has some potential for fixing inefficiency with BGP and other basic routing protocols was highlighted during Networking Field Day 12 during the presentation from Teridion. They have a method for creating more efficiency between endpoints thanks to their agents. Founder Elad Rave explains more here:

I like the idea of getting “traffic conditions” from endpoints to avoid congestion. For users of cloud applications, those conditions are largely unknown. Even multipath routing confuses tried-and-true troubleshooting like traceroute. What needs to happen is a way to collect the data for congestion and other inputs and make faster decisions that aren’t beholden to the underlying routing structure.

Overlay networking has tried to do this for a while now. Build something that can take more than basic input and make decisions on that data. But overlays have issues with scaling, especially past the boundary of the enterprise network. Teridion has potential to help influence routing decisions in networks outside your control. Sadly, even the fastest enterprise network in the world is only as fast as an overloaded link between two level 3 interconnects on the way to a cloud application.

Teridion has the right idea here. Alternate pathways need to be identified and utilized. But that data needs to be evaluated and updated regularly. Much like the issues with Waze dumping traffic into residential neighborhoods when major arteries get congested, traffic monitors could cause overloads on alternate links if shifts happen unexpectedly.

The other reason why I like Teridion is because they are doing things without hardware boxes or the need to install software anywhere but the end host. Anyone working with cloud-based applications knows that the provider is very unlikely to provide anything outside of their standard offerings for you. And even if they manage, there is going to be a huge price tag. More often than not, that feature request will become a selling point for a new service in time that may be of marginal benefit until everyone starts using it. Then application performance goes down again. Since Teridion is optimizing communications between hosts it’s a win for everyone.


Tom’s Take

I think Teridion is on to something here. Crowdsourcing is the best way to gather information about traffic. Giving packets a better destination with shorter travel times means better application performance. Better performance means happier users. Happier users means more time spent solving other problems that have symptoms that aren’t “It’s slow” or “Your network sucks”. And that makes everyone happier. Even grumpy old network engineers.

Disclaimer

Teridion was a presenter during Networking Field Day 12 in San Francisco, CA. As a participant in Networking Field Day 12, my travel and lodging expenses were covered by Tech Field Day for the duration of the event. Teridion did not ask for nor where they promised any kind of consideration in the writing of this post. My conclusions here represent my thoughts and opinions about them and are mine and mine alone.