Category Archives: Voice

The Tyranny of Technical Debt, Numerically

Posted on by
2

A Candlestick Phone (image courtesy of WIkipedia)

This week on the Gestalt IT Rundown, I talked about the plan by Let’s Encrypt to reuse some reserved IP address space. I’ve talked about this before and I said it was a bad idea then for a lot of reasons, mostly related to the fact that modern operating systems are coded not to allow 240/4 as a valid address space, for example. Yes, I realize that when the address space was codified back in the early days of the Internet that decisions were made to organize things and we “lost” a lot of addresses for experimental reasons. However, this is not the only time this has happened. Nor is it the largest example. For that, we need to talk about the device that you’re very likely reading this post on right now: your phone.

By the Numbers

We’re going to be referring to the North American Numbering Plan (NANP) in this post, so my non-US readers are going to want to click that link to understand how phone numbering works in the US. The NANP was devised back in the 1940s by AT&T as a way to assign numbers to the telephone exchanges so they were easy to contact. The first big decision that was made was to disallow a 1 or a 0 digit at the start of the prefix code. On a pulse dialing system that sends electrical pulses instead of the tones we associate with DTMF the error rate for having a 1 as the first digit was pretty high. It was generally ignored by the equipment. A 0 was used as the signal to a switchboard operator to get assistance. So the decision was made to restrict the 1 and the 0 for technical reasons. That’s the first technical debt choice that was made.

Next, the decision was made to make it easier for the CO switchboard operators to memorize numbers using mnemonics. That meant assigning letters to the numbers on the phone rotary dial (later keypad). Every number got three letters, leaving out Q and Z because they’re weird. Every number, that is, except 1 and 0. Because they weren’t used as the start of a prefix code they didn’t get letters assigned as noted above. Every telephone exchange was named based on the numerical prefix, which could have a 0 or a 1 in the first two digits. For example, the exchanges that started with 47x would be named using a letter from the 4 followed by a digit from the 7. Those words started with GR, such as “GRanite”, GReenwood”, and so on. it was handy to remember those as long as the telephone system didn’t get too big. If you ever watch an old movie from the 50s and hear someone asking to talk to a person whose number is KLondike5-6000 that’s why. We’ll come back to KLondike in a minute. The use of words instead of number was slowly phased out starting in 1958 because it was just easy for people to use nothing but numbers.

So what about dialing outside of your local exchange? Well, for that you need an area code right? But that area code looks just like a prefix code. How can you tell someone that you want to dial a different state? Well, AT&T came up with an interesting rule there. Remember how no prefix code started with a 0 or a 1? AT&T said that area codes MUST have a 0 or a 1 as the middle number. That organization allowed for the switchboard operator or signaling equipment to know that the first three digit code was an area code, followed by seven digits for the prefix and handset number. Simple, right? Combined with using the 1 digit to signal a long-distance interchange call you had a system that worked well for a number of years. Almost fifty, in fact, from 1947 all the way to 1995.

What happened in 1995? The telephone administrators realized they were going to run out of room given the explosion of phone numbers. The phone companies, now more than just AT&T, realized all the way back in the 1960s that growth would eventually lead to them needing to do away with the rules about restricting the middle digit of an area code to a 1 or a 0. That’s why area codes created after 1995 have middle digits in the 2-9 range. Thanks to things like mobile phones we doubled or tripled the amount of phone numbers we were going to need. Which meant tearing up all those careful plans about how to use the numbers that were created back in the 1940s to solve technical challenges.

To The Klondike

What about that KLondike number I mentioned earlier? Well, KL is 55 on the dial/keypad, so a KLondike5 number actually starts with 555. Most movie buffs will tell you that any number that has a 555 in the prefix code is a fake number since that’s what you hear in movies all the time. Originally these number were assigned to local exchanges only and used for testing purposes, except for 555-1212 which is a universal directory assistance number in every area code. In 1994, people realized that there were a huge amount of numbers that could be added to the NANP pool by reclaiming those test numbers.

However, if someone gave you a 555 number in a business setting or at a bar or club, how likely would you be to say that it was a fake or invalid number? Likely pretty high given the preponderance of usage in film. In fact, 555-0100 through 555-0199 were still set aside for “fake” number or entertainment purposes, not unlike 192.0.2.0/24 being reserved for documentation purposes.

The 555-XXXX number range is the perfect example of why a plan like Let’s Encrypt and their suggestion to reclaim the space is going to ultimately fail. You’re going to have to do a significant amount of programming to get every operating system to not immediately reject 240/4 as invalid or not immediately assume 127/8 is a loopback address. Because those decisions were made long ago, relatively speaking, and are going to take time and effort to undo. Remember that AT&T knew they were going to need to change area code rules back in the 1960s and it still took 30 years to put it into practice.

Moreover, the plans by Let’s Encrypt and others seeking to implement the Schoen draft in the IETF are ignoring a very simple truth. If we’re going to spend all this time and energy rewriting half the networking stacks in use today, why aren’t we using that energy to implement IPv6 support universally instead? It’s already a requirement if you interface with the US federal government. Why are we going to spend so much time fixing a problem we know is broken and not scalable in the future? Is it because we’re holding on to some idea from the past the IP addresses should be four octets? Is it because we want to exhaust every possible resource before being dragged into the IPv6 future? Or is it because after all these years we don’t want to admit that a better solution exists and we’re just ready to move to the next version of IP after IPv6 and we don’t want to say it out loud?

Tom’s Take

Technical debt is the result of decisions we made at the time to do what needed to be done. It’s not malicious or even petty. It’s usually what had to happen and now we have to live with it. Instead of looking at technical debt as a crushing weight we need to decide how to best fix it when necessary. Like the NANP changes we can modify things to make them work better. However, we also need to examine how much work we’re doing to perpetuate something that needs to be rewritten and why we choose do it the way we do. Phone numbers are something that are going to persist for along time. IP addresses aren’t. Let’s fix the problem the right way instead of the comfortable way.

Tale From The Trenches: The Debug Of Damocles

Posted on by
Reply

My good friend and colleague Rich Stroffolino (@MrAnthropology) is collecting Tales from the Trenches about times when we did things that we didn’t expect to cause problems. I wanted to share one of my own here about the time I knocked a school offline with a debug command.

I Got Your Number

The setup for this is pretty simple. I was deploying a CallManager setup for a multi-site school system deployment. I was using local gateways at every site to hook up fax lines and fire alarms with FXS/FXO ports for those systems to dial out. Everything else got backhauled to a voice gateway at the high school with a PRI running MGCP.

I was trying to figure out why the station IDs that were being send by the sites weren’t going out over caller ID. Everything was showing up as the high school number. I needed to figure out what was being sent. I was at the middle school location across town and trying to debug via telnet. I logged into the router and figured I would make a change, dial my cell phone from the VoIP phone next to me, and see what happened. Simple troubleshooting, right?

I did just that. My cell phone got the wrong caller ID. And my debug command didn’t show me anything. So I figured I must be debugging the wrong thing. Not debug isdn q931 after all. Maybe it’s a problem with MGCP. I’ll check that. But I just need to make sure I’m getting everything. I’ll just debug it all.

debug mgcp packet detail

Can You Hear Me Now?

Veterans of voice are probably screaming at me right now. For those who don’t know, debug anything detail generates a ton of messages. And I’m not consoled into the router. I’m remote. And I didn’t realize how big of a problem that was until my console started scrolling at 100 miles an hour. And then froze.

Turns out, when you overwhelm a router CPU with debug messages, it shuts off the telnet window. It also shuts off the console as well, but I wouldn’t have known that because I was way far way from that port. But I did starting hearing people down the hall saying, “Hello? Hey, are you still there? Weird, it just went dead.”

Guess what else a router isn’t doing when it’s processing a tidal wave of debug messages? It’s not processing calls. At all. For five school sites. I looked down at my watch. It was 2:00pm. That’s bad. Elementary schools get a ton of phone calls within the last hour of being in session. Parents calling to tell kids to wait in a pickup line or ride a certain bus home. Parents wanting to check kids out early. All kinds of things. That need phones.

I raced out of my back room. I acknowledged the receptionists comment about the phones not working. I jumped in my car and raced across town to the high school. I managed not to break any speed limits, but I also didn’t loiter one bit. I jumped out of my car and raced into the building. The look on my face must have warded off any comments about phone system issues because no one stopped me before I got to the physical location of the voice gateway.

I knew things were bad. I didn’t have time to console in and remove the debug command. I did what ever good CCIE has been taught since the beginning of time when they need to remove a bad configuration that broke their entire lab.

I pulled the power cable and cycled the whole thing.

I was already neck deep in it. It would have taken me at least five minutes to get my laptop ready and consoled in. In hindsight, that would have been five wasted minutes since the MGCP debugger would have locked out the console anyway. As the router was coming back up, I nervously looked at the terminal screen for a login prompt. Now that the debugger wasn’t running, everything looked normal. I waiting impatiently for the MGCP process to register with CallManager once more.

I kept repeating the same status CLI command while I refreshed the gateway page in CallManager over and over. After a few more tense minutes, everything was back to normal. I picked up a phone next to the rack and dialed my cell phone. It rang. I was happy. I walked back to the main high school office and told them that everything was back to normal.

Tom’s Take

My post-mortem was simple. I did dumb things. I shouldn’t have debugged remotely. I shouldn’t have used the detail keyword for something so simple. In fact, watching my screen fill up with five sites worth of phone calls in a fraction of a second told me there was too much going on behind the scenes for me to comprehend anyway.

That was the last time I ever debugged anything in detail. I made sure from that point forward to start out small and then build from there to find my answers. I also made sure that I did all my debugging from the console and not a remote access window. And the next couple of times I did it were always outside of production hours with a hand ready to yank the power cable just in case.

I didn’t save the day. At best, all I did was cover for my mistake. If it had been a support call center or a hospital I probably would have been fired for it. I made a bad decision and managed to get back to operational without costing money or safety.

Remember when you’re doing your job that you need to keep an eye on how your job will affect everything else. We don’t troubleshoot in a vacuum after all.

The Voice of SD-WAN

Posted on by
Reply

SD-WAN is about migrating your legacy hardware away from silos like MPLS and policy-based routing and instead integrating everything under one dashboard and one central location to make changes and see the impacts that those changes have. But there’s one thing that SD-WAN can’t really do yet. And that’s prepare us the for the end of TDM voice.

Can You Hear Me Now?

Voice is a way of life for some people. Cisco spent years upon years selling CallManager into every office they could. From small two-line shops to global organizations with multiple PRIs and TEHO configured everywhere. It was a Cisco staple for years. Which also had Avaya following along quickly to get into the act too.

Today’s voice world is a little less clear. Millenials hate talking on the phone. Video is an oddity when it comes to communications. Asynchronous chat programs like WhatsApp or Slack rule the day today. People would rather communicate via text than voice. We all have mobile devices and the phone may be one of the least used apps on it.

Where does that leave traditional voice services? Not in a good place for sure. We still need phone lines for service-focused businesses or when we need to call a hotline for support. But the office phone system isn’t getting any new features anytime soon. The phone system is like the fax machine in the corner. It’s a feature complete system that is used when it has to be used by people that are forced to use it unhappily.

Voice systems are going to stay where they are by virtue of their ubiquity. They exist because TDM technology hasn’t really advanced in the past 20 years. We still have twisted pair connections to deliver FXO lines. We still have the most basic system in place to offer services to our potential customers and users. I know this personally because when I finally traded out my home phone setup for a “VoIP” offering from my cable provider, it was really just an FXS port on the back of a residential cable modem. That’s as high-tech as it gets. TDM is a solved problem.

Call If You WANt To

So, how does SD-WAN play into this? Well, as it turns out, SD-WAN is replacing the edge router very quickly. Devices that used to be Cisco ISRs are now becoming SD-WAN edge devices. They aggregate WAN connections and balance between them. They take MPLS and broadband and LTE instead of serial and other long-forgotten connection methods.

But you know what SD-WAN appliances can’t aggregate? TDM lines. They don’t have cards that can accept FXO, FXS, or even PRI lines. They don’t have a way to provide for DSP add-in cards or even come with onboard transcoding resources. There is no way for an SD-WAN edge appliance to function as anything other than a very advanced packet router.

This is a good thing for SD-WAN companies. It means that they have a focused, purpose built device that has more software features than hardware muscle. SD-WAN should be all about data packets. It’s not a multitool box. Even the SD-WAN vendors that ship their appliances with LTE cards aren’t trying to turn them into voice routers. They’re just easing the transition for people that want LTE backup for data paths.

Voice devices were moved out of the TDM station and shelf and into data routers as Cisco and other companies tried to champion voice over IP. We’re seeing the fallout from those decisions today. As the data routing devices become more specialized and focused on the software aspects of the technology, the hardware pieces that the ISR platform specialized in are now becoming a yoke holding the platform back. Now, those devices are causing those platforms to fail to evolve.

I can remember when I was first thinking about studying for my CCIE Voice lab back in 2007-2008. At the time, the voice lab still have a Catalyst 6500 switch running in it that needed to be configured. It had a single T1 interface on a line card that you had to get up and running in CallManager. The catch? That line card would only work with a certain Supervisor engine that only ran CatOS. So, you have to be intimately familiar with CatOS in order to run that lab. I decided that it wasn’t for me right then and there.

Hardware can hold the software back. ISRs can’t operate voice interfaces in SD-WAN mode. You can’t get all the advanced features of the software until you pare the hardware down to the bare minimum needed to route data packets. If you need to have the router function as a TDM aggregator or an SBC/IPIPGW you realize that the router really should be dedicated to that purpose. Because it’s functioning more as a TDM platform than a packet router at that point.

Tom’s Take

The world of voice that I lived in five or six years ago is gone. It’s been replaced with texting and Slack/Spark/WebEx Teams. Voice is dying. Cell phones connect us more than we’ve ever been before but yet we don’t want to talk to each other. That means that the rows and rows of desk phones we used to use are falling by the wayside. And so too are the routers that used to power them. Now, we’re replacing those routers with SD-WAN devices. And when the time finally comes for use to replace those TDM devices, what will we use? That future is very murky indeed.

Slacking Off

Posted on by
1
A Candlestick Phone (image courtesy of WIkipedia)

A Candlestick Phone (image courtesy of WIkipedia)

There’s a great piece today on how Slack is causing disruption in people’s work habits. Slack is a program that has dedicated itself to getting rid of email, yet we now find ourselves mired in Slack team after Slack team. I believe the real issue isn’t with Slack but instead with the way that our brains are wired to handle communication.

Interrupt Driven

People get interrupted all the time. It’s a fact of life if you work in business, not just IT. Even if you have your head down typing away at a keyboard and you’ve closed out all other forms of distraction, a pop up from an email or a ringing or vibrating phone will jar your concentration out of the groove and force your brain to deal with this new intruder into your solitude.

That’s evolution working against you. When we were hunters and gatherers our brain had to learn how to deal with external threats when we were focused on a task like stalking a mammoth or looking for sprouts on the forest floor. Our eyes are even developed to take advantage of this. Your peripheral vision will pick up movement first, followed by color, then finally it can discern the shape of an object. So when your email notifier slides out from the system tray or notification window it triggers your primitive need to address the situation.

In the modern world we don’t hunt mammoths or forage for shoots any longer. Instead, our survival instinct has been replaced by the need to answer communications as fast as possible. At first it was returning phone calls before the end of the day. Then it became answering emails expediently. That changed into sending an immediate email response that you had seen the email and were working on a response. Then came instant messaging for corporate environments and the idea of “presence”, which allows everyone to know what you’re doing and when you’re doing it. Which has led us to ever presence – the idea that we’re never really not available.

Think about the last time you saw someone was marked unavailable in a chat window and you sent the message anyway. Perhaps you thought they would see the message the next time they logged in or returned to their terminal. Or perhaps you guessed that they had set their status as away to avoid distraction. Either way, the first thought you had was that this person wasn’t really gone and was available.

Instant messaging programs like Slack bridge the gap because synchronous communications channels like phone calls and asynchronous channels like email. In the past, we could deal with phone calls because it required the attention of both parties involved. A single channel was opened and it was very apparent that you were holding a conversation, at least until the invention of call waiting. On the other hand, email is asynchronous by nature because we can put all of our thoughts down in a single message over the course of minutes or even hours and send it into the void. Reliable delivery ensures that it will make it to the destination but we don’t know when it will be read. We don’t know when the response will come or in what form. The receiving party may not even read your message!

The Need to Please

Think back to the last time you responded to an email. How often did you start your response with “Sorry for the delay” or some version of that phrase? In today’s society, we’ve become accustomed to instant responses to things. Amy Lewis (@CommsNinja) is famous for having an escalation process for reaching her:

  1. Text message
  2. Twitter DM
  3. Email
  4. Phone Call
  5. Anything else
  6. Voice mail

She prefers instant communication and rapid response. In a lot of cases, this is very crucial. If you need an answer to a question quickly there are ways to reach people for immediate reply. But the desire to have immediate response for all forms of communication is a bit much.

Our brains don’t help us in this matter. When we get an email or a communication from someone, we feel compelled to respond to it. It’s like a checkbox that needs to be checked. And so we will drop everything else to work on a reply even if it means we’re displeasing someone for a short time to please someone immediately.

Many of the time management systems that have been created to deal with massive email flows, such as GTD are centered on the idea of dealing with things as the come in and pigeonholing them until they can be dealt with appropriately. By treating everything the same you disappoint everyone equally until everything can evaluated. There are cutouts for high priority communications, but the methods themselves tell you to keep those exceptions small and rare so as not to disrupt the flow of things.

The key to having coherent and meaningful conversations with other people is the same online as it is in person. Rather than speaking before you think, you should take the time to consider your thoughts and respond with appropriately measured words. It’s easier to do this via email since there is built-in delay but it works just the same in instant message conversations as well. An extra minute of thought won’t make someone angry with you, but not taking that extra minute could make someone very cross with you down the road.

Tom’s Take

I agree with people that say Slack is great for small teams spread everywhere to help categorize thoughts and keep projects on track. It takes away the need for a lot of status update emails and digests of communications. It won’t entirely replace email for communications and it shouldn’t be seen that way. Instead, the important thing to realize about programs like Slack is that they will start pushing your response style more toward quick replies with little information. You will need to make a conscious decision to push back a bit to make measured responses to things with more information and less response for the sake of responding. When you do you’ll find that instant messaging tools augment your communications instead of complicating them.

My Thoughts On The Death Of IP Telephony

Posted on by
2
A Candlestick Phone (image courtesy of WIkipedia)

A Candlestick Phone (image courtesy of Wikipedia)

Greg Ferro (@EtherealMind) posted a thought provoking article about collaboration in his Human Infrastructure magazine (which you should be reading). He talks about the death of IP Telephony and the rise of asynchronous communications methods like Slack. He’s got a very interesting point of view. I just happen to disagree with a few of his assertions.

IP Telephony Is Only Mostly Dead

Greg’s stance that IP Telephony is dead is a bit pointed to say the least. He is correct that the market isn’t growing. It is also true that a great number of new workers entering the workforce prefer to use their smartphones for communications, especially the asynchronous kind. However, desk phones are a huge part of corporate communications going forward.

IT shops have a stilted and bizarre world view. If you have a workforce that has to be mobile, whether it be for making service calls or going to customer sites for visits, you have a disproportionately large number of mobile users for sure. But what about organizations that don’t have large mobile populations? What about financial firms or law offices or hospitals? What about retail organizations? These businesses have specific needs for communications, especially with external customers and users.

Imagine if your pharmacy replaced their phone with a chat system? How about your doctor’s office throwing out their PBX and going to an email-only system? How would you feel about that? A couple of you might cheer because they finally “get it”. But a large number of people, especially more traditional non-technical folks, would switch providers or move their business elsewhere. That’s because some organizations rely on voice communications. For every millennial dumping their office phone to use a mobile device there is still someone they need to call on the other end.

We’re not even talking about the important infrastructure that still needs a lot of specialized communications gear. Fax machines are still a huge part of healthcare and legal work. Interactive Voice Response (IVR) systems are still crucial to handle call volumes for things like support lines. These functions can’t be replaced on mobile devices easily. You can fake IVRs and call queuing with the right setup, but faxing things to a mobile device isn’t possible.

Yes, services do exist to capture fax information as a TIFF or JPG and email it to the destination. But for healthcare and legal, this breaks confidentiality clauses and other important legal structures. The area around secure faxing via email is still a bit murky, and most of the searches you can do for the topic revolve around companies trying to tell you that it’s acceptable and okay to use (as long as you use their product).

IP Telephony isn’t far removed from buggy whip manufacturers. The oft-cited example of a cottage industry has relevance here. At some point after the introduction of the automobile, buggy whip growth slowed and eventually halted. But they didn’t go away permanently. The market did contract and still exists to this day. It’s not as big as the 13,000-strong market it once was, but it does exist today to meet a need that people still have. Likewise, IP Telephony will still have solutions to meet needs for specific customers. Perhaps we’ll contract down to two or three providers at some point in the future, but it will never really go away.

I’ll Have My People IM Your People

Contacting people is an exercise today. There are so many ways to reach someone via various communications that you have to spend time figuring out how to reach them. Direct message, Text message, Phone call, Voice Mail, Email, and smoke signals are all valid communications forms. It is true that a lot of these communications are moving toward asynchronous methods. But as mentioned above, a lot of customer-facing businesses are still voice-only.

Sales is one of these areas that is driven by sound. The best way to sell something to someone is to do it face-to-face. But a phone call is a close second. Sales works because you can use your voice to influence buyers. You can share their elation and allay their fears. You can be soothing or exciting or any range of emotion in between. That’s something you don’t get through the cold text of instant messaging or email.

It’s also much harder to ignore phone calls. Sure, you can send read receipts with emails but these are rarely implemented and even more rarely used correctly in my experience. Phone calls alert people about intent. Even ignoring or delaying them means being send to a voice mail box or to another phone in the department where your call can be dealt with. The synchronous nature of the communication means there has to be a connection with someone. You can’t just launch bytes of text into the ether and hope it gets where it’s supposed to go.

It is true that these voice communications happen via mobile numbers more often than not in this day and age. But corporations still prefer those calls to go through some kind of enterprise voice system. Those systems can track communications and be audited. Records can be subpoenaed for legal reasons without needing to involve carriers.

It’s much easier for call centers to track productivity via phone logs than seeing who is on the phone. If you’ve ever worked in a corporate call center, you know there are metrics for everything you do on the phone. Average call time, average wait time, amount of non-call time, and so on. Each of these metrics can be tracked via a desk phone with a headset, not so with a mobile phone and an app.

Tom’s Take

I live on my mobile phone. I send emails and social media updates. I talk in Slack and Skype and other instant messaging platforms. But I still get on the phone at least three times a week to talk to someone. Most of those calls take place on a conference bridge. That’s because people want to hear someone’s voice. It’s still comforting and important to listen to someone.

Doing away with IP Telephony sounds like an interesting strategy for small businesses and startups. It’s a cost-reduction method that has benefits in the short term. But as companies grow and change they will soon find that having a centralized voice system to control and manipulate calls is a necessity. Given the changes in voice technology in the last few years, I highly expect that “centralized” voice will eventually be a pay-per-seat cloud leased model with specific executives and support personnel using traditional phones while non-critical employees have no voice communications device or choose to use their personal mobile device.

IP Telephony isn’t dead. It’s not even dying. But it’s well past the age where it needs to consider retirement and a long and fulfilling life concentrating on specific people instead of trying to make everyone happy.

 

 

Gathering No MOS

Posted on by
Reply

mossBall1

If you work in the voice or video world, you’ve undoubtedly heard about Mean Opinion Scores (MOS). MOS is a rough way of ranking the quality of the sound on a call. It’s widely used to determine the over experience for the user on the other end of the phone. MOS represents something important in the grand scheme of communications. However, MOS is quickly becoming a crutch that needs some explanation.

That’s Just Like Your Opinion

The first think to keep in mind when you look at MOS data is that the second word in the term is opinion. Originally, MOS was derived by having selected people listen to calls and rank them on a scale of 1 (I can’t hear you) to 5 (We’re sitting next to each other). The idea was to see if listeners could distinguish when certain aspects of the call were changed, such as pathing or exchange equipment. It was an all-or-nothing ranking. Good calls got a 4 or even rarely a 5. Most terrible calls got 2 or 3. You take the average of all your subjects and that gives your the overall MOS for your system.

voip-qualitypbx

When digital systems came along, MOS took on an entirely different meaning. Rather than being used to subjectively rank call quality, MOS became a yardstick for tweaking the codec used to digitally transform analog speech to digital packets. Since this has to happen in order for the data to be sent, all digital calls must have a codec somewhere. The first codecs were trying to approximate the quality of a long distance phone call, which was the gold standard for quality. After that target was reached, providers started messing around the codecs in question to reduce bandwidth usage.

G.711 is considered the baseline level of call quality from which all others are measure. It has a relative MOS of 4.1, which means very good voice quality. It also uses around 64 kbps of bandwidth. As developers started playing with encoding schemes and other factors, they started developing codecs which used significantly less bandwidth and had almost equivalent quality. G.729 uses only 8 kbps of bandwidth but has a MOS of 3.9. It’s almost as good as G.711 in most cases but uses an eighth of the resources.

MOS has always been subjective. That was until VoIP system providers found that certain network metrics have an impact on the quality of a call. Things like packet loss, delay, and jitter all have negative impacts on call quality. By measuring these values a system could give an approximation of MOS for an admin without needing to go through the hassle of making people actually listen to the calls. That data could then be provided through analytics dashboards as an input into the overall health of the system.

Like A Rolling Stone

The problem with MOS is that it has always been subjective. Two identical calls may have different MOS scores based on the listener. Two radically different codecs could have similar MOS scores because of simple factors like tonality or speech isolation. Using a subjective ranking matrix to display empirical data is unwieldy at best. The only reason to use MOS as a yardstick is because everyone understands what MOS is.

Enter R-values. R-values take inputs from the same monitoring systems that produce MOS and rank those inputs on a scale of 1 – 100. Those scores can then be ranked with more precision to determine call quality and VoIP network health. A call in the 90s is a great call. If things dip in the 70s or the 60s, there are major issues to identify. R-values solve the problem of trying to bolt empirical data onto a subjective system.

Now that communications is becoming more and more focused on things like video, the need for analytics around them is becoming more pronounced. People want to track the same kinds of metrics – codec quality, packet loss, delay, and jitter. But there isn’t a unified score that can be presented in green, yellow, and red to let people know when things are hitting the fan.

It has been suggested that MOS be adapted to reference video in addition to audio. While the idea behind using a traditional yardstick like MOS sounds good on the surface, the reality is that video is a much more complicated thing that can’t be encompassed by a 50-year-old ranking method like MOS.

Video calls can look horrible and sound great. They can have horrible sound and be crystal clear from a picture perspective. There are many, many subjective pieces that can go into ranking a video call. Trying to shoehorn that into a simple scale of 5 values is doing a real disservice to video codec manufacturers, not to mention the network operators that try and keep things running smoothly for video users.

R-value seems to be a much better way to classify analytics for video. It’s much more nuanced and capable of offering insight into different aspects of call and picture quality. It can still provide a ranked score for threshold measuring, but that rank is much more likely to mean something important for each number as opposed to the absolute values present in MOS.

Tom’s Take

MOS is an old fashioned idea that tries valiantly to tie the telecom of old to the digital age. People who understood subjective quality tried to pair it with objective analytics in an effort to keep the old world and the new world matched. But even communications is starting to eclipse these bounds. Phone calls have given way to email, texting, and video chats. Two of those are asynchronous and require no network reliability beyond up or down. Video, and all the other real-time digital communications, needs to have the right metrics and analytics to provide good feedback about how to improve the experience for users. And whatever we end up calling that composite metric or ranked algorithmic score, it shouldn’t be called MOS. Time to let that term grow some moss in the retirement bin.

 

Fixing E-Rate – SIP

Posted on by
5
Embed from Getty Images

I was talking to my friend Joshua Williams (@JSW_EdTech) about our favorite discussion topic: E-Rate.  I’ve written about E-Rate’s slow death and how it needs to be modernized.  One of the things that Joshua mentioned to me is a recent speech from Commissioner Ajit Pai in front of the FCC.  The short, short version of this speech is that the esteemed commissioner doesn’t want to increase the pool of money paid from the Universal Service Fund (USF) into E-Rate.  Instead, he wants to do away with “wasteful” services like wireline telephones and web hosting.  Naturally, when I read this my reaction was a bit pointed.

Commissioner Pai has his heart in the right place.  His staff gave him some very good notes about his interviews with school officials.  But he’s missed the boat completely about the “waste” in the program and how to address it.  His idea of reforming the program won’t come close to fixing the problems inherent in the system.

Voices Carry

Let’s look at the phone portion for moment.  Commissioner Pai says that E-Rate spends $600 million per year on funding wireline telephone services.  That is a pretty big number.  He says that the money we sink into phone services should go to broadband connections instead.  Because the problems in schools aren’t decaying phone systems or lack of wireless or even old architecture.  It’s faster Internet.  Never mind that broadband circuits are part of the always-funded Priority One pool of money.  Or that getting the equipment required to turn up the circuit is part of Priority Two.  No, the way to fix the problem is to stop paying for phones.

Commissioner Pai obviously emails and texts the principals and receptionists at his children’s schools.  He must have instant messaging communications with them regularly. Who in their right mind would call a school?  Oh, right.  Think of all the reasons that you might want to call a school.  My child forget their sweater.  I’m picking them up early for a doctor’s appointment.  The list is virtually endless.  There are so many reasons to call a school.  Telling the school that you’re no longer paying for phone service is likely to get your yelled at.  Or run out of town on a rail.

What about newer phone technologies?  Services that might work better with those fast broadband connections that Commissioner Pai is suggesting are sorely needed?  What about SIP trunking?  It seems like a no-brainer to me.  Take some of the voice service money and earmark it for new broadband connections.  However, it can only be used for a faster broadband connection if the telephone service is converted to a SIP trunk.  That’s a brilliant idea that would redirect the funding where it’s needed.

Sure, it’s likely going to require an upgrade of phone gear to support SIP and VoIP in general.  Yes, some rural phone companies are going to be forced to upgrade their circuits to support SIP.  But given that the major telecom companies have already petitioned the FCC to do away with wireline copper services in favor of VoIP, it seems that the phone companies would be on board with this.  It fixes many of the problems while still preserving the need for voice communications to the schools.

This is a win for the E-Rate integrators that are being targeted by Commissioner Pai’s statement that it’s too difficult to fill out E-Rate paperwork.  Those same integrators will be needed to take legacy phone systems and drag them kicking and screaming into the modern era.  This kind of expertise is what E-Rate should be paying for.  It’s the kind of specialized knowledge that school IT departments shouldn’t need to have on staff.

Tom’s Take

I spent a large part of my career implementing voice systems for education.  Many times I wondered why we would hook up a state-of-the-art CallManager to a cluster of analog voice lines.  The answer was almost always about money.  SIP was expensive.  SIP required a faster circuit.  Analog was cheap.  It was available.  It was easy.

Now schools have to deal with the real possibility of losing funding for E-Rate voice service because one of the commissioners thinks that no one uses voice any more.  I say we should take the money he wants to save and reinvest it into modernizing phone systems for all E-Rate eligible schools.  Doing so would go a long way toward removing the increasing maintenance costs for legacy phone systems as well as retiring circuits that require constant attention.  That would increase the pool of available money in future funding years.  The answer isn’t to kill programs.  It’s to figure out why they cost so much and find ways to make them more efficient.  And if you don’t think that’s what’s needed Commissioner Pai, give me a call.  I still have a working phone.

Is It Time To Eliminate Long Distance?

Posted on by
1
Embed from Getty Images

“What’s your phone number?”

It seems like an innocuous question.  But what are you expecting?  Phone numbers in the US can vary in length greatly depending up on where you live.  I grew up in a small town.  My first telephone line was a party line.  Because there were four families on the same line, phone numbers didn’t mean much beyond getting you to the general location.  When we moved into town we finally got our own telephone line.  But the number was only four digits, like a PBX extension.  Since all phones in two had the same prefix, all calls were switched via the last four digits.  The day finally came when we all had to dial the prefix along with the four-digit number.  Now were were up to seven.

If you ask someone their phone number, you’re likely to get any one of several number combinations.  Seven digits, ten digits, or even eleven digits for those that do international business.  Computer systems can be coded to automatically fill in the area code for small stores that need contact information.  Other nationwide chains ask for the area code every time.  And those international business people always start their number with “+1”, which may not even be an option on the system.  How do we standardize?

Cracking The Code

Part of our standardization issues come from the area codes we’ve been using for sixty years.  Originally conceived as a way to regionalize telephone exchanges, area codes have become something of a quandary.  In larger cities, we use 10-digit phone dialing because of overlay area codes.  Rather than using one code for all the users in a given area, the dial plan has grown so large that more codes were needed to serve the population.  In order to insure these codes are used correctly you must dial all ten digits of the phone number.

In smaller locations still served by one area code, the need for 10-digit dialing is less clear. In my home area code of 405, I don’t need to dial ten digits to reach the Oklahoma City metro area.  If I want to dial outside of my area code, I need to use the long distance prefix.  However, there are some areas in the 405 area code that are not long distance but require dialing 405.  These are technically Inter-LATA Intrastate long distance calls.  And the confusion over the area codes comes down to the long distance question.

Going the Distance

The long distance system in America is the cause of all the area code confusion.  Users universally assume that they need to dial a 1 before any number to cross area codes.  That is true in places where a given area code covers all users.  But users also need to dial the long distance code to access users on different phone systems and in different towns.  It’s difficult to remember the rules.  And when you dial a 1 and it’s not needed, you get the reorder tone from your telco provider.

Now add mobile phones into the equation.  My friend from college still has the same mobile number he had ten years ago in this area code.  He lives in Seattle now.  If I want to call and talk to him, it’s a local call on my home phone.  If his next door neighbor wants to call him it will be a long distance call.  Many people still have their first mobile number even though they have moved to area codes across the country.

Mobile phone providers don’t care about long distance calls.  A call to a phone next to you is no different than a call to a phone in Alaska.  This reinforces the importance on 10-digit dialing.  I give my mobile number as ten digits all the time, unless I give the 11-digit E.164 globalized E.164 number.  It’s quick and easy and people in large areas are used to it.

It’s time to do the same for landline phones.  I think the utility for landlines would increase immensely if long distance was no longer an issue.  If you force all users to dial ten digits they won’t mind so long as the calls can be routed anywhere in any area code.  When you consider that most phone providers give users free long distance plans or even service for just a few cents, holding on to the idea of long distance calls makes little to no sense.

Tom’s Take

As a former voice engineer, long distance always gave me fits.  People wanted to track long distance calls to assign charges, even when they had hundreds of minutes of free long distance.  The need to enter a long distance access code rendered my Cisco Cius unusable.  I longed for the day that long distance was abolished.

Now, local phone companies see users evaporating before their very eyes.  No one uses their home phone any more.  I know I never answer mine, since most of the calls are from people I don’t want to talk to.  I think the last actual call I made was to my mother, which just happened to be long distance.

If telcos want users to use landlines, they should abolish the idea of long distance and make the system work like a mobile phone.  Calling my neighbor with a 212 area code would just require a 10-digit call.  No long distance.  No crazy rules.  Just a simple phone call.  People would start giving 10-digit numbers.  Billing would be simplified.  The world would be a better place.

The CoR Issue

Posted on by
Reply
Image from John Welsh. Read his blog for more voice goodness.

Image from John Welsh. Read his blog for more voice goodness.

In my former life as a voice engineer, I spent a lot of my time explaining class of restriction (CoR) to users and administrators.  The same kinds of questions kept getting asked every time I setup a new system.  Users wanted to know how to make long distance calls.  Administrators wanted to restrict long distance calls.  In some cases, administration went to the extreme of asking if phones could be configured to have no dial tone during class periods or only have long distance enabled during break and lunch periods.

This kind of technology restriction leads to all kinds of behavioral issues.  The administrators may have had the best of intentions in the beginning.  Restricting long distance calls cuts down on billing issues.  Using access codes removes arguments about who dialed a specific number.  Removing dial tone from a handset during work hours encourages teachers and staff and employees to focus on their duties.  It all sounds great. Until the users get involved.

No Restrictions

Users are ingenious creatures.  Given a restriction, they will do everything they can to go around it.  Long distance codes will be shared around a department until an unrestricted one can be found and exploited.  Phones that have dial tone turned off will be ignored.  Worse yet, given a restrictive enough environment users will turn to personal devices to avoid complications.

I used to tell school officials the unvarnished truth.  If you disable a phone during class, teachers will just drag out their cell phone to make a call when needed.  They won’t wait for a break, especially if it is a disciplinary issue or an emergency.  Cell phones are pervasive enough now that most everyone carries one.  Do you think that an employee that has a restricted phone is going to accept it?  Or will they just use their own phone to make a long distance call or make a call during a restricted time?

Class of restriction needs to be rethought for phone systems in today’s environments.  We need to ensure that things like access codes are in place for transparency, not for behavior modification.  Given that we have options like extension mobility for user identification on a specific device, it makes sense that we should be abel to identify phone calls from a given user on a given extension with ease.  There should be no reason for a client matter code or forced authorization code.

Likewise, restricting dial tone on a phone should be discouraged.  Giving users a good reason to use non-controlled devices like cell phones isn’t really a good option.  Instead, you should be counseling the users to treat an in-room phone like any other corporate device.  It should be used when appropriate.  If direct inward dial (DID) is configured for the extension, users should be cautioned to only give the number to trusted parties.  DID is usually not configured for extensions in most of my deployments, so it’s not an issue.  That’s not to say it won’t come up in your deployment.

Tom’s Take

Class of restriction is a necessary evil in a phone system.  It prevents expensive toll calls like 900 numbers or international calls.  However, it should really on be used to curtail these kinds of problems and not to restrict normal user behavior, like long distance calls.  I can remember using my Cisco Cius for the first time only to discover that a firmware bug rendered it unusable due to CoR preventing me from entering a long distance code.  I had to shelve the unit until the bug was fixed.  Which just happened to be a few weeks before the device was officially killed off.  When you restrict the use of your device, users will choose to not use your device.  Giving users the largest number of options will encourage them to use everything at their disposal.  CoR shouldn’t create issues, it should allow users to solve them.

FaceTime Audio: The Beginning or The End?

Posted on by
4

BlackApple

The world of mobile devices is a curious one. Handset manufacturers are always raising the bar for features in both hardware and software in order to convince customers to use their device. Yet, no matter how much innovation goes into the handset the vendors are still very reliant upon the whims of the carriers. Apple knows this perhaps better than anyone

In Your FaceTime

FaceTime was the first protocol to feel the wrath of the carriers. Apple developed it as a way to facilitate video communication between parties. The idea was that face-to-face video communications could be simplified to create a seamless experience. And it did, for the most part. Except that AT&T decided that using FaceTime over 3G would put too much strain on their network. At first, they forced Apple to limit FaceTime to only work with wireless connections. That severely inhibited the utility of the protocol. If the only place that a you can video call someone is at home or in a coffee shop (or on crappy hotel wireless) that makes the video call much less useful.

Apple finally allowed FaceTime to operate over cellular networks in iOS 6, yet AT&T (and other carriers) restricted the use of the protocol to those customers on the most current data plans. This eliminated those on older, unlimited data plans from utilizing the service. The carriers eventually gave in to customer pressure and started rolling out the capability to all subscribers. By then, it was too late. Apple had decided to take a different track – replace the need for a carrier.

Message For You

The first shot in this replacement battle came with iMessage. Apple created a messaging protocol like the iChat system for Mac, only it ran on iPhones and iPads (and later Macs). It was enabled by default, which was genius. The first time you sent an Short Message Service (SMS) text to a friend, the system detected you were messaging another iPhone user on a compatible version of software. The system then flipped the messaging over to use iMessage instead of SMS and the chat bubbles turned blue instead of green. Now, you could send pictures of any size as well as texts on any length with no restrictions. 160-character limits were no longer a concern. Neither was paying your carrier for an SMS plan. So long as the people you spoke with were all iDevice users the service was completely free.

iMessage was Apple’s first attempt to sideline the carriers. It removed a huge portion of their profitability. According to an article published at the launch of iMessage, carriers were making $.20 per message outside of an SMS plan for data that would cost about $.0125 on a data plan. Worse yet, that message traversed a control channel that was always present for the user. There was no additional cost to the carrier beyond flipping a switch to enable message delivery to the phone. It was a pure-profit enterprise. Apple seized on the opportunity to erode that profitability.

Today, you can barely find a cellular plan that *doesn’t* include unlimited text messaging. The carriers can no longer reap the rewards of a high profit, low cost service like SMS because of Apple and iMessage. Carriers are instead including it as a quality of life feature that they make nothing from. Cupertino has eliminated one of the sources of carrier entanglement. And they’re poised to do it again in iOS 7.

You Can Hear Me Now

FaceTime Audio was one of the features of iOS 7 that got swept under the rug in favor of talking about flat design or parallax wallpaper. FaceTime Audio uses the same audio codec from FaceTime, AAC-ELD, to initiate a phone call between two iDevice users. Only it doesn’t use the 3G/LTE radio to make the call. It’s all done via the data connection.

I tested FaceTime Audio for the first time after my wife upgraded her phone to iOS 7. The results were beyond astonishing. The audio quality of the call was as crisp and clear as any I’d every heard. In fact, I would compare it to the use of Cisco’s Wideband G.722 codec on an enterprise voice system. My wife, a non-technical person even noticed the difference by remarking, “It’s like you’re right next to me in the same room!” I specifically tried it over 3G/LTE to make sure it wasn’t blocked like FaceTime video. Amazingly, it wasn’t.

The Mean Opinion Score (MOS) rating that telephony network use to rate call clarity runs from 1 to 5. A 1 means you can’t hear them at all. A 5 means there is no difference between talking on the phone and talking in the same room. Most of the “best” calls get a MOS rating in the 4.1-4.3 range. I would rate FaceTime audio at a 4.5 or higher. Not only could I hear my wife clearly on the calls we made, but I also heard background noise clearly when she turned her head to speak to someone. The clarity was so amazing that I even tweeted about it.

FaceTime Audio calling could be poised to do the same thing to voice minutes that iMessage did to SMS. I’ve already changed the favorite for my wife’s number to dial her via FaceTime Audio instead of her mobile phone number. The clarity makes that much of a difference. It also helps that I’m not using any of my plan minutes to call her. Yes, I realize that many carriers make mobile-to-mobile calls free already. However, I was also able to call my wife via FaceTime Audio from my iPad as a test that worked perfectly. Now, I not only don’t use voice minutes but have the flexibility to call from a device that previously had no capability to do so.

Who Needs A Phone?

Think about the iPod Touch. It is a device that is very similar to the iPhone. In fact, with the exception of the cellular radio one might say they’re identical. With iMessage, I can get texts on an iPod touch using my Apple ID. So long as I’m around a wireless connection (or have a 3G MiFi device) I’m connected to the world. With FaceTime audio, the same Apple ID now allows me to take phone calls. The only thing the carriers now have to provide is a data connection. You still can’t text or call non-Apple devices with iMessage and FaceTime. However, you can reduce the amount of money you are paying for their services due to a reduction in the amount of minutes and/or texts you are sending. That should have the mobile carriers running scared.

Tom’s Take

I once said I would never own a cellular phone because sometimes I didn’t want to be found. Today, I get nervous if mine isn’t with me at all times. I also didn’t get SMS messaging at first. Now I spend more time doing that than anything else. Mobile technology has changed our lives. We’ve spent far too much time chained to the carriers, however. They have dictated what when can do with our phones. They have enforced how much data we use and how much we can talk. With protocols like FaceTime Audio, the handset manufacturers are going to start deciding how best to use their own devices. No carrier will be able to institute limits on minutes or texts. I think that if FaceTime Audio takes off in the same way as iMessage, you’ll see mobile carriers offering unlimited talk plans alongside the unlimited text plans within the next two years. If 50% of your userbase is making calls on their data plans, they need for all those “rollover” minutes becomes spurious. People will start reducing their plans down to the minimum necessary to get good data coverage. And if a carrier decides to start gouging for data service? Just take your device to another carrier. Or drop you contact in favor of a MiFi or similar data-only connection. FaceTime Audio is the beginning of easy Voice over IP (VoIP) calling. It’s the end of the road for carrier dominance.

Related articles