Switchport Voice VLAN – What Does It Do?

One of the more tedious parts of any phone system deployment is configuring the access layer switches to support said phones.  The configuration in and of itself isn’t complicated, but every port that may receive a phone needs to be setup correctly.  In Cisco parlance, this is accomplished with the switchport voice vlan <ID> command.  I’ve typed that into the CLI a thousand times and never really knew what it did besides “make the phones work”.  After a little research, I finally found some answers.  I thought I’d share them with you.

In the old days, before the Catalyst 2950, configuring a switch port for use by a phone involved creating an explicit 802.1q trunk.  This made sense from the perspective that it allowed traffic from multiple VLANs to pass on a single link.  It also allowed the 802.1p priority bits for Quality of Service (QoS) tagging to be sent with the frames.  The downside is that it was very difficult for phone mobility.  You either needed to provision every phone-facing switchport in your organization to be an 802.1q trunk or you had to leave the phones were they were.  While the latter is usually the case in most of my deployments, the mobility provided by the ability to plug a phone in anywhere in the network and not worry about extra configuration is key to some clients.  Thankfully, Cisco fixed this starting in the 2950 with a little concept known as the Auxiliary VLAN.

The Auxiliary VLAN (AUX VLAN) is a specialized VLAN that sits beside a regular access VLAN configured on a switch (sometimes called a “normal” VLAN).  The purpose of the AUX VLAN is to allow IP phones to transmit their payloads along with the untagged data coming from a PC that might be plugged into a switchport on the back of the phone.  The AUX VLAN allows these two devices to transmit on the same port without the need to use an explicit trunk on the link.  In addition, since the port is not configured explicitly as an 802.1q trunk, extraneous VLANs will not be flooded over the port.  In essence, the port becomes a two VLAN trunk.  All the phone traffic is tagged with the ID of the AUX VLAN and the PC traffic is untagged.  Curiously, according to this document, the traffic in the AUX VLAN must also carry a Class of Service (CoS) of 5 along with the AUX VLAN ID.  Otherwise, the traffic is dropped.  So how does the phone get the ID of the AUX VLAN so it can start sending the traffic?  Ah, that’s where CDP comes in.

Cisco Discovery Protocol (CDP) is very crucial in the operation of a Cisco IP phone.  It not only provides the AUX (Voice) VLAN ID for the phone to being sending traffic on the AUX VLAN, it also allows the phone to automatically negotiate power settings.  This allows the phone to use less than the maximum 15.4 watts of power under the 802.3af PoE standard.  If you disable CDP on the port facing the phone/PC you will likely start pulling your hair out.  Even though the phone might have already assigned itself in the Voice VLAN, removing CDP from the switchport in question causes it to forget where to find the voice VLAN.  You’ll need to re-enable CDP and reboot the phone.  You could also statically configure an 802.1q trunk to fix the issue, but where’s the fun in that?

One other curious note is that I’ve always been told that the connection between the phone and the switch when switchport voice vlan is configured is a “special 802.1q trunk”.  Not that I’ve ever been able to see that configuration, as show interface trunk seems to think that the port isn’t trunking and show interface switchport says that it’s an access port.  The key is in Cisco’s documentation.  The correct term for a port with switchport voice vlan configured is a “multi-VLAN access port”.  The distinction between the two is that only the two vlans (voice and access) configured on the switchport will be accepted on the link.  If you were to do something silly like, oh I don’t know, plug another switch into the back of the phone and configure an access port on that switch to be in a different VLAN than the voice or PC access VLAN, traffic will not pass through the phone port to the switch.  Once again, that’s because this isn’t a real trunk.  The switch will only accept tagged frames from the Voice (AUX) VLAN.


Tom’s Take

I hope this was a little more insight into what the magical command switchport voice vlan does on a switch.  I’m often asked by people new to voice why this must be configured each time.  Before I blindly regurgitated lines like “special 802.1q trunk” and “do it or it won’t work.”  Now I have a very interesting story to tell and threaten people with if they don’t do it.

Cisco Unified Communications Manager 8: Expert Administration Cookbook – Review

When you spend as much time configuring Cisco Unified Communications Manager (CUCM) servers as I do, you do one of two things.  Either you spend a lot of time reading through documentation, or you write down the important steps as concisely as possible for later use.  Documentation has uses.  When you are first learning something or you need the explanation for exactly what a partition does, documentation is your best friend.  However, when you’ve configured a ton of servers already and know the basics cold, wading through page upon page of prose to find the missing parameter of your Automated Alternate Routing (AAR) configuration is time consuming and frustrating.  If only there was some book that you could keep with you that has the basic configurations spelled out in short snippets.  A book that would allow you to quickly look up a function or feature and get it up and running without a fifteen page lead-in.  Thankfully, such a book does exist:

Tanner Ezell (@tannerezell) does a great job of condensing the mountain of documentation that Cisco has produced to support CUCM into 285 pages of tips and tricks on configuring important features that you’ll run across every day.  Unlike the Cisco Press CUCM guide I reviewed previously, Tanner’s book doesn’t step through the details of configuring a partition or a calling search space (CSS) for the first time.  Instead, this book assumes that you are a professional that has done tasks like that many, many times before.  Instead, this book concentrates on some of the newer features in CUCM 8 that may or may not be something that the reader has configured before.  Things like E.164 normalized dialing using the “+” symbol or Cross-Cluster Extension Mobility.  In fact, after reading the first three recipes in the book, I configured plus-dialing on my production cluster with no fuss.  That’s not something I was comfortable doing after reading through the tome of configuration on Cisco’s website or in the Solution Reference Network Design (SRND) document.

Think of this book as a reference guide for the 20% of features that you may configure once or twice every six months.  Sure, I can create a North American Numbering Plan (NANP) route pattern list in my sleep.  However, when it comes time for me to configure AAR or setup the Real Time Monitoring Tool (RTMT) to email me when something breaks, I’m going to have to look up how to do that.  Now, all I need to do is flip open this book to the appropriate chapter and get right to work without using CTRL + F to sort through to what I need to know.

Tom’s Take

CUCM 8 Expert Administration Cookbook was a pretty quick read for me.  That’s because I’ve seen many of the things in here before.  The problem is that I don’t remember them since they aren’t things I do every day.  It’s nice to know that I have a good reference book that I can rely on to help me in those times of need when I have to have a feature up and running quickly and my mind has gone totally blank on it.  I commend Tanner Ezell for taking the time to boil the feature configuration down to the bare necessities needed to get everything operational and then put it into printed form for us to enjoy.  I’m sure that my copy of this book is going to be well worn for many deployments to come.

Review Disclaimer

The copy of CUCM 8: Expert Administration Cookboook that was reviewed was purchased by me from Amazon.  It was not provided by the publisher.  As such, neither the publisher nor the author were granted any consideration in the writing of this review.  The opinions and analysis contained herein are mine and mine alone.

Moving to CUCM 8.6 – You’ll Never Upgrade Me Alive COPpers!

Upgrades are a fact of life for network rock stars.  Whether we are patching bugs or adding new features to our systems, the installation of software never seems to end.  If you are a Cisco voice rock star, you all too often find yourself upgrading to newer releases of Cisco Unified Communications Manager (CUCM) to support new devices like the Cius or fix show stopping bugs like the 180-day uptime lockup.  However, if you are a user of CUCM 8.x and you’re trying to move to 8.6, you’ve probably had a couple of head scratching moments so far.

If you’ve popped a freshly burned 8.6(1) ISO into your DVD drive or copied it via SFTP, you kicked off you installation and likely saw the following error message:

09/18/2011 19:31:48 refresh_upgrade|********** Upgrade Failed **********|<LVL::Info>
09/18/2011 19:31:48 refresh_upgrade|*** Please install the Refresh Upgrade COP, and reattempt the upgrade ***|<LVL::Info>
09/18/2011 19:31:48 refresh_upgrade|************************************|<LVL::Info>

Huh? What’s a refresh upgrade?  Why isn’t this ISO file working?  Well, it turns out Cisco needs you to take an additional step first.

CUCM runs on an operating system.  Up until version 5, that was Windows 2000 with some hardening and customizations.  Cisco eventually ported CallManager 4.3 to Windows Server 2003, but in the end the decision was made to move to an appliance-based OS that utilized Linux.  The Telephony OS in CUCM 5.x was new for those used to working in Windows but somewhat familiar to those that have seen Linux before, even if the login shell looked nothing like bash.  Cisco provides patches for the OS with every release of CUCM software and the user never knows what’s going on because of the way the system installs the patches transparently.  However, much like the shift from Windows 2000 to Windows 2003, software eventually reaches the end of its life.  Development stops on the old version and it’s time to move to the new one.  Such is the case in CUCM.  With version 8.6, Cisco has moved away from an OS platform based on Redhat Enterprise Linux (RHEL) 4 and upgraded the underlying OS to RHEL 5.  This is good news that allows the system to stay current and support a larger variety of hardware.  The bad news is that the upgrade of the OS can be a bit destructive.  This is part of the reason for the extra steps in moving to CUCM 8.6

Firstly, Cisco wants you to install a special Cisco Options Package (COP) file on 8.5(1) systems.  This file is ciscocm.refresh_upgrade_v1.0.cop.sgn.  The 8.6 installer checks for the presence of this file and won’t kick off unless it’s present.  It needs to be installed on every server in the cluster.  It’s also going to reboot the server after installation.  As near as I can tell, it makes some changes to the Tomcat service on the server as well as adding two new fields to the Install/Upgrade window:


Notice the new options for email.  This allows the server to send you an email whenever the upgrade is completed.  Probably a long overdue option that comes in handy for those of us that spend more than a few stress-filled moments clicking the Refresh buttons on our web browsers waiting for CUCM to come back to life after an upgrade.  There’s another reason for putting this email field in here now, though.

It turns out that when you upgrade from 8.5 to 8.6, its going to take a while.  Quite a while, in fact.  The system is going to reboot no less than twice, perhaps even three times.  Considering that a CUCM reboot can take 15-20 minutes to complete each time during an upgrade, you’re looking at nearly an hour of rebooting time under certain circumstances.  During the upgrade, CUCM is going to do things in 3 phases:

Phase 1: Export all the pertinent CUCM data to a safe partition

Phase 2: Reboot and install RHEL 5, then reboot and install the CUCM applications

Phase 3: Import all the data from the export partition

On the 7825H3 MCS server, there isn’t enough hard drive space to contain the safe partition during the reformat and installation of CUCM 8.6.  In that case, you’re going to need to plug a 16 GB USB drive into the system to serve as a target for the data export.  If you’re trying to upgrade a CUCM Business Edition system on a 7828H3 server, you better bust out the credit card because you’re going to need a 128 GB USB drive to hold all the CUCMBE data during the upgrade.  The IBM servers aren’t affected by this little caveat, as I’ve done the 8.6 refresh upgrade on a 7825I4 and not had any issues.  Be sure to leave the USB drives plugged in the whole time the system is upgrading.  Also, whatever is on the drive is going to be overwritten without warning, so be sure it’s blank before you start.

After you’ve completed the whole installation with all the reboots, you’re going to have a fresh new system with CUCM 8.6 to support all kinds of wonderful things, like finally being able to use Google Chrome to administer things.

Tom’s Take

I kicked off an upgrade to 8.6 without reading the release notes or documentation.  Thankfully Cisco prevented me from screwing things up big time by halting the installation with the above error message.  The more I dug into things, the more interesting it was.  It also took me two hours to finish things up with many reboots and even more nail biting (Fun fact: I was doing the upgrade during Packet Pushers Show 56, which is one of the reasons why I was quiet – I was trying not to scream at my CUCM server).  However, I think I could have avoided some pain and stress if I’d just read the docs first or even searched for refresh upgrade before I got started.

Cisco Cius – My Long Overdue Review


Cisco has introduced a new unified communications endpoint into its portfolio of devices that it hopes will bring a new user experience to customers wanting to unify video and voice in the palms of their hands.  The Cisco Cius represents a large investment into the intersection of mobility, voice, and video.

I won a Cisco Cius at Cisco Live this year.  I was excited to get it into my hands and start playing with it.  I wanted to put it on my desktop and utilize every function I could.  It’s been four months since I won the device, and I’ve spent time on and off putting it through it’s paces.  Some of the things I found were good.  Others, no so much.

The Cius is an Android-based (Froyo 2.2.2) tablet.  It has a 7″ screen (1024×600) with an Atom Z615 processor and 1GB of RAM.  It has an 802.11 a/b/g/n radio and a 4G LTE radio in an upcoming model as well as front and rear cameras, the latter capable of capturing 720p video.  It is also capable of being docked with a port replicator and handset that allows for speakerphone as well as USB ports to drive a keyboard and mouse.  Why?  Because the Cius also includes a Virtual Infrastructure Experience (VXI) client for running a virtual desktop as a replacement for your desktop PC.


When I got the unit, I first had to cool my jets for a bit.  The unit had pre-production software that wasn’t quite up to specs yet.  One of the things that didn’t work was application installation.  The Cius provides its own app store, AppHQ, which can be controlled via corporate policy to restrict downloads to this store.  You can also sideload apps from the regular Android market, but if your admin overlords decree that you shant be able to do that, you’ll be locked into AppHQ.  I took my time poking around the interface and noting how different it was from my iPad.  This was my first attempt at using a Google tablet, so it did take a bit of getting used to, layout wise. As well, the construction was a little different and the unit felt more ‘solid’.  Not to say that Apple’s iPad feels cheap, but the Cius is a little more dense than the aluminum used on my gen 1 iPad.  However, due to the software difficulties I was unable to do much with the Cius.  I did use it to record my Ultimate Cisco Live Attendee video right before I packed it away for the trip home.  Here you can get a feel for the video quality from the front VGA camera:

After I got it home, I had many stops and starts trying to get the right firmware to update it to a point where I could install things.  Thanks to some help from my friend Jon Nelson, I was at least able to get the right software to register it with my CallManager server, which I finally had to upgrade to 8.5 to get everything working correctly.  When I got the new firmware load installed, I was able to browse to the Android Market and start installing apps.  The process was pretty straightforward, and every app was available for installation.  The 7″ screen did seem a little cramped from my 10″ iPad, but it was very usable for simple browsing tasks.  I also noticed that the media dock didn’t secure the unit when docked.  Normally, I expect to hear a click or a snap as the locks engage on something like that, but there was nothing here.  In fact, if you don’t pay attention when docking the unit, it will slip and slide right off into the floor.

After playing with the Cius for a few days, I hit my first show stopping bug.  In the current firmware load there is a problem with dialing calls that require Forced Authorization Codes (FACs).  The dialpad for the unit disappears when the dial string is completed and won’t show up again until the call is connected.  The problem for me is that all my long distance calls (which represent the majority of my office calls) require me to enter an access code when I dial.  Without a dialpad, I can’t enter the code to complete the call.  For this reason, the 9971 I normally use has stayed on my desk and the Cius has been relegated to the side desk where it gets tested on occasion.  I’m sure that Cisco has seen the oversight in not allowing me to have a dialpad during ring out and will be releasing a firmware to fix that in no time.  Oh, wait…

In order to expedite my firmware update desires, I signed up for the Cius developer program and gained access to the firmware update service for testing.  Never one to shy away from putting beta code on my devices, I followed the developer directions and waited patiently for my Cius to update.  It took a couple of hours to pull the new code and reboot.  Where it promptly locked up.  Every time I tried to install new code, it rebooted and hung on the restart, the Cisco logo taunting me for hours on end until I performed a hard reset.  Which of course reset the firmware back to the old version.  And erased anything I might have installed.  Oh, bother.

Figuring that beta firmware may be just a little too advanced, I decided to head over to Cisco’s website and pull down a new production firmware for CUCM so that I can update it like that.  Which is where I finally encountered the “You do not have a valid contract” error that has bitten so many people as of late, especially Ethan Banks.  Of course, I don’t have a SmartNet contract for this device since I didn’t buy it in the first place.  I figure I need to order one if I want to figure out why it keeps locking up or why I can’t get the dialpad to show up to make long distance calls.  I know the firmware I managed to load did fix some other transient issues, like the unit losing connection to CUCM every night and requiring a reboot to establish a connection again.  However, I’m going to need a lot of support to bring this device up to the point were I consider it a replacement for my 9971 deskphone.


Tom’s Take

If I had to use one word to describe the Cius, it would be potential.  Cisco has obviously invested a lot of money into this unit and sees it as a big step going forward to unify all of their cutting edge technology into a single portable unit.  It makes for a really nice demo and you can argue that it makes a statement sitting on the desk.  The hardware seems to be acceptable for use as a business communications endpoint.  However, software quirks show it to be an early 1.0 product release.  Difficulties in getting my unit into a usable condition hampered me from replacing my current desk phone.  Inability to get software to load without causing reboot loops has forced me to reformat more times than I care to count.  And short-sightedness at allowing me to download production firmware updates means that it will likely sit on the side of my desk until such time as someone decides that, as a Cisco partner, I am not a stinking filthy pirate and only want to get my Cius running so that I can show it off to coworkers and customers in the hopes that they buy a truckload of them.  However, until that day comes, my Cius will be relegated to little more than a curious desk ornament, right next to the Buckyballs and my stressball collection.  Let’s hope I can fix that sooner rather than later.


Disclaimer

The Cisco Cius I have was won in a contest at Cisco Live 2011.  I recieved a Cius and a media dock, as well as a Cisco-branded Jawbone Icon headset.  At no time did Cisco ask me to write a review of the device, nor did they place any restrictions on the content of any reviews written by me.  They did not ask for any consideration nor were they promised any by me in the crafting of this post.  The opinions and conclusions reached are mine and mine alone.

Dial Plan Considerations

A Candlestick Phone (image courtesy of WIkipedia)

Dial Plans are probably one of the hardest parts of learning about voice.  I consider it to be just like subnetting for network enginee…rock stars.  There are volumes upon volumes of how to stage and arrange your dial plans to speed call routing and minimize memory usage on voice over IP (VoIP) equipment.  However, there are a couple of things that I’ve found over the course of my career in voice that I want to pass along that I’ve never really found written down anywhere.  Consider these some of the “street smarts” of VoIP.

– Avoid Placing Extensions in the “9XXX” range.  This one seems to be the most popular issue.  No matter if you’re using 3-digit or 4-digit extensions, consider anything beginning with a “9” to be off limits.  There are actually a couple of reasons for this.  First and foremost, “9” is generally used at the PSTN access code (or escape code) for most PBX-style equipment in the world.  It’s also used as the escape code for Centrex phone service.  If any of the extensions on your Cisco phone system start with a “9”, the system will get a bit confused.  The external route patterns on your CUCM/CUCME system all start with “9” and have the “Provide Outside Dial Tone” box checked (at least they should).  If you have an extension that is 9640, for instance, CUCM will not play the pitch-changed PSTN dial tone until the number you are dialing explicitly matches a route pattern with the “Provide Outside Dial Tone” check box enabled.  In this example, if you are calling a long distance number, when you hit “9”, you won’t hear the higher-pitched tone.  You also won’t hear it if you follow with 1, 3, or 1.  Not until you dial the 5th digit of your long distance call that eliminates the above 9640 extension will the caller hear the PSTN dial tone.  While this doesn’t really affect the operation of the system, it really throws the users for a loop when they don’t hear that dial tone for accessing the PSTN.

The other crucial reason for avoiding extensions that start with “9” is to cut down on the number of misdialed emergency numbers (911 or 999).  I’ve talked about emergency numbers before and taking them into account here is just as critical.  I’ve even had to change the PSTN escape code to something other than “9” (like 8 or 7) in order to correct this emergency calling issue.  In those cases, I have to avoid putting extensions in the 9 range and the new code range to keep my PSTN dial tones and emergency calling behavior straight.

– The 1XXX range is your friend.  If you need a number range for your extensions or voice mail ports or other system directory numbers, anything starting with a “1” is a great idea.  Why?  Well, since the very beginning of phone systems two numbers have always been reserved and not used to start phone numbers.  One of these is “0”.  Zero has always been used as a signal to the phone company operator, so no number in the North American Numbering Plan (NANP) starts with a zero.  The other number is “1”.  One is a more curious case.  It turns out that the original “candlestick” phones had a bad habit of sending a fast pulse when they went off-hook.  In order to prevent a ton of misdialed calls, the system was configured to ignore any numbers that started with a “1”.  Again, no numbers in the NANP start with a “1”.  We now use One to signal a long distance telephone call, but that is really the only time it’s used.  If you use the 1XXX range for all your voice mail ports or park slots or even extensions, you never really have to worry about it colliding with other parts of your dialing plan.  If you’re setting up a home CUCME system, like I’m trying to do once I can convince my wife, you can put your extensions in the 1XX range and not need to worry about using a PSTN access code.  I’ll probably write a little more about this once my experiment is up and running.

– Create a local 10-digit dial peer.  I’ve mentioned this in passing once before, but if you still live in one of those areas that hasn’t switched to 10-digit dialing for all local calls, you should probably program an explicit local dial peer.  For example, in central Oklahoma calls are still dialed with 7 digits locally.  However, there are destinations that are not long distance (prefixed with a 1) that use 10-digits.  If you program a standard 10-digit dial peer (9.[2-9]XX[2-9]XXXXXX), when you dial 7-digit local calls the system must wait for the interdigit timeout to expire before dialing the call.  This is because those 7 digits can match two different dial peers (7-digit and 10-digit) and the system doesn’t know which one to use until you let the digit timeout expire, which could be up to 15 seconds.  That time is an eternity to your users.

Instead, until the time when your state figures out 10-digit dialing is what all the cool kids are doing, you should do this little work around.  Configure your regular 7-digit and long distance dialing codes.  Rather than creating a 10-digit route pattern though, just create a route pattern with your 10-digit local area code.  In the above example for central Oklahoma (area code 405), that explicit dial peer would be 9.405[2-9]XXXXXX.  This way, any 10-digit calls will route immediately.  Most of your 7-digit calls should route immediately as well when they match the 9.[2-9]XXXXXX route pattern.  The only issue you might have is if your local NANP prefix (the [2-9]XX part) is the same as your area code.  Chances are slim in that case, so your local calls won’t wait for the interdigit time to expire.  Just be sure to have the 10-digit dial peer for all local calls ready to go on the day you get switched over.  Otherwise you are going to have some confused and angry users.

Tom’s Take

If you are going to be a voice enginee…miracle worker, you are going to spend a lot of time learning about dial plans.  Before you know it, things will just be automatic and you’ll be able to churn them out without a second thought.  If you take my advice above into account as you’re learning about dial plans, you will have a much easier time when it comes to the strange corner cases you might run into like not hearing a PSTN dial tone or interdigit timeout issues for local calling.

Cisco Phone Cheat Codes

There are many things in this world that are hidden just beneath the surface that make our lives easier.  Whether it be the Secret Menu at In-n-Out Burger or the good old Konami Code, the good stuff that we need is often just out of reach unless you know the code.  This is also the case when dealing with Cisco phones.  There are three key combinations that will help you immensely when configuring these devices, provided you know what they are.

1.  Unlock Settings – *, *, #.  When you check the settings on a Cisco phone, you’ll notice that you can look at the values but you can’t change any of them.  Many of these values are set at the Cisco Unified Communications Manager (CUCM) level.  However, once common issue is the phone not being able to contact the CUCM server or the phone having the wrong address/TFTP server information from DHCP.  While there are a multitude of ways to correct these issues in the network, there is a quick method to unlock the phone to change the settings.

  • Go to the Settings page of the phone
  • While in the settings page, press *, *, # (star, star, pound) about 1/2 second apart
  • The phone will display “Settings Unlocked” and allow you to make changes

It’s that easy.  There won’t be a whole lot to do with the phone Telephony User Interface (TUI), but you can make quick changes to DHCP, IP address, or TFTP server address entries to verify the phone configuration is correct.  By the way, when putting in an IP address via TUI, the “*” key can be used to put a period in an IP address.  That should save you an extra keystroke or two.

2.  Hard Reset – *,*,#,*,*.  Sometimes, you just need to reboot.  There are a variety of things that can cause a phone to need to be reset.  Firmware updates, line changes, or even ring cadence necessitate reboots.  While you can trigger these from the CUCM GUI, there are also times that they may need to be done from the phone itself in the event of a communications issue.  Rebooting is also a handy method for beginning to troubleshoot issues.

But Tom?  Why not just pull the network cable from the back of the phone?  Won’t disconnecting the power reboot?

True, it will.  What if the phone is mounted to the wall?  Or if the phone is running from an external power supply?  Or positioned in such as way that only the keypad is visible?  Better to know a different way to reboot just in case.  Here’s where the reboot cheat code comes in handy.

  • Go to the settings page of the phone
  • Press *,*,#,*,* (star, star, pound, star, star) about 1/2 second apart
  • The phone will display “Resetting” and perform a hard reset

This sequence will cause the phone to reboot as if the power cable had been unplugged and force it to pull a new configuration from CUCM.  Once common issue I find when entering this code is the keypresses not registering with the phone.  Try it a couple of times until you develop a rhythm for entering it about 1/2 second apart.  Much more than that and the phone won’t think you’re entering the code.  Quicker than that and the keys might not all register.

3.  Factory Reset – “1,2,3,4,5,6,7,8,9,*,0,#”.  When all else fails, nuke the phone from orbit.  It’s the only way to be sure.  Some settings are so difficult to change that it’s not worth it.  Or you’ve got a buggy firmware that needs to be erased.  In those cases, there is a way to completely reset a phone back to the shipping configuration.  You’ll need access to unplug the power cable, as well as enough dexterity to press buttons on the front as you plug it back in.

  • Unplug the power from the phone.
  • As you plug it back it, press and hold the “#” key.  If performed correctly, the Headset, Mute, and Speaker buttons in the lower right corner will start to flash in sequence.
  • When those three buttons start flashing in sequence, enter the following code: 1,2,3,4,5,6,7,8,9,*,0,#.  You’ll notice that’s every button on the keypad in sequence from left to right, top to bottom.
  • Phone will display “Upgrading” and erase the configuration.

Don’t worry if you press a key twice on accident.  The phone will still accept the code.  However, you do need to be quick about things.  The phone will only accept the factory reset code for 60 seconds after the Headset, Mute, and Speaker buttons start flashing in sequence.

Tom’s Take

I find myself using these cheat codes all the time.  Whether I’m correcting a bad TFTP server entry or setting a static IP on a subnet, the ability to manipulate a phone without resorting to using CUCM all the time is very useful.  You can also use these codes to impress your friends with your intimate knowledge of the way Cisco phones work.  Just be careful with that reset code.  About every 1 out of 1,000 times it gives you 30 lives instead.

Missing CUCM Configuration Files

Oy.  There’s always one trouble ticket that gives you difficulty and makes you want to throw things around the room.  When you solve it, you yell and dance down the hallway proclaiming how smart you are to have gotten it fixed.  Folks, let me introduce you to that issue.

A Cisco Unified Communications Manager Business Edition (CUCMBE) server started exhibiting strange behavior.  No phones registered and no web GUI.  Not the first time that this has happened, so I’ll just log in via SSH and reboot the server.  When it came back up, nothing.  Same thing.  When I poke around in the CLI, I find out the SSH services are started, but that’s about it.  When I try to start the Tomcat service, which is required for the web GUI, I get an error about the Service Manager not being started.  No problem, I’ll just start that one:

admin:utils service start Service Manager
Aborting servM startup due to invalid configuration files

Oh crap.

Uh, restore from backup?  Hah!  No backup here.  Boot off the recovery CD and check the disk with FSCK (which looks a lot like a curse word I was uttering at this point)?  Fixed a couple of file issues, but still no dice on the services.  No backup partition, as this server had never been upgraded.

Just great.  What now?

Well, if you’re impatient like me when you’re waiting on support engineers to get back with you and you know you’re probably going to have to reload anyway, you can try some crazy things on the off chance they might work.  I mean, what’s the worst that can happen, right?

WARNING!!!!!

The things I’m about to discuss are totally unsupported by Cisco.  I also am not going to support them.  It worked for me this time, but it could have very easily screwed things up.  Don’t come to me and tell me you did this and now you need to reformat and you want me to help you.

Okay, that being said, there are a multitude of ways to gain root access to your CUCM server.  Again, none of them are supported, so don’t do them if you are the least bit squeemish.  The first thing you should read is the great guide at blindhog.net about gaining root access on CUCM 5.x/6.x.  It’s a very handy way to show you that the underlying system in CUCM is actually RedHat Enterprise Linux.  Since I didn’t have a Linux boot disk handy, I instead stumbled across this post which talks about jailbreaking CUCM.  I didn’t have to go all the way through it, but it is a fascinating read nonetheless.

1.  Download PuTTY, PuTTYgen, and PSFTP from HERE.  The instructions at the above link use these files and you should too.

2.  Log into CUCM CLI via SSH as the administrator user.

3.  Type in “file dump sftpdetails ../.ssh/id_dsa” at the CLI.  You’re going to get a dump of the SSH private key for the sftpuser account.  Copy this information to a text file and save it somewhere on your system.

4.  You need to convert this SSH private key from OpenSSH to PuTTY’s SSH format using PuTTYgen.  Import the Private Key file and save it somewhere like c:\temp.  Be sure to save it with the .ppk extension.

5.  Launch PSFTP with this command string:

psftp -2 -i c:\TEMP\id.ppk sftpuser@cucm.example.com

The file location should be where you saved the private key and the user@server should reflect your server’s IP or hostname.  Be sure to type in sftpuser@<your server address here>.

6.  If you’ve logged into the server before and saved the RSA fingerprint, you may get a warning here about the key your using.  Just say “yes” and keep going.

7.  Voila!  You’ve logged into the system as the sftpuser account and you can now download files from the Linux file system or copy files to it.  In the above link, this is where you would jailbreak the system.  For my particular example, we won’t have to go quite that far.

8.  In my troubleshooting case, I changed directories to “/usr/local/platform/conf/” which is where the configuration files live.  I noticed that “server.conf” was missing, but there was a “server.conf.bak” in the same directory.  I typed in “mv server.conf.bak server.conf” since I couldn’t copy the file.  Then I tried to start the Service Manager service again from a SSH CLI session.

SUCCESS!!!

Tom’s Take

I do stupid things all the time.  Like voiding warranties, which is what my little procedure above will do to your CUCM system if you try it.  I was desperate and impatient and it paid off for me this time.  I also have experience on the Linux CLI so I’m not afraid to do things there, even knowing that the outcome for a little slipup could crater my system.  Don’t do what I do unless you know what you’re doing or you aren’t afraid to reload.

That being said, a little Internet searching followed by some practical application can save your bacon in a time of emergency.  Just remember that the Disaster Recovery Tool (DiRT) is there for a reason. Use it wisely and use it often and you shouldn’t find yourself needing to jailbreak your CUCM server anytime soon.

SIP Trunking – Review

When I first got started working with Voice-over-IP (VoIP), I was excited about all the possibilities of making calls over the Internet and moving away from my old reliance on Ma Bell.  However, the reality of my continued dependence on the good old phone company is an ever-present reminder that sometimes technology needs to mature a little before I can make bigger leaps.  That’s why the idea behind SIP trunking has me excited.  It brings back a little bit of that hopeful magic from my early days of VoIP possibilities.  Thanks to Christina Hattingh, Darryl Sladden, and ATM Zakaria Swapan and the good folks over at Cisco Press, I got my feet wet with SIP Trunking:

This is the “pound cake” of Cisco Press books.  It’s only about 300 pages and a bit on the thin size, but it’s a very dense read.  Part 1 covers the differences between traditional Time-Division Multiplexing (TDM) trunking and SIP trunking.  There is discussion of the cost and benefit of moving to a hybrid model or even to a pure SIP environment.  This is a good part to focus on if you aren’t familiar with SIP trunking in general or you are trying to convince your decision makers to give it a try.

Part 2 is all about planning.  One hundred plus pages of modeling and design and checklists.  An engineer’s dream.  You are going to spend a lot of time in here dissecting the cutover strategies and the list of questions that you need to ask your provider before delving into the SIP-infested waters.  In fact, I would recommend this book for Chapter 9 alone, the checklist chapter.  It goes into great detail about all the questions you need to ask your provider, along with a description of each question and why the answer would be so important to you.

Part 3 is the deployment guide.  No Cisco Press book is complete without some code examples, and Chapter 10 has them in spades.  One thing I did like about their examples of AT&T and Verizon configuration is that they are appropriately annotated with notes to be sure you understand why a particular setting was configured.  I want to see more of this in the networking-focused Cisco Press books, not just the planning ones.  There are also case studies to help you make decisions and a chapter on the future of Unified Communications.  This one’s kind of dubious, though, as most of the time the predictions either end up looking hilariously obvious in hindsight or wide of the mark.  You can’t fault the authors for wanting to put a little bit of vision in at the end of this read, though.

Tom’s Take

If you want to learn a little more about SIP trunking or you are planning to put one in in the next 6-8 months, grab a copy of this book.  Have a cup of coffee before you jump into it, as the material could be a little dry if you aren’t focused on the task at hand.  Make sure to dog-ear the first page of Chapter 9, as you’ll find yourself coming back here more and more as you start implementing your SIP trunk.

Disclaimer

This book was provided to me as a perk at Cisco Live for being a NetVet.  I chose this book from a list of the available titles and it was provided to me at no charge above the cost of the conference.  Cisco Press did not ask for nor did I promise any kind of consideration in the above review.  The thoughts and opinions expressed above represent my true and honest opinion of the material.

Less Than Dial Peer Zero

There was a good question posed by Karla Reyes a few days ago about the use of a voice dial peer that included the string incoming-called-number.  I did my best to figure out why one would use such a dial peer in production, and that led me to a search and discovery of all sorts of interesting things about dial peers and what happens when you are forced into defaults.

On a Cisco phone system, whether it be CallManager or CallManager Express, dial peers on your voice gateways route calls to the correct destinations, both inbound and outbound.  The outbound dial peers I’m familiar with, having configured them on numerous CUCME systems.  The inbound dial peers, not so much.  I configure most of my systems to use connection plar opx <extension> to ring analog circuits and send the calls to an operator or an auto attendant (AA).  Absent that configuration, you can still configure dial peers to evaluate incoming calls and route them to the proper locations.

Cisco voice gateways will evaluate the dial peers and choose the one the has the longest set of matching digits when deciding where to send a call.  So if one dial peer is 8… and the other is 85.., then if you call “8511”, the second dial peer will be used since it has the most matching digits of the two.  If you dial “8721”, the first dial peer is matched.  This is a simple, straight forward process.  When incoming calls are headed into the system, the same process is used.  So what happens if there isn’t a perfect match?

This is where Dial Peer 0 comes into play.  Dial Peer 0 exists on every voice gateway.  It’s a sneaky little default setting that no one pays much attention to until they stumble across it.  Part of the reason is that it’s invisible.  No reference to Dial Peer 0 pops up in the configuration.  Since it’s the default and it’s hard-coded, it’s always there.  It’s kind of like the implicit deny statement at the end of every access list.  And much like that deny statement, it often produces strange and undesirable results when it gets triggered.  That’s because Dial Peer 0 is designed to work with just about anything, so it has some of the silliest settings imaginable:

  • It supports any codec you throw at it.  The designers figured that rather than waste precious time negotiating whether or not your should use G.711 or G.729, Dial Peer 0 should just accept whatever it’s sent and move on.  Not so great when you are trying to keep bandwidth conserved on a slow link and someone sends you big fat voice packets.
  • DTMF packets are sent in-band, so if you decide to use G.729 as your signaling protocol, the tones can get mangled by the compression process.
  • All the voice signaling and data packets have their IP precedence settings remarked to 0.  All that pain and effort you went through getting your perfect Quality of Service (QoS) setup working gets chucked right out the window.
  • Resource Reservation Protocol (RSVP) is disabled.  That means if you spent even more time and pain setting up an Integrated Services QoS setup, it’s going to fail when the call hits Dial Peer 0.
  • IVR applications are disabled.  This means that any scripts that would be used for things like auto attendants are going to break.
  • Last but not least, Direct Inward Dial (DID) isn’t present.  This means your callers will get confused because they’ll hear a secondary dial tone when the call leg completes.  This is where they would enter your extension number to speak to you.  Instead, they’ll figure the call didn’t work then hang up and try again.  Enterprising hackers will hear the dial tone and realize what that means then try and transfer to another number, such as long distance or international calling.  That’ll run up your phone bill fast.

Sounds like a whole bunch of bad reasons to never use Dial Peer 0, right?  Except you can’t get rid of it.  It’s always there, lurking and waiting to screw up your day.  The only way to avoid using it is to configure a new default dial peer to match instead of letting the call fall all the way down to Dial Peer 0.  In order to understand that, you have to know the order in which Cisco matches incoming dial peers:

  1. Match the dialed number with incoming-called-address configured in the dial peer.
  2. Match the calling number with answer-address configured in the dial peer.
  3. Match the calling number with destination-pattern configured in the dial peer.
  4. Match an POTS dial peer with a port command configured in the dial peer.
  5. Take your chances with Dial Peer 0.

The easiest (and by far most popular) method of overriding Dial Peer 0 is to use #1 above like this:

dial-peer voice 1 pots
  incoming-called-number .
  direct-inward-dial

This ensures Dial Peer 1 is always matched ahead of Dial Peer 0, but not matched if a longer match exists.  It also uses DID to disable two-stage dialing, so those nefarious hackers won’t get a chance to call Cuba on your dime.

Tom’s Take

Defaults exist so that we don’t have to think about things sometimes.  Anyone who’s ever told someone how to install Microsoft Office by saying “Click Next, Next, I Agree, Next, Next” will know what defaults come in very handy.  However, if the default settings are baked in and unable to be changed, it makes life difficult if you can’t account for them and find a way to work around them.  At the very worst, it just means you get to spend a little more time on the phone with your favorite TAC engineers.

I’d like to thank Karla Reyes, Amy Arnold, and Aaron Conaway for their help on this post.

How to Transfer

The number one question I get asked when installing a new phone system has to be “How do I transfer a call?”  Seems that most people who have been using phones most of their adult lives are mystified by the little button that allows them to send a call somewhere else.  Especially when I put the complexity of a Voice over IP (VoIP) phone system in their hands.

The majority of people that have been using Private Branch Exchange (PBX) phone systems for the past few decades should be familiar with basic unsupervised transfer.  You may also hear this referred to as a blind transfer or a cold transfer.  In this implementation, the party initiating the transfer presses the transfer button, dials the number that the call needs to go to, and is then disconnected from the call.  The original caller is sent to the new target and when the target answers, they get an earful of whatever the original caller wishes to tell them.  They call this blind (or cold) because the other party has no idea it’s coming.  Some people, especially executives, get kind of upset by all this messy blind stuff.

Most modern phone systems use a method of supervised transfers.  These are also called consult transfers or warm transfers.  These kinds of transfers start out the same.  A called party presses the transfer button and dials the number of the person that needs to get the phone call.  In this case, though, the called party stays on the phone until the target picks up.  All the while, the original caller is hearing the dulcet tones of Streaming Winds (or whatever your music on hold happens to be).  When the target picks up, you get to give them all the gory details of the phone conversation so far, out of earshot of the transferred party.  Only when the person on the other end agrees to take the call do you complete the transfer by pressing the Transfer button again.  It’s a warm transfer because the target gets to hear a friendly voice before the original caller gets to launch into whatever they want to talk about.  It’s still possible for you to pull off a blind transfer in this setup.  When the target’s phone starts ringing, you press the transfer button to complete the transfer.  Now, the original caller will hear the target’s phone ringing and they get to start talking as soon as the phone picks up.  In some cases, depending on how the transfer caller ID works, this can be worse than a blind transfer, because the target will see the caller ID of the original called party and think it’s someone internally wanting to talk, only to get an outside caller when they answer.  It’s caused some embarrassment before, I can assure you.

For whatever reason, some people have a problem with warm transfers.  Especially if they are used to blind transfers.  They hang up on the original caller, thinking the transfer went through with no issues.  The problem is that the call won’t complete until the transfer button is pressed twice, meaning that hanging up without pressing it twice will disconnect the caller.  Then they get to call back, angry they’ve been disconnected.  No amount of training in this world has been able to break people of this bad habit.

Cisco Unified Communications Manager (CUCM) doesn’t have an option for blind transfers.  It’s a consult transfer system only.  However, there is an option you can set that will act like a blind system and allow your users to act as they always have without disconnecting callers.

  1. Log into your CUCM publisher and go to the System menu.
  2. Choose the Service Parameter option.
  3. Choose the publisher server from the dropdown menu.
  4. Choose the Cisco CallManager Service.
  5. Scroll down until you find the Clusterwide Parameters (Device-Phone)
  6. Look for the Transfer On-Hook Enabled setting.  By default, it’s set to FALSE.
  7. Change this setting to TRUE to allow On-hook transfers.

Here’s a picture in case you get lost:

Easy as cake.  If your phones are all SCCP-based, the setting will take effect immediately.  If you phones are SIP, you’re going to have to reset them to change this, so be sure to do it when you can reboot them all without affecting communications.

There you have it.  Once you’ve enabled this little setting, you can still use a warm transfer, but those users that can’t or won’t transfer with the Transfer button can still pull off a transfer by hanging up without disconnecting the caller.  If you can find a way to drill the concept of a warm transfer into the heads of those that just don’t seem to get it for some reason, you can make some real money.  Sure, it may involve some heavy duty mind control, but in the end I think it will all be worth it.