Glue Peddlers


There’s an old adage that says “A chain is only as strong as the weakest link.”  While people typically use this in terms on saying that teams are only as strong as their weakest member, I look at it through a different lens.  In my former life as a Value Added Reseller (VAR) engineer, I spent a lot of my time working with technologies that needed to be linked together like a chain.

You have probably seen the lamentations of a voice engineer complaining about fax machines.  If you haven’t, you should count yourself lucky.  Fax machines are the bane of the lives of many telecom folks.  They aren’t that difficult when you get right down to it.  They’re essentially printers with a 9600 baud modem attached for making phone calls.  Indeed, fax machines are probably one of the most robust pieces of technology that I’ve encountered.  I’ve seen faxes covered in dust and grime from a decade or more of use still dutifully churning out page after page of low resolution black-and-white print.

Faxes themselves aren’t the issue.  The problem is that their technology has been eclipsed to the point where interfacing them in the modern world is often difficult and time consuming.  I usually counsel my customers to leave their fax machines plugged directly into an analog landline to avoid issues.  For those times where that can’t be done, I have a whole bag of tricks to make it work with a voice over IP (VoIP) system.  Adaptors and relays and other such tricks help me figure out how to make this decades-old tech work with a modern PRI or SIP connection.  And don’t even get me started on interfacing a fire alarm with an IP phone system.

The best VARs in the world don’t make their money from reselling a pile of hardware to a customer.  The profits aren’t found in a bill of materials.  Instead, they make money in the glue business.  Tying two disparate technologies together via custom programming or knowledge of processes needed to make dissimilar technology work the right way is their real trade.  This is their “glue.”  I can remember having discussions with people regarding the hardest parts of an implementation.  It’s not in setting up a dial plan or configuring a VM cluster with the right IP address.  It’s usually in making some old piece of technology work correctly.  A fire alarm or a Novell server or an ancient wireless access point can quickly become the focus area of an entire project and consume all your time.

If you really want to differentiate yourself from the pack of “box pushers” out there just reselling equipment you need to concentrate on the point where the glue needs to be the stickiest.  That’s where the customer’s knowledge is the weakest.  That’s the point that will end up causing the most pain.  That’s where the money is waiting for the truly dedicated.  VARs have already figured this out.  If you want to make yourself valuable to a customer or to a VAR, be the best a gluing these technologies together.  Understand how to make old technology work with new tech.  There’s always going to be new technology coming out to replace what’s being used currently.  And there will always be a customer or two that want to keep using that old technology far past the expiration date.  If you are the one that can tie those too things together with a minimum of effort, you’ll find yourself the most popular peddler in the market.

CCIE Loses Its Voice

ccievThe world we live in is constantly adapting and changing to new communications methods.  I can still remember having a party line telephone when I was a kid.  I’ve graduated to using landlines, cellular phones, email, instant messaging, text messaging, and even the occasional video call.  There are more methods to contact people than I can count on both hands.  This change is also being reflected in the workforce as well.  People who just a few years ago felt comfortable having a desk phone and simple voice mail are now embracing instant messaging with presence integration and unified voice mail as well as single number reach to their mobile devices.  It’s a brave new world that a voice engineer is going to need to understand in depth.

To that end, Cisco has decided to retire the CCIE Voice in favor of an updated track that will be christened the CCIE Collaboration.  Note that they aren’t merely changing the blueprint like they have in the past with the CCIE SP or the CCIE R&S.  This is like the CCIE Storage being moved aside for the CCIE Data Center.  The radical shift in content of the exam should be a tip-off to the candidates that this isn’t going to be the same old voice stuff with a few new bells and whistles.

Name That Tune

The lab equipment and software list (CCO account required) includes a bump to CUCM 9.1 for the call processor, as well as various 9.x versions of Unity Connection, Presence, and CUCME.  There’s also a UCS C460, which isn’t too surprising with CUCM being a virtualized product now.  The hardware is rounded out with 2921 and 3925 routers as well as a 3750-X switch.  The most curious inclusion is the Cisco Jabber Video for Telepresence.  That right there is the key to the whole “collaboration” focus on this exam.  There is a 9971 phone listed as an item.  I can almost guarantee you’re going to have to make a video call from the 9971 to the video soft client in Cisco Jabber.  That’s all made possible thanks to Cisco’s integration of video in CUCM in 9.1.  This has been their strategy all along.

The CCIE Voice is considered one of the hardest certifications to get, even among the CCIE family.  It’s not that there is any one specific task to configure that just wrecks candidates.  The real issue is the amount of tasks that must be configured.  Especially when you consider that a simple 3-point task to get the remote site dial plan up and running could take a couple of hours of configuration.  Add in the integrated troubleshooting section that requires you to find a problem after you’ve already configured it incorrectly and you can see why this monster is such a hard test.  One has to wonder what adding video and other advanced topics like presence integration into the lab is going to do to the amount of time the candidate has to configure things.  It was already hard to get done in 8 hours.  I’m going to guess it’s downright impossible to do it in the CCIE Collaboration.  My best guess is that you are going to see versions of the test that are video-centric as well as ones that are voice-centric.  There’s going to be a lot of overlap between the two, but you can’t go into the lab thinking you’re guaranteed to get a video lab.

Hitting the Wrong Notes

There also seems to have been a lot of discussion about the retirement of the CCIE Voice track as opposed to creating a CCIE Voice version 4 track with added video.  In fact, there are some documents out there related to the CCIE Collaboration that reference a CCIE Voice v4.  The majority of discussion seems to be around the CCIE Voice folks getting “grandfathered” into a CCIE Collaboration title.  While I realize that the change in the name was mostly driven about the marketing of the greater collaboration story, I still don’t think that there should be any automatic granting of the Collaboration title.

The CCIE Collaboration is a different test.  While the blueprint may be 75% the same, there’s still the added video component to take into account (as well as cluster configuration for multiple CUCM servers).  People want an upgrade test to let the CCIE Voice become a CCIE Collaboration.  They have one already: the CCIE Collaboration lab exam.  If the title is that important, you should take that lab exam and pass it to earn your new credential.  The fact that there is precedent for this with the migration of the Storage track to Data Center shows that Cisco wants to keep the certifications current and fresh.  While Routing & Switching and Security see content refreshes, they are still largely the same at the core.  I would argue that the CCIE Collaboration will be a different exam in feel, even if not in blueprint or technology.  The focus on IM, presence and video means that there’s going to be an entirely different tone.  Cisco wants to be sure that the folks displaying the credential are really certified to work on it according to the test objectives.  I can tell you that there was serious consideration around allowing Storage candidates to take some sort of upgrade exam to get to the CCIE Data Center, but it looks like that was ultimately dropped in favor of making everyone go through the curriculum.  The retirement of the CCIE Voice doesn’t make you any less of a CCIE.  Like it or not, it looks like the only way to earn the CCIE Collaboration is going to be in the trenches.

It Ain’t Over Until…

The sunsetting officially starts on November 20th, 2013.  That’s the last day to take the CCIE Voice written.  Starting the next day (the 21st) you can only take the Collaboration written exam.  Thankfully, you can use either the Voice written or the Collaboration written exam to qualify for either lab.  That’s good until February 13, 2014.  That’s the last day to take the CCIE Voice lab.  Starting the next day (Valentine’s Day 2014), you will only be able to take the Collaboration lab exam.  If you want to get an idea of what is going to be tested on the lab exam, check out the document on the Cisco Learning Network (CCO account required).

If you’d like to read more about the changes from professional CCIE trainers, check out Vik  Malhi (@vikmalhi) on IPExpert’s blog.  You can also read Mark Snow’s (@highspeedsnow) take on things at INE’s blog.

Tom’s Take

Nothing lasts forever, especially in the technology world.  New gadgets and methods come out all the time to supplant the old guard.  In the world of communications and collaboration, Cisco is trying to blaze a trail towards business video as well as showing the industry that collaboration is more than just a desk phone and a voice mailbox.  That vision has seen some bumps along the way but Cisco seems to have finally decided on a course.  That means that the CCIE Voice has reached the apex of potential.  It is high time for something new and different to come along and push the collaboration agenda to the logical end.  Cisco has already created a new CCIE to support their data center ambitions.  I’m surprised it took them this long to bring business video and non-voice communications to the forefront.  While I am sad to see the CCIE Voice fade away, I’m sure the CCIE Collaboration is going to be a whole new barrel of fun.

Restricted CUCM – Rated R

If you’ve gone to download Cisco Unified Communications Manager (CUCM) software any time in the past couple of years, you’ve probable found yourself momentarily befuzzled by the option to download one of two different versions – Restricted and Unrestricted.  On the surface, without any research, you might be tempted to jump into the Unrestricted version. After all, no restrictions is usually a good thing, right?  In this case, that’s not what you want to do.  In fact, it could cause more problems than you think it might solve.

Prior to version 7.1(5), CUCM was an export restricted product.  Why would the government care about exporting a phone system?  The offending piece of code is in fact the media and signaling encryption that CUCM can provide in a secure RTP (SRTP) implementation.  Encryption has always been a very tightly controlled subject.  Initially developed heavily in World War II, the government needed to be sure to regulate the use of encryption (and cryptography) afterwards.  Normally, technology export is something that is controlled by the U.S. Department of Commerce.  However, since almost all applications for cryptography were military in nature it was classified as a munition by the military and therefore subject to regulation via the State Department.  And regulate it they did.  They decreed that no strong encryption software would be available to be exported out of the country without a hearing and investigation.  This usually meant that companies created “international versions” that contained the maximum strength encryption key that could be exported without a hearing – 40 bits.  This affected many programs in the early days of the Internet Age, such as Internet Explorer, Netscape Navigator, and even Windows itself.

In 1996, President Bill Clinton signed an order permitting cryptography software export rulings to be transferred to the Department of Commerce.  In fact, the order said that software was no longer to be treated as “technology” for the purposes of determining restrictions for export.  The Department of Commerce decided in 2000 to create new rules governing the export of strong encryption.  These restrictions were very permissive and have allowed encryption technology to flourish all over the world.  There are still a few countries on the Export Restriction list, such as those that are classified as terrorist states or rogue states as classified by the U.S. Government.  These countries may not be the recipient of strong encryption software.  In addition, even those countries that can receive such software are subject to inspection at any time by the U.S. Department of Commerce to ensure that the software is being used in line with the originally licensed purpose.  When you think of how many companies today have a multi-national presence, this could be a nightmare for regulatory compliance.

Cisco decided in CUCM 7.1(5) to create a version of software that eliminated the media and signaling encryption for voice traffic in an effort to avoid the need to police export destinations and avoid spot audits for CUCM software.  These Export Unrestricted versions are developed in parallel with other CUCM versions so all users can have the same functionality no matter their location.  CUCM Unrestricted versions do have a price when you install them, however.  Once you have upgraded a cluster to an Unrestricted version of CUCM, you can never go back to a Restricted (High Encryption) version.  You can’t migrate or insert any Restricted servers into the cluster.  The only way to go back is to blow everything away and reload from scratch.  Hence the reason you want to be very careful before you install the software.

If you’ve been running CUCM prior to version 7.1(5), you are running the Restricted version.  Unless you find yourself in a scenario where you need to install CUCM in a country that has Department of Commerce export restrictions or has some sort of import restriction on software (Russia is specifically called out in the Cisco release notes), you should stay on the Restricted version of CUCM.  There’s no real compelling reason for you to switch.  The cost is the same.  The licensing model is the same.  The only things you lose are the media encryption and the ability to ever upgrade to Restricted version.  Just like when going to the movies, all the good stuff is in the R-rated version.

Tom’s Take

I still get confused by the Restricted vs. Unrestricted thing from time to time.  Cisco needs to do a better job of explaining it on the download page.  I occasionally see references to the Unrestricted version being for places like Russia, but those warnings aren’t consistent between point releases, let along minor upgrades and major versions.  I think Cisco is trying to do the right thing by making this software as available to everyone in the world as they can.  With the rise of highly encrypted communications being used to launch things like command and control networks for massive botnets and distributed denial of service campaigns, I don’t doubt that we’ll see more restriction on cryptography and encryption coming sooner or later.  Until that time, we’ll just have to ensure we download the right version of CUCM to install on our servers.

On Demand Auto Attendant for CallManager Express


I’ve done my fair share of CallManager Express (CME) installations over the years, many of which were for small businesses.  I usually get to try and replace an old battleship of a phone system that has been running for a long time but has either finally given up the ghost or can’t be repaired due to the company being out of business.  When I do replace these units, the usual desire is to make it behave the same way as the old system.  For the most part, this is a pretty easy proposition.  That is, until it comes to auto attendants.  The automated recording that helps callers find the correct extension or leave a message is becoming an important part of the small business as employers start cutting back on expenses and use fewer people and more technology.  One case recently that had me baffled was a request for an on-demand auto attendant.

This particular customer had an old phone system that had finally failed.  They had decided on a CME system to replace it.  One feature they said they could not live without was the ability to toggle on a recording to handle calls.  This usually happened during lunch or during a meeting when all people at the office would be involved in some manner or another.  The receptionist wanted to push a button and enable the recording until the meeting or lunch had passed, then come back and toggle off the recording to allow calls to be answered by a human being again.  I nodded along slowly as the wheels started turning, because to my knowledge there was no feature inherent to the system that would do this.

After some thinking and planning and more than a few failed lab mockups, I finally found the answer in a combination of unlikely related features.  The first involved handling incoming calls to multiple phones in a manner that would allow redirection of calls.  This isn’t possible with parallel hunt groups in CME, as logging a phone into a hunt group changes all the forwarding behaviors of the phone.  It will only obey the hunt list settings and ignore almost everything else, include call-forward all.  The second issue was finding a way to have the auto attendant answer the call when invoked, as the standard method of using auto attendants either involve enabling it for all calls at all times or using a schedule to enable specific greetings after hours or on holidays.  As an aside, this is the real value in a solutions integrator.  It’s easy enough to check a few boxes and type a few lines to get something to work the way it says it will on the box.  A real integrator will make a system behave how the user wants it to behave, regardless of whether or not there’s a checkbox to do it.

Step 1: Fix Incoming Call Behavior

This ended up being the most technology-dependent part of the equation.  CME used to have a hard time handling a parallel (or broadcast) hunt group that rang a group of phones at one time.  Prior to CME 4.3, this feature was only available for SIP phones.  After 4.3, Cisco finally ported the parallel hunt group to SCCP phones (my preferred method for configuring phones in CME).  The only catch was that the phone hunting behavior followed the rules for hunt groups.  In order to make the incoming calls do something else, I had to find a way to make the calls ring multiple phones without a hunt group.  The answer actually came to me when I found an old page referencing a hacked together broadcast hunt group prior to CME 4.3.  This ingenious solution used a group of overlaid directory numbers (DNs) to mimic a broadcast hunt group.  A group of DNs was necessary because a DN in CME can only be single or dual-line.  With a dual line phone, two calls can hit the phone at once.  The third call is forced off to voice mail or some other behavior as dictated by the call forwarding configuration.  The second part of this solution was delivered in CME 4.0 – the octo line.

For those not familiar, the octo line creates a special DN capable of handling eight simultaneous incoming and outgoing calls across multiple extensions.  This looks to me like an attempt to create a basic form of call queuing in CME.  By creating a construct to handle more than two calls at once, you’ve in effect created something to can do basic call center call routing.  In this case, I created one octo-line DN and put it on the two phones used by reception at this business:

ephone-dn  1 octo-line
  number 100
  description Outside Call
  name Outside Call

Now I can make the calls ring on two phones without creating a hunt group.  That also means I can call-forward the phones as needed.

Step 2: Invoke Auto Attendant On Demand

This one was a bit trickier.  Enabling an auto attendant for a dialed number is easy.  How do we make that number only work when toggled?  Time schedules were out for this customer, as they were never sure when they were going to need to enable the auto attendant.  That means I have to find a way to call the auto attendant DN when needed.  But how to do that on CME?  The answer came to me in a flash of insight – night service.

Night service is a configuration setting that allows a system to be configured for a time schedule when the participating phones will ring in a special manner or pattern.  The idea is that when a business is closed, a designated phone can be monitored by personnel, such as janitorial staff or second shift, and be answered without modifying the open hours configuration.  In this case, we’re going to use the night service code to invoke the night service configuration when needed.  Normally, this command would be used when night service is active in order to disable it.  Here, we’re doing the exact opposite.  Also one more thing to note – the night service code command requires the code to be prefixed with an asterisk.  That works well, as the asterisk isn’t usually dialed as part of a number, so this signals that it’s something special.  I usually use either the extension number (as below) or the last four digits of the main telephone number as a mnemonic trigger.  The first part of the config is easy:

  night-service code *100

Now, we need to go back to the octo-line DN that we previously configured and add an additional setting to control the night service function.  In this instance, I’m using 501 as the pre-configured auto attendant dial-in number:

ephone-dn 1 octo-line
  call-forward night-service 501

The only remaining task to make this a true “push button” service is to enable a speed dial on the ephone itself.  That part is also easy:

ephone 1
  speed-dial 1 *100 label Auto Attendant

Now all the user needs to do is push the button on their phone labeled “Auto Attendant” and it will enable night service for all incoming calls.  Pushing the button again will disable it.  You can also add the command night-service bell to the ephone-dn in order to display a message that night service is active.

There are a number of other tricks that you can do with the basic building blocks presented by CME to make it behave just like a customer’s old phone system.  This should allow you to ease any transition and allay any fears they might have.  After all the users are comfortable with the new phones and phone behavior, you can start introducing new features to them like unified messaging or single number reach.  People are very open to change once they figure out nothing has really changed.

Cisco – Borderless Speed Dating

The first presentation of the final day of Network Field Day 4 brought us to the mothership on Tasman Drive.  The Cisco Borderless team had a lineup of eleven different presenters ready to show us everything they had.  For those of you not familar with the term, Borderless Networks inside Cisco essentially means “everything that isn’t data center or voice.”  Yeah, that means routing and switching and security and wireless and everything else.  That also meant that we got a very diverse group of people presenting to us and a lot of short twenty minute videos of their products.  In a way, it’s very much like speed dating. With little time to get the point across, you tend to shed the unnecessary pleasantries and get right to the important stuff.

First up was the UCS team with new E-series servers.  These are blades that are designed to slide into a ISR G2 router and provide a full-featured x86 platform.  It’s a great idea in search of an application.  I can still remember the AxP modules and how they were going to change my life.  That never really materialized.  The payoff use case that you are looking for is the second video above.  Cisco is starting to push for the idea that you can contain a whole branch office in a single router and run not only the phone system and networking routing and VPN, but now a light-duty server as well.  I’m not sure how many people will be looking to do that with virtualized server resources residing in the data center, but there was some discussion of using this a temporary failover type of environment to push the branch server to the edge in the event of some kind of disaster or outage.  That might work better to me that running the entire branch on the router.  Of course, as you can tell, the demo gremlins found Cisco as well.

The next presentation was the new darling Cloud Services Router (CSR) 1000v.  This little gem got some face time on stage with John Chambers at Cisco Live this year.  It’s a totally virtualized router (hence the “v”) that can move workloads into the cloud when needed.  I’m really curious as to why this is included with Borderless, as this is a very data center specific play right now.  I know that Cisco is pushing this device currently as a VPN concentrator or MPLS endpoint for WAN aggregation.  It makes more sense from some of their diagrams to have it running inside a cloud provider network carving up user space.  I’m going to keep an eye on this one to see where the development goes.

Now, we get to something fun.  Cisco FlexVPN is what happens when someone finally took a look at all the different methods for configuring VPNs on the various Cisco devices and said “WTF?!?”  FlexVPN utilizes IKEv2 to help speed configuration.  You can watch the short video and see all the stuff that we have to deal with to configure a VPN today.  Cisco finally took our complaints to heart and made things a lot more simple.  Of course there are drawbacks, and with FlexVPN that means it only works with IKEv2.  There’s no backwards compatibility.  Of course, if you’re going to have to be migrating everything anyway, you might as well make a clean break and rebuild it right.  That’s going to make things like hub-and-spoke VPN configuration a whole lot less painful in the near future.  Props to Cisco for fixing a pain point for us.

Okay, so maybe a I lied just a bit.  Since Cisco Unified Border Element runs on a router (even though it’s technically voice), we got a presentation about it!  I was in hog heaven here.  If you are looking at deploying a SIP trunk, you had better be looking at a CUBE box to handle the handoff.  Don’t think, just do it.  Listen to the voice of Amy Arnold (@amyengineer) and Erik Peterson (@ucgod).  You need this.  You just don’t know how much until you start banging your head against a wall.

More Voice!!!  By this point, I was practically crying tears of joy.  Two voice presentations in one day.  At a networking event no less!  This presentation on enhanced SRST shows how big of kludge SRST really is.  I’m not a huge fan of it, but I have to configure it to be sure that the phone systems work correctly in the event of a WAN outage.  It’s all still CLI and very annoying to configure and keep in sync.  Thankfully, with the ESRST manager highlighted in the video above, we can keep those configurations in sync and even have it automagically pull the necessary configurations out of CUCM.  This software runs on a Service Engine right now in the router, but I can’t wait to see if Cisco ports it to a virtual setup to run under a CUCMBE 6000 server or even on a UCS-E blade down the road.  Anything that I can do to make SRST less painful is a welcome change.

Okay, this had to be one of the more interesting presentations I’ve been involved in at an NFD event.  We got our AppNav presentation over Webex from a remote resource.  I know this a hot thing to do at Cisco offices to make sure we have the most talented people giving us the most up-to-date info about a particular subject.  However, I expect this when I’m in the middle of nowhere Oklahoma, not at the mothership in San Jose.  The Webex cut out now and then and there were times when we had to strain to hear what was being said in the room.  Looking back at the video, I marvel that the room mikes picked up as much as they did.  As for AppNav itself, it’s a virtual DC version of the Wide Area Application Services (WAAS).  My grasp of WAN acceleration isn’t as good as it should be, even from Infineta back at NFD3.  There’s some good info in here I’m sure.  I’m just going to have to go back and digest it to see where it fits into my needs.

Now it’s time for some switching talk.  We got a roadmap on the Catalyst line.  There are some interesting tidbits in the slides, such as a monster 9000W power supply for the 4500 to support UPoE (more on that in a minute).  The 4500 is also going to get VSS support and ISSU support.  Those two things alone are going to make me start considering the use of the 4500 in the core of most of my smaller networks.  The fixed configuration Catalyst switches also have some nice roadmaps, including UPoE support and lots of IPv6 enhancements.  As I move forward in 2013, I’m planning on doing a lot with IPv6, so knowing that I’m going to have switching support behind me is a nice comfort.  Of all the updates, the most talked about one was probably the Catalyst 6500.  A switch that has been rumored to be on the chopping block for many years now, the venerable Cat6K is getting more updates, including FabricPath support and 100Gig module support.  I think this switch may outlast my networking career at this rate.  There are lots of rumors as to why Cisco is renovating this campus core stalwart once more, but it’s clear that they are attempting to squeeze as much life out of it as they can right now.  To me, the idea of stretching FabricPath down into the campus presents some very tantalizing opportunities to finally get rid of spanning tree on all but the user-facing links.  Let’s hope that the Cat6k sticks around long enough to get a gold watch and a nice pension for all the work it’s given us over the years.

Our next discussion was around security and using Cisco TrustSec to do things a little differently that we’re used to.  By now, I think everyone has talked your ear off about BYOD.  Even I’ve done it a couple of times.  It’s a real issue for people in the dark security caves because our traditional methods of access lists and so forth don’t work the same way when you’ve got employees bringing their own laptops or asking you to give them access to data from tablets or phones.  What this has morphed into is a need to do more role-based authorization.  That’s what TrustSec means to me.  Of course, a lot of previous attempts to do this, like NAC, haven’t really hit the mark or have been so convoluted that it was almost impossible to get them working correctly.  Today, Cisco has rolled all the functionality of NAC and ACS into the Identity Services Engine (ISE).  I’ve had a very brief encounter with ISE, so I know it has a lot of potential.  I want to see how Cisco will incorporate it into the bigger TrustSec picture to make everything work across my various platforms.

Time to turn up the juice.  Cisco brought out Universal Power over Ethernet (UPoE), which is their solution to pump up to 60 watts of power across a standard Ethernet cable to power…well, whatever it is that eats 60w of power.  Cisco’s doing this by taking 802.3at PoE+, which can pump 30w down the cable, and pushing an additional 30w of power down the other unused pairs.  Interestingly, Cisco talked to the people behind the ISO and EIA/TIA standards and found that when you have a bunch of unstructured cables running about around 50 watts (which is the 60w number above minus cable loss), you get a temperature in the cable bundle about 8-10 degrees above the ambient room temperature.  In reality, this means that 60w is the max amount of power you’re likely to ever get out of a Cat5e cable unless you chill it or have some kind of new material that can reduce the heating effect.  Cisco seems to be targeting UPoE to drive things like monitors, thin client desktops, and even those crazy command center touch pads that you see littered across the floor of a trading house or stock exchange.  This last item really makes me believe that UPoE is going to be positioned in the same vein as the ultra-low latency Nexus 3548 – financial markets.  Thin clients and command center touch panels are likely to be the kind of mission-critical devices these companies are willing to pay big buck to power.  With the above-mentioned 9000w PS for the Catalyst 4500, you can see why we’re going to soon need to put a nuclear reactor in to drive these things.

Cisco Smart Operations dropped by to talk to us about Cisco Smart Install.  This is the feature that I tend to turn off when I see it by the telltale sign of “Error opening tftp://”  The Smart Operations team is doing its best to create an environment where an IT department that doesn’t have the headcount to send technicians to deploy remote site switches can leverage software tools to have those devices auto-provision themselves.  You can also configure them to automatically configure things like Smartport roles, which has never really been one of my favorite switch features.  Overall, I can appreciate where Cisco is wanting to go with this technology.  But, as a CLI jockey, I’m still a bit jaded when it comes to having part of my job replaced by a TFTP script.

The final Cisco NFD4 presentation was about application visibility and control.  This is a lot of the intelligence that is built into the Cisco Prime monitoring software that was demoed for us back at NFD3.  If you can identify the particular fingerprints of a given application, such as Telepresence, you can better determine when those fingerprints are out of whack.  I’m also excited because fingerprinting apps is going to be a huge part of security in the near future, as evidenced by Palo Alto’s app-based firewall and the others like Sonicwall and Watchguard that have followed along.  Even the Cisco ASA-CX is starting to come around to the idea of stopping apps and not protocols.

If you’d like to learn more about Cisco Borderless Networks, check them out at  You can see an archive of the presentations and associated data sheets at  You should also follow the Cisco Borderless team on Twitter as @CiscoEnterprise and @CiscoGeeks.

Tom’s Take

There you have it.  Lots of presenters.  Hours of video.  A couple of thousand words from me on all of it.  It’s almost exhausting to see that much information in a short span of time.  Some of the things that Cisco did with this presentation were great.  There were technologies that only needed a bit of time.  There were others that we could have spent an hour or more on.  I think that the next NFD presenters that want to try something along these lines should setup the first three hours with rapid fire presentations and reserve the last hour for us to call back to earlier presenters and hit them with additional questions.  That way, we don’t run out of time and we get to talk about the things that interest us the most.  Bravo overall to the Cisco Borderless team for breaking out of the mold and trying something new to keep the NFD delegates hooked in.

Tech Field Day Disclaimer

Cisco was a sponsor of Network Field Day 4.  As such, they were responsible for covering a portion of my travel and lodging expenses while attending Network Field Day 4.  In addition, they provided me with an 8GB USB drive with marketing collateral and data sheets. They did not ask for, nor where they promised any kind of consideration in the writing of this review.  The opinions and analysis provided within are my own and any errors or omissions are mine and mine alone.

What’s My Cisco ATA Second Line MAC Address?

In the world of voice, not everything is wine and roses.  As much as we might want to transition everything over to digital IP phones and soft clients, the fact remains that there are some analog devices that still need connectivity on a new phone system.  The more common offender of this is the lowly fax machine.  Yes, even in this day and age we still need to rely on the tried-and-true facsimile machine to send photostatic copies of documents across the PSTN to a waiting party.  Never mind email or Dropbox or even carrier pigeon.  Fax machines seem to be the most important device connected to a phone system.  Normally, I leave the fax connections and their POTS lines intact without touching anything.  However, there are times when I don’t have that luxury.

In the case of the Cisco VoIP systems, that means relying on the Analog Terminal Adapter, or ATA.  The ATA allows you to connect an analog device to the unit, whether it be a fax machine or a cordless analog phone or even a fire alarm or postage machine.  It has many uses.  The configuration of the ATA is fairly straightforward under any CUCM system.  However, if you have a multitude of analog devices that you need to connect, you might opt to use the second analog port on the ATA.  The ATA 186 of the past and its current replacement, the ATA 187, both have 2 analog ports on the back.  There’s only one Ethernet port, though.  This is where the interesting part comes in to play.  If there are two analog ports but only one Ethernet port, how to I configure the MAC address for the second port?  All phone devices in CUCM must be identified by MAC address.  On an ATA, the primary MAC address printed on the bottom or the side of the box is the address for the first port.

If you want to use the second port, you’re going to have to do a little bit of disassembly.  Cisco uses a standard method to create a new MAC address:

1.  Take the MAC address for port 1.  For example, 00:00:DE:AD:BE:EF.

2.  Drop the first two digits from the MAC address.  In the example, 00:DE:AD:BE:EF.

3.  Append “01” to the end of the 10-digit address.  Example, 00:DE:AD:BE:EF:01.

Once you’ve completed those steps, take the MAC address you’ve just created and plug it into CUCM as a new ATA device.  Once you’ve completed the necessary steps to create the new device, it will register with the DN you’ve assigned to it.  Then you can start calling or faxing it to your heart’s content.

There’s no mention of the secondary MAC address anywhere on the web interface.  You’d figure it wouldn’t be hard to write some HTML function to read the MAC address and do the above operation.  The Cisco documentation buries this information deep inside the setup document.  I’ve even search Cisco’s very own support forums and found all manner of advice that doesn’t work correctly.  I decided that it was time to jot this information down in a handy place for the next time I need to remember how to configure the ATA’s second port.  I hope you find it useful as well.

A Talk in the Park – Using Call Park and Directed Call Park

Anyone that has ever used a phone is familiar with being placed on hold.  Most of the time, you get to hear nice, soothing music while the person on the other end of the line tries to figure out something without either shouting into the phone or having a long period of uncomfortable silence.  The hold button is usually the most worn-out button on the key systems that I replace.  However, on the newer PBXs that I install, the hold button is quickly losing its usefulness.  Hold work well when every line on your phone system is on every phone, like it is in a key system.  Placing Line 1 on hold at the reception desk phone allows Line 1 to be picked up by the CEO at their phone.  However, what happens in a PBX environment when the incoming lines don’t appear on every phone?  The hold button will still place the caller on hold, but only on the phone where the call was initially held.  In order to move that caller to another phone, you’ll need to transfer the caller or have the CEO come up to the reception desk.  These may not be the most effective solutions for most people.  What if there was another way?

My first experience with a “modern” PBX was with the call park feature.  Rather than relying on the hold button and tying up the lines coming into the building, the park button takes a different approach.  When a caller wants to speak to someone other than the person that they called, the receiver of the call can “park” the caller.  Parking a call is like placing the call on hold, but on a phantom extension that can be accessed system-wide.  Now, rather than having the CEO go to the reception desk to retrieve the call, the CEO can dial an extension and retrieve the parked call whenever it’s convenient.  Call park is a great solution for places where not everyone has a phone or usually isn’t near their extension.  Think of a warehouse or a retail sales floor.  These users may not have ready access to a phone to take a call.  It’s better for them to find an extension and take the call when possible.  That’s where the genius of call park comes into play.  Without a doubt, call park is the number one feature on my office phone system.  If it stopped working for some reason, there might just be a riot.  For users that don’t use it already, telling them about the feature when I’m doing the installation is like a ray of sunshine for them.

Configuring Call Park

I’m going to show you how to configure Call Park on Cisco equipment, seeing as that’s the one that I work on most of the time.  Your mileage may vary on your flavor of system.  If you’re using Cisco Unified Call Manager:

1.  After logging into the system, head to Call Routing -> Call Park.

2.  You’ll see a screen that looks like this:

The Call Park Number/Range field can accept either a single park number or a range (using the same X wildcard as a route pattern).  I’d recommend a range to give yourself some flexibility.  Be aware, though, that the limit for a single range of park slots is 100.  If you need more than that, you’ll need to create a different pattern.  I usually set aside 10 or so.  The description field is pretty self-evident.  The partition should be one that’s dialable from the phones that you want to access the park feature.  I usually just put it in my cluster resources or internal DNs partition.  The CUCM field gives you a bit of control over which cluster you assign the park slots.

3.  Once you’ve created the park slots, be sure to check the Phone Button Template that the phones are using to ensure there is a Park softkey available for use by the users.  I tend to move the key to the first row of softkeys to ensure that it gets used instead of the hold button.  Just be aware that changing the softkey template will require you to restart the phones to make the settings take effect.  When your users press the Park softkey, the system will pick the first open park slot ascending in the park pattern created and leave the call there.  The screen will display the park slot that is holding the call for about ten seconds.  You can tweak this timer under System -> Service Parameters -> Cisco CallManager

If you find yourself on a CUCME system, configuring a park slot is as easy as this:

ephone-dn  40 
 number 601 secondary 600 
 park-slot timeout 60 limit 10 
 no huntstop 
ephone-dn  41 
 number 602 secondary 600 
 park-slot timeout 60 limit 10 
 no huntstop 
ephone-dn  42 
 number 603 secondary 600 
 park-slot timeout 60 limit 10 
 no huntstop 
ephone-dn  43 
 number 604 secondary 600 
 park-slot timeout 60 limit 10 
 no huntstop

This will setup four park slots with their own number and a shortcut.  One other quick note here.  If you accidentally configure an ephone-dn as a park slot and later need to use it for a phone DN, you’re going to need to remove the whole DN and add it back in with the right configuration.  For whatever reason, marking a DN as a park slot is one-way job until it’s been deleted.

Directed Call Park

As much as I love call park, it does have one downside.  Once a call has been parked in a call park slot, there’s no real way to monitor it.  The call park slot is basically a phantom extension with no way to watch what’s going on.  While that may be fine for a small office with five or six slots, what happens when an enterprise with thirty slots needs a little more control?  What if you want to ensure that you always park an executive’s calls on the same slot?  You can’t do that with a regular call park slot.  That’s where directed call park comes into play.  Directed Call Park allows you to designate a range of park extensions that can be monitored via Busy Lamp Field (BLF) buttons.  You can also use those same BLF buttons as a speed dial to rapidly park calls in the same slots every time.  This makes a lot more sense for a large enterprise switchboard.  The configuration is very similar, with only a couple of extra fields:

Most of it looks the same.  The new fields include the reversion number and CSS.  This is where the call will be sent if no one picks it up in a certain period of time.  Normal call park sends the call back to the extension that parked it.  With directed call park, you will usually want the call to go to a central switchboard or receptionist.  If you leave these optional fields blank, it will behave just like the normal call park slot.  You’ll also notice that the Retrieval Prefix is a required field.  Directed Call Park requires you to prefix the park slot with a code for access.  If you don’t include the prefix, the system does nothing, as it thinks you’re trying to park a non-existant call in an occupied park slot.  If your call is parked on 601 and the retrieval prefix is 55, you will need to dial 55601 to pick up the call.  When you want to park a call in a directed call park slot, you need to do a consultative transfer to that slot.  In the above case, transfer the call to 601, then hit transfer again to send the caller to the park slot.  The Park softkey doesn’t do anything here for directed call park and in fact will send the caller to a regular park slot if they are configured.

Tom’s Take

Call park is a make-or-break feature for me.  I always talk about it in the phone system training that I give to people when first setting up their systems.  I caution them against using hold any longer.  The only time I use hold on my own phone is when I’m looking something up or I need to step away from the phone for a few seconds.  I treat the hold button like a mute button with music.  Call park is the new hold.  Call park gives you everything that the hold button ever could and more.  You can move calls where you need them to be without worrying about which phone has the call.  When you add in directed call park to give your switchboard the flexibility to monitor calls and control where calls are parked people will being to wonder how they ever lived without it.  You may even find that you can remove the Hold softkey from your phone button templates.  And then your job really will be a walk in the park.

The Card Type Command – Don’t Flop

If you’ve ever found yourself staring at a VWIC2-1MFT-T1/E1 or a NM1-T3/E3 module, you know that you’ve got some configuration work ahead of you.  Whether it be for a PRI circuit to hook up that new VoIP system or a DS3 to get a faster network connection, the T1/T3 circuit still exists in many places today.  However, I’ve seen quite a few people that have been stymied in their efforts to get these humble interface cards connected to a router.  I have even returned a T1/E1 card myself when I thought that it was defective.  Imagine the egg on my face when I discovered that the error was mine.

It turns out that ordering the T1/E1 or T3/E3 module from Cisco requires a little more planning on the installation side of things.  These cards can have a dual identity because the delivery mechanism for these circuits is identical.  In the case of a T1/E1, the delivery mechanism is almost always over an unshielded twisted pair (UTP) cable.  Almost all of the T3/E3 circuits that I’ve installed have been delivered over fiber but terminated via coax cables with BNC connectors.  The magic, then, is in the location.  A T1 circuit is typically delivered in North America, while the E1 circuit is European version.  There are also differences in the specifics of each circuit.  A T1 is 24 channels of 64kbits each.  An E1 is 32 channels of the same size.  This means that a T1 has an effective data rate of 1.544 Mbits while an E1 is a bit faster a 2.048 Mbits.  There are also framing differences and a slightly different signaling structure.  The long and short of it is that T1 and E1 circuits are incompatible with each other.  So how does Cisco manage to ship a module that supports both circuit types?

The key is that you must choose which circuit you are going to support when you install the card.  The card can’t automatically flip back and forth based on circuit detection.  Where the majority of issues come from in my line of work is that the card doesn’t show up as a configurable interface until you force a circuit type.  This is accomplished by using the card type command:

RouterA(config)#card type ?
 e1 E1
 e3 E3
 t1 T1
 t3 T3

Choose your circuit type and away you go!  As soon as you enter the card type, the appropriate serial interface is created.  You will still need to enter the controller interface to set parameters like the framing and line code.  However, the controller interface only shows up when the card type has been set as well.  So unless you’ve done the first step, there isn’t going to be a place to enter any additional commands.

Tom’s Take

Sometimes there are things that seem so elementary that you forget to do them.  Checking a power plug, flipping a light switch, or even remembering to look for little blinking lights.  We don’t think about doing all the easy stuff because we’re concentrating on the hard problems.  After all our hard work, we know it has to be something really messed up otherwise it would be fixed by now.  In the case of T1/E1 cards, I made that mistake.  I forgot to check everything before declaring the card dead on arrival.  Now, I find myself spending a lot of time providing that voice of reason for others when they’re sure that it has to be something else.  The little voice of reason doesn’t always have to be loud, sometimes it just has to say something at the right time.

VADD – Video Attention Deficit Disorder

While I was at Cisco Live, I heard a lot about video.  In fact, “video is the new voice” was the center square on my John Chambers Keynote Bingo card.  With the advances that Cisco has been making with the various Jabber clients across all platforms, Cisco really wants to drive home the idea that customers want to use video in every aspect of their life.  This may even be borne out when you think about all the social networks that have been adding video capabilities, such as Facebook or Skype.  Then there’s the new launch of AirTime from the guys that brought you Napster.  AirTime is a social network that is built entirely around video and how you can interact with complete strangers that share your interests.

I started thinking about video and the involvement that it has in everyone’s life today.  It seems that everything has a video-capable camera now.  Mobile phones, tablets, and laptops come standard with them.  They are built into desktop monitors and all-in-one computers.  It seems that video has become ubiquitous.  So too have the programs that we use to display video.  I can remember all the pain and difficulty of trying to setup programs like AIM and Yahoo! Messenger to work with a webcam not all that long ago.  Now we have Skype and Facetime and Google+ Hangouts.  On the business side we have things like Cisco Jabber for Telepresence (formerly Movi) and Webex.  I even have a dedicated video endpoint on my desk.  However, the more and more I thought about it, the more that I realized that I hardly used video in my everyday life.  I’ve done maybe two Facetime calls with my family since my wife and I purchased Facetime-capable devices last year.  My Skype calls never involve video components.  My Webex sessions always have video features muted.  Even my EX90 gathers dust most of the time unless it gets dialed to test a larger Telepresence unit.  If video is so great, why does it feel so neglected?

For me, the key came in an article about AirTime.  In the press conference, the founders talked about how social media today consists of “asynchronous communications”.  We leave messages on walls and timelines.  We get email or instant messages when people try to communicate with us that sit there, beckoning to us to respond.  In some cases, we even have voicemail messages or transcriptions thereof that call to our attention.  The AirTime folks claim that this isn’t a natural method of communication and that video is how we really want to talk to people.  Nuances and body language, not text and typing.  That’s a good a noble goal, but when I thought about how many Facetime devices are out there and how many people I knew with the capability that weren’t really using it, something didn’t add up.  Why does everyone have access to video and yet not want to use it?  Why do we prefer to stick to things like Twitter timelines or instant messages via your favorite service?

I think it’s because people today have Video Attention Deficit Disorder (VADD).  People don’t like using video because it forces them to focus.  Now that all my communications can happen without direct dependence on someone else, I find my attention drifting to other things.  I can fire off emails or tweets aimed at people I want to communicate with and go on about my other tasks without waiting for an answer.  Think about how easy it is to just say something via instant message versus waiting for a response in real time.  Twitter doesn’t really have awkward silence during a conversation.  Twitter doesn’t require me to maintain eye contact with the person I’m talking to, and that’s when I can even figure out if I’m supposed to be looking at the camera or the eyes of the projected video image.  When I’m on a video camera, I have to worry about how I look and what I’m doing when I’m not talking to someone.  Every time I watch a Google+ hangout that consists of more than two or three people, I often see people not directly speaking having wandering attention spans.  They look around the room for something to grab their attention or get distracted by other things.  That’s why asynchronous communication is so appealing.  I can concentrate on my message and not on the way it’s delivered.  In real-time conversations, I often find myself subconsciously thinking about things like making eye contact or concentrating on the discussion instead of letting my focus drift elsewhere.  Sometimes I even miss things because I’m more focused on paying attention than what I should be paying attention to.  Video conversation is much the same way.  Add in the fact that most conversation takes place on a computer that provides significant distraction and you can see why video is not an easy thing for people like me.

Tom’s Take

I’ve wanted to have a video phone ever since I first watched Blade Runner.  The idea that I can see the person that I’m talking to while I converse them was so far out back then that I couldn’t wait for the future.  Now, that future is here and I find myself neglecting that amazing technology in favor of things like typing out emails and tweets.  I’d much rather spend my time concentrating on the message and not the presentation.  Video calling is a hassle because I can’t hide anymore.  For those that don’t like personal interaction, video is just as bad and being there.  While I don’t deny that video will eventually win out because of all the extra communication nuances that it provides, I doubt that it will be anytime soon.  I figure it will take another generation of kids growing up with video calling being ubiquitous and commonplace for it to see any real traction.  After all, it wasn’t that long ago that the idea of using a mobile phone not tied to a landline was pretty far fetched.  The generation prior to mine still has issues with fully utilizing those types of devices.  My generation uses them as if they’d always been around.  I figure my kids will one day make fun of me when they try to call me on their fancy video phone and their dad answers with video muted or throws a coat over the camera.  If they really want to talk to me, they can always just email me.  That’s about all the attention I can spare.

Upgrading to Cisco Unified Presence Server 8.6(4) – Caveat Jabber

With the Jabber for Everyone initiative that Cisco has been pushing as of late, the question about compatibility between the Jabber client and Cisco Unified Presence Server (CUPS) has come into question more than once.  Cisco has been pretty clear on the matter since May – you must upgrade to CUPS 8.6(4) [PDF Link] to take advantage of the Jabber for Everyone.  This version was released on June 16th, and being the diligent network engineer that I am, I had already upgraded to 8.6(3) previously.  This week I finally had enough down time to upgrade to 8.6(4) to support Jabber for Everyone as well as some of the newer features of the Jabber client for Windows.  Of course, that’s where all my nightmares started.

I read the release notes and found that 8.6(4) was a refresh upgrade.  I’ve already done one of these previously on my CUCM server so I knew what to expect.  I prepped the upgrade .COP file for download prior to installing the upgrade itself.  Luckily for me, 8.6(3) is the final version prior to the refresh upgrade, so it doesn’t require the upgrade .COP file to perform the upgrade.  The necessary schema extensions and notification fields are already present.  With all of the release note prerequisites satisfied, I fired up my FTP server and began the upgrade process.  As is my standard procedure, I didn’t let the server upgrade to the new version automatically.  I figured I’d let the upgrade run for a while and reboot afterwards.  After a couple of hours, I ordered the server to reboot and perform the upgrade.  Imagine my surprise when the server came back up with 8.6(4) loaded, but none of the critical services were running.  Instead, the server reported that only backups and restorations were possible.  I was puzzled at this, as the upgrade had appeared to work flawlessly.  After tinkering with things for a bit, I decided to revert my changes and roll back to 8.6(3).  After a quick reboot, the old version came back up.  Only this time, the critical services were stuck in the “starting” state.  Seemed that I was doomed either way.  After I verified the MD5 checksum of the upgrade file, I started the upgrade for the second time.  While I waited for the server to install the second time, I strolled over to the Internet to find out if anyone was having issues with this particular upgrade.

After some consulting, it turns out that Cisco pulled a bone-headed mistake with this upgrade.  Normally, one can be certain that any hardware-specific changes will be contained to major version upgrades.  For instance, upgrading from Windows XP to Windows 7 might entail hardware requirement changes, like additional RAM.  Point releases are a little more problematic.  Cisco uses the minor version to constitute bi-annual system releases.  So CUCM 8.0 had a certain set of hardware requirements, but CUCM 8.5 had different ones.  In that particular case, it was a higher RAM requirement.  However, for CUPS 8.6(4), the RAM requirement doubled to 4 GB.  For a sub-minor point release.  Worse yet, this information didn’t actually appear in the release notes themselves.  Instead, I had to stumble across a totally separate page that listed specific hardware requirements for MCS server types.  Even within that page, the particular model of server that I am using (MCS-7825-I3) is listed as compatible (with caveats).  Turns out that any 8.6(x) release is supposed to require more than 4GB of RAM to function correctly.  Except I was able to install 8.6(3) with no issues on 2GB of RAM.  Since I knew I was going to need to test 8.6(4), I rummaged around the office until I was able to dig up the required RAM (PC2-5300 ECC in case you’re curious).  Without the necessary amount of RAM, the server will only function in “bridge mode” for migrations to new hardware.  This means that your data is still intact on the CUPS server, but none of the services will start to begin processing user requests.  At least knowing that might prevent some stress.

For those of you that aren’t lucky enough to have RAM floating around the office and you’ve gotten as far as I have, reverting the server back to 8.6(3) isn’t the easiest thing to do.  Turns out that moving back to 8.6(x) from 8.6(4) requires a little intervention.  As found on the Cisco Support Forums, rolling back can only be accomplished by installing the ciscocm.cup.pe_db_install.cop file.  But there are two problems.  First, this file is not available anywhere on Cisco’s website.  The only way you can get your hands on it is to request it from TAC during a support call.  That’s fortunate, because problem number 2 is that the file is unsigned.  That means that it will fail the installation integrity check when you try to install it on the CUPS server.  You have to have TAC remote connect to the server and work some support voodoo to get it working.  Now, I suppose if you have a way to gain root access to a Cisco Telephony OS shell, you could do something like the outlined steps in the forum post (as follows):

Here what's required to temporarily install unsigned COP files

cd /usr/local/bin/base_scripts

Here what's required in Remote Access to remove the temporary fix

cd /usr/local/bin/base_scripts

Note: This is totally unsupported by me.  I’m putting it here for posterity.  Don’t call me if you blow up your server.  Also, I don’t have the TAC .COP file either, so don’t bother asking for it.

That being said, the above instructions should get you back up and running on 8.6(3) until you can buy some RAM from Newegg or your other preferred vendor.

Tom’s Take

Yes, I should have read the release notes a little more closely.  Yes, I should have verified the compatibility before I ran wild with this upgrade.  However, having fallen on my sword for my own mistakes, I think it’s well within my rights to call Cisco out on this one as well.  How do you not put a big, huge, blinking red line in the release notes warning people that you need to check the amount of RAM in the server before performing an upgrade?  You figure something like this would be pretty important to know?  Worse yet, why did you do this on a sub-minor point release?  When I install Windows 7 Service Pack 1 or OS X 10.7.4, I feel pretty confident that the system requirements for the original OS version will suffice for the minor service pack.  Why up the hardware requirements for CUPS for a minor upgrade at best?  Especially one that you’re driving all your people to be on to support your big Jabber initiative?  Why not hold off on the requirement until the CUCM 9 system release became final (which happened about a week later)?  If I’m moving from 8.6 to 9.0, I would at least expect a bunch of hardware to be retired and for things to not work correctly when moving to a new, big major version.  From now on I’m going to be a lot more careful when checking the release notes.  Cisco, you should be a lot more diligent in using the release notes to call attention to things that are important for that release.  The more caveats we know about up front, the less likely we are to jabber about them afterwards.