Upgrades are a fact of life for network rock stars. Whether we are patching bugs or adding new features to our systems, the installation of software never seems to end. If you are a Cisco voice rock star, you all too often find yourself upgrading to newer releases of Cisco Unified Communications Manager (CUCM) to support new devices like the Cius or fix show stopping bugs like the 180-day uptime lockup. However, if you are a user of CUCM 8.x and you’re trying to move to 8.6, you’ve probably had a couple of head scratching moments so far.
If you’ve popped a freshly burned 8.6(1) ISO into your DVD drive or copied it via SFTP, you kicked off you installation and likely saw the following error message:
09/18/2011 19:31:48 refresh_upgrade|********** Upgrade Failed **********|<LVL::Info> 09/18/2011 19:31:48 refresh_upgrade|*** Please install the Refresh Upgrade COP, and reattempt the upgrade ***|<LVL::Info> 09/18/2011 19:31:48 refresh_upgrade|************************************|<LVL::Info>
Huh? What’s a refresh upgrade? Why isn’t this ISO file working? Well, it turns out Cisco needs you to take an additional step first.
CUCM runs on an operating system. Up until version 5, that was Windows 2000 with some hardening and customizations. Cisco eventually ported CallManager 4.3 to Windows Server 2003, but in the end the decision was made to move to an appliance-based OS that utilized Linux. The Telephony OS in CUCM 5.x was new for those used to working in Windows but somewhat familiar to those that have seen Linux before, even if the login shell looked nothing like bash. Cisco provides patches for the OS with every release of CUCM software and the user never knows what’s going on because of the way the system installs the patches transparently. However, much like the shift from Windows 2000 to Windows 2003, software eventually reaches the end of its life. Development stops on the old version and it’s time to move to the new one. Such is the case in CUCM. With version 8.6, Cisco has moved away from an OS platform based on Redhat Enterprise Linux (RHEL) 4 and upgraded the underlying OS to RHEL 5. This is good news that allows the system to stay current and support a larger variety of hardware. The bad news is that the upgrade of the OS can be a bit destructive. This is part of the reason for the extra steps in moving to CUCM 8.6
Firstly, Cisco wants you to install a special Cisco Options Package (COP) file on 8.5(1) systems. This file is ciscocm.refresh_upgrade_v1.0.cop.sgn. The 8.6 installer checks for the presence of this file and won’t kick off unless it’s present. It needs to be installed on every server in the cluster. It’s also going to reboot the server after installation. As near as I can tell, it makes some changes to the Tomcat service on the server as well as adding two new fields to the Install/Upgrade window:
Notice the new options for email. This allows the server to send you an email whenever the upgrade is completed. Probably a long overdue option that comes in handy for those of us that spend more than a few stress-filled moments clicking the Refresh buttons on our web browsers waiting for CUCM to come back to life after an upgrade. There’s another reason for putting this email field in here now, though.
It turns out that when you upgrade from 8.5 to 8.6, its going to take a while. Quite a while, in fact. The system is going to reboot no less than twice, perhaps even three times. Considering that a CUCM reboot can take 15-20 minutes to complete each time during an upgrade, you’re looking at nearly an hour of rebooting time under certain circumstances. During the upgrade, CUCM is going to do things in 3 phases:
Phase 1: Export all the pertinent CUCM data to a safe partition
Phase 2: Reboot and install RHEL 5, then reboot and install the CUCM applications
Phase 3: Import all the data from the export partition
On the 7825H3 MCS server, there isn’t enough hard drive space to contain the safe partition during the reformat and installation of CUCM 8.6. In that case, you’re going to need to plug a 16 GB USB drive into the system to serve as a target for the data export. If you’re trying to upgrade a CUCM Business Edition system on a 7828H3 server, you better bust out the credit card because you’re going to need a 128 GB USB drive to hold all the CUCMBE data during the upgrade. The IBM servers aren’t affected by this little caveat, as I’ve done the 8.6 refresh upgrade on a 7825I4 and not had any issues. Be sure to leave the USB drives plugged in the whole time the system is upgrading. Also, whatever is on the drive is going to be overwritten without warning, so be sure it’s blank before you start.
After you’ve completed the whole installation with all the reboots, you’re going to have a fresh new system with CUCM 8.6 to support all kinds of wonderful things, like finally being able to use Google Chrome to administer things.
Tom’s Take
I kicked off an upgrade to 8.6 without reading the release notes or documentation. Thankfully Cisco prevented me from screwing things up big time by halting the installation with the above error message. The more I dug into things, the more interesting it was. It also took me two hours to finish things up with many reboots and even more nail biting (Fun fact: I was doing the upgrade during Packet Pushers Show 56, which is one of the reasons why I was quiet – I was trying not to scream at my CUCM server). However, I think I could have avoided some pain and stress if I’d just read the docs first or even searched for refresh upgrade before I got started.
The Networking Nerd 
Great article! What I find interesting as I am looking at installing a new version of Cisco Presence 8.6.3 in a virtualized environment that the OVA template for CUPS 8.6.3 uses Red Hat Enterprise Linux 4 (32-bit). To be consistent, I just wish the same 8.6 versions of CUCM and CUPS would use the same guest OS.
Thanks for the info, question though. Should you run the upgrade on the sub first, or the pub?
Best practice is to upgrade the publisher first so the database stay consistent. The phones fail over to the subscribers during the upgrade, then fail back to the publisher. When they fail back, they get firmware upgrades (unless this have been done ahead of time) and the subscribers can then be upgraded.
Is it even possible to upgrade the Sub before the Pub?
I haven’t checked, but it used to be the case (back in 6.X I think) that you would get an error message if you tried to run the upgrade on the Sub if the Pub hadn’t already been upgraded.
I’ve always followed the same best practice that you outlined above, so was just curious if the ability to upgrade the Sub first was opened up?
Thank you dude !!! This procedure will help me a lot !!!
Pingback: Guia de Upgrade para o CUCM 8.6 « Architecture for Voice, Video and Integrated Data
Pingback: Upgrade Guide to CUCM 8.6(1) « Architecture for Voice, Video and Integrated Data
Thanks for posting this! It gave me some piece of mind last Thursday night when I had an CUCM, CUP and Arc upgrade run over by 6 hours.
Hi, did you have any problems with certificates ( ITL issue ) ?
Nah, I ran into a couple of network issues and a hardware warning which stalled the upgrade on the remote Subscriber.
The upgrade was from 7.1(5) directly to 8.6(2) so didn’t have the ITL issue on this particular upgrade. A colleague of mine ran into it when upgrading from 8.0(4) to 8.6(2) though. He’s still got a couple of phones that won’t take an upgrade even after removing the certs.
Pingback: Upgrading Cisco Unified Communications Manager to 8.6(x) | Adventures in Unified Communications
Pingback: CCIECarl » Blog Archive » 8.6 upgrade
Hi – something to watch out for is the version of the ciscocm.refresh_upgrade_v1.0.cop.sgn file – all release notes for cucm 8.6.x refer to ciscocm.refresh_upgrade_v1.0.cop.sgn. There is a ciscocm.refresh_upgrade_v1.1.cop.sgn available to download, I couldn’t ifnd any information as to when to use this, so went ahead and used the exact version on the release notes: ciscocm.refresh_upgrade_v1.0.cop.sgn. Low and behold the publisher rebooted after the first phase of the upgrade and got caught into a loop of constantly rebooting as it tried to complete the upgrade. I eventually managed to revert to the previous 7.1.3 using the recovery cd and was advised by TAC that since I was using IBM hardware I should have used the ciscocm.refresh_upgrade_v1.1.cop.sgn……. however, once the 1.0 cop file is installed it cannot be uninstalled (i.e. it is used even when the 1.1 cop is installed). The process then is for TAC to gain root access, copy some script files from the sub and then install ciscocm.refresh_upgrade_v1.1.cop.sgn. to allo the upgrade to proceed.
Great Article. When upgraded a 7825H3, does the USB drive need to be used on all the nodes or just when upgrading the Publisher?
I have the same question:
When upgraded a 7825H3, does the USB drive need to be used on all the nodes or just when upgrading the Publisher?
Pingback: Upgrading to Cisco Unified Presence Server 8.6(4) – Caveat Jabber | The Networking Nerd
Pingback: CallManager/Unity shenangigans - dv8tor.com
FYI, for the 7825H3 upgrade the USB is required for each node in the cluster.
In addition be aware that there is a documented bug which I verified with TAC that requires the USB drive to be EXTERNALLY-POWERED – that means no cheap 16GB USB stick.
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCty27638
And of course the smallest drive I could even find with its own power supply was a minimum of 1TB. I am about to do this upgrade today so I will let you know of any other caveats that I run into.
Pingback: Upgrading Cisco Unified Communications Manager to 8.6(x) « Inane Geekery
any of you ever run in to this? I’m a couple of hours in to an upgrade from 8.0 to 8.6. I installed the cop file and have done every thing by the book so far. Starting to panic….
[SSH] Server Version OpenSSH_5.1
[SSH] Logged in (password)
/usr/local/platform/bin/startcliscript.sh: Permission denied
[SSH] INFO: DISCONNECT
Any idea what format the USB drive should have before we begin?
I don’t think it matters. The system will wipe out the USB drive when you start and put down the right format (which I’m pretty sure is ext3).
Thanks for the info and the great article! I hope to finish the upgrade tonight.
At long last, success!!! In addition to being a powered USB drive (which many others have noted), my drive had to be formatted for ext3. ext2, FAT32, and unpartitioned did NOT work. It was only after I attached an ext3 powered USB drive that the upgrade would go. Seems like an easy and obvious thing to put in the documentation.
I am having an awful time with this. The publisher did fine. The sub is frustrating me to death. TAC even appears stumped….—
admin:show cuc version
Active version: 8.0.3.20000-18
Inactive version: 8.6.2ES44.22900-44
admin:utils system switch-version
** There is no inactive side available **
admin:
Pingback: CallManager/Unity shenangigans - dv8tor.com
great read … too bad I only found it after the CUCM went down for one of its reboots and took for ever to came back up… I read this and went to sleep… got an email at about 3 AM confirming upgrade success. I am going to attempt upgrading the CUPS server tonight …. hopefully that works the same way.