Oy.  There’s always one trouble ticket that gives you difficulty and makes you want to throw things around the room.  When you solve it, you yell and dance down the hallway proclaiming how smart you are to have gotten it fixed.  Folks, let me introduce you to that issue.

A Cisco Unified Communications Manager Business Edition (CUCMBE) server started exhibiting strange behavior.  No phones registered and no web GUI.  Not the first time that this has happened, so I’ll just log in via SSH and reboot the server.  When it came back up, nothing.  Same thing.  When I poke around in the CLI, I find out the SSH services are started, but that’s about it.  When I try to start the Tomcat service, which is required for the web GUI, I get an error about the Service Manager not being started.  No problem, I’ll just start that one:

admin:utils service start Service Manager
Aborting servM startup due to invalid configuration files

Oh crap.

Uh, restore from backup?  Hah!  No backup here.  Boot off the recovery CD and check the disk with FSCK (which looks a lot like a curse word I was uttering at this point)?  Fixed a couple of file issues, but still no dice on the services.  No backup partition, as this server had never been upgraded.

Just great.  What now?

Well, if you’re impatient like me when you’re waiting on support engineers to get back with you and you know you’re probably going to have to reload anyway, you can try some crazy things on the off chance they might work.  I mean, what’s the worst that can happen, right?

WARNING!!!!!

The things I’m about to discuss are totally unsupported by Cisco.  I also am not going to support them.  It worked for me this time, but it could have very easily screwed things up.  Don’t come to me and tell me you did this and now you need to reformat and you want me to help you.

Okay, that being said, there are a multitude of ways to gain root access to your CUCM server.  Again, none of them are supported, so don’t do them if you are the least bit squeemish.  The first thing you should read is the great guide at blindhog.net about gaining root access on CUCM 5.x/6.x.  It’s a very handy way to show you that the underlying system in CUCM is actually RedHat Enterprise Linux.  Since I didn’t have a Linux boot disk handy, I instead stumbled across this post which talks about jailbreaking CUCM.  I didn’t have to go all the way through it, but it is a fascinating read nonetheless.

1.  Download PuTTY, PuTTYgen, and PSFTP from HERE.  The instructions at the above link use these files and you should too.

2.  Log into CUCM CLI via SSH as the administrator user.

3.  Type in “file dump sftpdetails ../.ssh/id_dsa” at the CLI.  You’re going to get a dump of the SSH private key for the sftpuser account.  Copy this information to a text file and save it somewhere on your system.

4.  You need to convert this SSH private key from OpenSSH to PuTTY’s SSH format using PuTTYgen.  Import the Private Key file and save it somewhere like c:\temp.  Be sure to save it with the .ppk extension.

5.  Launch PSFTP with this command string:

psftp -2 -i c:\TEMP\id.ppk sftpuser@cucm.example.com

The file location should be where you saved the private key and the user@server should reflect your server’s IP or hostname.  Be sure to type in sftpuser@<your server address here>.

6.  If you’ve logged into the server before and saved the RSA fingerprint, you may get a warning here about the key your using.  Just say “yes” and keep going.

7.  Voila!  You’ve logged into the system as the sftpuser account and you can now download files from the Linux file system or copy files to it.  In the above link, this is where you would jailbreak the system.  For my particular example, we won’t have to go quite that far.

8.  In my troubleshooting case, I changed directories to “/usr/local/platform/conf/” which is where the configuration files live.  I noticed that “server.conf” was missing, but there was a “server.conf.bak” in the same directory.  I typed in “mv server.conf.bak server.conf” since I couldn’t copy the file.  Then I tried to start the Service Manager service again from a SSH CLI session.

SUCCESS!!!

Tom’s Take

I do stupid things all the time.  Like voiding warranties, which is what my little procedure above will do to your CUCM system if you try it.  I was desperate and impatient and it paid off for me this time.  I also have experience on the Linux CLI so I’m not afraid to do things there, even knowing that the outcome for a little slipup could crater my system.  Don’t do what I do unless you know what you’re doing or you aren’t afraid to reload.

That being said, a little Internet searching followed by some practical application can save your bacon in a time of emergency.  Just remember that the Disaster Recovery Tool (DiRT) is there for a reason. Use it wisely and use it often and you shouldn’t find yourself needing to jailbreak your CUCM server anytime soon.