Network renumbering happens from time to time.  You outgrow a segment or buy a company and need to readdress things.  Or you start doing that new IPv6-thingy and need to renumber IPv4 to make more sense.  In any case, you go through everything that is attached to the network and make your changes.  The routers act just fine.  The switches couldn’t care less.  Even the toaster is still churning out perfect slices.  And then you get to CallManager.  As soon as you type in your password to the web administration page, ominous organ music starts playing in the back ground after a thunderclap.  You start to get the feeling that maybe this isn’t such a good idea.

There’s a good reason for that.  CallManager is very dependent on using IP addresses to communicate with the rest of the CUCM cluster (you did cluster your call processors, didn’t you???).  In fact, Cisco’s best practice is to use IP for communication with the cluster as opposed to relying on DNS.  You have to make the choice during the platform installation, and the only way to change is to completely reload the system.  For the purposes of this post, I’m going to assume you followed best practice and are using IP addresses for your communications.  When the time comes to change the IP addresses of your cluster members, you shouldn’t fret about the complexities.  That is, provided you have some patience and some familiarity with the CUCM command line interface.

1.  Make sure your cluster is healthy. This can’t be stressed enough.  If you’ve got database replication issues or network instability, this whole process is going to suck like a graviton-powered Hoover.  SSH to the publisher using your favorite SSH client and log in using the platform administrator login.  Once there, run this command:

show perf query class "Number of Replicates Created and State of Replication"

It should come back with the number of replicates created and what the replicate state is.  The state should say “2”.  This means your cluster is healthy and replicating like it should.  You could also see a “0” here if you only have one publisher and no subscribers.  This would be the case if you’re running a Business Edition (CUCMBE) all-in-one system.  Next, you should check the network connectivity with this command:

utils diagnose module validate_network

This command should return a status of “passed”.  If those command both succeed with no errors, your cluster is healthy and ready for the next step.

2.  Change the Subscriber Server IP in CUCM Administration. This is where my first IP change attempt failed spectacularly.  You need to change the IP address of the server in the CUCM Admin page before you do anything at the OS level.  This is due to the fact that the database services rely on the IP for a great number of things, and changing the IP at the OS level without changing the database first will cause the DB services to fail to start when you reboot. You need to change the subscribers before the publisher in order to maintain consistency.  First, login into the administration page, then go to System –> Server and select the node name for the subscriber you want to start with.  Change the IP address on this page to reflect the subscriber’s new IP address.  At this point, stop what you are doing and go get some coffee.  Just walk away from the PC for a few minutes.  While you’re up, I like my coffee with sugar and cream.

Back already?  And, I see you didn’t bring me any coffee.  Oh well.  We need to verify that the database has replicated the changes across the cluster before we start monkeying around with the OS configuration.  SSH into the publisher and run this command:

run sql select name,nodeid from ProcessNode

You should see the new IP address in the output table.  Now we need to actually change the subscriber’s IP.

3.  Change the Subscriber IP address in the OS. Fire up that SSH client again and connect to the subscriber you just changed in the publisher admin page.  Don’t worry, the address is still the same at the OS level, so you’ll be able to connect.  Once on the command line, you need to change the IP.  I recommend doing it this way to be sure the changes stick and make it a little easier to reboot the system afterwards.  If you’re moving the server to a different subnet, make sure to change the default gateway first like this:

set default gateway x.x.x.x

After that, or if you didn’t need to change the gateway, you need to change the IP address for the server.

set network ip eth0 x.x.x.x y.y.y.y

When you complete this command, you’re going to get a warning popup:

***   W A R N I N G   ***

If there are IP addresses (not hostnames)
configured in CallManager Administration
under System -> Servers
then you must change the IP address there BEFORE
changing it here or call processing will fail.
This will cause the system to restart
=======================================================
Note: To recognize the new IP address all nodes within
the cluster will have to be manually rebooted.
=======================================================

You can safely ignore this warning because you’re following these steps and you’ve already done this.  Type “Yes” at the prompt to force the subscriber to reboot.  Now would be a great time to enjoy that coffee.  It’s going to take a little longer than usual for the subscriber to come back up due to the changes to need to be made to the Tomcat server and the DB server.  At this point, if this is the only change you are making, Cisco recommends you reboot all the other cluster servers to change the name resolution and hosts file entries.  If you’ve still got more servers to do, go ahead and keep changing the subscribers as above.  Make sure the database replicates with the new IP before changing the OS IP address.  If your cluster is configured correctly, the phones should fail over to the active subscribers as you take down the other subscribers for the IP changes.  Once you’ve completed the subscriber IP changes, you’re almost done.

4.  Change the Publisher Server IP Addresses. The above steps are still correct for the the publisher server, except for one added step.  After you’ve returned with your fifth cup of coffee following the DB replication issues, you need to change the publisher IP on all the subscribers.  Go to the OS admin page of each one and go to Settings –> IP –> Publisher.  Once there, change the IP address of the Publisher server to the new IP you are going to set on the publisher.  If you happen to be a CLI junkie, you can do this from the comfort of your SSH program provided you’re on CUCM 6.1(2) or later with this command:

set network cluster publisher ip x.x.x.x

Be warned that you’ll need to reboot immediately after typing in that command.  Once that’s taken care of on all the subscribers, you can proceed to change the IP address in the OS admin of the publisher and then reboot.  After the publisher comes back up, communication should be restored and all the ominous organ music and thunderclapping in the background should stop.

Should you find yourself unlucky enough to use DNS for cluster identification, you’re going to want to start putting some Irish Crème in that coffee.  You’ll also want to refer to this page to get some information about the additional steps required to sort out the DNS mess on your system as you start changing IPs.

And there you have it.  With a little patience, a few cups of coffee, and a little CLI wizardry, even the mean, nasty CallManager can get a new IP address.  Just be sure to check the database replication after changing the IP in the system administration page.  Because if the DB changes haven’t replicated before you start changing OS settings, you’re going to have a bunch of fun getting things back to a good running state before you try again.  Good luck, and may the voice be with you.