Network renumbering happens from time to time. You outgrow a segment or buy a company and need to readdress things. Or you start doing that new IPv6-thingy and need to renumber IPv4 to make more sense. In any case, you go through everything that is attached to the network and make your changes. The routers act just fine. The switches couldn’t care less. Even the toaster is still churning out perfect slices. And then you get to CallManager. As soon as you type in your password to the web administration page, ominous organ music starts playing in the back ground after a thunderclap. You start to get the feeling that maybe this isn’t such a good idea.
There’s a good reason for that. CallManager is very dependent on using IP addresses to communicate with the rest of the CUCM cluster (you did cluster your call processors, didn’t you???). In fact, Cisco’s best practice is to use IP for communication with the cluster as opposed to relying on DNS. You have to make the choice during the platform installation, and the only way to change is to completely reload the system. For the purposes of this post, I’m going to assume you followed best practice and are using IP addresses for your communications. When the time comes to change the IP addresses of your cluster members, you shouldn’t fret about the complexities. That is, provided you have some patience and some familiarity with the CUCM command line interface.
1. Make sure your cluster is healthy. This can’t be stressed enough. If you’ve got database replication issues or network instability, this whole process is going to suck like a graviton-powered Hoover. SSH to the publisher using your favorite SSH client and log in using the platform administrator login. Once there, run this command:
show perf query class "Number of Replicates Created and State of Replication"
It should come back with the number of replicates created and what the replicate state is. The state should say “2”. This means your cluster is healthy and replicating like it should. You could also see a “0” here if you only have one publisher and no subscribers. This would be the case if you’re running a Business Edition (CUCMBE) all-in-one system. Next, you should check the network connectivity with this command:
utils diagnose module validate_network
This command should return a status of “passed”. If those command both succeed with no errors, your cluster is healthy and ready for the next step.
2. Change the Subscriber Server IP in CUCM Administration. This is where my first IP change attempt failed spectacularly. You need to change the IP address of the server in the CUCM Admin page before you do anything at the OS level. This is due to the fact that the database services rely on the IP for a great number of things, and changing the IP at the OS level without changing the database first will cause the DB services to fail to start when you reboot. You need to change the subscribers before the publisher in order to maintain consistency. First, login into the administration page, then go to System –> Server and select the node name for the subscriber you want to start with. Change the IP address on this page to reflect the subscriber’s new IP address. At this point, stop what you are doing and go get some coffee. Just walk away from the PC for a few minutes. While you’re up, I like my coffee with sugar and cream.
Back already? And, I see you didn’t bring me any coffee. Oh well. We need to verify that the database has replicated the changes across the cluster before we start monkeying around with the OS configuration. SSH into the publisher and run this command:
run sql select name,nodeid from ProcessNode
You should see the new IP address in the output table. Now we need to actually change the subscriber’s IP.
3. Change the Subscriber IP address in the OS. Fire up that SSH client again and connect to the subscriber you just changed in the publisher admin page. Don’t worry, the address is still the same at the OS level, so you’ll be able to connect. Once on the command line, you need to change the IP. I recommend doing it this way to be sure the changes stick and make it a little easier to reboot the system afterwards. If you’re moving the server to a different subnet, make sure to change the default gateway first like this:
set default gateway x.x.x.x
After that, or if you didn’t need to change the gateway, you need to change the IP address for the server.
set network ip eth0 x.x.x.x y.y.y.y
When you complete this command, you’re going to get a warning popup:
*** W A R N I N G *** If there are IP addresses (not hostnames) configured in CallManager Administration under System -> Servers then you must change the IP address there BEFORE changing it here or call processing will fail. This will cause the system to restart ======================================================= Note: To recognize the new IP address all nodes within the cluster will have to be manually rebooted. =======================================================
You can safely ignore this warning because you’re following these steps and you’ve already done this. Type “Yes” at the prompt to force the subscriber to reboot. Now would be a great time to enjoy that coffee. It’s going to take a little longer than usual for the subscriber to come back up due to the changes to need to be made to the Tomcat server and the DB server. At this point, if this is the only change you are making, Cisco recommends you reboot all the other cluster servers to change the name resolution and hosts file entries. If you’ve still got more servers to do, go ahead and keep changing the subscribers as above. Make sure the database replicates with the new IP before changing the OS IP address. If your cluster is configured correctly, the phones should fail over to the active subscribers as you take down the other subscribers for the IP changes. Once you’ve completed the subscriber IP changes, you’re almost done.
4. Change the Publisher Server IP Addresses. The above steps are still correct for the the publisher server, except for one added step. After you’ve returned with your fifth cup of coffee following the DB replication issues, you need to change the publisher IP on all the subscribers. Go to the OS admin page of each one and go to Settings –> IP –> Publisher. Once there, change the IP address of the Publisher server to the new IP you are going to set on the publisher. If you happen to be a CLI junkie, you can do this from the comfort of your SSH program provided you’re on CUCM 6.1(2) or later with this command:
set network cluster publisher ip x.x.x.x
Be warned that you’ll need to reboot immediately after typing in that command. Once that’s taken care of on all the subscribers, you can proceed to change the IP address in the OS admin of the publisher and then reboot. After the publisher comes back up, communication should be restored and all the ominous organ music and thunderclapping in the background should stop.
Should you find yourself unlucky enough to use DNS for cluster identification, you’re going to want to start putting some Irish Crème in that coffee. You’ll also want to refer to this page to get some information about the additional steps required to sort out the DNS mess on your system as you start changing IPs.
And there you have it. With a little patience, a few cups of coffee, and a little CLI wizardry, even the mean, nasty CallManager can get a new IP address. Just be sure to check the database replication after changing the IP in the system administration page. Because if the DB changes haven’t replicated before you start changing OS settings, you’re going to have a bunch of fun getting things back to a good running state before you try again. Good luck, and may the voice be with you.
Thanks for the info!
One question: as we will be moving our subscriber to a new building (thus the re-ip), at which step would you recommend powering it off for the move?
I.E before or after step 3.
Since you are going to need to reboot the subscriber during step 3 to complete the IP adress change, you should probably power the unit off at the “reboot” step and move it then. When the system comes back up, proceed to the next step to start sorting out any DB errors that have cropped up.
hi,I installed the call manager 7.0.1-11000-3 in the VMware workstation 8.0 ,after i change the eth0 ip address and reboot , the BD error comes up ,at the same time ,i cannot enter the system through web any more (although I have enterred the right admin number and key number ,the web shows that : the Database Error), so how could I handle the problem ? thank U very much !
I moved (changed IP) my subscriber last night and, all in all, it went okay.
I shut down the subscriber via command line at the end of step 3.
I physically moved the server and, after boot at the new location, replicate state was 2.
At this point my environment was in a strange split state where all gateways were registered with the subscriber (at new IP) but all phones were still on the publisher (where they failed over to when I took the subscriber down).
I waited for over an hour to see if things would resolve. I reset some phones but they’d come up registered to the publisher and with the old ip of the subscriber as the secondary (on the 7941 phone at settings/device/call manager configuration)
I restarted the publisher and, after several nervous minutes, it came up and phones started moving over to the subscriber. I had to reset some phones via Call Manager gui to force them over and a few other phones ended up in a ‘limbo’ state where
they were booted but not registered (due to not having a failover call manager while the publisher reloaded I assume). For these phones (~12 out of 800) it was necessary to disconnect/reconnect the Ethernet (switch interface shut/no shut for remote
Thanks for this page, it helped verify what I interpreted as the required steps from the somewhat unclear Cisco documentation. Hopefully some other poor soul dreading a Call Manager IP change will find this site!
Thanks again, Peace!
Pingback: ipcpu » Blog Archive » Changing CallManager's IP Address | The Networking Nerd
how to restart network in CUCM? I have setup CUCM on VM ware player.. every time i restarted the VN no network conectivty..
I’m have to set network gatway to restart the network,,,, 😦
Hi – I have a CUCM v8.5 cluster with 4 subs – I need to move and re-IP two of them (one is also running TFTP). Do I need to take any special steps around CTL/ITL files – I’ve heard horror stories of people having to manually delete ITL files on phones – please say this isn’t so!
I’ve done this Chris. You have to be careful but it isn’t as big of a nightmare as you might think. Phones should have primary and secondary TVS (trusted verification service?) servers to validate the regenerated ITL file and certificates. If a phone cannot contact the primary TVS server, it will fall back to the secondary server. The TVS servers are identified by the CM Group assigned to the phone.
Ensure that you change the IP address or hostname on only one server at a time. Go through all your reboots ands and restart the TFTP and TVS services on each node in the cluster after each IP address change. Then do a full reset of ALL phones as you want them to pull the newly regenerated ITL file. Make sure you check a few phones manually to ensure that the ITL file is correct on then. Then move on to the next server. If you use CTL files or tokens, re-run the CTL client after you change the IP address or hostname of the server, or after you change the DNS domain name.
If you aren’t using CTL files, you can also disable SBD (security by default) and restart TFTP/TVS then reset all the phones. This will essentially put your cluster and phones in a pre-v8 state. Don’t run in production very long as things like EM and directories will stop working. Once all the phones have blank ITL files and essentially don’t care about trusting systems, you can change IP addresses to your hearts content. Make sure when you are done to re-enable SBD (this is done in Enterprise parameters by enabling/disabling Prepare cluster for rollback to pre version 8). Restart TVS/TFTP and bounce your phones.
I’ve done it both ways.
Hi Jon, when you mentioned “perform a full reset on all of the phones” would it be enough by doing that just on the admin page of CUCM or do you need to do a factory reset on each phone?