CallManager not sending CDR to Billing Server – Troubleshooting Cisco CDR Repository Manager

A few days ago, one of our customers raised an issue.
The customer reported that they haven’t been receiving any CDR from the CallManager Cluster.
The customer has a remote Billing Server and the files were being sent to the server by FTP.

I mapped out three possible issues that could have suddenly caused this:

– issues with the CallManager Cluster (cluster consists of 3 nodes)
– issues with communication between the CallManager Cluster and the Billing Server (maybe even a firewall)
– issues with the Billing Server (permissions, home folder, wrong path e.t.c)

So I asked the customer the usual questions like “when was the last time you got any CDRs via FTP”, “has there been any maintenance”.

The customer reported that they stopped receiving CDRs on the Billing Server right after a Call Manager upgrade. The customer was recently upgraded from CallManager v6 to v7.

A quick check revealed some things:

– CDR was being generated by the server. This was good to know at least
– Cisco CAR Scheduler wasn’t running on the Publisher, so I started this immediately
– The two other node (Subscribers) were running Cisco CAR Scheduler
– Cisco CDR Agent was running on all the nodes
– Cisco CDR Repository Manager was running on all the nodes

At first glance, this didn’t look right. Some of the services running on the Subscribers were meant to run on the Publisher only.
In order to understand better, I turned to the Cisco Voice Engineer’s best friend –> CallManager Tracee (I actually hate it sometimes :-)!!!!)
The information in the traces didn’t show any attempts to send the CDR files via FTP

I did a bit of research and found a great link with some very useful info that confirmed what I was thinking:

Cisco CDR Repository Manager should be running on the First Node ONLY (the Publisher)
Cisco CAR Scheduler should be running on the First Node ONLY (the Publisher)
Cisco CDR Agent should be running on all the call processing nodes

The link Troubleshooting CUCM 5.x / 6.x CDR Analysis and Reporting “Data Missing” issues did a good job in explaining a few things.

Other important info included CLI commands that would let you know the following:

– whether the CDR Agent is moving files from the Subs to the Pub
– if the CDR files have been processed by the Publisher
– whether the Repository Manager was unable to transfer the files to the Billing Server

Another link I stumbled across, though it didn’t have much info that related to the issue was Troubleshooting CallManager 6.x 7.x CDR / CMR & how to fix error 10021. I gave it a quick glance and made a note of a few things.

I’ve just pulled a new set of CCM traces from the Cluster, and this time I can see the files being sent by FTP; see below:

INFO  [Thread-8] cdrrep.FtpManager (FtpManager.java:78) – FtpManager constructor 0: Establish server connection to host [x.x.x.x] successfully!
INFO  [Thread-8] cdrrep.CDRSender (CDRSender.java:345) – CDRREP before sending the file: JVM status: [Max = 246.56MB, Total = 6.62MB, Free = 3.91MB (59.00%), Used = 2.72MB (41.00%)]
INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:105) – CDRSender wakes up for destination 2
INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:105) – CDRSender wakes up for destination 3
INFO  [Thread-8] cdrrep.FtpManager (FtpManager.java:227) – FTP success: cdr_201005041150_1807 to x.x.x.x
INFO  [Thread-8] cdrrep.CDRSender (CDRSender.java:352) – CDRREP after sending the file: JVM status: [Max = 246.56MB, Total = 6.62MB, Free = 3.66MB (55.30%), Used = 2.96MB (44.70%)]
INFO  [Thread-8] cdrrep.CDRSender (CDRSender.java:345) – CDRREP before sending the file: JVM status: [Max = 246.56MB, Total = 6.62MB, Free = 3.57MB (53.85%), Used = 3.06MB (46.15%)]

One last thing I did was to go into CDR Management (CallManager Serviceability –> Tools –> CDR Management) and verify that the info there was correct.

So in all, it was quite interesting for me because I haven’t come across many CDR issues in my Cisco Life.

I hope someone finds this useful 🙂

Been ages….

Hey Folks!!

It’s been ages since I last posted.
I appreciate the folks that keep stopping by and even sending me e-mails and leaving comments!

I’ve been quite busy at work and I’ve literally not had much time to blog…..or even tweet  🙂
I’ve been working some crazy hours, so blogging has been on the back-burner.

Recent developments have prompted me to “upgrade” my lab and bring up to scratch.
One of the major accounts I work on have had a few of their clusters migrated to CallManager v7; I’ve also got three friends studying for the CCIE Voice v3 lab, so I’m trying to get in the study-flow of CUCM 7.

So far, I’ve setup my ESXi and installed CUCM 7.
I’ve had issues installing UCCX 7, but I’m hoping it should be sorted before the week runs out.

I’ll try to blog about the stuff I play with from time to time.

Right now, it’s 2:11am and I’ve just finished completed the pilot phase of a project I’m working on, so my eyes are getting really heavy 🙂

Be back sooooooooooooooooon!!!

RE: Got my Number

 I’ve been getting e-mails to write an update on my quest to obtain the CCIE number.

Apologies it has taken over 3 weeks, but I got plunged right into work the very next day I got back to the UK.

 After passing my CCIE, I felt like Daniel, who in the Bible was thrown in a lions’ den and miraculously came out alive.

I’ve had my eye on the CCIE for a while (right after I wrote my CCNA in 2001), however I wasn’t motivated enough to go the R&S way, though that was my main experience.

 I completed my CCVP in November 2008 and went straight on to write the CCIE Voice Written exam in December 2008.

I started the CCIE Voice Lab pursuit right after passing my written, however it was around the festive period, hence I couldn’t get much done in December.

 I kicked off in January trying to learn the technologies.

I spent a huge amount of time watching videos and reading some Cisco docs.

While I don’t recommend reading ALL of the Cisco docs (front-to-back) on the lab kit, I would suggest spending quality time to master certain areas. I remember getting a good grasp on IOS CoR using the Call Manager SRND (though I believe I used CCM v7 SRND).

 Another thing I remember doing was learning how to access Cisco docs using the “new” Doc CD (http://www.cisco.com/web/psa/products/index.html) that replaced the UniverCD.

 I started out the CCIE journey with 2 good friends, but I soon found that I was alone after they both reviewed their commitments and concluded that they would wait for the Version 3 exams.

After my friends dropped out, I started to spend less time doing online labs cos we used to work through labs together.

This gave me more time to study the technologies and I think it paid off.

 Along the way, I met James Key of Blindhog. He became a friend and mentor.

We would setup WebEx sessions on the fly and end up spending hours on the phone going through problems.

There were some areas I had absolutely no clue about which James was able to explain thoroughly.

 Another person I met through a CCIE friend was Dharm. In my own eyes, I was ready for CCIE…….well, until I met Dharm.

He gave me access to his kit remotely and also allowed me come over to his place to use the kit.

 I met Dharm about 5 weeks to my exam. I knew lots of stuff, but I had no strategy to pass the exam.

He helped breakdown lots of other concepts and I realised I was going to fail despite studying and labbing like a MAD MAN for 4 months.

In my opinion, I think this guy should have a training school 🙂

As I’m writing this, we’re waiting for the result of his own CCIE Voice exam…….I’m sure he’s going to pass!!!!

 I flew to San Jose to face the greatest challenge of my life up until now, and came back a VICTOR!!!!

 In all, I had about 5 months of solid preparation after my written. Yes I know it sounds crazy……CCIE-In-5-Months 😉

To be honest, I couldn’t tell people I had only prepared for 5 months……they would have thought I was crazy!

 One thing I’m still unable to quantify is how much work experience helped me. I believe experience from work really helped me with areas like Call Routing, Gatekeeper and Unity.

 If you’re married, with or without kids, you need to appreciate the fact that the CCIE pursuit would in many ways stand between you and your family. It’s very important to have your family’s support and consent before embarking on such a great journey.

 All I can say is that I’ve got the MOST supportive wife in the world!!!!!!!!!

 Wishing you all the best……God Bless!!!

 

Skybaba

To the Lion’s den and back!!!!

 
I sat my first CCIE Voice lab attempt on Tuesday, 26th May 2009.
To God be the Glory, I passed my exam!!!

The last few days have been hectic.
I’m currently in the US, and I’ve got a busy schedule.

I plan to write a comprehensive report on my prep and lab experience.

Right now, I guess I’m still basking in the euphoria of being part of the first 1000 people (in the world) that hold a CCIE in VOICE!

😉

THE JOY OF A FIRST IPT CALL

 

Do you remember the first IPT call you made in your lab or at work??

I’ve got a three year old daughter. You’ll be amazed how much one can learn from these little ones.
Apparently, she’s been watching me configure my “mini-voice-lab” and making calls between HQ and Branch 1.
One of my HQ phones is a 7940 and the BR1 phone is an IP Blue soft-phone running on my laptop.

Now, either she knew what the number to dial, or she actually knew the redial button…….but all I heard was “Daddy!!!!! It is WORKING!!!!!!!”

Apparently, she made a call between the HQ and BR1 IP Phones and she could hear her voice coming out of my laptop……EUREKA!!!!!!!
I couldn’t get her away from the phone after that. I think the most fascinating thing for her was that she could hear herself through my laptop speaker. LOL!!  She kept saying “I can hear my voice”.

Now that feeling took me back to 2002…..the first time EVER I configured VoIP!!!!
The truth was I had no clue what I was doing back then.
I was a zealous intern working at a company called United Telesys (now called Emperion West Africa).

I was working with Sly, a senior engineer tasked with setting up voip for two branches of a bank (I think it was either Intercontinental Bank or Diamond Bank). One branch was located in Trans-Amadi, while the other branch was located in Olu-Obasanjo, both in Port Harcourt, Nigeria.

I remember we simulated the bank’s WLAN and had two 1750’s on each end of the wireless connection.
One 1750 had an FXO, while the other had an FXS………..I can remember the feeling too well……..I was so overwhelmed when the phones rang.

I’m sure every IPT/UC Engineer has had this 1st Call feeling before……..please don’t forget it cos it’s a memory to cherish 😉

IMPLEMENTING GATEKEEPERS WITH HOP-OFF TECH-PREFIX

In the course of my CCIE studies, I’ve found the “Gatekeeper Address Resolution on ARQ” chart extremely useful.
Having a good understanding of the call-flow displayed the chart is key to understanding the debug gatekeeper main 10 and how the gatekeeper logically looks through configured parameters to decide how a call will be routed.

The chart (shown below) should be available in the CCIE Voice Lab exam. It can be found on Page 10-40 (pg 350 figure 10-14) in the CCM 4.1 SRND, which we’ve been told would be available in the lab.

 

 

Now back to the Hop-off Tech Prefix.
While there are many ways to route calls on a gatekeeper, we may need to use the Hop-off Tech-prefix configuration scenario in the lab if we’re specifically asked to do that, or if we’re instructed not to use the following:

 – Tech prefix
 – Default gw-type-prefix
 – E.164 Aliases (including Alias Static commands for CCM numbers)
 – Zone Prefix

Looking at the chart, we’ll need to match Step 1. Though it’s typically used to route calls to Remote Zones via a Location Request (LRQ), the gatekeeper can also generate an ACF at this stage if the Hop-off points to a Local Zone.

There isn’t much configuration needed to get this working; however, for your configuration to “stay working“, you would need to “hardcode” the signalling port used by your GK-Controlled Trunk (on Call Manager) to always use port 1720 as its Gateway Signalling Port.
This is quite important because CCM dynamically assigns port numbers for gateway signalling and for RAS.
These ports are set to 1720 and 1719 respectively on the gatekeeper by default.

If you don’t do this and the proctor resets your Trunks before grading (or maybe he/she even reboots the CCM box), then you’re screwed big-time 🙂

You can verify the Signalling and RAS ports CCM registered with on the gatekeeper with the show gatekeeper endpoints command:

 
GK-RTR#sh gatekeeper endpoints
                    GATEKEEPER ENDPOINT REGISTRATION
                    ================================
CallSignalAddr  Port  RASSignalAddr   Port  Zone Name         Type    Flags
————— —– ————— —– ———         —-    —–
10.10.50.103    2789  10.10.50.103    4318  gk-zone1          VOIP-GW
    H323-ID: CCM_GKCT_1
    Voice Capacity Max.=  Avail.=  Current.= 0
172.16.30.254   1720  172.16.30.254   57634 gk-zone1          VOIP-GW
    H323-ID: CME_Trunk
    Voice Capacity Max.=  Avail.=  Current.= 0
Total number of active registrations = 2

 
As shown above, CME registers with 1720 by default, but CCM registers with 2789.
If CCM_GKCT trunk is reset, the port number would change to something totally different.
To configure CCM to use the port 1720 (signalling port) for a particular gatekeeper-controlled trunk through the CCM Service Parameters:

Navigate to the CCM Service Parameters and do a search for 1720. You’ll find the section under Clusterwide Parameters (Device – H323).
Change the name from None to the name of your trunk. In my case, my trunk is called CCM_GKCT.
Once this is done, you’ll notice that the port number for your CCM trunk would now be 1720 after you reset it.

 

 

We would now need to type in 3 commands to cater for the prefixes in HQ, BR1 and BR2.
gatekeeper
 zone local gk-zone1 myccielab.com 172.16.40.254
 gw-type-prefix 2* hopoff gk-zone1 gw ipaddr 10.10.50.103 1720
 gw-type-prefix 3* hopoff gk-zone1 gw ipaddr 10.10.50.103 1720
 gw-type-prefix 4* hopoff gk-zone1 gw ipaddr 172.16.30.254 1720
 no shutdown

 

Depending on whether we’ve been instructed to provide redundancy, we may need to include the secondary call-processing engine in the command for HQ and BR1 prefixes.
I’ve only got one call-manager running in my home lab, but here’s how you’ll configure it with two call-managers (assuming your second call-manager IP Address is 10.10.50.104)

 
 gw-type-prefix 2* hopoff gk-zone1 gw ipaddr 10.10.50.103 1720 gw ipaddr 10.10.50.104 1720
 gw-type-prefix 3* hopoff gk-zone1 gw ipaddr 10.10.50.103 1720 gw ipaddr 10.10.50.104 1720

 

You can verify your configuration by using the show gatekeeper gw-type-prefix command (or by making calls ;-))

 

GK-RTR(config-gk)#do sh gatekeeper gw-type-prefix                         
GATEWAY TYPE PREFIX TABLE
=========================
Prefix: 2*    (Hopoff zone gk-zone1)
  Statically-configured gateways (not necessarily currently registered):
    10.10.50.103:1720
  Zone gk-zone1 master gateway list:
    10.10.50.103:1720 CCM_GKCT_1

Prefix: 3*    (Hopoff zone gk-zone1)
  Statically-configured gateways (not necessarily currently registered):
    10.10.50.103:1720
  Zone gk-zone1 master gateway list:
    10.10.50.103:1720 CCM_GKCT_1

Prefix: 4*    (Hopoff zone gk-zone1)
  Statically-configured gateways (not necessarily currently registered):
    172.16.30.254:1720
  Zone gk-zone1 master gateway list:
    172.16.30.254:1720 CME_Trunk

 

PS: I’ve assumed your CME side has the following configuration:

dial-peer voice 1000 voip
 destination-pattern [23]…$
 session target ras
 incoming called-number .

 

Now, that’s a long post!!!!!

H.323 GATEKEEPERS

H.323 GATEKEEPERS

Gatekeepers are interesting devices that can be used for a variety of purposes within our Dial-Plans.
Some mandatory gatekeeper functions include Address Translation, Bandwidth Control and Zone Management.
One thing to note is that gatekeepers do NOTget involved with Codec Negotiation and any form of Digit Manipulation.

According to IP Expert instructor, Mark Snow “most students I’ve taught have the biggest problems with H323 Gatekeepers and……..“.

This statement is so true, considering that there are thousands of Call Manager deployments out there that don’t use gatekeepers.
I’ve been quite fortunate to work on an account that simply cannot function without gatekeepers because of the sheer amount of intra-cluster and inter-cluster communications spanning the entire globe.

I plan to dedicate a couple of blog-posts to this monstrous component of any large-scale dial-plan deployment.

MWI, or should I say Phantom MWI’s

We’ve just deployed some Unified Messaging (UM) servers with Primary and Failover pairs, so we’re currently migrating users from Unity Voicemail only servers (VMO) to Unified Messaging servers (UM).
The current VMO servers (3 of them) currently serve 3 clusters, consisting of about 40 sites, in a multi-tenant deployment, hence we’re running this as a phased migration.

While the migration itself is going on without stress, we started getting MWI issues on different site right after cutting them over. This had me and my colleague pulling our hair out in some occasions cos users were raising tickets like: “Can’t retrieve my messages“, “No messages but light won’t go off” “I’ve  got voicemails in my e-mail box, but the lights on my phone isn’t coming on“…….the list is endless!!!

After some troubleshooting, we found out that users weren’t deleting their messages on the VMO servers prior to migration (even though they had all been advised to listen to messages and delete them on the eve of the cutover).
Because we are cutting-over the sites in phases, we can’t take the VMO servers out of service, at least no yet cos they were still being used. Also, the project lead doesn’t want to delete the subscribers off the VMO, just in case we needed to regress.

My colleague and I came up with a simple solution with three steps:

– we launched Message Store Manager within the Unity Tools Depot and ran a script to move unread messages from inbox into deleted items or Unity Dumpster for subscribers that had been cutover from VMO to UM

– Secondly, we used Bulk Edit tool to uncheck USE MWI for Message Notification (under the messages tab)

– Lastly, we resync’ed the MWI on the new UM servers (under UTIM).

That was the last we saw of the Phanton MWIs  😉

How it all began……..

It all started while I was a student @ Lagos State University, LASU as it was popularly called. I wasn’t satisfied with what school had to offer. Though I was in my 2nd year of a 5-year engineering course (Electronics & Computer Engineering) I just felt unfulfilled. I always believed the Uni couldn’t offer me what I wanted. I felt there was a bigger picture, and indeed there was!

I remember attending a Big Tent church meeting where Jacquie McCullough preached “No more abortions” and I knew I couldn’t even afford to drop out of school at that stage……there had to be another way!!!!

I started my IT journey with Microsoft 1999. Back in the days, folks were running NT 3.51 and NT 4.0. I passed my first Microsoft certification and got my MCP in October 2000.

Somewhere along the line I stumbled on serious networking (cos I don’t consider Microsoft as networking ;-)) and delved into CompTIA Network Plus in the year 2001.

Later in 2001, I came across CISCO and I said to myself this is it. Eight years later I’ve got CCNA, CCNP and CCVP under my belt.
I should have gone for the R & S CCIE a while ago, but I wasn’t motivated enough to even start to study for it.
Somewhere along the line I stumbled on Asterisk and Cisco voice and the rest is history.
I’m now working towards CCIE VOICE and would like to put my thoughts together and present it to the world to see. The CCIE journey has been quite exciting and challenging. Since I started my CCIE pursuit I’ve learnt so much; practically, I’ve learnt something new everyday. I’ve also learnt that I know so little, so little it could discourage, but I can’t let the passion die…….NEVER!!!!
The passion has brought me thus far in my love-affair with Cisco………..WELCOME TO MY BLOG!!!!!!