Automated IVR Testing

All software development technologies provide testing utilities so you can make changes run tests to make sure that a) your changes are doing what you expect, and b) you didn’t break anything.  I haven’t come across any unit testing features built in to IVR development environments, although all-encompassing solutions do exist (Hammer comes to mind).

The code sample provided is a simple unit testing IVR application, so you can create as many tests as you need, to be comfortable your IVR application is working.

State Machines (It’s not so bad, really!)

Before we dive in to the example, let’s go over the concept of “State Machines,” because we use a simple state table to set up a test.

Let’s say this is our basic application.  It has a few actions that it performs:

OK, so that’s pretty clear.  Let’s add details on how we connect one action to another.  The “reaction” is a description of the user’s response.  For example, for collect_pin, “OK” means the user entered a valid PIN.

“Wait,” you say, “but, there are many ‘reactions’ for a given action.  What if the user doesn’t provide a valid account number?”  OK, let’s add rows for that:

Just to make sure you understand the concept, what happens for the following “Collect_account” action when there is “Bad Input?”

It will go to “Bad Account Number” which probably plays a helpful message, and then returns to the state of “collect_account.”  Hey, did you see that?  I finally mentioned “state!”  Simply put, the actions column is your “state,” the reaction column is your “event,” and the next column is the “transition.”

So, to throw one more curve at you, let’s restructure the state table above using JSON.

testScript = {

    “Collect_account”: {

        “OK”: {next:”Collect_pin”},

        “No Input”: {next:”Collect_Account”},

        “Bad Input”: {next:”Bad_Account\_Number”},

        “Hang Up”: {next:”Exit”}

    },

    “Bad_Account\_Number”: {

        “OK”: {next:”Collect_Account”},

        “Hang Up”: {next:”Exit”}

    }

};

On line 1, testScript is the state table.  On line 2, “Collect_account” is the state.  On line 3, “OK” is the event, and “{next:’Collect_pin’}“ is the transition.

Well, that will surely bulk up our state table.  From here on, let’s keep it simple and program our test to take one true path; if that path is not completed, the test fails.

Flipping State Tables!

No, I am not cursing state tables.  We do need to flip two of the columns at this point.  The reason is this:  Above, I described an IVR application interacting with a caller, right?  Well, our test IVR is basically a caller interacting with an IVR.

To do so, there are two minor additions to the IVR application:

1. You have to “trigger” the IVR to realize that the inbound call is a test call.  This can be accomplished by the IVR app recognizing the ANI, the DNIS, some other feature, or having the IVR test app provide a “secret code” (which is what our example does).
2. Once in “test” mode, the IVR needs to generate DTMF tones at the end of each dialog.  This is so our testing app recognizes where the IVR application currently is.

Here is a visualization of what the IVR will then sound like if you call in:

You hear, “thank you for calling,” so you enter 987 to trigger the IVR test mode.  Then you hear, “please enter your account number (and it also generates DTMF 0001).”  After entering the account number, you hear, “please provide your 6 to 8 digit PIN (and it also sends DTMF 0002).”    After entering your PIN, you hear, “Main Menu.  To check your balance….(and it sends DTMF 0100)”

Our IVR test app will call the IVR app, and enter the test mode trigger (987) automatically when the IVR answers the call.  With that in mind, here’s the state table for our test:

So, when we program our test app with this information, upon reaching the “Exit,” transition, it knows the test is complete and to log this call as a success.

FYI, in the code, the State has a little more detail, so the IVR knows what “Enter Account Number” means.  Speaking of code...

Demo Code

You can find a working demonstration of this code  here.   I ran it like so:

1. Copy the code files to a Voxeo Evolution user account.
2. Add in inbound application for the simple_app.vxml page, and assign a phone number to it.
3. Add another application for the tester.vxml page, and set up an outbound token in the app.
4. Using the outbound token (on the app’s “contact” tab), launch a call to the phone number assigned to then inbound app.
5. Review the logs, for the test entry.  Here’s what I see logged:

TRACE=start-0001;a-0002;e-0002;b-0005;c-0004;RESULT=SUCCESS

Items for Discussion

In the IVR test mode, playing the DTMF before vs. after the prompts:

Up front, the application test can run faster, but you have two issues to contend with:

1. A prompt may not allow barge-in, so the test app would tone the DTMF too early.
2. If the dialog prompt has an error in some <value> being presented, the error may not be reached because of the barged-in DTMF.

Therefore, it’s better to have your DTMF debug tones at the end of your prompts, not before.

Techniques for Reporting Success/Errors

1. On an unexpected transition, add an email “action,” and the IVR test app can send an email.  The problem with this, is that you have to catch and handle ALL possible events, including disconnects and other system events.  Of course, you can have one “catch-all” state that handles this, but you still have to catch everything.  Ugh!
2. Send an exit code to the launching CCXML script, signifying success.  If there is no “success,” then you have to trigger an alert via email or some URL.  This a little cleaner, but you have to have a good knowledge of CCXML.
3. Use log monitoring tools to watch for issues.  This is the simplest to implement.  But you have to depend on another technology to monitor your logs, which now makes this a two-man job.  Personally, I prefer this one.  It is easy to monitor your logs with tools like SiteScope and Splunk (or simply Perl), and send alerts when the logs aren’t showing what is expected.

Speech Apps

This technique will work for speech applications, too.  Instead of having the state’s action be DTMF digits, provide a wav file name instead.  By adding a couple of lines to the IVR test app, it can then test DTMF or speech apps.

You can see/run the sample code here, or you can clone my repo:

git clone https://bitbucket.org/ericp56/ericp56.bitbucket.org