The first symptom that you will notice when you are running into problems is that the client would not behave the way you want it to behave. As an example, there would be no output, or the wrong output.

Since the outcome of running a REST client depends on the request that you send over the wire and the response that you receive over the wire, one of the first things is to capture the messages and verify that those are in the correct expected format.

REST Services and clients interact using messages, usually in pairs of request and response. So if there are problems, they are caused by errors in the messages being exchanged.

Sometimes the user only has control over a REST client and does not have access to the implementation details of the service. Sometimes the user will implement the REST service for others to consume the service. Sometimes the Web browser can act as a client. Sometimes a PHP application on a server can act as a REST client. Irrespective of where the client is and where the service is, you can use message capturing tools

to capture messages and try to figure out the problem. Thanks to the fact that the service and client use messages to interact with each other, we can always use a message capturing tool in the middle to capture messages. It is not that we must run the message capturing tool on the same machine where the client is running or the service is running; the message capturing tool can be run on either machine, or it can be run on a third machine.

The following figure illustrates how the message interaction would look with a message capturing tool in place.

If the REST client is a Web browser and we want to capture the request and response involved in a message interaction, we would have to point the Web browser to message capturing tool and let the tool send the request to the service on behalf of the Web browser. Then, since it is the tool that sent the request to the service, the service would respond to the tool. The message capturing tool in turn would send the response it received from the service to the Web browser. In this scenario, the tool in the middle would gain access to both the request and response. Hence it can reveal those messages for us to have a look.

When you are not seeing the client to work, here is the list of things that you might need to look for:

In order to check for the above, you would require a message-capturing tool to trace the messages.

There are multiple tools that you can use to capture the messages that are sent from the client to the service and vice versa. Wireshark (http://www.wireshark.org/) is one such tool that can be used to capture any network traffic. It is an open-source tool and is available under the GNU General Public License version 2. However this tool can be a bit complicated if you are looking for a simple tool.

Apache TCPMon (http://ws.apache.org/commons/tcpmon/) is another tool that is designed to trace web services messages. This is a Java based tool that can be used with web services to capture the messages. Because TCPMon is a message capturing tool, it can be used to intercept messages sent between client and service, and as explained earlier, can be run on the client machine, the server machine or on a third independent machine. The only catch is that you need Java installed in your system to run this tool. You can also find a C-based implementation of a similar tool with Apache Axis2/C (http://ws.apache.org/axis2/c). However, that tool does not have a graphical user interface.

There is a set of steps that you need to follow, which are more or less the same across all of these tools, in order to prepare the tool for capturing messages.

Target host name is the name of the host machine on which the service is running. As an example, if we want to debug the request sent to the Yahoo spelling suggestion service, hosted at http://search.yahooapis.com/WebSearchService/V1/spellingSuggestion, the host name would be search.yahooapis.com. We can either use the name of the host or we can use the IP address of the host because the tools are capable of dealing with both formats in place of the host name. As an example, if the service is hosted on the local machine, we could either use localhost or 127.0.0.1 in place of the host name.

Target port number is the port number on which the service hosting web server is listening; usually this is 80. As an example, for the Yahoo spelling suggestion service, hosted at http://search.yahooapis.com/WebSearchService/V1/spellingSuggestion, the target port number is 80. Note that, when the service URL does not mention any number, we can always use the default number. If it was running on a port other than port 80, we can find the port number followed by the host name and preceded with caracter ':'. As an example, if we have our web server running on port 8080 on the local machine, we would have service URL similar to http://localhost:8080/rest/04/library/book.php. Here, the host name is localhost and the target port is 8080.

Listen port is the port on which the tool will be listening to capture the messages from the client before sending it to the service. For an example, say that we want to use port 9090 as our listen port to capture the messages while using the Yahoo spelling suggestion service. Under normal circumstances, we will be using a URL similar to the following with the web browser to send the request to the service.

http://search.yahooapis.com/WebSearchService/V1/spellingSuggestion?appid=YahooDemo&query=apocalipto

When we want to send this request through the message capturing tool and since we decided to make the tools listen port to be 9090 with the tool in the middle and assuming that the tool is running on the local machine, we would now use the following URL with the web browser in place of the original URL.

http://localhost:9090/WebSearchService/V1/spellingSuggestion?appid=YahooDemo&query=apocalipto

Note that we are not sending this request directly to search.yahooapis.com, but rather to the tool listening on port 9090 on local host. Once the tool receives the request, it will capture the request, forward that to the target host, receive the response and forward that response to the web browser.

The following figure shows the Apache TCPMon tool. You can see localhost being used as the target host, 80 being the target port number and 9090 being the listening port number. Once you fill in these fields you can see a new tab being added in the tool showing the messages being captured.

Once you click on the Add button, you will see a new pane as shown in the next figure where it will show the messages and pass the messages to and from the client and service.

Before you can capture the messages, there is one more step. That is to change the client code to point to the port number 9090, since our monitoring tool is now listening on that port. Originally, we were using port 80.

$url = 'http://localhost:80/rest/04/library/book.php';

or just

$url = 'http://localhost/rest/04/library/book.php';

because the default port number used by a web server is port 80, and the client was directly talking to the service. However, with the tool in place, we are going to make the client talk to the tool listening on port 9090. The tool in turn will talk to the service. Note that in this sample we have all three parties, the client, the service, and the tool running on the same machine. So we will keep using localhost as our host name.

Now we are going to change the service endpoint address used by the client to contain port 9090. This will make sure that the client will be talking to the tool.

$url = 'http://localhost:9090/rest/04/library/book.php';

As you can see, the tool has captured the request and the response. The request appears at the top and the response at the bottom. The request is a GET request to the resource located at /rest/04/library/book.php. The response is a success response, with HTTP 200 OK code. And after the HTTP headers, the response body, which is in XML follows.

As mentioned earlier, the first step in debugging is to verify if the client has sent a request and if the service responded. In the above example, we have both the request and response in place. If both were missing then we need to check what is wrong on either side.

If the client request was missing, you can check for the following in the code.

If the response was missing, you can check for the following.

Here is a sample error scenario.

As you can see, the response is 404 not found. And if you look at the request, you see that there is a typo in the request. We have missed 'k' from our resource URL, hence we have sent the request to /rest/04/library/boo.php, which does not exist, whereas the correct resource URL is /rest/04/library/book.php.

Next let us look at the Yahoo search example that was discussed earlier to identify some advanced concepts. We want to capture the request sent by the web browser and the response sent by the server for the request. http://search.yahooapis.com/WebSearchService/V1/spellingSuggestion?appid=YahooDemo&query= apocalipto.

As discussed earlier, the target host name is search.yahooapis.com. The target port number is 80. Let's use 9091 as the listen.

Let us use the web browser to send the request through the tool so that we can capture the request and response. Since the tool is listening on port 9091, we would use the following URL with the web browser. http://localhost:9091/WebSearchService/V1/spellingSuggestion?appid=YahooDemo&query=apocalipto

When you use the above URL with the web browser, the web browser would send the request to the tool and the tool will get the response from the service and forward that to the web browser. We can see that the web browser gets the response.

However, if we have a look at the TCPMon tool's captured messages, we see that the service has sent some binary data instead of XML data even though the Web browser is displaying the response in XML format.

So what went wrong? In fact, nothing is wrong. The service sent the data in binary format because the web browser requested that format. If you look closely at the request sent you will see the following.

GET /WebSearchService/V1/spellingSuggestion?appid=YahooDemo&query=apocalipto HTTP/1.1
Host: search.yahooapis.com:9091
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.7,zh-cn;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

In the request, the web browser has used the HTTP header.

Accept-Encoding: gzip,deflate

This tells the service that the web browser can handle data that comes in gzip compressed format. Hence the service sends the data compressed to the web browser. Obviously, it is not possible to look into the XML messages and debug them if the response is compressed. Hence we should ideally capture the messages in XML format. To do this, we can modify the request message on the TCPMon pane itself and resend the message.

First remove the line

Accept-Encoding: gzip,deflate

Then click on the Resend button.

Once we click on the Resend button, we will get the response in XML format.

While forming XML as request payload or response payload, we can run into errors through simple mistakes. Some would be easy to spot but some are not. Most of the XML errors could be avoided by following a simple rule of thumb-each opening XML tag should have an equivalent closing tag. That is the common mistake that can happen while building XML payloads.

In the above diagram, if you look carefully in the circle, the ending tag for the book element is missing. A new starting tag for a new book is started before the first book is closed. This would cause the XML parsing on the client side to fail. In this case I am using the Library system sample and here is the PHP source code causing the problem.

echo "<books>";
while ($line = mysql_fetch_array($result, MYSQL_ASSOC)) {
echo "<book>";
foreach ($line as $key => $col_value) {
echo "<$key>$col_value</$key>";
}
//echo "</book>";
}
echo "</books>";

Here I have intentionally commented out printing the closing tag to demonstrate the error scenario. However, while writing this code, I could have missed that as well, causing the system to be buggy.

While looking for XML related errors, you can use the manual technique that we just used. Look for missing tags. If the process looks complicated and you cannot seem to find any XML errors in the response or request that you are trying to debug, you can copy the XML captured with the tool and run it through an XML validator tool. For example, you can use an online tool such as http://www.w3schools.com/XML/ xml_validator.asp. You can also check if the XML file is well formed using an XML parser.

There also can be situations where you could make mistakes while parsing an incoming XML message. As an example, have a look at the following code.

foreach ($xml->book as $book) {
echo "$book->id, $book->name, $book->author, $book->isbn <br/>
";
}

Here, the parsing logic assumes the following XML format.

<books>
<book>
<id>1</id>
<name>Book1</name>
<author>Auth1</author>
<isbn>ISBN0001</isbn>
</book>
</books>

There are two common possibilities where typos can happen while parsing an XML. First, the foreach statement.

foreach ($xml->book as $book) {

Here, the common mistake is to forget that the root element, book in this example corresponds to the $xml element. It is a common mistake to use

$xml->books->book

in place of

$xml->book

The second possibility is to use incorrect element names while using the child elements of a given element. In this example, we use

echo "$book->id, $book->name, $book->author, $book->isbn <br/>
";

If we were to use

$book->ISBN

instead of

$book->isbn

The client would not behave as expected and would not print the ISBN number in the output. This is because XML is case sensitive and the incoming XML has the element name<isbn> and not<ISBN>.

Best practices would help us avoid common mistakes while using REST style services and clients.

We looked into the use of tools to trace and look into the messages to figure out possible problems with request and response pairs passed between clients and services. We also looked into how we could look at the XML to figure out possible problems in building XML. We also discussed how we can locate problems in parsing an incoming XML.