In this section, we will troubleshoot the error code that exists with the donut emoji (refer to Chapter 15, Installing Linkerd, for more information). Since the microservice is throwing an error, using Linkerd's dashboard, we will investigate the HTTP traffic routes across its service, deployment, and pods to debug this issue.

Switch to the emojivoto application in your browser by going to http://emojivoto.linkerd.local. Vote for the doughnut emoji (third on the top row) and notice the HTTP 404 error. Now, follow these steps:

1. Go to Resources |  Namespaces |  All in http://dashboard.linkerd.local. Notice that the success rate is less than 100% for the emojivoto namespace, which is due to the doughnut emoji error:

2. Click on the emojivoto link under the HTTP metrics section to view the service discovery graph. This shows the dependency information of the emojivoto microservice application:

In the preceding screenshot, note that the emoji deployment shows a 100% success rate while voting, while web doesn't.

3. Click the web deployment. You will see a live traffic metrics visualization with indicators such as success rate (SR), P99 latency, and request per second (RPS) for each microservice with a dependency relationship:

4. Scroll down to see a feed of Live Calls of requests. vote-bot generates continuous traffic to the application:

Notice the call to the /emojivoto.v1.VotingService/VoteDoughnut path, which results in a 0% success rate. Through Linkerd, by looking at the live feed of the data, we now know which REST API endpoint is failing.

5. On the far side of this line from the previous screenshot, click on the tap icon, which will only show the live list of requests from the endpoint.

Notice that the gRPC status is Unknown. With this, we have drilled down to the exact REST API call, which is failing.

6. Linkerd shows a tap command line, along with an argument, which is very nice to watch without using the UI:

$ linkerd tap deployment/web --namespace emojivoto 
--to deployment/voting --to-namespace emojivoto 
--path /emojivoto.v1.VotingService/VoteDoughnut

After running the preceding command, press Ctrl + C after a few lines of output.

Now, let's switch to the booksapp application called http://booksapp.linkerd.local. Booksapp comes with a traffic generator that keeps on sending traffic to the application to explore service discovery capabilities. Let's explore service discovery through Linkerd's dashboard:

1. Switch back to the Linkerd dashboard and click Resources | Namespaces | All. In the right pane, look for linkerd-lab and click on it to open the namespace:

Notice that the service discovery traffic service is sending traffic to the webapp service, which is sending traffic to both books and authors. The books service communicates with the authors service.

Notice that the deployments and success rate is not 100%, indicating that something is wrong. This is intentional and designed by the authors of the application to show the built-in troubleshooting capabilities of Linkerd.

2. Switch to the tab running http://booksapp.linkerd.local, and scroll down to the Add a Book section. Here, write any title, select an author and page count, and hit Add Book:

The book may be added, or you may receive an Internal Server Error. You may have to try a few times to add the book. Note that if you are unsuccessful in adding a book after a few attempts, continue to the next step for troubleshooting. 

Now, let's go back to the dashboard to see if we can find the source of the problem.

3. Click on the webapp deployment to see more details:

Notice that deploy/authors shows a 100% success rate – this service is working fine, but the deploy/books success rate is less than 100% and so are the success rates of deploy/webapp and deploy/traffic.

4. Scroll further down and view the live traffic feed for webapp:

Here, we can see that deploy/traffic (generator) is sending POST requests to /books. 

5. Click the pop-out arrow next to deploy/traffic and check the name of the source and destination deployment microservices, which is from deploy/traffic to deploy/webapp.

Similarly, check for deploy/books, which is from deploy/webapp to deploy/books. For operations, this information is crucial and needs to be communicated to developers.

6. Click the tap icon on the line showing /books.json to drill down and view only the live requests coming to details/book: