Installing Jaeger

All information captured on each microservice should be reported to a server that will collect and store this information, so it can be queried later.

So, before instrumenting the source code of our microservices, first we need to install the Jaeger server and its components. Jaeger provides an all-in-one distribution composed of the Jaeger UI, collector, query, and agent, with an in-memory storage component.

We can install this distribution with the following command:

$ oc process -f 
http://raw.githubusercontent.com/jaegertracing/jaeger-openshift
/master/
all-in-one/jaeger-all-in-one-template.yml | oc create -f -

deployment.extensions "jaeger" created
service "jaeger-query" created
service "jaeger-collector" created
service "jaeger-agent" created
service "zipkin" created
route.route.openshift.io "jaeger-query" created

That’s it! Now we can start modifying our microservices to report the tracing information to this server.

Modifying Microservices for Distributed Tracing

Now that we have our Jaeger server installed, it’s time to modify our microservices to report information to it.

All Java OpenTracing libraries can be configured using environment variables. The only environment variable that is required is JAEGER_SERVICE_NAME, which tells Jaeger the name of the service. This will be declared as an ENV instruction in each microservice’s Dockerfile. Because we will also need other environment variables on all microservices to configure the tracing collector, they will be declared using a Kubernetes feature called a ConfigMap, and then will be consumed by all the microservices.

<!-- OpenTracing -->
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-opentracing-starter</artifactId>
</dependency>
<dependency>
    <groupId>io.jaegertracing</groupId>
    <artifactId>jaeger-tracerresolver</artifactId>
    <version>0.32.0</version>
</dependency>
<dependency>
    <groupId>io.jaegertracing</groupId>
    <artifactId>jaeger-thrift</artifactId>
    <version>0.32.0</version>
</dependency>

@SpringBootApplication
@CamelOpenTracing
public class MySpringBootApplication {

    /**
     * A main method to start this application.
     */
    public static void main(String[] args) {
        SpringApplication.run(MySpringBootApplication.class, args);
    }

}

FROM fabric8/java-alpine-openjdk8-jdk
ENV JAVA_APP_JAR api-gateway-1.0.jar
ENV AB_OFF true
ENV JAEGER_SERVICE_NAME API-Gateway
ADD target/api-gateway-1.0.jar /deployments/

$ mvn clean package
$ docker build -t rhdevelopers/api-gateway:1.0 .
$ oc delete pod -l app=api-gateway

<!-- OpenTracing -->
<dependency>
    <groupId>io.opentracing.contrib</groupId>
    <artifactId>opentracing-spring-web-starter</artifactId>
    <version>1.0.1</version>
</dependency>
<dependency>
    <groupId>io.jaegertracing</groupId>
    <artifactId>jaeger-tracerresolver</artifactId>
    <version>0.32.0</version>
</dependency>
<dependency>
    <groupId>io.jaegertracing</groupId>
    <artifactId>jaeger-thrift</artifactId>
    <version>0.32.0</version>
</dependency>

@RestController
@RequestMapping("/api")
@ConfigurationProperties(prefix="greeting")
public class GreeterRestController {

    private RestTemplate template = new RestTemplate();

    private String saying, backendServiceHost;

    private int backendServicePort;

    @RequestMapping(value = "/greeting",
            method = RequestMethod.GET, produces = "text/plain")
    @HystrixCommand(fallbackMethod = "fallback")
    public String greeting(){
        template.setInterceptors(
                Collections.singletonList(
                        new TracingRestTemplateInterceptor(
                                TracerResolver.resolveTracer())));

        String backendServiceUrl = String.format(
                "http://%s:%d/api/backend?greeting={greeting}",
                backendServiceHost, backendServicePort);

        System.out.println("Sending to: " + backendServiceUrl);


        BackendDTO response = template.getForObject(
                backendServiceUrl, BackendDTO.class, saying);

        return response.getGreeting() + " at host: " +
                response.getIp();
    }

    //fallback method and setters
}

FROM fabric8/java-alpine-openjdk8-jdk
ENV JAVA_APP_JAR hello-springboot-1.0.jar
ENV AB_OFF true
ENV JAEGER_SERVICE_NAME hello-springboot
ADD target/hello-springboot-1.0.jar /deployments/

$ mvn clean package -DskipTests
$ docker build -t rhdevelopers/hello-springboot:1.0 .
$ oc delete pod -l app=hello-springboot

<!-- OpenTracing -->
<dependency>
    <groupId>io.thorntail</groupId>
    <artifactId>microprofile-opentracing</artifactId>
</dependency>
<dependency>
    <groupId>io.thorntail</groupId>
    <artifactId>jaeger</artifactId>
</dependency>

@Path("/api")
public class GreeterRestController {

    @Inject
    @ConfigProperty(name="greeting.saying",
        defaultValue = "Hello")
    private String saying;

    @Inject
    @ConfigProperty(name = "greeting.backendServiceHost",
        defaultValue = "localhost")
    private String backendServiceHost;

    @Inject
    @ConfigProperty(name = "greeting.backendServicePort",
        defaultValue = "8080")
    private int backendServicePort;

    @GET
    @Produces("text/plain")
    @Path("greeting")
    @CircuitBreaker
    @Timeout
    @Fallback(fallbackMethod = "fallback")
    @Traced(operationName = "greeting")
    public String greeting() {
        String backendServiceUrl = String.format("http://%s:%d",
                backendServiceHost,backendServicePort);

        System.out.println("Sending to: " + backendServiceUrl);

        Client client = ClientTracingRegistrar
                .configure(ClientBuilder.newBuilder()).build();

        BackendDTO backendDTO = client.target(backendServiceUrl)
                .path("api")
                .path("backend")
                .queryParam("greeting", saying)
                .request(MediaType.APPLICATION_JSON_TYPE)
                .get(BackendDTO.class);

        return backendDTO.getGreeting()
                + " at host: " + backendDTO.getIp();
    }

    public String fallback(){
        return saying + " at host "  +
        System.getenv("HOSTNAME") + " - (fallback)";
    }

}

FROM fabric8/java-alpine-openjdk8-jdk
ENV JAVA_APP_JAR demo-thorntail.jar
ENV AB_OFF true
ENV JAEGER_SERVICE_NAME hello-microprofile
ADD target/demo-thorntail.jar /deployments/

$ mvn clean package -DskipTests
$ docker build -t rhdevelopers/hello-microprofile:1.0 .
$ oc delete pod -l app=hello-microprofile

<!-- OpenTracing -->
<dependency>
    <groupId>io.jaegertracing</groupId>
    <artifactId>jaeger-client</artifactId>
    <version>0.32.0</version>
</dependency>

//Extract the parent Span from the headers
SpanContext parentSpan = tracer
        .extract(Format.Builtin.HTTP_HEADERS,
                new TextMapExtractAdapter(headers));

//Start a new Span as a child of the Parent Span
Scope scope = tracer
        .buildSpan("backend-servlet")
        .asChildOf(parentSpan)
        .startActive(true);

//Perform work

 scope.span().finish();

@WebServlet(urlPatterns = {"/api/backend"})
public class BackendHttpServlet extends HttpServlet {

    private Tracer tracer = TracerResolver.resolveTracer();

    @Override
    protected void doGet(HttpServletRequest req,
        HttpServletResponse resp)
            throws ServletException, IOException {

        //Place the HTTP headers in a HashMap
        final HashMap<String, String> headers = new HashMap<>();
        Enumeration<String> headerNames = req.getHeaderNames();
        while (headerNames.hasMoreElements()){
            String name = headerNames.nextElement();
            String value = req.getHeader(name);
            headers.put(name, value);
        }
        //Extract the parent Span from the headers
        SpanContext parentSpan = tracer
                .extract(Format.Builtin.HTTP_HEADERS,
                        new TextMapExtractAdapter(headers));

        //Start a new Span as a child of the parent Span
        Scope scope = tracer
                .buildSpan("backend-servlet")
                .asChildOf(parentSpan)
                .startActive(true);

        resp.setContentType("application/json");

        ObjectMapper mapper = new ObjectMapper();
        String greeting = req.getParameter("greeting");

        ResponseDTO response = new ResponseDTO();
        response.setGreeting(greeting + " from cluster Backend");
        response.setTime(System.currentTimeMillis());
        response.setIp(getIp());

        PrintWriter out = resp.getWriter();
        mapper.writerWithDefaultPrettyPrinter()
                .writeValue(out, response);

        scope.span().finish();
    }

    private String getIp() {
        String hostname = null;
        try {
            hostname = InetAddress.getLocalHost()
                                    .getHostAddress();
        } catch (UnknownHostException e) {
            hostname = "unknown";
        }
        return hostname;
    }
}

FROM jboss/wildfly:10.0.0.Final
ENV JAEGER_SERVICE_NAME backend
ADD target/ROOT.war /opt/jboss/wildfly/standalone/deployments/

$ mvn clean package -DskipTests
$ docker build -t rhdevelopers/backend:1.0 .
$ oc delete pod -l app=backend

Configuring Microservices Using ConfigMap

As we discussed previously, the configuration of Jaeger Java clients is done through environment variables. In any case, the only environment variable that is required is JAEGER_SERVICE_NAME, which we defined in every Dockerfile.

If you look at the logs of any microservices, you should see a message like the following:

Initialized tracer=JaegerTracer(
    version=Java-0.32.0,
    serviceName=API-Gateway,
    reporter=RemoteReporter(
        sender=UdpSender(),
        closeEnqueueTimeout=1000), sampler=
            RemoteControlledSampler(
            maxOperations=2000,
            manager=HttpSamplingManager(
                hostPort=localhost:5778),
                sampler=ProbabilisticSampler(
                    tags={sampler.type=probabilistic,
                    sampler.param=0.001})),
...

This means that the default configuration for the tracer uses a UDP Sender that sends the tracing information to localhost:5778. The ProbabilisticSampler defines that only 0.1% (0.001) of the requests will be traced. Tracing only 0.1% of the requests seems fine for production usage. However, for our tutorial we will change the tracer to collect all requests.

According to the environment variable definitions in the jaeger-core module, we will need to configure the following keys/values for all microservices:

• JAEGER_ENDPOINT: http://jaeger-collector:14268/api/traces

• JAEGER_REPORTER_LOG_SPANS: true

• JAEGER_SAMPLER_TYPE: const

• JAEGER_SAMPLER_PARAM: 1

These environment variables configure the tracer to send an HTTP report to http://jaeger-collector:14268/api/traces. Every tracer report will be logged, and we will use a constant sampler that collects 100% of the requests (1 of 1).

We could use the command oc set env for every microservice, but we want to try something more advanced. We will create a Configmap Kubernetes object to hold this configuration. Later we will consume the configurations using environment variables, but don’t worry about the details right now.

$ oc set env deployment --all --from=configmap/jaeger-config
deployment.extensions/api-gateway updated
deployment.extensions/backend updated
deployment.extensions/hello-microprofile updated
deployment.extensions/hello-springboot updated
deployment.extensions/jaeger updated

Note that it will cause the deployment of every microservice and that the logs now for any microservice will contain different information about the tracer:

Initialized tracer=JaegerTracer(
    version=Java-0.32.0,
    serviceName=API-Gateway,
    reporter=CompositeReporter(
        reporters=[RemoteReporter(
            sender=HttpSender(),
            closeEnqueueTimeout=1000),
            LoggingReporter(
...
                sampler=ConstSampler(
                  decision=true,
                  tags={sampler.type=const, sampler.param=true}),
                  tags={hostname=api-gateway-78f6f8dcd7-wckvx,
                  jaeger.version=Java-0.32.0,
                  ip=172.17.0.16},
...

Wait for the pods to come alive, and try making a request to the microservice:

$ curl http://api-gateway-tutorial.$(minishift ip).nip.io
  /api/gateway
["Hello from cluster Backend at host: 172.17.0.13",
 "Hello Spring Boot from cluster Backend at host: 172.17.0.13"]

You should see something like this in the logs:

i.j.internal.reporters.LoggingReporter
: Span reported: d716584c2fab233d:d716584c2fab233d:0:1

Analyzing the Tracing in Jaeger

Now that we’ve made a request using the curl command, and we’ve seen in the logs that this request generated a tracer Span that was reported to Jaeger, we can open the Jaeger UI to look at some important information. To open the UI in your browser, use the following command:

$ minishift openshift service jaeger-query --in-browser

In the top menu, select Dependencies, and then select DAG. Note that the generated dependency graph is similar to what we expected (Figure 7-1 from Jaeger and Figure 7-2 from our architecture show the same pattern). The number 1 in the Jaeger DAG indicates the number of requests between the microservices.

Figure 7-1. Jaeger dependencies

Figure 7-2. Calling another service

Now click Search in the top menu, and select the API-Gateway service. Scroll down the page, and click the Find Traces button. You should see the tracing generated by your request with the curl command, as shown in Figure 7-3.

Figure 7-3. Jaeger tracing

Click on the trace, and Jaeger will open the details. It’s easy to see that the api-gateway service made parallel requests to hello-microprofile and hello-springboot. You can click on the details of each Span to verify the path walked by the request inside the Camel routes until it reached the microservice. Figure 7-4 shows the Span details.

Figure 7-4. Span details

Feel free to go ahead and search for the backend service spans.

Where to Look Next

In this chapter, you learned about distributed tracing, the CNCF OpenTracing specification, and the Jaeger implementation. You also learned how to instrument different technologies to collect and report tracing information, and learned how to use ConfigMaps to store and spread the configuration. Tracing is a complex subject, and we just covered the basics without going deeper into how the tracing happens. Check out the following links for more information:

• OpenTracing

• Jaeger

• The camel-opentracing component

• Jaeger bindings for the Java OpenTracing API

• “Using OpenTracing with Jaeger to Collect Application Metrics in Kubernetes” by Diane Mueller-Klingspor

• “OpenShift Commons Briefing #82: Distributed Tracing with Jaeger & Prometheus on Kubernetes” by Diane Mueller