In the last chapter, you created a simple echo server and client, demonstrating some of the power of Bazel to navigate and manage multiple languages with minimal setup. A noted shortcoming from that example stems from the definition of the transmitted object: both languages required independent definitions of the object. Over time, this easily can cause a literal breakdown in communication as two (or more) definitions of the transmitted object drift out of sync.
In this chapter, we are going to introduce a construct to handle this very problem, the Protocol Buffer (often referred to as protobuf). Yet another creation from Google, Protocol Buffers provide a way to describe the structure of objects in a declarative and type-safe fashion and provide a wire format for serialization. Protocol Buffer definitions are intrinsically language agnostic. Once created, a Protocol Buffer definition can then be compiled into a particular language (there is vast language support for Protocol Buffers) in order to read/write from the wire format into a language native object.
While Protocol Buffers are not wedded to Bazel per se, Bazel provides some fantastic support for them, making it easy to add them to a project and make use of them across multiple languages. In addition, due to Bazel’s dependency management, it is also very easy to make a change at the Protocol Buffer definition and ensure that all the dependent projects can at least compile against the new definition.
Setting Up Your Workspace
Adding support for Protocol Buffers
Creating Your First Protocol Buffer
Note
An astute reader will note that in the prior chapter, we had started off with creating our BUILD file before working with the code. This switch here is intentional, and we will address it very shortly.
Defining a simple Protocol Buffer
Save that to transmission_object.proto.
Let’s take a brief moment to examine what we have written. The initial line (syntax) is used to specify to the compiler what version of Protocol Buffers is being used here (at the time of this writing, the latest version of Protocol Buffers is version 3).
The following line (package) is used to define the conceptual package that this Protocol Buffer is defined within. This is very similar to the Java or Go notions of packages.
Finally, we have the definition of the message itself. This most closely resembles a C-style struct, with a name given to the object (TransmissionObject) and a basic set of type-specified fields (i.e., value and message, with types float and string, respectively). The one minor addition here is the addition of a field number; this is used to define unique identifiers to each of the members of the message. This is important since these are used to add and remove data members while keeping backward compatibility.
Creating the build target for the Protocol Buffer
Save your BUILD file.
Note
Those already familiar with Bazel might find the explicit inclusion of proto_library to be strange. The proto_library rule used to be a part of the core of Bazel itself. This is an example of the evolution of Bazel by moving specific constructs out of the core and into explicit packages.
One of the first things that you should notice is that, for at least the first time you build the Protocol Buffer target, there is a noticeable delay compared to executions in prior chapters. In this case, the dependency for the Protocol Buffer compiler is both being pulled down and being compiled for your target machine. Once the protobuf compiler is itself compiled, then it can then compile your protobuf definition.
Don’t worry. This particular slowdown should only be limited to the first time you actually run the Protocol Buffer compiler; once created, the Protocol Buffer compiler will remain cached (until you change a dependency or execute bazel clean).
Using the Protocol Buffer in Java
Although we have successfully compiled the Protocol Buffer, all we really have done is create a language-agnostic descriptor for it; in order to really make use of it, we will need to create a language-specific target for it. We will start with creating one in Java. Once again, we take advantage of the fact that Java, being one of the built-in languages of Bazel, comes with support built-in for Java-based Protocol Buffers.
Creating the Java Proto Library Target
Creating the Java Protocol Buffer library
Congratulations! You have a target that we can actually use in a Java program.
Note
In this case, you did not need to explicitly load java_proto_library. However, given Bazel’s evolution, bear in mind that some future iteration may require you to explicitly load the rule.
Using Your Java Protocol Buffer Target
In the last chapter, you had created a simple Java echo client using JSON. Here, we will make use of almost the same code for the Protocol Buffer example, with only a few minor changes.
Protocol Buffer version of the echo client
Save this to EchoClient.java.
- transmission_object
This is the package as specified within the original transmission_object.proto file.
- TransmissionObjectOuterClass
This is a class generated to encapsulate any messages contained within the Protocol Buffer definition.
This is an artifact of Java’s one (outer)-class-per-file rule; technically, we could have had multiple messages within our Protocol Buffer file, but we can only have a single class within a Java file.
This allows us to create multiple Protocol Buffer messages for use in Java.
- TransmissionObject
The actual Java object that represents the original Protocol Buffer message.
Within the code itself, the Java Protocol Buffer instance is created using a builder pattern, which allows you to set the various fields and then generate an invariant instance of the TransmissionObject. This object is then able to directly write itself to an output stream as well as parse itself from an input stream.
Creating the BUILD target for the EchoClient
Congratulations! You’ve successfully updated your client to use Protocol Buffers. However, once again you have a client with nothing to connect to. Now, we will make the necessary changes to the server side to also handle our Protocol Buffer definition.
Note
One might be tempted to run this new version of our client against the last chapter’s server. Although you are welcome to do so, it is important to know that this will not work, since the client and server are talking into different protocols (JSON vs. Protocol Buffer); the bytes are going to be interpreted differently.
Although Protocol Buffers do support translation to/from JSON, you would need to explicitly specify that within the code.
Using the Protocol Buffer in Go
In the last section, we were able to take advantage of the fact that Java is one of the built-in languages of Bazel to jump right into development. However, since Go is not one of those core languages, we will need to do some additional setup. Fortunately, most of this will look familiar from prior chapters.
Adding the Go rules to the project
Save your WORKSPACE file.j
Note
In this particular instance, we specified to load io_bazel_rules_go prior to rules_proto. The reason is that there can be conflicts between the underlying dependencies between these two packages. Ordering them in this fashion removes the issue. However, this is an item to watch out for as you construct your WORKSPACE dependencies moving forward.
Creating the Go Proto Library Target
As with the inclusion of the Go functionality into our project, we will need to explicitly bring the necessary rules into our BUILD file as we create our Go proto library target.
Creating the Go proto library target
As with your prior experience when creating the Java Protocol Buffer target, you likely will notice a slightly longer-than-normal build time. Once again, this is normal, since the language-specific (i.e., Go) plug-in is being compiled; as before, after the first time, this is cached, and later builds will go much quicker.
Once again, congratulations, since we now have a target that we can actually use in our Go program. Now let’s modify our echo server to take advantage of this.
Using Your Go Protocol Buffer Target
As with the echo client, in the last chapter, you had created a version of the echo server that bounced back the received JSON message (with some modifications). As before, we are going to be able to make some slight changes to our original program to handle Protocol Buffers.
Protocol Buffer version of the Go server
Save this to echo_server.go.
While the changes are not a complete drop-in replacement for the JSON, the final result is extremely close to what we had in the previous chapter.
Of particular note, we have to bring in a dependency on the proto library itself (github.com/golang/protobuf/proto) in order to perform the unmarshaling/marshaling of the object from/to the data streams. Unlike the previous dependency on the encoding package in Go, we will need to account for this when we specify the dependencies within the BUILD file.
Adding the echo server build target
Save your BUILD file.
Dependencies From Dependencies
An astute reader will notice that the dependency we have specified for the go_default_library is not actually specified within the WORKSPACE file; however, the preceding code still compiles without complaint.
The source of this additional dependency stems from the function that we called to set up the additional dependencies for the Go rules (i.e., go_rules_dependencies), which pulled in additional dependencies, including the above-listed one.
Although technically this is “explicitly” specified within the WORKSPACE file, it is obfuscated by the use of the dependencies function. In this case, we are taking advantage of the fact that all of these versions of the particular dependencies are meant to work in concert.
If this is too implicit, then a couple things can be done: (1) Explicitly specify a dependency within the WORKSPACE file; this will replace the version of the implicit dependency. (2) Pull the dependency into your project (e.g., through a third_party directory).
The decision on which route to pursues relates to how tightly you want to control your dependencies. (1) may be easier as a way to quickly get up and running and make it easier to change dependencies later on. However, again, (2) provides the strongest guarantee for build reproducibility.
Echo Using Protocol Buffers
Having reconstructed our echo client and server with Protocol Buffers, we are now ready to have them start talking to each other again.
Congratulations! You’ve recreated the echo client/server using Protocol Buffers.
Dependency Tracking and Management
Compared to the previous chapter, there are some slight formatting differences, but the outputs are effectively the same. This begs an obvious question: Why did we reinvent everything from the last chapter? The answer lies in how we manage changes to our selected transmission object; this, in turn, showcases the ability of Bazel to perform dependency management, even across multiple languages.
Dependency trees for the JSON echo client and server
As noted earlier, a major downfall is that the definitions for the JSON objects are specific to each language without reference to one another. Any change in the API (i.e., by changing the JSON object) needs to be done in both locations, making it prone to errors when you change it in one place and not the other.
Dependency tree for the Protocol Buffer echo client and server
Now the dependency tree for the Protocol Buffer echo client and server is still relatively simple and likely familiar to anyone coding at scale. However, the remarkable aspects of it are the following: (a) we are tying together dependencies across three languages (i.e., Java, Go, Protocol Buffer); (b) by doing so, we are solving the API change management problem from the JSON client; and (c) we have done so using relatively little setup code.
Change Management in Action
Having set up our build dependency tree, now let’s see it work in practice. First, we will ensure that all of our targets are already up to date.
Changes to the echo_server only affect a single target
Removing the Message field from TransmissionObject
Now, let’s attempt to rebuild both echo_client and echo_server:
Note
You will notice that we add the flag keep_going (or a shortened version of -k) to the build command. Without this, the “build everything” command would stop at the first failure; using it, we see all targets that are failing.
Changes to the transmission_object_proto affect all targets
If you’d like, you can double check the modification date after you fix the code by restoring the field.
Final Word
In this chapter, you were able to very simply add and use the necessary functionality for Protocol Buffers. Along the way, you also got to see firsthand the abilities of Bazel to very easily and powerfully manage build dependencies, even between code written in multiple languages. Although the examples here truly only scratched the surface, already you should be able to see the possibilities provided by a simple and standard declarative build language.
Protocol Buffers also reinforced Bazel’s capabilities at handling multiple languages with ease. At the same time, you also got a glimpse into easily using Protocol Buffers for serialization across multiple languages.
In later chapters, we will be returning to more use of Protocol Buffers with Bazel. For the moment, however, we will take a step back and look at some facilities that Bazel provides for code organization.