Until now, we have discussed how to write a protocol buffer file that is previously written in JSON or another data format. But, how do we actually integrate it into our programs? Remember that protocol buffers are data formats, no more than that. They are a format of communication between various systems, similar to JSON. These are the practical steps we follow for using protobufs in our Go programs:

1. Install the protoc command-line tool and the proto library.
2. Write a protobuf file with the .proto extension.
3. Compile it to target a programming language (here, it is Go).
4. Import structs from the generated target file and serialize the data.
5. On a remote machine, receive the serialized data and decode it into a struct or class. 

Take a look at the following diagram:

 

The first step is to install the protobuf compiler on our machine. For this, download the protobuf package from https://github.com/google/protobuf/releases. On macOS X, we can install protobuf using this command:

brew install protobuf

On Ubuntu or Linux, we can copy protoc to the /usr/bin folder:

# Make sure you grab the latest version
curl -OL https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip
# Unzip
unzip protoc-3.3.0-linux-x86_64.zip -d protoc3
# Move only protoc* to /usr/bin/
sudo mv protoc3/bin/protoc /usr/bin/protoc

On Windows, we can just copy the executable (.exe) from https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-win32.zip to the PATH environment variable. Let us write a simple protocol buffer to illustrate how to compile and use structs from the target file. Create a folder called protofiles in $GOPATH/src/github.com/narenaryan (this is the place for our Go projects) using the following command:

mkdir $GOPATH/src/github.com/narenaryan/protofiles

Here, create a file called person.proto, which models a person's information. Add a few messages to it, as shown in the following code snippet:

syntax = "proto3";
package protofiles;

message Person {
  string name = 1;
  int32 id = 2;  // Unique ID number for this person.
  string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
  }

  repeated PhoneNumber phones = 4;
}

// Our address book file is just one of these.
message AddressBook {
  repeated Person people = 1;
}

We created two main messages called AddressBook and Person. AddressBook has a list of persons. A Person has a name, id, email, and phone Number.  In the second line, we declared the package as protofiles like this:

package protofiles;

This tells the compiler to add the generating file in relation to the given package name. Go cannot consume this .proto file directly.  We need to compile it to a valid Go file. When compiled, this package name protofiles will be used to set the  package of the output file (Go in this case). To compile this protocol buffer file, traverse to the protofiles directory and run this command: 

protoc --go_out=. *.proto

This command converts the given protocol buffer file(s) to the Go file(s) with the same name. You will see that, after running this command, there is a new file created in the same directory:

[16:20:27] naren:protofiles git:(master*) $ ls -l
total 24
-rw-r--r-- 1 naren staff 5657 Jul 15 16:20 person.pb.go
-rw-r--r--@ 1 naren staff 433 Jul 15 15:58 person.proto

The new file name is person.pb.go. If we open and inspect this file, it contains the following important block:

........
type Person_PhoneType int32

const (
  Person_MOBILE Person_PhoneType = 0
  Person_HOME   Person_PhoneType = 1
  Person_WORK   Person_PhoneType = 2
)

var Person_PhoneType_name = map[int32]string{
  0: "MOBILE",
  1: "HOME",
  2: "WORK",
}
var Person_PhoneType_value = map[string]int32{
  "MOBILE": 0,
  "HOME":   1,
  "WORK":   2,
}
.......

This is just a part of that file. There will be many getter and setter methods created for the given structs such as Person and AddressBook in the output file. This code is automatically generated. We need to consume this code in the main program to create protocol buffer strings. Now, let us create a new directory called protobufs. This holds the main.go file that uses the Person struct from the person.pb.go file:

mkdir $GOPATH/src/github.com/narenaryan/protobufs

Now, for Go to serialize a struct to the protobinary format, we need to install the Go proto driver. Install it using the go get command:

go get github.com/golang/protobuf/proto

After this, let us compose main.go:

package main

import (
  "fmt"

  "github.com/golang/protobuf/proto"
  pb "github.com/narenaryan/protofiles"
)

func main() {
  p := &pb.Person{
    Id:    1234,
    Name:  "Roger F",
    Email: "[email protected]",
    Phones: []*pb.Person_PhoneNumber{
      {Number: "555-4321", Type: pb.Person_HOME},
    },
  }

  p1 := &pb.Person{}
  body, _ := proto.Marshal(p)
  _ = proto.Unmarshal(body, p1)
  fmt.Println("Original struct loaded from proto file:", p, "
")
  fmt.Println("Marshaled proto data: ", body, "
")
  fmt.Println("Unmarshaled struct: ", p1)
}

We are importing the protocol buffer (pb) from the protofiles package. There are structs that are mapped to the given protobuf in the proto files. We used the Person struct and initialized it. Then, we serialized the struct using the proto.Marshal function. If we run this program, the output looks like this:

go run main.go
Original struct loaded from proto file: name:"Roger F" id:1234 email:"[email protected]" phones:<number:"555-4321" type:HOME >

Marshaled proto data: [10 7 82 111 103 101 114 32 70 16 210 9 26 14 114 102 64 101 120 97 109 112 108 101 46 99 111 109 34 12 10 8 53 53 53 45 52 51 50 49 16 1]

Unmarshaled struct: name:"Roger F" id:1234 email:"[email protected]" phones:<number:"555-4321" type:HOME >

The second output of marshaled data is not intuitive because the proto library serializes data into binary bytes. Another good thing about protocol buffers in Go is that the structs generated by compiling the proto files can be used to generate JSON on the fly. Let us modify the preceding example to this. Call it main_json.go:

package main

import (
  "fmt"

  "encoding/json"
  pb "github.com/narenaryan/protofiles"
)

func main() {
  p := &pb.Person{
    Id:    1234,
    Name:  "Roger F",
    Email: "[email protected]",
    Phones: []*pb.Person_PhoneNumber{
      {Number: "555-4321", Type: pb.Person_HOME},
    },
  }
  body, _ := json.Marshal(p)
  fmt.Println(string(body))
}

If we run this, it prints a JSON string that can be sent to any client that can understand JSON:

go run main_json.go

{"name":"Roger F","id":1234,"email":"[email protected]","phones":[{"number":"555-4321","type":1}]}

Any other language or platform can easily load this JSON string and use the data instantly. So, what is the benefit of using protocol buffers instead of JSON? First of all, protocol buffers are intended for two backend systems to communicate with each other with less overhead. Since the size of the binary is less than text, protocol marshaled data is of less size than JSON.

But, protocol buffers are just a data format. They don't have any importance if we don't communicate. So here, protocol buffers are used to pass messages between two end systems in the form of RPC. We saw how RPC works and also created an RPC client and server in the previous chapters. Now, we are going to extend that knowledge to use Google Remote Procedure Call (GRPC) with protocol buffers to scale our microservice communications. A server and client, in this case, can talk with each other in protocol buffer format.