Protocol Buffers

Protocol buffers, commonly known as protobuf, are a mechanism that allow for serializing structured data.

Why?

Type safety

The protocol buffer language allows you to specify a schema for your structured data that will be sent over a network. This schema can then be used to generate code to unpack the data in a multitude of programming languages. The existence of a schema allows for type safety, as the types are built into the fields--a message cannot be sent that doesn't adhere to the schema. This reduces the need to write complex type validation code in your request handlers, as when you work with protobufs, you've already validated your requests against the schema at compile time.

The ideal use case here is internal APIs, where we have full control over both the client and server code. While it's good practice to have a human-friendly JSON API for external users to consume, the internal APIs of backend services that need to operate closely with each other to accomplish their tasks can more predictably interact with each other if they share a protobuf definition for the expected request and response structures.

Flexibility

Having a schema allows us to more clearly add and deprecate fields in our requests and responses, and if needed we can even version our entire API with ease.

Speed

While JSON is a text-based format, Protobuf is a binary format, which means that it will be much faster to transfer due to the more compact size of the message.

Because of this performance benefit, protobuf is a good choice for internal APIs that need to care about high throughput, like when you have a complex web of internal microservices that need to do all kinds of low-latency communications with each other before returning a JSON response to an external client. Popular distributed systems like Kubernetes, Mesos, and etcd all rely on protobuf to communicate between their components, due to their need for high-speed coordination.

So what are the files in this directory?

log.proto is the protobuf definition, where you define the schema you want requests and responses for your app to follow. Note the numbered fields. Don't change the numbers; just add new fields if you want to do something like change an existing field's type.

log.pb.go is the generated Go code that was generated based on the schema in log.proto. This was generated by running the make compile target. Check out the Makefile to see the actual generation command being run under the hood; the Makefile just makes it easier to run that long command in your terminal -- anytime you change the protobuf definition, you'll need to regenerate the code, so you'll likely be using this command fairly often! Note that this example is just generating Go code, but if you install protobuf runtimes for different languages, you would be able to use protoc to generate code for those as well.