Protocol buffer and gRPC: Beginners guide

Intro

Modern software applications rarely operate in isolation. Rather, they are connected with each other through computer networks and communicate and coordinate their actions by passing messages to one another.

Wether you are looking to weather in your iPhone, or Paying a service/product in websites, inter and public communications between different code bases is always required to implement such functionalities.

The problem

There is more than a thousand programming languages, and lot of different communications

What are protocol buffers?

https://developers.google.com/protocol-buffers

https://github.com/protocolbuffers/protobuf

https://en.wikipedia.org/wiki/Protocol_Buffers

Accordring to Protocol buffer official doocumentation:

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

Accordring to Wikipedia:

Protocol Buffers (Protobuf) is a free and open-source cross-platform data format used to serialize structured data. It is useful in developing programs to communicate with each other over a network or for storing data. The method involves an interface description language that describes the structure of some data and a program that generates source code from that description for generating or parsing a stream of bytes that represents the structured data.

As for comparison:

Comparison of data-serialization formats - Wikipedia

syntax = "proto3" // this defines which version of protocol buffer we use

// the messages we send is defined with "message"
// field type (int, string, boolean)
// field name
// field tag
message myMessage {
    int32 id = 1;
    string first_name = 2;
    string last_name = 3;
}

Scalar types:

Language Guide (proto3) | Protocol Buffers | Google Developers

https://developers.google.com/protocol-buffers/docs/proto3#scalar

For numbers we find this types:

double, float, int32, int64, uint32, uint64, sint32, sint64, fixed32, fixed64, sfixed64.

We find other common ones:

bool for boolean,

string for strings, which are UTF-8 or 7-bit ASCII text.

and bytes. or bytes array, we can use bytes for maybe a small image or something like that.

All types are zero valued

When a message is parsed, if the encoded message does not contain a particular singular element, the corresponding field in the parsed object is set to the default value for that field. These defaults are type-specific:

• For strings, the default value is the empty string.
• For bytes, the default value is empty bytes.
• For bools, the default value is false.
• For numeric types, the default value is zero.
• For Enum types, default value is first value.
• For repeated values, empty list.

syntax = "proto3"

message Person {
    uint32 age = 1;
    string first_name = 2;
    string last_name = 3;
    bytes small_picture = 4;
    bool profile_verified = 5;
    float height = 6;
}

Tags

As you can see, each field in the message definition has a unique number. These field numbers are used to identify your fields in the message binary format, and should not be changed once your message type is in use. Note that field numbers in the range 1 through 15 take one byte to encode, including the field number and the field's type (you can find out more about this in Protocol Buffer Encoding). Field numbers in the range 16 through 2047 take two bytes. So you should reserve the numbers 1 through 15 for very frequently occurring message elements. Remember to leave some room for frequently occurring elements that might be added in the future.

The smallest field number you can specify is 1, and the largest is 229 - 1, or 536,870,911. You also cannot use the numbers 19000 through 19999 (FieldDescriptor::kFirstReservedNumber through FieldDescriptor::kLastReservedNumber), as they are reserved for the Protocol Buffers implementation—the protocol buffer compiler will complain if you use one of these reserved numbers in your .proto. Similarly, you cannot use any previously reserved field numbers.

Tags in protocol buffers are a building block, they can be anywhere from

Smallest tag 1

Largest tag = 536 870 911 or 2 ex 29 -1

There is a special range between 19000 19999

Tags from 1 to 15 use 1 byte in space, so using them for frequently populated fields to maximise efficiency.

Tags from 16 to 2047 use 2 bytes.

Tag rules

• singular: a well-formed message can have zero or one of this field (but not more than one). And this is the default field rule for proto3 syntax.
• repeated: this field can be repeated any number of times (including zero) in a well-formed message. The order of the repeated values will be preserved.

syntax = "proto3"

/* multi
line
comment */
message Person {
    uint32 age = 1;
    string first_name = 2;
    string last_name = 3;
    bytes small_picture = 4;
    bool profile_verified = 5;
    float height = 6;
    repeated string address = 7;

    // enums of the possible eye color
    enum EyeColor {
        UKNOWN_EYE_COLOR = 0;
        EYE_GREEN = 1;
        EYE_BROWN = 2;
        EYE_BLUE = 3;
    }
    EyeColor eye_color = 8

}

Given we want to add dates, we can define the date message and then just use it in the message we want

syntax = "proto3"

/* multi
line
comment */
message Person {
    uint32 age = 1;
    string first_name = 2;
    string last_name = 3;
    bytes small_picture = 4;
    bool profile_verified = 5;
    float height = 6;
    repeated string address = 7;

    // enums of the possible eye color
    enum EyeColor {
        UKNOWN_EYE_COLOR = 0;
        EYE_GREEN = 1;
        EYE_BROWN = 2;
        EYE_BLUE = 3;
    }
    EyeColor eye_color = 8
    Date birthday = 9
}

message Date{
    int32 year = 1;
    int32 month = 2;
    int32 day = 3;
}

Import from other proto files

To import a type from another file, we have to use import syntax, where we start from the folder directory

import "my_root_project/date.proto";

message Person {
    uint32 age = 1;
    string first_name = 2;
    string last_name = 3;
    bytes small_picture = 4;
    bool profile_verified = 5;
    float height = 6;
    repeated string address = 7;

    // enums of the possible eye color
    enum EyeColor {
        UKNOWN_EYE_COLOR = 0;
        EYE_GREEN = 1;
        EYE_BROWN = 2;
        EYE_BLUE = 3;
    }
    EyeColor eye_color = 8
    Date birthday = 9
}

Packages

WE can add packages to have percise type naming convetino

// mydate is the pacakge name
// Date is message definiton
message Address {
    string first_name = 1;
    my.date.Date bithday = 2;
}

Getting started with ProtoBuf

Basics

Protocol buffers uses .proto files, in which a message is the message we send or receive, a basic message look like this

message Product {
    string title = 1;
    float price = 2;
}

Proto files has 3 versions, to indicate the version in the beginning of the file, we declare the version we want to use, the lastest one is version 3.

syntax = "proto3"

message Product {
    string title = 1;
    float price = 2;
    bool available = 3;
}

The scalar types

double, float, int32, int64, uint32, uint64, sint32, sint64, fixed32, fixed64, sfixed64.

We find other common ones:

bool for boolean,

string for strings, which are UTF-8 or 7-bit ASCII text.

and bytes. or bytes array, we can use bytes for maybe a small image or something like that.