Welcome back on how to create stable and efficient service definitions using Protobuf. In the first part of this series, we provided an overview of Protobuf and its benefits for defining gRPC services. Today, we will dive deeper into this powerful tool and guide you through the process of using Protobuf to create robust and efficient service definitions. So, if you're ready to supercharge your APIs, let's get started with Part 2 of our guide to using Protobuf for defining gRPC services.
Creating the First Service Description with Protobuf
Creating a service description with Protobuf is the kick-off to bring a gRPC service to life. Here, we'll take the retail domain model as a starting point.
Assuming we start from OpenAPI, we establish an initial description for the Product Service.
1syntax = "proto3";
2
3message Product {
4 string name = 2;
5 string description = 3;
6 float price = 4;
7 string currency = 5;
8 string image = 6;
9 int32 stock = 7;
10}
11
12message ProductId {
13 string id = 1;
14}
15
16message Stock {
17 string id = 1;
18 int32 stock = 2;
19}
20
21service ProductService {
22 rpc AddProduct(Product) returns (ProductId) {}
23 rpc GetProductById(ProductId) returns (Product) {}
24 rpc UpdateProductStock(Stock) returns (ProductId) {}
25}
Now the description is checked with buf lint to review style, syntax, and best practices. The linter identifies some style issues and emphasises adherence to best practices.
1retail/v1/retail_based_openapi.proto:24:5:"retail.v1.ProductId" is used as the request or response type for multiple RPCs.
2retail/v1/retail_based_openapi.proto:24:20:RPC request type "Product" should be named "AddProductRequest" or "ProductServiceAddProductRequest".
3retail/v1/retail_based_openapi.proto:24:38:RPC response type "ProductId" should be named "AddProductResponse" or "ProductServiceAddProductResponse".
4retail/v1/retail_based_openapi.proto:25:5:"retail.v1.ProductId" is used as the request or response type for multiple RPCs.
5retail/v1/retail_based_openapi.proto:25:5:RPC "GetProductById" has the same type "retail.v1.ProductId" for the request and response.
6retail/v1/retail_based_openapi.proto:25:24:RPC request type "ProductId" should be named "GetProductByIdRequest" or "ProductServiceGetProductByIdRequest".
7retail/v1/retail_based_openapi.proto:25:44:RPC response type "ProductId" should be named "GetProductByIdResponse" or "ProductServiceGetProductByIdResponse".
8retail/v1/retail_based_openapi.proto:26:5:"retail.v1.ProductId" is used as the request or response type for multiple RPCs.
9retail/v1/retail_based_openapi.proto:26:28:RPC request type "Stock" should be named "UpdateProductStockRequest" or "ProductServiceUpdateProductStockRequest".
10retail/v1/retail_based_openapi.proto:26:44:RPC response type "ProductId" should be named "UpdateProductStockResponse" or "ProductServiceUpdateProductStockResponse".
We take a closer look at these best practices to better understand how gRPC services are fundamentally designed. The aim is to learn from the style guide and tips to develop a meaningful design for Protobuf descriptions in the organisation. Because what is generally regarded as best practice may not always fit every organisation.
What makes a good Protobuf description, and thus a good gRPC service?
Here are some tips and tricks to kickstart your journey with Protobuf definitions.
1. Naming matters:
- Name your messages, services, and fields in a way that makes sense.
- Use Camel Case for message and service names, and snake_case for field names.
2. Avoid breaking things:
- Don’t mess with the tag numbers of fields. They are crucial for serialization.
- Don’t delete fields. If you don’t need one anymore, just mark it as deprecated.
- New fields? Simply use the next available number.
3. Choose the right data types:
- Pick the data type that fits best. For instance, use int32 instead of int64 if the value remains small.
- enum is great for fields with fixed values.
4. Don’t overdo optional and required labels:
- In proto3, all fields are optional. In proto2, the required label can be tricky at times.
5. Build smartly:
- Prefer creating multiple small messages over one large message. It’s clearer and more flexible.
6. Avoid nested messages:
- Sure, Protobuf allows it, but it can quickly get messy.
7. Be cautious with map types:
- map is cool, but has its quirks. For example, don’t use repeated fields as key or value.
8. Comments are invaluable:
- Document what you are doing. It helps you and others immensely later on.
9. Structure is key:
- Organize your Protobuf definitions into packages. It keeps things neat and tidy.
10. Don’t forget default values:
- Protobuf has default values, like 0 for numbers. If a field is missing, the default value is used. So, pay attention!
As can be seen, practice and experience are needed to create really good Protobuf definitions. But if you stick to proven practices, your definitions will be stable, future-proof, and easily understandable by other developers.
Protobuf Description, Second Iteration
After the initial linting, we revised the description to eliminate any hints or errors. This way, it successfully passed the first syntax validation. We can see the updated description below.
1syntax = "proto3";
2
3package retail.v1;
4
5message AddProductRequest {
6 string name = 2;
7 string description = 3;
8 float price = 4;
9 string currency = 5;
10 string image = 6;
11 int32 stock = 7;
12}
13
14message AddProductResponse {
15 string id = 1;
16}
17
18message GetProductByIdRequest {
19 string id = 1;
20}
21
22message GetProductByIdResponse {
23 string name = 2;
24 string description = 3;
25 float price = 4;
26 string currency = 5;
27 string image = 6;
28 int32 stock = 7;
29}
30
31message UpdateProductStockRequest {
32 string id = 1;
33 int32 stock = 2;
34}
35
36message UpdateProductStockResponse {
37 string id = 1;
38}
39
40service ProductService {
41 rpc AddProduct(AddProductRequest) returns (AddProductResponse) {}
42 rpc GetProductById(GetProductByIdRequest) returns (GetProductByIdResponse) {}
43 rpc UpdateProductStock(UpdateProductStockRequest) returns (UpdateProductStockResponse) {}
44}
Building upon the service for the product, we also created services for orders (Order) and customers (Customer), along with their respective messages (Messages).
1syntax = "proto3";
2
3package retail.v1;
4
5message OrderItem {
6 int32 product_id = 1;
7 int32 quantity = 2;
8 }
9
10message AddOrderRequest {
11 int32 customer_id = 1;
12 repeated OrderItem items = 2;
13}
14
15message AddOrderResponse {
16 int32 id = 1;
17}
18
19message GetOrderByIdRequest {
20 int32 id = 1;
21}
22
23message GetOrderByIdResponse {
24 int32 id = 1;
25 int32 customer_id = 2;
26 repeated OrderItem items = 3;
27}
28
29service OrderService {
30 rpc AddOrder(AddOrderRequest) returns (AddOrderResponse) {}
31 rpc GetOrderById(GetOrderByIdRequest) returns (GetOrderByIdResponse) {}
32}
1syntax = "proto3";
2
3package retail.v1;
4
5message RegisterCustomerRequest {
6 string name = 1;
7 string email = 2;
8 string address = 3;
9 }
10
11 message RegisterCustomerResponse {
12 int32 id = 1;
13 }
14
15 message GetCustomerByIdRequest {
16 int32 id = 1;
17 }
18
19 message GetCustomerByIdResponse {
20 int32 id = 1;
21 string name = 2;
22 string email = 3;
23 string address = 4;
24 }
25
26 service CustomerService {
27 rpc RegisterCustomer(RegisterCustomerRequest) returns (RegisterCustomerResponse) {}
28 rpc GetCustomerById(GetCustomerByIdRequest) returns (GetCustomerByIdResponse) {}
29}
This leads to the folder structure.
1.
2├── buf.yaml
3└── retail
4 ├── buf.gen.yaml
5 └── v1
6 ├── customer.proto
7 ├── order.proto
8 └── product.proto
9
103 directories, 5 files
From Local to Remote – Working with the Registry
With the Buf CLI, you can deploy the package to the Buf Schema Registry (BSR). After creating a repository and logging in through the console with buf registry login buf.build --username "<user-name>", you're basically ready to make your first push to the repository. But wait! Before you do that, you need to make a small adjustment in the configuration file buf.yaml (see Listing 8). Just add the key name along with the URI of your repo, and you're good to go!
1version: v1
2name: buf.build/cc-danielkocot/playground
3breaking:
4 use:
5 - FILE
6lint:
7 use:
8 - DEFAULT
Finally, a buf push will move the module and package to the registry.
Now, we can generate the gRPC services using remote packages, as the information about the packages can be directly fetched from the registry. This often simplifies teamwork significantly.
Generating gRPC Services
With the Buf CLI, you can generate the necessary source files for gRPC services based on the Protobuf definition. First, you need to create a configuration file named buf.gen.yaml in the package folder.
1version: v1
2managed:
3 enabled: true
4 java_multiple_files: false
5 java_package_prefix:
6 default: de.codecentric
7plugins:
8 - plugin: buf.build/grpc/java
9 out: gen/java
10 # dependencies
11 - plugin: buf.build/protocolbuffers/java
12 out: gen
To generate the code, we can use both local and remote plugins. In our example, we use the latter, ensuring that the team always uses the correct version for generation. With the Buf CLI and a configuration file, generating code is quite straightforward for developers.
Not only with the CLI, but also with tools like Maven and Gradle, you can generate code. Stay tuned for more information on this and other aspects of Protobuf and gRPC in the upcoming posts on the codecentric blog.
Conclusion
In summary, Protobuf is a powerful and efficient serialization format that has proven to be a valuable tool for developers since its introduction. It stands out from many other serialization formats due to its speed, efficiency, and flexibility. Since its release, Protobuf has been used in a wide range of projects and systems, especially excelling in high-performance applications and systems that require fast and efficient communication between services. Creating a gRPC service definition with Protobuf involves several steps, including defining data structures and service interfaces. With tools like the Buf CLI and the Buf Schema Registry (BSR), the development environment for working with Protobuf has been further optimized. Properly designing Protobuf definitions requires practice and experience, but by following best practices, you can create stable, forward-compatible, and easily understandable definitions. Generating gRPC services based on Protobuf definitions can be easily done with the Buf CLI and other tools like Maven and Gradle.
References
Was this post helpful?
More articles
fromDaniel Kocot
13.11.2024 Your job at codecentric?
Jobs
Agile Developer und Consultant (w/d/m)
Alle Standorte
More articles in this subject area
Discover exciting further topics and let the codecentric world inspire you.
Gemeinsam bessere Projekte umsetzen.
Wir helfen deinem Unternehmen.
Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.
Hilf uns, noch besser zu werden.
Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.
Blog author
Daniel Kocot
Senior Solution Architect / Head of API Consulting
Do you still have questions? Just send me a message.
Do you still have questions? Just send me a message.