Welcome to the developer documentation for protocol buffers – a language-neutral, platform-neutral, extensible way of serializing structured data for use in communications protocols, data storage, and more.
This documentation is aimed at Java, C++, or Python developers who want to use protocol buffers in their applications. This overview introduces protocol buffers and tells you what you need to do to get started – you can then go on to follow the tutorials or delve deeper into protocol buffer encoding. API reference documentation is also provided for all three languages, as well as language and style guides for writing .proto files.
What are protocol buffers?
Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. You can even update your data structure without breaking deployed programs that are compiled against the "old" format.
How do they work?
You specify how you want the information you're serializing to be structured by defining protocol buffer message types in .proto files. Each protocol buffer message is a small logical record of information, containing a series of name-value pairs. Here's a very basic example of a .proto file that defines a message containing information about a person:
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
}
As you can see, the message format is simple – each message type has one or more uniquely numbered fields, and each field has a name and a value type, where value types can be numbers (integer or floating-point), booleans, strings, raw bytes, or even (as in the example above) other protocol buffer message types, allowing you to structure your data hierarchically. You can specify optional fields, required fields, and repeated fields. You can find more information about writing .proto files in the Protocol Buffer Language Guide.
Once you've defined your messages, you run the protocol buffer compiler for your application's language on your .proto file to generate data access classes. These provide simple accessors for each field (like name() and set_name()) as well as methods to serialize/parse the whole structure to/from raw bytes – so, for instance, if your chosen language is C++, running the compiler on the above example will generate a class called Person. You can then use this class in your application to populate, serialize, and retrieve Person protocol buffer messages. You might then write some code like this:
Person person;
person.set_name("John Doe");
person.set_id(1234);
person.set_email("jdoe@example.com");
fstream output("myfile", ios::out | ios::binary);
person.SerializeToOstream(&output);
Then, later on, you could read your message back in:
fstream input("myfile", ios::in | ios::binary);
Person person;
person.ParseFromIstream(&input);
cout << "Name: " << person.name() << endl;
cout << "E-mail: " << person.email() << endl;
You can add new fields to your message formats without breaking backwards-compatibility; old binaries simply ignore the new field when parsing. So if you have a communications protocol that uses protocol buffers as its data format, you can extend your protocol without having to worry about breaking existing code.
You'll find a complete reference for using generated protocol buffer code in the API Reference section, and you can find out more about how protocol buffer messages are encoded in Protocol Buffer Encoding.
Why not just use XML?
Protocol buffers have many advantages over XML for serializing structured data. Protocol buffers:
- are simpler
- are 3 to 10 times smaller
- are 20 to 100 times faster
- are less ambiguous
- generate data access classes that are easier to use programmatically
For example, let's say you want to model a person with a name and an email. In XML, you need to do:
<person>
<name>John Doe</name>
<email>jdoe@example.com</email>
</person>
while the corresponding protocol buffer message (in protocol buffer text format) is:
# Textual representation of a protocol buffer.
# This is *not* the binary format used on the wire.
person {
name: "John Doe"
email: "jdoe@example.com"
}
When this message is encoded to the protocol buffer binary format (the text format above is just a convenient human-readable representation for debugging and editing), it would probably be 28 bytes long and take around 100-200 nanoseconds to parse. The XML version is at least 69 bytes if you remove whitespace, and would take around 5,000-10,000 nanoseconds to parse.
Also, manipulating a protocol buffer is much easier:
cout << "Name: " << person.name() << endl;
cout << "E-mail: " << person.email() << endl;
Whereas with XML you would have to do something like:
cout << "Name: "
<< person.getElementsByTagName("name")->item(0)->innerText()
<< endl;
cout << "E-mail: "
<< person.getElementsByTagName("email")->item(0)->innerText()
<< endl;
However, protocol buffers are not always a better solution than XML – for instance, protocol buffers would not be a good way to model a text-based document with markup (e.g. HTML), since you cannot easily interleave structure with text. In addition, XML is human-readable and human-editable; protocol buffers, at least in their native format, are not. XML is also – to some extent – self-describing. A protocol buffer is only meaningful if you have the message definition (the .proto file).
編譯器
binary format <--- the text format
https://github.com/protocolbuffers/protobuf/releases/tag/v3.7.1
Protocol Buffers source code is hosted on GitHub: https://github.com/protocolbuffers/protobuf.
Our old Google Code repository is: https://code.google.com/p/protobuf/. We moved to GitHub on Aug 26, 2014 and no future changes will be made on the Google Code site. For latest code updates/issues, please visit our GitHub site.
Compiling Your Protocol Buffers
https://developers.google.com/protocol-buffers/docs/pythontutorial
protoc -I=$SRC_DIR --python_out=$DST_DIR $SRC_DIR/addressbook.proto
D:\pyPlusGlang\protoc-3.7.1-win64\bin>protoc -I=D:\pyPlusGlang\mysite\polls\ --python_out=D:\pyPlusG
lang\mysite\polls\ D:\pyPlusGlang\mysite\polls\addressbook.proto
D:\pyPlusGlang\mysite\polls\addressbook_pb2.py
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# source: addressbook.proto
import sys
_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
from google.protobuf import descriptor as _descriptor
from google.protobuf import message as _message
from google.protobuf import reflection as _reflection
from google.protobuf import symbol_database as _symbol_database
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
DESCRIPTOR = _descriptor.FileDescriptor(
name='addressbook.proto',
package='tutorial',
syntax='proto3',
serialized_options=_b('\n\024com.example.tutorialB\021AddressBookProtos\252\002$Google.Protobuf.Examples.AddressBook'),
serialized_pb=_b('\n\x11\x61\x64\x64ressbook.proto\x12\x08tutorial\x1a\x1fgoogle/protobuf/timestamp.proto\"\x87\x02\n\x06Person\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\n\n\x02id\x18\x02 \x01(\x05\x12\r\n\x05\x65mail\x18\x03 \x01(\t\x12,\n\x06phones\x18\x04 \x03(\x0b\x32\x1c.tutorial.Person.PhoneNumber\x12\x30\n\x0clast_updated\x18\x05 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x1aG\n\x0bPhoneNumber\x12\x0e\n\x06number\x18\x01 \x01(\t\x12(\n\x04type\x18\x02 \x01(\x0e\x32\x1a.tutorial.Person.PhoneType\"+\n\tPhoneType\x12\n\n\x06MOBILE\x10\x00\x12\x08\n\x04HOME\x10\x01\x12\x08\n\x04WORK\x10\x02\"/\n\x0b\x41\x64\x64ressBook\x12 \n\x06people\x18\x01 \x03(\x0b\x32\x10.tutorial.PersonBP\n\x14\x63om.example.tutorialB\x11\x41\x64\x64ressBookProtos\xaa\x02$Google.Protobuf.Examples.AddressBookb\x06proto3')
,
dependencies=[google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,])
_PERSON_PHONETYPE = _descriptor.EnumDescriptor(
name='PhoneType',
full_name='tutorial.Person.PhoneType',
filename=None,
file=DESCRIPTOR,
values=[
_descriptor.EnumValueDescriptor(
name='MOBILE', index=0, number=0,
serialized_options=None,
type=None),
_descriptor.EnumValueDescriptor(
name='HOME', index=1, number=1,
serialized_options=None,
type=None),
_descriptor.EnumValueDescriptor(
name='WORK', index=2, number=2,
serialized_options=None,
type=None),
],
containing_type=None,
serialized_options=None,
serialized_start=285,
serialized_end=328,
)
_sym_db.RegisterEnumDescriptor(_PERSON_PHONETYPE)
_PERSON_PHONENUMBER = _descriptor.Descriptor(
name='PhoneNumber',
full_name='tutorial.Person.PhoneNumber',
filename=None,
file=DESCRIPTOR,
containing_type=None,
fields=[
_descriptor.FieldDescriptor(
name='number', full_name='tutorial.Person.PhoneNumber.number', index=0,
number=1, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=_b("").decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR),
_descriptor.FieldDescriptor(
name='type', full_name='tutorial.Person.PhoneNumber.type', index=1,
number=2, type=14, cpp_type=8, label=1,
has_default_value=False, default_value=0,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR),
],
extensions=[
],
nested_types=[],
enum_types=[
],
serialized_options=None,
is_extendable=False,
syntax='proto3',
extension_ranges=[],
oneofs=[
],
serialized_start=212,
serialized_end=283,
)
_PERSON = _descriptor.Descriptor(
name='Person',
full_name='tutorial.Person',
filename=None,
file=DESCRIPTOR,
containing_type=None,
fields=[
_descriptor.FieldDescriptor(
name='name', full_name='tutorial.Person.name', index=0,
number=1, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=_b("").decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR),
_descriptor.FieldDescriptor(
name='id', full_name='tutorial.Person.id', index=1,
number=2, type=5, cpp_type=1, label=1,
has_default_value=False, default_value=0,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR),
_descriptor.FieldDescriptor(
name='email', full_name='tutorial.Person.email', index=2,
number=3, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=_b("").decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR),
_descriptor.FieldDescriptor(
name='phones', full_name='tutorial.Person.phones', index=3,
number=4, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR),
_descriptor.FieldDescriptor(
name='last_updated', full_name='tutorial.Person.last_updated', index=4,
number=5, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR),
],
extensions=[
],
nested_types=[_PERSON_PHONENUMBER, ],
enum_types=[
_PERSON_PHONETYPE,
],
serialized_options=None,
is_extendable=False,
syntax='proto3',
extension_ranges=[],
oneofs=[
],
serialized_start=65,
serialized_end=328,
)
_ADDRESSBOOK = _descriptor.Descriptor(
name='AddressBook',
full_name='tutorial.AddressBook',
filename=None,
file=DESCRIPTOR,
containing_type=None,
fields=[
_descriptor.FieldDescriptor(
name='people', full_name='tutorial.AddressBook.people', index=0,
number=1, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR),
],
extensions=[
],
nested_types=[],
enum_types=[
],
serialized_options=None,
is_extendable=False,
syntax='proto3',
extension_ranges=[],
oneofs=[
],
serialized_start=330,
serialized_end=377,
)
_PERSON_PHONENUMBER.fields_by_name['type'].enum_type = _PERSON_PHONETYPE
_PERSON_PHONENUMBER.containing_type = _PERSON
_PERSON.fields_by_name['phones'].message_type = _PERSON_PHONENUMBER
_PERSON.fields_by_name['last_updated'].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP
_PERSON_PHONETYPE.containing_type = _PERSON
_ADDRESSBOOK.fields_by_name['people'].message_type = _PERSON
DESCRIPTOR.message_types_by_name['Person'] = _PERSON
DESCRIPTOR.message_types_by_name['AddressBook'] = _ADDRESSBOOK
_sym_db.RegisterFileDescriptor(DESCRIPTOR)
Person = _reflection.GeneratedProtocolMessageType('Person', (_message.Message,), dict(
PhoneNumber = _reflection.GeneratedProtocolMessageType('PhoneNumber', (_message.Message,), dict(
DESCRIPTOR = _PERSON_PHONENUMBER,
__module__ = 'addressbook_pb2'
# @@protoc_insertion_point(class_scope:tutorial.Person.PhoneNumber)
))
,
DESCRIPTOR = _PERSON,
__module__ = 'addressbook_pb2'
# @@protoc_insertion_point(class_scope:tutorial.Person)
))
_sym_db.RegisterMessage(Person)
_sym_db.RegisterMessage(Person.PhoneNumber)
AddressBook = _reflection.GeneratedProtocolMessageType('AddressBook', (_message.Message,), dict(
DESCRIPTOR = _ADDRESSBOOK,
__module__ = 'addressbook_pb2'
# @@protoc_insertion_point(class_scope:tutorial.AddressBook)
))
_sym_db.RegisterMessage(AddressBook)
DESCRIPTOR._options = None
# @@protoc_insertion_point(module_scope)
https://developers.google.com/protocol-buffers/docs/pythontutorial
protobuf2 protobuf3 差別
addressbook.proto: Explicit default values are not allowed in proto3.
addressbook.proto: Required fields are not allowed in proto3.
addressbook.proto:8:12: Explicit 'optional' labels are disallowed in the Proto3 syntax. To define 'o
ptional' fields in Proto3, simply remove the 'optional' label, as fields are 'optional' by default.
Language Guide | Protocol Buffers | Google Developers
https://developers.google.com/protocol-buffers/docs/proto
讀寫消息 protobuf二進制文件
D:\pyPlusGlang\LProtocol-buffers>python pb2ReadingAMessage.py myADDRESS_BOOK_FILE Person ID: 1 Name: a001 Can't test non-submessage field "Person.email" for presence in proto3. Mobile phone #: 152001 D:\pyPlusGlang\LProtocol-buffers>python pb2WritingAMessage.py myADDRESS_BOOK_FILE ['pb2WritingAMessage.py', 'myADDRESS_BOOK_FILE'] Enter person ID number: 002 Enter name: a002 Enter email address (blank for none): b@qq.cn Enter a phone number (or leave blank to finish): 142002 Is this a mobile, home, or work phone? home Enter a phone number (or leave blank to finish): D:\pyPlusGlang\LProtocol-buffers>python pb2ReadingAMessage.py myADDRESS_BOOK_FILE Person ID: 1 Name: a001 Can't test non-submessage field "Person.email" for presence in proto3. Mobile phone #: 152001 Person ID: 2 Name: a002 Can't test non-submessage field "Person.email" for presence in proto3. Home phone #: 142002 D:\pyPlusGlang\LProtocol-buffers>
address_book = addressbook_pb2.AddressBook()
寫protobuf文件
# Add an address.
PromptForAddress(address_book.people.add())
# Write the new address book back to disk.
f = open(fname, "wb")
f.write(address_book.SerializeToString())
讀protobuf文件
# Read the existing address book.
f = open(fname, "rb")
address_book.ParseFromString(f.read())
一階
python生成的protobuf文件被golang讀取 反序列化
生成golang
golang/protobuf: Go support for Google's protocol buffers
https://github.com/golang/protobuf
- Grab the code from the repository and install the
protopackage. The simplest way is to rungo get -u github.com/golang/protobuf/protoc-gen-go. The compiler plugin,protoc-gen-go, will be installed in$GOPATH/binunless$GOBINis set. It must be in your$PATHfor the protocol compiler,protoc, to find it.
protoc -I=$SRC_DIR --go_out=$DST_DIR $SRC_DIR/addressbook.proto
protoc -I=D:\pyPlusGlang\LProtocol-buffers --go_out=D:\pyPlusGlang\LProtocol-buffers D:\pyPlusGlang\LProtocol-buffers\SearchRequest.proto
protoc.exe 須要protoc-gen-go.exe，會從當前目錄開始查找
python
成功讀寫文件
D:\pyPlusGlang\LProtocol-buffers>python pb3ReadingAMessage.py sRqs_pb3 query: appleVSbanana page_number: 1 result_per_page: 4 D:\pyPlusGlang\LProtocol-buffers>python pb3WritingAMessage.py sRqs_pb3 ['pb3WritingAMessage.py', 'sRqs_pb3'] Enter query: aANDb Enter page_number: 123 Enter result_per_page: 789 D:\pyPlusGlang\LProtocol-buffers>python pb3ReadingAMessage.py sRqs_pb3 query: appleVSbanana page_number: 1 result_per_page: 4 query: aANDb page_number: 123 result_per_page: 789 D:\pyPlusGlang\LProtocol-buffers>
syntax = "proto3";
message SearchRequest {
string query = 1;
int32 page_number = 2;
int32 result_per_page = 3;
}
message SearchRequestList {
repeated SearchRequest sR = 1;
}
D:\pyPlusGlang\LProtocol-buffers\SearchRequestList_pb2.py
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# source: SearchRequestList.proto
import sys
_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
from google.protobuf import descriptor as _descriptor
from google.protobuf import message as _message
from google.protobuf import reflection as _reflection
from google.protobuf import symbol_database as _symbol_database
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor.FileDescriptor(
name='SearchRequestList.proto',
package='',
syntax='proto3',
serialized_options=None,
serialized_pb=_b('\n\x17SearchRequestList.proto\"L\n\rSearchRequest\x12\r\n\x05query\x18\x01 \x01(\t\x12\x13\n\x0bpage_number\x18\x02 \x01(\x05\x12\x17\n\x0fresult_per_page\x18\x03 \x01(\x05\"/\n\x11SearchRequestList\x12\x1a\n\x02sR\x18\x01 \x03(\x0b\x32\x0e.SearchRequestb\x06proto3')
)
_SEARCHREQUEST = _descriptor.Descriptor(
name='SearchRequest',
full_name='SearchRequest',
filename=None,
file=DESCRIPTOR,
containing_type=None,
fields=[
_descriptor.FieldDescriptor(
name='query', full_name='SearchRequest.query', index=0,
number=1, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=_b("").decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR),
_descriptor.FieldDescriptor(
name='page_number', full_name='SearchRequest.page_number', index=1,
number=2, type=5, cpp_type=1, label=1,
has_default_value=False, default_value=0,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR),
_descriptor.FieldDescriptor(
name='result_per_page', full_name='SearchRequest.result_per_page', index=2,
number=3, type=5, cpp_type=1, label=1,
has_default_value=False, default_value=0,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR),
],
extensions=[
],
nested_types=[],
enum_types=[
],
serialized_options=None,
is_extendable=False,
syntax='proto3',
extension_ranges=[],
oneofs=[
],
serialized_start=27,
serialized_end=103,
)
_SEARCHREQUESTLIST = _descriptor.Descriptor(
name='SearchRequestList',
full_name='SearchRequestList',
filename=None,
file=DESCRIPTOR,
containing_type=None,
fields=[
_descriptor.FieldDescriptor(
name='sR', full_name='SearchRequestList.sR', index=0,
number=1, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR),
],
extensions=[
],
nested_types=[],
enum_types=[
],
serialized_options=None,
is_extendable=False,
syntax='proto3',
extension_ranges=[],
oneofs=[
],
serialized_start=105,
serialized_end=152,
)
_SEARCHREQUESTLIST.fields_by_name['sR'].message_type = _SEARCHREQUEST
DESCRIPTOR.message_types_by_name['SearchRequest'] = _SEARCHREQUEST
DESCRIPTOR.message_types_by_name['SearchRequestList'] = _SEARCHREQUESTLIST
_sym_db.RegisterFileDescriptor(DESCRIPTOR)
SearchRequest = _reflection.GeneratedProtocolMessageType('SearchRequest', (_message.Message,), dict(
DESCRIPTOR = _SEARCHREQUEST,
__module__ = 'SearchRequestList_pb2'
# @@protoc_insertion_point(class_scope:SearchRequest)
))
_sym_db.RegisterMessage(SearchRequest)
SearchRequestList = _reflection.GeneratedProtocolMessageType('SearchRequestList', (_message.Message,), dict(
DESCRIPTOR = _SEARCHREQUESTLIST,
__module__ = 'SearchRequestList_pb2'
# @@protoc_insertion_point(class_scope:SearchRequestList)
))
_sym_db.RegisterMessage(SearchRequestList)
# @@protoc_insertion_point(module_scope)
D:\pyPlusGlang\LProtocol-buffers\pb3WritingAMessage.py
#! /usr/bin/python
import SearchRequestList_pb2
import sys
def PromptForSearchRequest(searchRequest):
searchRequest.query = input("Enter query: ")
searchRequest.page_number = int(input("Enter page_number: "))
searchRequest.result_per_page = int(input("Enter result_per_page: "))
print(sys.argv)
if len(sys.argv) != 2:
print("here is :", __file__, sys._getframe().f_lineno) # @print@ "Usage:", sys.argv[0], "SearchRequest__FILE"
sys.exit(-1)
searchRequestList = SearchRequestList_pb2.SearchRequestList()
# Read the existing SearchRequest book.
try:
f = open(sys.argv[1], "rb")
searchRequestList.ParseFromString(f.read())
f.close()
except IOError:
print("here is :", __file__,
sys._getframe().f_lineno) # @print@ sys.argv[1] + ": Could not open file. Creating a new one."
# Add an SearchRequest.
PromptForSearchRequest(searchRequestList.sR.add())
# Write the new SearchRequest book back to disk.
f = open(sys.argv[1], "wb")
f.write(searchRequestList.SerializeToString())
f.close()
D:\pyPlusGlang\LProtocol-buffers\pb3ReadingAMessage.py
#! /usr/bin/python
import SearchRequestList_pb2
import sys
# Iterates though all people in the AddressBook and prints info about them.
def ListSr(searchRequestList):
for sR in searchRequestList.sR:
print("query:", sR.query)
print("page_number:", sR.page_number)
print("result_per_page:", sR.result_per_page)
# Main procedure: Reads the entire address book from a file and prints all
# the information inside.
if len(sys.argv) != 2:
print("Usage:", sys.argv[0], "searchRequestList")
sys.exit(-1)
searchRequestList = SearchRequestList_pb2.SearchRequestList()
# Read the existing address book.
f = open(sys.argv[1], "rb")
searchRequestList.ParseFromString(f.read())
f.close()
ListSr(searchRequestList)