什麼是 Google Protocol Buffer?java
Google Protocol Buffer( 簡稱 Protobuf) 是 Google 公司內部的混合語言數據標準,目前已經正在使用的有超過 48,162 種報文格式定義和超過 12,183 個 .proto 文件。他們用於 RPC 系統和持續數據存儲系統。git
Protocol Buffers 是一種輕便高效的結構化數據存儲格式,能夠用於結構化數據串行化,或者說序列化。它很適合作數據存儲或 RPC 數據交換格式。可用於通信協議、數據存儲等領域的語言無關、平臺無關、可擴展的序列化結構數據格式。目前提供了 多種語言的API,包括C++、 C# 、GO、 JAVA、 PYTHONgithub
若是你並不瞭解Protobuf能作什麼,建議結合google搜索關鍵字,看一下入門級別的文章,或者看一下官方文檔中的Developer Guide,或者中文的開發指南 .官方的文檔中有各類語言相關的示例,能夠結合代碼看一下實際的用法。shell
不少人說爲何不用json(或者xml), 答案很簡單,Protobuf更小,更簡潔,並且序列化和反序列化更快!json
谷歌最新開源的gRpc框架就是默認使用Protobuf做爲數據傳輸格式和服務描述文件。對於gRpc 就不作詳細介紹了,有興趣的能夠看一下官網。c#
言歸正傳,在實際使用Protobuf過程當中,我發現Protobuf不但能夠編寫描述消息(Message)的內容,同時能夠表述其餘方法(相似Rpc中的方法),主要是gRpc中看到的。同時在Protobuf 代碼生成工具的包中,有一個這樣的目錄,一致以來都沒搞明白是作什麼用的,以下圖:markdown
在目錄中存在大量已經定義好的proto文件,其實這些文件是Protobuf的描述文件,相似元數據。用自己的語法描述自己,同時經過這些文件生成對應的語言的元數據類等代碼,好比在C#版本的Google.Protobuf中就能看到上述描述文件生成的類,以下圖所示框架
而這些描述文件中最重要的文件 就是descriptor.proto
這個文件,這個文件是整個proto語法的描述類,描述了實際Protobuf各層次語法的結構,來一塊兒看一下這個文件的一些代碼, 上面這個代碼描述了proto文件定義的語法定義,如前面兩個字段意思是可選的name,可選的package字段,中間是描述可多個message_type(Message),service(Rpc Service) ,enum_type(枚舉)等定義,而後一層層分解下去。 基本上就能夠了解Protobuf語法的全貌和擴展點了異步
message FileDescriptorProto { optional string name = 1; // file name, relative to root of source tree optional string package = 2; // e.g. "foo", "foo.bar", etc. // Names of files imported by this file. repeated string dependency = 3; // Indexes of the public imported files in the dependency list above. repeated int32 public_dependency = 10; // Indexes of the weak imported files in the dependency list. // For Google-internal migration only. Do not use. repeated int32 weak_dependency = 11; // All top-level definitions in this file. repeated DescriptorProto message_type = 4; repeated EnumDescriptorProto enum_type = 5; repeated ServiceDescriptorProto service = 6; repeated FieldDescriptorProto extension = 7; optional FileOptions options = 8; // This field contains optional information about the original source code. // You may safely remove this entire field without harming runtime // functionality of the descriptors -- the information is needed only by // development tools. optional SourceCodeInfo source_code_info = 9; // The syntax of the proto file. // The supported values are "proto2" and "proto3". optional string syntax = 12; }
同時在compiler目錄下 還有一個plugin的目錄,其中的plugin.proto文件很回味無窮,先來看下這個文件中的內容async
syntax = "proto3"; package google.protobuf.compiler; option java_package = "com.google.protobuf.compiler"; option java_outer_classname = "PluginProtos"; option csharp_namespace = "Google.Protobuf.Compiler"; option go_package = "plugin_go"; import "google/protobuf/descriptor.proto"; message CodeGeneratorRequest { repeated string file_to_generate = 1; string parameter = 2; repeated FileDescriptorProto proto_file = 15; } message CodeGeneratorResponse { string error = 1; message File { string name = 1; string insertion_point = 2; string content = 15; } repeated File file = 15; }
刪除了非必要的註釋後,咱們能夠看到這個文件裏面其實只定義了兩個類型,一個是代碼生成請求,一個是代碼生成響應,而在CodeGeneratorRequest
中又有以前咱們在descriptor.proto
中看到的FileDescriptorProto
這個類的信息,用大腿均可以想到這裏應該就是代碼生成插件獲取元數據的入口了,那麼怎麼作呢?
從gRpc 的代碼生成示例中 咱們能夠看到 其實Protobuf是支持自定義生成代碼插件的,以下所示:
%PROTOC% -I../../protos --csharp_out Greeter ../../protos/helloworld.proto --grpc_out Greeter --plugin=protoc-gen-grpc=%PLUGIN%
按理咱們能夠實現本身的插件來生成咱們須要的任意格式,包括各類代碼,甚至是文檔。可是這個資料卻很是少,幾乎沒有多少相關的文章,後來終於找到一片關於plugin的文章http://www.expobrain.net/2015/09/13/create-a-plugin-for-google-protocol-buffer/ ,你們有興趣的能夠看看,不過文章的重點是這句:
The core part is the interface code to read a request from the
stdin
, traverse the AST and write the response on thestdout
.
原來插件的接口代碼實際上是從標準輸入中讀取流,而後再把你要生成的內容輸出到標準輸出中。這些終於知道怎麼用了。。
撩起袖子開始幹,經過protoc命令行生成plugin.proto的代碼
protoc-I../../protos --csharp_out test ../../protos/plugin.proto
新建一個控制檯項目,把代碼copy 到項目中,並在Program.cs代碼中添加測試的代碼
using Google.Protobuf; using Google.Protobuf.Compiler; using System; namespace DotBPE.ProtobufPlugin { class Program { static void Main(string[] args) { Console.OutputEncoding = System.Text.Encoding.UTF8; var response = new CodeGeneratorResponse(); try { CodeGeneratorRequest request; using (var inStream = Console.OpenStandardInput()) { request = CodeGeneratorRequest.Parser.ParseFrom(inStream); } ParseCode(request, response); } catch (Exception e) { response.Error += e.ToString(); } using (var output = Console.OpenStandardOutput()) { response.WriteTo(output); output.Flush(); } } private static void ParseCode(CodeGeneratorRequest request, CodeGeneratorResponse response) { DotbpeGen.Generate(request,response); } } }
哈哈 開始編譯,然而編譯不經過!,坑爹啊! 原來C#版本中 Google.Protobuf已經生成好的類 都是internal訪問權限,不能從外部引用。。。可是Google.Protobuf是開源的。。並且我須要用的類 我也能夠經過protoc命令本身生成到同一個項目中,或者設置成public訪問權限。。方便起見,我直接copy了Google.Protobuf的源碼到咱們的項目中,此次再次編譯 ,代碼就完美運行了,接下來的工做 不過是填充DotbpeGen.Generate
的代碼了,這不過是體力活。
至於CodeGeneratorRequest和CodeGeneratorResponse 到底有什麼方法,其實看proto文件就能知道。如下是我本身在項目中使用的生成代碼類 供你們參考
using Google.Protobuf.Compiler; using Google.Protobuf.Reflection; using System; using System.Collections.Generic; using System.IO; using System.Linq; using System.Text; namespace DotBPE.ProtobufPlugin { public class DotbpeGen { public static void Generate(CodeGeneratorRequest request, CodeGeneratorResponse response) { foreach (var protofile in request.ProtoFile) { try{ GenerateByProtoFile(protofile, response); } catch(Exception ex){ using (Stream stream = File.Create("./error.txt")) { byte[] err = Encoding.UTF8.GetBytes(ex.Message+ex.StackTrace); stream.Write(err,0,err.Length); } response.Error += ex.Message; } } } private static void GenerateSourceInfo(FileDescriptorProto protofile, CodeGeneratorResponse response) { bool genericDoc; protofile.Options.CustomOptions.TryGetBool(DotBPEOptions.GENERIC_MARKDOWN_DOC,out genericDoc); if (!genericDoc) { return; } StringBuilder sb = new StringBuilder(); foreach (var location in protofile.SourceCodeInfo.Location) { string path = String.Join(",", location.Path); string span = String.Join(",", location.Span); string leadingDetachedComments = String.Join("\r", location.LeadingDetachedComments); string trailingComments = String.Join("\r", location.TrailingComments); sb.AppendLine("{\"Path\",\""+path+"\",\"Span\",\""+span+"\",\"LeadingComments\",\""+ location.LeadingComments + "\",\"LeadingDetachedComments\",\""+ leadingDetachedComments + "\",\"TrailingComments\",\""+ trailingComments + "\"}"); } var nfile = new CodeGeneratorResponse.Types.File { Name = GetFileName(protofile.Name) + "SI.txt", Content = sb.ToString() }; response.File.Add(nfile); } private static void GenerateByProtoFile(FileDescriptorProto protofile, CodeGeneratorResponse response) { GenerateSourceInfo(protofile, response); GenerateServer(protofile, response); GenerateClient(protofile, response); } private static void GenerateServer(FileDescriptorProto protofile, CodeGeneratorResponse response) { bool genericServer; protofile.Options.CustomOptions.TryGetBool(DotBPEOptions.DISABLE_GENERIC_SERVICES_SERVER, out genericServer); if (genericServer) { return; } if (protofile.Service == null || protofile.Service.Count <= 0) return; //生成文件頭 StringBuilder sb = new StringBuilder(); sb.AppendLine("// Generated by the protocol buffer compiler. DO NOT EDIT!"); sb.AppendLine($"// source: {protofile.Name}"); //還能夠生成註釋 sb.AppendLine("#region Designer generated code"); sb.AppendLine(""); sb.AppendLine("using System; "); sb.AppendLine("using System.Threading.Tasks; "); sb.AppendLine("using DotBPE.Rpc; "); sb.AppendLine("using DotBPE.Protocol.Amp; "); sb.AppendLine("using Google.Protobuf; "); sb.AppendLine(""); string ns = GetFileNamespace(protofile); sb.AppendLine("namespace " + ns + " {"); //生成代碼 foreach (ServiceDescriptorProto t in protofile.Service) { t.Options.CustomOptions.TryGetBool(DotBPEOptions.DISABLE_GENERIC_SERVICES_SERVER, out genericServer); if (genericServer) { continue; } sb.AppendLine(""); sb.AppendLine("//start for class Abstract"+t.Name); GenerateServiceForServer(t, sb); sb.AppendLine("//end for class Abstract"+t.Name); } sb.AppendLine("}\n"); sb.AppendLine("#endregion\n"); var nfile = new CodeGeneratorResponse.Types.File { Name = GetFileName(protofile.Name) + "Server.cs", Content = sb.ToString() }; response.File.Add(nfile); } private static void GenerateClient(FileDescriptorProto protofile, CodeGeneratorResponse response) { bool genericClient; protofile.Options.CustomOptions.TryGetBool(DotBPEOptions.DISABLE_GENERIC_SERVICES_CLIENT, out genericClient); if (genericClient) { return; } if (protofile.Service == null || protofile.Service.Count <= 0) return; //生成文件頭 StringBuilder sb = new StringBuilder(); sb.AppendLine("// Generated by the protocol buffer compiler. DO NOT EDIT!"); sb.AppendLine($"// source: {protofile.Name}"); //還能夠生成註釋 sb.AppendLine("#region Designer generated code"); sb.AppendLine(""); sb.AppendLine("using System; "); sb.AppendLine("using System.Threading.Tasks; "); sb.AppendLine("using DotBPE.Rpc; "); sb.AppendLine("using DotBPE.Protocol.Amp; "); sb.AppendLine("using DotBPE.Rpc.Exceptions; "); sb.AppendLine("using Google.Protobuf; "); sb.AppendLine(""); string ns = GetFileNamespace(protofile); sb.AppendLine("namespace " + ns + " {"); //生成代碼 foreach (ServiceDescriptorProto t in protofile.Service) { t.Options.CustomOptions.TryGetBool(DotBPEOptions.DISABLE_GENERIC_SERVICES_CLIENT, out genericClient); if (genericClient) { continue; } sb.AppendLine(""); sb.AppendLine("//start for class "+t.Name+"Client"); GenerateServiceForClient(t, sb); sb.AppendLine("//end for class "+t.Name+"Client"); } sb.AppendLine("}"); sb.AppendLine("#endregion"); //生成文件 var nfile = new CodeGeneratorResponse.Types.File { Name = GetFileName(protofile.Name) + "Client.cs", Content = sb.ToString() }; response.File.Add(nfile); } private static void GenerateServiceForClient(ServiceDescriptorProto service, StringBuilder sb) { int serviceId; bool hasServiceId = service.Options.CustomOptions.TryGetInt32(DotBPEOptions.SERVICE_ID, out serviceId); if (!hasServiceId || serviceId <= 0) { throw new Exception("Service=" + service.Name + " ServiceId NOT_FOUND"); } if (serviceId >= ushort.MaxValue) { throw new Exception("Service=" + service.Name + "ServiceId too large"); } sb.AppendFormat("public sealed class {0}Client : AmpInvokeClient \n",service.Name); sb.AppendLine("{"); //構造函數 sb.AppendLine($"public {service.Name}Client(IRpcClient<AmpMessage> client) : base(client)"); sb.AppendLine("{"); sb.AppendLine("}"); //循環方法 foreach (var method in service.Method) { int msgId ; bool hasMsgId= method.Options.CustomOptions.TryGetInt32(DotBPEOptions.MESSAGE_ID,out msgId); if (!hasMsgId || msgId <= 0) { throw new Exception("Service" + service.Name + "." + method.Name + " ' MessageId NOT_FINDOUT "); } if (msgId >= ushort.MaxValue) { throw new Exception("Service" + service.Name + "." + method.Name + " is too large"); } //異步方法 string outType = GetTypeName(method.OutputType); string inType = GetTypeName(method.InputType); sb.AppendLine($"public async Task<{outType}> {method.Name}Asnyc({inType} request,int timeOut=3000)"); sb.AppendLine("{"); sb.AppendLine($"AmpMessage message = AmpMessage.CreateRequestMessage({serviceId}, {msgId});"); sb.AppendLine("message.Data = request.ToByteArray();"); sb.AppendLine("var response = await base.CallInvoker.AsyncCall(message,timeOut);"); sb.AppendLine("if (response != null && response.Data !=null)"); sb.AppendLine("{"); sb.AppendLine($"return {outType}.Parser.ParseFrom(response.Data);"); sb.AppendLine("}"); sb.AppendLine("throw new RpcException(\"請求出錯,請檢查!\");"); sb.AppendLine("}"); sb.AppendLine(); sb.AppendLine("//同步方法"); sb.AppendLine($"public {outType} {method.Name}({inType} request)"); sb.AppendLine("{"); sb.AppendLine($"AmpMessage message = AmpMessage.CreateRequestMessage({serviceId}, {msgId});"); sb.AppendLine("message.Data = request.ToByteArray();"); sb.AppendLine("var response = base.CallInvoker.BlockingCall(message);"); sb.AppendLine("if (response != null && response.Data !=null)"); sb.AppendLine("{"); sb.AppendLine($"return {outType}.Parser.ParseFrom(response.Data);"); sb.AppendLine("}"); sb.AppendLine("throw new RpcException(\"請求出錯,請檢查!\");"); sb.AppendLine("}"); } //循環方法end sb.AppendLine("}"); //類結束 } private static void GenerateServiceForServer(ServiceDescriptorProto service, StringBuilder sb) { int serviceId; bool hasServiceId = service.Options.CustomOptions.TryGetInt32(DotBPEOptions.SERVICE_ID, out serviceId); if(!hasServiceId || serviceId<=0){ throw new Exception("Service="+service.Name+" ServiceId NOT_FOUND"); } if(serviceId>=ushort.MaxValue){ throw new Exception("Service="+service.Name+ "ServiceId too large" ); } sb.AppendFormat("public abstract class {0}Base : IServiceActor<AmpMessage> \n", service.Name); sb.AppendLine("{"); sb.AppendLine("public string Id => \""+serviceId+"$0\";"); StringBuilder sbIfState = new StringBuilder(); //循環方法 foreach (var method in service.Method) { int msgId ; bool hasMsgId= method.Options.CustomOptions.TryGetInt32(DotBPEOptions.MESSAGE_ID,out msgId); if(!hasMsgId || msgId<=0){ throw new Exception("Service"+service.Name+"."+method.Name+" ' MessageId NOT_FINDOUT "); } if(msgId>=ushort.MaxValue){ throw new Exception("Service" + service.Name+"."+method.Name+" is too large"); } //異步方法 string outType = GetTypeName(method.OutputType); string inType = GetTypeName(method.InputType); sb.AppendLine("//調用委託"); sb.AppendLine( $"private async Task Receive{method.Name}Async(IRpcContext<AmpMessage> context, AmpMessage req)"); sb.AppendLine("{"); sb.AppendLine($"var request = {inType}.Parser.ParseFrom(req.Data);"); sb.AppendLine($"var data = await {method.Name}Async(request);"); sb.AppendLine("var response = AmpMessage.CreateResponseMessage(req.ServiceId, req.MessageId);"); sb.AppendLine("response.Sequence = req.Sequence;"); sb.AppendLine("response.Data = data.ToByteArray();"); sb.AppendLine("await context.SendAsync(response);"); sb.AppendLine("}"); sb.AppendLine(); sb.AppendLine("//抽象方法"); sb.AppendLine($"public abstract Task<{outType}> {method.Name}Async({inType} request);"); //拼裝if調用語句 sbIfState.AppendFormat("//方法{0}.{1}\n",service.Name,method.Name); sbIfState.AppendLine("if(req.MessageId == "+msgId+"){return this.Receive"+method.Name+"Async(context, req);}"); } //循環方法end //生成主調用代碼 sb.AppendLine("public Task ReceiveAsync(IRpcContext<AmpMessage> context, AmpMessage req)"); sb.AppendLine("{"); sb.Append(sbIfState); sb.AppendLine("return Task.CompletedTask;"); sb.AppendLine("}"); sb.AppendLine("}"); //類結束 } private static string GetFileNamespace(FileDescriptorProto protofile) { string ns = protofile.Options.CsharpNamespace; if (string.IsNullOrEmpty(ns)) { throw new Exception("" + protofile.Name + ".proto did not set csharp_namespace"); } return ConvertCamelCase(ns); } private static string GetFileName(string fileProto) { string nomalName = fileProto.Split('.')[0]; return ConvertCamelCase(nomalName); } private static string ConvertCamelCase(string nomalName) { return String.Join("", nomalName.Split('_').Select(_ => _.Substring(0, 1).ToUpper() + _.Substring(1))); } private static string GetTypeName(string typeFullName) { return ConvertCamelCase(typeFullName.Split('.').Last()); } } }
而後咱們編寫一個proto文件測試如下
//benchmark.proto syntax = "proto3"; package dotbpe; option csharp_namespace = "DotBPE.IntegrationTesting"; import public "dotbpe_option.proto"; option optimize_for = SPEED; //Benchmark測試服務 service BenchmarkTest{ option (service_id)= 50000 ;//設定服務ID //測試發送Echo消息 rpc Echo (BenchmarkMessage) returns (BenchmarkMessage){ option (message_id)= 1 ;//設定消息ID };//Echo尾部的註釋 // 測試發送退出消息 rpc Quit (Void) returns (Void){ option (message_id)= 10000 ;//設定消息ID };//Quit尾部的註釋 } //我是void消息 message Void { } //我是BenchmarkMessage消息 message BenchmarkMessage { //字段前的註釋 string field1 = 1; //字段後的註釋 //字段前的註釋 多行 //字段前的字數多行 int32 field2 = 2; //字段後的註釋 /** * 字段前註釋特殊格式 * 字段前註釋特殊格式多行 */ int32 field3 = 3; string field4 = 4; repeated fixed64 field5 = 5; string field9 = 9; string field18 = 18; bool field80 = 80; bool field81 = 81; int32 field280 = 280 ; int32 field6 = 6; int64 field22 = 22 ; bool field59 = 59 ; string field7 = 7; int32 field16 = 16 ; int32 field130 = 130 ; bool field12 = 12 ; bool field17 = 17; bool field13 = 13; bool field14 = 14; int32 field104 = 104 ; int32 field100 = 100 ; int32 field101 = 101 ; string field102 = 102; string field103 = 103; int32 field29 = 29 ; bool field30 = 30 ; int32 field60 = 60 ; int32 field271 = 271 ; int32 field272 = 272; int32 field150 = 150; int32 field23 = 23; bool field24 = 24; int32 field25 = 25 ; bool field78 = 78 ; int32 field67 = 67; int32 field68 = 68 ; int32 field128 = 128 ; string field129 = 129 ; int32 field131 = 131 ; }
// dotbpe_option.proto // [START declaration] syntax = "proto3"; package dotbpe; // [END declaration] // [START csharp_declaration] option csharp_namespace = "DotBPE.ProtoBuf"; // [END csharp_declaration] import "google/protobuf/descriptor.proto"; //擴展服務 extend google.protobuf.ServiceOptions { int32 service_id = 51001; bool disable_generic_service_client = 51003; //是否生成客戶端代碼 bool disable_generic_service_server = 51004; //是否生成服務端代碼 } extend google.protobuf.MethodOptions { int32 message_id = 51002; } extend google.protobuf.FileOptions { bool disable_generic_services_client = 51003; //是否生成客戶端代碼 bool disable_generic_services_server = 51004; //是否生成服務端代碼 bool generic_markdown_doc = 51005; //是否生成文檔 }
上面的dotbpe_option.proto 咱們proto文件進行了自定義的擴展,添加一些本身須要的額外信息,其實全部擴展都是對descriptor.proto
中消息的擴展。
而後咱們經過命令來生成一下,這裏有個特殊的約定,必定要注意當咱們設置
protoc-gen-dotbpe=../../tool/ampplugin/dotbpe_amp.exe 插件的名稱protoc-gen-dotbpe時,那麼輸出的目錄必定要寫成--dotbpe_out ,兩個名字一點要匹配哦
set -ex cd $(dirname $0)/../../test/IntegrationTesting/ PROTOC=protoc PLUGIN=protoc-gen-dotbpe=../../tool/ampplugin/dotbpe_amp.exe IntegrationTesting_DIR=./DotBPE.IntegrationTesting/ $PROTOC -I=./protos --csharp_out=$IntegrationTesting_DIR --dotbpe_out=$IntegrationTesting_DIR \ ./protos/benchmark.proto --plugin=$PLUGIN
差很少就結束了,相關的代碼能夠在https://github.com/xuanye/dotbpe/tree/develop/src/tool 查看到,這是我最近在寫的一個C#的rpc框架,如今完成了基本的功能,還須要進一步完善,有機會再介紹把。
descriptor.proto信息挖掘
咱們注意到在descriptor.proto文件中包含有這樣的一個message: SourceCodeInfo, 這個消息體裏有以下字段
optional string leading_comments = 3; optional string trailing_comments = 4; repeated string leading_detached_comments = 6;
這是很是有意思的定義,意思是能夠在運行時獲取到proto文件中的註釋。這能夠幫助咱們生成 文檔或者代碼註釋,可是讀取邏輯比較複雜,其內部有一個經過Path和Span來定位元素的邏輯。由於在實際的狀況中,通常都是要獲取Service和Message上的註釋,那麼就來專門討論一下如何獲取這兩個類型的註釋吧。
下面是 SourceCodeInfo.Location
中咱們須要用到Path示例
* [4, m] - Message的註釋 * [4, m, 2, f] - Message 中 字段(field)的註釋 * [6, s] - Service的註釋 * [6, s, 2, r] - Service中Rpc方法的註釋
where:
m
- proto文件中Message的索引(就是第幾個定義的Message), 從0開始f
- Message中Field字段的索引(就是第幾個字段), 從0開始s
- proto文件中Service的索引, 從0開始r
- Service中Rpc方法的索引, 從0開始like this:
// [4, 0] 就是這裏的註釋 message MyMessage { // [4, 0, 2, 0] 在這裏 int32 field1 = 1; // [4, 0, 2, 0] 也在這裏 }// [4, 0] 就是這裏的註釋 // [6, 0] 在這裏! service MyService { // [6, 0, 2, 0] 在這裏! rpc (MyMessage) returns (MyMessage); }
想要了解所有內容能夠去看下descriptor.proto
中的註釋內容 吧