【Protobuf專題】（四）proto開發神器——Jprotobuf

時間 2021-02-15

標籤 java git github 數組 maven ide 工具性能測試 gradle 欄目微服務简体版

原文原文鏈接

0 前言

在JAVA團隊內部交互數據時，有幾種經常使用的方法，好比JAVA原生序列化方法、Json序列化方法、Protobuf序列化方法等，對比實驗詳見：幾種序列化方式的性能比較。java

從對比實驗可知protobuf性能最佳，由於其高效的序列化性能以及優秀的跨平臺能力，被普遍應用於跨平臺之間的數據交互。但在不須要考慮跨平臺的數據交互時，protobuf的繁瑣流程成了煩人的「污點」。git

基於ptoro傳輸數據的通常流程是，先編寫proto文件，再手動編譯proto文件生成java類，而後實例化java類並填充數據，最後編譯成二進制數據進行傳輸。其中編寫及手動編譯proto文件是個較繁瑣的過程（好比複雜項目中proto之間的相互引用），並且由proto生成的java類使用的是建立者模式（即Bulider模式），其實例化後的賦值過程與項目歷史代碼風格不一致，爲帶來較大的改動。github

爲解決這個問題，本文將介紹百度團隊開發的Jprotobuf工具，並總結使用過程當中的注意事項，以及經過實驗驗證其與谷歌團隊開發的Protobuf功能的一致性。數組

1 Jprotobuf介紹

1.1 定義

jprotobuf是針對JAVA程序開發一套簡易類庫，目的是簡化JAVA語言對protobuf類庫的使用。使用jprotobuf能夠無需再去了解proto文件操做與語法，直接使用JAVA註解定義字段類型便可。maven

github地址： https://github.com/jhunters/j...

1.2 原理

jprotobuf工做原理以下：ide

掃描類上的註解的信息，進行分析(與protobuf讀取proto文件進行分析過程類似)
根據註解分析的結果，動態生成java代碼進行protobuf序列化與反序列化的功能實現
使用JDK6及以上的 code compile API進行編譯後加載到classloader

1.3 性能

jprotobuf 主要性能消耗在掃描類上註解，動態生成代碼編譯的過程。在執行序列化與反序列化的過程當中，幾乎與protobuf生成的代碼效率等同。若是使用預編譯插件，則無需在運行中進行代碼生成與編譯，效率更高。工具

1.4 特色

無需編寫proto文件及繁瑣的手工編譯過程，支持基於POJO對象的註解方式，方便快捷。
支持protobuf全部類型，包括對象嵌套，數組，枚舉類型
提供根據proto文件，動態生成代理對象，可省去POJO對象的編寫工做。
完整支持proto文件全部功能，包括內聯對象，匿名對象，枚舉類型
提供從POJO對象的註解方式自動生成proto文件的功能，方便proto描述文件的管理與維護
提供預編譯Maven插件，進一步提高運行性能
新增預編譯gradle插件
2.x版本。支持TimeStamp類型, 與原生protobuf保持一致。支持Date類型，使用long類型傳遞 docs

2 Jprotobuf安裝

maven代碼以下：性能

<!-- jprotobuf -->
<dependency>
    <groupId>com.baidu</groupId>
    <artifactId>jprotobuf</artifactId>
    <version>2.4.5</version>
</dependency>
<dependency>
    <groupId>com.baidu</groupId>
    <artifactId>jprotobuf-precompile-plugin</artifactId>
    <version>2.2.2</version>
</dependency>

3 案例數據

測試思路：經過各自的方法構建相同的測試數據，進行序列化和反序列比較二者之間的差別。
注意事項：設計的案例數據儘可能包含全部的數據類型。測試

3.1 Protobuf數據定義

組織結構如圖，其中data依賴student，student依賴person，teacher做爲第三方擴展包，用於測試擴展功能。
gradle

data.proto代碼以下：

// Google Protocol Buffers Version 3.
syntax = "proto3";
// Package name.
package prcticeProto.messages;
// Options for code generation.
option java_package = "learnProto.practiceTest.protoModel";
option java_outer_classname = "SchoolModel";
option java_multiple_files = true;
// import packages
import "google/protobuf/any.proto";
import "practiceProto/categories/student.proto";
message School {
  message Location{
    string name=1;
    uint32 id=2;
  }
  Location schoolLocation = 1;
  bool isOpen =2;
  repeated categories.Student allStudents = 3;
  google.protobuf.Any extend =4;
}

student.proto代碼以下：

// Google Protocol Buffers Version 3.
syntax = "proto3";
// Package name.
package prcticeProto.categories;
// Options for code generation.
option java_package = "learnProto.practiceTest.protoModel";
option java_outer_classname = "StudentModel";
option java_multiple_files = true;
// import packages
import "practiceProto/base/person.proto";
message Student {
  base.Person baseInfo = 1;
  fixed32 calssId = 2;
  sint32 score = 3;
}

person.proto代碼以下：

// Google Protocol Buffers Version 3.
syntax = "proto3";
// Package name.
package prcticeProto.base;
// Options for code generation.
option java_package = "learnProto.practiceTest.protoModel";
option java_outer_classname = "PersonModel";
option java_multiple_files = true;
message Person{
  message Location{
    string placeName=1;
    uint64 placeId=2;
  }
  enum Gender{
    man=0;
    woman=1;
  }
  string name = 1;
  int32 age=2;
  Gender gender=3;
  float height=4;
  double weight=5;
  Location location=6;
}

teacher.proto代碼以下：

// Google Protocol Buffers Version 3.
syntax = "proto3";
// Options for code generation.
option java_package = "learnProto.practiceTest.protoModel";
option java_outer_classname = "TeacherModel";
option java_multiple_files = true;
message Teacher{
  string name = 1;
  int32 age=2;
}

3.2 Jprotobuf數據定義

組織結構如圖所示。

Jprotobuf的定義語法詳見： https://github.com/jhunters/j...

school代碼以下：

package learnProto.jprotobuf.model;
import com.baidu.bjf.remoting.protobuf.Any;
import com.baidu.bjf.remoting.protobuf.FieldType;
import com.baidu.bjf.remoting.protobuf.annotation.Protobuf;
import com.baidu.bjf.remoting.protobuf.annotation.ProtobufClass;
import java.util.List;

@ProtobufClass
public class School {
 @ProtobufClass
 public static class Location{
        @Protobuf(fieldType= FieldType.STRING, order=1)
        private String name;
        @Protobuf(fieldType= FieldType.UINT32, order=2)
        private Integer id;
        public String getName() {
            return name;
        }
        public Location setName(String name) {
            this.name = name;
            return this;
        }
        public Integer getId() {
            return id;
        }
        public Location setId(Integer id) {
            this.id = id;
            return this;
        }
    }
    @Protobuf(fieldType= FieldType.OBJECT, order=1)
    private Location schoolLocation;
    @Protobuf(fieldType= FieldType.BOOL, order=2)
    private Boolean isOpen;
    @Protobuf(fieldType= FieldType.OBJECT, order=3)
    private List<Student> allStudents;
    @Protobuf(fieldType = FieldType.OBJECT,order = 4)
    private Any extend;
    public Any getExtend() {
        return extend;
    }
    public School setExtend(Any extend) {
        this.extend = extend;
        return this;
    }
    public Location getSchoolLocation() {
        return schoolLocation;
    }
    public School setSchoolLocation(Location schoolLocation) {
        this.schoolLocation = schoolLocation;
        return this;
    }
    public Boolean getOpen() {
        return isOpen;
    }
    public School setOpen(Boolean open) {
        isOpen = open;
        return this;
    }
    public List<Student> getAllStudents() {
        return allStudents;
    }
    public School setAllStudents(List<Student> allStudents) {
        this.allStudents = allStudents;
        return this;
    }
}

student代碼以下：

package learnProto.jprotobuf.model;
import com.baidu.bjf.remoting.protobuf.FieldType;
import com.baidu.bjf.remoting.protobuf.annotation.Protobuf;
import com.baidu.bjf.remoting.protobuf.annotation.ProtobufClass;

@ProtobufClass
public class Student{
    @Protobuf(fieldType=FieldType.OBJECT, order=1)
    private Person baseInfo;
    @Protobuf(fieldType=FieldType.FIXED32, order=2)
    private Integer calssId;
    @Protobuf(fieldType=FieldType.SINT32, order=3)
    private Integer score;
    public Person getBaseInfo() {
        return baseInfo;
    }
    public Student setBaseInfo(Person baseInfo) {
        this.baseInfo = baseInfo;
        return this;
    }
    public Integer getCalssId() {
        return calssId;
    }
    public Student setCalssId(Integer calssId) {
        this.calssId = calssId;
        return this;
    }
    public Integer getScore() {
        return score;
    }
    public Student setScore(Integer score) {
        this.score = score;
        return this;
    }
}

person代碼以下：

package learnProto.jprotobuf.model;
import com.baidu.bjf.remoting.protobuf.FieldType;
import com.baidu.bjf.remoting.protobuf.annotation.Protobuf;
import com.baidu.bjf.remoting.protobuf.annotation.ProtobufClass;

@ProtobufClass
public class Person {
    @ProtobufClass
 public enum Gender {
        MAN(0),WOMAN(1);
        @Protobuf(fieldType= FieldType.INT32, order=1)
        private final Integer value;
        Gender(Integer value) { this.value = value; }
        public Integer toValue() { return this.value; }
        public Integer value() {
            return toValue();
        }
    }
 @ProtobufClass
 public static class Location{
        @Protobuf(fieldType= FieldType.STRING, order=1)
        private String placeName;
        @Protobuf(fieldType= FieldType.UINT64, order=2)
        private Long placeId;
        public String getPlaceName() {
            return placeName;
        }
        public Location setPlaceName(String placeName) {
            this.placeName = placeName;
            return this;
        }
        public Long getPlaceId() {
            return placeId;
        }
        public Location setPlaceId(Long placeId) {
            this.placeId = placeId;
            return this;
        }
    }
    @Protobuf(fieldType= FieldType.STRING, order=1)
    private String name;
    @Protobuf(fieldType= FieldType.INT32, order=2)
    private Integer age;
    @Protobuf(fieldType= FieldType.ENUM, order=3)
    private Gender gender;
    @Protobuf(fieldType= FieldType.FLOAT, order=4)
    private Float height;
    @Protobuf(fieldType= FieldType.DOUBLE, order=5)
    private Double weight;
    @Protobuf(fieldType= FieldType.OBJECT, order=6)
    private Location personLocation;
    public String getName() {
        return name;
    }
    public Person setName(String name) {
        this.name = name;
        return this;
    }
    public Integer getAge() {
        return age;
    }
    public Person setAge(Integer age) {
        this.age = age;
        return this;
    }
    public Gender getGender() {
        return gender;
    }
    public Person setGender(Gender gender) {
        this.gender = gender;
        return this;
    }
    public Float getHeight() {
        return height;
    }
    public Person setHeight(Float height) {
        this.height = height;
        return this;
    }
    public Double getWeight() {
        return weight;
    }
    public Person setWeight(Double weight) {
        this.weight = weight;
        return this;
    }
    public Location getPersonLocation() {
        return personLocation;
    }
    public Person setPersonLocation(Location personLocation) {
        this.personLocation = personLocation;
        return this;
    }
}

teacher代碼以下：

package learnProto.jprotobuf.model;
import com.baidu.bjf.remoting.protobuf.annotation.ProtobufClass;

@ProtobufClass
public class Teacher {
    private Integer id;
    private String name;
    public Integer getId() {
        return id;
    }
    public Teacher setId(Integer id) {
        this.id = id;
        return this;
    }
    public String getName() {
        return name;
    }
    public Teacher setName(String name) {
        this.name = name;
        return this;
    }
}

4 案例測試

4.1 基本數據類型測試

public class jprotobufTest {
    public static void writeFileByte(byte[] b,String path) {
        File file = new File(path);
        long size = file.length();
        try (FileOutputStream fos = new FileOutputStream(file)) {
            fos.write(b);
            fos.flush();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
    public static byte[] readFileByte(String path) {
        File file = new File(path);
        long size = file.length();
        byte[] b = new byte[(int) size];
        try (InputStream fis = new FileInputStream(file)) {
            if (fis.read(b) < 0) {
                return null;
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
        return b;
    }
    public void protobuff(String path){
        // 實例化
 learnProto.practiceTest.protoModel.Person.Builder builderPerson = learnProto.practiceTest.protoModel.Person.newBuilder().setAge(10).setGender(learnProto.practiceTest.protoModel.Person.Gender.woman).setName("Tom").setHeight(100.00f).setWeight(100.00d).setLocation(
                learnProto.practiceTest.protoModel.Person.Location.newBuilder().setPlaceId(123l).setPlaceName("hubei")
        );
        learnProto.practiceTest.protoModel.School school = learnProto.practiceTest.protoModel.School.newBuilder()
                .setIsOpen(true)
                .setSchoolLocation(learnProto.practiceTest.protoModel.School.Location.newBuilder().setId(123).setName("hubei"))
                .addAllStudents(Student.newBuilder().setBaseInfo(builderPerson).setCalssId(10).setScore(-1))
                .addAllStudents(Student.newBuilder().setBaseInfo(builderPerson).setCalssId(10).setScore(1))
                .build();
        try {
            // 序列化
 byte[] bytes = school.toByteArray();
            writeFileByte(bytes,path);
            System.out.println("protobuf序列化後的數據：" + Arrays.toString(bytes)+",字節個數："+bytes.length);
            //反序列化
 byte[] bytes1 = readFileByte(path);
            learnProto.practiceTest.protoModel.School parseFrom = learnProto.practiceTest.protoModel.School.parseFrom(bytes1);
        } catch (InvalidProtocolBufferException e) {
            e.printStackTrace();
        }
    }
    public void jprotobuf(String path){
        // 實例化
 Person person = new Person().setAge(10).setGender(Person.Gender.MAN).setName("Tom").setHeight(100.00f).setWeight(100.00d).setPersonLocation(
                new Person.Location().setPlaceId(123l).setPlaceName("hubei")
        );
        ArrayList<learnProto.jprotobuf.model.Student> studentArrayList = new ArrayList<>();
        studentArrayList.add(new learnProto.jprotobuf.model.Student().setCalssId(10).setScore(-1).setBaseInfo(person));
        studentArrayList.add(new learnProto.jprotobuf.model.Student().setCalssId(10).setScore(1).setBaseInfo(person));
        School sch = new School().setOpen(true).setAllStudents(studentArrayList).setSchoolLocation(new School.Location().setId(123).setName("hubei"));
        try {
            // 序列化
 Codec<School> simpleTypeCodec = ProtobufProxy.create(School.class);
            byte[] b = simpleTypeCodec.encode(sch);
            writeFileByte(b,path);
            System.out.println("Jprotobuf序列化後的數據：" + Arrays.toString(b)+",字節個數："+b.length);
            // 反序列化
 byte[] bytes =readFileByte(path);
            School newStt = simpleTypeCodec.decode(bytes);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
    @Test
 public void test(){
        protobuff("C:UsersadminDesktopProtodemo.pack");
        jprotobuf("C:UsersadminDesktopJprotodemo.pack");
    }
}

運行結果：

protobuf序列化後的數據：[10, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 16, 1, 26, 43, 10, 34, 10, 3, 84, 111, 109, 16, 10, 24, 1, 37, 0, 0, -56, 66, 41, 0, 0, 0, 0, 0, 0, 89, 64, 50, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 21, 10, 0, 0, 0, 24, 1, 26, 43, 10, 34, 10, 3, 84, 111, 109, 16, 10, 24, 1, 37, 0, 0, -56, 66, 41, 0, 0, 0, 0, 0, 0, 89, 64, 50, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 21, 10, 0, 0, 0, 24, 2],字節個數：103
Jprotobuf序列化後的數據：[10, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 16, 1, 26, 43, 10, 34, 10, 3, 84, 111, 109, 16, 10, 24, 0, 37, 0, 0, -56, 66, 41, 0, 0, 0, 0, 0, 0, 89, 64, 50, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 21, 10, 0, 0, 0, 24, 1, 26, 43, 10, 34, 10, 3, 84, 111, 109, 16, 10, 24, 0, 37, 0, 0, -56, 66, 41, 0, 0, 0, 0, 0, 0, 89, 64, 50, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 21, 10, 0, 0, 0, 24, 2],字節個數：103

實驗代表，Jprotobuf的序列化方法與Protobuf一致，能夠放心使用。

4.2 第三方擴展測試

@Test
public void schooltest() throws IOException {
    // jproto
 Any any = Any.pack(new Teacher());
    School person = new School().setExtend(any);
    Codec<School> simpleTypeCodec = ProtobufProxy.create(School.class);
    try {
        // 序列化
 byte[] b = simpleTypeCodec.encode(person);
        writeFileByte(b,"C:UsersadminDesktopjproto.pack");
        System.out.println("jproto序列化後的數據：" + Arrays.toString(b)+",字節個數："+b.length);
        // 反序列化
 byte[] bytes =readFileByte("C:UsersadminDesktopjproto.pack");
        School newStt = simpleTypeCodec.decode(bytes);
    } catch (IOException e) {
        e.printStackTrace();
    }
    // proto
 com.google.protobuf.Any any1 = com.google.protobuf.Any.pack(learnProto.practiceTest.protoModel.Teacher.newBuilder().build());
    learnProto.practiceTest.protoModel.School builderPerson = learnProto.practiceTest.protoModel.School.newBuilder()
            .setExtend(any1)
            .build();
    try {
        // 序列化
 String path="C:UsersadminDesktopProtodemo.pack";
        byte[] bytes = builderPerson.toByteArray();
        writeFileByte(bytes,path);
        System.out.println("protobuf序列化後的數據：" + Arrays.toString(bytes)+",字節個數："+bytes.length);
        //反序列化
 byte[] bytes1 = readFileByte(path);
        learnProto.practiceTest.protoModel.School parseFrom = learnProto.practiceTest.protoModel.School.parseFrom(bytes1);
    } catch (InvalidProtocolBufferException e) {
        e.printStackTrace();
    }
}

運行結果：

jproto序列化後的數據：[34, 58, 10, 54, 116, 121, 112, 101, 46, 103, 111, 111, 103, 108, 101, 97, 112, 105, 115, 46, 99, 111, 109, 47, 108, 101, 97, 114, 110, 80, 114, 111, 116, 111, 46, 106, 112, 114, 111, 116, 111, 98, 117, 102, 46, 109, 111, 100, 101, 108, 46, 84, 101, 97, 99, 104, 101, 114, 18, 0],字節個數：60
protobuf序列化後的數據：[34, 29, 10, 27, 116, 121, 112, 101, 46, 103, 111, 111, 103, 108, 101, 97, 112, 105, 115, 46, 99, 111, 109, 47, 84, 101, 97, 99, 104, 101, 114],字節個數：31

實驗代表，Jprotobuf的擴展功能能夠正常使用，但其機制可能與protobuf有差別。

5 使用總結

一、字段的註解@Protobuf，能夠不寫，不寫的狀況下按照默認類型，order按照書寫的順序；也能夠寫，自定義類型和order。但必定不要有的字段寫，有的字段不寫，order會亂！

二、構建的@ProtobufClass不要帶有有參構造器，實例化時也不能帶參數構造。只能無參構造後，經過set或add賦值（與proto的實例化規範一致）。

三、能夠調用_getIDL_得到IDL，即proto文件，但全部的嵌套結構都被拆開成了單獨的結構。

四、 setter的返回類型能夠設定爲類自己，從而實現鏈式。如圖：

五、 定義基本數據類型的數據時，要用包裝類。好比對於整型，不要寫int，而應該寫INTGER，如上圖中的id，若是設置爲int，系統默認初始化了id=0，即便沒有賦值，該數值在序列化時會被寫入數據；若是設置爲INTGER，沒有賦值，不會被寫入序列化的字節序中。（此處與proto不一樣，proto中字段設定值等於默認值時，序列化時數據不會被寫入，反序列化後全部沒被賦值的字段都被賦對應類型的默認值；而jprotobuf徹底按照用戶輸入的意願，序列化以前賦值什麼，序列化時就有什麼，反序列化時也對應有什麼，若是字段沒被賦值，反序列化後爲null，enum爲第一個枚舉值）。

六、反序列化使用時注意：沒被賦值的字段反序列化後值爲null，並不是默認值。

七、設定了多個字段，有些沒有賦值，並不會影響數據序列化後的大小。