Protobuf、Avro

使用場景

• hadoop大文件塊序列化傳輸
• Spark-netty Shuffe機制序列化傳輸
• 大文件塊序列化傳輸，大Payload的RPC序列化

什麼是序列化：將對象或者文件轉換成0和1組成的字節數組，如[1,0,1,1,1,0],而後就能夠用經過網絡進行傳輸，不一樣序列化工具算法數據結構不一樣。性能，結果大小不一樣，具體使用什麼樣的序列化須要結合場景。

爲何用 protobuf、Arvo

• 對比JSON和Java原生，序列化性能更高，結果體積更小。適用於大數據場景，如Shuffe，RPC，數據容災冗餘。
• 跨語言，若是你使用Java原生序列化，其餘語言是沒法反序列化的。

Protobuf

既然是工具，用起來其實很簡單。使用步驟。

1. 定義以 .proto 爲結尾的Schema文件，這個文件申明瞭被序列化數據的格式，反序列化端只需有一樣內容.proto文件便可完成反序列化。
2. 編譯.proto爲Java源碼
3. 使用Protobuf JavaSDK進行數據傳輸。

Protobuf編譯器下載

github，下載本身系統對應版本便可。如我是Win64，就下載protoc-3.11.2-win64.zip，解壓後將bin目錄添加到環境變量。

pom .proto2

<!--核心SDK-->
<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-java</artifactId>
  <version>3.10.0</version>
</dependency>
<!--JSON工具-->
<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-java-util</artifactId>
  <version>3.10.0</version>
</dependency>
複製代碼

新建 addressbook.proto

idea新建Proto須要安裝插件 ProtoBuf Support

#申明proto語法版本爲2，hadoop3.X版本中用的是proto2
syntax = "proto2";

#申明proto文件所屬的包，相似於java的包管理。
package tutorial;

#聲明proto文件編譯產生java文件輸出包，輸出java類名
option java_package = "com.example.tutorial";
option java_outer_classname = "AddressBookProtos";

#聲明消息格式
message Person {
# required必需要求的，爲空則會報錯
# optional可選的
# repeated 可重複的，重複次數從0到無限
# 賦值爲 1 或 2表明其重要程度，對於不經常使用的能夠標記爲更大，大到15
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phones = 4;
}

message AddressBook {
  repeated Person people = 1;
}
複製代碼

編譯生成Java源文件

protoc -I={你proto文件放置的文件夾，如E:\ideaproject\test\src\main\proto\} --java_out={java輸出目錄，如E:\ideaproject\test\src\main\java}  {proto文件位置，如E:\ideaproject\test\src\main\proto\addressbook}
複製代碼

API操做

寫數據，比較簡單，API調用起來。

class Writer {
    // This function fills in a Person message based on user input.
    static AddressBookProtos.Person PromptForAddress(BufferedReader stdin, PrintStream stdout) throws IOException {
        AddressBookProtos.Person.Builder person = AddressBookProtos.Person.newBuilder();

        stdout.print("Enter person ID: ");
        person.setId(Integer.valueOf(stdin.readLine()));

        stdout.print("Enter name: ");
        person.setName(stdin.readLine());

        stdout.print("Enter email address (blank for none): ");
        String email = stdin.readLine();
        if (email.length() > 0) {
            person.setEmail(email);
        }

        while (true) {
            stdout.print("Enter a phone number (or leave blank to finish): ");
            String number = stdin.readLine();
            if (number.length() == 0) {
                break;
            }

            AddressBookProtos.Person.PhoneNumber.Builder phoneNumber =
                    AddressBookProtos.Person.PhoneNumber.newBuilder().setNumber(number);

            stdout.print("Is this a mobile, home, or work phone? ");
            String type = stdin.readLine();
            if (type.equals("mobile")) {
                phoneNumber.setType(AddressBookProtos.Person.PhoneType.MOBILE);
            } else if (type.equals("home")) {
                phoneNumber.setType(AddressBookProtos.Person.PhoneType.HOME);
            } else if (type.equals("work")) {
                phoneNumber.setType(AddressBookProtos.Person.PhoneType.WORK);
            } else {
                stdout.println("Unknown phone type. Using default.");
            }

            person.addPhones(phoneNumber);
        }

        return person.build();
    }

    // Main function: Reads the entire address book from a file,
    // adds one person based on user input, then writes it back out to the same
    // file.
    public static void main(String[] args) throws Exception {
        if (args.length != 1) {
            System.err.println("Usage: AddPerson ADDRESS_BOOK_FILE");
            System.exit(-1);
        }

        AddressBookProtos.AddressBook.Builder addressBook = AddressBookProtos.AddressBook.newBuilder();

        // Read the existing address book.
        try {
            addressBook.mergeFrom(new FileInputStream(args[0]));
        } catch (FileNotFoundException e) {
            System.out.println(args[0] + ": File not found. Creating a new file.");
        }

        // Add an address.
        addressBook.addPeople(
                PromptForAddress(new BufferedReader(new InputStreamReader(System.in)),
                        System.out));

        // Write the new address book back to disk.
        FileOutputStream output = new FileOutputStream(args[0]);
        addressBook.build().writeTo(output);
        output.close();
    }
}
複製代碼

讀數據，也是比較簡單的。

import com.example.tutorial.AddressBookProtos.AddressBook;
import com.example.tutorial.AddressBookProtos.Person;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.PrintStream;

class ListPeople {
  // Iterates though all people in the AddressBook and prints info about them.
  static void Print(AddressBook addressBook) {
    for (Person person: addressBook.getPeopleList()) {
      System.out.println("Person ID: " + person.getId());
      System.out.println(" Name: " + person.getName());
      if (person.hasEmail()) {
        System.out.println(" E-mail address: " + person.getEmail());
      }

      for (Person.PhoneNumber phoneNumber : person.getPhonesList()) {
        switch (phoneNumber.getType()) {
          case MOBILE:
            System.out.print(" Mobile phone #: ");
            break;
          case HOME:
            System.out.print(" Home phone #: ");
            break;
          case WORK:
            System.out.print(" Work phone #: ");
            break;
        }
        System.out.println(phoneNumber.getNumber());
      }
    }
  }

  // Main function:  Reads the entire address book from a file and prints all
  //   the information inside.
  public static void main(String[] args) throws Exception {
    if (args.length != 1) {
      System.err.println("Usage: ListPeople ADDRESS_BOOK_FILE");
      System.exit(-1);
    }

    // Read the existing address book.
    AddressBook addressBook =
      AddressBook.parseFrom(new FileInputStream(args[0]));

    Print(addressBook);
  }
}
複製代碼

protocbuf用起來仍是很是容易，只是須要額外的Proto文件描述數據的Schema。

Avro

基本和Protobuf一致，須要定義Schema文件，而後進行API操做。