Apache Thrift概念以及快速入門

thrift的全名叫作Apache thrift，是一款軟件開發RPC框架，能夠很高效地實現跨語言的RPC服務。

本文簡要介紹了thrift的背景、相關概念以及安裝流程。並給出了C++以及python版本的入門例子。其中背景概念部分翻譯自[1]。

1 Krzysztof Rakowski的Apache Thrift介紹

1.1 部分歷史與背景

Thrift由facebook在2007年提出。facebook的公司文化就是不限制你選擇任何開發語言，用最佳方案解決實際問題。毫無疑問，公司內部許多應用程序都是用不一樣語言編寫的。這也是爲何facebook須要一種工具，使這些應用程序可以相互通訊。而後他們花了一些精力尋找這樣的工具，惋惜並沒找到合適的。facebook發佈的白皮書也提到了他們曾經考察的工具，以及Thrift框架的設計思路。

固然啦，牛逼如facebook的程序員克服了一些挑戰，本身開發了該需求的解決方案——Thrift就是這樣誕生的。不久以後，他們開源了Thrift，而且託管到Apache基金會，後者開始負責Thrift的開發。如今Thrift不只僅被用在facebook（公司內部應用間通訊的主要工具），並且被其餘許多公司使用。（包括Evernote、Twitter和Netflix等知名公司）。facebook的工程師仍然會在fork版本FBThrift上繼續開發。而且有但願與Apache Thrift整合。

1.2 Apache Thrift到底是什麼

咱們想象一下這種情形：你有許多用不一樣語言開發的應用程序，好比一種很常見的狀況就是公司內部獨立的開發小組各自開發了一些應用程序，用來執行一些不一樣的內部任務。那怎麼讓這些應用程序相互通訊呢？固然啦，你能夠添加一些REST API。可是在許多狀況下——尤爲是你要傳輸二進制數據——這種方案並不能知足性能和可靠性要求。

1.3 Thrift如何工做

首先，讓咱們從Apache Thrift開發者的角度來看看。
Thrift框架的主要概念是服務，服務和麪向對象編程語言中的類很類似。每一個服務中都包含方法，也是OOP中的相似概念。Thrift還實現了須要數據類型。這些數據類型在每種編程語言中都有具體對應的類型，最簡單的例子好比Thrift中的int，在全部語言中都會映射成整型。可是複雜點的好比set，在php中會映射成array，在java中會映射成HashSet。編寫服務的文本文件也被稱爲Thrift文檔（後綴是.thrift），包括服務在內，文檔中的全部代碼都是接口定義語言(Interface Description Language ,IDL)，若是要了解詳細語法，請參考官方文檔[2]。（注：其實IDL語法很簡單，源碼中有一份自解釋的Thrift文件：tutorial.thrift）

# Thrift Tutorial
# Mark Slee (mcslee@facebook.com)
#
# This file aims to teach you how to use Thrift, in a .thrift file. Neato. The
# first thing to notice is that .thrift files support standard shell comments.
# This lets you make your thrift file executable and include your Thrift build
# step on the top line. And you can place comments like this anywhere you like.
#
# Before running this file, you will need to have installed the thrift compiler
# into /usr/local/bin.

/**
 * The first thing to know about are types. The available types in Thrift are:
 *
 *  bool        Boolean, one byte
 *  i8 (byte)   Signed 8-bit integer
 *  i16         Signed 16-bit integer
 *  i32         Signed 32-bit integer
 *  i64         Signed 64-bit integer
 *  double      64-bit floating point value
 *  string      String
 *  binary      Blob (byte array)
 *  map<t1,t2>  Map from one type to another
 *  list<t1>    Ordered list of one type
 *  set<t1>     Set of unique elements of one type
 *
 * Did you also notice that Thrift supports C style comments?
 */

// Just in case you were wondering... yes. We support simple C comments too.

/**
 * Thrift files can reference other Thrift files to include common struct
 * and service definitions. These are found using the current path, or by
 * searching relative to any paths specified with the -I compiler flag.
 *
 * Included objects are accessed using the name of the .thrift file as a
 * prefix. i.e. shared.SharedObject
 */
include "shared.thrift"

/**
 * Thrift files can namespace, package, or prefix their output in various
 * target languages.
 */

namespace cl tutorial
namespace cpp tutorial
namespace d tutorial
namespace dart tutorial
namespace java tutorial
namespace php tutorial
namespace perl tutorial
namespace haxe tutorial
namespace netstd tutorial

/**
 * Thrift lets you do typedefs to get pretty names for your types. Standard
 * C style here.
 */
typedef i32 MyInteger

/**
 * Thrift also lets you define constants for use across languages. Complex
 * types and structs are specified using JSON notation.
 */
const i32 INT32CONSTANT = 9853
const map<string,string> MAPCONSTANT = {'hello':'world', 'goodnight':'moon'}

/**
 * You can define enums, which are just 32 bit integers. Values are optional
 * and start at 1 if not supplied, C style again.
 */
enum Operation {
  ADD = 1,
  SUBTRACT = 2,
  MULTIPLY = 3,
  DIVIDE = 4
}

/**
 * Structs are the basic complex data structures. They are comprised of fields
 * which each have an integer identifier, a type, a symbolic name, and an
 * optional default value.
 *
 * Fields can be declared "optional", which ensures they will not be included
 * in the serialized output if they aren't set.  Note that this requires some
 * manual management in some languages.
 */
struct Work {
  1: i32 num1 = 0,
  2: i32 num2,
  3: Operation op,
  4: optional string comment,
}

/**
 * Structs can also be exceptions, if they are nasty.
 */
exception InvalidOperation {
  1: i32 whatOp,
  2: string why
}

/**
 * Ahh, now onto the cool part, defining a service. Services just need a name
 * and can optionally inherit from another service using the extends keyword.
 */
service Calculator extends shared.SharedService {

  /**
   * A method definition looks like C code. It has a return type, arguments,
   * and optionally a list of exceptions that it may throw. Note that argument
   * lists and exception lists are specified using the exact same syntax as
   * field lists in struct or exception definitions.
   */

   void ping(),

   i32 add(1:i32 num1, 2:i32 num2),

   i32 calculate(1:i32 logid, 2:Work w) throws (1:InvalidOperation ouch),

   /**
    * This method has a oneway modifier. That means the client only makes
    * a request and does not listen for any response at all. Oneway methods
    * must be void.
    */
   oneway void zip()

}

/**
 * That just about covers the basics. Take a look in the test/ folder for more
 * detailed examples. After you run this file, your generated code shows up
 * in folders with names gen-<language>. The generated code isn't too scary
 * to look at. It even has pretty indentation.
 */

接着，從這個Thrift文件——經過Apache Thrift編譯器——你能夠生成server和client的stub。這些自動生成的代碼被稱爲Apache Thrift庫，而後你藉助庫，按照你的需求，實現指定語言的server和client——過程很像代碼填空，填好本身相關部分便可。（好比對象建立、方法調用等語句），這樣就實現了不一樣應用程序相互通訊。你生成的server和client代碼嵌入到你的應用程序中。過程以下所示：

看實例以前，讓咱們大體看下Apache Thrift的架構。以下圖所示：

傳輸層的功能是從你使用的介質（通常是socket）中讀寫負載。協議層通常和傳輸層解耦，負責數據的編碼和解析，方便數據傳輸。經常使用協議包括：binary，compact（Thrift獨有）以及JSON。至於處理器層，這裏說的很模糊，主要是把用戶指定的輸入輸出協議做爲參數，而後從輸入讀取數據，按照用戶規則進行處理，最後向輸出寫入數據[3]，以下圖所示：

在server和client代碼中，這三層組合在一塊兒。若是你想兩個應用程序相互通訊，那麼server和client要使用相同的傳輸層和協議層集合，這樣才能正確編碼和解析數據。第三節有C++以及python版本的入門例子。

2 安裝

一、安裝依賴項，官方給出了不一樣系統下的依賴項安裝步驟。

二、編譯源代碼，安裝

# github mirror: https://github.com/apache/thrift
git clone https://git-wip-us.apache.org/repos/asf/thrift.git
cd thrift
./bootstrap.sh

# 禁用lua語言，查看更多參數：./configure --help
./configure --with-lua=no

# 編譯
make
# 安裝
sudo make install

configure這一步結束會在terminal輸出對不一樣語言的支持信息，檢查一下是否知足本身要求。

Building Plugin Support ...... : yes
Building C++ Library ......... : yes
Building C (GLib) Library .... : no
Building Java Library ........ : no
Building C# Library .......... : no
Building .NET Core Library ... : no
Building Python Library ...... : yes
Building Ruby Library ........ : no
Building Haxe Library ........ : no
Building Haskell Library ..... : no
Building Perl Library ........ : no
Building PHP Library ......... : yes
Building Dart Library ........ : no
Building Erlang Library ...... : no
Building Go Library .......... : no
Building D Library ........... : no
Building NodeJS Library ...... : no
Building Lua Library ......... : no

C++ Library:
   Build TZlibTransport ...... : yes
   Build TNonblockingServer .. : yes
   Build TQTcpServer (Qt4) .... : no
   Build TQTcpServer (Qt5) .... : no

Python Library:
   Using Python .............. : /usr/bin/python
   Using Python3 ............. : /usr/local/bin/python3

若是make這一步出現缺乏.a文件的問題：

g++: error: /usr/lib64/libboost_unit_test_framework.a: No such file or directory

首先檢查上一步的依賴項有沒有所有安裝，沒問題的話能夠看看/usr/local/lib/下有沒有該文件，再拷貝到make過程當中所尋找的路徑下。

$ ls /usr/local/lib/libboost_unit_test_framework.*
/usr/local/lib/libboost_unit_test_framework.a
/usr/local/lib/libboost_unit_test_framework.so
/usr/local/lib/libboost_unit_test_framework.so.1.53.0

$ ls /usr/lib64/libboost_unit_test_framework*
/usr/lib64/libboost_unit_test_framework-mt.so         /usr/lib64/libboost_unit_test_framework.so
/usr/lib64/libboost_unit_test_framework-mt.so.1.53.0  /usr/lib64/libboost_unit_test_framework.so.1.53.0

$ make -n | grep libboost_unit_test_framework
/bin/sh ../../../libtool  --tag=CXX   --mode=link g++ -Wall -Wextra -pedantic -g -O2 -std=c++11 -L/usr/lib64  -o processor_test processor/ProcessorTest.o processor/EventLog.o processor/ServerThread.o libprocessortest.la ../../../lib/cpp/libthrift.la ../../../lib/cpp/libthriftnb.la /usr/lib64/libboost_unit_test_framework.a -L/usr/lib64 -levent -lrt -lpthread

$ sudo cp /usr/local/lib/libboost_unit_test_framework.a /usr/lib64/
$ make
no error.

三、驗證

$ thrift -version
Thrift version 1.0.0-dev
$ which thrift
/usr/local/bin/thrift

3 入門例子

編寫一個簡單的thrift文件，定義相乘函數

$ ls
multiplication.thrift
$ cat multiplication.thrift
namespace py tutorial

service MultiplicationService
{
    i32 multiply(1:i32 n1, 2:i32 n2),
}

3.1 C++ server和client

編譯生成c++代碼

$ thrift --gen cpp multiplication.thrift

$ ls -R 
.:
gen-cpp  multiplication.thrift

./gen-cpp:
multiplication_constants.cpp    MultiplicationService.h    multiplication_types.h
multiplication_constants.h      MultiplicationService_server.skeleton.cpp
MultiplicationService.cpp       multiplication_types.cpp

接下來修改官方模版skeleton.cpp[4]

$ mv MultiplicationService_server.skeleton.cpp server.cpp
只須要修改server.cpp中int32_t multiply(const int32_t n1, const int32_t n2)函數
$ cat server.cpp
...(中間省略)...
  int32_t multiply(const int32_t n1, const int32_t n2) {
    // Your implementation goes here
    return n1*n2;
  }
...(中間省略)...

而後編寫client.cpp

#include "MultiplicationService.h"
#include <thrift/transport/TSocket.h>
#include <thrift/transport/TBufferTransports.h>
#include <thrift/protocol/TBinaryProtocol.h>

#include <iostream>
using namespace std;

using namespace apache::thrift;
using namespace apache::thrift::protocol;
using namespace apache::thrift::transport;

int main(int argc, char *argv[]) {
  boost::shared_ptr<TSocket> socket(new TSocket("localhost", 9090));
  boost::shared_ptr<TTransport> transport(new TBufferedTransport(socket));
  boost::shared_ptr<TProtocol> protocol(new TBinaryProtocol(transport));

  int res=0;
  MultiplicationServiceClient client(protocol);
  transport->open();
  res = client.multiply(10,10);
  cout << "res "<<res<<endl;
  transport->close();
  return 0;
}

編譯源碼

$ g++ -c MultiplicationService.cpp
$ g++ -c multiplication_constants.cpp multiplication_types.cpp server.cpp
$ g++ -lthrift *.o -o server.exe
$ g++ -c MultiplicationService.cpp multiplication_constants.cpp multiplication_types.cpp client.cpp
$ g++ -lthrift *.o -o client.exe

$ ls -R 
.:
gen-cpp  multiplication.thrift

./gen-cpp:
client.cpp  multiplication_constants.cpp  MultiplicationService.cpp  multiplication_types.cpp  server.cpp
client.exe  multiplication_constants.h    MultiplicationService.h    multiplication_types.h    server.exe
client.o    multiplication_constants.o    MultiplicationService.o    multiplication_types.o    server.o

運行

shell1> ./server.exe
shell2> ./client.exe
res 100

若是運行server.exe出現找不到共享庫的錯誤，解決辦法以下：

$ ./server 
./server: error while loading shared libraries: libthrift-1.0.0-dev.so: cannot open shared object file: No such file or directory
$ sudo find / -name "libthrift*so" #where is your libthrift
/usr/local/lib/libthrift-1.0.0-dev.so
/usr/local/lib/libthrift.so
...
$ echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/:/usr/lib/" >> ~/.bashrc  #set env var
$ . ~/.bashrc  #reload

3.2 python server和client

生成python代碼

$ thrift --gen py multiplication.thrift
$ ls -R gen-py
./gen-py:
__init__.py  tutorial

./gen-py/tutorial:
constants.py  __init__.pyc              MultiplicationService.pyc     ttypes.py
__init__.py   MultiplicationService.py  MultiplicationService-remote  ttypes.pyc

建立server.py和client.py

server.py

import glob
import sys
sys.path.append('gen-py')
sys.path.insert(0, glob.glob('../../lib/py/build/lib*')[0])
from tutorial import MultiplicationService
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from thrift.server import TServer

class MultiplicationServiceHandler(MultiplicationService.Iface):
    def __init__(self):
        self.log = {}

    def multiply(self, n1, n2):
        print('multiply(%d,%d)' % (n1,n2))
        return n1*n2

if __name__ == '__main__':
    handler = MultiplicationServiceHandler()
    processor = MultiplicationService.Processor(handler)
    transport = TSocket.TServerSocket(port=9090)
    tfactory = TTransport.TBufferedTransportFactory()
    pfactory = TBinaryProtocol.TBinaryProtocolFactory()

    server = TServer.TSimpleServer(processor, transport, tfactory, pfactory)

    # You could do one of these for a multithreaded server
    # server = TServer.TThreadedServer(
    #     processor, transport, tfactory, pfactory)
    # server = TServer.TThreadPoolServer(
    #     processor, transport, tfactory, pfactory)

    print('Starting the server...')
    server.serve()

client.py

import sys
import glob
sys.path.append('gen-py')
sys.path.insert(0, glob.glob('../../lib/py/build/lib*')[0])

from tutorial import MultiplicationService
from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol

def main():
    transport = TSocket.TSocket('localhost', 9090)
    # Buffering is critical. Raw sockets are very slow
    transport = TTransport.TBufferedTransport(transport)
    # Wrap in a protocol
    protocol = TBinaryProtocol.TBinaryProtocol(transport)
    # Create a client to use the protocol encoder
    client = MultiplicationService.Client(protocol)
    # Connect!
    transport.open()
    print(client.multiply(10,10))
    # Close!
    transport.close()

if __name__ == '__main__':
    try:
        main()
    except Thrift.TException as tx:
        print('%s' % tx.message)

運行

$ chmod a+x server.py
$ chmod a+x client.py

shell1> ./server.py
Starting the server...
multiply(10,10)
shell2> ./client.py
100

python client使用C++ server提供的服務

shell1> ./server.exe
shell2> ./client.py
100

References

1. http://www.thrift.pl/
2. http://thrift.apache.org/
3. http://thrift-tutorial.readthedocs.io
4. http://roclinux.cn/?p=3316
5. 你應該知道的 RPC 原理