使用Nginx代理thrift NIO實現SSL鏈路加密

時間 2019-11-06

標籤使用 nginx 代理 thrift nio 實現 ssl 鏈路加密欄目 Nginx 简体版

原文原文鏈接

1 目標說明

1.1 調研目的

本次調研主要爲了解決兩個問題：java

thrift提供的SSL API只支持BIO（阻塞式IO），而咱們使用的是NIO API，但願能在不改變IO模型的前提下對鏈路進行加密；
將來系統可能須要對thrift服務進行擴展，採用多個thrift服務進行負載均衡，以提高吞吐量。

結合這兩點，經過調研是否可使用nginx ssl代理來解決。同時熟悉下nginx對tcp代理的配置。nginx

1.2 目標網絡模型

但願達到的目標網絡模型以下：git

1.3 SSL說明

經過對SSL的學習，結合自身業務的考慮，對SSL的使用作以下說明：web

我這裏SSL使用TLSv1，而且服務端不須要校驗客戶端的身份合法性，則使用SSL單向認證方式，只須要服務端證書。另外咱們只須要用到SSL的鏈路加密，因此能夠設置客戶端對服務端證書保持永久信任windows

2 調研步驟

因爲對網絡相關的知識比較欠缺，因此採用以下步驟一一嘗試可行性。先測試nginx對普通tcp的代理，再測試nginx ssl代理在bio 和 nio IO模型下的使用，最後使用nginx ssl代理Thrift NIO。服務器

BIO：同步阻塞IO；NIO：同步非阻塞IO網絡

nginx代理 tcp bio socket server（Server -> BIO，Client -> BIO）；
nginx SSL 代理 tcp bio socket server（Server->BIO, Client -> BIO,SSL）；
nginx SSL 代理 tcp nio socket server（Server->NIO, Client->BIO,SSL）；
nginx SSL 代理 thrift nio server（Server-> thrift NIO, Client->thrift BIO）；

3 調研過程

3.1 nginx安裝

在windows7機器上安裝nginx-1.10.1，其中包括了ngx_stream_core_module模塊，可用於代理TCP協議，nginx具體安裝方法在此不詳述。負載均衡

3.2 nginx代理 tcp bio socket server

3.2.1 nginx配置

worker_processes  1;

events {
    worker_connections  1024;
}

stream {
	server {
		listen 9000;
		proxy_pass localhost:9091;
	}
}

3.2.2 服務端代碼

public class TcpServer {

    private ServerSocket serverSocket = null;


    public void guest (Socket socket) {
        Thread t = new Thread(new ServiceHandler(socket));
        t.start();
    }

    public void start(int port) throws IOException {

        try {
            serverSocket = new ServerSocket(port);
        } catch (IOException e) {
            throw e;
        }

        System.out.println("TCP server start, port -> " + port);

        while (true) {
            guest(serverSocket.accept());
            System.out.println("Guest client");
        }
    }

    class ServiceHandler implements Runnable {

        private Socket socket = null;
        private BufferedReader reader = null;
        private PrintWriter writer = null;

        public ServiceHandler(Socket socket) {
            this.socket = socket;
        }

        public Socket getSocket() {
            return socket;
        }

        @Override
        public void run() {

            try {
                reader = new BufferedReader(new InputStreamReader(this.socket.getInputStream()));
                writer = new PrintWriter(this.socket.getOutputStream());

                String line = null;
                while ((line = reader.readLine()) != null) {

                    if ("close".equals(line)) {
                        break;
                    }

                    System.out.println("c -> " + line);
                    writer.println("Received, t - " + new Date().toString());
                    writer.flush();
                }
            } catch (IOException e) {
                e.printStackTrace();
            } finally {
                if (reader != null) {
                    try {
                        reader.close();
                    } catch (IOException e) {
                    }
                }
                if (writer != null) {
                    writer.close();
                }

                if (this.socket != null) {
                    try {
                        this.socket.close();
                    } catch (IOException e) {
                    }
                }
            }
        }
    }

    public static void main(String[] args) {

        TcpServer tcpServer = new TcpServer();
        try {
            tcpServer.start(9091);
        } catch (IOException e) {
            e.printStackTrace();
        }

    }
}

3.2.3 客戶端代碼

public class TcpClient {

    private Socket socket = null;
    private BufferedReader reader = null;
    private PrintWriter writer = null;

    public void start(int port) throws IOException {
        try {
            socket = new Socket("localhost", port);
            System.out.println("Connected, port -> " + port);
        } catch (IOException e) {
            throw e;
        }

        try {
            reader = new BufferedReader(new InputStreamReader(this.socket.getInputStream()));

            Thread t = new Thread(new TcpReader(reader));
            t.setDaemon(true);
            t.start();

            writer = new PrintWriter(this.socket.getOutputStream());

            Scanner scanner = new Scanner(System.in);

            while (true) {
                System.out.println("input -> ");
                String input = scanner.next();
                writer.println(input);
                writer.flush();
            }
        } catch (IOException e) {
            throw e;
        } finally {
            if (reader != null) {
                try {
                    reader.close();
                } catch (IOException e) {
                }
            }
            if (writer != null) {
                writer.close();
            }

            if (this.socket != null) {
                try {
                    this.socket.close();
                } catch (IOException e) {
                }
            }
        }
    }

    public class TcpReader implements Runnable {

        private BufferedReader reader = null;

        public TcpReader(BufferedReader reader) {
            this.reader = reader;
        }

        @Override
        public void run() {
            String returnLine = null;

            while (true) {
                try {
                    returnLine = reader.readLine();
                    System.out.println(returnLine);
                } catch (IOException e) {
                    e.printStackTrace();
                    break;
                }
            }
        }
    }

    public static void main(String[] args) {
        TcpClient tcpClient = new TcpClient();
        try {
            tcpClient.start(9000);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

服務端開啓TCP監聽9091端口，nginx TCP代理9091端口，並監聽9000端口，客戶端鏈接9000端口，經測試鏈接成功，並可與服務端進行交互。socket

3.3 nginx SSL 代理 tcp bio socket server

3.3.1 nginx配置

worker_processes  1;

events {
    worker_connections  1024;
}

stream {
	server {
		listen 9000 ssl;
		proxy_pass localhost:9091;
		ssl_certificate       D:/server.crt;
		ssl_certificate_key   D:/_server.key;
	}
}

_server.key爲服務器私鑰，server.crt爲服務器證書，經過openssl生成，具體生成方法在此不詳述。tcp

3.3.2 服務器端代碼

同3.2.2

3.3.3 客戶端代碼

import com.spiro.test.net.common.Configuration;

import javax.net.ssl.*;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.security.cert.X509Certificate;
import java.util.Scanner;

/**
 * Created by tz0643 on 2016/6/17.
 */
public class SSLTcpClient {

    private SSLSocket socket = null;
    private BufferedReader reader = null;
    private PrintWriter writer = null;

    public void start(int port) throws Exception {

        // Create a trust manager that does not validate certificate chains
        TrustManager[] trustAllCerts = new TrustManager[]{
                new X509TrustManager() {
                    public java.security.cert.X509Certificate[] getAcceptedIssuers() {
                        return null;
                    }

                    public void checkClientTrusted(X509Certificate[] certs, String authType) {
                    }

                    public void checkServerTrusted(X509Certificate[] certs, String authType) {
                    }
                }
        };

        SSLContext sslContext = SSLContext.getInstance("TLSv1");
        sslContext.init(null, trustAllCerts, null);

        try {
            SSLSocketFactory factory = sslContext.getSocketFactory();
            socket = (SSLSocket) factory.createSocket("192.168.10.188", port);
            System.out.println("Connected, port -> " + port);
        } catch (IOException e) {
            throw e;
        }

        try {
            reader = new BufferedReader(new InputStreamReader(this.socket.getInputStream()));
            Thread t = new Thread(new TcpReader(reader));
            t.setDaemon(true);
            t.start();

            writer = new PrintWriter(this.socket.getOutputStream());

            Scanner scanner = new Scanner(System.in);

            while (true) {
                System.out.println("input -> ");
                String input = scanner.next();
                writer.println(input);
                writer.flush();
            }
        } catch (IOException e) {
            throw e;
        } finally {
            if (reader != null) {
                try {
                    reader.close();
                } catch (IOException e) {
                }
            }
            if (writer != null) {
                writer.close();
            }

            if (this.socket != null) {
                try {
                    this.socket.close();
                } catch (IOException e) {
                }
            }
        }
    }

    public class TcpReader implements Runnable {

        private BufferedReader reader = null;

        public TcpReader(BufferedReader reader) {
            this.reader = reader;
        }

        @Override
        public void run() {
            String returnLine = null;

            while (true) {
                try {
                    returnLine = reader.readLine();
                    System.out.println(returnLine);
                } catch (IOException e) {
                    e.printStackTrace();
                    break;
                }
            }
        }
    }

    public static void main(String[] args) {

        Configuration conf = Configuration.getInstance();

        try {
            conf.init();
        } catch (IOException e) {
            e.printStackTrace();
            System.exit(-1);
        }

        SSLTcpClient tcpClient = new SSLTcpClient();
        try {
            tcpClient.start(9000);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

服務端開啓BIO socket監聽9091端口，nginx TCP SSL代理9091端口，並監聽9000端口，客戶端BIO SSL socket鏈接9000端口，經測試鏈接成功，並可與服務端進行交互。

3.4 nginx SSL 代理 tcp nio socket server

3.4.1 nginx配置

同3.3.1

3.4.2 服務端代碼

import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.SocketChannel;
import java.util.Date;
import java.util.Iterator;

public class NIOServer {

    private Selector selector;

    public void initServer(int port) throws IOException {
        ServerSocketChannel serverChannel = ServerSocketChannel.open();
        serverChannel.configureBlocking(false);
        serverChannel.socket().bind(new InetSocketAddress(port));
        this.selector = Selector.open();
        serverChannel.register(selector, SelectionKey.OP_ACCEPT);
    }

    /**
     * @throws IOException
     */
    public void listen() throws IOException {
        System.out.println("Server started");
        while (true) {
            selector.select();
            Iterator ite = this.selector.selectedKeys().iterator();
            while (ite.hasNext()) {
                SelectionKey key = (SelectionKey) ite.next();
                ite.remove();

                if (key.isAcceptable()) {

                    System.out.println("Accept 1 socket");

                    ServerSocketChannel server = (ServerSocketChannel) key
                            .channel();

                    SocketChannel channel = server.accept();
                    channel.configureBlocking(false);
                    channel.register(this.selector, SelectionKey.OP_READ);

                } else if (key.isReadable()) {
                    read(key);
                }

            }

        }
    }

    public void read(SelectionKey key) throws IOException{
        SocketChannel channel = (SocketChannel) key.channel();
        ByteBuffer buffer = ByteBuffer.allocate(1024);
        channel.read(buffer);
        buffer.flip();
        byte[] data = new byte[buffer.remaining()];
        buffer.get(data);

        System.out.println("c -> "
                + new String(data).trim());

        String msg = "Received, t - " + new Date().toString() + "\n";
        ByteBuffer outBuffer = ByteBuffer.wrap(msg.getBytes());
        channel.write(outBuffer);

    }

    /**
     * @throws IOException
     */
    public static void main(String[] args) throws IOException {
        NIOServer server = new NIOServer();
        server.initServer(9091);
        server.listen();
    }
}

3.4.3 客戶端代碼

同3.3.3

服務端開啓NIO socket監聽9091端口，nginx TCP SSL代理9091端口，並監聽9000端口，客戶端BIO SSL socket鏈接9000端口，經測試鏈接成功，並可與服務端進行交互。

3.5 nginx SSL 代理 thrift nio server

3.5.1 nginx配置

同3.3.1

3.5.2 服務端代碼

public void serve() {
        try {
            TNonblockingServerTransport transport =
                    new TNonblockingServerSocket(port);
            TServer server = new TNonblockingServer(
                    new TNonblockingServer.Args(transport).processor(processor));

            System.out.println("Starting the simple nio server...");
            server.serve();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

查看完整代碼

3.5.3 客戶端代碼

因爲Thrift客戶端API 參數TSSLTransportParameters必須設置trustStore，故必須根據服務端證書生成trust store文件。其實也可本身從新實現TSSLTransportFactory從而達到不須要設置trustStore，即永久信任服務端證書，這裏暫時不實現。

protected void connectAndInvoke() {

        TTransport transport = null;
        try {
            TSSLTransportFactory.TSSLTransportParameters params
                    = new TSSLTransportFactory.TSSLTransportParameters();

            String truststoreFilename = Configuration.getInstance()
                    .getConf("ssl.truststore.filename");
            String truststorePassword = Configuration.getInstance()
                    .getConf("ssl.truststore.password");
            params.setTrustStore(truststoreFilename, truststorePassword, "SunX509", "JKS");

            transport = TSSLTransportFactory.getClientSocket("localhost", 9091, 0, params);
            transport.open();

            TProtocol protocol = new TBinaryProtocol(transport);
            Calculator.Client client = new Calculator.Client(protocol);

            perform(client);
        } catch (TException x) {
            x.printStackTrace();
        } finally {
            if (transport != null) {
                transport.close();
            }
        }
    }

查看完整代碼

服務端開啓NIO thrift服務監聽9091端口，nginx TCP SSL代理9091端口，並監聽9000端口，客戶端使用Thrift SSL API鏈接9000端口，經測試鏈接成功，RPC調用正常。

4 總結

通過調研，thrift服務端仍然使用NIO API，經過nginx ssl tcp代理對鏈路進行加密是可行的。只須要修改客戶端代碼爲 Thrift SSL API，同時這裏客戶端必須爲服務端證書生成trust store 文件，固然經過從新實現TSSLTransportFactory仍是能夠作到不須要這個trust store文件，只對鏈路進行加密不驗證服務端的合法性，這個待後續有時間再研究。

另外nginx ssl tcp代理也可用於進行負載均衡，這個相似對web http代理作負載均衡，這裏不作詳細介紹。