docker部署tensorflow serving以及模型替換

Using TensorFlow Serving with Docker

 

1.Ubuntu16.04下安裝docker ce

1-1：卸載舊版本的docker

sudo apt-get remove docker docker-engine docker.io

1-2：安裝可選內核模塊從 Ubuntu 14.04 開始，一部份內核模塊移到了可選內核模塊包 ( linux-image-extra-* ) ，以減小內核軟件包的體積。正常安裝的系統應該會包含可選內核模塊包，而一些裁剪後的            系統 可能會將其精簡掉。 AUFS 內核驅動屬於可選內核模塊的一部分，做爲推薦的 Docker 存儲層驅動，通常建議安裝可選內核模塊包以使用 AUFS 。若是沒有安裝的話能夠使用下面的命令安裝

sudo apt-get update
sudo apt-get install  linux-image-extra-$(uname -r)  linux-image-extra-virtual

1-3：安裝Docker CE因爲 apt 源使用 HTTPS 以確保軟件下載過程當中不被篡改。所以，咱們首先須要添加使用HTTPS 傳輸的軟件包以及 CA 證書。

sudo apt-get update
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

1-4：鑑於國內網絡問題，強烈建議使用國內源，官方源請在註釋中查看。爲了確認所下載軟件包的合法性，須要添加軟件源的 GPG 密鑰

curl -fsSL https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
# 官方源
#curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

1-5：而後，咱們須要向 source.list 中添加 Docker 軟件源

sudo add-apt-repository \
"deb [arch=amd64] https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu \
$(lsb_release -cs) \
stable"

# 官方源
#  sudo add-apt-repository \
# "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
# $(lsb_release -cs) \
# stable

1-6：更新 apt 軟件包緩存，並安裝 docker-ce

sudo apt-get update
sudo apt-get install docker-ce

1-7：啓動docker ce

sudo systemctl enable docker
sudo systemctl start docker

1-8：測試安裝是否正確

sudo docker run hello-world

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
ca4f61b1923c: Pull complete 
Digest: sha256:445b2fe9afea8b4aa0b2f27fe49dd6ad130dfe7a8fd0832be5de99625dad47cd
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://cloud.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

出現上述信息表示安裝正確！！

 

 

2.部署Tensorflow serving：latest-gpu

2-1：部署tensorflow serving：latest-gpu時，除了安裝docker以外，還須要安裝nvidia-docker

　  2-1-1：安裝適用於你係統的NVDIA驅動程序，去官網查詢下載，這裏不作演示

      2-1-2：安裝Tensorflow Model Server的GPU，不然會報錯： 

Cannot assign a device for operation 'a': Operation was explicitly assigned to /device:GPU：0

      2-1-3：安裝tensorflow-model-server

1.環境說明
操做系統：Ubuntu

2.安裝依賴
apt-get install -y python3 build-essential curl libcurl3-dev git libfreetype6-dev libzmq3-dev pkg-config python3-dev python3-numpy python3-pip software-properties-common swig zip zlib1g-dev

3.配置python環境
ln -s /usr/bin/python3 /usr/bin/python
ln -s /usr/bin/pip3 /usr/bin/pip

4.安裝tensorflow-serving-api
pip install tensorflow-serving-api

5.配置tensorflow-model-server倉庫
echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | tee /etc/apt/sources.list.d/tensorflow-serving.list
curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | apt-key add -

6.安裝tensorflow-model-server
apt-get remove tensorflow-model-server
apt-get install tensorflow-model-server

7.安裝執行tensorflow_mode_server命令測試是否安裝成功
tensorflow_model_server

Flags:
    --port=8500                         int32   Port to listen on for gRPC API
    --rest_api_port=0                   int32   Port to listen on for HTTP/REST API. If set to zero HTTP/REST API will not be exported. This port must be different than the one specified in --port.
    --rest_api_num_threads=32           int32   Number of threads for HTTP/REST API processing. If not set, will be auto set based on number of CPUs.
    --rest_api_timeout_in_ms=30000      int32   Timeout for HTTP/REST API calls.
    --enable_batching=false             bool    enable batching
    --batching_parameters_file=""       string  If non-empty, read an ascii BatchingParameters protobuf from the supplied file name and use the contained values instead of the defaults.
    --model_config_file=""              string  If non-empty, read an ascii ModelServerConfig protobuf from the supplied file name, and serve the models in that file. This config file can be used to specify multiple models to serve and other advanced parameters including non-default version policy. (If used, --model_name, --model_base_path are ignored.)
    --model_name="default"              string  name of model (ignored if --model_config_file flag is set
    --model_base_path=""                string  path to export (ignored if --model_config_file flag is set, otherwise required)
    --file_system_poll_wait_seconds=1   int32   interval in seconds between each poll of the file system for new model version
    --flush_filesystem_caches=true      bool    If true (the default), filesystem caches will be flushed after the initial load of all servables, and after each subsequent individual servable reload (if the number of load threads is 1). This reduces memory consumption of the model server, at the potential cost of cache misses if model files are accessed after servables are loaded.
    --tensorflow_session_parallelism=0  int64   Number of threads to use for running a Tensorflow session. Auto-configured by default.Note that this option is ignored if --platform_config_file is non-empty.
    --ssl_config_file=""                string  If non-empty, read an ascii SSLConfig protobuf from the supplied file name and set up a secure gRPC channel
    --platform_config_file=""           string  If non-empty, read an ascii PlatformConfigMap protobuf from the supplied file name, and use that platform config instead of the Tensorflow platform. (If used, --enable_batching is ignored.)
    --per_process_gpu_memory_fraction=0.000000  float   Fraction that each process occupies of the GPU memory space the value is between 0.0 and 1.0 (with 0.0 as the default) If 1.0, the server will allocate all the memory when the server starts, If 0.0, Tensorflow will automatically select a value.
    --saved_model_tags="serve"          string  Comma-separated set of tags corresponding to the meta graph def to load from SavedModel.
    --grpc_channel_arguments=""         string  A comma separated list of arguments to be passed to the grpc server. (e.g. grpc.max_connection_age_ms=2000)
    --enable_model_warmup=true          bool    Enables model warmup, which triggers lazy initializations (such as TF optimizations) at load time, to reduce first request latency.

　　

2-2：GPU綁定官網的一個模型，並使用REST API調用它

        拉取最新的tensorflow的GPU docker鏡像文件：

docker pull tensorflow/serving:latest-gpu

 

2-3：接下來，咱們將使用一個名爲的玩具模型Half Plus Two，它0.5 * x + 2爲x咱們提供的預測值生成。此模型將操做綁定到GPU設備，而且不會在CPU上運行。

mkdir -p /tmp/tfserving
cd /tmp/tfserving
git clone https://github.com/tensorflow/serving

2-4：運行Tensorflow serving容器，並打開REST API端口（8501）

docker run --runtime=nvidia -p 8501:8501 \
  --mount type=bind,\
  source=/tmp/tfserving/serving/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_gpu,\
  target=/models/half_plus_two \
  -e MODEL_NAME=half_plus_two -t tensorflow/serving:latest-gpu &

參數說明：

--mount：   表示要進行掛載
source：    指定要運行部署的模型地址， 也就是掛載的源，這個是在宿主機上的模型目錄
target:     這個是要掛載的目標位置，也就是掛載到docker容器中的哪一個位置，這是docker容器中的目錄
-t:         指定的是掛載到哪一個容器
-p:         指定主機到docker容器的端口映射
docker run: 啓動這個容器並啓動模型服務（這裏是如何同時啓動容器中的模型服務的還不太清楚）
 
綜合解釋：
         將source目錄中的例子模型，掛載到-t指定的docker容器中的target目錄，並啓動

注意：若是執行報錯沒法識別type=bind， 那應該是source的路徑有問題

注意：若是出現這樣的錯誤提示，表示傳入的參數不能有空格：表示target、source等參數不能有空格

2-5：提示：在查詢模型以前，請務必等到看到以下所示的消息，代表服務器已準備好接收請求：

2018-07-27 00:07:20.773693: I tensorflow_serving/model_servers/main.cc:333]
Exporting HTTP/REST API at:localhost:8501 ...

2-6：傳入參數，預測結果

$ curl -d '{"instances": [1.0, 2.0, 5.0]}' \
  -X POST http://localhost:8501/v1/models/half_plus_two:predict

2-7：獲得的返回值：

{ "predictions": [2.5, 3.0, 4.5] }

 

到此爲止，用REST API端口，部署tensorflow serving服務已經完成了，並且已經完成了最基本的預測！

 

 

————————————————————————————————————————————————————————————————————

3.模型替換

3-1：下面就是替換模型，上述的模型是git clone官方的最簡單的模型文件，下面就是須要替換模型，替換爲本身訓練好的pb模型文件！

注意：可能有些人訓練完成以後，保存模型，格式看似爲.pb格式的文件，實際上有些人保存的pb格式的文件是frozen model，啓用替換模型的話，會報錯：

E tensorflow_serving/core/aspired_versions_manager.cc:358] Servable {name: mnist version: 1} cannot be loaded: Not found: Could not find meta graph def matching supplied tags: { serve }. To inspect available tag-sets in the SavedModel, please use the SavedModel CLI: `saved_model_cli`

那是由於tensorflow serving但願讀取的pb模型爲saved model格式，因而須要將frozen model 轉化爲 saved model 格式：

 64 from tensorflow.python.saved_model import signature_constants
 65 from tensorflow.python.saved_model import tag_constants
 66 
 67 export_dir = '/data/codebase/Keyword-fenci/brand_recogniton_biLSTM/saved_model'
 68 graph_pb = '/data/codebase/Keyword-fenci/brand_recogniton_biLSTM/tensorflow_model/tensor_model.pb'
 69 
 70 builder = tf.saved_model.builder.SavedModelBuilder(export_dir)
 71 
 72 with tf.gfile.GFile(graph_pb, "rb") as f:
 73     graph_def = tf.GraphDef()
 74     graph_def.ParseFromString(f.read())
 75 
 76 sigs = {}
 77 
 78 with tf.Session(graph=tf.Graph()) as sess:
 79     # name="" is important to ensure we don't get spurious prefixing
 80     tf.import_graph_def(graph_def, name="")
 81     g = tf.get_default_graph()
 82     inp = g.get_tensor_by_name(net_model.input.name)
 83     out = g.get_tensor_by_name(net_model.output.name)
 84 
 85     sigs[signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY] = \
 86         tf.saved_model.signature_def_utils.predict_signature_def(
 87             {"in": inp}, {"out": out})
 88 
 89     builder.add_meta_graph_and_variables(sess,
 90                                          [tag_constants.SERVING],
 91                                          signature_def_map=sigs)
 92 
 93 builder.save()
--------------------- 
原文：https://blog.csdn.net/mouxiaoqiu/article/details/81220222 

這樣保存下來了以後，saved model文件夾下就有兩個文件：

saved_model.pb 和 variables文件夾（文件夾裏面包含了兩個文件）

這樣saved_model的pb模型文件，tensorflow serving纔可以讀取~~

目錄結構爲：

── 00000123
   ├── saved_model.pb └── variables ├── variables.data-00000-of-00001 └── variables.index

 

3-2：按照上述官方的source目錄，模仿建立相似的目錄結構：

        在saved_model_half_plus_two_gpu模型同級的目錄下建立了一個文件夾，名爲tfmodel， 這是我模型的名稱，而後：

$cd tfmodel
$mkdir 00000123
$cd 00000123
$mkdir variables

將這兩個文件放入variables文件夾內，而後將pb模型放入00000123文件夾內，至關於tfmodel文件夾內包含：1.pb模型 2.variables文件夾(這個文件夾包含兩個權重文件)

 

3-3：找到你建立的tfmodel文件夾，在source裏面替換你的模型路徑：

第一個tf_model爲路徑的文件夾，第二和第三個爲任意起的名字，方便後續訪問服務器的時候須要用着這個名字：

 

3-4：看到這個提示，表示您的服務已經起來了：

 

 到此爲止，用docker部署tensorflow serving以及對模型的替換已經完成了，若有漏洞，但願你們批評指正！