OtterTune配置記錄

0. 準備兩臺Ubuntu 18.04的虛擬機，安裝mysql（供server-side存儲調優數據用）和postgresql（供client-side存儲業務數據用，這裏以PostgreSQL爲例。由於ottertune官方只給了postgres的knob configuration.....）。而後按照官方配置（https://github.com/cmu-db/ottertune/wiki/Linux-Quick-Setup）安裝好必要的包。

Ref: http://www.javashuo.com/article/p-qjhgwbqx-u.html

　　http://www.javashuo.com/article/p-cbkdnyyq-ng.html

　　http://www.javashuo.com/article/p-vaolouwa-db.html

　　http://www.javashuo.com/article/p-nsoqoriw-br.html

　　https://blog.csdn.net/chengyuqiang/article/details/70153980?utm_source=blogxgwz5

 　　安裝配置postgresql：

 1 wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
 2 sudo add-apt-repository "deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main"
 3 sudo apt-get update
 4 sudo apt-get install postgresql-9.6
 5 sudo apt-get remove postgresql-10　　　　#安裝9.6時自動帶上了10...不知道爲啥。要卸載掉防止端口衝突。能夠用sudo lsof -i:5432看一下
 6 
 7 2、修改postgres數據庫用戶的密碼
 8 打開客戶端工具（psql）
 9 sudo -u postgres psql
10 其中，sudo -u postgres 是使用postgres 用戶登陸的意思。PostgreSQL數據默認會建立一個postgres的數據庫用戶做爲數據庫的管理員，密碼是隨機的
11 postgres=# ALTER USER postgres WITH PASSWORD '******';
12 postgres=#爲PostgreSQL下的命令提示符，--注意最後的分號；
13 
14 3、退出PostgreSQL psql客戶端
15 postgres=# \q
16  
17 4、修改ubuntu操做系統的postgres用戶的密碼（密碼要與數據庫用戶postgres的密碼相同）
18 su root
19 刪除PostgreSQL用戶密碼
20 sudo passwd -d postgres
21 passwd -d 是清空指定用戶密碼的意思
22 設置PostgreSQL系統用戶的密碼
23 sudo -u postgres passwd
24 按照提示，輸入兩次新密碼
25  
26 5、修改PostgresSQL數據庫配置實現遠程訪問
27 sudo gedit /etc/postgresql/9.6/main/postgresql.conf
28 1.監放任何地址訪問，修改鏈接權限
29 #listen_addresses = 'localhost' 改成 listen_addresses = '*'
30 2.啓用密碼驗證
31 #password_encryption = on 改成 password_encryption = on
32 3. 確保端口爲5432
33 port = 5432                # (change requires restart)
34 
35 sudo gedit /etc/postgresql/9.6/main/pg_hba.conf
36 在文檔末尾加上如下內容
37 host all all 0.0.0.0/0 trust
38 
39 6、重啓服務
40 /etc/init.d/postgresql restart
41 
42 7、5432端口的防火牆設置
43 5432爲postgreSQL默認的端口
44 sudo iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 5432 -j ACCEPT
45  
46 安裝好以後能夠
47 psql -U postgres -h 192.168.1.170
48 測試一下
49  
50 Ref: https://blog.csdn.net/zj0078/article/details/71156064

View Code

 

修改系統默認python版本

緣由：ubuntu系統安裝了python2.7和python3.6，默認使用的是python 2.7。致使從2.7找不到包
解決辦法：使用如下命令來修改默認python版本：

sudo cp /usr/bin/python /usr/bin/python_bak
sudo rm /usr/bin/python
sudo ln -s /usr/bin/python3.6 /usr/bin/python

這樣在終端中運行Python時，默認啓動的就是3.6版本了

 

若是在C2步驟中出錯，那麼還須要修改默認java版本

sudo update-alternatives --config java

選擇/usr/lib/jvm/jdk1.8.0_212/bin/java 所在的number

 

OtterTune分爲兩部分：server side: 包括一個MySQL數據庫（用於存儲調優數據，供ml model用），Django（FrontEnd User Interface），Celery（用於調度ML task）；client side: Target_DBMS（存儲用戶的業務數據用，支持多種DBMS），Controller（用於控制target DBMS），Driver（用於調用controller，入口是fabfile.py）

Ref: https://github.com/cmu-db/ottertune/wiki/Getting-Started-ottertune

　　https://blog.csdn.net/weixin_40449300/article/details/87904128

　　https://blog.csdn.net/u013385018/article/details/84028360

　　https://blog.csdn.net/u013385018/article/details/84201771 

說實話見到不少帖子都在討論它，但大多都是空泛的複述論文，實際上不多有人真正deploy過......這點仍是挺讓我驚訝的

 

-------------------------------------------Test OtterTune with real target DBMS-------------------------------------------

這裏以postgreSQL爲例。

按照S1, S2, C1, C2, C3, C4, C5, S3, S4, C6的順序執行以下步驟便可

 

-----------------------------------------------------Server-side配置-----------------------------------------------------

S1. 新建名爲ottertune的數據庫

mysqladmin create -u root -p ottertune

 

S2. 編輯配置文件

cd /ottertune/server/website

cp website/settings/credentials_TEMPLATE.py website/settings/credentials.py

並填入USERNAME, PASSWORD等信息，設置DEBUG=True

示例文件：

 1 #
 2 # OtterTune - credentials_TEMPLATE.py
 3 #
 4 # Copyright (c) 2017-18, Carnegie Mellon University Database Group
 5 #
 6 """
 7 Private/custom Django settings for the OtterTune project.
 8 
 9 """
10 # pylint: disable=invalid-name
11 
12 # ==============================================
13 # SECRET KEY CONFIGURATION
14 # ==============================================
15 
16 # SECURITY WARNING: keep the secret key used in production secret!
17 SECRET_KEY = 'ADD ME!!'
18 
19 # ==============================================
20 # DATABASE CONFIGURATION
21 # ==============================================
22 
23 DATABASES = {
24     'default': {
25         'ENGINE': 'django.db.backends.mysql',
26         'NAME': 'ottertune',
27         'USER': 'root',
28         'PASSWORD': '1',
29         'HOST': '',
30         'PORT': '',
31         'OPTIONS': {
32             'init_command': "SET sql_mode='STRICT_TRANS_TABLES',innodb_strict_mode=1",
33         },
34     }
35 }
36 
37 # ==============================================
38 # DEBUG CONFIGURATION
39 # ==============================================
40 
41 # Can override the DEBUG setting here
42 DEBUG = True
43 
44 # ==============================================
45 # MANAGER CONFIGURATION
46 # ==============================================
47 
48 # Admin and managers for this project. These people receive private
49 # site alerts.
50 ADMINS = (
51     # ('Your Name', 'your_email@example.com'),
52 )
53 MANAGERS = ADMINS
54 
55 # ==============================================
56 # GENERAL CONFIGURATION
57 # ==============================================
58 
59 # Hosts/domain names that are valid for this site; required if DEBUG is False
60 # See https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts
61 ALLOWED_HOSTS = []

View Code

 

S3：Set up Django（用於監控）

Firstly, you need to migrate the Django models into the database. （至關於配置Django網站後端，將須要的表放進mysql的OtterTune數據庫內）

python3 manage.py makemigrations website
python3 manage.py migrate

Create the super user

python manage.py createsuperuser

這一步操做完成後，將在mysql的OtterTune數據庫中創建以下表：

值得注意的是website_knobcatalog和website_metriccatalog這兩個表，分別存儲了待觀測的knob和metric名稱。這是dump出來的表內容（以postgresql爲例）

 

S4：Start Celery（用於調度ML task）

OtterTune uses Celery to schedule machine learning tasks. Before staring the celery worker, you may want to start a message broker required by celery. In our case, we use RabbitMq.

sudo rabbitmq-server -detached

Then you can start the celery worker:

python3 manage.py celery worker --loglevel=info --pool=threads

Then you can start Django Server:

python manage.py runserver 0.0.0.0:8000

OtterTune has periodical ML tasks (i.e. knob identification and metrics pruning), which run every time period. You can use celery beat to start the periodical tasks.

python3 manage.py celerybeat --verbosity=2 --loglevel=info

完成後打開http://127.0.0.1:8080，在裏面創建一個tuning session，並記下upload code

 

-----------------------------------------------------Client-side配置-----------------------------------------------------

首先將ottertune、oltpbench都git clone到home/username/下面

C1. 編輯/ottertune/client/driver/driver_config.json，填入target DBMS類型、用戶名密碼等信息，並填寫其中涉及到的全部配置文件

注意

　　1. 對於save_path這一項，要事先創建好對應的文件夾

　　2. 對於upload code這一項，要和S4中分配給的upload code一致

示例文件：

 1 -------------/ottertune/client/driver/driver_config.json-------------
 2 {
 3   "database_type" : "postgres",
 4   "database_name" : "tpcc",
 5   "database_disk": "/dev/sda1",
 6   "database_conf": "/etc/postgresql/9.6/main/postgresql.conf",
 7   "database_save_path": "/home/tidb/ottertune",
 8   "username" : "postgres",
 9   "password" : "******",
10   "oltpbench_home": "/home/tidb/oltpbench",
11   "oltpbench_config": "/home/tidb/oltpbench/config/tpcc_config_postgres.xml",
12   "oltpbench_workload": "tpcc",
13   "oltpbench_log" : "/home/tidb/ottertune/client/driver/oltp.log",
14   "controller_config": "/home/tidb/ottertune/client/controller/config/sample_postgres_config.json",
15   "controller_log" : "/home/tidb/ottertune/client/driver/controller.log",
16   "save_path": "/home/tidb/ottertune/server/results",
17   "upload_url" : "http://127.0.0.1:8000",
18   "upload_code" : "0B58NG1HBQZ4UH24Q00Q",
19   "lhs_knob_path" : "/home/tidb/ottertune/client/driver/knobs/postgres-96.json",
20   "lhs_save_path" : "/home/tidb/ottertune/client/driver/configs"
21 }
22 
23 
24 -------------/home/tidb/ottertune/client/controller/config/sample_postgres_config.json-------------
25 {
26   "database_type" : "postgres",
27   "database_url" : "jdbc:postgresql://localhost:5432/postgres",
28   "username" : "postgres",
29   "password" : "asdfgh",
30   "upload_code" : "DEPRECATED",
31   "upload_url" : "DEPRECATED",
32   "workload_name" : "tpcc"
33 }
34 
35 -------------/oltpbench/config/tpcc_config_postgres.xml-------------
36 <?xml version="1.0"?>
37 <parameters>
38     
39     <!-- Connection details -->
40     <dbtype>postgres</dbtype>
41     <driver>org.postgresql.Driver</driver>
42     <DBUrl>jdbc:postgresql://localhost:5432/tpcc</DBUrl>
43     <username>postgres</username>
44     <password>******</password>
45     <isolation>TRANSACTION_READ_COMMITTED</isolation>
46     
47     <!-- Scale factor is the number of warehouses in TPCC -->
48     <scalefactor>2</scalefactor>
49     
50     <!-- The workload -->
51     <terminals>2</terminals>
52     <works>
53         <work>
54           <time>10</time>
55           <rate>10000</rate>
56           <ratelimited bench="tpcc">true</ratelimited>
57           <weights>45,43,4,4,4</weights>
58         </work>
59     </works>
60     
61     <!-- TPCC specific -->  
62        <transactiontypes>
63         <transactiontype>
64             <name>NewOrder</name>
65         </transactiontype>
66         <transactiontype>
67             <name>Payment</name>
68         </transactiontype>
69         <transactiontype>
70             <name>OrderStatus</name>
71         </transactiontype>
72         <transactiontype>
73             <name>Delivery</name>
74         </transactiontype>
75         <transactiontype>
76             <name>StockLevel</name>
77         </transactiontype>
78        </transactiontypes>    
79 </parameters>

View Code

 

C2. 配置oltpbench（用於週期性在target DBMS上run benchmark）：參考https://github.com/oltpbenchmark/oltpbench/wiki#Environment%20Setup

若是出現錯誤，sudo update-alternatives --config java，選擇/usr/lib/jvm/jdk1.8.0_212/bin/java 所在的number

 

C3. 在target DBMS（postgres）中新建名爲tpcc的數據庫，供oltpbench用

psql -U postgres -h 192.168.1.170

postgres=# create database tpcc owner postgres;

postgres=# grant all privileges on database tpcc to postgres;

 

C4. load oltpbench into target DBMS

./oltpbenchmark -b tpcc -c config/tpcc_config_postgres.xml --create=true --load=true

 

C5. Build controller

　　cd ottertune/client/controller

　　gradle build

 

C6. 運行

運行時發現幾個bug：

　　1.  在ottertune/client/driver/fabfile.py line236, 'Output into file' 須要改成 'Output Raw data into file'

　　2.  loop一次以後發現這玩意把個人postgreSQL的配置（postgresql.conf）改崩潰了......感受要本身魔改一番了...

　　　　目測是ottertune在修改時沒有考慮好數據類型（原版的random_page_cost值是4.0    科學計數法在這裏確定是認不出來的啊）

　　　　

After setting the above three configurations, you can run the loop in the description above. In each loop, it collects target DBMS info, uploads to the server, gets new recommended configuration, installs the config and restarts DBMS. Users can continue to run loops until they are satisfied with the recommended configuration. Functions are defined in the driver file ottertune/client/driver/fabfile.py

fab loop　　　　　　　　　　　　　　 runs one single loop.

fab run_loops:max_iter=10　　　　runs 10 loops. You can set max_iter to change the maximum iterations.

注意在剛開始運行時，由於ML model缺少足夠的訓練數據，因此生成的configuration多是random的：

 

 

 

-------------------------------------------Test OtterTune with fake workload data-------------------------------------------

這裏fake workload data是用data_generator.py本身random生成出來的，不是真的在target DBMS上跑出來的knob-metric samples。

我的感受這個就是用來測試下django interface的顯示效果....實際意義不大

F0：首先完成上面的S1, S2, C1, C2, C3, C4操做

F1：生成一些fake workload data

　　cd /ottertune/server/website/script/controller_simulator/

　　python data_generator.py 2 5

這裏2表示workload數量（模擬在target DBMS跑過的不一樣workload）

5表示對於每一個workload，會被觀測到的訓練樣本數（knob/metric samples. 模擬每一個workload在target DBMS上觀測到的結果）

生成好的數據會被放在generated_data文件夾下

F2：Initialize Django Server，初始化ottertune MySQL數據庫

　　cd /ottertune/server/website fab

　　fab create_test_website

F3：啓動Django

　　fab start_debug_server

F4：上傳fake workload data到Django interface

　　cd /ottertune/server/website/script/controller_simulator/

　　python upload_data.py ./generated_data 0987654321

[其中0987654321是upload code，和下圖網頁上的upload code對應。詳細可參考 https://github.com/cmu-db/ottertune/wiki/Finding-a-Session's-Upload-Code ]

在bash中能夠看到，當上傳好後，系統就會自動運行ML pipeline

 

F5：進入http://192.168.1.172:8000，用user/abcd123登錄，單擊test_project -> tuning_session.

左邊能夠選擇workload和要優化的metric，右邊就能夠看到metric在不一樣knob configuration下的表現。點擊下面具體的Result ID就能夠看到這一metric value對應的knob