最近在測試OpenStack Havana RDO版本的時候,發現過一段時間後,整個平臺會變更的愈來愈慢。剛剛開始還沒什麼動力去trouble shooting,畢竟就一demo的環境。昨天竟然給讓我沒法訪問,這就不能忍了,得看看是什麼臭蟲在搗鬼。mysql
無心中發如今Nova API調用的過程中,在向keystone獲取Token的時間會異常的長。而當我剛剛安裝完keystone的時候,整個速度仍是很不錯的。查閱了相關的資料,發現Token這塊在數據庫永久存放,並且增加速度仍是很快,對keystone的數據庫的表分析以下:sql
切換到MySQL自帶管理庫information_schema數據庫
mysql> use information_schema; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed
查看Token的大小:後端
mysql> select data_length,index_length from tables where table_schema='keystone' -> and table_name='token'; +-------------+--------------+ | data_length | index_length | +-------------+--------------+ | 916455424 | 17924096 | +-------------+--------------+ 1 row in set (0.01 sec)
用比較直觀的方式查看:bash
mysql> select concat(round(sum(data_length/1024/1024),2),'MB') as data_length_MB, -> concat(round(sum(index_length/1024/1024),2),'MB') as index_length_MB -> from tables where -> table_schema='keystone' -> and table_name='token'; +----------------+-----------------+ | data_length_MB | index_length_MB | +----------------+-----------------+ | 874.00MB | 17.09MB | +----------------+-----------------+ 1 row in set (0.01 sec)
大概兩週的時間,竟然有近1G的數據,固然,1G對於數據庫來講也是小case,但若是是一年,這個數據量也是不容忽視的。服務器
在看一下keystone對於token的配置keystone.conf
,主要有幾部分:ide
[token] # Provides token persistence. driver = keystone.token.backends.sql.Token # driver = keystone.token.backends.memcache.Token # Controls the token construction, validation, and revocation operations. # Core providers are keystone.token.providers.[pki|uuid].Provider # provider = # Amount of time a token should remain valid (in seconds) expiration = 86400
從上述的配置能夠token默認存放在數據庫中,默認過時時間爲爲一天。固然,在launchpad上有不少朋友提到用腳本清除數據庫便可。memcached
這邊的話,也借鑑了一下他們的經驗去解決這個問題,下面是清除token表的腳本。性能
#!/bin/bash mysql_user= mysql_password= mysql_host= #這裏你們能夠考慮一下清除數據的時間的問題,爲啥是2天 mysql -u${mysql_user} -p${mysql_password} -h${mysql_host} -e 'USE keystone ; DELETE FROM token WHERE NOT DATE_SUB(CURDATE(),INTERVAL 2 DAY) <= expires;'
固然,若是你在清除token的時候不幸遇到以下的錯誤:測試
ERROR 1205 (HY000) at line 1: Lock wait timeout exceeded; try restarting transaction
只須要增長innodb_lock_wait_timeout
的值:
mysql> show variables like 'innodb_lock_wait_timeout'; +--------------------------+-------+ | Variable_name | Value | +--------------------------+-------+ | innodb_lock_wait_timeout | 50 | +--------------------------+-------+ 1 row in set (0.00 sec) mysql> set innodb_lock_wait_timeout=100; Query OK, 0 rows affected (0.00 sec) mysql> show variables like 'innodb_lock_wait_timeout'; +--------------------------+-------+ | Variable_name | Value | +--------------------------+-------+ | innodb_lock_wait_timeout | 100 | +--------------------------+-------+
從新運行上面的腳本便可。
固然,兄弟們不能每次本身手動去服務器上執行這個腳本,採用crontab+腳本的方式,腳本clearToken.sh
。
#!/bin/bash mysql_user=keystone mysql_password=******** mysql_host= mysql=$(which mysql) logger -t keystone-cleaner "Starting Keystone 'token' table cleanup" logger -t keystone-cleaner "Starting token cleanup" mysql -u${mysql_user} -p${mysql_password} -h${mysql_host} -e 'USE keystone ; DELETE FROM token WHERE NOT DATE_SUB(CURDATE(),INTERVAL 2 DAY) <= expires;' valid_token=$($mysql -u${mysql_user} -p${mysql_password} -h${mysql_host} -e 'USE keystone ; SELECT * FROM token;' | wc -l) logger -t keystone-cleaner "Finishing token cleanup, there is still $valid_token valid tokens..." exit 0
crontab的定時任務:
[root@controller01 keystone]# cat /etc/crontab SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root HOME=/ # For details see man 4 crontabs # Example of job definition: # .---------------- minute (0 - 59) # | .------------- hour (0 - 23) # | | .---------- day of month (1 - 31) # | | | .------- month (1 - 12) OR jan,feb,mar,apr ... # | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat # | | | | | # * * * * * user-name command to be executed 0 1 * * * /opt/clearToken.sh
爲了給MySQL的token表瘦身,能夠後端採用Memcached做爲Token的存儲後端。但採用Memcached做爲後端的存儲。但Memcached這塊,目前仍是有些問題:
Token在Memcached中是否永久存儲?
當Memcached宕機時,Cache沒法持久化
Memcached採用集羣複製,如何去作?
Memcached效率如何?
安裝Memcached服務:
yum -y install memcached
修改keystone.conf
配置文件:
[token] driver = keystone.token.backends.memcache.Token
重啓服務:
service memcached restart service openstack-keystone restart
檢查是否work:
[root@controller01 ~]# lsof -i :11211 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME keystone- 6658 keystone 10u IPv4 20381693 0t0 TCP localhost:41865->localhost:memcache (ESTABLISHED) keystone- 6658 keystone 12u IPv4 19947698 0t0 TCP localhost:33995->localhost:memcache (ESTABLISHED) keystone- 6658 keystone 14u IPv4 20140990 0t0 TCP localhost:37472->localhost:memcache (ESTABLISHED) keystone- 6658 keystone 20u IPv4 20333522 0t0 TCP localhost:40973->localhost:memcache (ESTABLISHED) memcached 28336 memcached 126u IPv4 3050900 0t0 TCP *:memcache (LISTEN) memcached 28336 memcached 127u IPv4 3050902 0t0 UDP *:memcache memcached 28336 memcached 128u IPv4 20381696 0t0 TCP localhost:memcache->localhost:41865 (ESTABLISHED) memcached 28336 memcached 129u IPv4 19947701 0t0 TCP localhost:memcache->localhost:33995 (ESTABLISHED) memcached 28336 memcached 130u IPv4 20140993 0t0 TCP localhost:memcache->localhost:37472 (ESTABLISHED) memcached 28336 memcached 133u IPv4 20333525 0t0 TCP localhost:memcache->localhost:40973 (ESTABLISHED)
能夠看到keystone已經和memcached創建了TCP鏈接。
能夠用telnet查看memcached中的token的信息:
root@controller01 ~]# telnet 127.0.0.1 11211 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. stats items STAT items:9:number 1 STAT items:9:age 19766 STAT items:9:evicted 0 STAT items:9:evicted_nonzero 0 STAT items:9:evicted_time 0 STAT items:9:outofmemory 0 STAT items:9:tailrepairs 0 STAT items:11:number 62 STAT items:11:age 81982 STAT items:11:evicted 0 STAT items:11:evicted_nonzero 0 STAT items:11:evicted_time 0 STAT items:11:outofmemory 0 STAT items:11:tailrepairs 0 STAT items:15:number 1 STAT items:15:age 20365 STAT items:15:evicted 0 STAT items:15:evicted_nonzero 0 STAT items:15:evicted_time 0 STAT items:15:outofmemory 0 STAT items:15:tailrepairs 0 STAT items:17:number 1 STAT items:17:age 20287 STAT items:17:evicted 0 STAT items:17:evicted_nonzero 0 STAT items:17:evicted_time 0 STAT items:17:outofmemory 0 STAT items:17:tailrepairs 0 STAT items:18:number 8558 STAT items:18:age 7675 STAT items:18:evicted 0 STAT items:18:evicted_nonzero 0 STAT items:18:evicted_time 0 STAT items:18:outofmemory 0 STAT items:18:tailrepairs 0 STAT items:22:number 1 STAT items:22:age 81605 STAT items:22:evicted 0 STAT items:22:evicted_nonzero 0 STAT items:22:evicted_time 0 STAT items:22:outofmemory 0 STAT items:22:tailrepairs 0 STAT items:28:number 1 STAT items:28:age 81982 STAT items:28:evicted 0 STAT items:28:evicted_nonzero 0 STAT items:28:evicted_time 0 STAT items:28:outofmemory 0 STAT items:28:tailrepairs 0 STAT items:36:number 1 STAT items:36:age 82020 STAT items:36:evicted 0 STAT items:36:evicted_nonzero 0 STAT items:36:evicted_time 0 STAT items:36:outofmemory 0 STAT items:36:tailrepairs 0 END stats cachedump 18 100 ITEM token-9be5243e6765408bb67cd73f66367cdd [3827 b; 1387340643 s] ITEM token-499d93bec0a54ddebdc28ef0941ba7df [3827 b; 1387340702 s] ITEM token-186a1394b2474c998dafc752fd80d058 [3827 b; 1387340700 s] ITEM token-3de99c3d4f174d15a1546ce321d1e3a1 [3824 b; 1387340576 s] ITEM token-fcab58d23ff94369b27a8dc335340028 [3824 b; 1387340697 s] ITEM token-7092f681ebe34cae9830eba8ae7d45f3 [3824 b; 1387340636 s] ITEM token-977cf2ab85e54920b3d3b582dabba7d8 [3824 b; 1387340634 s] ITEM token-49df94c6ef7a4a1582c7539f5829b354 [3824 b; 1387340567 s] ITEM token-7b1f9646734542d5a30dd0cbc8d1937d [3824 b; 1387340569 s] ……
能夠看到,目前items:18存放的是token信息。
你能夠看到,關於keystone其實在實際用過的過程中會遇到不少的問題,這些問題怎麼去處理將是在生產環境是否能真正用好的關鍵。目前的OpenStack不應處於實驗室的產品,應該更向生產環境邁進,那後面咱們該作什麼,怎麼去作?
對於keystone的performance的問題,其實社區也在討論這塊的東西,如何去作performance的測試,怎麼去tunning,這都是一個話題,後面會針對於keystone這塊如何去測試,若是調整性能作一些分析,但願對你們有幫助。