docker高級應用之cpu與內存資源限制（轉）
標籤：docker資源限制 docker cpu限制 docker內存限制 php
以前介紹過如何使用ovs的qos技術對docker容器作網絡資源限制，本次介紹如何使用docker自己的參數進行cpu與內存資源限制。html
默認docker自己提供在建立容器的時候，進行內存、swap、cpu限制，但使用起來就得有些技巧須要注意，具體以下：python
一、內存限制linux
默認docker內存限制可使用-m參數進行限制，但若是僅-m參數，不添加其餘，那麼內存限制不會限制死，好比你-m 256m內存，那麼容器里程序能夠跑到256m*2=512m後纔會被oom給殺死，nginx
緣由是源碼裏memory.memsw.limit_in_bytes 值是被設置成咱們指定的內存參數的兩倍。git
源碼地址是https://github.com/docker/libcontainer/blob/v1.2.0/cgroups/fs/memory.go#L39github
內容以下docker
// By default, MemorySwap is set to twice the size of RAM.
// If you want to omit MemorySwap, set it to `-1‘.
if d.c.MemorySwap != -1 {
if err := writeFile(dir, "memory.memsw.limit_in_bytes", strconv.FormatInt(d.c.Memory*2, 10)); err != nil {
return err
}
固然以上的內容都是有人發過的，不是我原創，地址是https://goldmann.pl/blog/2014/09/11/resource-management-in-docker/shell
下面是介紹一下個人測試過程centos
先限制內存爲100m
docker run --restart always -d -m 100m --name=‘11‘ docker.ops-chukong.com:5000/centos6-base:5.0 /usr/bin/supervisord
而後登錄容器後使用stress進行內存測試
PS：怕你們不知道如何安裝stress，我也簡單介紹一下
rpm -Uvh http://pkgs.repoforge.org/stress/stress-1.0.2-1.el7.rf.x86_64.rpm
這一個命令就安裝完成了
使用stress進行測試
stress --vm 1 --vm-bytes 190M --vm-hang 0
這個命令是跑190mM的內存
下面是跑的結果
10:11:35 # stress --vm 1 --vm-bytes 190M --vm-hang 0 stress: info: [282] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd stress: FAIL: [282] (420) <-- worker 283 got signal 9 stress: WARN: [282] (422) now reaping child worker processes stress: FAIL: [282] (456) failed run completed in 1s
能夠看到程序死了，想查看程序死的緣由能夠去/var/log/messages裏查看
May 22 10:11:06 ip-10-10-125-11 docker: 2015/05/22 10:11:06 http: multiple response.WriteHeader calls
May 22 10:11:45 ip-10-10-125-11 kernel: stress invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 22 10:11:45 ip-10-10-125-11 kernel: stress cpuset=docker-6ae152aa189f2596249a56a2e9c414ae70addc55a2fe495b93e7420fc64758d5.scope mems_allowed=0-1
May 22 10:11:45 ip-10-10-125-11 kernel: CPU: 7 PID: 37242 Comm: stress Not tainted 3.10.0-229.el7.x86_64 #1
May 22 10:11:45 ip-10-10-125-11 kernel: Hardware name: Dell Inc. PowerEdge R720/01XT2D, BIOS 2.0.19 08/29/2013
May 22 10:11:45 ip-10-10-125-11 kernel: ffff88041780a220 0000000019d4767f ffff8807c189b808 ffffffff81604b0a
May 22 10:11:45 ip-10-10-125-11 kernel: ffff8807c189b898 ffffffff815ffaaf 0000000000000046 ffff880418462d80
May 22 10:11:45 ip-10-10-125-11 kernel: ffff8807c189b870 ffff8807c189b868 ffffffff81159fb6 ffff8803f55fc9a8
May 22 10:11:45 ip-10-10-125-11 kernel: Call Trace:
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff81604b0a>] dump_stack+0x19/0x1b
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff815ffaaf>] dump_header+0x8e/0x214
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff81159fb6>] ? find_lock_task_mm+0x56/0xc0
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff8115a44e>] oom_kill_process+0x24e/0x3b0
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff8107bdde>] ? has_capability_noaudit+0x1e/0x30
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff811bd5ae>] __mem_cgroup_try_charge+0xb9e/0xbe0
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff811bdd80>] ? mem_cgroup_charge_common+0xc0/0xc0
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff811bdd19>] mem_cgroup_charge_common+0x59/0xc0
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff811bf96a>] mem_cgroup_cache_charge+0x8a/0xb0
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff811571a2>] __add_to_page_cache_locked+0x52/0x260
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff81157407>] add_to_page_cache_lru+0x37/0xb0
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff812070f5>] mpage_readpages+0xb5/0x160
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffffa020b310>] ? _ext4_get_block+0x1b0/0x1b0 [ext4]
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffffa020b310>] ? _ext4_get_block+0x1b0/0x1b0 [ext4]
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffffa0208f0c>] ext4_readpages+0x3c/0x40 [ext4]
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff8116333c>] __do_page_cache_readahead+0x1cc/0x250
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff81163981>] ra_submit+0x21/0x30
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff81158f1d>] filemap_fault+0x11d/0x430
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff8117e63e>] __do_fault+0x7e/0x520
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff81182995>] handle_mm_fault+0x3d5/0xd70
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff81189396>] ? mmap_region+0x186/0x610
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff8160fe06>] __do_page_fault+0x156/0x540
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff81189b25>] ? do_mmap_pgoff+0x305/0x3c0
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff81173f89>] ? vm_mmap_pgoff+0xb9/0xe0
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff8161020a>] do_page_fault+0x1a/0x70
May 22 10:11:45 ip-10-10-125-11 kernel: [<ffffffff8160c408>] page_fault+0x28/0x30
May 22 10:11:45 ip-10-10-125-11 kernel: Task in /system.slice/docker-6ae152aa189f2596249a56a2e9c414ae70addc55a2fe495b93e7420fc64758d5.scope killed as a result of limit of /system.slice/docker-6ae152aa189f2596249a56a2e9c414ae70addc55a2fe495b93e7420fc64758d5.scope
May 22 10:11:45 ip-10-10-125-11 kernel: memory: usage 101144kB, limit 102400kB, failcnt 10934
May 22 10:11:45 ip-10-10-125-11 kernel: memory+swap: usage 204800kB, limit 204800kB, failcnt 37
May 22 10:11:45 ip-10-10-125-11 kernel: kmem: usage 0kB, limit 9007199254740991kB, failcnt 0
May 22 10:11:45 ip-10-10-125-11 kernel: Memory cgroup stats for /system.slice/docker-6ae152aa189f2596249a56a2e9c414ae70addc55a2fe495b93e7420fc64758d5.scope: cache:80KB rss:101064KB rss_huge:90112KB mapped_file:0KB swap:103656KB inactive_anon:72136KB active_anon:28928KB inactive_file:56KB active_file:24KB unevictable:0KB
May 22 10:11:45 ip-10-10-125-11 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
May 22 10:11:45 ip-10-10-125-11 kernel: [34515] 0 34515 24879 302 54 1947 0 supervisord
May 22 10:11:45 ip-10-10-125-11 kernel: [34531] 0 34531 16674 0 38 179 0 sshd
May 22 10:11:45 ip-10-10-125-11 kernel: [34542] 0 34542 5115 9 14 141 0 crond
May 22 10:11:45 ip-10-10-125-11 kernel: [34544] 0 34544 42841 0 20 101 0 rsyslogd
May 22 10:11:45 ip-10-10-125-11 kernel: [35344] 0 35344 4359 0 11 52 0 anacron
May 22 10:11:45 ip-10-10-125-11 kernel: [36682] 0 36682 2873 38 10 58 0 bash
May 22 10:11:45 ip-10-10-125-11 kernel: [37241] 0 37241 1643 1 8 22 0 stress
May 22 10:11:45 ip-10-10-125-11 kernel: [37242] 0 37242 50284 23693 103 24637 0 stress
May 22 10:11:45 ip-10-10-125-11 kernel: Memory cgroup out of memory: Kill process 37242 (stress) score 945 or sacrifice child
May 22 10:11:45 ip-10-10-125-11 kernel: Killed process 37242 (stress) total-vm:201136kB, anon-rss:94772kB, file-rss:0kB
顯示內存使用超過2倍限制的內存，因此進行oom kill掉stress的進程，但容器不會死，仍然能正常運行。
但若是你想寫死內存，可使用
-m 100m --memory-swap=100m
這樣就直接限制死，只要程序內存使用超過100m，那麼就會被oom給kill。
二、cpu限制
目前cpu限制可使用綁定到具體的線程，或者是在綁定線程基礎上對線程資源權重分配。
綁定線程可使用參數--cpuset-cpus=7（注意1.6版本纔是--cpuset-cpus，1.6版本如下是--cpuset）
分配線程資源使用可使用參數-c
具體的cpu如何使用與測試，參考上面國外朋友寫的文章，很全面。
我這裏注意介紹一下我這裏平臺cpu在容器裏是如何使用？
我這裏平臺是直接把docker弄成iaas平臺，而後在這個iaas基礎上跑paas；其餘的docker雲基本都是在openstack的iaas或者其餘的iaas裏跑docker作paas。
內存使用我是直接限制爲256m、512m、1g，cpu默認都是0.2c。
c 的概念是cpu線程，好比你top的時候，使用1命令能夠直接看到系統裏有多少線程，以16個線程爲例，我保留2個c給宿主機系統使用，另外14個c給 docker使用，一個c有5個容器共同使用，最少可使用20%資源，也就是0.2c，最多可使用100%資源，也就是1c。
這樣好處是線程不忙的時候，容器能夠獨享100%資源，資源緊張的時候，也就保證本身有20%的資源使用，另外同一臺宿主機的容器即便某個資源使用多，也很差太影響其餘c的容器。
我這裏測試環境出現過某個項目，在進行壓力測試的時候，因爲沒有對容器作資源限制，致使測試中把宿主機壓測掛了，致使同臺宿主機全部容器都沒法訪問。
錯誤以下圖
講了一大堆內存與cpu資源限制，基本別人都多多少少的結束過，我在介紹一下我獨有的。
三、模塊化進行cpu、內存、網絡資源限制
可使用先是腳本進行一鍵化的配置下面功能容器：
一、持久化獨立ip；
二、能夠作系統描述符配置；
三、能夠限制內存使用資源；
四、能夠限制網絡使用資源；
五、能夠自動綁定cpu線程；
六、能夠出現故障時自動重啓容器；
七、支持容器使用密碼或者key驗證；
下面是建立容器的過程
一、建立密碼驗證的
內存是256m，網絡資源1m（bit），使用密碼驗證，文件描述符是65535
[root@ip-10-10-125-8 docker_code]# python create_docker_container_multi.py test1 docker.ops-chukong.com:5000/centos6-base:5.0 /usr/bin/supervisord 256m 1 passwd "123123" 65535
{‘Container_id‘: ‘522b5a91838375657c5ebba6f8b43d737dd7290cfad46803ba663a70ef789555‘, ‘Container_gateway_ip‘: u‘172.16.0.14‘, ‘Memory_limit‘: ‘256m‘, ‘Cpu_share‘: ‘3‘, ‘Container_status‘: ‘running‘, ‘Physics_ip‘: ‘10.10.125.8‘, ‘Container_create‘: ‘2015.06.09-10:47:41‘, ‘Container_name‘: ‘test1‘, ‘Cpu_core‘: ‘14‘, ‘Network_limit‘: ‘1‘, ‘Container_gateway_bridge‘: ‘ovs2‘, ‘Container_ip‘: u‘172.16.2.164/16‘}
測試
[root@ip-10-10-125-8 docker_code]# ssh 172.16.2.164
root@172.16.2.164‘s password:
Last login: Fri Mar 13 13:53:37 2015 from 172.17.42.1
root@522b5a918383:~
10:48:56 # ifconfig eth1
eth1 Link encap:Ethernet HWaddr 86:44:D4:F5:5C:01
inet addr:172.16.2.164 Bcast:0.0.0.0 Mask:255.255.0.0
inet6 addr: fe80::8444:d4ff:fef5:5c01/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:96 errors:0 dropped:0 overruns:0 frame:0
TX packets:75 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:12261 (11.9 KiB) TX bytes:10464 (10.2 KiB) root@522b5a918383:~ 10:48:59 # ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 128596 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 65535 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1048576 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
內存的話，使用docker stats模塊查看
[root@ip-10-10-125-8 docker_code]# docker stats test1 CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O test1 0.00% 30.7 MiB/256 MiB 11.99% 0 B/0 B
能夠看到就是256m的內存限制
網絡的話，我經過wget下載iso文件來測試
root@522b5a918383:~ 10:50:20 # wget -S -O /dev/null http://mirror.neu.edu.cn/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1503-01.iso --2015-06-09 10:51:04-- http://mirror.neu.edu.cn/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1503-01.iso Resolving mirror.neu.edu.cn... 202.118.1.64, 2001:da8:9000::64 Connecting to mirror.neu.edu.cn|202.118.1.64|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK Server: nginx/1.0.4 Date: Tue, 09 Jun 2015 02:44:48 GMT Content-Type: application/octet-stream Content-Length: 4310695936 Last-Modified: Wed, 01 Apr 2015 00:05:50 GMT Connection: keep-alive Accept-Ranges: bytes Length: 4310695936 (4.0G) [application/octet-stream] Saving to: 「/dev/null」 0% [ ] 884,760 117K/s eta 9h 54m
能夠看到下載速度是1m（bit）/8=117k（byte）。
在測試一下持久化固定ip，重啓容器
[root@ip-10-10-125-8 docker_code]# docker restart test1
test1
[root@ip-10-10-125-8 docker_code]# docker exec test1 ifconfig
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
能夠看到重啓後容器ip沒了，但我有另外的一個腳本，會在發現容器沒有ip後自動的把ip以及以前的配置從新給予（在crontab裏每分鐘運行一次）。
[root@ip-10-10-125-8 docker_code]# docker exec test1 ifconfig
eth1 Link encap:Ethernet HWaddr B6:6D:BC:CC:CD:69
inet addr:172.16.2.164 Bcast:0.0.0.0 Mask:255.255.0.0
inet6 addr: fe80::b46d:bcff:fecc:cd69/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:648 (648.0 b) TX bytes:690 (690.0 b)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
[root@ip-10-10-125-8 docker_code]#
能夠看到ip從新賦予了，實現了持久化固定ip功能。
下面是建立過程當中使用的腳本.你們參考就行，不必直接複製使用，每一個人的需求都是不同的。
[root@ip-10-10-125-8 docker_code]# cat create_docker_container_multi.py
#!/usr/bin/env python
#-*- coding: utf-8 -*-
#author:Deng Lei
#email: dl528888@gmail.com
import sys
import etcd
import time
import socket, struct, fcntl
from docker import Client
import subprocess
from multiprocessing import cpu_count
#get local host ip
def get_local_ip(iface = ‘em1‘):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sockfd = sock.fileno()
SIOCGIFADDR = 0x8915
ifreq = struct.pack(‘16sH14s‘, iface, socket.AF_INET, ‘\x00‘*14)
try:
res = fcntl.ioctl(sockfd, SIOCGIFADDR, ifreq)
except:
return None
ip = struct.unpack(‘16sH2x4s8x‘, res)[2]
return socket.inet_ntoa(ip)
def docker_container_all():
docker_container=docker_client.containers(all=True)
container_name=[]
container_stop_name=[]
for i in docker_container:
container_name.append(i[‘Names‘])
for b in container_name:
for c in b:
container_stop_name.append(c[1:])
return container_stop_name
def get_now_cpu_core():
#get recycle cpu use in etcd
recycle_cpu_key=‘%s%s/resource_limit/cpu_limit/now_cpu_core_avaiable‘%(docker_etcd_key,local_ip)
try:
all_recycle_cpu_result=eval(etcd_client.read(recycle_cpu_key).value)
except etcd.EtcdKeyNotFound:
all_recycle_cpu_result=‘‘
if len(all_recycle_cpu_result) > 0:
now_core={}
now_core[‘now_core‘]=all_recycle_cpu_result[0].split(‘-‘)[0]
now_core[‘now_core_share‘]=all_recycle_cpu_result[0].split(‘-‘)[1]
all_recycle_cpu_result.remove(all_recycle_cpu_result[0])
etcd_client.write(recycle_cpu_key,all_recycle_cpu_result)
return now_core
else:
#get local core use in etcd
now_core={‘count‘:cpu_core,‘now_core‘:cpu_core-1,‘core_share_limit‘:cpu_core_share_limit,‘now_core_share‘:-1}
try:
now_core_use=eval(etcd_client.read(‘%s%s/resource_limit/cpu_limit/now_cpu_core‘%(docker_etcd_key,local_ip)).value)
except KeyError:
now_core_use=now_core
etcd_client.write(‘%s%s/resource_limit/cpu_limit/now_cpu_core‘%(docker_etcd_key,local_ip),now_core)
time.sleep(1)
except etcd.EtcdKeyNotFound:
now_core_use=now_core
etcd_client.write(‘%s%s/resource_limit/cpu_limit/now_cpu_core‘%(docker_etcd_key,local_ip),now_core)
time.sleep(1)
if int(now_core_use[‘now_core‘]) == int((cpu_core_retain - 1)):
msg={‘status‘:1,‘resource‘:‘cpu‘,‘result‘:‘the avaiable cpu core is less than retain:%s,i can not create new container!‘%cpu_core_retain} print msg sys.exit(1) else: if int(now_core_use[‘now_core_share‘]+1) >= int(now_core_use[‘core_share_limit‘]): now_core_use[‘now_core‘]=now_core_use[‘now_core‘]-1 if int(now_core_use[‘now_core‘]) == int((cpu_core_retain - 1)): msg={‘status‘:1,‘resource‘:‘cpu‘,‘result‘:‘the avaiable cpu core is less than retain:%s,i can not create new container!‘%cpu_core_retain} print msg sys.exit(1) else: now_core_use[‘now_core_share‘]=0 else: now_core_use[‘now_core_share‘]=now_core_use[‘now_core_share‘]+1 etcd_client.write(‘%s%s/resource_limit/cpu_limit/now_cpu_core‘%(docker_etcd_key,local_ip),now_core_use) return now_core_use def get_container_ip(): #get avaiable container ip recycle_ip_key=‘%scontainer/now_ip_avaiable‘%docker_etcd_key try: all_recycle_ip_result=eval(etcd_client.read(recycle_ip_key).value) except etcd.EtcdKeyNotFound: all_recycle_ip_result=‘‘ if len(all_recycle_ip_result) > 0: now_recycle_ip=all_recycle_ip_result[0] all_recycle_ip_result.remove(all_recycle_ip_result[0]) etcd_client.write(recycle_ip_key,all_recycle_ip_result) return now_recycle_ip else: #get new container ip try: now_container_ip=etcd_client.read(‘%scontainer/now_ip‘%(docker_etcd_key)).value except KeyError: now_container_ip=default_container_ip except etcd.EtcdKeyNotFound: now_container_ip=default_container_ip ip=now_container_ip.split(‘/‘)[0] netmask=now_container_ip.split(‘/‘)[1] if int(ip.split(‘.‘)[-1]) <254: new_container_ip=‘.‘.join(ip.split(‘.‘)[0:3])+‘.‘+str(int(ip.split(‘.‘)[-1])+1)+‘/‘+netmask elif int(ip.split(‘.‘)[-1]) >=254 and int(ip.split(‘.‘)[-2]) <254: if int(ip.split(‘.‘)[-1]) ==254: last_ip=‘1‘ else: last_ip=int(ip.split(‘.‘)[-1])+1 new_container_ip=‘.‘.join(ip.split(‘.‘)[0:2])+‘.‘+str(int(ip.split(‘.‘)[-2])+1)+‘.‘+last_ip+‘/‘+netmask elif int(ip.split(‘.‘)[-2]) <=254 and int(ip.split(‘.‘)[-3])<254: if int(ip.split(‘.‘)[-1]) == 254: last_ip=‘1‘ else: last_ip=int(ip.split(‘.‘)[-1])+1 new_container_ip=‘.‘.join(ip.split(‘.‘)[0:2])+‘.‘+str(int(ip.split(‘.‘)[2])+1)+‘.‘+last_ip+‘/‘+netmask elif int(ip.split(‘.‘)[-2]) ==255 and int(ip.split(‘.‘)[-1])==254: print ‘now ip had more than ip pool!‘ sys.exit(1) etcd_client.write(‘%scontainer/now_ip‘%docker_etcd_key,new_container_ip) return new_container_ip if __name__ == "__main__": bridge=‘ovs2‘ default_physics_ip=‘172.16.0.1/16‘ default_container_ip=‘172.16.1.1/16‘ email=‘244979152@qq.com‘ local_ip=get_local_ip(‘ovs1‘) if local_ip is None: local_ip=get_local_ip(‘em1‘) docker_etcd_key=‘/app/docker/‘ try: container=sys.argv[1] if container == "-h" or container == "-help": print ‘Usage: container_name container_image command‘ sys.exit(1) except IndexError: print ‘print input container name!‘ sys.exit(1) try: images=sys.argv[2] except IndexError: print ‘print input images!‘ sys.exit(1) try: daemon_program=sys.argv[3] except IndexError: print ‘please input run image commmand!‘ sys.exit(1) try: memory_limit=sys.argv[4] except IndexError: print ‘please input memory limit where unit = b, k, m or g!‘ sys.exit(1) try: network_limit=sys.argv[5] except IndexError: print ‘please input network limit where unit = M!‘ sys.exit(1) try: ssh_method=sys.argv[6] except IndexError: print ‘please input container ssh auth method by key or passwd!‘ sys.exit(1) try: ssh_auth=sys.argv[7] except IndexError: print ‘please input container ssh auth passwd or key_context!‘ sys.exit(1) try: ulimit=sys.argv[8] except IndexError: ulimit=65535 etcd_client=etcd.Client(host=‘127.0.0.1‘, port=4001) cpu_core=cpu_count() cpu_core_retain=2 cpu_core_share_limit=5 local_dir=sys.path[0] #get local core use in etcd now_core=get_now_cpu_core() docker_client = Client(base_url=‘unix://var/run/docker.sock‘, version=‘1.15‘, timeout=10) docker_container_all_name=docker_container_all() #check input container exist if container in docker_container_all_name: print ‘The container %s is exist!‘%container sys.exit(1) #check pipework software exist check_pipework_status=((subprocess.Popen("which pipework &>>/dev/null && echo 0 || echo 1",shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip(‘\n‘) if check_pipework_status != "0": print ‘I can not find pipework in this host!‘ sys.exit(1) #check ovs bridge check_bridge_status=((subprocess.Popen("ovs-vsctl list-br |grep %s &>>/dev/null && echo 0 || echo 1"%bridge,shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip(‘\n‘) if check_bridge_status !="0": subprocess.Popen("ovs-vsctl add-br %s"%bridge,shell=True,stdout=subprocess.PIPE) #check default physics ip try: default_physics_ip=etcd_client.read(‘%sphysics/now_ip‘%(docker_etcd_key)).value except KeyError: default_physics_ip=default_physics_ip etcd_client.write(‘%sphysics/now_ip‘%docker_etcd_key,default_physics_ip) time.sleep(1) except etcd.EtcdKeyNotFound: default_physics_ip=default_physics_ip etcd_client.write(‘%sphysics/now_ip‘%docker_etcd_key,default_physics_ip) time.sleep(1) #check physics ip try: new_physics_ip=etcd_client.read(‘%sphysics/%s‘%(docker_etcd_key,local_ip)).value except KeyError: now_physics_ip=etcd_client.read(‘%sphysics/now_ip‘%(docker_etcd_key)).value ip=now_physics_ip.split(‘/‘)[0] netmask=now_physics_ip.split(‘/‘)[1] if int(ip.split(‘.‘)[-1]) <254: new_physics_ip=‘.‘.join(ip.split(‘.‘)[0:3])+‘.‘+str(int(ip.split(‘.‘)[-1])+1)+‘/‘+netmask elif int(ip.split(‘.‘)[-1]) >=254: print ‘physics gateway ip more than set up ip pool!‘ sys.exit(1) etcd_client.write(‘%sphysics/now_ip‘%docker_etcd_key,new_physics_ip) etcd_client.write(‘%sphysics/%s‘%(docker_etcd_key,local_ip),new_physics_ip) except etcd.EtcdKeyNotFound: now_physics_ip=etcd_client.read(‘%sphysics/now_ip‘%(docker_etcd_key)).value ip=now_physics_ip.split(‘/‘)[0] netmask=now_physics_ip.split(‘/‘)[1] if int(ip.split(‘.‘)[-1]) <254: new_physics_ip=‘.‘.join(ip.split(‘.‘)[0:3])+‘.‘+str(int(ip.split(‘.‘)[-1])+1)+‘/‘+netmask elif int(ip.split(‘.‘)[-1]) >=254: print ‘physics gateway ip more than set up ip pool!‘ sys.exit(1) etcd_client.write(‘%sphysics/now_ip‘%docker_etcd_key,new_physics_ip) etcd_client.write(‘%sphysics/%s‘%(docker_etcd_key,local_ip),new_physics_ip) #set physics ip #run_command="awk -v value=`ifconfig %s|grep inet|grep -v inet6|awk ‘{print $2}‘|wc -l` ‘BEGIN{if(value<1) {system(‘/sbin/ifconfig %s %s‘)}}‘"%(bridge,bridge,new_physics_ip) set_physics_status=((subprocess.Popen("/sbin/ifconfig %s %s && echo 0 || echo 1"%(bridge,new_physics_ip),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip(‘\n‘) if set_physics_status !="0": print ‘config local physics ip:%s is fail!‘%new_physics_ip sys.exit(1) docker_version=docker_client.version()[‘Version‘][0:3] if float(docker_version) >= 1.6 : cpu_tag=‘--cpuset-cpus‘ container_ulimit="--ulimit nofile=%s:%s"%(ulimit,ulimit) else: cpu_tag=‘--cpuset‘ container_ulimit=‘‘ #create new container get_container_id=((subprocess.Popen("docker run --restart always %s -d -m %s --memory-swap=%s %s=%s --net=‘none‘ --name=‘%s‘ %s %s"%(container_ulimit,memory_limit,memory_limit,cpu_tag,now_core[‘now_core‘],container,images,daemon_program),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip(‘\n‘) check_container_status=((subprocess.Popen("docker inspect %s|grep Id &>>/dev/null && echo 0 || echo 1"%(container),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip(‘\n‘) if check_container_status !="0": print ‘Create container %s is fail!‘%container sys.exit(1) #get container ip new_container_ip=get_container_ip() #check container network new_physics_ip=new_physics_ip.split(‘/‘)[0] set_container_status=((subprocess.Popen("`which pipework` %s %s %s@%s &>>/dev/null && echo 0 || echo 1"%(bridge,container,new_container_ip,new_physics_ip),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip(‘\n‘) if set_container_status != "0": print ‘config container:%s network is fail!‘%container sys.exit(1) #set container network limit set_container_network_limit=((subprocess.Popen("/bin/bash %s/modify_docker_container_network_limit.sh limit %s %s &>>/dev/null && echo 0 || echo 1"%(local_dir,container,network_limit),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip(‘\n‘) if set_container_network_limit != "0": print ‘config container:%s network limit:%s is fail!‘%(container,network_limit) sys.exit(1) #write container network limit to etcd etcd_client.write(‘%s%s/resource_limit/network_limit/%s‘%(docker_etcd_key,local_ip,container),network_limit) #modify container ssh auth if ssh_method == ‘key‘: cmd=‘docker exec %s sh -c "echo \‘%s\‘ >>/root/.ssh/authorized_keys && echo 0 || echo 1"‘%(container,ssh_auth) elif ssh_method == ‘passwd‘: cmd=‘docker exec %s sh -c "echo "%s" | passwd --stdin root &>>/dev/null && echo 0 || echo 1"‘%(container,ssh_auth) check_ssh_status=((subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip(‘\n‘) if check_ssh_status != "0": print ‘config container:%s ssh auth method:%s is fail!‘%(container,ssh_method) delete_cmd=‘docker rm $(docker stop %s)‘%container delete_ssh_status=((subprocess.Popen(delete_cmd,shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip(‘\n‘) sys.exit(1) #write container info to etcd key=‘%s%s/container/%s‘%(docker_etcd_key,local_ip,container) value={‘Physics_ip‘:‘%s‘%local_ip,‘Container_id‘:get_container_id,‘Container_name‘:container,‘Container_ip‘:‘%s‘%new_container_ip,‘Container_gateway_bridge‘:bridge,‘Container_gateway_ip‘:new_physics_ip,‘Container_create‘:time.strftime(‘%Y.%m.%d-%T‘),‘Container_status‘:‘running‘,‘Cpu_core‘:now_core[‘now_core‘],‘Cpu_share‘:now_core[‘now_core_share‘],‘Memory_limit‘:memory_limit,‘Network_limit‘:network_limit} etcd_client.write(key,value) #print ‘create container:%s is success.follow is container info.‘%container print value
這個腳本是建立容器並進行資源限制的
下面腳本是進行網絡資源限制，上面腳本會調用作網絡限制
[root@ip-10-10-125-8 docker_code]# cat modify_docker_container_network_limit.sh
#!/bin/bash
#filename:modify_docker_container_network_limit.sh
#author:Deng Lei
#email:dl528888@gmail.com
op=$1
container=$2
limit=$3 # Mbits/s
if [ -z $1 ] || [ -z $2 ]; then
echo "Usage: operation container_name limit(default:5m)"
echo "Example1: I want limit 5m in the container:test"
echo "The command is: bash `basename $0` limit test 5"
echo "Example2: I want delete network limit in the container:test"
echo "The command is: bash `basename $0` ulimit test"
exit 1
fi
if [ -z $3 ];then
limit=‘5m‘
fi
if [ `docker inspect --format "{{.State.Pid}}" $container &>>/dev/null && echo 0 || echo 1` -eq 1 ];then
echo "no this container:$container"
exit 1
fi
ovs_prefix=‘veth1pl‘
container_id=`docker inspect --format "{{.State.Pid}}" $container`
device_name=`echo ${ovs_prefix}${container_id}`
docker_etcd_key=‘/app/docker/‘
local_ip=`ifconfig ovs1|grep inet|awk ‘{print $2}‘|head -n1`
if [ $op == ‘limit‘ ];then
for v in $device_name; do
ovs-vsctl set interface $v ingress_policing_rate=$((limit*1000))
ovs-vsctl set interface $v ingress_policing_burst=$((limit*100))
ovs-vsctl set port $v qos=@newqos -- --id=@newqos create qos type=linux-htb queues=0=@q0 other-config:max-rate=$((limit*1000000)) -- --id=@q0 create queue other-config:min-rate=$((limit*1000000)) other-config:max-rate=$((limit*1000000)) &>>/dev/null && echo ‘0‘ || echo ‘1‘
done
#save result to etcd
`which etcdctl` set ${docker_etcd_key}${local_ip}/resource_limit/network_limit/${container} $limit &>>/dev/null
elif [ $op == ‘ulimit‘ ];then
for v in $device_name; do
ovs-vsctl set interface $v ingress_policing_rate=0
ovs-vsctl set interface $v ingress_policing_burst=0
ovs-vsctl clear Port $v qos &>>/dev/null && echo ‘0‘ || echo ‘1‘
done
`which etcdctl` rm ${docker_etcd_key}${local_ip}/resource_limit/network_limit/${container} &>>/dev/null
fi
相關文章
- 1. docker高級應用之cpu與內存資源限制（轉）
- 2. docker高級應用之cpu與內存資源限制
- 3. docker 內存資源限制
- 4. docker高級應用之資源監控
- 5. Docker容器：磁盤&內存&CPU資源限制實戰
- 6. docker-compose限制內存, cpu資源以及固定ip寫法
- 7. docker存儲資源限制
- 8. kubernetes(k8s)：容器資源限制（內存限制、cpu限制、namespace限制）
- 9. Docker容器CPU、memory資源限制
- 10. docker的memory和cpu資源限制
- 更多相關文章...
- • Docker 資源彙總 - Docker教程
- • Hibernate的級聯與反轉 - Hibernate教程
- • Docker容器實戰(六) - 容器的隔離與限制
- • 使用阿里雲OSS+CDN部署前端頁面與加速靜態資源
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
歡迎關注本站公眾號,獲取更多信息
