cmdb事件推送實現zabbix資產同步
前言
《事件推送網關：讓cmdb告別「花瓶」》 一文咱們將cmdb事件推送的參數進行了解析，並結合redis作了去重，避免事件的重複推送。接下來的任務咱們計劃是cmdb和zabbix進行資產同步，功能以下：php
- cmdb的業務、集羣、模塊分組信息同步至zabbix主機分組；
- cmdb資產刪除同步至zabbix進行相應的主機刪除；
- cmdb資產新建同步至zabbix，並根據相應的信息綁定不一樣的模板；
此時cmdb只是爲zabbix提供了資產新建、分組、變動等基礎數據，zabbix的告警信息如何與其關聯，讓運維可以經過告警內容就能直接知道哪塊業務出現問題纔是咱們真正想要的。python
爲此，咱們作了以下定義：nginx
- 定義zabbix的分組規則爲：業務_集羣_模塊；
- zabbix agent的Hostname參數統一使用ip地址，方便事件推送網關經過zabbix api根據進行查詢及其餘相關操做；
- 定義zabbix 主機的可見名稱規則爲：集羣_模塊_ip；因爲咱們的集羣基本都放在同一個業務下，可見名稱規則中不加業務所屬；
經過以上規則，咱們能夠將告警主機的ip、分組、可見名稱自由添加到告警動做中，實現了告警信息的可讀性。redis
其中：django
- 集羣：中間件；
- 模塊：nginx-lan；
- IP：10.164.193.189；
經過告警內容，就能夠快速定位問題，後期可結合故障自愈實現，更好的提升咱們處理問題的效率。json
目錄結構
開發過程當中隨着功能的不斷增多，咱們須要對不一樣狀況生成相應格式化參數。vim
D:\WORK\BLUEKING\GATEWAY\GATEWAY
│ manage.py
│
├─.vscode
│ settings.json
│
├─gateway
│ asgi.py
│ settings.py
│ urls.py
│ wsgi.py
│ __init__.py
│
└─gw_cmdb
│ admin.py
│ apps.py
│ models.py
│ tests.py
│ urls.py
│ views.py
│ __init__.py
│
├─common
│ hostidentifier_cmdb.py
│ main.py
│ module_cmdb.py
│ select_cmdb.py
└─zabbix
group.py
host.py
main.py
template.py
複製代碼
其中： common目錄是與cmdb相關的模塊：api
- main 接收事件推送網關推送過來的參數；
- hostidentifier_cmdb 針對主機相關事件推送返回格式化參數；
- module_cmdb 針對模塊相關事件推送返回格式化參數；
- select_cmdb 查詢cmdb內容如集羣、業務、操做系統等輔助信息；
zabbix目錄是與zabbix相關的模塊：markdown
- main 解析格式化參數，對zabbix作相應的處理；
- group 與主機分組操做相關的模塊，如查詢、建立、刪除、更新等；
- host 與主機操做相關模塊，如查詢、建立、更新、刪除等；
- template 與模板操做相關的模塊，如查詢等；
因爲分組同步能夠在不影響zabbix使用的狀況下操做，所以咱們在此着重介紹此功能。架構
分組同步
分組同步主要有如下幾個功能需求：
- 業務、集羣、模塊分組同步，如：建立、更新、刪除，對應分組名爲：業務_集羣_模塊；
- 主機模塊更新同步，主機模塊轉移須要在zabbix中修改主機相應的分組，保持和cmdb一致；
注意：分組同步的前提是經過zabbix的IP查詢並操做主機，所以要求zabbix agent 的Hostname必須爲IP，而且和cmdb中的IP一一對應。
經過分組同步最終實現的效果以下：
1.接收cmdb推送參數
cmdb事件推送將參數發送給網關，由views.py接收
vim gw_cmdb/views.py
from django.http import HttpResponse
from .common.main import main
from .zabbix.main import zabbix_main
from .jumpserver.main import jumpserver_main
import json
import logging
logger = logging.getLogger('log')
# Create your views here
def cmdb_request(request):
if request.method == 'POST':
data = json.loads(request.body)
logger.info('cmdb發送消息：{}'.format(data))
## 整理cmdb數據
res=main(data)
##是否須要聯動zabbix及jumpserver
if res['result'] == 1:
return HttpResponse("ok")
else:
logger.info(res)
#zabbix同步
zabbix_request = zabbix_main(res)
logger.info('zabbix 同步完畢:{}'.format(zabbix_request))
#jumpserver同步
return HttpResponse("ok")
else:
logger.info('本接口只支持POST模式')
return HttpResponse("本接口只支持POST模式")
複製代碼
因爲cmdb推送過來的參數內容過多，所以咱們須要將其進行格式化而且將其解析成不一樣的動做，以便zabbix進行操做。
2.解析參數
（1）common將views接收的請求進行
vim common/main.py
import logging
from .hostidentifier_cmdb import hostidentifier
from .module_cmdb import module_action
logger = logging.getLogger('log')
def main(data):
result = {'result': 1,'data': 0}
## 模塊操做
if data['obj_type'] == 'module':
return module_action(data)
## 主機標識操做
elif data['obj_type'] == 'hostidentifier':
if data['action'] == 'update' and data['event_type'] == 'relation' :
logger.info("主機標識更新: {}".format(data))
return hostidentifier(data)
else:
logger.info("主機標識未知操做: {}".format(data))
else:
logger.info("未知操做: {}".format(data))
return result
複製代碼
注意：cmdb側事件推送的觸發動做比較多，如主機、業務、組織架構等，但其實在zabbix側咱們沒有必要一一和以上事件進行對應，而是隻須要和最小操做保持一致便可。
舉例：
-
根據分組規則：業務_集羣_模塊，最小級的模塊存在的前提確定是有集羣和業務，所以只要最小級的模塊建立、更新、刪除時，觸發zabbix動做建立主機羣組便可，而空業務、空集羣對zabbix操做沒有實際意義。
-
對於錄入cmdb的主機，不管是在資源池或業務下的空閒資源池，此時都沒有必要觸發zabbix操做，而是當主機轉移到相應的模塊下，標識已經分配使用了，此時咱們將觸發zabbix操做，如建立主機、綁模板、增長主機分組等操做。
所以zabbix側動做觸發，只須要在模塊和主機標識更新狀況下觸發便可。
（2）模塊操做解析
vim common/module_cmdb.py
import json
import logging
from .select_cmdb import select_cmdb_biz,select_cmdb_set
logger = logging.getLogger('log')
def module_action(data):
logger.info("模塊更新: {}".format(data))
##定義數據格式
datajson= {'action': '','obj_type': '','data': {'cur_data': {'group': ''},'pre_data': {'group': ''}},'result': 1}
if data['action'] == "create":
pass
elif data['action'] == "update":
datajson['obj_type']=data['obj_type']
datajson['action']= data['action']
groupname_biz_name = select_cmdb_biz(data['data'][0]['cur_data']['bk_biz_id'])
groupname_set_name = select_cmdb_set(data['data'][0]['cur_data']['bk_biz_id'], data['data'][0]['cur_data']['bk_set_id'])
groupname_module_new_name = data['data'][0]['cur_data']['bk_module_name']
groupname_module_old_name = data['data'][0]['pre_data']['bk_module_name']
logger.info("業務id:{},集羣id:{},模塊新名:{},模塊舊名:{}".format(groupname_biz_name,groupname_set_name,groupname_module_new_name,groupname_module_old_name))
if groupname_module_new_name != groupname_module_old_name:
datajson['data']['cur_data']['group']= groupname_biz_name+"_"+groupname_set_name+"_"+groupname_module_new_name
datajson['data']['pre_data']['group']= groupname_biz_name+"_"+groupname_set_name+"_"+groupname_module_old_name
result = {
'result': 0,
'data': datajson
}
return result
elif data['action'] == "delete":
datajson['obj_type']=data['obj_type']
datajson['action']= data['action']
groupname_biz_name = select_cmdb_biz(data['data'][0]['pre_data']['bk_biz_id'])
groupname_set_name = select_cmdb_set(data['data'][0]['pre_data']['bk_biz_id'], data['data'][0]['pre_data']['bk_set_id'])
groupname_module_name = data['data'][0]['pre_data']['bk_module_name']
datajson['data']['pre_data']['group']= groupname_biz_name+"_"+groupname_set_name+"_"+groupname_module_name
result = {
'result': 0,
'data': datajson
}
return result
else:
pass
return datajson
複製代碼
（3）主機標識解析
cmdb主機模塊轉移可能會觸發屢次請求，所以咱們須要藉助redis將請求進行去重。
vim common/hostidentifier_cmdb.py
import redis
import json
import hashlib
import logging
logger = logging.getLogger('log')
r = redis.StrictRedis(host='127.0.0.1',port=6379,db=1)
## 模塊變動獲取主機全部模板定製分組
def hostidentifier(data):
##定義數據格式
datajson= {'tex_id': '','action': data['action'],'obj_type': data['obj_type'],'data': {'cur_data': {'ip': '','group': []},'bk_host_id':data['data']['cur_data']['bk_host_id'],'pre_data': 'None'},'result': 1}
## 獲取主機組信息，並清理記錄reids去除重複會話
for i in data['data']:
datajson['data']['cur_data']['ip'] = i['cur_data']['bk_host_innerip']
grouplist = i['cur_data']['associations']
for j in grouplist:
groupname = grouplist[j]['bk_biz_name']+"_"+grouplist[j]['bk_set_name']+"_"+grouplist[j]['bk_module_name']
datajson['data']['cur_data']['group'].append(groupname)
datajson['tex_id']= hashlib.md5((data['request_id']+ i['cur_data']['bk_host_innerip']).encode('utf-8')).hexdigest()
rkey = r.hget('cmdb',datajson['tex_id'])
logger.info(rkey)
if rkey is None:
r.hset('cmdb',datajson['tex_id'],json.dumps(datajson['data']))
datajson['result'] = 0
logger.info(datajson)
return datajson
複製代碼
3.zabbix操做
當common將cmdb事件的參數解析後，就可傳給zabbix模塊進行相應的分組同步操做了。
（1）zabbix操做入口
vim zabbix/main.py
import json
import logging
from django.conf import settings
from urllib import request, parse
from .group import select_zabbix_group,create_zabbix_group,main_zabbix_group, update_zabbix_group, delete_zabbix_group
from .host import select_zabbix_host, select_zabbix_host_group,update_zabbix_host_group
logger = logging.getLogger('log')
def zabbix_auth(ZABBIX_USERNAME,ZABBIX_PASSWORD,ZABBIX_URL):
url = "{}/api_jsonrpc.php".format(ZABBIX_URL)
header = {"Content-Type": "application/json"}
# auth user and password
data = {
"jsonrpc": "2.0",
"method": "user.login",
"params": {
"user": ZABBIX_USERNAME,
"password": ZABBIX_PASSWORD
},
"id": 1,
}
# 因爲API接收的是json字符串，故須要轉化一下
value = json.dumps(data).encode('utf-8')
# 對請求進行包裝
req = request.Request(url, headers=header, data=value)
# 驗證並獲取Auth ID
try:
# 打開包裝過的url
result = request.urlopen(req)
except Exception as e:
logger.error("Auth Failed, Please Check Your Name And Password:", e)
else:
response = result.read()
# 上面獲取的是bytes類型數據，故須要decode轉化成字符串
page = response.decode('utf-8')
# 將此json字符串轉化爲python字典
page = json.loads(page)
result.close()
logger.info("Auth Successful. The Auth ID Is: {}".format(page.get('result')))
return page.get('result')
def zabbix_main(result):
# 獲取zabbix 認證
auth_code = zabbix_auth(settings.ZABBIX_USERNAME,settings.ZABBIX_PASSWORD,settings.ZABBIX_URL)
logger.info("zabbix受權id:{}".format(auth_code))
# 模塊操做
if result['obj_type'] == 'module':
#####zabbix舊組是否存在，不存在直接建立新組
result_data = select_zabbix_group(result['data']['pre_data']['group'],auth_code,settings.ZABBIX_URL)
###zabbix建立
if result['action'] == 'create' :
logger.info("zabbix組建立: {}".format(result))
###zabbix修改
elif result['action'] == 'update' and result['data']['pre_data'] != None:
if len(result_data) == 0:
logger.info("zabbix組舊模塊不存在: {}".format(result['data']['pre_data']))
res = create_zabbix_group(result['data']['cur_data']['group'],auth_code,settings.ZABBIX_URL)
if "err" in res:
logger.error('zabbix新增長組失敗:{}'.format(result['data']['cur_data']))
else:
logger.info('zabbix新增長組成功:{}'.format(result['data']['cur_data']['group']))
return result['data']['cur_data']['group']
else:
res = update_zabbix_group(result_data[0]['groupid'],result['data']['cur_data']['group'],auth_code,settings.ZABBIX_URL)
logger.info('zabbix組修改完畢:{}'.format(result['data']['cur_data']['group']))
return result['data']['cur_data']['group']
###模塊刪除
elif result['action'] == 'delete':
logger.info("zabbix組刪除: {}".format(result))
result_data = select_zabbix_group(result['data']['pre_data']['group'],auth_code,settings.ZABBIX_URL)
if len(result_data) == 0:
logger.info("zabbix組舊模塊不存在: {}".format(result['data']['pre_data']))
else:
res = delete_zabbix_group(result_data[0]['groupid'],auth_code,settings.ZABBIX_URL)
logger.info("zabbix組刪除完畢: {}".format(result))
###模塊其餘操做
else:
logger.info("模塊未知操做: {}".format(result))
## 主機標識操做
elif result['obj_type'] == 'hostidentifier':
if result['action'] == 'update':
logger.info("主機標識更新: {}".format(result))
result_Groupidlist = main_zabbix_group(result,auth_code)
if result_Groupidlist['result'] == 0:
res = update_zabbix_host_group(result_Groupidlist['hostid'],result_Groupidlist['grouplistid'],auth_code,settings.ZABBIX_URL)
logger.info('數據同步成功：{}'.format(res))
else:
logger.info("主機標識未知操做: {}".format(result))
else:
logger.info("未知操做: {}".format(result))
return result
複製代碼
（2）主機羣組操做
vim zabbix/group.py
import json
import logging
from urllib import request, parse
from django.conf import settings
from .template import select_zabbix_template
from .host import create_zabbix_host, select_zabbix_host
from common.select_cmdb import select_cmdb_host
logger = logging.getLogger('log')
def select_zabbix_group(Groupname,auth_code,ZABBIX_URL):
url = "{}/api_jsonrpc.php".format(ZABBIX_URL)
header = {"Content-Type": "application/json"}
data = {
"jsonrpc": "2.0",
"method": "hostgroup.get",
"params": {
"output": "extend",
"filter": {
"name": Groupname
}
},
"auth": auth_code,
"id": 1
}
value = json.dumps(data).encode('utf-8')
req = request.Request(url, headers=header, data=value)
try:
# 打開包裝過的url
result = request.urlopen(req)
except Exception as e:
logger.error("result封裝：", e)
else:
response = json.loads(result.read().decode('utf-8'))
result.close()
logger.info("group_result: {}".format(response['result']))
return response['result']
def create_zabbix_group(Groupname,auth_code,ZABBIX_URL):
url = "{}/api_jsonrpc.php".format(ZABBIX_URL)
header = {"Content-Type": "application/json"}
data = {
"jsonrpc": "2.0",
"method": "hostgroup.create",
"params": {
"name": Groupname
},
"auth": auth_code,
"id": 1
}
value = json.dumps(data).encode('utf-8')
logger.info("zabbix組建立信息: {}".format(value))
req = request.Request(url, headers=header, data=value)
try:
# 打開包裝過的url
result = request.urlopen(req)
except Exception as e:
logger.error("主機組結構體建立失敗:", e)
else:
response = json.loads(result.read().decode('utf-8'))
result.close()
logger.info("group_result: {}".format(response))
return response
def update_zabbix_group(Groupid,Groupname,auth_code,ZABBIX_URL):
url = "{}/api_jsonrpc.php".format(ZABBIX_URL)
header = {"Content-Type": "application/json"}
data = {
"jsonrpc": "2.0",
"method": "hostgroup.update",
"params": {
"groupid": Groupid,
"name": Groupname,
},
"auth": auth_code,
"id": 1
}
value = json.dumps(data).encode('utf-8')
logger.info("zabbix組信息修改以下: {}".format(value))
req = request.Request(url, headers=header, data=value)
try:
# 打開包裝過的url
result = request.urlopen(req)
except Exception as e:
logger.error("result封裝：", e)
else:
response = json.loads(result.read().decode('utf-8'))
result.close()
logger.info("group_result: {}".format(response['result']))
def delete_zabbix_group(Groupid,auth_code,ZABBIX_URL):
url = "{}/api_jsonrpc.php".format(ZABBIX_URL)
header = {"Content-Type": "application/json"}
data = {
"jsonrpc": "2.0",
"method": "hostgroup.delete",
"params": [Groupid],
"auth": auth_code,
"id": 1
}
value = json.dumps(data).encode('utf-8')
logger.info("zabbix組刪除信息以下: {}".format(value))
req = request.Request(url, headers=header, data=value)
try:
# 打開包裝過的url
result = request.urlopen(req)
except Exception as e:
logger.error("result封裝：", e)
else:
response = json.loads(result.read().decode('utf-8'))
result.close()
logger.info("group_result: {}".format(response['result']))
def main_zabbix_group(result,auth_code):
zabbix_host_group_id = {'hostid':'','grouplistid':[],'zabbix_host_name':'','result':1}
Groupidlist = []
##獲取zabbix中的組是否存在
for i in result['data']['cur_data']['group']:
zabbix_host_name= "[" +i+ "_" + result['data']['cur_data']['ip'] + "]" + zabbix_host_name
Groupname_result = select_zabbix_group(i,auth_code,settings.ZABBIX_URL)
logger.info(Groupname_result)
if len(Groupname_result) == 0:
res = create_zabbix_group(i,auth_code,settings.ZABBIX_URL)
if "errr" in res:
logger.error('zabbix增長組失敗:{}'.format(i))
else:
logger.info('zabbix增長組:{}'.format(i))
Groupidlist.append(res['result']['groupids'][0])
else:
Groupidlist.append(Groupname_result[0]['groupid'])
logger.info('zabbix分組已存在:{}'.format(i))
# 查詢zabbix 主機信息獲取組id，及主機id
res = select_zabbix_host( result['data']['cur_data']['ip'],auth_code,settings.ZABBIX_URL)
if len(res) != 0:
logger.info('zabbix主機{}查詢信息以下:{}'.format(result['data']['cur_data']['ip'],res))
# 增長原始組id
#for i in res[0]['groups']:
# Groupidlist.append(i['groupid'])
zabbix_host_group_id['hostid'] = res[0]['hostid']
zabbix_host_group_id['grouplistid'] = Groupidlist
zabbix_host_group_id['result'] = 0
return zabbix_host_group_id
else:
logger.info('zabbix主機{}查詢信息爲空:{}'.format(result['data']['cur_data']['ip']))
cmdb_host_os_type = select_cmdb_host(result['data']['cur_data']['bk_host_id'])
zabbix_template_name = "Template OS " + cmdb_host_os_type
zabbix_template_id = select_zabbix_template(zabbix_template_name,auth_code,settings.ZABBIX_URL)
logger.info('zabbix建立主機：{},主機系統類型：{}，主機模板id：{}'.format(result['data']['cur_data']['ip'],cmdb_host_os_type,zabbix_template_id))
res = create_zabbix_host(result['data']['cur_data']['ip'],zabbix_host_name,Groupidlist,zabbix_template_id,auth_code,settings.ZABBIX_URL)
return zabbix_host_group_id
複製代碼
（3）主機操做
vim zabbix/host.py
import json
import logging
from urllib import request, parse
logger = logging.getLogger('log')
def select_zabbix_host(Host,auth_code,ZABBIX_URL):
url = "{}/api_jsonrpc.php".format(ZABBIX_URL)
header = {"Content-Type": "application/json"}
data = json.dumps({
"jsonrpc": "2.0",
"method": "host.get",
"params": {
"output": ["hostid","name","status","host","groupids"],
"selectGroups":"extend",
"selectInterfaces":["ip"],
"filter":{"host":Host}
},
"id": 1,
"auth": auth_code}).encode('utf-8')
req = request.Request(url, headers=header, data=data)
try:
# 打開包裝過的url
result = request.urlopen(req)
except Exception as e:
logger.error("result封裝：", e)
else:
response = json.loads(result.read().decode('utf-8'))
result.close()
logger.info("group_result: {}".format(response['result']))
return response['result']
def select_zabbix_host_group(Host,auth_code,ZABBIX_URL):
group_list = []
url = "{}/api_jsonrpc.php".format(ZABBIX_URL)
header = {"Content-Type": "application/json"}
data = json.dumps({
"jsonrpc": "2.0",
"method": "host.get",
"params": {
"output": ["hostid","name","status","host","groupids"],
"selectGroups":"extend",
"selectInterfaces":["ip"],
"filter":{"host":Host}
},
"id": 1,
"auth": auth_code
}).encode('utf-8')
req = request.Request(url, headers=header, data=data)
try:
# 打開包裝過的url
result = request.urlopen(req)
except Exception as e:
logger.error("result封裝：", e)
else:
response = json.loads(result.read().decode('utf-8'))
# 上面獲取的是bytes類型數據，故須要decode轉化成字符串
result.close()
logger.info("group_result: {}".format(response['result']))
if (response != 0) and (len(response) != 0):
groups = response['result'][0]['groups']
for group in groups:
var = {}
var['groupid'] = group['groupid']
group_list.append(var)
# return host['groups'][0]['groupid']
return group_list
else:
return ""
return group_list
def update_zabbix_host():
pass
def update_zabbix_host_group(Hostid,Host_Group_list,auth_code, ZABBIX_URL):
url = "{}/api_jsonrpc.php".format(ZABBIX_URL)
header = {"Content-Type": "application/json"}
data = json.dumps({
"jsonrpc": "2.0",
"method": "host.update",
"params": {
# "hostid": '10202',
"hostid": Hostid,
"groups": Host_Group_list,
"status": 0,
},
"auth": auth_code,
"id": 1
}).encode('utf-8')
logger.info("更新數據json以下：{}".format(data))
req = request.Request(url, headers=header, data=data)
try:
# 打開包裝過的url
result = request.urlopen(req)
except Exception as e:
logger.error("result封裝：", e)
else:
response = json.loads(result.read().decode('utf-8'))
logger.info("返回數據以下：{}".format(response))
result.close()
if 'error' in response.keys():
logger.info("host組信息修改失敗")
else:
return response['result']
def create_zabbix_host(Hostip,Hostname,Host_Group_list,templateid,auth_code,ZABBIX_URL):
url = "{}/api_jsonrpc.php".format(ZABBIX_URL)
header = {"Content-Type": "application/json"}
data = json.dumps({
"jsonrpc": "2.0",
"method": "host.create",
"params": {
"host": Hostip,
"name": Hostname,
"interfaces": [
{
"type": 1,
"main": 1,
"useip": 1,
"ip": Hostip,
"dns": "",
"port": "10050"
}
],
"groups": Host_Group_list,
"templates": [
{
"templateid": templateid
}
]
},
"id": 1,
"auth": auth_code}).encode('utf-8')
logger.info("更新數據json以下：{}".format(data))
req = request.Request(url, headers=header, data=data)
try:
# 打開包裝過的url
result = request.urlopen(req)
except Exception as e:
logger.error("result封裝：", e)
else:
response = json.loads(result.read().decode('utf-8'))
logger.info("返回數據以下：{}".format(response))
result.close()
if 'error' in response.keys():
logger.info("zabbix主機建立失敗：{}".format(response))
else:
return response['result']
return response['result']
複製代碼
總結
經過事件推送網關咱們初步實現了cmdb和zabbix的分組同步，咱們只須要在cmdb一側就能夠管理主機資源，不只節省了咱們跨平臺處理的時間，並且規範了咱們基礎設施的管理。
相關文章
- 1. cmdb事件推送實現jumpserver資產分組同步
- 2. wordpress實現主動推送+熊掌號推送同步進行
- 3. CMDB資產採集
- 4. 20180720-同步推送
- 5. 事件推送網關：讓cmdb告別「花瓶」
- 6. cmdb資產管理2
- 7. CMDB資產管理系統
- 8. CMDB資產採集方案
- 9. 184_FreeRTOS使用事件組實現同步
- 10. zabbix 二次開發之同步CMDB主機和模板綁定
- 更多相關文章...
- • 現實生活中的 XML - XML 教程
- • Hibernate實現增刪改查 - Hibernate教程
- • ☆基於Java Instrument的Agent實現
- • Spring Cloud 微服務實戰(三) - 服務註冊與發現
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
- 1. No provider available from registry 127.0.0.1:2181 for service com.ddbuy.ser 解決方法
- 2. Qt5.7以上調用虛擬鍵盤(支持中文)，以及源碼修改(可拖動，水平縮放)
- 3. 軟件測試面試- 購物車功能測試用例設計
- 4. ElasticSearch（概念篇）：你知道的, 爲了搜索…
- 5. redux理解
- 6. gitee創建第一個項目
- 7. 支持向量機之硬間隔（一步步推導，通俗易懂）
- 8. Mysql 異步複製延遲的原因及解決方案
- 9. 如何在運行SEPM配置嚮導時將不可認的複雜數據庫密碼改爲簡單密碼
- 10. windows系統下tftp服務器使用
歡迎關注本站公眾號,獲取更多信息
相關文章