archery 平臺升級部署實踐

時間  2019-11-07
標籤 archery 平臺 升級 部署 實踐 简体版
原文   原文鏈接

目錄html

v1.3.8 → v1.5.0python

1.安裝、升級python3 venv 環境

1.1.安裝 python36

全新安裝須要執行，升級安裝可忽略mysql

https://www.cnblogs.com/David-domain/p/11165734.htmlnginx

1.2.建立Python3 venv 環境（Python>=3.6.5，建議使用虛擬環境 ）

全新安裝須要執行，升級安裝也建議從新部署一個venv 環境，便於進行切換。c++

不推薦在原venv環境中直接進行升級。git

pip3 install virtualenv

cd /opt/
virtualenv venv4archery150 --python=python36
source /opt/venv4archery150/bin/activate

#確認命令在 $PATH 中，不然安裝包時報錯
which mysql_config     #確認命令在 $PATH 中，不然安裝包時報錯

# 下載 archery 源碼
yum install -y gcc gcc-c++ epel-release python-pip python-devel unixODBC-devel
cd /opt
wget https://github.com/hhyo/archery/archive/v1.3.8.tar.gz
tar -zxvf v1.5.0.tar.gz
cd archery150
pip3 install -r requirements.txt （-i https://mirrors.ustc.edu.cn/pypi/web/simple/）

1.3.報錯處理參考

1.3.1. pyodbc 安裝失敗

# 安裝 unixODBC-devel 包
sudo yum install -y epel-release python-pip gcc-c++ python-devel unixODBC-devel

2.組件安裝配置

2.1.MySQL 安裝、配置

2.2.數據庫帳號受權

2.3. Redis 安裝——新增

  • 參照wiki 文檔安裝單實例 redis

2.4.inception 安裝、配置、啓動

2.5.Percona-toolkit 安裝

略github

2.6.Nginx 安裝、配置——有變動

yum install -y nginx
vim /etc/nginx/conf.d/nginx.conf
server{
        listen 9999; # 監聽的端口，未正式使用時，先試用臨時端口
        server_name archery150;
        client_max_body_size 20M; #新增
        proxy_read_timeout 600s;     #超時時間與gunicorn超時時間設置一致，主要用於在線查詢
        location / {
            proxy_pass http://127.0.0.1:8888/; # 8888 要與 startup.sh 腳本中端口對應
            proxy_set_header Host $host:9999;      #解決重定向404的問題
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
        location /static {
            alias /opt/archery150/common/static/;        #此處指向settings.py配置項STATIC_ROOT目錄的絕對路徑，用於nginx收集靜態資源
        }
        error_page 404 /404.html;
            location = /40x.html {
        }
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

2.7.上傳 archery 第三方工具

2.8. SQLAdvisor 安裝

2.8.1.報錯處理

2.9. SchemaSync 安裝

3.安裝 archery 環境

3.1. 1.2.步已下載源碼，略過

略web

3.2.修改 archery/settings.py 文件 DATABASES 配置項——有變動

  • 此次升級加入了redis cache，配置略有改動。django_q 使用默認配置便可。
vim /opt/archery/archery150/archery/setting.py
# 12 ~ 13
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = False #關閉調試模式
# 118 ~ 132
# 該項目自己的mysql數據庫地址
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'archery150',
        'USER': 'xxx',
        'PASSWORD': 'xxx',
        'HOST': '10.xx.xx.xx',
        'PORT': '3306',
        'OPTIONS': {
            'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
            'charset': 'utf8mb4'
        }
    }
}
# 145 ~ 158
# Django-Q 默認配置便可，也可參考django-q文檔修改
Q_CLUSTER = {
    'name': 'archery',
    'workers': 4,
    'recycle': 500,
    'timeout': 60,
    'compress': True,
    'cpu_affinity': 1,
    'save_limit': 0,
    'queue_limit': 50,
    'label': 'Django Q',
    'django_redis': 'default',
    'sync': False  # 本地調試能夠修改成True，使用同步模式
}
# 160 ~ 169
# 緩存配置
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://10.33.66.241:50001/0", #單實例
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
        }
    }
}
# 171 ~ 191
# LDAP
ENABLE_LDAP = True
if ENABLE_LDAP:
    import ldap
    from django_auth_ldap.config import LDAPSearch
    AUTHENTICATION_BACKENDS = (
        'django_auth_ldap.backend.LDAPBackend',  # 配置爲先使用LDAP認證，如經過認證則再也不使用後面的認證方式
        'django.contrib.auth.backends.ModelBackend',  # django系統中手動建立的用戶也可以使用，優先級靠後。注意這2行的順序
    )
    AUTH_LDAP_SERVER_URI = "ldap://xx.xx.xx.com"
    AUTH_LDAP_BIND_DN = "CN=auth,CN=Users,DC=xxx,DC=xxx,DC=com"
    AUTH_LDAP_BIND_PASSWORD = "xxxx"
    AUTH_LDAP_USER_SEARCH = LDAPSearch("ou=xxx,dc=xxx,dc=xxx,dc=com",ldap.SCOPE_SUBTREE, "(sAMAccountName=%(user)s)")
    AUTH_LDAP_ALWAYS_UPDATE_USER = True  # 每次登陸從ldap同步用戶信息
    AUTH_LDAP_USER_ATTR_MAP = {  # key爲archery.sql_users字段名，value爲ldap中字段名，用戶同步信息
        "username": "xxxxx",
        "display": "cn",
        "email": "mail"
    }

3.3.數據庫初始化

  • 以下命令在測試環境中執行，生產環境直接執行腳本。
# 數據庫初始化
python3 manage.py makemigrations sql
python3 manage.py migrate 

# 編譯翻譯文件
python3 manage.py compilemessages    #新增步驟
  • 官方文檔雖說能夠沒必要初始化，可是因爲權限的更新，權限ID 對應關係可能錯亂，因此使用1.38數據庫升級後，遷移到1.50數據庫的方式。
/*
 Navicat Premium Data Transfer

 Source Server         : archery150
 Source Server Type    : MySQL
 Source Server Version : 50722
 Source Host           : 
 Source Schema         : archery_new

 Target Server Type    : MySQL
 Target Server Version : 50722
 File Encoding         : 65001

 Date: 17/04/2019 15:13:10
*/
create database archery150 charset utf8;
use archery150;
SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;

-- ----------------------------
-- Table structure for aliyun_access_key
-- ----------------------------
DROP TABLE IF EXISTS `aliyun_access_key`;
CREATE TABLE `aliyun_access_key` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `ak` varchar(50) NOT NULL,
  `secret` varchar(100) NOT NULL,
  `is_enable` tinyint(4) NOT NULL COMMENT '是否啓用',
  `remark` varchar(50) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for aliyun_rds_config
-- ----------------------------
DROP TABLE IF EXISTS `aliyun_rds_config`;
CREATE TABLE `aliyun_rds_config` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `instance_id` int(11) NOT NULL,
  `rds_dbinstanceid` varchar(100) NOT NULL,
  `is_enable` tinyint(4) NOT NULL COMMENT '是否啓用',
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_uni_iid` (`instance_id`),
  CONSTRAINT `fk_instanid__sql_instance_id` FOREIGN KEY (`instance_id`) REFERENCES `sql_instance` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for auth_group
-- ----------------------------
DROP TABLE IF EXISTS `auth_group`;
CREATE TABLE `auth_group` (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主鍵',
  `name` varchar(80) NOT NULL COMMENT '組',
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='權限組';

-- ----------------------------
-- Table structure for auth_group_permissions
-- ----------------------------
DROP TABLE IF EXISTS `auth_group_permissions`;
CREATE TABLE `auth_group_permissions` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `group_id` int(11) NOT NULL,
  `permission_id` int(11) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_uni_gid_pid` (`group_id`,`permission_id`),
  KEY `idx_pid` (`permission_id`),
  CONSTRAINT `fk_groupid__auth_group_id` FOREIGN KEY (`group_id`) REFERENCES `auth_group` (`id`),
  CONSTRAINT `fk_perid__auth_permission_id` FOREIGN KEY (`permission_id`) REFERENCES `auth_permission` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for auth_permission
-- ----------------------------
DROP TABLE IF EXISTS `auth_permission`;
CREATE TABLE `auth_permission` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(255) NOT NULL,
  `content_type_id` int(11) NOT NULL,
  `codename` varchar(100) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_uni_ctid` (`content_type_id`,`codename`),
  CONSTRAINT `fk_ctypeid__django_content_type_id` FOREIGN KEY (`content_type_id`) REFERENCES `django_content_type` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=137 DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for data_masking_columns
-- ----------------------------
DROP TABLE IF EXISTS `data_masking_columns`;
CREATE TABLE `data_masking_columns` (
  `column_id` int(11) NOT NULL AUTO_INCREMENT,
  `rule_type` int(11) NOT NULL,
  `active` tinyint(4) NOT NULL COMMENT '激活狀態',
  `instance_id` int(11) NOT NULL,
  `table_schema` varchar(64) NOT NULL,
  `table_name` varchar(64) NOT NULL,
  `column_name` varchar(64) NOT NULL,
  `column_comment` varchar(1024) NOT NULL,
  `create_time` datetime(6) NOT NULL,
  `sys_time` datetime(6) NOT NULL,
  PRIMARY KEY (`column_id`),
  KEY `idx_iid` (`instance_id`),
  CONSTRAINT `fk_instance_id__sql_instance_id` FOREIGN KEY (`instance_id`) REFERENCES `sql_instance` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for data_masking_rules
-- ----------------------------
DROP TABLE IF EXISTS `data_masking_rules`;
CREATE TABLE `data_masking_rules` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `rule_type` int(11) NOT NULL,
  `rule_regex` varchar(255) NOT NULL,
  `hide_group` int(11) NOT NULL,
  `rule_desc` varchar(100) NOT NULL,
  `sys_time` datetime(6) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_uni_rule_type` (`rule_type`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for django_admin_log
-- ----------------------------
DROP TABLE IF EXISTS `django_admin_log`;
CREATE TABLE `django_admin_log` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `action_time` datetime(6) NOT NULL,
  `object_id` longtext,
  `object_repr` varchar(200) NOT NULL,
  `action_flag` smallint(5) unsigned NOT NULL,
  `change_message` longtext NOT NULL,
  `content_type_id` int(11) DEFAULT NULL,
  `user_id` int(11) NOT NULL,
  PRIMARY KEY (`id`),
  KEY `idx_ctid` (`content_type_id`),
  KEY `idx_uid` (`user_id`),
  CONSTRAINT `fk_ctid__django_content_type_id` FOREIGN KEY (`content_type_id`) REFERENCES `django_content_type` (`id`),
  CONSTRAINT `fk_users_id__sql_users_id` FOREIGN KEY (`user_id`) REFERENCES `sql_users` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for django_content_type
-- ----------------------------
DROP TABLE IF EXISTS `django_content_type`;
CREATE TABLE `django_content_type` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `app_label` varchar(100) NOT NULL,
  `model` varchar(100) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_uni_app_label__model` (`app_label`,`model`)
) ENGINE=InnoDB AUTO_INCREMENT=34 DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for django_migrations
-- ----------------------------
DROP TABLE IF EXISTS `django_migrations`;
CREATE TABLE `django_migrations` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `app` varchar(255) NOT NULL,
  `name` varchar(255) NOT NULL,
  `applied` datetime(6) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=25 DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for django_q_ormq
-- ----------------------------
DROP TABLE IF EXISTS `django_q_ormq`;
CREATE TABLE `django_q_ormq` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `key` varchar(100) NOT NULL,
  `payload` longtext NOT NULL,
  `lock` datetime(6) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for django_q_schedule
-- ----------------------------
DROP TABLE IF EXISTS `django_q_schedule`;
CREATE TABLE `django_q_schedule` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `func` varchar(256) NOT NULL,
  `hook` varchar(256) DEFAULT NULL,
  `args` longtext,
  `kwargs` longtext,
  `schedule_type` varchar(1) NOT NULL,
  `repeats` int(11) NOT NULL,
  `next_run` datetime(6) DEFAULT NULL,
  `task` varchar(100) DEFAULT NULL,
  `name` varchar(100) DEFAULT NULL,
  `minutes` smallint(5) unsigned DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for django_q_task
-- ----------------------------
DROP TABLE IF EXISTS `django_q_task`;
CREATE TABLE `django_q_task` (
  `id` varchar(32) NOT NULL,
  `name` varchar(100) NOT NULL,
  `func` varchar(256) NOT NULL,
  `hook` varchar(256) DEFAULT NULL,
  `args` longtext,
  `kwargs` longtext,
  `result` longtext,
  `started` datetime(6) NOT NULL,
  `stopped` datetime(6) NOT NULL,
  `success` tinyint(4) NOT NULL,
  `group` varchar(100) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for django_session
-- ----------------------------
DROP TABLE IF EXISTS `django_session`;
CREATE TABLE `django_session` (
  `session_key` varchar(40) NOT NULL,
  `session_data` longtext NOT NULL,
  `expire_date` datetime(6) NOT NULL,
  PRIMARY KEY (`session_key`),
  KEY `idx_expire_date` (`expire_date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for param_history
-- ----------------------------
DROP TABLE IF EXISTS `param_history`;
CREATE TABLE `param_history` (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主鍵',
  `instance_id` int(11) NOT NULL COMMENT '實例ID',
  `variable_name` varchar(64) NOT NULL COMMENT '參數名',
  `old_var` varchar(1024) NOT NULL COMMENT '修改前參數值',
  `new_var` varchar(1024) NOT NULL COMMENT '修改後參數值',
  `set_sql` varchar(1024) NOT NULL COMMENT '在線變動配置執行的SQL語句',
  `user_name` varchar(30) NOT NULL COMMENT '修改人',
  `user_display` varchar(50) NOT NULL COMMENT '修改人中文名',
  `update_time` datetime(6) NOT NULL COMMENT '修改時間',
  PRIMARY KEY (`id`),
  KEY `idx_iid` (`instance_id`),
  CONSTRAINT `fk_instanceid__sql_instance_id` FOREIGN KEY (`instance_id`) REFERENCES `sql_instance` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for param_template
-- ----------------------------
DROP TABLE IF EXISTS `param_template`;
CREATE TABLE `param_template` (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主鍵',
  `db_type` varchar(10) NOT NULL COMMENT '數據庫類型，mysql、mssql、redis、pgsql',
  `variable_name` varchar(64) NOT NULL COMMENT '參數名',
  `default_value` varchar(1024) NOT NULL COMMENT '默認參數值',
  `editable` tinyint(4) NOT NULL COMMENT '是否支持修改',
  `valid_values` varchar(1024) NOT NULL COMMENT '有效參數值',
  `description` varchar(1024) NOT NULL COMMENT '參數描述',
  `create_time` datetime(6) NOT NULL COMMENT '建立時間',
  `sys_time` datetime(6) NOT NULL COMMENT '建立時間',
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_uni_db_type__variable_name` (`db_type`,`variable_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for query_log
-- ----------------------------
DROP TABLE IF EXISTS `query_log`;
CREATE TABLE `query_log` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `instance_name` varchar(50) NOT NULL,
  `db_name` varchar(64) NOT NULL,
  `sqllog` longtext NOT NULL,
  `effect_row` bigint(20) NOT NULL,
  `cost_time` varchar(10) NOT NULL,
  `username` varchar(30) NOT NULL,
  `user_display` varchar(50) NOT NULL,
  `priv_check` tinyint(4) NOT NULL COMMENT '查詢權限是否正常校驗',
  `hit_rule` tinyint(4) NOT NULL COMMENT '查詢是否命中脫敏規則',
  `masking` tinyint(4) NOT NULL COMMENT '查詢結果是否正常脫敏',
  `create_time` datetime(6) NOT NULL,
  `sys_time` datetime(6) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for query_privileges
-- ----------------------------
DROP TABLE IF EXISTS `query_privileges`;
CREATE TABLE `query_privileges` (
  `privilege_id` int(11) NOT NULL AUTO_INCREMENT COMMENT '權限id',
  `user_name` varchar(30) NOT NULL COMMENT '用戶',
  `user_display` varchar(50) NOT NULL COMMENT '下拉菜單篩選名',
  `instance_id` int(11) NOT NULL,
  `table_name` varchar(64) NOT NULL COMMENT '表',
  `db_name` varchar(64) NOT NULL COMMENT '數據庫',
  `valid_date` date NOT NULL COMMENT '有效時間',
  `limit_num` int(11) NOT NULL COMMENT '結果集',
  `priv_type` tinyint(4) NOT NULL COMMENT '權限級別',
  `is_deleted` tinyint(4) NOT NULL COMMENT '刪除標記',
  `create_time` datetime(6) NOT NULL COMMENT '申請時間',
  `sys_time` datetime(6) NOT NULL COMMENT '系統時間',
  PRIMARY KEY (`privilege_id`),
  KEY `idx_uname__iid__db_name__vdate` (`user_name`,`instance_id`,`db_name`,`valid_date`),
  KEY `idx_iid` (`instance_id`),
  CONSTRAINT `fk_instid__sql_instance_id` FOREIGN KEY (`instance_id`) REFERENCES `sql_instance` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for query_privileges_apply
-- ----------------------------
DROP TABLE IF EXISTS `query_privileges_apply`;
CREATE TABLE `query_privileges_apply` (
  `apply_id` int(11) NOT NULL AUTO_INCREMENT,
  `group_id` int(11) NOT NULL,
  `group_name` varchar(100) NOT NULL,
  `title` varchar(50) NOT NULL,
  `user_name` varchar(30) NOT NULL,
  `user_display` varchar(50) NOT NULL,
  `instance_id` int(11) NOT NULL,
  `db_list` longtext NOT NULL,
  `table_list` longtext NOT NULL,
  `valid_date` date NOT NULL,
  `limit_num` int(11) NOT NULL,
  `priv_type` tinyint(4) NOT NULL,
  `status` int(11) NOT NULL,
  `audit_auth_groups` varchar(255) NOT NULL,
  `create_time` datetime(6) NOT NULL,
  `sys_time` datetime(6) NOT NULL,
  PRIMARY KEY (`apply_id`),
  KEY `idx_iid` (`instance_id`),
  CONSTRAINT `fk_insid__sql_instance_id` FOREIGN KEY (`instance_id`) REFERENCES `sql_instance` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for resource_group
-- ----------------------------
DROP TABLE IF EXISTS `resource_group`;
CREATE TABLE `resource_group` (
  `group_id` int(11) NOT NULL AUTO_INCREMENT,
  `group_name` varchar(100) NOT NULL,
  `group_parent_id` bigint(20) NOT NULL,
  `group_sort` int(11) NOT NULL,
  `group_level` int(11) NOT NULL,
  `ding_webhook` varchar(255) NOT NULL,
  `is_deleted` tinyint(4) NOT NULL,
  `create_time` datetime(6) NOT NULL,
  `sys_time` datetime(6) NOT NULL,
  PRIMARY KEY (`group_id`),
  UNIQUE KEY `idx_uni_group_name` (`group_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for resource_group_relations
-- ----------------------------
DROP TABLE IF EXISTS `resource_group_relations`;
CREATE TABLE `resource_group_relations` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `object_type` tinyint(4) NOT NULL,
  `object_id` int(11) NOT NULL,
  `object_name` varchar(100) NOT NULL,
  `group_id` int(11) NOT NULL,
  `group_name` varchar(100) NOT NULL,
  `create_time` datetime(6) NOT NULL,
  `sys_time` datetime(6) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_uni_oid__gid__object_type` (`object_id`,`group_id`,`object_type`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for sql_config
-- ----------------------------
DROP TABLE IF EXISTS `sql_config`;
CREATE TABLE `sql_config` (
  `item` varchar(50) NOT NULL,
  `value` varchar(200) NOT NULL,
  `description` varchar(200) NOT NULL,
  PRIMARY KEY (`item`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for sql_instance
-- ----------------------------
DROP TABLE IF EXISTS `sql_instance`;
CREATE TABLE `sql_instance` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `instance_name` varchar(50) NOT NULL,
  `type` varchar(6) NOT NULL,
  `db_type` varchar(10) NOT NULL,
  `host` varchar(200) NOT NULL,
  `port` int(11) NOT NULL,
  `user` varchar(100) NOT NULL,
  `password` varchar(300) NOT NULL,
  `create_time` datetime(6) NOT NULL,
  `update_time` datetime(6) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_uni_instance_name` (`instance_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for sql_permission
-- ----------------------------
DROP TABLE IF EXISTS `sql_permission`;
CREATE TABLE `sql_permission` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for sql_users
-- ----------------------------
DROP TABLE IF EXISTS `sql_users`;
CREATE TABLE `sql_users` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `PASSWORD` varchar(128) NOT NULL COMMENT '密碼',
  `last_login` datetime(6) DEFAULT NULL COMMENT '上次登陸',
  `is_superuser` tinyint(4) NOT NULL COMMENT '超級用戶狀態:1是,0否',
  `username` varchar(150) NOT NULL COMMENT '用戶名',
  `first_name` varchar(30) NOT NULL COMMENT '名,無值',
  `last_name` varchar(150) NOT NULL COMMENT '姓,無值',
  `email` varchar(254) NOT NULL COMMENT '電子郵箱地址',
  `is_staff` tinyint(4) NOT NULL COMMENT '職員狀態(是否能管理django後臺):1是,0否',
  `is_active` tinyint(4) NOT NULL COMMENT '有效(禁用用戶標籤):1是,0否',
  `date_joined` datetime(6) NOT NULL COMMENT '加入日期(第一次登陸時間)',
  `display` varchar(50) NOT NULL COMMENT '顯示的中文名',
  `failed_login_count` int(11) NOT NULL COMMENT '登錄失敗次數',
  `last_login_failed_at` datetime DEFAULT NULL COMMENT '上次失敗登陸時間',
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_uni_username` (`username`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for sql_users_groups
-- ----------------------------
DROP TABLE IF EXISTS `sql_users_groups`;
CREATE TABLE `sql_users_groups` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `users_id` int(11) NOT NULL,
  `group_id` int(11) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_uni_uid__gid` (`users_id`,`group_id`),
  KEY `idx_gid` (`group_id`),
  CONSTRAINT `fk_gid__auth_group_id` FOREIGN KEY (`group_id`) REFERENCES `auth_group` (`id`),
  CONSTRAINT `fk_usersid__sql_users_id` FOREIGN KEY (`users_id`) REFERENCES `sql_users` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for sql_users_user_permissions
-- ----------------------------
DROP TABLE IF EXISTS `sql_users_user_permissions`;
CREATE TABLE `sql_users_user_permissions` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `users_id` int(11) NOT NULL,
  `permission_id` int(11) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_uni_uid__pid` (`users_id`,`permission_id`),
  KEY `idx_pid` (`permission_id`),
  CONSTRAINT `fk_pid__auth_permission_id` FOREIGN KEY (`permission_id`) REFERENCES `auth_permission` (`id`),
  CONSTRAINT `fk_uid__sql_users_id` FOREIGN KEY (`users_id`) REFERENCES `sql_users` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for sql_workflow
-- ----------------------------
DROP TABLE IF EXISTS `sql_workflow`;
CREATE TABLE `sql_workflow` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `workflow_name` varchar(50) NOT NULL,
  `group_id` int(11) NOT NULL,
  `group_name` varchar(100) NOT NULL,
  `instance_id` int(11) NOT NULL,
  `db_name` varchar(64) NOT NULL COMMENT '數據庫',
  `engineer` varchar(30) NOT NULL,
  `engineer_display` varchar(50) NOT NULL,
  `audit_auth_groups` varchar(255) NOT NULL,
  `create_time` datetime(6) NOT NULL,
  `finish_time` datetime(6) DEFAULT NULL,
  `STATUS` varchar(50) NOT NULL,
  `is_backup` tinyint(4) NOT NULL COMMENT '是否備份',
  `is_manual` tinyint(4) NOT NULL,
  `syntax_type` tinyint(4) NOT NULL COMMENT '工單類型 一、DDL，二、DML',
  PRIMARY KEY (`id`),
  KEY `idx_iid` (`instance_id`),
  CONSTRAINT `fk_iid__sql_instance_id` FOREIGN KEY (`instance_id`) REFERENCES `sql_instance` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for sql_workflow_content
-- ----------------------------
DROP TABLE IF EXISTS `sql_workflow_content`;
CREATE TABLE `sql_workflow_content` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `workflow_id` int(11) NOT NULL COMMENT 'SQL工單ID',
  `sql_content` longtext NOT NULL COMMENT '提交的SQL文本',
  `review_content` longtext NOT NULL COMMENT '自動審覈內容的JSON格式',
  `execute_result` longtext NOT NULL COMMENT '執行結果的JSON格式',
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_uni_workflow_id` (`workflow_id`),
  CONSTRAINT `fk_wfid__sql_workflow_id` FOREIGN KEY (`workflow_id`) REFERENCES `sql_workflow` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for workflow_audit
-- ----------------------------
DROP TABLE IF EXISTS `workflow_audit`;
CREATE TABLE `workflow_audit` (
  `audit_id` int(11) NOT NULL AUTO_INCREMENT,
  `group_id` int(11) NOT NULL,
  `group_name` varchar(100) NOT NULL,
  `workflow_id` bigint(20) NOT NULL,
  `workflow_type` tinyint(4) NOT NULL,
  `workflow_title` varchar(50) NOT NULL,
  `workflow_remark` varchar(140) NOT NULL,
  `audit_auth_groups` varchar(255) NOT NULL,
  `current_audit` varchar(20) NOT NULL,
  `next_audit` varchar(20) NOT NULL,
  `current_status` tinyint(4) NOT NULL,
  `create_user` varchar(30) NOT NULL,
  `create_user_display` varchar(50) NOT NULL,
  `create_time` datetime(6) NOT NULL,
  `sys_time` datetime(6) NOT NULL,
  PRIMARY KEY (`audit_id`),
  UNIQUE KEY `idx_uni_wfid__workflow_type` (`workflow_id`,`workflow_type`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for workflow_audit_detail
-- ----------------------------
DROP TABLE IF EXISTS `workflow_audit_detail`;
CREATE TABLE `workflow_audit_detail` (
  `audit_detail_id` int(11) NOT NULL AUTO_INCREMENT,
  `audit_id` int(11) NOT NULL,
  `audit_user` varchar(30) NOT NULL,
  `audit_time` datetime(6) NOT NULL,
  `audit_status` tinyint(4) NOT NULL,
  `remark` varchar(140) NOT NULL,
  `sys_time` datetime(6) NOT NULL,
  PRIMARY KEY (`audit_detail_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for workflow_audit_setting
-- ----------------------------
DROP TABLE IF EXISTS `workflow_audit_setting`;
CREATE TABLE `workflow_audit_setting` (
  `audit_setting_id` int(11) NOT NULL AUTO_INCREMENT,
  `group_id` int(11) NOT NULL,
  `group_name` varchar(100) NOT NULL,
  `workflow_type` tinyint(4) NOT NULL,
  `audit_auth_groups` varchar(255) NOT NULL,
  `create_time` datetime(6) NOT NULL,
  `sys_time` datetime(6) NOT NULL,
  PRIMARY KEY (`audit_setting_id`),
  UNIQUE KEY `idx_uni_gid__workflow_type` (`group_id`,`workflow_type`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for workflow_log
-- ----------------------------
DROP TABLE IF EXISTS `workflow_log`;
CREATE TABLE `workflow_log` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主鍵',
  `audit_id` bigint(20) NOT NULL COMMENT '工單審批id',
  `operation_type` tinyint(4) NOT NULL COMMENT '操做類型，0提交/待審覈、1審覈經過、2審覈不經過、3審覈取消/取消執行、4定時執行、5執行工單、6執行結束',
  `operation_type_desc` char(10) NOT NULL COMMENT '操做類型描述',
  `operation_info` varchar(200) NOT NULL COMMENT '操做信息',
  `operator` varchar(30) NOT NULL COMMENT '操做人',
  `operator_display` varchar(50) NOT NULL COMMENT '操做人中文名',
  `operation_time` datetime(6) NOT NULL COMMENT '操做時間',
  PRIMARY KEY (`id`),
  KEY `idx_aid` (`audit_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

SET FOREIGN_KEY_CHECKS = 1;

3.4.建立管理用戶

cd /opt/archery150/
python3 manage.py createsuperuser
Username: archery_admin    # 手動輸入
Email address: xx@xx.com
Password: xxxx       #輸入密碼
Password (again): xxxxx      #確認密碼
Superuser created successfully.

4.數據庫升級

4.1.導出線上庫

# 10.xx.xx.xx
sudo su -
mysqldump -S xx/mysql.sock --single-transaction --master-data=2 --set-gtid-purged=OFF archery > archery138to150.sql

4.2.在目標實例上恢復數據

# 10.xx.xx.xx
sudo su -
mysql -S xx.sock
create database archery138to150 charset utf8mb4;
\q
mysql -S xx/mysql.sock archery138to150 < archery138to150.sql

4.3.中間庫執行數據庫升級腳本

# 10.xx.xx.xx
sudo su -
cd /opt/archery150/src/init_sql
mysql -h 10.xx.xx.xx -P3306 -uxx -p archery138to150 < v1.3.8_v1.4.0.sql
mysql -h 10.xx.xx.xx -P3306 -uxx -p archery138to150 < v1.4.2_v1.4.3.sql
mysql -h 10.xx.xx.xx -P3306 -uxx -p archery138to150 < v1.4.3_v1.4.5.sql
mysql -h 10.xx.xx.xx -P3306 -uxx -p archery138to150 < v1.4.5_v1.5.0.sql

4.4.數據庫標準化更新——新增——新版本數據庫執行

4.4.1.添加慢日誌收集表、添加阿里雲參數

mysql -h 10.xx.xx.xx -P3306 -uxx -p archery150 < mysql_slow_query_review.sql
mysql -h 10.xx.xx.xx -P3306 -uxx -p archery150 < rds_param_template.sql

4.4.2.標準化SQL腳本，調整索引名稱、外鍵名稱、字段類型等redis

4.4.3.字段添加註釋SQL腳本，已PR到1.5.3版本代碼中。

mysql -h 10.xx.xx.xx -P3306 -uxx -p archery138to150 < 1.5.3_comment.sql

4.4.4.導出恢復庫升級數據，並導入 archery 數據庫。

SET FOREIGN_KEY_CHECKS = 0;
insert into archery150.aliyun_rds_config select * from archery138to150.aliyun_rds_config;
insert into archery150.auth_group select * from archery138to150.auth_group;
insert into archery150.data_masking_columns select * from archery138to150.data_masking_columns;
insert into archery150.data_masking_rules select * from archery138to150.data_masking_rules;
insert into archery150.django_admin_log select * from archery138to150.django_admin_log;
insert into archery150.django_session select * from archery138to150.django_session;
insert into archery150.mysql_slow_query_review select * from archery138to150.mysql_slow_query_review;
insert into archery150.mysql_slow_query_review_history select * from archery138to150.mysql_slow_query_review_history;
insert into archery150.query_log select * from archery138to150.query_log;
insert into archery150.query_privileges select * from archery138to150.query_privileges;
insert into archery150.query_privileges_apply select * from archery138to150.query_privileges_apply;
insert into archery150.resource_group select * from archery138to150.resource_group;
insert into archery150.resource_group_relations select * from archery138to150.resource_group_relations;
insert into archery150.sql_instance select * from archery138to150.sql_instance;
truncate table archery150.sql_users;
insert into archery150.sql_users select * from archery138to150.sql_users;
insert into archery150.sql_users_groups select * from archery138to150.sql_users_groups;
insert into archery150.sql_workflow select * from archery138to150.sql_workflow;
insert into archery150.sql_workflow_content select * from archery138to150.sql_workflow_content;
insert into archery150.workflow_audit select * from archery138to150.workflow_audit;
insert into archery150.workflow_audit_detail select * from archery138to150.workflow_audit_detail;
insert into archery150.workflow_audit_setting select * from archery138to150.workflow_audit_setting;
insert into archery150.workflow_log select * from  archery138to150.workflow_log;

5.啓動

5.1. 方式 1（推薦）：gunicorn+nginx啓動

server{
        listen 9999; # 監聽的端口
        server_name archery150;
        client_max_body_size 20M; # 處理Request Entity Too Large
        proxy_read_timeout 600s;  # 超時時間與Gunicorn超時時間設置一致，主要用於在線查詢
        location / {
            proxy_pass http://127.0.0.1:8888;
            proxy_set_header Host $host:9123; # 解決重定向404的問題，和listen端口保持一致，若是是docker則和宿主機映射端口保持一致
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
        location /static {
            alias /opt/archery/static; # 此處指向settings.py配置項STATIC_ROOT目錄的絕對路徑，用於nginx收集靜態資源
        }
        error_page 404 /404.html;
            location = /40x.html {
        }
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
cd /opt/archery/archery-1.5.0/
source /opt/venv4archery150/bin/activate
sed -i 's#8000#8888#g' startup.sh
bash startup.sh

5.1.1.報錯

  • 啓動不正常先檢查nginx 是否安裝、是否正常啓動
  • 再檢查nginx 配置是否正確
  • 檢查 qcluster 啓動是否正常，是否能夠看到進程
  • 查看 startup.sh 的 nohup 日誌信息

5.2.方式2（不推薦）——runserver 啓動:

# 殺掉已有進程
ps aux | grep python3 | grep -v grep | awk '{print $2}' | xargs kill -9
cd /opt/archery/archery150/
source /opt/venv4archery/bin/activate
python3 manage.py qcluster &
source /opt/venv4archery/bin/activate
python3 manage.py runserver 0.0.0.0:9999  --insecure &

6.配置archery

6.1.Inception配置

訪問 http://10.xx.xx.xx:9999sql

INCEPTION_HOST 10.33.66.241
INCEPTION_PORT 6669
REMOTE_BACKUP_HOST 10.33.66.231
REMOTE_BACKUP_PORT 3306
REMOTE_BACKUP_USER incep
REMOTE_BACKUP_PASSWORD xxxx

6.2.功能模塊配置

6.2.1.SQL上線

CRITICAL_DDL_REGEX  ^drop|^truncate|^rename
AUTO_REVIEW_WRONG  2
AUTO_REVIEW OFF

6.2.2.SQL查詢

QUERY_CHECK  ON
DISABLE_STAR
DATA_MASKING  ON
MAX_EXECUTION_TIME
ADMIN_QUERY_LIMIT

6.2.3.SQL優化

SQLADVISOR_PATH  /opt/archery/sqladvisor
SOAR_PATH  /opt/archery/soar
SOAR_TEST_DSN  xxx:xxxx@10.xx.xx.xxx:3307/archery150

6.3.通知配置

ARCHERY_BASE_URL  https://xx.xx.xx.com
MAIL  ON
MAIL_SSL  ON
MAIL_SMTP_SERVER  xx.xx.com
MAIL_SMTP_PORT  xx
MAIL_SMTP_USER  xx@xx.com
MAIL_SMTP_PASSWORD  
DDL_NOTIFY_AUTH_GROUP  dinglu@ybm100.com
DING  ON

6.4.其餘配置

SCHEMASYNC  /opt/archery/schemasync
BINLOG2SQL  /opt/archery/binlog2sql.py
DEFAULT_AUTH_GROUP  RD
DEFAULT_RESOURCE_GROUP  ALL
LOCK_TIME_THRESHOLD
LOCK_CNT_THRESHOLD  
SIGN_UP_ENABLED  OFF

7.修改

修改後須要重啓 qcluster 或者重啓python3服務，才能生效。



相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程