gitlab數據遷移至其餘gitlb服務器上

需求：

A : 待遷移服務器，上邊存有數據

B：接收項目得服務器，自己存在數據

驗證方案：

一，搭建gitlab8.15.2

OS:rhel7.4

yum install policycoreutils-python.x86_64

wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-8.15.2-ce.0.el7.x86_64.rpm

chmod 777 gitlab-ce-8.15.2-ce.0.el7.x86_64.rpm

rpm -ivh gitlab-ce-8.15.2-ce.0.el7.x86_64.rpm

########################更新配置文件########修改數據庫爲mysql###################

1）安裝MySQL

wget -i -c  http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm

yum -y install mysql57-community-release-el7-10.noarch.rpm

yum -y install mysql-community-server

yum install mysql-devel

2）修改mysql密碼

systemctl start mysqld.service

grep "password" /var/log/mysqld.log   ===》得到臨時密碼

mysql -uroot -p     # 回車後會提示輸入密碼

alter user 'root'@'localhost' identified by '123qweASD!';   ===》修改密碼

set global validate_password_policy=0;  ===》關閉MySQL密碼驗證

set global validate_password_length=1;

 

注：MySQL須要開啓binlog

vim /etc/my.cnf

添加

log-bin=/var/lib/mysql/mysql-bin   ====>路徑注意權限，不然會啓動報錯

server-id=10

 

3）建立必要數據庫

CREATE USER 'git'@'localhost' IDENTIFIED BY '123456';

SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda, innodb_large_prefix=1;

CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_general_ci`;

GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES, REFERENCES ON `gitlabhq_production`.* TO 'git'@'localhost';

flush privileges;

4）修改配置文件

vim /etc/gitlab/gitlab.rb

 

修改內容以下

external_url 'http://10.2.177.31:8090'  ===》AIP

gitlab_rails['db_adapter'] = "mysql2"
gitlab_rails['db_encoding'] = "utf8"
gitlab_rails['db_database'] = "gitlabhq_production"
gitlab_rails['db_username'] = "git"
gitlab_rails['db_password'] = "123456"
gitlab_rails['db_host'] = "127.0.0.1"
gitlab_rails['db_port'] = 3306
postgresql['enable'] = false

 

vim /opt/gitlab/embedded/service/gitlab-rails/.bundle/config

修改內容以下：

BUNDLE_WITHOUT: "development:test:postgres"  ===》MySQL改成postgres

 

5）重構

cd /opt/gitlab/embedded/bin/

./gem install mysql2 --version="0.3.20"

./gem install -i /opt/gitlab/embedded/service/gem/ruby/2.3.0/ mysql2 --version="0.3.20"

//gitlab-rake gitlab:setup

gitlab-ctl reconfigure

gitlab-ctl restart

6)檢查

gitlab-rake gitlab:check

#################一樣搭建另外一臺#####################

保險起見，先備份gitlab

 gitlab-rake gitlab:backup:create  ====>備份目錄/var/opt/gitlab/backups/

##################################################

二，備份庫以及其餘文件

key目錄：

/var/opt/gitlab/.ssh/authorized_keys

====》cp authorized_keys /opt

倉庫目錄：

/var/opt/gitlab/git-data/repositories

  ===》tar cvf repositories_A.tar repositories   ===》cp repositories_A.tar /opt/

mysql庫備份：

先建立一個備份數據庫&複製到備份數據庫gitlab

mysqldump gitlabhq_production -uroot -p123qweASD! | mysql gitlab -uroot -p123qweASD!

導出備份數據庫：

mysqldump --single-transaction -u root -p -B xx > A.sql  ===》cp A.sql /opt

傳送文件：將上述3個文件拷貝至接收服務器B：/opt目錄下，其餘目錄也能夠

解析binlog

cd mysql-binlog的目錄

 mysqlbinlog --base64-output=decode-rows -v -d gitlabhq_production mysql-bin.000001 > t.binlog

 

 

三，遷移文件

gitlab命令參考：

https://wiki.archlinux.org/index.php/GitLab

 

1）合併MySQL數據庫

查看t.binlog中的sql語句，按照順序在mysql 上進行操做

 涉及的表爲users,namespaces,routes,audit_events,members,notification_settings,project_features,projects,events,project_authorizations,

　　protected_branches,protected_branch_merge_access_levels,protected_branch_push_access_levels,events 共13個表

插入數據時注意各個表的id不要與已存數據衝突

 

2）合併repository數據庫

cd /var/opt/gitlab/git-data/

tar xvf repositories_B.tar  ===>將B倉庫合併至A中

以後執行gitlab-rake gitlab:import:repos

 

或者執行

#chmod -R ug+rwX,o-rwx 倉庫path
#chmod -R ug-s 倉庫path
#find 倉庫path -type d -print0 |  xargs -0 chmod g+s
#chown -R git:git 倉庫path

 

3）合併authorized_keys  ====>視狀況====》我驗證時不用

cat authorized_keys_B >> /var/opt/gitlab/.ssh/authorized_keys

 

############################### ########################MySQL數據庫表合併##################################

1）建立組涉及4個表

namespaces,routes,members,notification_settings

驗證：假設建立空組B，SQL執行順序以下

　　1.1）namespaces 先爲組B分配namespace_id=11，path=B

　　1.2）routes爲組B指定path,  routes_id=15,  routes.source_id=namespaces_id ,routes.path=namespaces.path

　　1.3) members 爲組B建立member_id=10, members.source_id=namespaces.id

　　1.4) notification_settings爲組B建立id=10, notification_settings.source_id=namespace_id

測試，修改namespace_id以及其餘關聯id,驗證可否生成新的組

要點：routes.source_id=namespaces.id=members.source_id=notification_settings.id

　　   routes.path=namespaces.path

####################測試實例############################

  INSERT INTO gitlabhq_production.namespaces values(11,'B','B',NULL,'2018-11-24 04:09:59','2018-11-24 04:09:59','Group','',NULL,0,20,0,NULL,1,NULL,NULL);
  INSERT INTO gitlabhq_production.routes values(15,11,'Namespace','B','2018-11-24 04:09:59','2018-11-24 04:09:59');
  INSERT INTO gitlabhq_production.members values(10,50,11,'Namespace',1,3,'GroupMember','2018-11-24 04:09:59','2018-11-24 04:09:59',NULL,NULL,NULL,NULL,NULL,NULL);
  INSERT INTO gitlabhq_production.notification_settings values(10,1,11,'Namespace',3,'2018-11-24 04:09:59','2018-11-24 04:09:59','{"new_note":false,"new_issue":false,"reopen_issue":false,"close_issue":false,"reassign_issue":false,"new_merge_request":false,"reopen_merge_request":false,"close_merge_request":false,"reassign_merge_request":false,"merge_merge_request":false,"failed_pipeline":false,"success_pipeline":false}');

###################################################

結果：測試經過

2)建立用戶涉及3個表

users,namespaces,routes

驗證：假設建立用戶UserA,密碼123qweASD!，SQL執行順序以下

　　1.1) users爲用戶建立user_id, email，authentication_token具備惟一性

　　1.2)namespaces建立namespace_id ,namespaces.owner_id=users.id ，namespaces.name=users.username    namespace的name默認=path

　　1.3)routes建立routes_id, routes.source_id=namespaces.id,routes.path=namespaces.path

　　1.4) users更新密碼   

　　密碼會過時，要執行UPDATE gitlabhq_production.users set password_expires_at=NULL where id= 'x'

測試：修改相關關聯id，是否會生成新組

　　要點：users_id=owner_id      routes.source_id=namespaces.id

###############################測試實例####

  INSERT INTO gitlabhq_production.users values(4,'zhangyu512@sohu.com','$2a$10$AibtkRnWuicxnioQ5enWReVnO8upwoVED94Mm3E/U6RQLstggLexm','c601f60659a30dcb49cfd978f5bd0f73b3591cdb562a100f6a0c4d92f65d0ccd','2018-11-24 05:35:52',NULL,0,NULL,NULL,NULL,NULL,'2018-11-24 05:35:52','2018-11-24 05:35:52','zhang',0,10,'','','','CCEPy4qr-kGUUNCzdzmi',2,NULL,0,NULL,'zhangyu',1,0,'active',1,NULL,1,NULL,NULL,NULL,'2018-11-24 05:35:52',NULL,NULL,0,'','zhangyu512@sohu.com',0,0,NULL,NULL,NULL,NULL,0,NULL,'',0,0,NULL,0,0,NULL,NULL,0,0,NULL,'c8ksp6mszdo9is7r9w4cq3t8f',NULL);
  INSERT INTO gitlabhq_production.namespaces values(15,'zhangyu','zhangyu',4,'2018-11-24 05:35:52','2018-11-24 05:35:52',NULL,'',NULL,0,20,0,NULL,NULL,NULL,NULL);
  INSERT INTO gitlabhq_production.routes values(19,15,'Namespace','zhangyu','2018-11-24 05:35:52','2018-11-24 05:35:52');
UPDATE gitlabhq_production.users set encrypted_password='$2a$10$6iZkb/N/mJ4lmJIjUlOSvuRZeAH26yyIY.7OB7RfrsykJ3mpiFDRW',reset_password_token=NULL,reset_password_sent_at=NULL,updated_at='2018-11-24 05:36:06',password_expires_at='2018-11-24 05:36:06' where id= '4'

UPDATE gitlabhq_production.users set encrypted_password='$2a$10$UGo9UflSX9LC9rGGtJqD/Om5b0xBhwPmgBh2Lvapz5T2XE34RK6oO',updated_at='2018-11-24 05:37:13' where id= '4'
UPDATE gitlabhq_production.users set password_expires_at=NULL where id= '4'

##########################################　

測試結果：經過

3）將用戶添加至組

涉及2個表 members,notification_settings

　　3.1)members表爲用戶和組創建關聯    members.source_id=namespaces.id    members.user_id=users.id

　　3.2)notification_settings爲用戶和組創建關聯  notificatiion_settings.user_id=users.id   notification_setting.source_id=namespaces.id

 

 

##################測試實例##########

 INSERT INTO gitlabhq_production.members values(13,10,12,'Namespace',2,3,'GroupMember','2018-11-24 05:59:22','2018-11-24 05:59:22',1,NULL,NULL,NULL,NULL,NULL);
 INSERT INTO gitlabhq_production.notification_settings values(13,2,12,'Namespace',3,'2018-11-24 05:59:22','2018-11-24 05:59:22','{"new_note":false,"new_issue":false,"reopen_issue":false,"close_issue":false,"reassign_issue":false,"new_merge_request":false,"reopen_merge_request":false,"close_merge_request":false,"reassign_merge_request":false,"merge_merge_request":false,"failed_pipeline":false,"success_pipeline":false}');

###############################

測試結果：經過

 4）建立project（建立是直接指認組）

涉及project_features,project,routes,events,project_authorizations,users

　　4.1)project_features 建立2個id ,  指定snippets_access_level=0

　　4.2)projects建立project_id ,name,path

　　4.3)routes 建立 routes_id,routes.source_id=projects.id

　　4.4)project_features 設置第二個的project_features.project_id=projects.id

　　4.5)project_authorizations插入 user_id=1,project_id=projects.id

　　　　　　　　　　　　    插入user_id=組.用戶的ID，project_id=projects.id

　　4.6)users  設置對應用戶authorized_projects_populated=1

　　4.7)projects設置has_external_issue_tracker=0  has_external_wiki=0

######## 測試實例######################

 INSERT INTO gitlabhq_production.project_features values(9,NULL,20,20,20,20,20,'2018-11-24 06:21:40','2018-11-24 06:21:40',20);
 UPDATE gitlabhq_production.project_features set snippets_access_level=0 where id= '9'
 INSERT INTO gitlabhq_production.project_features values(10,NULL,20,20,20,20,20,'2018-11-24 06:22:44','2018-11-24 06:22:44',20);
 UPDATE gitlabhq_production.project_features set snippets_access_level=0 where id= '10'
 INSERT INTO gitlabhq_production.projects values(1,'A','A','','2018-11-24 06:22:44','2018-11-24 06:22:44',1,11,NULL,NULL,20,0,NULL,'none',0                  ,0,NULL,NULL,0,NULL,NULL,1,'BrGVBu9duy_xm5tfpKc6',NULL,1,3600,0,1,NULL,NULL,1,0,NULL,'default',0,NULL,NULL,'',0);
 INSERT INTO gitlabhq_production.routes values(21,1,'Project','B/A','2018-11-24 06:22:44','2018-11-24 06:22:44');
 UPDATE gitlabhq_production.projects set last_activity_at='2018-11-24 06:22:44' where id= '1'
 UPDATE gitlabhq_production.project_features set project_id=1 where id= '10'
 INSERT INTO gitlabhq_production.events values(9,NULL,NULL,NULL,NULL,1,'2018-11-24 06:22:45','2018-11-24 06:22:45',1,1);
 INSERT INTO gitlabhq_production.project_authorizations values(2,1,1,50);
 INSERT INTO gitlabhq_production.project_authorizations values(1,5,1,10);
UPDATE gitlabhq_production.users set authorized_projects_populated=1 where id= '5'
UPDATE gitlabhq_production.projects set has_external_issue_tracker=0 where id= '1'
 UPDATE gitlabhq_production.projects set has_external_wiki=0 where id= '1'

######################################

測試結果：經過