Mongodb基於oplog恢復至任意時間

時間  2019-11-13
標籤 mongodb 基於 oplog 恢復 任意 時間 欄目 MongoDB 简体版
原文   原文鏈接

 

 

 

背景：python

　　最近後端基於mongo的項目愈來愈多，MySQL基於冷備份+binlog能夠恢復至任意時間點，那麼mongo是否有一樣的功能呢？通過調研發現能夠經過dump+oplog能夠實現粒度更細緻的恢復。linux

官方文檔關於oplog的說明 ： <a class="mongo-link" href="https://docs.mongodb.com/manual/core/replica-set-oplog/"></a>

 

　　如下是實驗步驟，附腳本內容。mongodb

一、 製做測試數據而且獲取冷備json

二、模擬線上兩個冷備份之間的數據寫入後端

三、備份oplogbash

四、冷備+oplog恢復至臨時實例，確認沒問題後，經過臨時實例恢復到主庫  app

一、oop

dump比較重要的幾個參數
#mongodump --help
Usage:
  mongodump <options>

Export the content of a running server into .bson files.

Specify a database with -d and a collection with -c to only dump that database or collection.

See http://docs.mongodb.org/manual/reference/program/mongodump/ for more information.


connection options:
  -h, --host=<hostname>                                     mongodb host to connect to (setname/host1,host2 for replica sets)
      --port=<port>                                         server port (can also use --host hostname:port)
 
authentication options:
  -u, --username=<username>                                 username for authentication
  -p, --password=<password>                                 password for authentication
      --authenticationDatabase=<database-name>              database that holds the user's credentials
      --authenticationMechanism=<mechanism>                 authentication mechanism to use
 
      --gzip                                                compress archive our collection output with Gzip
      --oplog                                               use oplog for taking a point-in-time snapshot

 

 

　　
 

 
 
test_1234:PRIMARY> show dbs
admin  0.000GB
local  2.724GB
test_1234:PRIMARY> use test
switched to db test
test_1234:PRIMARY> for (var i=0;i<=5000;i++)(db.test.insert({"test":i}))
WriteResult({ "nInserted" : 1 })
test_1234:PRIMARY> db.test.count()
5001
當前test.test有5001條數據，獲取冷備份並查看當前時間戳
 
  
 
#date +%s
1553499626測試
 
 
 
 
 
 

 

 
 

 
 

二、 這個時候模擬下冷備以後的數據寫入，刪除一些數據並記錄oplogfetch
 

 
 

 
 
test_1234:PRIMARY> use test
switched to db test
test_1234:PRIMARY> 
test_1234:PRIMARY> 
test_1234:PRIMARY> show tables
test
test_1234:PRIMARY> for (var i=0;i<=5000;i++)(db.after.insert({"after":i}))
WriteResult({ "nInserted" : 1 })
test_1234:PRIMARY> db.test.remove({"test":{$gt:1500}})
WriteResult({ "nRemoved" : 3500 })
test_1234:PRIMARY> db.test.count()
1501
test_1234:PRIMARY> db.after.count()
5001
test_1234:PRIMARY> 
 
 
#date +%s
1553499686
 

 

　　
 

 
 

三、備份oplog
 

 
 
./mongoha_dump_oplog.sh 

options:
 -S      [Start-BSON-Timestamp]
 -E      [End-BSON-Timestamp] 
 is running like mongoha_dump_oplog.sh -S  -E 

 #
 
 
 
 

 

　 
 

 
 
./mongoha_dump_oplog.sh -S 1553499626 -E 1553499686
2019-03-25T15:42:07.579+0800 writing local.oplog.rs to 
2019-03-25T15:42:10.576+0800 local.oplog.rs 0
2019-03-25T15:42:13.575+0800 local.oplog.rs 0
2019-03-25T15:42:16.575+0800 local.oplog.rs 0
2019-03-25T15:42:19.575+0800 local.oplog.rs 0
2019-03-25T15:42:22.575+0800 local.oplog.rs 0
2019-03-25T15:42:25.576+0800 local.oplog.rs 0
2019-03-25T15:42:26.267+0800 local.oplog.rs 8507
2019-03-25T15:42:26.267+0800 done dumping local.oplog.rs (8507 documents)
Mongo oplog backup [2019-03-25 15:40:26 - 2019-03-25 15:41:26] -->Sucess

 

 

　　
 

 
 

 
 

四、 新建臨時實例利用剛纔冷備+oplog恢復，確認數據沒問題以後，恢復至主庫。
 

 
 
1）新建臨時端口
mongo_install.sh -T 1 -P 6666 -M 1
 2019-03-25 15:32:57 start install
mongodb-linux-x86_64-rhel62-3.4.0/README
mongodb-linux-x86_64-rhel62-3.4.0/THIRD-PARTY-NOTICES
mongodb-linux-x86_64-rhel62-3.4.0/MPL-2
mongodb-linux-x86_64-rhel62-3.4.0/GNU-AGPL-3.0
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongodump
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongorestore
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongoexport
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongoimport
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongostat
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongotop
mongodb-linux-x86_64-rhel62-3.4.0/bin/bsondump
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongofiles
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongooplog
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongoreplay
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongoperf
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongod
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongos
mongodb-linux-x86_64-rhel62-3.4.0/bin/mongo
/usr/local/mongodb is exists ..
mongo data,log dir create sucess


2019-03-25 15:32:57 ----> mongodb6666 start sucess 
note: noprealloc may hurt performance in many applications
about to fork child process, waiting until server is ready for connections.
forked process: 8394
child process started successfully, parent exiting

2）恢復冷備+oplog
 
 
mongoha_fullrestore.sh
 
 
options:
 -P specify the port
 -p specify the restore db path ,must abspath
 is running like mongoha_backup.sh -P 27017 -p /test
 
 
mongoha_fullrestore.sh -P 6666 -p /data1/backup/mongodb/2019/03/25/_2019-03-25_15\:15/
at 2019-03-25_15:34 to do restore for mongo
2019-03-25T15:34:48.505+0800 preparing collections to restore from
2019-03-25T15:34:48.506+0800 reading metadata for test.test from /data1/backup/mongodb/2019/03/25/_2019-03-25_15:15/test/test.metadata.json
2019-03-25T15:34:48.544+0800 restoring test.test from /data1/backup/mongodb/2019/03/25/_2019-03-25_15:15/test/test.bson
2019-03-25T15:34:48.573+0800 restoring indexes for collection test.test from metadata
2019-03-25T15:34:48.573+0800 finished restoring test.test (5001 documents)
2019-03-25T15:34:48.573+0800 replaying oplog
2019-03-25T15:34:48.573+0800 done
Mongo restore() -->Sucess 
 
 
冷備恢復成功
 
 
臨時節點查看下數據
 
 
 
 
恢復到冷備以前的數據了，可是以前刪除的數據及後插入的after document還未恢復，這時候咱們恢復oplog
 
 
 
 
 
./mongoha_restore_oplog.sh
 
 
options:
 -P specify the port
 -p specify the restore db path ,must abspath
 is running like ./mongoha_restore_oplog.sh -P 臨時端口 -p /data1/backup/mongodb/oplog/2019/03/25/local/oplog.rs.bson
 
 
 再次查看臨時庫數據
 
 
 
 
臨時庫數據以恢復，這時候就能夠備份出來恢復至線上了。
 

 

 
 

 
 

  附mongoha_fullbackup.sh腳本
 

 
 
 #!/bin/bash
#auth liding@zlongame.com



dat=`date "+%Y-%m-%d_%H:%M"`
day=`date "+%Y/%m/%d" -d now`
#localip=`ip a | grep "\binet\b" |awk '{print $2}'| egrep -e "^10|172" | awk -F"/" '{print $1; exit;}'`
basedir="/data1/backup/mongodb/$day"



helpfunc(){
        echo 
        echo "options:"
        echo " -P      specify the port"
        echo " is running like mongoha_backup.sh -P 27017 -D test,test1"
}

while getopts "P:" Option
do
        case $Option in
                P) ports=$OPTARG;;
                *) helpfunc; exit 1; ;;
        esac
done

dbname=`echo $dbname | sed 's/,/ /g'`
dobackup(){
        echo "at $dat to do bakcup for mongo$i"
        mkdir -p "$basedir"
        #mongodump -u$bacuser -p$pswd --port $i --oplog  -o "$basedir"/mongo"$i"
        if /usr/local/mongodb/bin/mongodump -h 127.0.0.1 --port $ports   --oplog  -o "$basedir"/"$i"_$dat;then
                echo "Mongo Backup($i) -->Sucess"
        else
                echo "Mongo Backup($i) -->faild"
        fi


}





check(){
person=`whoami`
        if [ "$person" != "root" ]; then
                echo "the user is now $person, checkout to 'root' to continue"
                exit 1
        fi

        if [ -z "$ports" ] ;then
                helpfunc
                exit 1
        fi

}

cleanbackup(){
rmdate=`date "+%Y/%m/%d" -d "7 days ago"`
if [ -d $1 ] ;then
        rm -rf $1
        echo "rm -$1"
elif [ -d $basedir/$rmdate ] ;then
        rm -rf $basedir/$rmdate
        echo "rm -rf $basedir/$rmdate"
else
        rm -rf $basedir/mongo$1
        echo "rm -rf $basedir/mongo$1"
        exit 1
fi
}
main(){

check
dobackup
}
main

 

 

　　
 

  附mongoha_fullrestore.sh腳本 
 

 
 
mongoha_fullrestore.sh 
#!/bin/bash
 


dat=`date "+%Y-%m-%d_%H:%M"`
day=`date "+%Y/%m/%d" -d now`
#localip=`ip a | grep "\binet\b" |awk '{print $2}'| egrep -e "^10|172" | awk -F"/" '{print $1; exit;}'`
basedir="/data1/backup/mongodb/$day"



helpfunc(){
        echo 
        echo "options:"
        echo " -P      specify the port"
        echo " -p      specify the restore db path ,must abspath"
        echo " is running like mongoha_backup.sh -P 27017  -p $backdir/test"
}

while getopts "P:p:" Option
do
        case $Option in
                P) ports=$OPTARG;;
                p) path=$OPTARG;;
                *) helpfunc; exit 1; ;;
        esac
done

#mongorestore -h 127.0.0.1:27017 -d test test/
dorestore(){
        echo "at $dat to do restore for mongo${dbname}"
        if /usr/local/mongodb/bin/mongorestore --drop --oplogReplay -h 127.0.0.1:${ports} $path;then
                echo "Mongo restore(${dbname}) -->Sucess"
        else
                echo "Mongo restore(${dbname}) -->faild"
        fi


}





check(){
person=`whoami`
        if [ "$person" != "root" ]; then
                echo "the user is now $person, checkout to 'root' to continue"
                exit 1
        fi

        if [ -z "$ports" ] ;then
                helpfunc
                exit 1
        fi

}

main(){

check
dorestore
}
main

 

 

  附mongoha_dump_oplog.sh 腳本 
 

 
 
cat mongoha_dump_oplog.sh 
#!/bin/bash
 


dat=`date "+%Y-%m-%d_%H:%M"`
day=`date "+%Y/%m/%d" -d now`



helpfunc(){
        echo 
        echo "options:"
        echo " -S      [Start-BSON-Timestamp]"
        echo " -E      [End-BSON-Timestamp] "
        echo " is running like mongoha_dump_oplog.sh -S  -E "
}
if [ $# -lt 2 ] ;
    then
        helpfunc
         exit 1
else

    while getopts "S:E:" Option
    do
            case $Option in
                    S) start_t=$OPTARG;;
                    E) stop_t=$OPTARG;;
                    *) helpfunc; exit 1; ;;
            esac
    done
fi

start_time="Timestamp(${start_t}, 0)"
stop_time="Timestamp(${stop_t}, 0)"
date_now=`date "+%Y%m%d%H_%M" -d now`
backdir="/data1/backup/mongodb/oplog"
#dump
function data_fetch(){
mkdir -p $backdir
query='{ts:{$gt:'$start_time',$lt: '$stop_time' }}'
flags='-d local -c oplog.rs'
if /usr/local/mongodb/bin/mongodump $flags --query "$query" -o $backdir/$date_now;then
    echo "Mongo oplog backup [`date -d @${start_t} "+%Y-%m-%d %H:%M:%S"` - `date -d @${stop_t} "+%Y-%m-%d %H:%M:%S"`] -->Sucess"
else
    echo "Mongo oplog backup [`date -d @${start_t} "+%Y-%m-%d %H:%M:%S"` - `date -d @${stop_t} "+%Y-%m-%d %H:%M:%S"`] -->Faild"
fi

}

main(){
data_fetch
}
main

 

 

　　 附mongoha_restore_oplog.sh 腳本
 

 
 
cat mongoha_restore_oplog.sh 
#!/bin/bash
 


dat=`date "+%Y-%m-%d_%H:%M"`
day=`date "+%Y/%m/%d" -d now`
#localip=`ip a | grep "\binet\b" |awk '{print $2}'| egrep -e "^10|172" | awk -F"/" '{print $1; exit;}'`
backdir="/data1/backup/mongodb/oplog/$day"



helpfunc(){
        echo 
        echo "options:"
        echo " -P      specify the port"
        echo " -p      specify the restore db path ,must abspath"
        echo " is running like $0 -P 臨時端口  -p $backdir/local/oplog.rs.bson"
}

if [ $# -lt 2 ] ;
    then
        helpfunc
         exit 1
else
    while getopts "P:p:" Option
    do
            case $Option in
                    P) port=$OPTARG;;
                    p) path=$OPTARG;;
                    *) helpfunc; exit 1; ;;
            esac
    done
fi

function data_restore(){
flags='-d local -c oplog.rs'
if /usr/local/mongodb/bin/mongorestore --port $port $flags $path;then
    echo "Mongo restore[$port] -->Sucess"
else
    echo "Mongo restore[$port] -->Faild"
fi

}
data_restore

 



        

            

        	





    

		

            
相關文章

            

                

                

                    

                

            


            
相關標籤/搜索

            
        


    

        

            

        

        

        
        

        

        

    




	

    





    

    	

    

        每日一句
    

    
    
    	每一个你不满意的现在，都有一个你没有努力的曾经。
    







    

    


    



    

        本站公眾號
    

    

           歡迎關注本站公眾號,獲取更多信息

        
    




    

    




	


	







    

        聯繫我們
        最近搜索
        最新文章
        
        沪ICP备13005482号-10

            MyBatis教程
            SQL 教程
            MySQL教程
            Java 教程
            Thymeleaf 教程
            Hibernate教程
            Spring教程 
            Redis教程