Jenkins系列之-—07 實現SQL腳本批量執行

時間 2019-11-16

標籤 jenkins 系列實現 sql 腳本批量執行欄目自動化構建简体版

原文原文鏈接

公司內部推廣DevOps，全部目前在維護階段和開發階段項目所有配置上了自動發佈。採用Jenkins+SVN+ANT，以後批量執行SQL語句的實現提上日程

1、環境

Linux環境
安裝ANT工具，且下載ant擴展包---ant-contrib-1.0b3.jar，maven連接
下載oracle鏈接ojdbc5.jar包。

2、思路

步驟1：jenkins 從SVN下載全部SQL腳本sql

步驟2：編寫shell腳本：獲取今天的sql腳本，若是是今日已經執行過且執行成功的文件（即存在於bak文件夾中的），就不重複執行；shell

步驟3：編寫ant腳本：對待執行腳本排序後，循環執行每條sql語句，執行成功的sql腳本備份到bak文件夾，方便後續執行時排除重複。數據庫

3、具體實現

3.1 執行shell

year=`date +%Y`
today=`date +%Y%m%d`
sqlfolder=${WORKSPACE}/svn/${year}/${today}
bakfolder=${WORKSPACE}/sqlbak/${year}/${today}
logfolder=${WORKSPACE}/sqllog

if [ -d "runSqlInFolder" ];then
echo '清空/runSqlInFolder文件夾'
rm -rf runSqlInFolder/*
else
mkdir -p ${WORKSPACE}/runSqlInFolder
fi

if [ ! -d "${logfolder}" ];then
  mkdir ${logfolder}
else
  echo dir ${logfolder} exist
fi

echo '提交到SVN的sql語句中，今日未執行過的拷貝到runSqlInFolder文件夾'
cp -r ${sqlfolder}/* runSqlInFolder
find runSqlInFolder/ -name '*.jar' |xargs rm -r
find runSqlInFolder/ -type d | grep .svn$ | xargs rm -r
if [ -d "${bakfolder}" ];then
echo "備份文件夾已存在"
cd ${bakfolder}
find * -type f -name *.sql|xargs -i rm -f ../../../runSqlInFolder/{}
else
echo "備份文件夾不存在，建立備份文件夾： ${bakfolder}"
mkdir -p ${bakfolder}
fi

View Code

3.2 Invoke ant

3.2.1 定義一個target：runSqlInFolder

使用try catch包裹for 循環，for循環中調用execSQL標籤（自定義公共方法塊，即宏），順序逐條執行SQL腳本。apache

    <taskdef resource="net/sf/antcontrib/antlib.xml" classpath="${lib}/ant-contrib-1.0b3.jar"/>
    <target name="runSqlInFolder">
        <echo>Run the SQL at Folder: ${sqlfolder}</echo>
        <echo>DB Host: ${v7uatdb.host}</echo>
        <echo>DB Name: ${v7uatdb.name}</echo>
        <echo>DB User: ${v7uatdb.user}</echo>
        <trycatch property="errMsg">
            <try>            
                <for param="folder">
                    <path>
                        <sort xmlns:rcmp="antlib:org.apache.tools.ant.types.resources.comparators">
                            <dirset dir="${sqlfolder}" includes="*" />                        
                        </sort>
                    </path>
                    <sequential>
                    <echo>SQL Folder: @{folder}</echo>    
                    <for param="file">
                        <path>
                            <sort xmlns:rcmp="antlib:org.apache.tools.ant.types.resources.comparators">
                                <fileset dir="@{folder}" includes="*/*.sql" casesensitive="false"/>                                            
                            </sort>
                        </path>
                        <sequential>
                        <echo>SQL: @{file}</echo>                             
                        <execsql
                            dbhost="${v7uatdb.host}"    
                            dbport="${v7uatdb.port}"    
                            dbname="${v7uatdb.name}" 
                            dbuser="${v7uatdb.user}" 
                            dbpwd="${v7uatdb.pwd}"
                            sqlfile="@{file}"
                            logfile="${Sqllogfile}"/>
                        </sequential>
                        <!--<move file="@{file}" todir="${sqlbakdir}/@{folder}"/>
                        folder 包含路徑和文件名，因此直接複製file還有點問題，須要截取文件名--目前待研究 -->
                    </for>
                    <move file="@{folder}" todir="${sqlbakdir}"/> 
                    </sequential>    
                </for>
                <echo>Finished running all SQL</echo>
                <echo>File moved to backup folder:</echo>
                <echo>${sqlbakdir}</echo>
            </try>
            <catch>
                <echo>Error found when running SQL</echo>
                <echo>Log file can be found in:</echo>
                <echo>${sqlbakdir}/err</echo>
                <move file="${Sqllogfile}" todir="${sqlbakdir}/err"/>
                <fail>Error Occur</fail>
            </catch>
            <finally>
            </finally>
        </trycatch>
    </target>

3.2.2 定義execsql標籤

經過sql標籤執行sql文件，經過record標籤記錄這段執行的日誌而且輸出。oracle

注意：若是執行procedure就須要設置delimiter，本例中經過SQL文件的命名來區分是不一樣SQL仍是procedure。maven

    <macrodef name="execsql" description="Run single SQL file.">
        <attribute name="dbhost" description="Host Name/ IP of the DB"/>
        <attribute name="dbport" description="DB Port"/>
        <attribute name="dbname" description="DB name"/>
        <attribute name="dbuser" description="DB User name"/>
        <attribute name="dbpwd" description="DB Password"/>
        <attribute name="sqlfile" description="SQL file to be run"/>
        <attribute name="logfile" default="sql.log" description="Log file"/>
        <sequential>
            <echo>Log file @{logfile}</echo>                    
            <record name="@{logfile}" action="start"/>    
            <if>
                <contains string="@{sqlfile}" substring="PROCEDURE"/>
                <then>
                    <sql driver="${oracleDriver}"
                        url="jdbc:oracle:thin:@@@{dbhost}:@{dbport}:@{dbname}"
                        userid="@{dbuser}"
                        password="@{dbpwd}"
                        classpathref="classpath"
                        encoding="${encoding}"
                        print="true"
                        autocommit="true"
            delimiter="/"
            delimitertype="row">
                        <transaction src="@{sqlfile}"/>
                    </sql>
                </then>
                <else>
                    <sql driver="${oracleDriver}"
                        url="jdbc:oracle:thin:@@@{dbhost}:@{dbport}:@{dbname}"
                        userid="@{dbuser}"
                        password="@{dbpwd}"
                        encoding="${encoding}"
                        classpathref="classpath"
                        autocommit="true"
                        print="true">
                        <transaction src="@{sqlfile}"/>
                    </sql>
                </else>
            </if>
            <record name="@{logfile}" action="stop"/> 
        </sequential>
    </macrodef>

View Code

更新後ide

    <macrodef name="execsql" description="Run single SQL file.">
        <attribute name="dbhost" description="Host Name/ IP of the DB"/>
        <attribute name="dbport" description="DB Port"/>
        <attribute name="dbname" description="DB name"/>
        <attribute name="dbuser" description="DB User name"/>
        <attribute name="dbpwd" description="DB Password"/>
        <attribute name="sqlfile" description="SQL file to be run"/>
        <attribute name="logfile" default="sql.log" description="Log file"/>
        <sequential>
            <echo>Log file @{logfile}</echo>    
            <record name="@{logfile}" action="start"/>    
            <if>
                <contains string="@{sqlfile}" casesensitive="no" substring="PROCEDURE"/>
                <then>
                    <sql driver="${oracleDriver}"
                        url="jdbc:oracle:thin:@@@{dbhost}:@{dbport}:@{dbname}"
                        userid="@{dbuser}"
                        password="@{dbpwd}"
                        classpathref="classpath"
                        encoding="${encoding}"
                        print="true"
                        autocommit="true"
                        delimiter="/"
                        delimitertype="row">
                        <transaction src="@{sqlfile}"/>
                    </sql>
                </then>
            <elseif>
                <contains string="@{sqlfile}" casesensitive="no" substring="DECLARE"/>
                <then>
                    <sql driver="${oracleDriver}"
                        url="jdbc:oracle:thin:@@@{dbhost}:@{dbport}:@{dbname}"
                        userid="@{dbuser}"
                        password="@{dbpwd}"
                        classpathref="classpath"
                        encoding="${encoding}"
                        print="true"
                        autocommit="true"
                        delimiter=";;">
                        <transaction src="@{sqlfile}"/>
                    </sql>
                </then>
             </elseif>    
            <else>
                    <sql driver="${oracleDriver}"
                        url="jdbc:oracle:thin:@@@{dbhost}:@{dbport}:@{dbname}"
                        userid="@{dbuser}"
                        password="@{dbpwd}"
                        encoding="${encoding}"
                        classpathref="classpath"
                        autocommit="true"
                        print="true">
                        <transaction src="@{sqlfile}"/>
                    </sql>
            </else>
            </if>
            <record name="@{logfile}" action="stop"/> 
        </sequential>
    </macrodef>

View Code

3.3 SVN文件夾設置

3.4 執行效果

4、遇到的問題處理

4.1【問題】若是執行的SQL語句中，有中文，執行完在數據庫裏是亂碼

GBK、UTF-8是最經常使用的兩種編碼方式，是不少編碼中的兩種，也是兩種風格(ANSI和UNICODE)中的表明
因爲公司數據庫採用GBK編碼，因此將encoding改成GBK，亂碼問題解決svn

<property name="encoding" value="GBK" />工具

4.2【問題】若是執行的SQL語句中，有declare，聲明變量的，執行declare過程當中會報錯

delete zssys.WEB_APP_TEMPLATE where C_INTERFACE_ID = '0000-GDCARVA';;
declare
  response clob;
begin
  response := 'aaa';

insert into zssys.WEB_APP_TEMPLATE (C_PK_ID, C_INTERFACE_ID, C_SYSCODE, C_APP_NAME, C_TEMPLATE_REQUEST, C_TEMPLATE_RESPONSE)
values (SYS_GUID(), '0000-GDCARVA', '*', 'test', null, response);
end;