Azure Data Factory（三）集成 Azure Devops 實現CI/CD

時間 2020-11-12

標籤 html git web shell 編程 json windows 架構 ide 工具欄目 HTML 简体版

原文原文鏈接

一，引言

　　因爲上一節文章內容過長，沒法分享Azure Data Factory 的持續集成，持續發佈。今天將着重介紹一下在使用 Azure DevOps Pipeline 發佈，自動進行持續集成，而且已自動化發佈，將Azure Data Factory 部署到多個環境中。html

　　其實你們也沒必要驚訝，這裏的部署其實也沒有多麼神祕的，咱們在ADF中的 master 分支發佈以後，其實會將ADF中全部的配置信息打包，編譯到adf_master 分支下面，若是你們仔細看過發佈以後的代碼，就很很容易發現，都是一些 ARM 模板資源。以當前我建立的demo爲例git

這是就有人在問了，什麼是ARM模板？這裏就先見到你的概規一下：web

　　針對於Azure上一些基礎設施資源的部署，能夠經過運用基礎架構即代碼，但是實現部署自動化，在代碼中定義須要部署的基礎架構資源。將這些基礎架構資源代碼變成項目的一部分，與應用程序代碼同樣。經過代碼管理工具，管理起來，方便部署，經過代碼的方式建立、刪除資源。而ARM模板就是基礎架構即代碼的一種形式（另一種是 Terraform）,該模板是一個定義項目基礎結構和配置的 JavaScript 對象表示法 (JSON) 文件。該模板使用聲明性語法，使你能夠指明要部署的內容，而不須要編寫一系列編程命令來建立內容。在該模板中，指定要部署的資源以及這些資源的屬性。shell

說人話，就是ARM模板中描述了咱們須要部署的雲資源以及其資源所須要的參數，好比說經過使用ARM不是一臺VM，那麼ARM模板中就描述了VM資源以及其建立VM所須要的必要的參數。編程

回到ADF中，也就是說咱們最終經過master分支發佈到 adf_master 分支的代碼，其實就是一堆描述ADF資源以及ADF配置的屬性及參數。咱們今天的內容也就是經過ARM去實現ADF的 UAT，PRO 環境的部署。json

--------------------我是分割線--------------------windows

1，Azure Data Factory（一）入門簡介

2，Azure Data Factory（二）複製數據

3，Azure Data Factory（三）集成 Azure Devops 實現CI/CD

4，Azure Data Factory（四）集成 Logic App 的郵件通知提醒

二，正文

1，建立ADF的 UAT 環境

咱們經過資源名+UAT的方式來模擬測試環境，以下新建立了一個叫 "ADF-CnBateBlogWeb-UAT" 的 Azure Data Factory 和兩個UAT 環境的 Blob Storage架構

」cnbateblogwebaccount1dev「做爲UAT 環境的數據源，咱們也爲UAT環境建立叫「resformfolder」的容器ide

「cnbateblogwebaccount2dev」做爲UAT 環境的目標源，咱們也爲UAT環境建立叫「restofolder」的容器工具

2，初始化部署先後腳本

實際中，咱們在部署ADF 甚至其餘項目代碼時，都是須要將AFD的觸發器中止，等到部署完成後，咱們就得重啓ADF中配置的觸發器。微軟有幫助咱們提供了部署先後可使用的腳本

  1 param
  2 (
  3     [parameter(Mandatory = $false)] [String] $armTemplate,
  4     [parameter(Mandatory = $false)] [String] $ResourceGroupName,
  5     [parameter(Mandatory = $false)] [String] $DataFactoryName,
  6     [parameter(Mandatory = $false)] [Bool] $predeployment=$true,
  7     [parameter(Mandatory = $false)] [Bool] $deleteDeployment=$false
  8 )
  9 
 10 function getPipelineDependencies {
 11     param([System.Object] $activity)
 12     if ($activity.Pipeline) {
 13         return @($activity.Pipeline.ReferenceName)
 14     } elseif ($activity.Activities) {
 15         $result = @()
 16         $activity.Activities | ForEach-Object{ $result += getPipelineDependencies -activity $_ }
 17         return $result
 18     } elseif ($activity.ifFalseActivities -or $activity.ifTrueActivities) {
 19         $result = @()
 20         $activity.ifFalseActivities | Where-Object {$_ -ne $null} | ForEach-Object{ $result += getPipelineDependencies -activity $_ }
 21         $activity.ifTrueActivities | Where-Object {$_ -ne $null} | ForEach-Object{ $result += getPipelineDependencies -activity $_ }
 22         return $result
 23     } elseif ($activity.defaultActivities) {
 24         $result = @()
 25         $activity.defaultActivities | ForEach-Object{ $result += getPipelineDependencies -activity $_ }
 26         if ($activity.cases) {
 27             $activity.cases | ForEach-Object{ $_.activities } | ForEach-Object{$result += getPipelineDependencies -activity $_ }
 28         }
 29         return $result
 30     } else {
 31         return @()
 32     }
 33 }
 34 
 35 function pipelineSortUtil {
 36     param([Microsoft.Azure.Commands.DataFactoryV2.Models.PSPipeline]$pipeline,
 37     [Hashtable] $pipelineNameResourceDict,
 38     [Hashtable] $visited,
 39     [System.Collections.Stack] $sortedList)
 40     if ($visited[$pipeline.Name] -eq $true) {
 41         return;
 42     }
 43     $visited[$pipeline.Name] = $true;
 44     $pipeline.Activities | ForEach-Object{ getPipelineDependencies -activity $_ -pipelineNameResourceDict $pipelineNameResourceDict}  | ForEach-Object{
 45         pipelineSortUtil -pipeline $pipelineNameResourceDict[$_] -pipelineNameResourceDict $pipelineNameResourceDict -visited $visited -sortedList $sortedList
 46     }
 47     $sortedList.Push($pipeline)
 48 
 49 }
 50 
 51 function Get-SortedPipelines {
 52     param(
 53         [string] $DataFactoryName,
 54         [string] $ResourceGroupName
 55     )
 56     $pipelines = Get-AzDataFactoryV2Pipeline -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
 57     $ppDict = @{}
 58     $visited = @{}
 59     $stack = new-object System.Collections.Stack
 60     $pipelines | ForEach-Object{ $ppDict[$_.Name] = $_ }
 61     $pipelines | ForEach-Object{ pipelineSortUtil -pipeline $_ -pipelineNameResourceDict $ppDict -visited $visited -sortedList $stack }
 62     $sortedList = new-object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSPipeline]
 63     
 64     while ($stack.Count -gt 0) {
 65         $sortedList.Add($stack.Pop())
 66     }
 67     $sortedList
 68 }
 69 
 70 function triggerSortUtil {
 71     param([Microsoft.Azure.Commands.DataFactoryV2.Models.PSTrigger]$trigger,
 72     [Hashtable] $triggerNameResourceDict,
 73     [Hashtable] $visited,
 74     [System.Collections.Stack] $sortedList)
 75     if ($visited[$trigger.Name] -eq $true) {
 76         return;
 77     }
 78     $visited[$trigger.Name] = $true;
 79     if ($trigger.Properties.DependsOn) {
 80         $trigger.Properties.DependsOn | Where-Object {$_ -and $_.ReferenceTrigger} | ForEach-Object{
 81             triggerSortUtil -trigger $triggerNameResourceDict[$_.ReferenceTrigger.ReferenceName] -triggerNameResourceDict $triggerNameResourceDict -visited $visited -sortedList $sortedList
 82         }
 83     }
 84     $sortedList.Push($trigger)
 85 }
 86 
 87 function Get-SortedTriggers {
 88     param(
 89         [string] $DataFactoryName,
 90         [string] $ResourceGroupName
 91     )
 92     $triggers = Get-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName
 93     $triggerDict = @{}
 94     $visited = @{}
 95     $stack = new-object System.Collections.Stack
 96     $triggers | ForEach-Object{ $triggerDict[$_.Name] = $_ }
 97     $triggers | ForEach-Object{ triggerSortUtil -trigger $_ -triggerNameResourceDict $triggerDict -visited $visited -sortedList $stack }
 98     $sortedList = new-object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSTrigger]
 99     
100     while ($stack.Count -gt 0) {
101         $sortedList.Add($stack.Pop())
102     }
103     $sortedList
104 }
105 
106 function Get-SortedLinkedServices {
107     param(
108         [string] $DataFactoryName,
109         [string] $ResourceGroupName
110     )
111     $linkedServices = Get-AzDataFactoryV2LinkedService -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName
112     $LinkedServiceHasDependencies = @('HDInsightLinkedService', 'HDInsightOnDemandLinkedService', 'AzureBatchLinkedService')
113     $Akv = 'AzureKeyVaultLinkedService'
114     $HighOrderList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService]
115     $RegularList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService]
116     $AkvList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService]
117 
118     $linkedServices | ForEach-Object {
119         if ($_.Properties.GetType().Name -in $LinkedServiceHasDependencies) {
120             $HighOrderList.Add($_)
121         }
122         elseif ($_.Properties.GetType().Name -eq $Akv) {
123             $AkvList.Add($_)
124         }
125         else {
126             $RegularList.Add($_)
127         }
128     }
129 
130     $SortedList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService]($HighOrderList.Count + $RegularList.Count + $AkvList.Count)
131     $SortedList.AddRange($HighOrderList)
132     $SortedList.AddRange($RegularList)
133     $SortedList.AddRange($AkvList)
134     $SortedList
135 }
136 
137 $templateJson = Get-Content $armTemplate | ConvertFrom-Json
138 $resources = $templateJson.resources
139 
140 #Triggers 
141 Write-Host "Getting triggers"
142 $triggersInTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/triggers" }
143 $triggerNamesInTemplate = $triggersInTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)}
144 
145 $triggersDeployed = Get-SortedTriggers -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
146 
147 $triggersToStop = $triggersDeployed | Where-Object { $triggerNamesInTemplate -contains $_.Name } | ForEach-Object { 
148     New-Object PSObject -Property @{
149         Name = $_.Name
150         TriggerType = $_.Properties.GetType().Name 
151     }
152 }
153 $triggersToDelete = $triggersDeployed | Where-Object { $triggerNamesInTemplate -notcontains $_.Name } | ForEach-Object { 
154     New-Object PSObject -Property @{
155         Name = $_.Name
156         TriggerType = $_.Properties.GetType().Name 
157     }
158 }
159 $triggersToStart = $triggersInTemplate | Where-Object { $_.properties.runtimeState -eq "Started" -and ($_.properties.pipelines.Count -gt 0 -or $_.properties.pipeline.pipelineReference -ne $null)} | ForEach-Object { 
160     New-Object PSObject -Property @{
161         Name = $_.name.Substring(37, $_.name.Length-40)
162         TriggerType = $_.Properties.type
163     }
164 }
165 
166 if ($predeployment -eq $true) {
167     #Stop all triggers
168     Write-Host "Stopping deployed triggers`n"
169     $triggersToStop | ForEach-Object {
170         if ($_.TriggerType -eq "BlobEventsTrigger") {
171             Write-Host "Unsubscribing" $_.Name "from events"
172             $status = Remove-AzDataFactoryV2TriggerSubscription -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
173             while ($status.Status -ne "Disabled"){
174                 Start-Sleep -s 15
175                 $status = Get-AzDataFactoryV2TriggerSubscriptionStatus -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
176             }
177         }
178         Write-Host "Stopping trigger" $_.Name
179         Stop-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name -Force
180     }
181 }
182 else {
183     #Deleted resources
184     #pipelines
185     Write-Host "Getting pipelines"
186     $pipelinesADF = Get-SortedPipelines -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
187     $pipelinesTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/pipelines" }
188     $pipelinesNames = $pipelinesTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)}
189     $deletedpipelines = $pipelinesADF | Where-Object { $pipelinesNames -notcontains $_.Name }
190     #dataflows
191     $dataflowsADF = Get-AzDataFactoryV2DataFlow -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
192     $dataflowsTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/dataflows" }
193     $dataflowsNames = $dataflowsTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40) }
194     $deleteddataflow = $dataflowsADF | Where-Object { $dataflowsNames -notcontains $_.Name }
195     #datasets
196     Write-Host "Getting datasets"
197     $datasetsADF = Get-AzDataFactoryV2Dataset -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
198     $datasetsTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/datasets" }
199     $datasetsNames = $datasetsTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40) }
200     $deleteddataset = $datasetsADF | Where-Object { $datasetsNames -notcontains $_.Name }
201     #linkedservices
202     Write-Host "Getting linked services"
203     $linkedservicesADF = Get-SortedLinkedServices -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
204     $linkedservicesTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/linkedservices" }
205     $linkedservicesNames = $linkedservicesTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)}
206     $deletedlinkedservices = $linkedservicesADF | Where-Object { $linkedservicesNames -notcontains $_.Name }
207     #Integrationruntimes
208     Write-Host "Getting integration runtimes"
209     $integrationruntimesADF = Get-AzDataFactoryV2IntegrationRuntime -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
210     $integrationruntimesTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/integrationruntimes" }
211     $integrationruntimesNames = $integrationruntimesTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)}
212     $deletedintegrationruntimes = $integrationruntimesADF | Where-Object { $integrationruntimesNames -notcontains $_.Name }
213 
214     #Delete resources
215     Write-Host "Deleting triggers"
216     $triggersToDelete | ForEach-Object { 
217         Write-Host "Deleting trigger "  $_.Name
218         $trig = Get-AzDataFactoryV2Trigger -name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName
219         if ($trig.RuntimeState -eq "Started") {
220             if ($_.TriggerType -eq "BlobEventsTrigger") {
221                 Write-Host "Unsubscribing trigger" $_.Name "from events"
222                 $status = Remove-AzDataFactoryV2TriggerSubscription -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
223                 while ($status.Status -ne "Disabled"){
224                     Start-Sleep -s 15
225                     $status = Get-AzDataFactoryV2TriggerSubscriptionStatus -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
226                 }
227             }
228             Stop-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name -Force 
229         }
230         Remove-AzDataFactoryV2Trigger -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 
231     }
232     Write-Host "Deleting pipelines"
233     $deletedpipelines | ForEach-Object { 
234         Write-Host "Deleting pipeline " $_.Name
235         Remove-AzDataFactoryV2Pipeline -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 
236     }
237     Write-Host "Deleting dataflows"
238     $deleteddataflow | ForEach-Object { 
239         Write-Host "Deleting dataflow " $_.Name
240         Remove-AzDataFactoryV2DataFlow -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 
241     }
242     Write-Host "Deleting datasets"
243     $deleteddataset | ForEach-Object { 
244         Write-Host "Deleting dataset " $_.Name
245         Remove-AzDataFactoryV2Dataset -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 
246     }
247     Write-Host "Deleting linked services"
248     $deletedlinkedservices | ForEach-Object { 
249         Write-Host "Deleting Linked Service " $_.Name
250         Remove-AzDataFactoryV2LinkedService -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 
251     }
252     Write-Host "Deleting integration runtimes"
253     $deletedintegrationruntimes | ForEach-Object { 
254         Write-Host "Deleting integration runtime " $_.Name
255         Remove-AzDataFactoryV2IntegrationRuntime -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 
256     }
257 
258     if ($deleteDeployment -eq $true) {
259         Write-Host "Deleting ARM deployment ... under resource group: " $ResourceGroupName
260         $deployments = Get-AzResourceGroupDeployment -ResourceGroupName $ResourceGroupName
261         $deploymentsToConsider = $deployments | Where { $_.DeploymentName -like "ArmTemplate_master*" -or $_.DeploymentName -like "ArmTemplateForFactory*" } | Sort-Object -Property Timestamp -Descending
262         $deploymentName = $deploymentsToConsider[0].DeploymentName
263 
264        Write-Host "Deployment to be deleted: " $deploymentName
265         $deploymentOperations = Get-AzResourceGroupDeploymentOperation -DeploymentName $deploymentName -ResourceGroupName $ResourceGroupName
266         $deploymentsToDelete = $deploymentOperations | Where { $_.properties.targetResource.id -like "*Microsoft.Resources/deployments*" }
267 
268         $deploymentsToDelete | ForEach-Object { 
269             Write-host "Deleting inner deployment: " $_.properties.targetResource.id
270             Remove-AzResourceGroupDeployment -Id $_.properties.targetResource.id
271         }
272         Write-Host "Deleting deployment: " $deploymentName
273         Remove-AzResourceGroupDeployment -ResourceGroupName $ResourceGroupName -Name $deploymentName
274     }
275 
276     #Start active triggers - after cleanup efforts
277     Write-Host "Starting active triggers"
278     $triggersToStart | ForEach-Object { 
279         if ($_.TriggerType -eq "BlobEventsTrigger") {
280             Write-Host "Subscribing" $_.Name "to events"
281             $status = Add-AzDataFactoryV2TriggerSubscription -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
282             while ($status.Status -ne "Enabled"){
283                 Start-Sleep -s 15
284                 $status = Get-AzDataFactoryV2TriggerSubscriptionStatus -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
285             }
286         }
287         Write-Host "Starting trigger" $_.Name
288         Start-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name -Force
289     }
290 }

adf_ci_cd.sh

咱們將當前powershell 腳本添加到代碼中。文件命名爲「adf_ci_cd.sh」

將，上面的腳本文件的內容貼進去，點擊「Commit」，而且提交保存到「master」分支中

3，配置Azure DevOps Pipeline 環境

回到Azure DevOps 中，選擇「Pipelines=》Releases」，點擊「New pipeline」

此時須要咱們選擇模板，咱們先點擊「Empty job」建立一個空的 job

修改補助名稱爲「UAT」

接下來選擇添加「artifact」，先添加一個afd_master 發佈產品源

Source Type 選擇：」Azure Repos Git「

Project 選擇：」CnBateBlogWeb_Proj「

Source(reposity)選擇：」CnBateBlogWeb_Proj「

Default branch：」adf_publish「

Source alias（源別名）：」_CnBateBlogWeb_Proj_Publish「

點擊」Add「進行添加操做。

再添加「artifact」，先添加一個ADF的」master「分佈做爲產品發佈源，你們注意裏面的參數

Default branch：」master「

Source alias：」_CnBateBlogWeb_Proj_Master「

點擊」OK「進行添加操做。

最後，讓咱們給」UAT「 stages 添加 task，點擊圖中圈起來的部分

點擊」+「，進行添加」Azure PowerShell「 Task，同時PowerShell 腳本去中止目前全部正在運行的 Triggers

對於當前中止Trigger的 Task，有些須要咱們留意的屬性配置

1，Display name 能夠改成」Stop Triggers「（表面當前Task具體幹了什麼事）

2，選擇」UAT「環境所在的Azure 訂閱

3，Script Path（腳本路徑）

4，Script Arguments（腳本參數）

上面提到的腳本文件說明一下，就是咱們剛剛在」master「分支提交的 adf_ci_cd.sh 這個文件，咱們能夠進行選擇

部署前，須要中止全部Trigger，關於腳本參數（中止Trigger）

-armTemplate "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -ResourceGroupName <your-resource-group-name> -DataFactoryName <your-data-factory-name> -predeployment $true -deleteDeployment $false

部署完成後，須要啓動全部Trigger，關於腳本參數（啓動Trigger）

-armTemplate "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -ResourceGroupName <your-resource-group-name> -DataFactoryName <your-data-factory-name> -predeployment $false -deleteDeployment $true

你們將上面對應的的腳本參數貼上本身的實際的UAT環境所在的資源組，資源名稱，以及模板所在路徑

如下是我當前的模板參數；你們能夠做爲參考：

-armTemplate "$(System.DefaultWorkingDirectory)/_CnBateBlogWeb_Proj_Publish/ADF-CnBateBlogWeb-Dev/ARMTemplateForFactory.json" -ResourceGroupName "Web_Test_DF_RG_UAT" -DataFactoryName "ADF-CnBateBlogWeb-AUT" -predeployment $true -deleteDeployment $false

接下來就是添加 ARM 模板部署了，搜索」ARM template deployment「，找到 ARM 模板部署，點擊」Add「進行添加

修改當前Task屬性

Display name：」ARM Template deployment：ADF Deploy「

Subscription 選擇：當前須要UAT環境所在的訂閱

Resource group 選擇：」Web_Test_DF_RG_UAT「（也就是當前UAT環境 ADF所在的資源組）

Loaction 選擇：」East Asia「

Template 以及 Template parameters 分別選擇」_CnBateBlog_Proj_Publish「下的」ARMTemplateForFactory.json「,"ARMTemplateParametersForFactory.json"

而 Override template parameters（替換模板參數）則指咱們須要把模板參數文件中的一些參數替換掉，如須要部署的ADF的名稱，數據源、目標源的blob Storage 的連接字符串

以下，我演示的覆蓋模板參數實例（你們須要將UAT環境的ADF 名稱換成本身建立命名的，以及兩個連接字符參數名稱須要在模板參數文件中找到本身對應的名稱，連接字符串複製粘貼本身UAT環境的兩個Blob Storage的連接字符串）：

-factoryName "ADF-CnBateBlogWeb-AUT" -CnBateBlogFromBlobStorage_connectionString "DefaultEndpointsProtocol=https;AccountName=cnbateblogwebaccount1uat;AccountKey=XF7XkFsnEdIGZT8CCUJs7At1E/Ni9k/XJBtwWaKvwAuBnp3cCN6e2CYiV2uzsq2iI0vI2eZVHS8Kh9CvcuoNcg==;EndpointSuffix=core.windows.net" -CnBateBlobToBlobStorage_connectionString "DefaultEndpointsProtocol=https;AccountName=cnbateblogwebaccount2uat;AccountKey=E5z+q7XL7+8xTlqHsyaIGr0Eje/0DhT9+/E+oro4D57tsSuEchmnjLiK8zftTtyvQKLQUvTGJEsAOFCBGdioHw==;EndpointSuffix=core.windows.net"

最近就是添加劇啓全部 Trigger 的 PowerShell 的 Task，具體參數以下圖

Script Path 選擇：」_CnBateBlogWeb_Proj_Master「文件下的「adf_ci_cd.sh」文件

Script Path （須要清理資源、重啓全部Trigger的腳本參數）：

-armTemplate "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -ResourceGroupName <your-resource-group-name> -DataFactoryName <your-data-factory-name> -predeployment $false -deleteDeployment $true

你們將上面對應的的腳本參數貼上本身的實際的UAT環境所在的資源組，資源名稱，以及模板所在路徑

如下是我當前的模板參數；你們能夠做爲參考：

-armTemplate "$(System.DefaultWorkingDirectory)/_CnBateBlogWeb_Proj_Publish/ADF-CnBateBlogWeb-Dev/ARMTemplateForFactory.json" -ResourceGroupName "Web_Test_DF_RG_UAT" -DataFactoryName "ADF-CnBateBlogWeb-AUT" -predeployment $true -deleteDeployment $true

修改當前pipeline 的名稱，點擊 "Save」進行保存。

接下來，就是設置 pipeline 的觸發條件

開啓持續部署觸發，每次在所選存儲庫中發生Git推送時觸發pipeline，接下來添加分支篩選條件

Type：Include，Branch：「adf_master」，也就是說每當「adf_publish」發生git 推送的時候，觸發此 pipeline

設置完畢後，點擊「Save」進行保存

咱們能夠手動「建立Release」，測試 pipeline 狀態顯示正常，成功

回到Azure UAT 環境中，咱們能夠發現 UAT 環境的ADF的 pipeline dataset 等也已經部署配置成功

4，測試ADF 自動化部署

咱們回到ADF的 Dev 環境，咱們嘗試在ADF Dev 環境建立一個的新的「Feature Branch」嘗試更改 pipeline 中複製數據的名稱

咱們將上圖中的Copy Data 的名稱由「Copy data1」改成「Copy data2」，而且進行保存，驗證操做。

驗證沒有問題後，咱們就能夠將當前「allen/Feature_1」分支上的更改合併到「master」分支上

經過提交「Create pull request」

接下來我就再也不演示了，有什麼的不懂的，你們能夠去看上一篇文章。等待合併完成後，咱們回到Dev 環境的ADF 中就能夠在「master」分支進行發佈了。當發佈完成後，就會將最新更改推送到 "adf_master" 分支上的 ARM 模板以及參數等。這個時候就會觸發以前設置好的Azure DevOps pipeline 了，就能夠自動就這些更改自動部署到UA環境了。

三，結尾

　　今天的內容着重介紹了一下Azure Data Factory 與Azure Pipeline 的集成和自動化部署，可是一樣的也遺留了一個問題，就是在部署ARM 模板的時候，咱們時直接將Blob Storage 的連接字符串直接粘貼到覆蓋模板參數一欄中，若是ADF 中添加不少數據源的化，使用直接這種方式也是不合適的，這個問題怎麼解決，下一節咱們繼續。*★,°*:.☆(￣▽￣)/$:*.°★* 。

做者：Allen

版權：轉載請在文章明顯位置註明做者及出處。如發現錯誤，歡迎批評指正。