Spring Cloud Data Flow用Shell來操做，方便創建CICD

1 前言

歡迎訪問 南瓜慢說 www.pkslow.com獲取更多精彩文章！

以前咱們用兩篇文章講解了Spring Cloud Data Flow，例子都是用UI操做的，但咱們在Linux系統上常常是沒法提供界面來操做，集成在Jenkins上也沒法使用UI。好在官方提供了Data Flow Shell工具，能夠在命令行模式下進行操做，很是方便。

相關文章可參考：

Spring Cloud Data Flow初體驗，以Local模式運行

把Spring Cloud Data Flow部署在Kubernetes上，再跑個任務試試

Spring Cloud Data Flow Server提供了可操做的REST API，因此這個Shell工具的本質仍是經過調用REST API來交互的。

2 經常使用操做

2.1 啓動

首先要確保咱們已經安裝有Java環境和下載了可執行的jar包：spring-cloud-dataflow-shell-2.5.3.RELEASE.jar

而後啓動以下：

$ java -jar spring-cloud-dataflow-shell-2.5.3.RELEASE.jar

默認是鏈接了http://localhost:9393的Server，能夠經過--dataflow.uri=地址來指定。若是須要認證信息，須要加上--dataflow.username=用戶 --dataflow.password=密碼。

好比咱們鏈接以前安裝在Kubernetes上的Server以下：

$ java -jar spring-cloud-dataflow-shell-2.5.3.RELEASE.jar --dataflow.uri=http://localhost:30093

2.2 Application操做

介紹一下Application相關操做：

列出全部目前註冊的app：

dataflow:>app list
╔═══╤══════╤═════════╤════╤════════════════════╗
║app│source│processor│sink│        task        ║
╠═══╪══════╪═════════╪════╪════════════════════╣
║   │      │         │    │composed-task-runner║
║   │      │         │    │timestamp-batch     ║
║   │      │         │    │timestamp           ║
╚═══╧══════╧═════════╧════╧════════════════════╝

查看某個app的信息：

dataflow:>app info --type task timestamp

清除app註冊信息：

dataflow:>app unregister --type task timestamp
Successfully unregistered application 'timestamp' with type 'task'.

清除全部app註冊信息：

dataflow:>app all unregister
Successfully unregistered applications.
dataflow:>app list 
No registered apps.
You can register new apps with the 'app register' and 'app import' commands.

註冊一個app：

dataflow:>app register --name timestamp-pkslow --type task --uri docker:springcloudtask/timestamp-task:2.1.1.RELEASE
Successfully registered application 'task:timestamp-pkslow'
dataflow:>app list
╔═══╤══════╤═════════╤════╤════════════════╗
║app│source│processor│sink│      task      ║
╠═══╪══════╪═════════╪════╪════════════════╣
║   │      │         │    │timestamp-pkslow║
╚═══╧══════╧═════════╧════╧════════════════╝

批量導入app，能夠從一個URL或一個properties文件導入：

dataflow:>app import https://dataflow.spring.io/task-docker-latest
Successfully registered 3 applications from [task.composed-task-runner, task.timestamp.metadata, task.composed-task-runner.metadata, task.timestamp-batch.metadata, task.timestamp-batch, task.timestamp]

須要注意的是，在註冊或導入app時，若是重複的話，默認是沒法導入的，不會覆蓋。若是想要覆蓋，能夠加參數--force。

dataflow:>app register --name timestamp-pkslow --type task --uri docker:springcloudtask/timestamp-task:2.1.1.RELEASE
Command failed org.springframework.cloud.dataflow.rest.client.DataFlowClientException: The 'task:timestamp-pkslow' application is already registered as docker:springcloudtask/timestamp-task:2.1.1.RELEASE
The 'task:timestamp-pkslow' application is already registered as docker:springcloudtask/timestamp-task:2.1.1.RELEASE

dataflow:>app register --name timestamp-pkslow --type task --uri docker:springcloudtask/timestamp-task:2.1.1.RELEASE --force
Successfully registered application 'task:timestamp-pkslow'

2.3 Task操做

列出task：

dataflow:>task list
╔════════════════╤════════════════════════════════╤═══════════╤═══════════╗
║   Task Name    │        Task Definition         │description│Task Status║
╠════════════════╪════════════════════════════════╪═══════════╪═══════════╣
║timestamp-pkslow│timestamp                       │           │COMPLETE   ║
║timestamp-two   │<t1: timestamp || t2: timestamp>│           │ERROR      ║
║timestamp-two-t1│timestamp                       │           │COMPLETE   ║
║timestamp-two-t2│timestamp                       │           │COMPLETE   ║
╚════════════════╧════════════════════════════════╧═══════════╧═══════════╝

刪除一個task，這裏咱們刪除的是一個組合task，因此會把子task也一併刪除了：

dataflow:>task destroy timestamp-two
Destroyed task 'timestamp-two'

刪除全部task，會有風險提示：

dataflow:>task all destroy 
Really destroy all tasks? [y, n]: y
All tasks destroyed

dataflow:>task list
╔═════════╤═══════════════╤═══════════╤═══════════╗
║Task Name│Task Definition│description│Task Status║
╚═════════╧═══════════════╧═══════════╧═══════════╝

建立一個task：

dataflow:>task create timestamp-pkslow-t1 --definition "timestamp --format=\"yyyy\"" --description "pkslow timestamp task"
Created new task 'timestamp-pkslow-t1'

啓動一個task並查看狀態，啓動時須要記錄執行ID，而後經過執行ID來查詢狀態：

dataflow:>task launch timestamp-pkslow-t1
Launched task 'timestamp-pkslow-t1' with execution id 8
dataflow:>task execution status 8

查看全部task執行並查看執行日誌：

dataflow:>task execution list 


dataflow:>task execution log 8

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::       (v2.1.13.RELEASE)

2020-08-01 17:20:51.626  INFO 1 --- [       Thread-5] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
2020-08-01 17:20:51.633  INFO 1 --- [       Thread-5] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.

2.4 Http請求

能夠進行http請求：

dataflow:>http get https://www.pkslow.com

dataflow:>http post --target https://www.pkslow.com --data "data"
> POST (text/plain) https://www.pkslow.com data
> 405 METHOD_NOT_ALLOWED

Error sending data 'data' to 'https://www.pkslow.com'

2.5 讀取並執行文件

先準備一個腳本文件，用來放Data Flow Shell命令，文件名爲pkslow.shell，內容以下：

version
date
app list

執行與結果以下：

dataflow:>script pkslow.shell
version
2.5.3.RELEASE
date
Sunday, August 2, 2020 1:59:34 AM CST
app list
╔═══╤══════╤═════════╤════╤════════════════════╗
║app│source│processor│sink│        task        ║
╠═══╪══════╪═════════╪════╪════════════════════╣
║   │      │         │    │timestamp-pkslow    ║
║   │      │         │    │composed-task-runner║
║   │      │         │    │timestamp-batch     ║
║   │      │         │    │timestamp           ║
╚═══╧══════╧═════════╧════╧════════════════════╝

Script required 0.045 seconds to execute
dataflow:>

但其實咱們在CI/CD的pipeline中，並不想先啓動一個shell命令行，而後再執行一個腳本。咱們想一步到位，直接執行，執行完畢後退出shell命令行。這也是有辦法的，能夠在啓動的時候經過 --spring.shell.commandFile指定文件，若是有多個文件則用逗號,分隔。以下所示：

$ java -jar spring-cloud-dataflow-shell-2.5.3.RELEASE.jar --dataflow.uri=http://localhost:30093 --spring.shell.commandFile=pkslow.shell
Successfully targeted http://localhost:30093
2020-08-02T02:03:49+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - 2.5.3.RELEASE
2020-08-02T02:03:49+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - Sunday, August 2, 2020 2:03:49 AM CST
2020-08-02T02:03:49+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:309 - 
╔═══╤══════╤═════════╤════╤════════════════════╗
║app│source│processor│sink│        task        ║
╠═══╪══════╪═════════╪════╪════════════════════╣
║   │      │         │    │timestamp-pkslow    ║
║   │      │         │    │composed-task-runner║
║   │      │         │    │timestamp-batch     ║
║   │      │         │    │timestamp           ║
╚═══╧══════╧═════════╧════╧════════════════════╝
$

執行完畢後，不會在shell命令行模式裏，而是退回linux的終端。這正是咱們所須要的。

咱們來準備一個註冊應用——建立任務——執行任務的腳本試試：

version
date
app register --name pkslow-app-1 --type task --uri docker:springcloudtask/timestamp-task:2.1.1.RELEASE
task create pkslow-task-1 --definition "pkslow-app-1"
task launch pkslow-task-1

執行與結果以下：

$ java -jar spring-cloud-dataflow-shell-2.5.3.RELEASE.jar --dataflow.uri=http://localhost:30093 --spring.shell.commandFile=pkslow.shell
Successfully targeted http://localhost:30093
2020-08-02T02:06:41+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - 2.5.3.RELEASE
2020-08-02T02:06:41+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - Sunday, August 2, 2020 2:06:41 AM CST
2020-08-02T02:06:41+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - Successfully registered application 'task:pkslow-app-1'
2020-08-02T02:06:42+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - Created new task 'pkslow-task-1'
2020-08-02T02:06:51+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - Launched task 'pkslow-task-1' with execution id 9

這樣，咱們就能夠實現自動化打包與部署運行了。

3 一些使用技巧

強大的shell工具提供了許多命令，其實不用一一記住，能夠經過help命令查看全部命令：

dataflow:>help

若是隻對特定的一類命令感興趣，能夠經過help xxx的方式獲取幫助：

dataflow:>help version
* version - Displays shell version

dataflow:>help app
* app all unregister - Unregister all applications
* app default - Change the default application version
* app import - Register all applications listed in a properties file
* app info - Get information about an application
* app list - List all registered applications
* app register - Register a new application
* app unregister - Unregister an application

shell還支持tab鍵補全命令。

4 總結

本文的命令比較多，不想形成冗長，部分執行結果就不貼出來了，原文可到官網參考。

---

歡迎關注微信公衆號<南瓜慢說>，將持續爲你更新...

多讀書，多分享；多寫做，多整理。