StarlingX 補丁升級功能

Patch功能說明

StarlingX系統具備升級的能力，這個特性叫作「patching」，提供從2個版本之間升級的能力，主要用戶bug修復、安全補丁和特性加強等等。

Patching支持兩種補丁, In-Service補丁和Reboot-required補丁。In-Service補丁不須要主機節點重啓，只須要服務進程重啓便可。Reboot-required補丁須要重啓主機以實現補丁生效。在升級Reboot-required補丁時，須要先對主機進行lock操做，等待補丁applied，再unlock使其生效。

這篇介紹文檔，主要面向開發人員使用補丁功能，而不是產品用戶指南。它更着重介紹補丁修復功能，而不是包含補丁的各個方面。

簡要的說，補丁修復包含2個階段，建立補丁和應用補丁。下面進行詳細介紹這兩部分。

建立補丁

一個StarlingX補丁包括一個或多個系統升級所須要的rpm包。在開始建立補丁以前須要驗證rpm包已經在已部署的StarlingX上安裝了。如下步驟能夠幫助咱們確認。

1. 肯定已部署系統的軟件版本，有兩種方式
   • 在horizon界面上 Admin -> Platform
   -> System Configuration -> Systems
   • 使用命令行 system show
   controller-0:~$ . /etc/platform/openrc
   [sysadmin@controller-0 ~(keystone_admin)]$ system show
   +----------------------+--------------------------------------+
   | Property | Value |
   +----------------------+--------------------------------------+
   | contact | None |
   | created_at | 2019-10-14T03:10:50.862114+00:00 |
   | description | None |
   | https_enabled | False |
   | location | None |
   | name | 608dfe48-9a05-4b21-afc1-ea122574caa7 |
   | region_name | RegionOne |
   | sdn_enabled | False |
   | security_feature | spectre_meltdown_v1 |
   | service_project_name | services |
   | software_version | 19.09 |
   | system_mode | duplex |
   | system_type | All-in-one |
   | timezone | UTC |
   | updated_at | 2019-10-14T03:12:41.983029+00:00 |
   | uuid | 2639ad15-08a7-4f1b-a372-f927a5e4ab31 |
   | vswitch_type | none |
   +----------------------+--------------------------------------+

2. 檢查最新構建，找到針對這個版本須要升級的rpm包，選擇須要的rpm生成補丁
   一旦肯定須要升級/安裝的rpm包，下一步就是準備補丁構建環境。做爲StarlingX開發人員，最簡單的辦法是使用StarlingX Building 容器，咱們只須要對容器進行小小的修改就能夠了。StarlingX Building 容器可使用構建教程生成。

如今假設StarlingX的源碼已經下載好了，須要升級安裝的rpm包也準備好了，如今咱們開始構造補丁構建環境。再次聲明，這個教程主要針對開發人員，而不是產品。

1. 安裝2個cgcs-patch依賴包，crypto和pycrypto
   sudo pip install crypto pycrypto

2. 使用腳本$MY_REPO/stx/stx-update/extras/scripts/patch_build.sh建立補丁。

在這個腳本中，它從release-info.inc 文件中獲取PLATFORM_RELEASE參數，並把PYTHONPATH指向repo中的cgcs-patch包，避免了安裝cgcs-patch和手動指定PLATFORM_RELEASE參數。可使用下面命令查看構建腳本的使用說明。

$ $MY_REPO/stx/stx-update/cgcs-patch/bin/patch_build --help
Usage: patch_build [ <args> ] ... <rpm list>
Options:
    --id <id>                   Patch ID
    --release <version>         Platform release version
    --status <status>           Patch Status Code (ie. O, R, V)
    --unremovable               Marks patch as unremovable
    --reboot-required <Y|N>     Marks patch as reboot-required (default=Y)
    --summary <summary>         Patch Summary
    --desc <description>        Patch Description
    --warn <warnings>           Patch Warnings
    --inst <instructions>       Patch Install Instructions
    --req <patch_id>            Required Patch
    --controller <rpm>          New package for controller
    --worker <rpm>              New package for worker node
    --worker-lowlatency <rpm>   New package for worker-lowlatency node
    --storage <rpm>             New package for storage node
    --controller-worker <rpm>   New package for combined node
    --controller-worker-lowlatency <rpm>   New package for lowlatency
                                combined node
    --all-nodes <rpm>           New package for all node types

使用這個腳本能夠指定patch id、reboot required、depended patches、rpm list等等，若是系統上沒有的，須要新安裝的包須要指定節點，好比 --controller 指定是在控制節點上新裝包。腳本執行完後，能夠獲得名字爲「<patch-id>.patch」的文件。

下面深刻研究下這個補丁文件。

1. 首先，這個補丁文件是個gzip壓縮包。咱們能夠經過file命令去檢查下。
   $ file 001.patch
   001.patch: gzip compressed data, was "001.patch", last modified:
   Fri Aug 16 05:56:59 2019, max compression

2. 解壓出來後，能夠看到如下文件
   $ tar -xf 001.patch
   $ tree
   ├── 001.patch
   ├── metadata.tar
   ├── signature
   ├── signature.v2
   └── software.tar

3. 解壓 software.tar，能夠發現它包含了全部須要安裝的rpm包。注意：全部的rpm包在補丁構建時用下面的key進行簽名。

$MY_REPO/build-tools/signing/ima_signing_key.priv

4. 在metadata.tar中只有一個文件metadata.xml，包含補丁構建的全部信息。StarlingX集羣系統會讀取這個文件信息。

5. signature文件包含software.tar和metadata.tar的MD5的組合。

6. signature.v2是爲software.tar和metadata.tar的簽名文件，在當前環境中，它由$MY_REPO/build-tools/signing/dev-private-key.pemkey文件生成。

安裝補丁

補丁生成後，能夠手動安裝補丁到指定的StarlingX系統，同時支持界面和命令行安裝操做。補丁的生命週期包括四個狀態： Available,Partial-Apply, Applied 和 Partial-Remove.

• Available：表示補丁已經上傳到補丁存儲庫裏了，可是尚未到軟件升級的倉庫，一樣也沒有在任何主機上安裝。

• Partial-Apply：表示補丁升級程序已經被觸發（sw-patch apply），已經在部分主機上安裝，可是尚未在全部須要安裝的主機上安裝。

• Applied：表示已經在全部須要安裝的主機上安裝完成。

• Partial-Remove：表示補丁正在被移除，經過命令觸發(sw-patch remove),正在移除，可是還沒徹底移除。

若是須要用命令行安裝補丁，須要把補丁拷貝到active的控制節點上。StarlingX集羣提供客戶端命令sw-patch。補丁操做都是經過這個命令完成，這個命令提供了不少功能，包括upload, apply, query,host-install, delete, remove等等。

controller-0:~$ sw-patch --help
usage: sw-patch [--debug]
                  <subcommand> ...
Subcomands:
    upload:         Upload one or more patches to the patching system.
    upload-dir:     Upload patches from one or more directories to the
                    patching system.
    apply:          Apply one or more patches. This adds the specified
                    patches to the repository, making the update(s)
                    available to the hosts in the system. Use --all to
                    apply all available patches.
                    Patches are specified as a space-separated list of
                    patch IDs.
    remove:         Remove one or more patches. This removes the specified
                    patches from the repository.
                    Patches are specified as a space-separated list of
                    patch IDs.
    delete:         Delete one or more patches from the patching system.
                    Patches are specified as a space-separated list of
                    patch IDs.
    query:          Query system patches. Optionally, specify 'query
                    applied' to query only those patches that are applied,
                    or 'query available' to query those that are not.
    show:           Show details for specified patches.
    what-requires:  List patches that require the specified patches.
    query-hosts:    Query patch states for hosts in the system.
    host-install:   Trigger patch install/remove on specified host. To
                    force install on unlocked node, use the --force option.
    host-install-async: Trigger patch install/remove on specified host. To
                    force install on unlocked node, use the --force option.
                    Note: This command returns immediately upon dispatching
                    installation request.
    install-local:  Trigger patch install/remove on the local host. This
                    command can only be used for patch installation prior
                    to initial configuration.
    drop-host:      Drop specified host from table.
    query-dependencies: List dependencies for specified patch. Use
                    --recursive for recursive query.
    is-applied:     Query Applied state for list of patches. Returns True
                    if all are Applied, False otherwise.
    report-app-dependencies: Report application patch dependencies,
                    specifying application name with --app option, plus a
                    list of patches. Reported dependencies can be dropped
                    by specifying app with no patch list.
    query-app-dependencies: Display set of reported application patch
                    dependencies.
    commit:         Commit patches to free disk space. WARNING: This
                    action is irreversible!
    --os-region-name: Send the request to a specified region

下面演示如何使用這個命令去安裝補丁。演示用的補丁是須要安裝在全部主機上的In-Service補丁，須要升級的StarlingX環境是 2+2+2的標準環境。

1. 上傳補丁文件
   controller-0:~$ sudo sw-patch upload 001.patch
   001 is now available
   檢查補丁狀態
   controller-0:~$ sudo sw-patch query
   Patch ID RR Release Patch State
   ======== == ======= ===========
   001 N 19.09 Available
   檢查全部主機的的升級狀態
   controller-0:/$ sudo sw-patch query-hosts
   Hostname IP Address Patch Current Reboot Required Release State
   ============ ============== ============= =============== ====== =====
   compute-0 192.178.204.7 Yes No 19.09 idle
   compute-1 192.178.204.9 Yes No 19.09 idle
   controller-0 192.178.204.3 Yes No 19.09 idle
   controller-1 192.178.204.4 Yes No 19.09 idle
   storage-0 192.178.204.12 Yes No 19.09 idle
   storage-1 192.178.204.11 Yes No 19.09 idle

Patch Current 表示當前主機是否有補丁安裝，Yes表示沒有安裝補丁，No表示至少有一個補丁在安裝

2. 當補丁狀態available後，能夠觸發補丁安裝
   controller-0:/$ sudo sw-patch apply 001
   001 is now in the repo
   檢查補丁狀態
   controller-0:~$ sudo sw-patch query
   Patch ID RR Release Patch State
   ======== == ======= =============
   001 N 19.09 Partial-Apply
   檢查節點狀態
   controller-0:~$ sudo sw-patch query-hosts
   Hostname IP Address Patch Current Reboot Required Release State
   ============ ============== ============= =============== ====== =====
   compute-0 192.178.204.7 No No 19.09 idle
   compute-1 192.178.204.9 No No 19.09 idle
   controller-0 192.178.204.3 No No 19.09 idle
   controller-1 192.178.204.4 No No 19.09 idle
   storage-0 192.178.204.12 No No 19.09 idle
   storage-1 192.178.204.11 No No 19.09 idle

3. 在每一個節點上安裝補丁，因爲是in-service 補丁，因此不須要執行lock操做。
   controller-0:~$ sudo sw-patch host-install controller-0
   ...
   Installation was successful.
   檢查主機升級狀態
   controller-0:~$ sudo sw-patch query-hosts
   Hostname IP Address Patch Current Reboot Required Release State
   ============ ============== ============= =============== ====== =====
   compute-0 192.178.204.7 No No 19.09 idle
   compute-1 192.178.204.9 No No 19.09 idle
   controller-0 192.178.204.3 Yes No 19.09 idle
   controller-1 192.178.204.4 No No 19.09 idle
   storage-0 192.178.204.12 No No 19.09 idle
   storage-1 192.178.204.11 No No 19.09 idle

   在全部節點上安裝補丁，須要爲每一個節點執行命令
   controller-0:~$ sudo sw-patch host-install controller-1
   ....
   Installation was successful.
   controller-0:~$ sudo sw-patch host-install compute-0
   ....
   Installation was successful.
   controller-0:~$ sudo sw-patch host-install compute-1
   ....
   Installation was successful.
   controller-0:~$ sudo sw-patch host-install storage-0
   ...
   Installation was successful.
   controller-0:~$ sudo sw-patch host-install storage-1
   ...
   Installation was successful.

4. 全部節點按照完畢後，能夠看到下面狀態
   controller-0:~$ sudo sw-patch query
   Patch ID RR Release Patch State
   ======== == ======= ===========
   001 N 19.09 Applied
   controller-0:~$ sudo sw-patch query-hosts
   Hostname IP Address Patch Current Reboot Required Release State
   ============ ============== ============ =============== ======= =====
   compute-0 192.178.204.7 Yes No 19.09 idle
   compute-1 192.178.204.9 Yes No 19.09 idle
   controller-0 192.178.204.3 Yes No 19.09 idle
   controller-1 192.178.204.4 Yes No 19.09 idle
   storage-0 192.178.204.12 Yes No 19.09 idle
   storage-1 192.178.204.11 Yes No 19.09 idle
   此時補丁升級程序完成

除了補丁升級，StarlingX還支持補丁回退和刪除，經過下面兩個命令實現sw-patch remove和sw-patch host-install，和補丁安裝有點相似。

補丁編排

在上面的例子中，演示了在集羣中補丁升級的功能。可是在大規模集羣中，整個升級過程會持續很長的時間。特別是reboot-required補丁，這個方案會很糟糕，效率很低並且給管理員帶來不少工做。所以StarlingX提供了另外一個高級特性「補丁編排」。它支持集羣經過一些簡單的操做達到升級的目的，極大減小管理員的工做負擔和較少出錯。這個功能有三種方式使用，客戶端CLI、界面Horizon和VIM Restful API。

1. 客戶端CLI。StarlingX提供客戶端工具sw-manager， 能夠用於補丁編排。以下所示，能夠經過建立和應用補丁策略來升級整個集羣

   controller-0:~$ sw-manager patch-strategy -h
   usage: sw-manager patch-strategy [-h] ...
   optional arguments:
   -h, --help show this help message and exit
   Software Patch Commands:
   create Create a strategy
   delete Delete a strategy
   apply Apply a strategy
   abort Abort a strategy
   show Show a strategy
   controller-0:~$ sw-manager patch-strategy create -h
   usage: sw-manager patch-strategy create [-h]
   [--controller-apply-type {serial,ignore}]
   [--storage-apply-type {serial,parallel,ignore}]
   [--worker-apply-type {serial,parallel,ignore}]
   [--max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10,
   11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,
   28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,
   45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,
   62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,
   79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,
   96,97,98,99,100}]
   [--instance-action {migrate,stop-start}]
   [--alarm-restrictions {strict,relaxed}]
   optional arguments:
   -h, --help show this help message and exit
   --controller-apply-type {serial,ignore}
   defaults to serial
   --storage-apply-type {serial,parallel,ignore}
   defaults to serial
   --worker-apply-type {serial,parallel,ignore}
   defaults to serial
   --max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,
   17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,
   37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,
   57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,
   77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,
   97,98,99,100}
   maximum worker hosts to patch in parallel
   --instance-action {migrate,stop-start}
   defaults to stop-start
   --alarm-restrictions {strict,relaxed}
   defaults to strict

2. 界面Horizon。打開Admin -> Platform
   -> Software Management -> Patch Orchestration 標籤

3. VIM API。<http://<oam_ip>:4545>;
   +--------+---------------------------------------+----------------------------+
   | Method | URI | Description |
   +========+=======================================+============================+
   | Post | /api/orchestration/sw-update/strategy | Create a patch strategy |
   +--------+---------------------------------------+----------------------------+
   | Delete | /api/orchestration/sw-update/strategy | Delete current patch |
   | | | strategy |
   +--------+---------------------------------------+----------------------------+
   | Get | /api/orchestration/sw-update/strategy | Get detailed information of|
   | | | current patch strategy |
   +--------+---------------------------------------+----------------------------+
   | Post | /api/orchestration/sw-update/strategy/| Apply or abort a patch |
   | | actions | strategy |
   +--------+---------------------------------------+----------------------------+

在補丁安裝時，補丁編排要求集羣處於一個良好的狀態。
• 全部主機必須處於unlocked-enabled-available狀態
• 系統沒有告警
• 足夠的空間用於VM遷移

當前開發狀態

• 全部的源碼都在StarlingX倉庫裏開源，包括「update」和「nfv」• in-service補丁和reboot-required補丁的生成和安裝已經通過驗證• 補丁編排還沒通過驗證