StreamSets 設計Edge pipeline

edge pipeline 運行在edge 執行模式，咱們可使用 data collector UI 進行edge pipeline 設計，
設計完成以後，你能夠部署對應的pipeline到edge 設備

能夠設計的edge pipeline

• edge 發送pipeline

edge 發送pipeline 使用特定的orgin讀取edge設備上的數據，這個pipeline 能夠在將數據發送到data collector 以前進行數據的處理

• edge 接收pipeline

接收pipeline能夠接收來自edge 設備或者 data collector pipeline的數據

orgin 組件

• Dev Random Record Source
• Dev Raw Data Source
• Directory Edge pipelines do not support multithreaded processing.
  In an edge pipeline, the Directory origin always creates a single thread to read the files even if you configure it to use multiple threads.
• File Tail
  In edge pipelines, the File Tail origin can read a single set of files.
  If you configure multiple sets of files for the origin, the origin reads only the files configured in the first set.
• HTTP Client
  In edge pipelines, the HTTP Client origin does not support batch processing mode, pagination, or OAuth2 authorization.
• HTTP Server Edge pipelines do not support multithreaded processing.
  In an edge pipeline, the HTTP Server origin always creates a single thread to read the files even if you configure it to use multiple threads.
• MQTT Subscriber Edge pipelines that use MQTT stages require using an intermediary MQTT broker.
  For example, an edge sending pipeline uses an MQTT Publisher destination to write to an MQTT broker. The MQTT broker temporarily stores the data until the MQTT Subscriber origin in the edge receiving pipeline reads the data.
• Sensor Reader
• System Metrics
• WebSocket Client
• Windows Event Log

processsor 組件

• Delay
• Dev Identity
• Expression Evaluator
• Field Remover
• JavaScript Evaluator In edge pipelines, the JavaScript Evaluator processor does not support the sdcFunctions scripting object.
• Stream Selector

destinations 組件

• CoAP Client
• HTTP Client
• Kafka Producer
• MQTT Publisher Edge pipelines that use MQTT stages require using an intermediary MQTT broker.
  For example, an edge sending pipeline uses an MQTT Publisher destination to write to an MQTT broker. The MQTT broker temporarily stores the data until the MQTT Subscriber origin in the Data Collector receiving pipeline reads the data.
• Trash
• WebSocket Client

錯誤記錄處理

• Discard 丟踢
  The pipeline discards the record.
• Write to File 寫到文件
  The pipeline writes error records and related details to a local directory on the edge device. Create another edge pipeline with a Directory origin to process the error records written to the file.
• Write to MQTT 寫到mqtt
  The pipeline publishes error records and related details to a topic on an MQTT broker. Create another edge or standalone Data Collector pipeline with an MQTT Subscriber origin to process the error records published to the broker.

支持的數據格式

• json
• text

限制

• Email and webhook notifications cannot be sent by edge pipelines.
• Rules and alerts cannot be defined for edge pipelines.
• Edge pipelines support a limited number of record, math, pipeline, and string functions.
• Edge pipelines do not support dataflow triggers.
• Edge pipelines do not support multithreaded processing.
• You cannot capture snapshots for edge pipelines.

參考資料

https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Edge_Mode/EdgePipelineTypes.html#concept_c14_m4r_4bb