memsql filesystem pipeline 試用

一些功能相似drill ，好比s3，file 。。。

建立file pipeline

• 準備file

mkdir -p /opt/db/
touch books.txt
內容以下：
The Catcher in the Rye, J.D. Salinger, 1945
Pride and Prejudice, Jane Austen, 1813
Of Mice and Men, John Steinbeck, 1937
Frankenstein, Mary Shelley, 1818

• 建立表

memsql

CREATE DATABASE books;

USE books;

CREATE TABLE classic_books
(
title VARCHAR(255),
author VARCHAR(255),
date VARCHAR(255)
);

• 建立pipeline

CREATE PIPELINE library
AS LOAD DATA FS '/opt/db/*'
INTO TABLE `classic_books`
FIELDS TERMINATED BY ',';

啓用pipeline

• 啓動

START PIPELINE library;

• 查看狀態

SHOW PIPELINES;

測試結果

幾個問題

• Paused due to error. Run START PIPELINE or consider setting pipelines_stop_on_error to false

配置參數修改
SET GLOBAL pipelines_stop_on_error = false;

• 注意文件的權限，同時文件必須是每一個節點的，否則一直看不到數據（我沒注意一直在master，就有問題）
• 常常有如下提示

Data volume has significantly changed since the last time ANALYZE TABLE was run. Run <a target="_blank" href="https://docs.memsql.com/ops-redir/analyze/?utm_source=ops&amp;utm_medium=link&amp;utm_campaign=ref" data-reactid=".0.3.1.2.0.2.0.0.0.3" style="background:rgb(250, 250, 250);color:rgb(18, 135, 186);text-decoration:none;font-size:1.067rem;"><b data-reactid=".0.3.1.2.0.2.0.0.0.3.0">ANALYZE TABLE</b></a> on each table to improve query performance andrefresh schema.

解決方法

按照提示操做便可，這個可能和個人系統沒有進行參數優化有關，具體的能夠參考下面的資料的安裝最佳實踐

參考資料

https://docs.memsql.com/memsql-pipelines/v6.0/filesystem-pipelines-quickstart/
https://docs.memsql.com/memsql-pipelines/v6.0/filesystem-pipelines-overview/
https://docs.memsql.com/tutorials/v6.0/installation-best-practices/