kettle之mongodb數據同步

需求：

1.源數據庫新增一條記錄，目標庫同時新增一條記錄；

2.源數據庫修改一條記錄，目標庫同時修改該條記錄；

示例用到三個Kettle組件

 

下面詳細說下每一個組件的配置

Source:

本示例鏈接的是Mongodb數據庫，四個字段，ID默認爲主鍵，_id會系統自動生成暫時先無論。

本節點的詳細說明，可見官網：http://wiki.pentaho.com/display/EAI/MongoDB+Input

值映射：

本步在本示例做用不大，只是爲了測試效果。按照截圖上進行配置便可

MongoDbOutput:

關鍵是這一步的配置

官網上對這個tab頁的解釋是這樣的：

2.2 Selecting the write mode

The MongoDb output step provides a number of options that control what and how data is written to the target Mongo document collection. By default, data is inserted into the target collection. If the specified collection doesn't exist, it will be created before data is inserted. Selecting the Truncate option will delete any existing data in the target collection before inserting begins. Unless unique indexes are being used (see section on indexing below) then Mongo DB will allow duplicate records to be inserted. Mongo DB allows for fast bulk insert operations - the batch size can be configured using the Batch insert size field. If no value is supplied here, then the default size of 100 rows is used.

Selecting the Upsert option changes the write mode from insert to upsert (i.e. update if a match is found, otherwise insert a new record). Information on defining how records are matched can be found in the next section. Standard upsert replaces a matched record with an entire new record based on all the incoming fields specified in the Mongo document fields tab. Modifier update enables modifier ($ operators) to be used to mutate individual fields within matching documents. This type of update is fast and involves minimal network traffic; it also has the ability to update all matching documents, rather than just the first, if the Multi-update option is enabled

我的理解就是默認都不選擇時往目標集合添加數據，若是指定的集合不存在則先建立集合而後插入數據；

Truncate:插入數據以前先把集合裏的全部數據刪除。（至關於數據刪除同步）；

Update:修改數據，存在則修改，不存在不操做。

upsert:找到匹配項則修改，找不到匹配項則添加。

勾選上紅色圈着的選項以後，源數據修改、添加了，在目標庫裏都會有相應的操做。不過還要設置下面的一步

ID爲主鍵match field for update時必定要選擇Y不然運行時出錯。

同步過程最主要的就是上邊列出的幾步設置，固然若是想要再設置更強大的功能，可詳細去研究官網的API

官網API地址：http://wiki.pentaho.com/display/EAI/

示例kri文件下載地址：http://files.cnblogs.com/nyzhai/mongodbTran.rar