實現MySQL表數據全量索引和增量索引，基於Solr DIH組件實現起來比較簡單，只須要重複使用Solr的DIH（Data Import Handler）組件，對data-config.xml進行簡單的修改便可。Solr DIH組件的實現類爲org.apache.solr.handler.dataimport.DataImportHandler，在Solr的 solrconfig.xml中配置兩個handler，配置分別說明以下。html
全量索引mysql
solrconfig.xml配置以下：sql
1 |
<requestHandler name="/dataimport" |
2 |
class="org.apache.solr.handler.dataimport.DataImportHandler"> |
4 |
<str name="config">data-config.xml</str> |
上面這個是針對全量索引的，主要是配置data-config.xml文件，示例以下所示：數據庫
02 |
<dataSource name="jdbc" driver="com.mysql.jdbc.Driver" |
04 |
user="developer" password="sedept@shiyanjun.cn"/> |
05 |
<document name="mkt_data"> |
06 |
<entity name="marketing_data" pk="id" |
07 |
query="select * from marketing_data limit ${dataimporter.request.length} offset ${dataimporter.request.offset}" |
08 |
transformer="RegexTransformer"> |
09 |
<field column="id" name="id" /> |
10 |
<field column="domain" name="domain" /> |
11 |
<field column="alex_rank" name="alex_rank" /> |
12 |
<field column="server_port" name="server_port" /> |
13 |
<field column="cert_validity_notBefore" name="cert_validity_notBefore" /> |
14 |
<field column="cert_validity_notAfter" /> |
15 |
<field column="cert_validity_notAfter_yyyyMMdd" regex="(.*?)\s+.*" name="cert_validity_notAfter_yyyyMMdd" sourceColName="cert_validity_notAfter"/> |
16 |
<field column="cert_issuer_brand" name="cert_issuer_brand" /> |
17 |
<field column="cert_validation" name="cert_validation" /> |
18 |
<field column="cert_isMultiDomain" name="cert_isMultiDomain" /> |
19 |
<field column="cert_issuer_brand_isXRelated" name="cert_issuer_brand_isXRelated" /> |
20 |
<field column="cert_isWildcard" name="cert_isWildcard" /> |
21 |
<field column="cert_notAfter" name="cert_notAfter" /> |
22 |
<field column="special_ssl" name="special_ssl" /> |
23 |
<field column="competitor_logo" name="competitor_logo" /> |
24 |
<field column="segment" name="segment" /> |
上面主要是經過內置變量 「${dataimporter.request.length}」和 「${dataimporter.request.offset}」來設置一個批次索引的數據表記錄數，請求的URL示例以下：apache
上面表示，對數據表中id範圍爲[10000000, 1100000]的記錄進行索引，由於數據表可能達到千萬記錄數，並且線上有業務在操做數據庫，因此要選擇分批進行索引。dom
增量索引url
solrconfig.xml配置以下：code
01 |
<requestHandler name="/deltaimport" class="org.apache.solr.handler.dataimport.DataImportHandler"> |
03 |
<str name="config">delta-data-config.xml</str> |
06 |
上面定義請求的接口爲「deltaimport」，對應的配置文件delta- data-config.xml內容以下所示： |
08 |
<dataSource name="jdbc" driver="com.mysql.jdbc.Driver" |
10 |
user="developer" password="sedept@shiyanjun.cn"/> |
11 |
<document name="mkt_data"> |
12 |
<entity name="marketing_data" pk="id" |
13 |
query="select * from marketing_data" |
14 |
deltaImportQuery="select * from marketing_data where id='${dih.delta.id}'" |
15 |
deltaQuery="select id from marketing_data where updated_at > '${dih.last_index_time}'" |
16 |
transformer="RegexTransformer"> |
17 |
<field column="id" name="id" /> |
18 |
<field column="domain" name="domain" /> |
19 |
<field column="alex_rank" name="alex_rank" /> |
20 |
<field column="server_port" name="server_port" /> |
21 |
<field column="cert_validity_notBefore" name="cert_validity_notBefore" /> |
22 |
<field column="cert_validity_notAfter" /> |
23 |
<field column="cert_validity_notAfter_yyyyMMdd" regex="(.*?)\s+.*" name="cert_validity_notAfter_yyyyMMdd" sourceColName="cert_validity_notAfter"/> |
24 |
<field column="cert_issuer_brand" name="cert_issuer_brand" /> |
25 |
<field column="cert_validation" name="cert_validation" /> |
26 |
<field column="cert_isMultiDomain" name="cert_isMultiDomain" /> |
27 |
<field column="cert_issuer_brand_isXRelated" name="cert_issuer_brand_isXRelated" /> |
28 |
<field column="cert_isWildcard" name="cert_isWildcard" /> |
29 |
<field column="cert_notAfter" name="cert_notAfter" /> |
30 |
<field column="special_ssl" name="special_ssl" /> |
31 |
<field column="competitor_logo" name="competitor_logo" /> |
32 |
<field column="segment" name="segment" /> |
上面主要是經過內置變量「${dih.delta.id}」和 「${dih.last_index_time}」來記錄本次索引的id和最後索引時間。這裏，會保存在deltaimport.properties文件中，示例以下：orm
1 |
#Sat Feb 16 17:48:49 CST 2013 |
2 |
last_index_time=2013-02-16 17\:48\:48 |
3 |
marketing_data.last_index_time=2013-02-16 17\:48\:48 |
請求增量索引的接口爲「deltaimport」，能夠寫一個定時腳本，例如每隔一天執行一次 （一天索引一次，只索引數據表中更新的記錄），請求URL示例以下：server
1 |
172.0.8.212:8080/search-server/core0/dataimport?command=delta-import |
有關「query」，「deltaImportQuery」， 「deltaQuery」的解釋，引用官網說明，以下所示：
The query gives the data needed to populate fields of the Solr document in full-import
The deltaImportQuery gives the data needed to populate fields when running a delta-import
The deltaQuery gives the primary keys of the current entity which have changes since the last index time
還有更靈活的索引方式，之後再講。
參考連接
