Solr/Lucene分佈式搜索,Solr Integrate katta step3

前面的兩篇介紹了安裝katta及ZooKeeper,後邊來介紹katta的Node.

咱們回到step1 後邊提到的solr-katta-plugin項目,源碼導入後會出現不少的錯誤,在項目中繼承了solr-core,和solrj中的類嘗試着把訪問修飾private改成protected.

如:solr-core org.apache.solr.handler.component.SearchHandler類中的shardHandlerFactory成員變量

protected ShardHandlerFactory shardHandlerFactory = new HttpShardHandlerFactory();

同時借鑑 https://issues.apache.org/jira/browse/SOLR-1395 Tomliu的作法,把

the bugs is :
1. solr's ShardDoc.java, ShardFieldSortedHitQueue line 210 :
        final float f1 = e1.score == null ? 0.00f : e1.score;
        final float f2 = e2.score == null ? 0.00f : e2.score;

等等.直到項目基本錯誤解決.

下載solr而且copy其中的apache-solr-3.6.1/example/solr到//data下.

目錄結構如:

solr
  --bin/
  --conf/
    --schama.xml
    --solrconfig.xml
    --...
  --data/
    --index/
    --spellchecker/
  --solr.xml
  --README.txt

咱們視/data/solr 爲solr.home,改變${solr.home}/solr.xml爲:

<solr persistent="false">

  <!--
  adminPath: RequestHandler path to manage cores.  
    If 'null' (or absent), cores will not be manageable via request handler
  -->
  <cores adminPath="/admin/cores" defaultCoreName="proxy">
    <core name="proxy" instanceDir="proxy" />
  </cores>
</solr>

改變後的目錄結構爲[ 在solr.home 中新建proxy文件夾並把conf/ data/放到proxy文件夾下面],如:

solr/
  --proxy/
    --bin/
    --conf/
      --schama.xml
      --solrconfig.xml
      --....
    --data/
      --index/
      --spellchecker
  --solr.xml

上面的步驟在我嘗試的時候費神了好久,爲何是這樣的?我在代碼發現的:org.apache.solr.katta.DeployableSolrKattaServer

final public static String serverNameDefault = "proxy";
	final public static String serverNameProperty = "solr.server.name";

	final public static String solrHomeDefault = "solrHome";
	final public static String solrHomeProperty = "solr.home";

	final public static String solrConfigFileDefault = "solr.xml";
	final public static String solrConfigFileProperty = "solr.config.name";

public static String getServerName() {
      return System.getProperty(serverNameProperty, serverNameDefault);
 }

因此solr.home/solr.xml中配置的默認的solr coreName必須爲proxy,不然啓動Node的時候須要加上-Dsolr.server.name=youname,這裏我不更改了使用它的配置

更改solrconfig.xml中增長[可加在<requestHandler name="search" class="solr.SearchHandler">以前,並去掉search的default=true]:

<requestHandler name="standard" class="solr.MultiEmbeddedSearchHandler" default="true">
    <!-- default values for query parameters -->
     <lst name="defaults">
       <str name="echoParams">explicit</str>
       <int name="rows">10</int>
     </lst>
  </requestHandler>

ok,咱們在把配置好了的solr複製一份爲solrhome1.這時咱們有了/data/solrhome1

如今copy ${katta_install}/conf/ katta.node.properties, katta.zk.properties, log4j.properties到solr-katta-plugin項目的test/下[test爲測試源碼目錄],更改katta.node.properties爲

#node.server.class=net.sf.katta.lib.lucene.LuceneServer
node.server.class=org.apache.solr.katta.DeployableSolrKattaServer

建立一個啓動類:

public class Launcher {

    public final static DateFormat DATE_FORMAT = new SimpleDateFormat("yy_MM_dd_HH_mm_ss");

    static {
        System.setProperty("solr.home", "/data/solr");

        System.setProperty("solr.server.name", "proxy");

        System.setProperty("solr.directoryFactory", "solr.MMapDirectoryFactory");

        System.setProperty("katta.log.dir", "/data/logs");

        System.setProperty("katta.log.file", "katta_" + DATE_FORMAT.format(new Date()) + ".log");

    }

    public Launcher() {


    }

    public static void main(String[] args) throws Exception {
        Katta.main(new String[]{"startNode", "-c", "org.apache.solr.katta.DeployableSolrKattaServer"});
    }
}

運行該類,咱們能夠看到它加載了solr

10-18 14:22:02 [INFO 級別][ core.SolrCore ]類 (83 行) QuerySenderListener done.
10-18 14:22:02 [INFO 級別][ component.SpellCheckComponent ]類 (673 行) Loading spell index for spellchecker: default
10-18 14:22:02 [INFO 級別][ core.SolrCore ]類 (1325 行) [] Registered new searcher Searcher@70f5d656 main
10-18 14:22:02 [INFO 級別][ ipc.Server ]類 (328 行) Starting SocketReader
10-18 14:22:02 [INFO 級別][ node.Node ]類 (226 行) DeployableSolrKattaServer server started on : zhenqin-K45VM:20000
10-18 14:22:02 [INFO 級別][ ipc.Server ]類 (598 行) IPC Server Responder: starting
10-18 14:22:02 [INFO 級別][ ipc.Server ]類 (434 行) IPC Server listener on 20000: starting
10-18 14:22:02 [INFO 級別][ ipc.Server ]類 (1358 行) IPC Server handler 0 on 20000: starting

而後,回到終端執行:$bin/katta addIndex solrhome1 file:///data/solrhome1

固然,咱們也能夠把該solrhome1打包放入HDFS,$bin/katta addIndex solrhome1 hdfs://localhost:9000/solr/solrhome1.zip

$ bin/katta addIndex solrhome1 file:///data/solrhome1
.....
deployed index 'solrhome1' in 4840 ms

這是能夠看見Eclipse控制檯輸出了:

10-18 14:24:02 [INFO 級別][ node.Node ]類 (279 行) executing ShardDeployOperation:3cca2475:[solrhome1#proxy]
10-18 14:24:02 [INFO 級別][ node.AbstractShardOperation ]類 (55 行) deploy shard 'solrhome1#proxy'
10-18 14:24:02 [INFO 級別][ node.ShardManager ]類 (104 行) install shard 'solrhome1#proxy' from file:/media/Study/data/katta/testIndexA/proxy
10-18 14:24:03 [WARN 級別][ util.NativeCodeLoader ]類 (52 行) Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
zhenqin-K45VM:20000 addShard solrhome1#proxy solrhome:/media/Study/data/katta/katta-shards/zhenqin-K45VM_20000/solrhome1#proxy
10-18 14:24:04 [INFO 級別][ core.SolrResourceLoader ]類 (103 行) new SolrResourceLoader for directory: '/media/Study/data/katta/katta-shards/zhenqin-K45VM_20000/solrhome1#proxy/'
10-18 14:24:04 [INFO 級別][ core.CoreContainer ]類 (448 行) Creating SolrCore 'solrhome1#proxy' using instanceDir: /media/Study/data/katta/katta-shards/zhenqin-K45VM_20000/solrhome1#proxy
10-18 14:24:04 [INFO 級別][ core.SolrResourceLoader ]類 (103 行) new SolrResourceLoader for directory: '/media/Study/data/katta/katta-shards/zhenqin-K45VM_20000/solrhome1#proxy/'
10-18 14:24:05 [WARN 級別][ core.SolrConfig ]類 (148 行) <indexDefaults> and <mainIndex> configuration sections are deprecated (but still work). Please use <indexConfig> instead.
10-18 14:24:05 [INFO 級別][ core.SolrConfig ]類 (162 行) Using Lucene MatchVersion: LUCENE_36
10-18 14:24:05 [INFO 級別][ core.Config ]類 (248 行) Loaded SolrConfig: solrconfig.xml
10-18 14:24:05 [INFO 級別][ schema.IndexSchema ]類 (413 行) Reading Solr Schema
10-18 14:24:05 [INFO 級別][ schema.IndexSchema ]類 (427 行) Schema name=collection1
10-18 14:24:05 [INFO 級別][ node.AbstractShardOperation ]類 (75 行) publish shard 'solrhome1#proxy'
10-18 14:24:05 [INFO 級別][ core.SolrCore ]類 (1325 行) [solrhome1#proxy] Registered new searcher Searcher@61db327f main
10-18 14:24:05 [INFO 級別][ core.SolrCore ]類 (43 行) QuerySenderListener sending requests to Searcher@48b4e8e2 main
10-18 14:24:05 [INFO 級別][ core.SolrCore ]類 (1386 行) [solrhome1#proxy] webapp=null path=null params={event=firstSearcher&q=solr+first+searching..} hits=0 status=0 QTime=6 
10-18 14:24:05 [INFO 級別][ core.SolrCore ]類 (83 行) QuerySenderListener done.
10-18 14:24:05 [INFO 級別][ component.SpellCheckComponent ]類 (673 行) Loading spell index for spellchecker: default
10-18 14:24:05 [INFO 級別][ search.SolrIndexSearcher ]類 (256 行) Closing Searcher@61db327f main
	fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
	filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
	queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=1,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
	documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
10-18 14:24:05 [INFO 級別][ core.SolrCore ]類 (1325 行) [solrhome1#proxy] Registered new searcher Searcher@48b4e8e2 main

至此,一個Node加載一個Shard完成. step3 is over!