Nutch環境搭建

1. 環境準備

HOST：Ubuntu12.04LTS

JDK： jdk-7u45-linux-i586.rpm

Nutch：apache-nutch-1.7-bin.tar.gz

Solr：solr-4.6.0-src.tgz

工做目錄： /home/zephyr/

 

1.1JDK安裝

爲了下載快點，一下腦殘下了rpm.

可Ubuntu上沒有rpm工具apt-get install rpm 後rpm -ivh jdk-7u45-linux-i586.rpm

提示要使用alien

apt-get install alien alien -i jdk-7u45-linux-i586.rpm

alien -d jdk-7u45-linux-i586.rpm 生成deb文件 不可用

無奈，從新使用apt-get install oracle-java7-installer* 直接使用失敗

add-apt-repository ppa:webupd8team/java

apt-get update

apt-get install oracle-java7-installer

到/etc/profile 文件的最後增長上如下幾行

export JAVA_HOME=/usr/java/jdk1.7.0_45

export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar

export NUTCH_HOME=/home/zephyr/apache-nutch-1.7

export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$NUTCH_HOME/bin:$PATH

 

1.2Tomcat安裝

apt-get install tomcat7 下載tomcat

修改tomcat的端口使用狀況爲

vi /etc/tomcat7/server.xml

<Connector port="8088" protocol="HTTP/1.1"

connectionTimeout="20000"

URIEncoding="UTF-8"

redirectPort="8443" />

/etc/init.d/tomcat7 start

 

1.3Nutch 安裝

工做目錄下解壓 tar xzvf apache-nutch-1.7-bin.tar.gz

測試Nutch 是否能正常工做

在urls目錄touch一個seed.txt echo http://nutch.apache.org/ >> seed.txt

修改conf/regex-urlfilter.txt

   # accept anything else

     +.

爲

   +^http://([a-z0-9]*\.)*nutch.apache.org/

編輯conf/nutch-site.xml

<property>

 <name>http.agent.name</name>

 <value>My Nutch Spider</value>

</property>

.nutch crawl ../urls -dir ../test -depth 1 -topN 10

報錯

java/lang/NoClassDefFoundError: java/lang/Object

Exception in thread "main" java.lang.InternalError: internal error: SHA-1 not available

修正JDK

cd /usr/java/jdk1.7.0_45/lib 將pack結尾的文件都轉成 jar 如：unpack200 tools.pack tools.jar

/usr/java/jdk1.7.0_45/jre/lib 也作相同操做
再操做成功。

 

1.4 Solr安裝

tar xzvf solr-4.6.0-src.tgz

ivy未安裝，先安裝ivy, apt-get install ivy

ant compile 失敗，修正ln -s -T /usr/share/java/ivy.jar /usr/share/ant/lib

/ivy.jar 編譯無響應

tar xzvf solr-4.6.0.tgz

把Solr 預設的core 範例collection1 複製爲core-nutch

把Nutch 提供的conf/schema-solr4.xml 覆蓋掉Solr core-nutch 的conf/schema.xml

將Solr core-nutch conf/schema.xml補上一行漏掉的欄位設定<field name=」_version_」 type=」long」 stored=」true」 indexed=」true」 multiValued=」false」/>

將core.properties中的name 改成 core-nutch

nutch crawl /home/zephyr/apache-nutch-1.7/urls -solr http://192.168.153.128:8983/solr/core-nutch/ -threads 20 -depth 1 -topN 10

也能夠使用nutch solrindex http://192.168.153.128:8983/solr/core-nutch/ /home/zephyr/apache-nutch-1.7/test/crawldb -linkdb /home/zephyr/apache-nutch-1.7/test/linkdb/ /home/zephyr/apache-nutc

h-1.7/test/segments/*

 

1.5 solr經過tomcat7部署

apt-get install tomcat7-admin

/etc/tomcat7編輯tomcat-users.xml

<role rolename="manager-gui"/>

<role rolename="admin-gui"/>

<user username="tomcat" password="tomcat" roles="manager-gui,admin-gui"/>

cp /home/zephyr/solr-4.6.0/example/webapps/solr.war /var/lib/tomcat7/webapps/solr.war

cp –r /home/zephyr/solr-4.6.0/example/solr /var/lib/tomcat7/solr

cd /var/lib/tomcat7/conf/Catalina/localhost

vi solr.xml

<Context docBase="/var/lib/tomcat7/webapps/solr.war" debug="0" crossContext="true" >
   <Environment name="solr/home" type="java.lang.String" value="/var/lib/tomcat7/solr" override="true" />
</Context>

cp /home/zephyr/solr-4.6.0/example/lib/ext/*.jar /usr/share/tomcat7/lib

cp /home/zephyr/solr-4.6.0/example/resources/log4j.properties /usr/share/tomcat7/lib/

vi log4j.properties 修改 solr.log=/var/logs/tomcat7/

不然solr沒法在tomcat啓動

「One of the things about running Solr in a container other than the embedded Jetty is that we need to setup the logging. This step is crucial, without doing this solr will never start and you will pull all of your hair out trying to figure out why」

啓動運行提示出錯：

SolrCore 'collection1' is not available due to init failure: Index locked for write for core collection1

cd /var/lib/tomcat7/solr/ collection1

chown tomcat7 data -R

cd /var/lib/tomcat7/solr/core-nutch

chown tomcat7 data –R

 

1.6 Nutch添加中文支持

編輯/home/zephyr/apache-nutch-1.7/conf/nutch-site.xml

<property>

<name>http.accept.language</name>

<value>ja-jp, en-us, zh-cn,en-gb,en;q=0.7,*;q=0.3</value>

<description>Value of the 「Accept-Language」 request header field.

   This allows selecting non-English language as default one to retrieve.

   It is a useful setting for search engines build for certain national group.

</description>

</property>