ElasticSearch ik,elasticsearch-jdbc 使用和 yii2 實例

時間 2019-11-07

標籤 elasticsearch jdbc 使用 yii2 yii 實例欄目日誌分析简体版

原文原文鏈接

1.ES安裝+ik插件

mac安裝和使用:php

a.因爲ES爲java開發因此首先安裝下載jdk,因爲最新版本對jdk要求>1.8.0,因此能夠經過brew 升級jdkhtml

brew cask install caskroom/versions/java8

b.brew 安裝ESjava

$ brew update
$ brew search elasticsearch
elasticsearch            elasticsearch@2.4        elasticsearch@5.6        
$ brew install elasticsearch@2.4  #版本不須要安裝過高,否則有的插件用不了
....
$ brew info elasticsearch@2.4 #安裝成功後信息
elasticsearch@2.4: stable 2.4.6 [keg-only]
Distributed search & analytics engine
https://www.elastic.co/products/elasticsearch
/usr/local/Cellar/elasticsearch@2.4/2.4.6 (78 files, 38.5MB)
  Built from source on 2017-12-15 at 11:49:26
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/elasticsearch@2.4.rb
==> Requirements
Required: java >= 1.7 ✔
==> Caveats
Data:    /usr/local/var/elasticsearch/elasticsearch_lph/
Logs:    /usr/local/var/log/elasticsearch/elasticsearch_lph.log
Plugins: /usr/local/opt/elasticsearch@2.4/libexec/plugins/
Config:  /usr/local/etc/elasticsearch/
plugin script: /usr/local/opt/elasticsearch@2.4/libexec/bin/plugin

This formula is keg-only, which means it was not symlinked into /usr/local,
because this is an alternate version of another formula.

If you need to have this software first in your PATH run:
  echo 'export PATH="/usr/local/opt/elasticsearch@2.4/bin:$PATH"' >> ~/.bash_profile


To have launchd start elasticsearch@2.4 now and restart at login:
  brew services start elasticsearch@2.4
Or, if you don't want/need a background service you can just run:
  elasticsearch

$ elasticsearch #啓動elasticsearch

啓動 elasticsearch 後,在瀏覽器地址:127.0.0.1:9200,瀏覽器會顯示一段json數據node

這裏出現一個錯誤: 再次用brew 啓動 elasticsearch時候mysql

 $ brew services start elasticsearch@2.4 
 或者
 $ elasticsearch
 #報錯
 ...
 failed to obtain node locks, tried [[/usr/local/var/lib/elasticsearch/elasticsearch_lph]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
 ...

127.0.0.1:9200不能進行訪問,這是由於已經有個節點在線程中跑了,能夠查看線程linux

$ ps aux | grep "elastics"
lph              86699   0.0  0.0  2442020    816 s000  S+    2:02下午   0:00.00 grep elastics
lph              86662   0.0 11.0  5079720 920492   ??  S     2:01下午   0:19.68 /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home....
$ kill -9 86662 #殺死jdk跑的elasticsearch 線程

以後 elasticsearch就能正常啓動了git

 $ brew services start elasticsearch@2.4

修改elasticsearch配置github

在安裝完成後 brew info elasticsearch,顯示了對於配置文件目錄位置 /usr/local/etc/elasticsearch 打開elasticsearch.ymlweb

$ vim /usr/local/etc/elasticsearch/elasticsearch.yml
...
...
cluster.name : test  #集羣名稱
...
node.name: master-1   #當前節點
...
network.host: 127.0.0.1

保存修改後,啓動elasticsearchsql

$ brew services start elasticsearch@2.4

外網訪問elasticsearch配置

1.elasticsearch.yml 中network.host: 127.0.0.1 改成 network.host: 0.0.0.0

2.開啓unbuntu 防火牆對9200的容許

$ sudo ufw status #查看防火牆狀態
22                         ALLOW       Anywhere
80                         ALLOW       Anywhere

$ sudo ufw allow from youallowip to any port 9200 #youallowip爲你容許操做的ip,若是沒有ip操做限制直接寫成:sudo ufw allow 9200

3.用雲平臺的朋友須要考慮雲端是否設置了9200端口是否被容許訪問

安裝中文分詞插件 elasticsearch-analysis-ik

Versions

IK version	ES version
master	6.x -> master
6.0.0	6.0.0
5.6.4	5.6.4
5.5.3	5.5.3
5.4.3	5.4.3
5.3.3	5.3.3
5.2.2	5.2.2
5.1.2	5.1.2
1.10.6	2.4.6
1.9.5	2.3.5
1.8.1	2.2.1
1.7.0	2.1.1
1.5.0	2.0.0
1.2.6	1.0.0
1.2.5	0.90.x
1.1.3	0.20.x
1.0.0	0.16.2 -> 0.19.0

本例子安裝ES 版本爲2.4.6,因此對應下載1.10.6插件, 點擊releases 查看v1.10.6版本複製下載地址

$ cd /usr/local/opt/elasticsearch@2.4/libexec/plugins/ #切換到elasticsearch插件位置
$ cd mkdir ik #建立ik目錄存放插件
$ cd ik
$ wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v1.10.6/elasticsearch-analysis-ik-1.10.6.zip #下載安裝包
$ unzip elasticsearch-analysis-ik-1.10.6.zip #解壓安裝包
$ rm elasticsearch-analysis-ik-1.10.6.zip #刪除壓縮包
$ brew services restart elasticsearch@2.4  #重啓elasticsearch@2.4

重啓以後查看log 表示插件加載成功 plugins [analysis-ik]

[2017-12-15 13:28:50,371][INFO ][plugins                  ] [master1] modules [reindex, lang-expression, lang-groovy], plugins [analysis-ik], sites []

對比 ik 和自帶對比測試

$ curl -XPOST "http://127.0.0.1:9200/_analyze?analyzer=ik&pretty" -d "你好世界"
{
  "tokens" : [ {
    "token" : "你好",
    "start_offset" : 0,
    "end_offset" : 2,
    "type" : "CN_WORD",
    "position" : 0
  }, {
    "token" : "世界",
    "start_offset" : 2,
    "end_offset" : 4,
    "type" : "CN_WORD",
    "position" : 1
  } ]
}

$ curl -XPOST "http://127.0.0.1:9200/_analyze?analyzer=standard&pretty" -d "你好世界"
{
  "tokens" : [ {
    "token" : "你",
    "start_offset" : 0,
    "end_offset" : 1,
    "type" : "<IDEOGRAPHIC>",
    "position" : 0
  }, {
    "token" : "好",
    "start_offset" : 1,
    "end_offset" : 2,
    "type" : "<IDEOGRAPHIC>",
    "position" : 1
  }, {
    "token" : "世",
    "start_offset" : 2,
    "end_offset" : 3,
    "type" : "<IDEOGRAPHIC>",
    "position" : 2
  }, {
    "token" : "界",
    "start_offset" : 3,
    "end_offset" : 4,
    "type" : "<IDEOGRAPHIC>",
    "position" : 3
  } ]
}

ubuntu安裝:

首先安裝jdk

sudo add-apt-repository ppa:webupd8team/java
##報錯:
The program 'add-apt-repository' is currently not installed. You can install it by typing:
apt-get install software-properties-common
須要先執行:apt-get install software-properties-common 再執行sudo add-apt-repository ppa:webupd8team/java


sudo apt-get update
sudo apt-get install oracle-java8-installer #都選yes
sudo apt-get install oracle-java8-set-default   #都選yes
java -version
whereis java
which java （java執行路徑）
echo $JAVA_HOME

導入 Elasticsearch 公共 GPG 密鑰到 apt：

$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

建立 Elasticsearch 資源列表：

$ echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list

安裝Elasticsearch

$ sudo apt-get update
$ sudo apt-get -y install elasticsearch
$ service elasticsearch start  #啓動elasticsearch

運行如下命令將在系統啓動時啓動 Elasticsearch：

$ sudo update-rc.d elasticsearch defaults 95 10

查看版本

root@iZwz92p0vf7mrdbn3qwl40Z:~# curl -XGET localhost:9200
{
  "name" : "masterr-1",
  "cluster_name" : "deraceur",
  "cluster_uuid" : "VV0D289YTbamI939oKzkOA",
  "version" : {
    "number" : "2.4.6",#爲es版本號 ik 對應版本1.10.6
    "build_hash" : "5376dca9f70f3abef96a77f4bb22720ace8240fd",
    "build_timestamp" : "2017-07-18T12:17:44Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.4"
  },
  "tagline" : "You Know, for Search"
}

安裝maven (ik須要打包須要先按照maven)

1 下載安裝

$ wget http://mirror.bit.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz 
$ tar -zxvf apache-maven-3.3.9-bin.tar.gz 
$ sudo mv apache-maven-3.3.9 /usr/local/

2 爲maven配置環境變量

將以下內容添加到/etc/profile和/root/.bashrc中

export M2_HOME=/usr/local/apache-maven-3.3.9 
export PATH=$PATH:$M2_HOME/bin

執行以下指令使其上述配置生效或者重啓系統：

source /etc/profile

檢查安裝是否生效

$ mvn -v 
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00) 
Maven home: /usr/local/apache-maven-3.3.9 
Java version: 1.8.0_65, vendor: Oracle Corporation 
Java home: /usr/local/jdk/jdk1.8.0_65/jre 
Default locale: en_US, platform encoding: UTF-8 
OS name: 「linux」, version: 「3.19.0-25-generic」, arch: 「amd64」, family: 「unix」

安裝ik插件

git clone https://github.com/medcl/elasticsearch-analysis-ik.git
cd elasticsearch-analysis-ik/
git checkout -b v1.10.6
/usr/local/apache-maven-3.3.9/bin/mvn package  #mvn執行
mkdir /usr/share/elasticsearch/plugins/ik #建立ik
cp  target/releases/elasticsearch-analysis-ik-1.10.6.zip  /usr/share/elasticsearch/plugins/ik
cd  /usr/share/elasticsearch/plugins/ik
unzip elasticsearch-analysis-ik-1.10.6.zip

$ service elasticsearch restart  #從新啓動elasticsearch

修改elasticsearch配置

vim /etc/elasticsearch/elasticsearch.yml

# 添加如下內容
index.analysis.analyzer.default.type: ik

測試

root@dsfsd:~# curl -XGET 'http://127.0.0.1:9200/_analyze?pretty&analyzer=ik_max_word' -d '電動車不貼反光膜是違法行爲'
{
  "tokens" : [ {
    "token" : "電動車",
    "start_offset" : 0,
    "end_offset" : 3,
    "type" : "CN_WORD",
    "position" : 0
  }, {
    "token" : "電動",
    "start_offset" : 0,
    "end_offset" : 2,
    "type" : "CN_WORD",
    "position" : 1
  }, {
    "token" : "車",
    "start_offset" : 2,
    "end_offset" : 3,
    "type" : "CN_CHAR",
    "position" : 2
  }, {
    "token" : "不",
    "start_offset" : 3,
    "end_offset" : 4,
    "type" : "CN_CHAR",
    "position" : 3
  }, {
    "token" : "貼反",
    "start_offset" : 4,
    "end_offset" : 6,
    "type" : "CN_WORD",
    "position" : 4
  }, {
    "token" : "貼",
    "start_offset" : 4,
    "end_offset" : 5,
    "type" : "CN_WORD",
    "position" : 5
  }, {
    "token" : "反光膜",
    "start_offset" : 5,
    "end_offset" : 8,
    "type" : "CN_WORD",
    "position" : 6
  }, {
    "token" : "反光",
    "start_offset" : 5,
    "end_offset" : 7,
    "type" : "CN_WORD",
    "position" : 7
  }, {
    "token" : "膜",
    "start_offset" : 7,
    "end_offset" : 8,
    "type" : "CN_WORD",
    "position" : 8
  }, {
    "token" : "違法行爲",
    "start_offset" : 9,
    "end_offset" : 13,
    "type" : "CN_WORD",
    "position" : 9
  }, {
    "token" : "違法",
    "start_offset" : 9,
    "end_offset" : 11,
    "type" : "CN_WORD",
    "position" : 10
  }, {
    "token" : "違",
    "start_offset" : 9,
    "end_offset" : 10,
    "type" : "CN_WORD",
    "position" : 11
  }, {
    "token" : "法",
    "start_offset" : 10,
    "end_offset" : 11,
    "type" : "CN_CHAR",
    "position" : 12
  }, {
    "token" : "行爲",
    "start_offset" : 11,
    "end_offset" : 13,
    "type" : "CN_WORD",
    "position" : 13
  } ]
}

4.經過elasticsearch-jdbc操做數據(不推薦使用,這邊我只是記錄下)

使用elasticsearch-jdbc將現有MySQL數據批量導入至ElasticSearch(ubuntu環境運行)

下載並生成 mysql-import-product.sh

# wget http://xbib.org/repository/org/xbib/elasticsearch/importer/elasticsearch-jdbc/2.3.4.0/elasticsearch-jdbc-2.3.4.0-dist.zip  #若是下載速度很慢 能夠嘗試迅雷下載
# unzip elasticsearch-jdbc-2.3.4.0-dist.zip #解壓
# cd elasticsearch-jdbc-2.3.4.0-dist
# cd bin
# cp mysql-blog.sh mysql-import-product.sh
# vim mysql-import-product.sh

修改 mysql-import-product.sh

#!/bin/sh

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
bin=${DIR}/../bin
lib=${DIR}/../lib

echo '
{
    "type" : "jdbc",
    "jdbc" : {
        "url" : "jdbc:mysql://localhost:3306/test", #目標數據庫爲test
        "schedule" : "0 0-59 0-23 ? * *", #定時任務
        "user" : "root", #數據庫user
        "password" : "12345678",
        "sql" : "select productid,title from product" #導入數據語句
        "index" : "test", 
        "type" : "products",
       
        "elasticsearch" : {
             "cluster" : "test",#爲 elasticsearch config 中的 cluster.name
             "host" : "localhost",
             "port" : 9300 
        }   
    }
}
' | java \
    -cp "${lib}/*" \
    -Dlog4j.configurationFile=${bin}/log4j2.xml \
    org.xbib.tools.Runner \
    org.xbib.tools.JDBCImporter

保存後執行該腳本

# cd ../..
# bin/mysql-import-product.sh
bin/mysql-import-product.sh: 3: bin/mysql-import-product.sh: Bad substitution
bin/mysql-import-product.sh: 4: bin/mysql-import-product.sh: num: not found

#報錯 爲路勁錯誤,google不少說是jdk path問題,這邊沒有時間研究,直接修改路勁
# pwd  #顯示當前路勁
/root/elasticsearch-jdbc-2.3.4.0
# vim bin/mysql-import-product.sh  #修改以下
...
 bin=/root/elasticsearch-jdbc-2.3.4.0/bin
lib=/root/elasticsearch-jdbc-2.3.4.0/lib
....
# 退出保存
# bin/mysql-import-product.sh #繼續報錯 google後 爲jdk版本>1.8要求,本機爲1.7
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/xbib/tools/Runner : Unsupported major.minor version 52.0
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:803)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:442)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:64)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:354)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:312)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
# .... #papapa~ 升級jdk 升級方式請參考 http://blog.csdn.net/cuiaamay/article/details/51822308
# chmod 777 bin/mysql-import-product.sh #添加執行權限
# bin/mysql-import-product.sh 
# curl -XGET "http://127.0.0.1:9200/test/_search?pretty"
 #驗證導入時候成功
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 378,
  .......
  #有數據表示成功

指令記錄:

$  curl -XPOST "http://127.0.0.1:9200/test/_search?pretty"   #獲取全部
$  curl -XPOST "http://127.0.0.1:9200/test/_search?pretty" -d」@search.json」   #經過search.json檢索數據
$  curl -XDELETE "http://127.0.0.1:9200/test"    #刪除數據

查看更多 es指令操做

elasticsearch-jdbc自動導入增量數據

# cp mysql-import-product.sh mysql-delta-import-product.sh
# vim mysql-delta-import-product.sh

#!/bin/sh

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
bin=${DIR}/../bin
lib=${DIR}/../lib

echo '
{
    "type" : "jdbc",
    "jdbc" : {
        "url" : "jdbc:mysql://localhost:3306/test", #目標數據庫爲test
        "schedule" : "0 0-59 0-23 ? * *", #定時任務
        "user" : "root", #數據庫user
        "password" : "12345678",
        "sql" :[{
                   "statement": "select productid,title from product.updated_at > unix_timestamp(?) group by bshop_product_lang.productid",#當updated_at時間大於當前時間 執行數據導入
                    "parameter" : ["$metrics.lastexecutionstart"] #當前時間戳
        }],
        "index" : "test", 
        "type" : "products",
       
        "elasticsearch" : {
             "cluster" : "test",#爲 elasticsearch config 中的 cluster.name
             "host" : "localhost",
             "port" : 9300 
        }   
    }
}
' | java \
    -cp "${lib}/*" \
    -Dlog4j.configurationFile=${bin}/log4j2.xml \
    org.xbib.tools.Runner \
    org.xbib.tools.JDBCImporter

其中updated_at爲product更新或者建立時候的記錄時間戳.

添加mysql-delta-import-product.sh 腳本權限和寫進開機執行腳本

# chmod -X  mysql-delta-import-product.sh
# vim /etc/rc.local
....
sh /youpath/bin/mysql-delta-import-product.sh #ES數據增量 youpath爲文件絕對位置
exit 0
....

這樣開機將自動執行該腳本

5.經過 yii2-elasticsearch操做數據(推薦方法)

打開 yii2 根目錄下 composer.json

...
   "require": {
        ....
        "yiisoft/yii2-elasticsearch": "^2.0"
    },
...

保存後

$ composer update

完成後在yii2 配置文件中寫入

'components' => [
        'elasticsearch' => [
            'class' => 'yii\elasticsearch\Connection',
            'nodes' => [
                ['http_address' => '127.0.0.1:9200'],
                // configure more hosts if you have a cluster
            ],
        ],

建立common\models\ESProduct類繼承 yii\elasticsearch\ActiveRecord

<?php

namespace common\models;

use yii\elasticsearch\ActiveRecord;
/**
 * ProductSearch represents the model behind the search form about `common\models\Product`.
 */
class ESProductSearch extends ActiveRecord
{

 /**
     * @inheritdoc
     */

    //public  $index = "bshop";
    // public $type = "products";

    # 屬性
    public function attributes()
    {
        $mapConfig = self::mapConfig();
        return array_keys($mapConfig['properties']);
    }

    public function rules()
    {
        return [
            [["productid","title"],'required']
        ];
    }

    # mapping配置
  
    public static function mapConfig(){
        return [
            'properties' => [
                'productid'  => ['type' => 'double'],
                
                'title' => ['type' => 'string','analyzer'=>'ik', 'searchAnalyzer' => 'ik']
            ]
        ];
    }



    public static function mapping()
    {
        return [
            static::type() => self::mapConfig(),
        ];
    }



    //索引名稱
    public static function index()
    {
        return "test";
    }
    //索引類型
    public static function type()
    {
        return "products";
    }

    /**
     * Set (update) mappings for this model
     */
    public static function updateMapping()
    {
        $db = static::getDb();
        $command = $db->createCommand();
        $command->setMapping(static::index(), static::type(), static::mapping());
    }


    /**
     * Create this model's index
     */
    public static function createIndex()
    {
        $db = static::getDb();
        $command = $db->createCommand();
        $command->createIndex(static::index(), [
            'settings' => [ 'index' => ['refresh_interval' => '1s'] ],
            'mappings' => static::mapping(),
            //'warmers' => [ /* ... */ ],
            //'aliases' => [ /* ... */ ],
            //'creation_date' => '...'
        ]);
    }


    /**
     * Delete this model's index
     */
    public static function deleteIndex()
    {
        $db = static::getDb();
        $command = $db->createCommand();
        $command->deleteIndex(static::index(), static::type());
    }

    public static function updateRecord($productid, $columns){
        try {
            $record = self::get($productid);
            foreach($columns as $key => $value){
                $record->$key = $value;
            }
            return $record->update();
        }catch (\Exception $e){
            return false;
        }
    }

    //刪除記錄
    public static function deleteRecord($productid){
        try {
            $record = self::get($productid);
            $record->delete();
            return true;
        }catch (\Exception $e){
            return false;
        }
    }

    //保存數據(添加 || 更新)
    public function saveRecord($data){

        //加載數據
        if ($this->load($data) && $this->validate()){
            $isExist = false;
            try {
                $record = self::get($this->productid);
                //不存在 爲更新
                if (!$record){
                    $record = new self();
                    $record->setPrimaryKey($this->productid);
                }else{
                    $isExist = true;
                }
            }catch (\Exception $e){
                $record = new self();
                $record->setPrimaryKey($this->productid);
            }
            //遍歷數據到對象
            foreach ($data['ESProductSearch'] as $key => $value) {
                $record->$key = $value;
            }

            try {

                if (!$isExist){
                    //插入
                    $result = $record->insert();
                }else{
                    //更新
                    $result = $record->update();
                }

            }catch (\Exception $e){
                return false;

            }
            return true;
        }
    }



   


}

建立測試TestControlle.php

<?php
/**
 * Created by PhpStorm.
 * User: yidashi
 * Date: 2017/4/14
 * Time: 下午11:33
 */

namespace api\modules\v1\models;
use common\models\ESProductSearch;
class TestController extends Controller{

    //查詢
    public function actionTestES(){
        //es 測試
        //搜索詞
        $keyword = "測試";
        $res = ESProductSearch::find()->query([
            "multi_match" => [
                "query" => $keyword,
                "fields" => ["title"] //檢索屬性
            ],
        ])->all();
        foreach ($res as $k){
            var_dump($k->productid);//輸出id
        }
        exit;
    }

    //建立 或更新
    public function actionCreateES(){
        $ESArray = [
          'ESProductSearch' => [
                'productid' => '1',
                'title'     => 'hello world'
           ]
        ];

        $eSProductSearch = new ESProductSearch();
         if(!$eSProductSearch -> saveRecord($ESArray)){
                   throw new \Exception(Common::getLayoutMessageHtml("操做失敗"));
        }        
       
    }
    
    //刪除
    public function actionDeteleES($id){
        
        $eSProductSearch = new ESProductSearch();
         if(!$eSProductSearch->deleteByProduct($id)){
                   throw new \Exception(Common::getLayoutMessageHtml("刪除失敗"));
        }        
       
    }

}