Java Spark讀取ElasticSearch數據，並分析

時間 2019-11-11

標籤 java spark 讀取 elasticsearch 數據分析欄目 Java 简体版

原文原文鏈接

直接在ES服務器上獲取的數據以下所示java

使用的java代碼以下所示（能夠看到數據都被封裝成了tuple）：node

package mainPackage;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.VoidFunction;
import org.apache.spark.sql.SparkSession;
import org.elasticsearch.spark.rdd.api.java.JavaEsSpark;
import scala.Tuple2;

import java.util.Map;

public class MainClass {
    public static void main(String[] args){
        run1();
    }
    private static void run1(){
        SparkConf conf = new SparkConf();
        conf.setMaster("local[1]");
        conf.setAppName("SPARK ES");

        conf.set("es.index.auto.create", "true");
        conf.set("es.nodes", "roc-3");
        conf.set("es.port", "8086");

        JavaSparkContext sc = new JavaSparkContext(conf);
        sc.setLogLevel("ERROR");

        JavaPairRDD<String, Map<String, Object>> esRDD =
                JavaEsSpark.esRDD(sc, "my-index4/person");

        for(Tuple2 tuple:esRDD.collect()){
            System.out.print(tuple._1()+"----------");
            System.out.println(tuple._2());
        }

    }
}

運行的語句以下：es6

spark2-submit --master local[*] --packages org.elasticsearch:elasticsearch-hadoop:6.4.0 --class mainPackage.MainClass --repositories http://maven.aliyun.com/nexus/content/groups/public,http://conjars.org/repo/  AID-1.0-SNAPSHOT.jar

若是讀取的數據十分巨大，能夠用下面的執行語句sql

spark2-submit --master local[*]  --executor-memory 16G --driver-memory 32G  --packages org.elasticsearch:elasticsearch-hadoop:6.4.0 --class mainPackage.MainClass --repositories http://maven.aliyun.com/nexus/content/groups/public,http://conjars.org/repo/ --conf spark.yarn.executor.memoryOverhead=8196 --conf spark.rpc.message.maxSize=2000 --conf spark.driver.maxResultSize=32g --conf spark.kryoserializer.buffer.max.mb=2040 --conf spark.debug.maxToStringFields=200  AID-1.0-SNAPSHOT.jar

設置repository可使用指定的資源庫下載jar數據庫

最終結果數據apache

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
18/09/07 19:19:21 INFO SparkContext: Running Spark version 2.0.0
……
省略N行
……
18/09/07 19:19:23 INFO MemoryStore: MemoryStore started with capacity 893.1 MB
18/09/07 19:19:23 INFO SparkEnv: Registering OutputCommitCoordinator
18/09/07 19:19:23 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/09/07 19:19:23 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.116.16.141:4040
18/09/07 19:19:23 INFO Executor: Starting executor ID driver on host localhost
18/09/07 19:19:23 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 10681.
18/09/07 19:19:23 INFO NettyBlockTransferService: Server created on 10.116.16.141:10681
18/09/07 19:19:23 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.116.16.141, 10681)
18/09/07 19:19:23 INFO BlockManagerMasterEndpoint: Registering block manager 10.116.16.141:10681 with 893.1 MB RAM, BlockManagerId(driver, 10.116.16.141, 10681)
18/09/07 19:19:23 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.116.16.141, 10681)
1----------{desc=數據庫管理, user=張三, title=工程師}
7----------{desc=數據庫管理, user=張5, title=工程師}
13----------{desc=數據庫管理, user=張11, title=工程師}
16----------{desc=數據庫管理, user=張14, title=工程師}
18----------{desc=數據庫管理, user=張16, title=工程師}
2----------{desc=數據庫管理, user=張0, title=工程師}
4----------{desc=數據庫管理, user=張2, title=工程師}
6----------{desc=數據庫管理, user=張4, title=工程師}
15----------{desc=數據庫管理, user=張13, title=工程師}
20----------{desc=數據庫管理, user=張18, title=工程師}
14----------{desc=數據庫管理, user=張12, title=工程師}
19----------{desc=數據庫管理, user=張17, title=工程師}
5----------{desc=數據庫管理, user=張3, title=工程師}
8----------{desc=數據庫管理, user=張6, title=工程師}
9----------{desc=數據庫管理, user=張7, title=工程師}
10----------{desc=數據庫管理, user=張8, title=工程師}
12----------{desc=數據庫管理, user=張10, title=工程師}
21----------{desc=數據庫管理, user=張19, title=工程師}
3----------{desc=數據庫管理, user=張1, title=工程師}
11----------{desc=數據庫管理, user=張9, title=工程師}
17----------{desc=數據庫管理, user=張15, title=工程師}

Process finished with exit code 0

一些bug，配置版本和jackon時候須要注意，若是配置成以下就會出問題：json

2.4.4。api

而用這個版本就沒事兒bash

默認es6.4.0版本會使用2.6.5版本的jackson來解析json數據，這樣就OK服務器

實驗證實6.4.0版本es鏈接6.3.1取數不會遇到版本不兼容問題

最終成功的maven文件配置：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>SparkReadES</groupId>
    <artifactId>AID</artifactId>
    <version>1.0-SNAPSHOT</version>

    <repositories>
        <repository>
            <id>central</id>
            <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>


    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
        <spark.version>2.0.0</spark.version>
        <elasticsearch.version>6.4.0</elasticsearch.version>
    </properties>


    <dependencies>
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch-spark-20_2.11</artifactId>
            <version>${elasticsearch.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-yarn_2.11</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

    </dependencies>


    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.6.0</version>
                <configuration>
                    <encoding>utf-8</encoding><!-- 指定項目源文件編碼 -->
                    <source>${maven.compiler.source}</source><!-- 指定項目源文件jdk版本 -->
                    <target>${maven.compiler.target}</target><!-- 指定項目生成目標文件的jdk版本 -->
                </configuration>
            </plugin>

        </plugins>
    </build>
</project>

Done