Flink-1.8.1踩坑記錄 - HDFS的scheme不支持問題

前置環境說明：

centos 6.10

hadoop 2.7.3

oracle jdk1.8

 flink1.8.1

scala 2.11

安裝完集羣以後提交一個jar包中的類運行報錯 , （這個類裏面寫了操做HDFS的內容）

2019-09-02 18:42:01,651 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Job DimensionState (1b47949b019929e25b8bee89539f9f1d) switched from state FAILING to FAILED.
java.lang.RuntimeException: Error while creating FileSystem when initializing the state of the BucketingSink.
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:379)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:403)
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.createHadoopFileSystem(BucketingSink.java:1227)
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initFileSystem(BucketingSink.java:432)
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:376)
	... 8 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.
	at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:64)
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399)
	... 11 more
2019-09-02 18:42:01,652 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Could not restart the job DimensionState (1b47949b019929e25b8bee89539f9f1d) because the restart strategy prevented it.
java.lang.RuntimeException: Error while creating FileSystem when initializing the state of the BucketingSink.
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:379)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:403)
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.createHadoopFileSystem(BucketingSink.java:1227)
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initFileSystem(BucketingSink.java:432)
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:376)
	... 8 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.
	at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:64)
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399)
	... 11 more
2019-09-02 18:42:01,652 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Stopping checkpoint coordinator for job 1b47949b019929e25b8bee89539f9f1d.
2019-09-02 18:42:01,652 INFO  org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore  - Shutting down
2019-09-02 18:42:01,662 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Job 1b47949b019929e25b8bee89539f9f1d reached globally terminal state FAILED.
2019-09-02 18:42:01,704 INFO  org.apache.flink.runtime.jobmaster.JobMaster                  - Stopping the JobMaster for job DimensionState(1b47949b019929e25b8bee89539f9f1d).
2019-09-02 18:42:01,711 INFO  org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl      - Suspending SlotPool.
2019-09-02 18:42:01,711 INFO  org.apache.flink.runtime.jobmaster.JobMaster                  - Close ResourceManager connection 0f8cea6c8e8a3904e338bfe4b4aca43b: JobManager is shutting down..
2019-09-02 18:42:01,711 INFO  org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl      - Stopping SlotPool.
2019-09-02 18:42:01,711 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Disconnect job manager 00000000000000000000000000000000@akka.tcp://flink@localhost:6123/user/jobmanager_0 for job 1b47949b019929e25b8bee89539f9f1d from the resource manager.
2019-09-02 18:42:01,712 INFO  org.apache.flink.runtime.jobmaster.JobManagerRunner           - JobManagerRunner already shutdown.

 我提取一下上面關鍵字 ,方便搜索引擎爬取, 以試圖幫助其餘人:

1. while creating FileSystem when initializing the state of the BucketingSink.

2. Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.

3.org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.

1.8.1若是要操做hdfs的話，必需要在flink安裝目錄的 lib 下加上額外的jar包

官網有介紹:

https://ci.apache.org/projects/flink/flink-docs-release-1.8/release-notes/flink-1.8.html

因此這裏緣由就是在flink集羣的CLASS_PATH下缺乏了 HDFS相關的ja 

解決辦法:

jar包下載地址:

https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.7.5-7.0/flink-shaded-hadoop-2-uber-2.7.5-7.0.jar