[b0009] 玩Hadoop中碰到的各類錯誤

1. Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found

hadoop@ssmaster:~/java_program$ hadoop  jar hadoop_mapr_wordcount.jar  /input /output 
16/10/24 08:22:41 INFO client.RMProxy: Connecting to ResourceManager at ssmaster/192.168.249.144:8032
16/10/24 08:22:42 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/10/24 08:22:42 WARN mapreduce.JobResourceUploader: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
16/10/24 08:22:42 INFO input.FileInputFormat: Total input paths to process : 1
16/10/24 08:22:43 INFO mapreduce.JobSubmitter: number of splits:1
16/10/24 08:22:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1477315002921_0003
16/10/24 08:22:44 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
16/10/24 08:22:44 INFO impl.YarnClientImpl: Submitted application application_1477315002921_0003
16/10/24 08:22:44 INFO mapreduce.Job: The url to track the job: http://ssmaster:8088/proxy/application_1477315002921_0003/
16/10/24 08:22:44 INFO mapreduce.Job: Running job: job_1477315002921_0003
16/10/24 08:22:57 INFO mapreduce.Job: Job job_1477315002921_0003 running in uber mode : false
16/10/24 08:22:57 INFO mapreduce.Job:  map 0% reduce 0%
16/10/24 08:23:04 INFO mapreduce.Job: Task Id : attempt_1477315002921_0003_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2154)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:742)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2060)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2152)
    ... 8 more

16/10/24 08:23:12 INFO mapreduce.Job: Task Id : attempt_1477315002921_0003_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2154)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:742)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2060)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2152)
    ... 8 more

16/10/24 08:23:21 INFO mapreduce.Job: Task Id : attempt_1477315002921_0003_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2154)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:742)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2060)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2152)

View Code

背景：Hadoop2.6.4 Ubuntu， hadoop jar xxx.jar  運行程序時報錯

緣由：在代碼中沒有指定執行類

解決：

添加 job.setJarByClass(WordCount2.class);

解決

 

2  Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://ssmaster:9000/data/paper.txt, expected: file:///

Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://ssmaster:9000/data/paper.txt, expected: file:///
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
    at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80)
    at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:532)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:750)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:527)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:140)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
    at hadoop.hdfs.Hdfs_Download.main(Hdfs_Download.java:36)

View Code

背景：Hadoop2.6.4 Ubuntu, windows 7 eclipse 已經配置好win下運行hadoop插件環境，參考[0008] ，運行hdfs時報錯 (下載一個文件到windows本地)

       路徑設置以下 src = new Path("hdfs://ssmaster:9000/data/paper.txt" );

       out = new FileOutputStream ("c:\\paper.txt");

解決：

     將 core-site.xml，hdfs-site.xml，mapred-site.xml 複製到eclipse工程的src目錄下

解決

 

3  log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

View Code

背景： Hadoop2.6.4 Ubuntu, windows 7 eclipse 運行程序

解決： 工程下的src目錄新建一個文件 log4j.properties，添加內容（網上找)

解決

4 main"java.lang.UnsatisfiedLinkError:org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

 

背景：window 7 64，eclipse 執行 mapreduce wordcount 程序

緣由：win 下開發的問題

解決：

源碼包 hadoop-2.6.4-src.tar.gz解壓，hadoop-2.6.4-src\hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\io\nativeio下NativeIO.java 複製到對應的Eclipse的project

       修改以下地方

 

若是沒有解決，按照 Y.2.1 中配置

[0008]  http://www.cnblogs.com/sunzebo/p/5991563.html