签到成功

知道了

CNDBA社区CNDBA社区

Hadoop 3.1.1 Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster 解决方法

2019-01-23 16:02 8690 0 原创 Hadoop
作者： dave
本文链接：http://www.cndba.cn/cndba/dave/article/3259

在Hadoop 3.1.1 平台执行wordcount 程序时，报如下错误：http://www.cndba.cn/cndba/dave/article/3259http://www.cndba.cn/cndba/dave/article/3259

http://www.cndba.cn/cndba/dave/article/3259
http://www.cndba.cn/cndba/dave/article/3259

[http://www.cndba.cn@hadoopmaster mapreduce]$ pwd
/home/cndba/hadoop/share/hadoop/mapreduce
[http://www.cndba.cn@hadoopmaster mapreduce]$ hadoop jar hadoop-mapreduce-examples-3.1.1.jar wordcount /dave/LICENSE.txt output
2019-01-23 23:25:48,336 INFO client.RMProxy: Connecting to ResourceManager at hadoopmaster/192.168.20.80:8032
2019-01-23 23:25:48,782 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/cndba/.staging/job_1548242934753_0001
2019-01-23 23:25:49,106 INFO input.FileInputFormat: Total input files to process : 1
2019-01-23 23:25:49,226 INFO mapreduce.JobSubmitter: number of splits:1
2019-01-23 23:25:49,298 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2019-01-23 23:25:49,467 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1548242934753_0001
2019-01-23 23:25:49,470 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-01-23 23:25:49,687 INFO conf.Configuration: resource-types.xml not found
2019-01-23 23:25:49,688 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2019-01-23 23:25:50,210 INFO impl.YarnClientImpl: Submitted application application_1548242934753_0001
2019-01-23 23:25:50,292 INFO mapreduce.Job: The url to track the job: http://hadoopmaster:8088/proxy/application_1548242934753_0001/
2019-01-23 23:25:50,293 INFO mapreduce.Job: Running job: job_1548242934753_0001
2019-01-23 23:25:56,360 INFO mapreduce.Job: Job job_1548242934753_0001 running in uber mode : false
2019-01-23 23:25:56,361 INFO mapreduce.Job:  map 0% reduce 0%
2019-01-23 23:25:56,382 INFO mapreduce.Job: Job job_1548242934753_0001 failed with state FAILED due to: Application application_1548242934753_0001 failed 2 times due to AM Container for appattempt_1548242934753_0001_000002 exited with  exitCode: 1
Failing this attempt.Diagnostics: [2019-01-23 23:25:55.635]Exception from container-launch.
Container id: container_1548242934753_0001_02_000001
Exit code: 1

[2019-01-23 23:25:55.684]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>

[2019-01-23 23:25:55.685]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>

For more detailed output, check the application tracking page: http://hadoopmaster:8088/cluster/app/application_1548242934753_0001 Then click on links to logs of each attempt.
. Failing the application.
2019-01-23 23:25:56,417 INFO mapreduce.Job: Counters: 0
[http://www.cndba.cn@hadoopmaster mapreduce]$

虽然错误很多，但输出日志提示的非常明显。~etc/hadoop/mapred-site.xml 配置文件缺少如下内容：

<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>

修改后，重新执行，正常结束：http://www.cndba.cn/cndba/dave/article/3259http://www.cndba.cn/cndba/dave/article/3259

http://www.cndba.cn/cndba/dave/article/3259 

[http://www.cndba.cn@hadoopmaster hadoop]$ cat mapred-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
 <!-- Configurations for MapReduce Applications: -->
  <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
   </property>
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=/home/cndba/hadoop</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=/home/cndba/hadoop</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=/home/cndba/hadoop</value>
</property>

</configuration>

执行结果：http://www.cndba.cn/cndba/dave/article/3259

http://www.cndba.cn/cndba/dave/article/3259
http://www.cndba.cn/cndba/dave/article/3259

[http://www.cndba.cn@hadoopmaster mapreduce]$ hadoop jar hadoop-mapreduce-examples-3.1.1.jar wordcount /dave/LICENSE.txt output
2019-01-23 23:55:14,527 INFO client.RMProxy: Connecting to ResourceManager at hadoopmaster/192.168.20.80:8032
2019-01-23 23:55:14,944 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/cndba/.staging/job_1548242934753_0003
2019-01-23 23:55:15,344 INFO input.FileInputFormat: Total input files to process : 1
2019-01-23 23:55:15,461 INFO mapreduce.JobSubmitter: number of splits:1
2019-01-23 23:55:15,538 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2019-01-23 23:55:15,749 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1548242934753_0003
2019-01-23 23:55:15,751 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-01-23 23:55:15,967 INFO conf.Configuration: resource-types.xml not found
2019-01-23 23:55:15,967 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2019-01-23 23:55:16,050 INFO impl.YarnClientImpl: Submitted application application_1548242934753_0003
2019-01-23 23:55:16,106 INFO mapreduce.Job: The url to track the job: http://hadoopmaster:8088/proxy/application_1548242934753_0003/
2019-01-23 23:55:16,107 INFO mapreduce.Job: Running job: job_1548242934753_0003
2019-01-23 23:55:23,242 INFO mapreduce.Job: Job job_1548242934753_0003 running in uber mode : false
2019-01-23 23:55:23,244 INFO mapreduce.Job:  map 0% reduce 0%
2019-01-23 23:55:28,328 INFO mapreduce.Job:  map 100% reduce 0%
2019-01-23 23:55:34,369 INFO mapreduce.Job:  map 100% reduce 100%
2019-01-23 23:55:34,380 INFO mapreduce.Job: Job job_1548242934753_0003 completed successfully
2019-01-23 23:55:34,524 INFO mapreduce.Job: Counters: 53
    File System Counters
        FILE: Number of bytes read=46271
        FILE: Number of bytes written=521743
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=147250
        HDFS: Number of bytes written=34795
        HDFS: Number of read operations=8
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Launched reduce tasks=1
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=3346
        Total time spent by all reduces in occupied slots (ms)=3103
        Total time spent by all map tasks (ms)=3346
        Total time spent by all reduce tasks (ms)=3103
        Total vcore-milliseconds taken by all map tasks=3346
        Total vcore-milliseconds taken by all reduce tasks=3103
        Total megabyte-milliseconds taken by all map tasks=3426304
        Total megabyte-milliseconds taken by all reduce tasks=3177472
    Map-Reduce Framework
        Map input records=2746
        Map output records=21463
        Map output bytes=228869
        Map output materialized bytes=46271
        Input split bytes=106
        Combine input records=21463
        Combine output records=2965
        Reduce input groups=2965
        Reduce shuffle bytes=46271
        Reduce input records=2965
        Reduce output records=2965
        Spilled Records=5930
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=102
        CPU time spent (ms)=2300
        Physical memory (bytes) snapshot=518160384
        Virtual memory (bytes) snapshot=5637390336
        Total committed heap usage (bytes)=431489024
        Peak Map Physical memory (bytes)=314851328
        Peak Map Virtual memory (bytes)=2815950848
        Peak Reduce Physical memory (bytes)=203309056
        Peak Reduce Virtual memory (bytes)=2821439488
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=147144
    File Output Format Counters 
        Bytes Written=34795
[http://www.cndba.cn@hadoopmaster mapreduce]$

版权声明：本文为博主原创文章，未经博主允许不得转载。

用户评论
* 以下用户言论只代表其个人观点，不代表CNDBA社区的观点或立场
dave

dave

关注

人的一生应该是这样度过的：当他回首往事的时候，他不会因为虚度年华而悔恨，也不会因为碌碌无为而羞耻；这样，在临死的时候，他就能够说：“我的整个生命和全部精力，都已经献给世界上最壮丽的事业....."

  • 2262
    原创
  • 3
    翻译
  • 578
    转载
  • 192
    评论
  • 访问：8067956次
  • 积分：4349
  • 等级：核心会员
  • 排名：第1名
精华文章

    达梦数据库运维实战

    Oracle 18c 必须掌握的新特性：管理与实战

    Oracle数据库问题解决方案和故障排除手册
    最新问题
    查看更多+
    热门文章
      热门用户
      推荐用户
        友情链接： CNDBA社区 | openGauss | openEuler | openAnolis | 广告联系: database@ustc.edu
        Copyright © 2016 All Rights Reserved. Powered by CNDBA · 皖ICP备2022006297号-1·

        QQ交流群

        注册联系QQ