spark-submit ?

问答 maxy ⋅ 于 2018-02-02 14:55:11 ⋅ 最后回复由 Xibaibai 2018-02-07 15:35:56 ⋅ 4594 阅读

同样的脚本, 同一个环境, 提交 spark任务时:
--master yarn-cluster \ 这个模式提交失败, 报错
Diagnostics: Exception from container-launch.
Container id: container_1517482621865_0004_02_000001
Exit code: 15
--master yarn-client \ 这个模式是正常的, 这是啥原因?? 请大牛指点

成为第一个点赞的人吧 :bowtie:
回复数量: 7
  • 青牛 国内首批大数据从业者,就职于金山,担任大数据团队核心研发工程师
    2018-02-02 15:18:10

    有没有更详细的错误日志?

  • maxy
    2018-02-02 15:21:43

    18/02/02 15:20:25 INFO yarn.Client: Application report for application_1517482621865_0005 (state: ACCEPTED)
    18/02/02 15:20:26 INFO yarn.Client: Application report for application_1517482621865_0005 (state: ACCEPTED)
    18/02/02 15:20:27 INFO yarn.Client: Application report for application_1517482621865_0005 (state: FAILED)
    18/02/02 15:20:27 INFO yarn.Client:
    client token: N/A
    diagnostics: Application application_1517482621865_0005 failed 2 times due to AM Container for appattempt_1517482621865_0005_000002 exited with exitCode: 15
    For more detailed output, check application tracking page:http://master:8088/proxy/application_1517482621865_0005/Then, click on links to logs of each attempt.
    Diagnostics: Exception from container-launch.
    Container id: container_1517482621865_0005_02_000001
    Exit code: 15
    Stack trace: ExitCodeException exitCode=15:
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
    at org.apache.hadoop.util.Shell.run(Shell.java:455)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

    Container exited with a non-zero exit code 15
    Failing this attempt. Failing the application.
    ApplicationMaster host: N/A
    ApplicationMaster RPC port: -1
    queue: default
    start time: 1517555966590
    final status: FAILED
    tracking URL: http://master:8088/cluster/app/application_1517482621865_0005
    user: root
    Exception in thread "main" org.apache.spark.SparkException: Application application_1517482621865_0005 finished with failed status
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1029)
    at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1076)
    at org.apache.spark.deploy.yarn.Client.main(Client.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)

  • 青牛 国内首批大数据从业者,就职于金山,担任大数据团队核心研发工程师
    2018-02-02 15:25:54

    @maxy hdfs上传spark的jar包了吗?

  • maxy
    2018-02-02 15:27:04

    有啊, client 模式是可以的, 我打的是 依赖包

  • 青牛 国内首批大数据从业者,就职于金山,担任大数据团队核心研发工程师
    2018-02-02 16:44:55

    @maxy 你这是appmaster没起来吧,我角得是少JAR包了

  • maxy
    2018-02-02 16:54:59

    搞了好长时间了, 郁闷了!

  • Xibaibai
    2018-02-07 15:35:56

    得去hdfs看详细的任务日志吧?我感觉你这个像nohup.out里的日志.........QAQ

暂无评论~~
  • 请注意单词拼写,以及中英文排版,参考此页
  • 支持 Markdown 格式, **粗体**、~~删除线~~、`单行代码`, 更多语法请见这里 Markdown 语法
  • 支持表情,可用Emoji的自动补全, 在输入的时候只需要 ":" 就可以自动提示了 :metal: :point_right: 表情列表 :star: :sparkles:
  • 上传图片, 支持拖拽和剪切板黏贴上传, 格式限制 - jpg, png, gif,教程
  • 发布框支持本地存储功能,会在内容变更时保存,「提交」按钮点击时清空
Ctrl+Enter