足迹

第 197 位会员
注册于 2017-02-12 17:53:20
活跃于 2018-05-28 16:01:31


  • Hadoop 部署集群时节点无法启动问题? at 2018-01-19 17:30:07

    @青牛 环境变量应该没问题。挺奇怪的。

    file
    file

  • Hadoop 部署集群时节点无法启动问题? at 2018-01-19 16:08:24

    @青牛
    参考青牛哥的配置改了一下

    file

    不过还是没有启动,部分dubug信息,markdown语法目前没有高亮显示:

    [hadoop@master hadoop]$ sh -x start-dfs.sh
    + this=/usr/local/hadoop/sbin/start-dfs.sh
    +++ dirname -- /usr/local/hadoop/sbin/start-dfs.sh
    ++ cd -P -- /usr/local/hadoop/sbin
    ++ pwd -P
    + bin=/usr/local/hadoop-3.0.0/sbin
    + [[ -n /usr/local/hadoop ]]
    + HADOOP_DEFAULT_LIBEXEC_DIR=/usr/local/hadoop/libexec
    + HADOOP_LIBEXEC_DIR=/usr/local/hadoop/libexec
    + HADOOP_NEW_CONFIG=true
    + [[ -f /usr/local/hadoop/libexec/hdfs-config.sh ]]
    + . /usr/local/hadoop/libexec/hdfs-config.sh
    ++ [[ -z /usr/local/hadoop/libexec ]]
    ++ [[ -n '' ]]
    ++ [[ -e /usr/local/hadoop/libexec/hadoop-config.sh ]]
    ++ . /usr/local/hadoop/libexec/hadoop-config.sh
    +++ [[ -z 4 ]]
    +++ [[ 4 -lt 3 ]]
    +++ [[ 4 -eq 3 ]]
    +++ [[ -z /usr/local/hadoop/libexec ]]
    +++ [[ -n '' ]]
    +++ [[ -e /usr/local/hadoop/libexec/hadoop-functions.sh ]]
    +++ . /usr/local/hadoop/libexec/hadoop-functions.sh
    ++++ declare -a HADOOP_SUBCMD_USAGE
    ++++ declare -a HADOOP_OPTION_USAGE
    ++++ declare -a HADOOP_SUBCMD_USAGE_TYPES
    /usr/local/hadoop/libexec/hadoop-functions.sh:行398: 未预期的符号 `<' 附近有语法错误
    /usr/local/hadoop/libexec/hadoop-functions.sh:行398: `  done < <(for text in "${input[@]}"; do'
    +++ hadoop_deprecate_envvar HADOOP_PREFIX HADOOP_HOME
    /usr/local/hadoop/libexec/hadoop-config.sh:行70: hadoop_deprecate_envvar: 未找到命令
    +++ [[ -n '' ]]
    +++ [[ -e /usr/local/hadoop/libexec/hadoop-layout.sh ]]
    +++ hadoop_bootstrap
    /usr/local/hadoop/libexec/hadoop-config.sh:行87: hadoop_bootstrap: 未找到命令
    +++ HADOOP_USER_PARAMS=("$@")
    +++ hadoop_parse_args
    /usr/local/hadoop/libexec/hadoop-config.sh:行104: hadoop_parse_args: 未找到命令
    +++ shift ''
    /usr/local/hadoop/libexec/hadoop-config.sh: 第 105 行:shift: : 需要数字参数
    +++ hadoop_find_confdir
    /usr/local/hadoop/libexec/hadoop-config.sh:行110: hadoop_find_confdir: 未找到命令
    +++ hadoop_exec_hadoopenv
    /usr/local/hadoop/libexec/hadoop-config.sh:行111: hadoop_exec_hadoopenv: 未找到命令
    +++ hadoop_import_shellprofiles
    /usr/local/hadoop/libexec/hadoop-config.sh:行112: hadoop_import_shellprofiles: 未找到命令
    +++ hadoop_exec_userfuncs
    /usr/local/hadoop/libexec/hadoop-config.sh:行113: hadoop_exec_userfuncs: 未找到命令
    +++ hadoop_exec_user_hadoopenv
    /usr/local/hadoop/libexec/hadoop-config.sh:行119: hadoop_exec_user_hadoopenv: 未找到命令
    +++ hadoop_verify_confdir
    /usr/local/hadoop/libexec/hadoop-config.sh:行120: hadoop_verify_confdir: 未找到命令
    +++ hadoop_deprecate_envvar HADOOP_SLAVES HADOOP_WORKERS
    /usr/local/hadoop/libexec/hadoop-config.sh:行122: hadoop_deprecate_envvar: 未找到命令
    +++ hadoop_deprecate_envvar HADOOP_SLAVE_NAMES HADOOP_WORKER_NAMES
    /usr/local/hadoop/libexec/hadoop-config.sh:行123: hadoop_deprecate_envvar: 未找到命令
    +++ hadoop_deprecate_envvar HADOOP_SLAVE_SLEEP HADOOP_WORKER_SLEEP
    /usr/local/hadoop/libexec/hadoop-config.sh:行124: hadoop_deprecate_envvar: 未找到命令
    +++ hadoop_os_tricks
    /usr/local/hadoop/libexec/hadoop-config.sh:行129: hadoop_os_tricks: 未找到命令
    +++ hadoop_java_setup
    /usr/local/hadoop/libexec/hadoop-config.sh:行131: hadoop_java_setup: 未找到命令
    +++ hadoop_basic_init
    /usr/local/hadoop/libexec/hadoop-config.sh:行133: hadoop_basic_init: 未找到命令
    +++ declare -F hadoop_subproject_init
    +++ hadoop_subproject_init
    +++ [[ -z '' ]]
    +++ [[ -e /hdfs-env.sh ]]
    +++ hadoop_deprecate_envvar HADOOP_HDFS_LOG_DIR HADOOP_LOG_DIR
    /usr/local/hadoop/libexec/hdfs-config.sh:行38: hadoop_deprecate_envvar: 未找到命令
    +++ hadoop_deprecate_envvar HADOOP_HDFS_LOGFILE HADOOP_LOGFILE
    /usr/local/hadoop/libexec/hdfs-config.sh:行40: hadoop_deprecate_envvar: 未找到命令
    +++ hadoop_deprecate_envvar HADOOP_HDFS_NICENESS HADOOP_NICENESS
    /usr/local/hadoop/libexec/hdfs-config.sh:行42: hadoop_deprecate_envvar: 未找到命令
    +++ hadoop_deprecate_envvar HADOOP_HDFS_STOP_TIMEOUT HADOOP_STOP_TIMEOUT
    /usr/local/hadoop/libexec/hdfs-config.sh:行44: hadoop_deprecate_envvar: 未找到命令
    +++ hadoop_deprecate_envvar HADOOP_HDFS_PID_DIR HADOOP_PID_DIR
    /usr/local/hadoop/libexec/hdfs-config.sh:行46: hadoop_deprecate_envvar: 未找到命令
    +++ hadoop_deprecate_envvar HADOOP_HDFS_ROOT_LOGGER HADOOP_ROOT_LOGGER
    /usr/local/hadoop/libexec/hdfs-config.sh:行48: hadoop_deprecate_envvar: 未找到命令
    +++ hadoop_deprecate_envvar HADOOP_HDFS_IDENT_STRING HADOOP_IDENT_STRING
    /usr/local/hadoop/libexec/hdfs-config.sh:行50: hadoop_deprecate_envvar: 未找到命令
    +++ hadoop_deprecate_envvar HADOOP_DN_SECURE_EXTRA_OPTS HDFS_DATANODE_SECURE_EXTRA_OPTS
    /usr/local/hadoop/libexec/hdfs-config.sh:行52: hadoop_deprecate_envvar: 未找到命令
    +++ hadoop_deprecate_envvar HADOOP_NFS3_SECURE_EXTRA_OPTS HDFS_NFS3_SECURE_EXTRA_OPTS
    /usr/local/hadoop/libexec/hdfs-config.sh:行54: hadoop_deprecate_envvar: 未找到命令
    +++ hadoop_deprecate_envvar HADOOP_SECURE_DN_USER HDFS_DATANODE_SECURE_USER
    /usr/local/hadoop/libexec/hdfs-config.sh:行56: hadoop_deprecate_envvar: 未找到命令
    +++ hadoop_deprecate_envvar HADOOP_PRIVILEGED_NFS_USER HDFS_NFS3_SECURE_USER
    /usr/local/hadoop/libexec/hdfs-config.sh:行58: hadoop_deprecate_envvar: 未找到命令
    +++ HADOOP_HDFS_HOME=/usr/local/hadoop
    +++ export HDFS_AUDIT_LOGGER=INFO,NullAppender
    +++ HDFS_AUDIT_LOGGER=INFO,NullAppender
    +++ export HDFS_NAMENODE_OPTS=-Dhadoop.security.logger=INFO,RFAS
    +++ HDFS_NAMENODE_OPTS=-Dhadoop.security.logger=INFO,RFAS
    +++ export HDFS_SECONDARYNAMENODE_OPTS=-Dhadoop.security.logger=INFO,RFAS
    +++ HDFS_SECONDARYNAMENODE_OPTS=-Dhadoop.security.logger=INFO,RFAS
    +++ export HDFS_DATANODE_OPTS=-Dhadoop.security.logger=ERROR,RFAS
    +++ HDFS_DATANODE_OPTS=-Dhadoop.security.logger=ERROR,RFAS
    +++ export HDFS_PORTMAP_OPTS=-Xmx512m
    +++ HDFS_PORTMAP_OPTS=-Xmx512m
    +++ export 'HDFS_DATANODE_SECURE_EXTRA_OPTS=-jvm server'
    +++ HDFS_DATANODE_SECURE_EXTRA_OPTS='-jvm server'
    +++ export 'HDFS_NFS3_SECURE_EXTRA_OPTS=-jvm server'
    +++ HDFS_NFS3_SECURE_EXTRA_OPTS='-jvm server'
    +++ hadoop_shellprofiles_init
    /usr/local/hadoop/libexec/hadoop-config.sh:行140: hadoop_shellprofiles_init: 未找到命令
    +++ hadoop_add_javalibpath /usr/local/hadoop/build/native
    /usr/local/hadoop/libexec/hadoop-config.sh:行143: hadoop_add_javalibpath: 未找到命令
    +++ hadoop_add_javalibpath /usr/local/hadoop/
    /usr/local/hadoop/libexec/hadoop-config.sh:行144: hadoop_add_javalibpath: 未找到命令
    +++ hadoop_shellprofiles_nativelib
    /usr/local/hadoop/libexec/hadoop-config.sh:行146: hadoop_shellprofiles_nativelib: 未找到命令
    +++ hadoop_add_common_to_classpath
    /usr/local/hadoop/libexec/hadoop-config.sh:行152: hadoop_add_common_to_classpath: 未找到命令
    +++ hadoop_shellprofiles_classpath
    /usr/local/hadoop/libexec/hadoop-config.sh:行153: hadoop_shellprofiles_classpath: 未找到命令
    +++ hadoop_exec_hadooprc
    /usr/local/hadoop/libexec/hadoop-config.sh:行157: hadoop_exec_hadooprc: 未找到命令
    +++ [[ -z true ]]
    + [[ 0 -ge 1 ]]
    + nameStartOpt=' '
    ++ /usr/local/hadoop/bin/hdfs getconf -namenodes
    + NAMENODES=master.hadoop
    + [[ -z master.hadoop ]]
    + echo 'Starting namenodes on [master.hadoop]'
    Starting namenodes on [master.hadoop]
    + hadoop_uservar_su hdfs namenode /usr/local/hadoop/bin/hdfs --workers --config '' --hostnames master.hadoop --daemon start namenode
    + declare program=hdfs
    + declare command=namenode
    + shift 2
    + declare uprogram
    + declare ucommand
    + declare uvar
    + declare svar
    + hadoop_privilege_check
    + [[ 1000 = 0 ]]
    + /usr/local/hadoop/bin/hdfs --workers --config '' --hostnames master.hadoop --daemon start namenode
    ERROR: No parameter provided for --config
    Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
  • Hadoop 部署集群时节点无法启动问题? at 2018-01-18 17:50:46

    是不是哪里配置错了

  • Hadoop 部署集群时节点无法启动问题? at 2018-01-18 17:48:39

    @青牛
    嗯,ssh是好的。

    file

  • Hadoop 50070 端口没有监听? at 2018-01-17 18:12:34

    问题已解决。
    DateNode报错是因为格式化了集群2次导致clusterID不一致。
    修改datanode的version的id与namenode中一致后重启即可。
    页面无法访问是由于Linux防火墙未关闭。感谢青牛大哥。

    file

  • Hadoop 50070 端口没有监听? at 2018-01-15 23:01:32

    @青牛 青牛大哥,详细日志已上传

  • 海牛部落 Java 基础教程 (15) 多线程 at 2017-11-27 15:02:51

    学习

  • 信息被删除或无权限查看
  • 万能的胶水语言 python at 2017-11-14 21:10:48

    学习

  • 海牛部落 Hadoop 系列教程(九):hadoop 的 Windows 伪分布式环境部署 at 2017-07-31 09:09:59

    谢谢分享

  • 海牛部落 Hadoop 系列教程(十):Windows 开发 wordcount at 2017-07-04 23:20:07

    看看windows

  • 海牛部落 Hadoop 系列教程(六):HDFS 原理以及常用命令 at 2017-07-04 23:17:46

    笔记非常好

  • 海牛部落 Hadoop 系列教程(五):高可用(HA)HDFS 安装 at 2017-07-04 23:17:26

    资料不错

  • 海牛部落 Hadoop 系列教程(四):编译 hadoop 源码 at 2017-07-04 23:15:03

    文章写的很详细

  • 海牛部落 Hadoop 系列教程(三):zookeeper at 2017-07-04 23:14:20

    学习一下看看