V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
V2EX 提问指南
crouchred
V2EX  ›  问与答

hadoop2 中找不到 println 或者 log 的内容?

  •  
  •   crouchred · 2016-05-21 15:10:57 +08:00 · 2024 次点击
    这是一个创建于 3132 天前的主题,其中的信息可能已经有所发展或是发生改变。

    hadoop2.6.3 , intellij ,没有用 maven 。

    写最简单的 wordcount ,想用 System.out.println 或者 log.info 输出点东西,但是 master:19888/jobhistory 里面的 log 找不到。

    代码很简单的:

    import java.io.IOException;
    import java.util.StringTokenizer;
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.Mapper;
    import org.apache.hadoop.mapreduce.Reducer;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    import org.apache.hadoop.util.GenericOptionsParser;
    
    import org.apache.commons.logging.Log;
    import org.apache.commons.logging.LogFactory;
    
    public class WordCount {
    
    
        public static class TokenizerMapper
                extends Mapper<Object, Text, Text, IntWritable>{
    
            private final static IntWritable one = new IntWritable(1);
            private Text word = new Text();
            private static final Log LOG = LogFactory.getLog(TokenizerMapper.class);
    
            public void map(Object key, Text value, Context context
            ) throws IOException, InterruptedException {
                LOG.info("this is log.info");
                LOG.error("this is log.error");
                System.out.println("this is system.out.println");
                StringTokenizer itr = new StringTokenizer(value.toString());
                while (itr.hasMoreTokens()) {
                    word.set(itr.nextToken());
                    context.write(word, one);
                }
            }
        }
    
    
        public static class IntSumReducer
                extends Reducer<Text,IntWritable,Text,IntWritable> {
            private IntWritable result = new IntWritable();
    
            public void reduce(Text key, Iterable<IntWritable> values,
                               Context context
            ) throws IOException, InterruptedException {
                int sum = 0;
                for (IntWritable val : values) {
                    sum += val.get();
                }
                result.set(sum);
                context.write(key, result);
            }
        }
    
    
        public static void main(String[] args) throws Exception {
            Configuration conf = new Configuration();
            String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
            if (otherArgs.length != 2) {
                System.err.println("Usage: wordcount <in> <out>");
                System.exit(2);
            }
            Job job = Job.getInstance(conf, "word count");
            job.setJarByClass(WordCount.class);
            job.setMapperClass(TokenizerMapper.class);
            job.setCombinerClass(IntSumReducer.class);
    
            job.setReducerClass(IntSumReducer.class);
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(IntWritable.class);
            FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
            FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
            System.exit(job.waitForCompletion(true) ? 0 : 1);
        }
    }
    

    log 是这样的:

    Log Type: stderr
    Log Upload Time: 20-05-2016 17:12:41
    Log Length: 545
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/local/hadoop/tmp/nm-local-dir/usercache/chenxuyuan/appcache/application_1463402812846_0004/filecache/10/job.jar/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
    
    Log Type: stdout
    Log Upload Time: 20-05-2016 17:12:41
    Log Length: 0
    
    Log Type: syslog
    Log Upload Time: 20-05-2016 17:12:41
    Log Length: 3255
    2016-05-16 22:21:48,909 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
    2016-05-16 22:21:49,159 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
    .......(omit)
    

    在 stackoverflow 上搜了一圈,有人说 local/standalone 模式下肯定是有输出的,但是即使是 local mode 也啥都找不到..

    目前尚无回复
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   3460 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 33ms · UTC 05:01 · PVG 13:01 · LAX 21:01 · JFK 00:01
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.