MultipleOutputs Zero Reducer

My input folder has 200 files. I want MultipleOutputs to write my parsed inputs from each file(identified using "map.input.file") into an output file with the same name.Since, I don't have any aggregation to perform and so using 0 reducer option (conf.setNumReduceTasks(0)). Ideally, I should get 200 output files. But, my output has around 5000+ files - each file containing only one line (of streaming output).Clearly, it is not aggregating.My assumption is, Ideally, in zero reducer - mapper output should be aggregated . Help is appreciated. Thanks! public static void main(String[] args) throws IOException { if (args.length != 2) { System.err.println("Usage: MaxTemperature "); System.exit(-1); } JobConf conf = new JobConf(MultipleOutputEx.class); conf.setJobName("Duration Count"); FileInputFormat.addInputPath(conf, new Path(args[0])); FileOutputFormat.setOutputPath(conf, new Path(args[1])); conf.setNumReduceTasks(0); conf.setMapperClass(MultipleOutputExMapper.class); conf.setReducerClass(MultipleOutputExReducer.class); conf.setMapOutputKeyClass(NullWritable.class); MultipleOutputs.addMultiNamedOutput(conf,"mofiles", TextOutputFormat.class, NullWritable.class, Text.class); JobClient.runJob(conf); } And my Mapper class is, public class MultipleOutputExMapper extends MapReduceBase implements Mapper { MultipleOutputs mos = null; Text fileKey = new Text(); String line = ""; private JobConf conf; @Override public void configure(JobConf conf) { this.conf = conf; mos = new MultipleOutputs(conf); } public void map(LongWritable key, Text value, OutputCollector output, Reporter reporter) throws IOException { try { String filename = conf.get("map.input.file"); fileKey.set(filename); OutputCollector collector = mos.getCollector( "mofiles", key.toString(), reporter); collector.collect(NullWritable.get(), value); } catch (ArrayIndexOutOfBoundsException E) { E.printStackTrace(); } catch (Exception E) { System.out.println(line); E.printStackTrace(); } } @Override public void close() throws IOException { mos.close(); }
How many distinct keys do you have? It looks like you're creating a new filename for each key.

以上就是MultipleOutputs Zero Reducer的详细内容,更多请关注web前端其它相关文章!

赞(0) 打赏
未经允许不得转载:web前端首页 » JavaScript 答疑

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址

前端开发相关广告投放 更专业 更精准