cascading : how to define every map-reduce job in configuration? - configuration

Question

My code is below. This is cascading code.
And it has 8 jobs. I don't know how to configure every job.
Code below configure 8 jobs togerher.
But what i want to do is let last job one reduce.
I want to ask how to recoginse this 8 jobs, and how to configure them seperatly?
thanks.
private static void Demo(String[] args) {
/* Tap sourceTap = new Hfs(new TextLine(), "D:/test/file");
Tap finalResultTap = new Hfs(new TextLine(), "D:/test/result", true);
*/
Tap sourceTap = new Hfs(new TextLine(), args[0], SinkMode.KEEP);
Tap finalResultTap = new Hfs(new TextLine(), args[1], SinkMode.REPLACE);
Tap trap = new Hfs(new TextLine(), args[2], SinkMode.REPLACE);
Pipe sourcePipe = new Pipe("sourcePipe");
sourcePipe = getFilterPipe(sourcePipe);
Pipe vvResultPipe = new Pipe("vvResultPipe", sourcePipe);
vvResultPipe = getVVResultPipe(sourcePipe);
Pipe clickResultPipe = new Pipe("clickResultPipe", sourcePipe);
clickResultPipe = getClickResultPipe(clickResultPipe);
Pipe stClickResultPipe = new Pipe("stClickResultPipe", sourcePipe);
stClickResultPipe = getStClickResultPipe(sourcePipe);
//连接3个pipe的结果
Pipe resultPipe = new Pipe("resultPipe");
resultPipe = new CoGroup(vvResultPipe, new Fields("vid"), clickResultPipe, new Fields("referVid"),
new Fields("vid", "totalVV", "referVid", "totalClick"), new LeftJoin());
resultPipe = new CoGroup(resultPipe, new Fields("vid"), stClickResultPipe, new Fields("referVid"),
new Fields("vid", "totalVV", "referVid", "totalClick", "referVid2", "st1","st2","st3", "st4","st6", "st8"), new LeftJoin());
resultPipe = new Each(resultPipe, new Fields("vid", "totalVV", "totalClick", "st1","st2","st3", "st4","st6", "st8"),
new Identity(Fields.ARGS));
Fields sortClickFields = new Fields("totalVV");
resultPipe = new GroupBy(resultPipe, Fields.NONE, sortClickFields );
sortClickFields.setComparators(Collections.reverseOrder());
/* Limit limit = new Limit(200);
resultPipe = new Each(resultPipe, limit);
*/
JobConf conf = new JobConf();
conf.setJarByClass(Main.class);
//Properties properties = new Properties();
Properties properties = AppProps.appProps().buildProperties(conf);
properties.setProperty("user.group", "d_sdo_data");
properties.setProperty("mapred.job.queue.name", "cug_d_sdo_data");
properties.setProperty("mapred.fairscheduler.pool", "cug_d_sdo_data");
properties.setProperty("cascading.tmp.dir", "/home/hdfs/cluster-data/tmp/mapred/staging/recommend_user/tmp");
properties.setProperty("mapreduce.job.complete.cancel.delegation.tokens", "false");
properties.setProperty("mapred.reduce.tasks", "30");
properties.setProperty("mapred.map.tasks", "200");
//AppProps.setApplicationJarClass(properties, Main.class);
FlowConnector flowConnector = new HadoopFlowConnector(properties);
FlowDef flowDef = FlowDef.flowDef()
.setName( "tfidf" )
.addSource( sourcePipe, sourceTap )
.addTailSink( resultPipe, finalResultTap )
.addTrap( "assertions", trap );
Flow flow = flowConnector.connect(flowDef);
flow.complete();
}

Answer 1

This question is three years old, but I ran across it while looking for this solution. Here's what I finally ended up with:
pipe.getStepConfigDef().setProperty("mapreduce.job.reduces", "1");
This goes after you've defined the step you want to configure.
Note that this is for Hadoop 2.6.4 - if you're using mapred.reduce.tasks property instead. See here:
https://hadoop.apache.org/docs/r2.6.4/hadoop-project-dist/hadoop-common/DeprecatedProperties.html