刚刚开始学习hadoop，搭好环境跑了一下wordcount程序。出现一个问题。我的输入数据有5.8G，采用mapred.map.tasks=2,mapred.reduce.tasks=2时，没有问题。当mapred.map.task2=2,maprd.reduce.tasks=1时，在reduce阶段出现问题，reduce一直进行到89%，就进行不下去，然后retry，最后任务失败。
tasknode的错误如下。
java.io.IOException:subprocessstillrunning
R/W/S=340694604/0/0in:439605=340694604/775[rec/s]out:0=0/775[rec/s]
minRecWrittenToEnableSkip_=9223372036854775807LOGNAME=null
HOST=null
USER=search
HADOOP_USER=null
lastHadoopinput:|null|
lasttooloutput:|null|
Date:ThuFeb0216:08:02CST2012
Brokenpipe
atorg.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:131)
atorg.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:518)
atorg.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:419)
atorg.apache.hadoop.mapred.Child$4.run(Child.java:259)
atjava.security.AccessController.doPrivileged(NativeMethod)
atjavax.security.auth.Subject.doAs(Subject.java:416)
atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
atorg.apache.hadoop.mapred.Child.main(Child.java:253)
stderrlogs
Traceback(mostrecentcalllast):
File"/taskTracker/search/jobcache/job_201202021401_0005/attempt_201202021401_0005_r_000000_0/work/./reduce.py",line16,in?
wc[word]=count
MemoryError
map的源码如下：
#!/usr/bin/python
importsys
forlineinsys.stdin:
line=line.strip()
words=line.split()
forwordinwords:
print'%s\t%s'%(word,1)
reduce的源码如下
#!/usr/bin/python
importsys
wc={}
forlineinsys.stdin:
line=line.strip()
word,count=line.split('\t',1)
try:
count=int(count)
exceptError:
pass
ifwc.has_key(word):
wc[word+=count
else:
wc[word]=count
forkeyinwc.keys():
print'%s\t%s'%(key,wc[key])
分 --> |
