Have something wrong with 'Run mapreduce on a Hadoop Cluster'

조회 수: 1 (최근 30일)
Jingyu Ru
Jingyu Ru 2015년 8월 16일
댓글: Rick Amos 2015년 8월 17일
Hadoop version 1.2.1 Matlab version 2015a
Linux ubuntu 14.
I install Hadoop with a small cluster(one master and one slave).
It is success to run the example 'wordcount'in Hadoop.
And it is success to read the data from the HDFS through the Matlab.
But when I try to run the example in Matlab 'Run mapreduce on a Hadoop Cluster',I failed.
It shows that,
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
run_mapreduce_on_a_hadoop
ans =
ArrDelay
________
8
8
21
13
4
59
3
11
Parallel mapreduce execution on the Hadoop cluster:
******************************
  • MAPREDUCE PROGRESS *
******************************
Map 0% Reduce 0%
Map 100% Reduce 33%
Map 100% Reduce 71%
Map 100% Reduce 100%
Error using mapreduce (line 100)
All the input files must be Sequence files.
Invalid file: ''
Error in run_mapreduce_on_a_hadoop (line 24)
meanDelay = mapreduce(ds,@meanArrivalDelayMapper,@meanArrivalDelayReducer,mr,...
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
There is my Matlab codes
setenv('HADOOP_HOME','/usr/local/hadoop');
cluster = parallel.cluster.Hadoop;
cluster.HadoopProperties('mapred.job.tracker') = 'ubuntu:50031';
cluster.HadoopProperties('fs.default.name') = 'hdfs://ubuntu:8020';
outputFolder = '/home/rjy/logs/hadooplog';
mr = mapreducer(cluster);
ds = datastore('airlinesmall.csv','TreatAsMissing','NA','SelectedVariableNames','ArrDelay','ReadSize',1000);
preview(ds)
meanDelay = mapreduce(ds,@meanArrivalDelayMapper,@meanArrivalDelayReducer,mr,...
'OutputFolder',outputFolder);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
What's ' All the input files must be Sequence files. Invalid file: '' ' mean?
I have never seen it before. I just copy the code in the Matlab document.
I wish to know how to solve this problem. I have tried many methods to solve it.
Please give me some suggestions. Thanks.

답변 (0개)

카테고리

Help CenterFile Exchange에서 Tall Arrays and mapreduce에 대해 자세히 알아보기

제품

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by