Matlab + Hadoop Integration ??

We Followed standard doc for integrating Hadoop-2.7.2 with Matlab-R2016b. * Link for document* : http://in.mathworks.com/help/mdce/configure-a-hadoop-cluster.html

We Completed up to this stage but we are confuse with the rest part....

The requirements are:

1. MATLAB® Distributed Computing Server™ must be installed or available on the cluster nodes. See Install Products and Choose Cluster Configuration.
2. If the cluster is running in Kerberos authentication that requires the Java Cryptography Extension, you must download and install the Oracle version of this extension to each MATLAB Distributed Computing Server installation. You must also perform this step for the MATLAB client installation. To install the extension, place the Java Cryptography Extension jar files into the folder ${MATLABROOT}/sys/jre/${ARCH}/jre/lib/security.
3. You must have a Hadoop installation on the MATLAB client machine, that can submit normal (non-MATLAB) jobs to the cluster.
4. The cluster must identify its user home directory as a valid location that the nodes can access. You must choose a local filesystem path and typically use a local folder such as /tmp/hduserhome or /home/${USER}. Set yarn.nodemanager.user-home-dir for Hadoop version 2.X.
5. There is one Hadoop property that must not be "final." (If properties are "final", they are locked to a fixed predefined value, and jobs cannot alter them.)
6. The software needs to append a value to this property so that task processes are able to correctly run MATLAB. This property is passed as part of the job metadata given to Hadoop during job submission.

This property is mapred.child.env, which controls environment variables for the job's task processes.

You must provide necessary information to the parallel.cluster.Hadoop object in the MATLAB client session. For example, see Run mapreduce on a Hadoop Cluster (Parallel Computing Toolbox) and Use Tall Arrays on a Spark Enabled Hadoop Cluster (Parallel Computing Toolbox).

*MAIN QUESTION IS .....

1. We are not able to see any Cluster Configuration in Home->Parallel->Manage_Cluster_Profile ???

2.what is Role of MJS in this and how to configure it.

3.we are not understanding what we have to export and import for rest of the worker (slave )and master node* ??

Thanks

[Walter Roberson]

Which MATLAB release are you using?

For me, in R2017a, look in the MATLAB toolbar, in the column that has "Preferences" at the top and "Set Path" below that. The column is between the Layout and the Add-Ons columns. The third item in the column is Parallel; click on that and it is a menu that includes Manage Cluster Profiles.

[Pulkesh Haran]

I am using R2016b Matlab and Hadoop-2.7.2. We did not finding any info there (on Path you Mention)..

Answer 1

[Kojiro Saito]

1. When integrating with Hadoop, MATLAB does not use a cluster profile. So, it's not an issue that Hadoop cluster profile is not listed in "Manage Cluster Profiles".
2. When integrating with Hadoop, MJS is not used. MATLAB uses Hadoop's job scheduler, so you don't need to configure in MATLAB side.
3. For the rest of workers and nodes, I don't think you need to export and import.

If you have any question, please ask us.

[Pulkesh Haran]

Then we integrated Correctly Matlab and Hadoop. we can see Running program on Hadoop Interface.. But I have following questions ...

1. Matlab+Hadoop cluster (with 50 Nodes) taking more time for running map-reduce Job compare to single machine Matlab Map-Reduce ?

2. What is role of MDCS ??

3. How to define or configure property of MDCS ??

4. In Matlab GUI how can we see Matlab+Hadoop Cluster or Program running on Cluster with detailed info.

5. Matlab+hadoop CLuster with 50 nodes taking more time compare to normal matlab map-reduce running on single machine. How to can we resolve that problem.

6.We are getting Errors like :

      while (Converting 50K images into Sequence File)

    i. Unable to read MAT file ... : not a binary File .

    ii. we are getting serialization error.

how to resolve them ??

Thanks

[Kojiro Saito]

1. It's seems strange.

2. MDCS does parallel execution of mapreduce with Hadoop.

3. In order to use MDCS with Hadoop, you need to set the following parameters.

In Hadoop 2.x settings, set "yarn.nodemanager.user-home-dir" ($HADOOP_HOME/etc/hadoop/yarn-site.xml)

and in MATLAB, setenv 'HADOOP_HOME' and create a parallel.cluster.Hadoop and mapreducer.

This example will help you.

4. You can monitor the status using Hadoop Web UI( http://YOUR_HADOOP_HOST:8088/ ) or by the following command in terminal,

yarn application -status <application ID>

5.6 Could you give us more detail MATLAB scripts? or did you also ask to MathWorks technical support ?

[Pulkesh Haran]

This Problem we are Facing while Running our Matlab Script on Cluster for 50K images.

We are Creating Sequence File for 50 thousand images. * I am attaching Matlab Code and other details. Please help us doing same.*

------------------------------- Matlab Error1--------------------------------------------

Parallel mapreduce execution on the Hadoop cluster:

******************************

• MAPREDUCE PROGRESS *******************************

Map 0% Reduce 0%

Map 1% Reduce 0%

Map 33% Reduce 0%

Map 93% Reduce 0%

Error using mapreduce (line 118)

Unable to read MAT-file /tmp/tp1ce5fe8e_0189_4e64_85a3_b671c61453a4/task_0_675_MAP_4.mat: not a binary MAT-file.

Error in create_seq (line 101)

seqds = mapreduce(imageDS, @identityMap, @identityReduce,'OutputFolder',output_identity);

Error -2

> whos -file '/home/nitw_viper_user/task_0_1081_MAP_4.mat'

Name       Size            Bytes  Class         Attributes

Error      1x1              2336  MException

>> Error

Error =

MException with properties:

    identifier: 'parallel:internal:DeserializationException'

       message: 'Deserialization threw an exception.'

         cause: {0×1 cell}

         stack: [3×1 struct]

Error -3

>> create_seq

Hadoop with properties:

                 HadoopInstallFolder: '/home/nitw_viper_user/hadoop-

2.7.2'

             HadoopConfigurationFile: ''

                  SparkInstallFolder: ''

                    HadoopProperties: [2×1 containers.Map]

                     SparkProperties: [0×1 containers.Map]

                   ClusterMatlabRoot: '/usr/local/MATLAB/R2016b'

    RequiresMathWorksHostedLicensing: 0

                       LicenseNumber: ''

                     AutoAttachFiles: 1

                       AttachedFiles: {}

                    AdditionalPaths: {}

Parallel mapreduce execution on the Hadoop cluster:

****************************** * MAPREDUCE PROGRESS *******************************

Map 0% Reduce 0%

Map 1% Reduce 0%

Map 2% Reduce 0%

Map 22% Reduce 0%

Map 40% Reduce 0%

Map 80% Reduce 0%

Error using mapreduce (line 118)

The HADOOP job failed to complete.

Error in create_seq (line 101)

seqds = mapreduce(imageDS, @identityMap, @identityReduce,'OutputFolder',output_identity);

Caused by:

    Error using distcompdeserialize

    Deserialization threw an exception.

    Error using distcompdeserialize

    Deserialization threw an exception.

    Error using distcompdeserialize

    Deserialization threw an exception.

    Error using distcompdeserialize

    Deserialization threw an exception.

    Error using distcompdeserialize

    Deserialization threw an exception.

   Error using distcompdeserialize

    Deserialization threw an exception.

    Error using distcompdeserialize

    Deserialization threw an exception.

    Error using distcompdeserialize

    Deserialization threw an exception.

>>

date:07-05-2017

[WARN] BlockReaderFactory - I/O error constructing remote block reader. <java.io.IOException: Got error, status message opReadBlock BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461, for OP_READ_BLOCK, self=/192.168.192.128:60935, remote=/192.168.193.177:50010, for file /fv_13l_2/.matlaberror /task_0_797_MAP_1.mat, for pool BP-581788350-127.0.1.1-1490718884252 block 1075155042_1414461>java.io.IOException: Got error, status message opReadBlock BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461, for OP_READ_BLOCK, self=/192.168.192.128:60935, remote=/192.168.193.177:50010, for file /fv_13l_2/.matlaberror /task_0_797_MAP_1.mat, for pool BP-581788350-127.0.1.1-1490718884252 block 1075155042_1414461

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.ch eckBlockOpStatus(DataTransferProtoUtil.java:140)

at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockRea der2.java:456)

at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockR eader2.java:424)

at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockR eaderFactory.java:818)

at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp (BlockReaderFactory.java:697)

at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.ja va:355)

at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java :656)

at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream .java:882)

at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)

at java.io.DataInputStream.read(Unknown Source)

at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)

at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)

at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)

[WARN] DFSClient - Failed to connect to /192.168.193.177:50010 for block, add to deadNodes and continue. java.io.IOException: Got

error, status message opReadBlock BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461, for OP_READ_BLOCK, self=/192.168.192.128:60935, remote=/192.168.193.177:50010, for file /fv_13l_2/.matlaberror /task_0_797_MAP_1.mat, for pool BP-581788350-127.0.1.1-1490718884252 block 1075155042_1414461 <java.io.IOException: Got error, status message opReadBlock BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461, for OP_READ_BLOCK, self=/192.168.192.128:60935, remote=/192.168.193.177:50010, for file /fv_13l_2/.matlaberror /task_0_797_MAP_1.mat, for pool BP-581788350-127.0.1.1-1490718884252 block 1075155042_1414461>java.io.IOException: Got error, status message opReadBlock BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461, for OP_READ_BLOCK, self=/192.168.192.128:60935, remote=/192.168.193.177:50010, for file /fv_13l_2/.matlaberror /task_0_797_MAP_1.mat, for pool BP-581788350-127.0.1.1-1490718884252 block 1075155042_1414461

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.ch eckBlockOpStatus(DataTransferProtoUtil.java:140)

at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockRea der2.java:456)

at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockR eader2.java:424)

at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockR eaderFactory.java:818)

at o

rg.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp( BlockReaderFactory.java:697)

at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:656) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934) at java.io.DataInputStream.read(Unknown Source) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)

[INFO] DFSClient - Successfully connected to /192.168.193.167:50010 for BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461 Error using mapreduce (line 118) Unable to read MAT-file /tmp/tp3c745940_f508_4326_93f7_fc5f6fb9ef06/task_0_805_MAP_1.mat: not a binary MAT-file.

Error in main (line 270) res = mapreduce(seqds, @Ltrp_db1_seq_file_mapper, @Ltrp_db1_reducer, 'OutputFolder', Ltrp_db1_seq_file_result); bold

[Kojiro Saito]

I tested create_seq.m.

Is the following correct?

seqds = mapreduce(imageDS, @identityMap, @identityReduce,'OutputFolder',output_identity);

I've modified as the following,

seqds = mapreduce(imageDs, @identityMapper, @identityReducer, 'OutputFolder', output_identity);

and get no error. I'm not sure the content of main.m, so I couldn't test init.m nor main.m.

[Pulkesh Haran]

We are working on (Matlab+Hadoop)Cluster consist of 110 nodes each having 8GB RAM.

We tested making of sequence file for 40 thousand images and it is working fine. For More than 40 thousand images it is giving following error:

1. Error : Error using distcompdeserialize

               *Deserialization threw an exception.*

2.Error : Unable to read MAT-file /tmp/tp1ce5fe8e_0189_4e64_85a3_b671c61453a4/task_0_675_MAP_4.mat: not a binary MAT-file.

• MException with properties:*

    identifier: 'parallel:internal:DeserializationException'

       message: 'Deserialization threw an exception.'

         cause: {0×1 cell}

         stack: [3×1 struct]

_ We request you to please provide solution for this Error._ Thanks

[Walter Roberson]

You should open a support case about this.

[Kojiro Saito]

Could you attach the log of Hadoop failed job?

yarn logs -applicationId APPLICATION_ID

Anyway, I think it's better to contact to MathWorks support, too.

[Pulkesh Haran]

This is log of Hadoop Failed Job :

[WARN] BlockReaderFactory - I/O error constructing remote block reader. <java.io.IOException: Got error, status message opReadBlock BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461, for OP_READ_BLOCK, self=/192.168.192.128:60935, remote=/192.168.193.177:50010, for file /fv_13l_2/.matlaberror /task_0_797_MAP_1.mat, for pool BP-581788350-127.0.1.1-1490718884252 block 1075155042_1414461>java.io.IOException: Got error, status message opReadBlock BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461, for OP_READ_BLOCK, self=/192.168.192.128:60935, remote=/192.168.193.177:50010, for file /fv_13l_2/.matlaberror /task_0_797_MAP_1.mat, for pool BP-581788350-127.0.1.1-1490718884252 block 1075155042_1414461

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.ch eckBlockOpStatus(DataTransferProtoUtil.java:140)

at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockRea der2.java:456)

at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockR eader2.java:424)

at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockR eaderFactory.java:818)

at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp (BlockReaderFactory.java:697)

at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.ja va:355)

at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java :656)

at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream .java:882)

at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)

at java.io.DataInputStream.read(Unknown Source)

at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)

at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)

at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)

[WARN] DFSClient - Failed to connect to /192.168.193.177:50010 for block, add to deadNodes and continue. java.io.IOException: Got

error, status message opReadBlock BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461, for OP_READ_BLOCK, self=/192.168.192.128:60935, remote=/192.168.193.177:50010, for file /fv_13l_2/.matlaberror /task_0_797_MAP_1.mat, for pool BP-581788350-127.0.1.1-1490718884252 block 1075155042_1414461 <java.io.IOException: Got error, status message opReadBlock BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461, for OP_READ_BLOCK, self=/192.168.192.128:60935, remote=/192.168.193.177:50010, for file /fv_13l_2/.matlaberror /task_0_797_MAP_1.mat, for pool BP-581788350-127.0.1.1-1490718884252 block 1075155042_1414461>java.io.IOException: Got error, status message opReadBlock BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-581788350-127.0.1.1-1490718884252:blk_1075155042_1414461, for OP_READ_BLOCK, self=/192.168.192.128:60935, remote=/192.168.193.177:50010, for file /fv_13l_2/.matlaberror /task_0_797_MAP_1.mat, for pool BP-581788350-127.0.1.1-1490718884252 block 1075155042_1414461

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.ch eckBlockOpStatus(DataTransferProtoUtil.java:140)

at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockRea der2.java:456)

at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockR eader2.java:424)

at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockR eaderFactory.java:818)

at o

rg.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp( BlockReaderFactory.java:697)

at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:656) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934) at java.io.DataInputStream.read(Unknown Source) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)

[Kojiro Saito]

What information does "task_0_797_MAP_1.mat" file contain?

[Pulkesh Haran]

Hello Sir, I attaching file and Error Screen picture with this.Please Check it and guide us further

[Kojiro Saito]

It seems datanode disk check failed that's why MDCS produced deserialization error. Could you check datanode's log and out files and find any error message?

[Pulkesh Haran]

In datanode logs... No error is showing.. All container are clear... Only .matlaberror folder is created with list of map task failed

[Kojiro Saito]

All right. Is there any corrupted images in 50K files?

If not, you should contact to MathWorks Technical Support.

Answer 2

[lov kumar]

How to fix this error. Please help

Error using mapreduce (line 124)

The HADOOP job failed to submit. It is possible that there is some issue with the HADOOP configuration.