datastoreToRDD

Class: matlab.compiler.mlspark.SparkContext
Package: matlab.compiler.mlspark

Convert MATLAB datastore to a Spark RDD

expand all in page

Syntax

rdd = datastoreToRDD(sc,ds)

Description

rdd = datastoreToRDD(sc,ds) converts a MATLAB^® datastore object ds to a Spark™ RDD.

Input Arguments

expand all

`sc` — SparkContext to use
`SparkContext` object

The SparkContext to use, specified as a SparkContext object.

`ds` — Datastore to be converted
MATLAB `datastore`

Datastore to be converted to an RDD, specified as a MATLAB datastore object.

Output Arguments

expand all

`rdd` — Output RDD
`RDD` object

Output RDD representing the converted datastore object, returned as a RDD object.

Examples

expand all

Convert MATLAB Datastore to Spark RDD

Convert a MATLAB datastore object to a Spark RDD.

% Setup Spark Properties as a MATLAB Map object using a containers.Map class
sparkProp = containers.Map({'spark.executor.cores'}, {'1'}); 

% Create SparkConf object
conf = matlab.compiler.mlspark.SparkConf(...
    'AppName'        , 'myApp', ...
    'Master'         , 'local[1]', ...
    'SparkProperties',  sparkProp );
     
% Create a SparkContext
sc = matlab.compiler.mlspark.SparkContext(conf);

% Create a MATLAB datastore 
ds = datastore('airlinesmall.csv','TreatAsMissing','NA');

% Convert MATLAB datastore to Spark RDD 
rdd = datastoreToRDD(sc,ds);

% Alternate object usage:
rdd = sc.datastoreToRDD(ds);