edu.iu.dsc.tws.tset.env

Class BatchEnvironment

java.lang.Object

edu.iu.dsc.tws.tset.env.TSetEnvironment

edu.iu.dsc.tws.tset.env.BatchEnvironment

Direct Known Subclasses:

BatchChkPntEnvironment, BatchTSetCDFWEnvironment, BeamBatchTSetEnvironment

public class BatchEnvironment
extends TSetEnvironment

Implementation of TSetEnvironment for batch OperationMode.

There are 2 ways a tset be executed. 1. running a tset (at the completion, execution will be closed) 2. evaluating a tset (execution will be kept alive until 'finishEval' method is called)

And there are 3 execution options. 1. Running/ evaluating a subgraph/ DAG from a specified TSet 2. Running/ evaluating a tset and update another with the results 3. Running just a single source TSet

Constructor Summary

Constructors

Constructor and Description
BatchEnvironment()
BatchEnvironment(WorkerEnvironment wEnv)

Method Summary

Modifier and Type	Method and Description
BaseTSet<java.lang.Integer>	createArrowSource(java.lang.String filePath, int parallelism)
SourceTSet<java.lang.Object>	createArrowSource(java.lang.String filePath, java.lang.String fileName, int parallelism, java.lang.String schema)
SourceTSet<java.lang.String[]>	createCSVSource(java.lang.String filePath, int datasize, int parallelism, java.lang.String type)
<K,V,F extends <any>> SourceTSet<Tuple<K,V>>	createHadoopSource(Configuration configuration, java.lang.Class<F> inputFormat, int parallel)
<K,V,F extends <any>,I> SourceTSet<I>	createHadoopSource(Configuration configuration, java.lang.Class<F> inputFormat, int parallel, MapFunc<I,Tuple<K,V>> mapFunc)
<K,V,F extends <any>,K2,V2> KeyedSourceTSet<K2,V2>	createKeyedHadoopSource(Configuration configuration, java.lang.Class<F> inputFormat, int parallel, MapFunc<Tuple<K2,V2>,Tuple<K,V>> mapFunc)
<K,V> KeyedSourceTSet<K,V>	createKeyedSource(SourceFunc<Tuple<K,V>> source, int parallelism) Creates a Keyed Source TSet based on the SourceFunc that produces a Tuple
<K,V> KeyedSourceTSet<K,V>	createKeyedSource(java.lang.String name, SourceFunc<Tuple<K,V>> source, int parallelism) Same as above, but a source tset name can be provided at the instantiation
RowSourceTSet	createRowSource(java.lang.String name, SourceFunc<Row> source, int parallelism)
<T> SourceTSet<T>	createSource(SourceFunc<T> source, int parallelism) Creates a source TSet based on the SourceFunc
<T> SourceTSet<T>	createSource(java.lang.String name, SourceFunc<T> source, int parallelism) Same as above, but a source tset name can be provided at the instantiation
SourceTSet<java.lang.String>	createTextSource(java.lang.String filePath, int dataSize, int parallelism, java.lang.String type)
void	eval(BaseTSet evalTSet) Evaluates the TSet using iterative execution in the executor.
<T,ST extends BaseTSet<T> & StorableTBase<T>> void	evalAndUpdate(ST evalTSet, ST updateTSet) Similar to eval, but here, the data produced by the evaluation will be passed on to the updateTSet
void	finishEval(BaseTSet evalTset) Completes iterative execution for evaluated TSet
<T> DataObject<T>	getData(java.lang.String key)
OperationMode	getOperationMode() Returns the OperationMode
void	run(BaseTSet leafTset) Runs a subgraph of TSets from the specified TSet
<T,ST extends BaseTSet<T> & StorableTBase<T>> void	runAndUpdate(ST runTSet, ST updateTSet) Runs a subgraph of TSets from the specified TSet and output results as a tset
void	runOne(BaseTSet tSet) Runs a single TSet (NO subgraph execution!)

Methods inherited from class edu.iu.dsc.tws.tset.env.TSetEnvironment

addInput, close, executeBuildContext, getConfig, getDefaultParallelism, getGraph, getInputs, getNoOfWorkers, getTaskExecutor, getTSetGraph, getWorkerEnv, getWorkerID, initBatch, initCheckpointing, initStreaming, isCheckpointingEnabled, parallelize, parallelize, parallelize, parallelize, setDefaultParallelism, settBaseGraph

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BatchEnvironment

public BatchEnvironment(WorkerEnvironment wEnv)

BatchEnvironment

public BatchEnvironment()

Method Detail

getOperationMode

public OperationMode getOperationMode()

Description copied from class: TSetEnvironment

Returns the OperationMode

Specified by:

getOperationMode in class TSetEnvironment

Returns:

operation mode

createSource

public <T> SourceTSet<T> createSource(SourceFunc<T> source,
                                      int parallelism)

Description copied from class: TSetEnvironment

Creates a source TSet based on the SourceFunc

Specified by:

createSource in class TSetEnvironment

Type Parameters:

T - data type

Parameters:

source - source function

parallelism - parallelism

Returns:

Source TSet

createCSVSource

public SourceTSet<java.lang.String[]> createCSVSource(java.lang.String filePath,
                                                      int datasize,
                                                      int parallelism,
                                                      java.lang.String type)

Specified by:

createCSVSource in class TSetEnvironment

createArrowSource

public SourceTSet<java.lang.Object> createArrowSource(java.lang.String filePath,
                                                      java.lang.String fileName,
                                                      int parallelism,
                                                      java.lang.String schema)

createTextSource

public SourceTSet<java.lang.String> createTextSource(java.lang.String filePath,
                                                     int dataSize,
                                                     int parallelism,
                                                     java.lang.String type)

Specified by:

createTextSource in class TSetEnvironment

createArrowSource

public BaseTSet<java.lang.Integer> createArrowSource(java.lang.String filePath,
                                                     int parallelism)

Specified by:

createArrowSource in class TSetEnvironment

createSource

public <T> SourceTSet<T> createSource(java.lang.String name,
                                      SourceFunc<T> source,
                                      int parallelism)

Description copied from class: TSetEnvironment

Same as above, but a source tset name can be provided at the instantiation

Specified by:

createSource in class TSetEnvironment

Type Parameters:

T - data type

Parameters:

name - name for the tset

source - source function

parallelism - parallelism

Returns:

Source TSet

createKeyedSource

public <K,V> KeyedSourceTSet<K,V> createKeyedSource(SourceFunc<Tuple<K,V>> source,
                                                    int parallelism)

Description copied from class: TSetEnvironment

Creates a Keyed Source TSet based on the SourceFunc that produces a Tuple

Specified by:

createKeyedSource in class TSetEnvironment

Type Parameters:

K - key type

V - value type

Parameters:

source - source function

parallelism - parallelism

Returns:

Keyed Source TSet

createKeyedSource

public <K,V> KeyedSourceTSet<K,V> createKeyedSource(java.lang.String name,
                                                    SourceFunc<Tuple<K,V>> source,
                                                    int parallelism)

Description copied from class: TSetEnvironment

Same as above, but a source tset name can be provided at the instantiation

Specified by:

createKeyedSource in class TSetEnvironment

Type Parameters:

K - key type

V - value type

Parameters:

name - name for the tset

source - source function

parallelism - parallelism

Returns:

Keyed Source TSet

createRowSource

public RowSourceTSet createRowSource(java.lang.String name,
                                     SourceFunc<Row> source,
                                     int parallelism)

createHadoopSource

public <K,V,F extends <any>> SourceTSet<Tuple<K,V>> createHadoopSource(Configuration configuration,
                                                                       java.lang.Class<F> inputFormat,
                                                                       int parallel)

createHadoopSource

public <K,V,F extends <any>,I> SourceTSet<I> createHadoopSource(Configuration configuration,
                                                                java.lang.Class<F> inputFormat,
                                                                int parallel,
                                                                MapFunc<I,Tuple<K,V>> mapFunc)

createKeyedHadoopSource

public <K,V,F extends <any>,K2,V2> KeyedSourceTSet<K2,V2> createKeyedHadoopSource(Configuration configuration,
                                                                                  java.lang.Class<F> inputFormat,
                                                                                  int parallel,
                                                                                  MapFunc<Tuple<K2,V2>,Tuple<K,V>> mapFunc)

getData

public <T> DataObject<T> getData(java.lang.String key)

runOne

public void runOne(BaseTSet tSet)

Runs a single TSet (NO subgraph execution!)

Parameters:

tSet - tset to run

run

public void run(BaseTSet leafTset)

Runs a subgraph of TSets from the specified TSet

Parameters:

leafTset - TSet to be run

runAndUpdate

public <T,ST extends BaseTSet<T> & StorableTBase<T>> void runAndUpdate(ST runTSet,
                                                                       ST updateTSet)

Runs a subgraph of TSets from the specified TSet and output results as a tset

Type Parameters:

T - type of the output data object

Parameters:

runTSet - TSet to be run

updateTSet - TSet to be updated

eval

public void eval(BaseTSet evalTSet)

Evaluates the TSet using iterative execution in the executor. The task graph generated by the evaluation will be cached and reused in subsequent evaluation calls of that particular TSet. To complete the iterative execution, call @finishEval method

Parameters:

evalTSet - TSet to be evaluated

evalAndUpdate

public <T,ST extends BaseTSet<T> & StorableTBase<T>> void evalAndUpdate(ST evalTSet,
                                                                        ST updateTSet)

Similar to eval, but here, the data produced by the evaluation will be passed on to the updateTSet

Type Parameters:

T - type

Parameters:

evalTSet - TSet to be evaluated

updateTSet - TSet to be updated

finishEval

public void finishEval(BaseTSet evalTset)

Completes iterative execution for evaluated TSet

Parameters:

evalTset - TSet to be evaluated