edu.iu.dsc.tws.tset.env

Class BatchTSetEnvironment

java.lang.Object

edu.iu.dsc.tws.tset.env.TSetEnvironment

edu.iu.dsc.tws.tset.env.BatchTSetEnvironment

Direct Known Subclasses:

BatchTSetCDFWEnvironment, BeamBatchTSetEnvironment, CheckpointingTSetEnv

public class BatchTSetEnvironment
extends TSetEnvironment

Implementation of TSetEnvironment for batch OperationMode.

There are 2 ways a tset be executed. 1. running a tset (at the completion, execution will be closed) 2. evaluating a tset (execution will be kept alive until 'finishEval' method is called)

And there are 3 execution options. 1. Running/ evaluating a subgraph/ DAG from a specified TSet 2. Running/ evaluating a tset and update another with the results 3. Running just a single source TSet

Constructor Summary

Constructors

Constructor and Description
BatchTSetEnvironment()
BatchTSetEnvironment(WorkerEnvironment wEnv)

Method Summary

Modifier and Type	Method and Description
<K,V,F extends <any>> SourceTSet<Tuple<K,V>>	createHadoopSource(Configuration configuration, java.lang.Class<F> inputFormat, int parallel)
<K,V,F extends <any>,I> SourceTSet<I>	createHadoopSource(Configuration configuration, java.lang.Class<F> inputFormat, int parallel, MapFunc<I,Tuple<K,V>> mapFunc)
<K,V,F extends <any>,K2,V2> KeyedSourceTSet<K2,V2>	createKeyedHadoopSource(Configuration configuration, java.lang.Class<F> inputFormat, int parallel, MapFunc<Tuple<K2,V2>,Tuple<K,V>> mapFunc)
<K,V> KeyedSourceTSet<K,V>	createKeyedSource(SourceFunc<Tuple<K,V>> source, int parallelism) Creates a Keyed Source TSet based on the SourceFunc that produces a Tuple
<K,V> KeyedSourceTSet<K,V>	createKeyedSource(java.lang.String name, SourceFunc<Tuple<K,V>> source, int parallelism) Same as above, but a source tset name can be provided at the instantiation
<T> SourceTSet<T>	createSource(SourceFunc<T> source, int parallelism) Creates a source TSet based on the SourceFunc
<T> SourceTSet<T>	createSource(java.lang.String name, SourceFunc<T> source, int parallelism) Same as above, but a source tset name can be provided at the instantiation
void	eval(BaseTSet evalTSet) Evaluates the TSet using iterative execution in the executor.
<T,ST extends BaseTSet<T> & StorableTBase<T>> void	evalAndUpdate(ST evalTSet, ST updateTSet) Similar to eval, but here, the data produced by the evaluation will be passed on to the updateTSet
void	finishEval(BaseTSet evalTset) Completes iterative execution for evaluated TSet
<T> DataObject<T>	getData(java.lang.String key)
OperationMode	getOperationMode() Returns the OperationMode
void	run(BaseTSet leafTset) Runs a subgraph of TSets from the specified TSet
<T,ST extends BaseTSet<T> & StorableTBase<T>> void	runAndUpdate(ST runTSet, ST updateTSet) Runs a subgraph of TSets from the specified TSet and output results as a tset
void	runOne(BaseTSet tSet) Runs a single TSet (NO subgraph execution!)

Methods inherited from class edu.iu.dsc.tws.tset.env.TSetEnvironment

addInput, close, executeBuildContext, getConfig, getDefaultParallelism, getGraph, getInputs, getNoOfWorkers, getTaskExecutor, getTSetGraph, getWorkerID, initBatch, initStreaming, isCheckpointingEnabled, parallelize, parallelize, parallelize, parallelize, setDefaultParallelism, settBaseGraph

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BatchTSetEnvironment

public BatchTSetEnvironment(WorkerEnvironment wEnv)

BatchTSetEnvironment

public BatchTSetEnvironment()

Method Detail

getOperationMode

public OperationMode getOperationMode()

Description copied from class: TSetEnvironment

Returns the OperationMode

Specified by:

getOperationMode in class TSetEnvironment

Returns:

operation mode

createSource

public <T> SourceTSet<T> createSource(SourceFunc<T> source,
                                      int parallelism)

Description copied from class: TSetEnvironment

Creates a source TSet based on the SourceFunc

Specified by:

createSource in class TSetEnvironment

Type Parameters:

T - data type

Parameters:

source - source function

parallelism - parallelism

Returns:

Source TSet

createSource

public <T> SourceTSet<T> createSource(java.lang.String name,
                                      SourceFunc<T> source,
                                      int parallelism)

Description copied from class: TSetEnvironment

Same as above, but a source tset name can be provided at the instantiation

Specified by:

createSource in class TSetEnvironment

Type Parameters:

T - data type

Parameters:

name - name for the tset

source - source function

parallelism - parallelism

Returns:

Source TSet

createKeyedSource

public <K,V> KeyedSourceTSet<K,V> createKeyedSource(SourceFunc<Tuple<K,V>> source,
                                                    int parallelism)

Description copied from class: TSetEnvironment

Creates a Keyed Source TSet based on the SourceFunc that produces a Tuple

Specified by:

createKeyedSource in class TSetEnvironment

Type Parameters:

K - key type

V - value type

Parameters:

source - source function

parallelism - parallelism

Returns:

Keyed Source TSet

createKeyedSource

public <K,V> KeyedSourceTSet<K,V> createKeyedSource(java.lang.String name,
                                                    SourceFunc<Tuple<K,V>> source,
                                                    int parallelism)

Description copied from class: TSetEnvironment

Same as above, but a source tset name can be provided at the instantiation

Specified by:

createKeyedSource in class TSetEnvironment

Type Parameters:

K - key type

V - value type

Parameters:

name - name for the tset

source - source function

parallelism - parallelism

Returns:

Keyed Source TSet

createHadoopSource

public <K,V,F extends <any>> SourceTSet<Tuple<K,V>> createHadoopSource(Configuration configuration,
                                                                       java.lang.Class<F> inputFormat,
                                                                       int parallel)

createHadoopSource

public <K,V,F extends <any>,I> SourceTSet<I> createHadoopSource(Configuration configuration,
                                                                java.lang.Class<F> inputFormat,
                                                                int parallel,
                                                                MapFunc<I,Tuple<K,V>> mapFunc)

createKeyedHadoopSource

public <K,V,F extends <any>,K2,V2> KeyedSourceTSet<K2,V2> createKeyedHadoopSource(Configuration configuration,
                                                                                  java.lang.Class<F> inputFormat,
                                                                                  int parallel,
                                                                                  MapFunc<Tuple<K2,V2>,Tuple<K,V>> mapFunc)

getData

public <T> DataObject<T> getData(java.lang.String key)

runOne

public void runOne(BaseTSet tSet)

Runs a single TSet (NO subgraph execution!)

Parameters:

tSet - tset to run

run

public void run(BaseTSet leafTset)

Runs a subgraph of TSets from the specified TSet

Parameters:

leafTset - TSet to be run

runAndUpdate

public <T,ST extends BaseTSet<T> & StorableTBase<T>> void runAndUpdate(ST runTSet,
                                                                       ST updateTSet)

Runs a subgraph of TSets from the specified TSet and output results as a tset

Type Parameters:

T - type of the output data object

Parameters:

runTSet - TSet to be run

updateTSet - TSet to be updated

eval

public void eval(BaseTSet evalTSet)

Evaluates the TSet using iterative execution in the executor. The task graph generated by the evaluation will be cached and reused in subsequent evaluation calls of that particular TSet. To complete the iterative execution, call @finishEval method

Parameters:

evalTSet - TSet to be evaluated

evalAndUpdate

public <T,ST extends BaseTSet<T> & StorableTBase<T>> void evalAndUpdate(ST evalTSet,
                                                                        ST updateTSet)

Similar to eval, but here, the data produced by the evaluation will be passed on to the updateTSet

Type Parameters:

T - type

Parameters:

evalTSet - TSet to be evaluated

updateTSet - TSet to be updated

finishEval

public void finishEval(BaseTSet evalTset)

Completes iterative execution for evaluated TSet

Parameters:

evalTset - TSet to be evaluated