edu.iu.dsc.tws.tset.env

Class TSetEnvironment

java.lang.Object

edu.iu.dsc.tws.tset.env.TSetEnvironment

Direct Known Subclasses:

BatchEnvironment, StreamingEnvironment

public abstract class TSetEnvironment
extends java.lang.Object

Entry point to tset operations. This is a singleton which initializes as BatchEnvironment or StreamingEnvironment

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	TSetEnvironment() Used to construct the TSet environment when in the connected data flow mode.
protected	TSetEnvironment(WorkerEnvironment wEnv)

Method Summary

Modifier and Type	Method and Description
void	addInput(java.lang.String tSetID, java.lang.String inputTSetID, java.lang.String inputKey) Adds a TSet to another TSet as an input that will be identified by the inputKey
void	close() Close the environment
abstract BaseTSet<java.lang.Integer>	createArrowSource(java.lang.String filePath, int parallelism)
abstract BaseTSet<java.lang.String[]>	createCSVSource(java.lang.String filePath, int dataSize, int parallelism, java.lang.String type)
abstract <K,V> TupleTSet<K,V>	createKeyedSource(SourceFunc<Tuple<K,V>> source, int parallelism) Creates a Keyed Source TSet based on the SourceFunc that produces a Tuple
abstract <K,V> TupleTSet<K,V>	createKeyedSource(java.lang.String name, SourceFunc<Tuple<K,V>> source, int parallelism) Same as above, but a source tset name can be provided at the instantiation
abstract <T> BaseTSet<T>	createSource(SourceFunc<T> source, int parallelism) Creates a source TSet based on the SourceFunc
abstract <T> BaseTSet<T>	createSource(java.lang.String name, SourceFunc<T> source, int parallelism) Same as above, but a source tset name can be provided at the instantiation
abstract BaseTSet<java.lang.String>	createTextSource(java.lang.String filePath, int dataSize, int parallelism, java.lang.String type)
protected void	executeBuildContext(BuildContext buildContext) Executes data flow graph wrapped by a BuildContext
Config	getConfig() returns the config object received from the iWorker Config
int	getDefaultParallelism() Default parallelism
TBaseGraph	getGraph() Returns the tset graph
java.util.Map<java.lang.String,java.lang.String>	getInputs(java.lang.String tSetID) Returns the map of inputs of a particular TSet
int	getNoOfWorkers() No of twister2 workers
abstract OperationMode	getOperationMode() Returns the OperationMode
TaskExecutor	getTaskExecutor()
TBaseGraph	getTSetGraph()
WorkerEnvironment	getWorkerEnv()
int	getWorkerID() Running worker ID
static BatchEnvironment	initBatch(WorkerEnvironment wEnv) initialize the Tset environment in batch OperationMode
static BatchChkPntEnvironment	initCheckpointing(WorkerEnvironment wEnv) initialize the Tset environment for checkpointing OperationMode
static StreamingEnvironment	initStreaming(WorkerEnvironment wEnv) initialize the Tset environment in streaming OperationMode
boolean	isCheckpointingEnabled() Checks if checkpointing is enabled
<T> BaseTSet<T>	parallelize(java.util.List<T> list, int parallelism) This method will create a source based on the list and each source will read only a part of the list specified.
<K extends java.lang.Comparable,V> TupleTSet<K,V>	parallelize(java.util.Map<K,V> map, int parallelism)
<K,V> TupleTSet<K,V>	parallelize(java.util.Map<K,V> map, int parallelism, java.util.Comparator<K> keyComparator)
<T> BaseTSet<T>	parallelize(java.lang.String name, java.util.List<T> list, int parallelism) This method will create a source based on the list and each source will read only a part of the list specified.
void	setDefaultParallelism(int newDefaultParallelism) Overrides the default parallelism.
void	settBaseGraph(TBaseGraph tBaseGraph) Sets TBaseGraph based on the OperationMode

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TSetEnvironment

protected TSetEnvironment(WorkerEnvironment wEnv)

TSetEnvironment

protected TSetEnvironment()

Used to construct the TSet environment when in the connected data flow mode.

Method Detail

getWorkerEnv

public WorkerEnvironment getWorkerEnv()

getOperationMode

public abstract OperationMode getOperationMode()

Returns the OperationMode

Returns:

operation mode

createSource

public abstract <T> BaseTSet<T> createSource(SourceFunc<T> source,
                                             int parallelism)

Creates a source TSet based on the SourceFunc

Type Parameters:

T - data type

Parameters:

source - source function

parallelism - parallelism

Returns:

Source TSet

createSource

public abstract <T> BaseTSet<T> createSource(java.lang.String name,
                                             SourceFunc<T> source,
                                             int parallelism)

Same as above, but a source tset name can be provided at the instantiation

Type Parameters:

T - data type

Parameters:

name - name for the tset

source - source function

parallelism - parallelism

Returns:

Source TSet

createCSVSource

public abstract BaseTSet<java.lang.String[]> createCSVSource(java.lang.String filePath,
                                                             int dataSize,
                                                             int parallelism,
                                                             java.lang.String type)

createTextSource

public abstract BaseTSet<java.lang.String> createTextSource(java.lang.String filePath,
                                                            int dataSize,
                                                            int parallelism,
                                                            java.lang.String type)

createArrowSource

public abstract BaseTSet<java.lang.Integer> createArrowSource(java.lang.String filePath,
                                                              int parallelism)

parallelize

public <T> BaseTSet<T> parallelize(java.lang.String name,
                                   java.util.List<T> list,
                                   int parallelism)

This method will create a source based on the list and each source will read only a part of the list specified.

Type Parameters:

T - data type of the list

Parameters:

name - name of the tset

list - list to be parallelized

parallelism - no of sources to be created

Returns:

Source TSet

parallelize

public <T> BaseTSet<T> parallelize(java.util.List<T> list,
                                   int parallelism)

This method will create a source based on the list and each source will read only a part of the list specified.

Type Parameters:

T - data type of the list

Parameters:

list - list to be parallelized

parallelism - no of sources to be created

Returns:

Source TSet

parallelize

public <K extends java.lang.Comparable,V> TupleTSet<K,V> parallelize(java.util.Map<K,V> map,
                                                                     int parallelism)

parallelize

public <K,V> TupleTSet<K,V> parallelize(java.util.Map<K,V> map,
                                        int parallelism,
                                        java.util.Comparator<K> keyComparator)

createKeyedSource

public abstract <K,V> TupleTSet<K,V> createKeyedSource(SourceFunc<Tuple<K,V>> source,
                                                       int parallelism)

Creates a Keyed Source TSet based on the SourceFunc that produces a Tuple

Type Parameters:

K - key type

V - value type

Parameters:

source - source function

parallelism - parallelism

Returns:

Keyed Source TSet

createKeyedSource

public abstract <K,V> TupleTSet<K,V> createKeyedSource(java.lang.String name,
                                                       SourceFunc<Tuple<K,V>> source,
                                                       int parallelism)

Same as above, but a source tset name can be provided at the instantiation

Type Parameters:

K - key type

V - value type

Parameters:

name - name for the tset

source - source function

parallelism - parallelism

Returns:

Keyed Source TSet

getGraph

public TBaseGraph getGraph()

Returns the tset graph

Returns:

tset graph

setDefaultParallelism

public void setDefaultParallelism(int newDefaultParallelism)

Overrides the default parallelism. Default is 1

Parameters:

newDefaultParallelism - new parallelism

getDefaultParallelism

public int getDefaultParallelism()

Default parallelism

Returns:

default parallelism

getConfig

public Config getConfig()

returns the config object received from the iWorker Config

Returns:

config

getWorkerID

public int getWorkerID()

Running worker ID

Returns:

workerID

getNoOfWorkers

public int getNoOfWorkers()

No of twister2 workers

isCheckpointingEnabled

public boolean isCheckpointingEnabled()

Checks if checkpointing is enabled

Returns:

bool

addInput

public void addInput(java.lang.String tSetID,
                     java.lang.String inputTSetID,
                     java.lang.String inputKey)

Adds a TSet to another TSet as an input that will be identified by the inputKey

Parameters:

tSetID - TSet ID

inputTSetID - input TSet ID

inputKey - key given to the input TSet

getInputs

public java.util.Map<java.lang.String,java.lang.String> getInputs(java.lang.String tSetID)

Returns the map of inputs of a particular TSet

Parameters:

tSetID - TSet ID

Returns:

map of inputs that maps inputTSetDD --> inputKey

initBatch

public static BatchEnvironment initBatch(WorkerEnvironment wEnv)

initialize the Tset environment in batch OperationMode

Parameters:

wEnv - worker environment

Returns:

BatchTSetEnvironment

initStreaming

public static StreamingEnvironment initStreaming(WorkerEnvironment wEnv)

initialize the Tset environment in streaming OperationMode

Parameters:

wEnv - worker environment

Returns:

StreamingTSetEnvironment

initCheckpointing

public static BatchChkPntEnvironment initCheckpointing(WorkerEnvironment wEnv)

initialize the Tset environment for checkpointing OperationMode

Parameters:

wEnv - worker environment

Returns:

CheckpointingTSetEnv

settBaseGraph

public void settBaseGraph(TBaseGraph tBaseGraph)

Sets TBaseGraph based on the OperationMode

Parameters:

tBaseGraph - TBase graph

executeBuildContext

protected void executeBuildContext(BuildContext buildContext)

Executes data flow graph wrapped by a BuildContext

Parameters:

buildContext - data flow graph wrapped by BuildContext

getTSetGraph

public TBaseGraph getTSetGraph()

getTaskExecutor

public TaskExecutor getTaskExecutor()

close

public void close()

Close the environment