edu.iu.dsc.tws.api.tset.sets

Interface TSet<T>

Type Parameters:

T - type of the data set

All Superinterfaces:

java.io.Serializable, TBase

All Known Subinterfaces:

BatchTSet<T>, StreamingTSet<T>

All Known Implementing Classes:

BatchTSetImpl, CachedTSet, CheckpointedTSet, ComputeTSet, PersistedTSet, SComputeTSet, SourceTSet, SSinkTSet, SSourceTSet, StoredTSet, StreamingTSetImpl, WindowComputeTSet

public interface TSet<T>
extends TBase

Twister data set. A TSet would abstract a Task level computation (Source/ Compute or Sink) in a more user friendly API. A TSet would be followed by a TLink that would expose the communication level operations performed on the computation. Note the extensions of this interface BatchTSet and StreamingTSet. These would intimately separate out the operations based on the OperationMode of the data flow graph. This interface only specifies the common operations for Batch and Streaming operations.

Method Summary

Modifier and Type	Method and Description
TLink<?,T>	allGather() Same as gather, but all the target TSet instances would receive the gathered result in the runtime.
TLink<?,T>	allReduce(ReduceFunc<T> reduceFn) Similar to reduce, but all instances of the target TSet would receive the reduced result.
TLink<?,T>	direct() Returns a Direct TLink that corresponds to the communication operation where the data will be transferred to another TSet directly.
TLink<?,T>	gather() Returns a Gather TLink that would gather data to the target TSet instance with index 0 (in the runtime).
<K,V> TupleTSet<K,V>	mapToTuple(MapFunc<Tuple<K,V>,T> mapToTupleFn) Creates a TupleTSet based on the MapFunc provided.
TLink<?,T>	partition(PartitionFunc<T> partitionFn) Same as above, but the parallelism will be preserved in the target TSet.
TLink<?,T>	partition(PartitionFunc<T> partitionFn, int targetParallelism) Returns a Partition TLink that would partition data according based on a function provided.
TLink<?,T>	reduce(ReduceFunc<T> reduceFn) Returns a Reduce TLink that reduce data on to the target TSet instance (in the runtime) with index 0.
TLink<?,T>	replicate(int replicas) Returns a Replicate TLink that would clone/broadcast the data from this TSet.
TSet<T>	setName(java.lang.String name) Sets the name
TSet<T>	union(java.util.Collection<TSet<T>> tSets) Same as above, but accepts a Collection of TSets.
TSet<T>	union(TSet<T> unionTSet) Returns a single TSet that create a union of data in both TSets.
TSet<T>	withSchema(Schema dataType) Sets the data type of the TSet output.

Methods inherited from interface edu.iu.dsc.tws.api.tset.TBase

getId, getName

Method Detail

setName

TSet<T> setName(java.lang.String name)

Sets the name

Specified by:

setName in interface TBase

direct

TLink<?,T> direct()

Returns a Direct TLink that corresponds to the communication operation where the data will be transferred to another TSet directly.

Returns:

Direct TLink

reduce

TLink<?,T> reduce(ReduceFunc<T> reduceFn)

Returns a Reduce TLink that reduce data on to the target TSet instance (in the runtime) with index 0.

Parameters:

reduceFn - Reduce function

Returns:

Reduce TLink

allReduce

TLink<?,T> allReduce(ReduceFunc<T> reduceFn)

Similar to reduce, but all instances of the target TSet would receive the reduced result.

Parameters:

reduceFn - Reduce function

Returns:

AllReduce TLink

partition

TLink<?,T> partition(PartitionFunc<T> partitionFn,
                     int targetParallelism)

Returns a Partition TLink that would partition data according based on a function provided. The parallelism of the target TSet can also be specified.

Parameters:

partitionFn - Partition function

targetParallelism - Target parallelism

Returns:

Partition TLink

partition

TLink<?,T> partition(PartitionFunc<T> partitionFn)

Same as above, but the parallelism will be preserved in the target TSet.

Parameters:

partitionFn - Partition function

Returns:

Partition TLink

gather

TLink<?,T> gather()

Returns a Gather TLink that would gather data to the target TSet instance with index 0 (in the runtime).

Returns:

Gather TLink

allGather

TLink<?,T> allGather()

Same as gather, but all the target TSet instances would receive the gathered result in the runtime.

Returns:

AllGather TLink

mapToTuple

<K,V> TupleTSet<K,V> mapToTuple(MapFunc<Tuple<K,V>,T> mapToTupleFn)

Creates a TupleTSet based on the MapFunc provided. This will an entry point to keyed communication operations from a non-keyed TSet.

Type Parameters:

K - type of key

V - type of value

Parameters:

mapToTupleFn - Map function

Returns:

Tuple TSet

replicate

TLink<?,T> replicate(int replicas)

Returns a Replicate TLink that would clone/broadcast the data from this TSet. Note that the parallelism of this TSet should be 1.

Parameters:

replicas - Replicas of data (= target TSet parallelism)

Returns:

Replicate TLink

union

TSet<T> union(TSet<T> unionTSet)

Returns a single TSet that create a union of data in both TSets. In order for this to work both TSets should be of the same type

Parameters:

unionTSet - TSet to union with

Returns:

Union TSet

union

TSet<T> union(java.util.Collection<TSet<T>> tSets)

Same as above, but accepts a Collection of TSets.

Parameters:

tSets - a collection of TSet's to union with

Returns:

Union TSet

withSchema

TSet<T> withSchema(Schema dataType)

Sets the data type of the TSet output. This will be used in the packers for efficient SER-DE operations in the following TLinks

Parameters:

dataType - data type as a MessageType

Returns:

this TSet