edu.iu.dsc.tws.tset.sets.batch

Class BatchTupleTSetImpl<K,V>

java.lang.Object

edu.iu.dsc.tws.tset.sets.BaseTSet<T>

edu.iu.dsc.tws.tset.sets.BaseTSetWithSchema<V>

edu.iu.dsc.tws.tset.sets.batch.BatchTupleTSetImpl<K,V>

Type Parameters:

K - key type

V - data (value) type

All Implemented Interfaces:

AcceptingData<Tuple<K,V>>, BatchTupleTSet<K,V>, TupleTSet<K,V>, StoringData<Tuple<K,V>>, TBase, Buildable, BuildableTSet, java.io.Serializable

Direct Known Subclasses:

KeyedSourceTSet, KeyedStoredTSet, KeyedTSet

public abstract class BatchTupleTSetImpl<K,V>
extends BaseTSetWithSchema<V>
implements BatchTupleTSet<K,V>

Attaches a key to the oncoming data.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class edu.iu.dsc.tws.tset.sets.BaseTSet

BaseTSet.StateType

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BatchTupleTSetImpl()

Method Summary

Modifier and Type	Method and Description
BatchTupleTSetImpl<K,V>	addInput(java.lang.String key, StorableTBase<?> input) Adds inputs to BatchTupleTSets
KeyedCachedTSet<K,V>	cache() Runs this TSet and caches the data to an in-memory DataPartition and exposes the data as another TSet.
TupleSchema	getOutputSchema() Output schema of a BatchTupleTSet is definitely a KeyedSchema.
BatchEnvironment	getTSetEnv() tset env
<VR> JoinTLink<K,V,VR>	join(BatchTupleTSet<K,VR> rightTSet, CommunicationContext.JoinType type, java.util.Comparator<K> keyComparator) Joins with another BatchTupleTSet.
<VR> JoinTLink<K,V,VR>	join(BatchTupleTSet<K,VR> rightTSet, CommunicationContext.JoinType type, java.util.Comparator<K> keyComparator, TaskPartitioner<K> partitioner) Joins with another BatchTupleTSet.
KeyedDirectTLink<K,V>	keyedDirect() Direct/pipe communication
KeyedGatherTLink<K,V>	keyedGather() Gathers data by key for BatchTupleTSets
KeyedGatherTLink<K,V>	keyedGather(PartitionFunc<K> partitionFn) Gathers data by key for BatchTupleTSets
KeyedGatherTLink<K,V>	keyedGather(PartitionFunc<K> partitionFn, java.util.Comparator<K> comparator) Gathers data by key for BatchTupleTSets
KeyedGatherUngroupedTLink<K,V>	keyedGatherUngrouped() Gathers data by key for BatchTupleTSets without grouping data by keys
KeyedGatherUngroupedTLink<K,V>	keyedGatherUngrouped(PartitionFunc<K> partitionFn) Gathers data by key for BatchTupleTSets without grouping data by keys
KeyedGatherUngroupedTLink<K,V>	keyedGatherUngrouped(PartitionFunc<K> partitionFn, java.util.Comparator<K> comparator) Gathers data by key for BatchTupleTSets without grouping data by keys
KeyedPartitionTLink<K,V>	keyedPartition(PartitionFunc<K> partitionFn) Partitions data using a PartitionFunc based on keys
KeyedPipeTLink<K,V>	keyedPipe() Pipe implementation
KeyedReduceTLink<K,V>	keyedReduce(ReduceFunc<V> reduceFn) Reduces data by key for BatchTupleTSets
KeyedCachedTSet<K,V>	lazyCache() Performs caching lazily.
KeyedPersistedTSet<K,V>	lazyPersist() Performs persisting lazily.
KeyedPersistedTSet<K,V>	persist() Similar to cache, but the data is stored in a disk based DataPartition.
BatchTupleTSetImpl<K,V>	setName(java.lang.String n) Name of the tset
BatchTupleTSetImpl<K,V>	withSchema(TupleSchema schema) Sets the data type of the TupleTSet output.

Methods inherited from class edu.iu.dsc.tws.tset.sets.BaseTSetWithSchema

getInputSchema, setOutputSchema

Methods inherited from class edu.iu.dsc.tws.tset.sets.BaseTSet

addChildToGraph, addChildToGraph, equals, getId, getInputs, getName, getParallelism, getStateType, hashCode, isMutable, rename, setMutable, setStateType, setTSetEnv, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface edu.iu.dsc.tws.api.tset.TBase

getId, getName

Methods inherited from interface edu.iu.dsc.tws.tset.sets.BuildableTSet

build, getINode

Methods inherited from interface edu.iu.dsc.tws.tset.Buildable

generateID, getTBaseGraph

Constructor Detail

BatchTupleTSetImpl

protected BatchTupleTSetImpl()

Method Detail

getTSetEnv

public BatchEnvironment getTSetEnv()

Description copied from interface: Buildable

tset env

Specified by:

getTSetEnv in interface Buildable

Overrides:

getTSetEnv in class BaseTSet<V>

Returns:

tset env

keyedDirect

public KeyedDirectTLink<K,V> keyedDirect()

Description copied from interface: BatchTupleTSet

Direct/pipe communication

Specified by:

keyedDirect in interface BatchTupleTSet<K,V>

Specified by:

keyedDirect in interface TupleTSet<K,V>

Returns:

Keyed Direct TLink

keyedPipe

public KeyedPipeTLink<K,V> keyedPipe()

Description copied from interface: BatchTupleTSet

Pipe implementation

Specified by:

keyedPipe in interface BatchTupleTSet<K,V>

Returns:

Keyed Pipe TLink

keyedReduce

public KeyedReduceTLink<K,V> keyedReduce(ReduceFunc<V> reduceFn)

Description copied from interface: BatchTupleTSet

Reduces data by key for BatchTupleTSets

Specified by:

keyedReduce in interface BatchTupleTSet<K,V>

Parameters:

reduceFn - the reduce function

Returns:

Keyed Reduce TLink

keyedPartition

public KeyedPartitionTLink<K,V> keyedPartition(PartitionFunc<K> partitionFn)

Description copied from interface: BatchTupleTSet

Partitions data using a PartitionFunc based on keys

Specified by:

keyedPartition in interface BatchTupleTSet<K,V>

Specified by:

keyedPartition in interface TupleTSet<K,V>

Parameters:

partitionFn - partition function

Returns:

Keyed Partition TLink

keyedGather

public KeyedGatherTLink<K,V> keyedGather()

Description copied from interface: BatchTupleTSet

Gathers data by key for BatchTupleTSets

Specified by:

keyedGather in interface BatchTupleTSet<K,V>

Returns:

Keyed Gather TLink

keyedGather

public KeyedGatherTLink<K,V> keyedGather(PartitionFunc<K> partitionFn)

Description copied from interface: BatchTupleTSet

Gathers data by key for BatchTupleTSets

Specified by:

keyedGather in interface BatchTupleTSet<K,V>

Parameters:

partitionFn - partition function to partition data based on key

Returns:

Keyed Gather TLink

keyedGather

public KeyedGatherTLink<K,V> keyedGather(PartitionFunc<K> partitionFn,
                                         java.util.Comparator<K> comparator)

Description copied from interface: BatchTupleTSet

Gathers data by key for BatchTupleTSets

Specified by:

keyedGather in interface BatchTupleTSet<K,V>

Parameters:

partitionFn - partition function to partition data based on key

comparator - custom key comparator

Returns:

Keyed Gather TLink

keyedGatherUngrouped

public KeyedGatherUngroupedTLink<K,V> keyedGatherUngrouped()

Description copied from interface: BatchTupleTSet

Gathers data by key for BatchTupleTSets without grouping data by keys

Specified by:

keyedGatherUngrouped in interface BatchTupleTSet<K,V>

Returns:

this TSet

keyedGatherUngrouped

public KeyedGatherUngroupedTLink<K,V> keyedGatherUngrouped(PartitionFunc<K> partitionFn)

Description copied from interface: BatchTupleTSet

Gathers data by key for BatchTupleTSets without grouping data by keys

Specified by:

keyedGatherUngrouped in interface BatchTupleTSet<K,V>

Parameters:

partitionFn - partition function to partition data based on key

Returns:

Keyed Gather TLink

keyedGatherUngrouped

public KeyedGatherUngroupedTLink<K,V> keyedGatherUngrouped(PartitionFunc<K> partitionFn,
                                                           java.util.Comparator<K> comparator)

Description copied from interface: BatchTupleTSet

Gathers data by key for BatchTupleTSets without grouping data by keys

Specified by:

keyedGatherUngrouped in interface BatchTupleTSet<K,V>

Parameters:

partitionFn - partition function to partition data based on key

comparator - custom key comparator

Returns:

Keyed Gather TLink

join

public <VR> JoinTLink<K,V,VR> join(BatchTupleTSet<K,VR> rightTSet,
                                   CommunicationContext.JoinType type,
                                   java.util.Comparator<K> keyComparator,
                                   TaskPartitioner<K> partitioner)

Description copied from interface: BatchTupleTSet

Joins with another BatchTupleTSet. Note that this TSet will be considered the left TSet

Specified by:

join in interface BatchTupleTSet<K,V>

Type Parameters:

VR - value type of the right tset

Parameters:

rightTSet - right tset

type - CommunicationContext.JoinType

keyComparator - key comparator

partitioner - partitioner for keys

Returns:

Joined TLink

join

public <VR> JoinTLink<K,V,VR> join(BatchTupleTSet<K,VR> rightTSet,
                                   CommunicationContext.JoinType type,
                                   java.util.Comparator<K> keyComparator)

Description copied from interface: BatchTupleTSet

Joins with another BatchTupleTSet. Note that this TSet will be considered the left TSet

Specified by:

join in interface BatchTupleTSet<K,V>

Type Parameters:

VR - value type of the right tset

Parameters:

rightTSet - right tset

type - CommunicationContext.JoinType

keyComparator - key comparator

Returns:

Joined TLink

setName

public BatchTupleTSetImpl<K,V> setName(java.lang.String n)

Description copied from interface: BatchTupleTSet

Name of the tset

Specified by:

setName in interface BatchTupleTSet<K,V>

Specified by:

setName in interface TupleTSet<K,V>

Specified by:

setName in interface TBase

Parameters:

n - name

Returns:

this set

cache

public KeyedCachedTSet<K,V> cache()

Description copied from interface: StoringData

Runs this TSet and caches the data to an in-memory DataPartition and exposes the data as another TSet.

Specified by:

cache in interface StoringData<Tuple<K,V>>

Returns:

Cached TSet

lazyCache

public KeyedCachedTSet<K,V> lazyCache()

Description copied from interface: StoringData

Performs caching lazily. i.e. cache operation would only be performed when the TSet is evaluated explicitly.

Specified by:

lazyCache in interface StoringData<Tuple<K,V>>

Returns:

Cached TSet

persist

public KeyedPersistedTSet<K,V> persist()

Description copied from interface: StoringData

Similar to cache, but the data is stored in a disk based DataPartition. This method would also expose the checkpointing ability to TSets.

Specified by:

persist in interface StoringData<Tuple<K,V>>

Returns:

Persisted / Checkpointed TSets

lazyPersist

public KeyedPersistedTSet<K,V> lazyPersist()

Description copied from interface: StoringData

Performs persisting lazily.

Specified by:

lazyPersist in interface StoringData<Tuple<K,V>>

Returns:

Persisted / Checkpointed TSets

addInput

public BatchTupleTSetImpl<K,V> addInput(java.lang.String key,
                                        StorableTBase<?> input)

Description copied from interface: BatchTupleTSet

Adds inputs to BatchTupleTSets

Specified by:

addInput in interface AcceptingData<Tuple<K,V>>

Specified by:

addInput in interface BatchTupleTSet<K,V>

Parameters:

key - identifier for the input

input - input tset

Returns:

this tset

withSchema

public BatchTupleTSetImpl<K,V> withSchema(TupleSchema schema)

Description copied from interface: TupleTSet

Sets the data type of the TupleTSet output. This will be used in the packers for efficient SER-DE operations in the following TLinks

Specified by:

withSchema in interface TupleTSet<K,V>

Parameters:

schema - data type as a MessageType

Returns:

this TupleTSet

getOutputSchema

public TupleSchema getOutputSchema()

Output schema of a BatchTupleTSet is definitely a KeyedSchema.

Overrides:

getOutputSchema in class BaseTSetWithSchema<V>

Returns:

keyed schema