edu.iu.dsc.tws.tset.sets.batch

Class KeyedPersistedTSet<K,V>

java.lang.Object

edu.iu.dsc.tws.tset.sets.BaseTSet<T>

edu.iu.dsc.tws.tset.sets.BaseTSetWithSchema<V>

edu.iu.dsc.tws.tset.sets.batch.BatchTupleTSetImpl<K,V>

edu.iu.dsc.tws.tset.sets.batch.KeyedStoredTSet<K,V>

edu.iu.dsc.tws.tset.sets.batch.KeyedPersistedTSet<K,V>

All Implemented Interfaces:

AcceptingData<Tuple<K,V>>, BatchTupleTSet<K,V>, StorableTBase<Tuple<K,V>>, TupleTSet<K,V>, StoringData<Tuple<K,V>>, TBase, Buildable, BuildableTSet, java.io.Serializable

Direct Known Subclasses:

KeyedCheckpointedTSet

public class KeyedPersistedTSet<K,V>
extends KeyedStoredTSet<K,V>

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class edu.iu.dsc.tws.tset.sets.BaseTSet

BaseTSet.StateType

Field Summary

Fields inherited from class edu.iu.dsc.tws.tset.sets.batch.KeyedStoredTSet

storedSource

Constructor Summary

Constructors

Constructor and Description
KeyedPersistedTSet(BatchEnvironment tSetEnv, SinkFunc<java.util.Iterator<Tuple<K,V>>> storingSinkFn, int parallelism, KeyedSchema inputSchema)

Method Summary

Modifier and Type	Method and Description
KeyedPersistedTSet<K,V>	addInput(java.lang.String key, StorableTBase<?> input) Adds inputs to BatchTupleTSets
KeyedCachedTSet<K,V>	cache() Runs this TSet and caches the data to an in-memory DataPartition and exposes the data as another TSet.
KeyedCachedTSet<K,V>	lazyCache() Performs caching lazily.
KeyedPersistedTSet<K,V>	lazyPersist() Performs persisting lazily.
KeyedPersistedTSet<K,V>	persist() Similar to cache, but the data is stored in a disk based DataPartition.
KeyedPersistedTSet<K,V>	setName(java.lang.String n) Name of the tset
KeyedPersistedTSet<K,V>	withSchema(TupleSchema schema) Sets the data type of the TupleTSet output.

Methods inherited from class edu.iu.dsc.tws.tset.sets.batch.KeyedStoredTSet

getData, getINode, getInputSchema, getStoredSourceTSet, join, join, keyedDirect, keyedGather, keyedGather, keyedGather, keyedGatherUngrouped, keyedGatherUngrouped, keyedGatherUngrouped, keyedPartition, keyedPipe, keyedReduce

Methods inherited from class edu.iu.dsc.tws.tset.sets.batch.BatchTupleTSetImpl

getOutputSchema, getTSetEnv

Methods inherited from class edu.iu.dsc.tws.tset.sets.BaseTSetWithSchema

setOutputSchema

Methods inherited from class edu.iu.dsc.tws.tset.sets.BaseTSet

addChildToGraph, addChildToGraph, equals, getId, getInputs, getName, getParallelism, getStateType, hashCode, isMutable, rename, setMutable, setStateType, setTSetEnv, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface edu.iu.dsc.tws.api.tset.sets.StorableTBase

getDataObject

Methods inherited from interface edu.iu.dsc.tws.api.tset.TBase

getId, getName

Methods inherited from interface edu.iu.dsc.tws.tset.sets.BuildableTSet

build

Methods inherited from interface edu.iu.dsc.tws.tset.Buildable

generateID, getTBaseGraph

Constructor Detail

KeyedPersistedTSet

public KeyedPersistedTSet(BatchEnvironment tSetEnv,
                          SinkFunc<java.util.Iterator<Tuple<K,V>>> storingSinkFn,
                          int parallelism,
                          KeyedSchema inputSchema)

Method Detail

setName

public KeyedPersistedTSet<K,V> setName(java.lang.String n)

Description copied from interface: BatchTupleTSet

Name of the tset

Specified by:

setName in interface BatchTupleTSet<K,V>

Specified by:

setName in interface TupleTSet<K,V>

Specified by:

setName in interface TBase

Overrides:

setName in class BatchTupleTSetImpl<K,V>

Parameters:

n - name

Returns:

this set

addInput

public KeyedPersistedTSet<K,V> addInput(java.lang.String key,
                                        StorableTBase<?> input)

Description copied from interface: BatchTupleTSet

Adds inputs to BatchTupleTSets

Specified by:

addInput in interface AcceptingData<Tuple<K,V>>

Specified by:

addInput in interface BatchTupleTSet<K,V>

Overrides:

addInput in class BatchTupleTSetImpl<K,V>

Parameters:

key - identifier for the input

input - input tset

Returns:

this tset

withSchema

public KeyedPersistedTSet<K,V> withSchema(TupleSchema schema)

Description copied from interface: TupleTSet

Sets the data type of the TupleTSet output. This will be used in the packers for efficient SER-DE operations in the following TLinks

Specified by:

withSchema in interface TupleTSet<K,V>

Overrides:

withSchema in class BatchTupleTSetImpl<K,V>

Parameters:

schema - data type as a MessageType

Returns:

this TupleTSet

cache

public KeyedCachedTSet<K,V> cache()

Description copied from interface: StoringData

Runs this TSet and caches the data to an in-memory DataPartition and exposes the data as another TSet.

Specified by:

cache in interface StoringData<Tuple<K,V>>

Overrides:

cache in class BatchTupleTSetImpl<K,V>

Returns:

Cached TSet

lazyCache

public KeyedCachedTSet<K,V> lazyCache()

Description copied from interface: StoringData

Performs caching lazily. i.e. cache operation would only be performed when the TSet is evaluated explicitly.

Specified by:

lazyCache in interface StoringData<Tuple<K,V>>

Overrides:

lazyCache in class BatchTupleTSetImpl<K,V>

Returns:

Cached TSet

persist

public KeyedPersistedTSet<K,V> persist()

Description copied from interface: StoringData

Similar to cache, but the data is stored in a disk based DataPartition. This method would also expose the checkpointing ability to TSets.

Specified by:

persist in interface StoringData<Tuple<K,V>>

Overrides:

persist in class BatchTupleTSetImpl<K,V>

Returns:

Persisted / Checkpointed TSets

lazyPersist

public KeyedPersistedTSet<K,V> lazyPersist()

Description copied from interface: StoringData

Performs persisting lazily.

Specified by:

lazyPersist in interface StoringData<Tuple<K,V>>

Overrides:

lazyPersist in class BatchTupleTSetImpl<K,V>

Returns:

Persisted / Checkpointed TSets