edu.iu.dsc.tws.tset.sets.batch

Class PersistedTSet<T>

java.lang.Object

edu.iu.dsc.tws.tset.sets.BaseTSet<T>

edu.iu.dsc.tws.tset.sets.BaseTSetWithSchema<T>

edu.iu.dsc.tws.tset.sets.batch.BatchTSetImpl<T>

edu.iu.dsc.tws.tset.sets.batch.StoredTSet<T>

edu.iu.dsc.tws.tset.sets.batch.PersistedTSet<T>

All Implemented Interfaces:

AcceptingData<T>, BatchTSet<T>, StorableTBase<T>, TSet<T>, StoringData<T>, TBase, Buildable, BuildableTSet, java.io.Serializable

Direct Known Subclasses:

CheckpointedTSet

public class PersistedTSet<T>
extends StoredTSet<T>

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class edu.iu.dsc.tws.tset.sets.BaseTSet

BaseTSet.StateType

Field Summary

Fields inherited from class edu.iu.dsc.tws.tset.sets.batch.StoredTSet

storedSource

Constructor Summary

Constructors

Constructor and Description
PersistedTSet(BatchEnvironment tSetEnv, SinkFunc<?> sinkFunc, int parallelism, Schema inputSchema)

Method Summary

Modifier and Type	Method and Description
PersistedTSet<T>	addInput(java.lang.String key, StorableTBase<?> input) Adds data to this TSet
CachedTSet<T>	cache() Runs this TSet and caches the data to an in-memory DataPartition and exposes the data as another TSet.
CachedTSet<T>	lazyCache() Performs caching lazily.
PersistedTSet<T>	lazyPersist() Performs persisting lazily.
PersistedTSet<T>	persist() Similar to cache, but the data is stored in a disk based DataPartition.
PersistedTSet<T>	setName(java.lang.String n) Sets the name for the TBase
PersistedTSet<T>	withSchema(Schema schema) Sets the data type of the TSet output.

Methods inherited from class edu.iu.dsc.tws.tset.sets.batch.StoredTSet

allGather, allReduce, direct, gather, getData, getDataObject, getINode, getStoredSourceTSet, mapToTuple, partition, partition, pipe, reduce, replicate, union, union

Methods inherited from class edu.iu.dsc.tws.tset.sets.batch.BatchTSetImpl

getTSetEnv

Methods inherited from class edu.iu.dsc.tws.tset.sets.BaseTSetWithSchema

getInputSchema, getOutputSchema, setOutputSchema

Methods inherited from class edu.iu.dsc.tws.tset.sets.BaseTSet

addChildToGraph, addChildToGraph, equals, getId, getInputs, getName, getParallelism, getStateType, hashCode, isMutable, rename, setMutable, setStateType, setTSetEnv, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface edu.iu.dsc.tws.api.tset.TBase

getId, getName

Methods inherited from interface edu.iu.dsc.tws.tset.sets.BuildableTSet

build

Methods inherited from interface edu.iu.dsc.tws.tset.Buildable

generateID, getTBaseGraph

Constructor Detail

PersistedTSet

public PersistedTSet(BatchEnvironment tSetEnv,
                     SinkFunc<?> sinkFunc,
                     int parallelism,
                     Schema inputSchema)

Method Detail

persist

public PersistedTSet<T> persist()

Description copied from interface: StoringData

Similar to cache, but the data is stored in a disk based DataPartition. This method would also expose the checkpointing ability to TSets.

Specified by:

persist in interface StoringData<T>

Overrides:

persist in class BatchTSetImpl<T>

Returns:

Persisted / Checkpointed TSets

lazyPersist

public PersistedTSet<T> lazyPersist()

Description copied from interface: StoringData

Performs persisting lazily.

Specified by:

lazyPersist in interface StoringData<T>

Overrides:

lazyPersist in class BatchTSetImpl<T>

Returns:

Persisted / Checkpointed TSets

cache

public CachedTSet<T> cache()

Description copied from interface: StoringData

Runs this TSet and caches the data to an in-memory DataPartition and exposes the data as another TSet.

Specified by:

cache in interface StoringData<T>

Overrides:

cache in class BatchTSetImpl<T>

Returns:

Cached TSet

lazyCache

public CachedTSet<T> lazyCache()

Description copied from interface: StoringData

Performs caching lazily. i.e. cache operation would only be performed when the TSet is evaluated explicitly.

Specified by:

lazyCache in interface StoringData<T>

Overrides:

lazyCache in class BatchTSetImpl<T>

Returns:

Cached TSet

setName

public PersistedTSet<T> setName(java.lang.String n)

Description copied from interface: TBase

Sets the name for the TBase

addInput

public PersistedTSet<T> addInput(java.lang.String key,
                                 StorableTBase<?> input)

Description copied from interface: BatchTSet

Adds data to this TSet

Specified by:

addInput in interface AcceptingData<T>

Specified by:

addInput in interface BatchTSet<T>

Overrides:

addInput in class StoredTSet<T>

Parameters:

key - the key used to store the given TSet

input - a @StorableTBase TSet to be added as an input

Returns:

this test

withSchema

public PersistedTSet<T> withSchema(Schema schema)

Description copied from interface: TSet

Sets the data type of the TSet output. This will be used in the packers for efficient SER-DE operations in the following TLinks

Specified by:

withSchema in interface TSet<T>

Overrides:

withSchema in class BatchTSetImpl<T>

Parameters:

schema - data type as a MessageType

Returns:

this TSet