edu.iu.dsc.tws.dataset.partition

Class BufferedCollectionPartition<T>

java.lang.Object

edu.iu.dsc.tws.dataset.partition.CollectionPartition<T>

edu.iu.dsc.tws.dataset.partition.BufferedCollectionPartition<T>

All Implemented Interfaces:

DataPartition<T>, java.io.Closeable, java.io.Serializable, java.lang.AutoCloseable

Direct Known Subclasses:

DiskBackedCollectionPartition, HDFSBackedCollectionPartition

public abstract class BufferedCollectionPartition<T>
extends CollectionPartition<T>
implements java.io.Closeable

See Also:

Serialized Form

Field Summary

Fields inherited from class edu.iu.dsc.tws.dataset.partition.CollectionPartition

dataList

Fields inherited from interface edu.iu.dsc.tws.api.dataset.DataPartition

DEFAULT_REF

Constructor Summary

Constructors

Constructor and Description
BufferedCollectionPartition(long maxFramesInMemory, Config config) Create a disk based partition with default buffer size(10MB) and without a MessageType If MessageType is not defined, it will assume as ObjectType.
BufferedCollectionPartition(long maxFramesInMemory, Config config, java.lang.String reference)
BufferedCollectionPartition(long maxFramesInMemory, MessageType dataType, Config config) Creates an instance with default bugger size 10MB
BufferedCollectionPartition(long maxFramesInMemory, MessageType dataType, long bufferedBytes, Config config, java.lang.String reference) Creates an instance of BufferedCollectionPartition
BufferedCollectionPartition(MessageType dataType, long bufferedBytes, Config config) Creates an 100% disk based instance.
BufferedCollectionPartition(MessageType dataType, long bufferedBytes, Config config, java.lang.String reference) Loads an disk based partition from file system.

Method Summary

Modifier and Type	Method and Description
void	add(T val)
void	addAll(java.util.Collection<T> frames)
void	clear()
void	close()
void	dispose() This method will clear the memory components of this partition by assigning buffer to null and making it garbage collectible.
void	flush()
T	get(int index)
DataPartitionConsumer<T>	getConsumer() Get the data consumer
protected abstract FileSystem	getFileSystem(Config config)
java.lang.String	getReference() Reference can be used to load a persisted partition from disk of hdfs
protected abstract Path	getRootPath(Config config)
boolean	hasIndexInMemory(int index)

Methods inherited from class edu.iu.dsc.tws.dataset.partition.CollectionPartition

getPartitionId, setId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.iu.dsc.tws.api.dataset.DataPartition

first, firstOrDefault

Constructor Detail

BufferedCollectionPartition

public BufferedCollectionPartition(long maxFramesInMemory,
                                   MessageType dataType,
                                   long bufferedBytes,
                                   Config config,
                                   java.lang.String reference)

Creates an instance of BufferedCollectionPartition

Parameters:

maxFramesInMemory - No of frames(elements) to keep in memory

dataType - MessageType of data frames

bufferedBytes - amount to buffer in memory before writing to the disk

BufferedCollectionPartition

public BufferedCollectionPartition(long maxFramesInMemory,
                                   Config config)

Create a disk based partition with default buffer size(10MB) and without a MessageType If MessageType is not defined, it will assume as ObjectType. This will make serialization and deserialization inefficient.

Parameters:

maxFramesInMemory - No of frames(elements) to keep in memory

BufferedCollectionPartition

public BufferedCollectionPartition(long maxFramesInMemory,
                                   Config config,
                                   java.lang.String reference)

BufferedCollectionPartition

public BufferedCollectionPartition(long maxFramesInMemory,
                                   MessageType dataType,
                                   Config config)

Creates an instance with default bugger size 10MB

Parameters:

maxFramesInMemory - No of frames(elements) to keep in memory

dataType - MessageType of data frames

BufferedCollectionPartition

public BufferedCollectionPartition(MessageType dataType,
                                   long bufferedBytes,
                                   Config config)

Creates an 100% disk based instance. This mode will not keep any data frame in memory

Parameters:

dataType - MessageType of data frames

bufferedBytes - amount to buffer in memory before writing to the disk

BufferedCollectionPartition

public BufferedCollectionPartition(MessageType dataType,
                                   long bufferedBytes,
                                   Config config,
                                   java.lang.String reference)

Loads an disk based partition from file system. This mode will not keep any data frame in memory.

Parameters:

dataType - MessageType of data frames

bufferedBytes - amount to buffer in memory before writing to the disk

reference - Partition reference

Method Detail

getFileSystem

protected abstract FileSystem getFileSystem(Config config)
                                     throws java.io.IOException

Throws:

java.io.IOException

getRootPath

protected abstract Path getRootPath(Config config)

add

public void add(T val)

Overrides:

add in class CollectionPartition<T>

addAll

public void addAll(java.util.Collection<T> frames)

Overrides:

addAll in class CollectionPartition<T>

getConsumer

public DataPartitionConsumer<T> getConsumer()

Description copied from interface: DataPartition

Get the data consumer

Specified by:

getConsumer in interface DataPartition<T>

Overrides:

getConsumer in class CollectionPartition<T>

Returns:

the consumer

dispose

public void dispose()

This method will clear the memory components of this partition by assigning buffer to null and making it garbage collectible. This partition shouldn't be used after disposing.

clear

public void clear()

Specified by:

clear in interface DataPartition<T>

Overrides:

clear in class CollectionPartition<T>

flush

public void flush()

hasIndexInMemory

public boolean hasIndexInMemory(int index)

get

public T get(int index)

close

public void close()

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

getReference

public java.lang.String getReference()

Description copied from interface: DataPartition

Reference can be used to load a persisted partition from disk of hdfs

Specified by:

getReference in interface DataPartition<T>