edu.iu.dsc.tws.dataset.partition

Class HDFSBackedCollectionPartition<T>

java.lang.Object

edu.iu.dsc.tws.dataset.partition.CollectionPartition<T>

edu.iu.dsc.tws.dataset.partition.BufferedCollectionPartition<T>

edu.iu.dsc.tws.dataset.partition.HDFSBackedCollectionPartition<T>

All Implemented Interfaces:

DataPartition<T>, java.io.Closeable, java.io.Serializable, java.lang.AutoCloseable

public class HDFSBackedCollectionPartition<T>
extends BufferedCollectionPartition<T>

See Also:

Serialized Form

Field Summary

Fields inherited from class edu.iu.dsc.tws.dataset.partition.CollectionPartition

dataList

Fields inherited from interface edu.iu.dsc.tws.api.dataset.DataPartition

DEFAULT_REF

Constructor Summary

Constructors

Constructor and Description
HDFSBackedCollectionPartition(int maxFramesInMemory, Config config)
HDFSBackedCollectionPartition(int maxFramesInMemory, MessageType dataType, Config config)
HDFSBackedCollectionPartition(int maxFramesInMemory, MessageType dataType, int bufferedBytes, Config config, java.lang.String reference)
HDFSBackedCollectionPartition(MessageType dataType, int bufferedBytes, Config config)
HDFSBackedCollectionPartition(MessageType dataType, int bufferedBytes, Config config, java.lang.String reference)

Method Summary

Modifier and Type	Method and Description
protected FileSystem	getFileSystem(Config config)
protected Path	getRootPath(Config config)
protected java.lang.String	getRootPathStr(Config config)

Methods inherited from class edu.iu.dsc.tws.dataset.partition.BufferedCollectionPartition

add, addAll, clear, close, dispose, flush, get, getConsumer, getReference, hasIndexInMemory

Methods inherited from class edu.iu.dsc.tws.dataset.partition.CollectionPartition

getPartitionId, setId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.iu.dsc.tws.api.dataset.DataPartition

first, firstOrDefault

Constructor Detail

HDFSBackedCollectionPartition

public HDFSBackedCollectionPartition(int maxFramesInMemory,
                                     MessageType dataType,
                                     int bufferedBytes,
                                     Config config,
                                     java.lang.String reference)

HDFSBackedCollectionPartition

public HDFSBackedCollectionPartition(int maxFramesInMemory,
                                     Config config)

HDFSBackedCollectionPartition

public HDFSBackedCollectionPartition(int maxFramesInMemory,
                                     MessageType dataType,
                                     Config config)

HDFSBackedCollectionPartition

public HDFSBackedCollectionPartition(MessageType dataType,
                                     int bufferedBytes,
                                     Config config)

HDFSBackedCollectionPartition

public HDFSBackedCollectionPartition(MessageType dataType,
                                     int bufferedBytes,
                                     Config config,
                                     java.lang.String reference)

Method Detail

getRootPathStr

protected java.lang.String getRootPathStr(Config config)

getFileSystem

protected FileSystem getFileSystem(Config config)

Specified by:

getFileSystem in class BufferedCollectionPartition<T>

getRootPath

protected Path getRootPath(Config config)

Specified by:

getRootPath in class BufferedCollectionPartition<T>