edu.iu.dsc.tws.data.api.splits

Class FileInputSplit<OT>

java.lang.Object

edu.iu.dsc.tws.data.api.splits.LocatableInputSplit<OT>

edu.iu.dsc.tws.data.api.splits.FileInputSplit<OT>

All Implemented Interfaces:

InputSplit<OT>, java.io.Serializable

Direct Known Subclasses:

ArrowInputSplit, BinaryInputSplit, CSVInputSplit, DelimitedInputSplit

public abstract class FileInputSplit<OT>
extends LocatableInputSplit<OT>

A file input split provides information on a particular part of a file, possibly hosted on a distributed file system and replicated among several hosts.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileInputSplit.InputSplitOpenThread Obtains a DataInputStream in an thread that is not interrupted.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	enumerateNestedFiles The flag to specify whether recursive traversal of the input directory structure is enabled.
protected static float	MAX_SPLIT_SIZE_DISCREPANCY The fraction that the last split may be larger than the others.
protected long	minSplitSize The minimal split size, set by the configure() method.
protected int	numSplits The desired number of splits, as set by the configure() method.
protected long	openTimeout Time to wait when opening a file.
protected long	splitLength Length of the current split
protected long	splitStart Start point of the current split
protected FSDataInputStream	stream The input data stream

Constructor Summary

Constructors

Constructor and Description
FileInputSplit(int num, Path file, long start, long length, java.lang.String[] hosts) Constructs a split with host information.
FileInputSplit(int num, Path file, java.lang.String[] hosts)

Method Summary

Modifier and Type	Method and Description
void	close() Method that marks the end of the life-cycle of an input split.
void	configure(Config parameters) Configure the split with parameters
boolean	equals(java.lang.Object obj)
long	getLength() Returns the number of bytes in the file to process.
long	getMinSplitSize()
int	getNumSplits()
Path	getPath() Returns the path of the file containing this split's data.
long	getStart() Returns the position of the first byte in the file to process.
int	hashCode()
boolean	isEnumerateNestedFiles()
void	open() Opens an input stream to the file defined in the input format.
void	open(Config cfg)
void	setEnumerateNestedFiles(boolean enumerateNestedFiles)
void	setMinSplitSize(long minSplitSize)
void	setNumSplits(int numSplits)
java.lang.String	toString()

Methods inherited from class edu.iu.dsc.tws.data.api.splits.LocatableInputSplit

getHostnames, getSplitNumber

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface edu.iu.dsc.tws.data.fs.io.InputSplit

nextRecord, reachedEnd

Field Detail

numSplits

protected int numSplits

The desired number of splits, as set by the configure() method.

enumerateNestedFiles

protected boolean enumerateNestedFiles

The flag to specify whether recursive traversal of the input directory structure is enabled.

MAX_SPLIT_SIZE_DISCREPANCY

protected static final float MAX_SPLIT_SIZE_DISCREPANCY

The fraction that the last split may be larger than the others.

See Also:

Constant Field Values

minSplitSize

protected long minSplitSize

The minimal split size, set by the configure() method.

splitStart

protected long splitStart

Start point of the current split

splitLength

protected long splitLength

Length of the current split

stream

protected FSDataInputStream stream

The input data stream

openTimeout

protected long openTimeout

Time to wait when opening a file.

Constructor Detail

FileInputSplit

public FileInputSplit(int num,
                      Path file,
                      long start,
                      long length,
                      java.lang.String[] hosts)

Constructs a split with host information.

Parameters:

num - the number of this input split

file - the file name

start - the position of the first byte in the file to process

length - the number of bytes in the file to process (-1 is flag for "read whole file")

hosts - the list of hosts containing the block, possibly null

FileInputSplit

public FileInputSplit(int num,
                      Path file,
                      java.lang.String[] hosts)

Method Detail

isEnumerateNestedFiles

public boolean isEnumerateNestedFiles()

setEnumerateNestedFiles

public void setEnumerateNestedFiles(boolean enumerateNestedFiles)

getMinSplitSize

public long getMinSplitSize()

setMinSplitSize

public void setMinSplitSize(long minSplitSize)

getNumSplits

public int getNumSplits()

setNumSplits

public void setNumSplits(int numSplits)

getPath

public Path getPath()

Returns the path of the file containing this split's data.

Returns:

the path of the file containing this split's data.

getStart

public long getStart()

Returns the position of the first byte in the file to process.

Returns:

the position of the first byte in the file to process

getLength

public long getLength()

Returns the number of bytes in the file to process.

Returns:

the number of bytes in the file to process

hashCode

public int hashCode()

Overrides:

hashCode in class LocatableInputSplit<OT>

equals

public boolean equals(java.lang.Object obj)

Overrides:

equals in class LocatableInputSplit<OT>

toString

public java.lang.String toString()

Overrides:

toString in class LocatableInputSplit<OT>

configure

public void configure(Config parameters)

Description copied from interface: InputSplit

Configure the split with parameters

Parameters:

parameters - the parameters

close

public void close()
           throws java.io.IOException

Description copied from interface: InputSplit

Method that marks the end of the life-cycle of an input split. Should be used to close channels and streams and release resources. After this method returns without an error, the input is assumed to be correctly read.

When this method is called, the input format it guaranteed to be opened.

Throws:

java.io.IOException - Thrown, if the input could not be closed properly.

open

public void open()
          throws java.io.IOException

Opens an input stream to the file defined in the input format. The stream is positioned at the beginning of the given split.

The stream is actually opened in an asynchronous thread to make sure any interruptions to the thread working on the input format do not reach the file system.

Throws:

java.io.IOException - Thrown, if the spit could not be opened due to an I/O problem.

open

public void open(Config cfg)
          throws java.io.IOException

Throws:

java.io.IOException