edu.iu.dsc.tws.data.api.splits

Class CSVInputSplit

java.lang.Object

edu.iu.dsc.tws.data.api.splits.LocatableInputSplit<OT>

edu.iu.dsc.tws.data.api.splits.FileInputSplit<java.lang.Object>

edu.iu.dsc.tws.data.api.splits.CSVInputSplit

All Implemented Interfaces:

InputSplit<java.lang.Object>, java.io.Serializable

public class CSVInputSplit
extends FileInputSplit<java.lang.Object>

This class is primarily responsible for reading the split of a CSV file. It receive the input as start and length of the file to be read for each split.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class edu.iu.dsc.tws.data.api.splits.FileInputSplit

FileInputSplit.InputSplitOpenThread

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_FIELD_DELIMITER
static java.lang.String	DEFAULT_LINE_DELIMITER
static java.lang.String	DEFAULT_TAB_DELIMITER
protected static java.lang.String	RECORD_DELIMITER The configuration key to set the record delimiter.

Fields inherited from class edu.iu.dsc.tws.data.api.splits.FileInputSplit

enumerateNestedFiles, MAX_SPLIT_SIZE_DISCREPANCY, minSplitSize, numSplits, openTimeout, splitLength, splitStart, stream

Constructor Summary

Constructors

Constructor and Description
CSVInputSplit(int num, Path file, long start, long length, java.lang.String[] hosts) Constructs a split with host information.

Method Summary

Modifier and Type	Method and Description
void	close() Closes the input by releasing all buffers and closing the file input stream.
void	configure(Config parameters) Configures this input format by reading the path to the file from the configuration and the string that defines the record delimiter.
int	getBufferSize()
java.nio.charset.Charset	getCharset() Get the character set used for the row delimiter.
java.lang.String	getCharsetName()
byte[]	getDelimiter()
java.lang.String	getDelimiterString()
int	getLineLengthLimit()
java.lang.String	nextRecord(java.lang.Object record) Reads the next record from the input.
void	open(Config cfg) Opens the given input split.
boolean	reachedEnd() Checks whether the current split is at its end.
protected boolean	readLine()
java.lang.String	readRecord(byte[] bytes, int readOffset, int numBytes)
void	setBufferSize(int bufferSize)
void	setCharsetName(java.lang.String charsetName)
void	setDelimiter(byte[] delimiter)
void	setDelimiterString(java.lang.String delimiterString)
void	setLineLengthLimit(int lineLengthLimit)

Methods inherited from class edu.iu.dsc.tws.data.api.splits.FileInputSplit

equals, getLength, getMinSplitSize, getNumSplits, getPath, getStart, hashCode, isEnumerateNestedFiles, open, setEnumerateNestedFiles, setMinSplitSize, setNumSplits, toString

Methods inherited from class edu.iu.dsc.tws.data.api.splits.LocatableInputSplit

getHostnames, getSplitNumber

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_LINE_DELIMITER

public static final java.lang.String DEFAULT_LINE_DELIMITER

See Also:

Constant Field Values

DEFAULT_FIELD_DELIMITER

public static final java.lang.String DEFAULT_FIELD_DELIMITER

See Also:

Constant Field Values

DEFAULT_TAB_DELIMITER

public static final java.lang.String DEFAULT_TAB_DELIMITER

See Also:

Constant Field Values

RECORD_DELIMITER

protected static final java.lang.String RECORD_DELIMITER

The configuration key to set the record delimiter.

See Also:

Constant Field Values

Constructor Detail

CSVInputSplit

public CSVInputSplit(int num,
                     Path file,
                     long start,
                     long length,
                     java.lang.String[] hosts)

Constructs a split with host information.

Parameters:

num - the number of this input split

file - the file name

start - the position of the first byte in the file to process

length - the number of bytes in the file to process (-1 is flag for "read whole file")

hosts - the list of hosts containing the block, possibly null

Method Detail

getDelimiter

public byte[] getDelimiter()

setDelimiter

public void setDelimiter(byte[] delimiter)

getDelimiterString

public java.lang.String getDelimiterString()

setDelimiterString

public void setDelimiterString(java.lang.String delimiterString)

getBufferSize

public int getBufferSize()

setBufferSize

public void setBufferSize(int bufferSize)

getCharsetName

public java.lang.String getCharsetName()

setCharsetName

public void setCharsetName(java.lang.String charsetName)

getLineLengthLimit

public int getLineLengthLimit()

setLineLengthLimit

public void setLineLengthLimit(int lineLengthLimit)

getCharset

public java.nio.charset.Charset getCharset()

Get the character set used for the row delimiter.

Returns:

the charset

configure

public void configure(Config parameters)

Configures this input format by reading the path to the file from the configuration and the string that defines the record delimiter.

Specified by:

configure in interface InputSplit<java.lang.Object>

Overrides:

configure in class FileInputSplit<java.lang.Object>

Parameters:

parameters - The configuration object to read the parameters from.

open

public void open(Config cfg)
          throws java.io.IOException

Opens the given input split. This method opens the input stream to the specified file, allocates read buffers and positions the stream at the correct position, making sure that any partial record at the beginning is skipped.

Specified by:

open in interface InputSplit<java.lang.Object>

Overrides:

open in class FileInputSplit<java.lang.Object>

Throws:

java.io.IOException

nextRecord

public java.lang.String nextRecord(java.lang.Object record)
                            throws java.io.IOException

Description copied from interface: InputSplit

Reads the next record from the input.

When this method is called, the input format it guaranteed to be opened.

Parameters:

record - Object that may be reused.

Returns:

Read record.

Throws:

java.io.IOException - Thrown, if an I/O error occurred.

readRecord

public java.lang.String readRecord(byte[] bytes,
                                   int readOffset,
                                   int numBytes)
                            throws java.io.IOException

Throws:

java.io.IOException

readLine

protected final boolean readLine()
                          throws java.io.IOException

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Closes the input by releasing all buffers and closing the file input stream.

Specified by:

close in interface InputSplit<java.lang.Object>

Overrides:

close in class FileInputSplit<java.lang.Object>

Throws:

java.io.IOException - Thrown, if the closing of the file stream causes an I/O error.

reachedEnd

public boolean reachedEnd()

Checks whether the current split is at its end.

Returns:

True, if the split is at its end, false otherwise.