edu.iu.dsc.tws.data.api.formatters

Class CSVInputPartitioner<OT>

java.lang.Object

edu.iu.dsc.tws.data.api.formatters.CSVInputPartitioner<OT>

All Implemented Interfaces:

InputPartitioner<OT,FileInputSplit<OT>>, java.io.Serializable

Direct Known Subclasses:

LocalCSVInputPartitioner

public abstract class CSVInputPartitioner<OT>
extends java.lang.Object
implements InputPartitioner<OT,FileInputSplit<OT>>

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Config	config
protected Path	filePath
protected int	numSplits

Constructor Summary

Constructors

Constructor and Description
CSVInputPartitioner(Path filePath, Config cfg)
CSVInputPartitioner(Path filePath, Config cfg, int datasize)

Method Summary

Modifier and Type	Method and Description
void	configure(Config parameters) Configures this input format.
FileInputSplit<OT>[]	createInputSplits(int minNumSplits) Create the input splits
protected abstract FileInputSplit	createSplit(int num, Path file, long start, long length, java.lang.String[] hosts)
InputSplitAssigner<OT>	getInputSplitAssigner(FileInputSplit<OT>[] inputSplits) Return the input split asigner

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

numSplits

protected transient int numSplits

filePath

protected Path filePath

config

protected Config config

Constructor Detail

CSVInputPartitioner

public CSVInputPartitioner(Path filePath,
                           Config cfg)

CSVInputPartitioner

public CSVInputPartitioner(Path filePath,
                           Config cfg,
                           int datasize)

Method Detail

configure

public void configure(Config parameters)

Description copied from interface: InputPartitioner

Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated input format.

Specified by:

configure in interface InputPartitioner<OT,FileInputSplit<OT>>

Parameters:

parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).

createInputSplits

public FileInputSplit<OT>[] createInputSplits(int minNumSplits)
                                       throws java.io.IOException

Description copied from interface: InputPartitioner

Create the input splits

Specified by:

createInputSplits in interface InputPartitioner<OT,FileInputSplit<OT>>

Parameters:

minNumSplits - Number of minimal input splits, as a hint.

Returns:

set of input splits

Throws:

java.io.IOException

createSplit

protected abstract FileInputSplit createSplit(int num,
                                              Path file,
                                              long start,
                                              long length,
                                              java.lang.String[] hosts)

getInputSplitAssigner

public InputSplitAssigner<OT> getInputSplitAssigner(FileInputSplit<OT>[] inputSplits)

Description copied from interface: InputPartitioner

Return the input split asigner

Specified by:

getInputSplitAssigner in interface InputPartitioner<OT,FileInputSplit<OT>>

Parameters:

inputSplits - the input splits to be assigned

Returns:

assigner