edu.iu.dsc.tws.data.api.formatters

Class FixedInputPartitioner<OT>

java.lang.Object

edu.iu.dsc.tws.data.api.formatters.FixedInputPartitioner<OT>

All Implemented Interfaces:

InputPartitioner<OT,FileInputSplit<OT>>, java.io.Serializable

Direct Known Subclasses:

LocalFixedInputPartitioner

public abstract class FixedInputPartitioner<OT>
extends java.lang.Object
implements InputPartitioner<OT,FileInputSplit<OT>>

The fixed input partitioner is really useful if the user exactly knows the number of lines to be generated or available in the input data file.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Config	config
protected Path	filePath
protected FSDataInputStream	stream

Constructor Summary

Constructors

Constructor and Description
FixedInputPartitioner(Path filePath)
FixedInputPartitioner(Path filePath, Config cfg)
FixedInputPartitioner(Path filePath, Config cfg, int datasize)

Method Summary

Modifier and Type	Method and Description
void	configure(Config parameters) Configures this input format.
FileInputSplit<OT>[]	createInputSplits(int minNumSplits) This method create the input splits which is based on the number of lines in the input and the parallelism value.
protected abstract FileInputSplit	createSplit(int num, Path file, long start, long length, java.lang.String[] hosts)
InputSplitAssigner<OT>	getInputSplitAssigner(FileInputSplit<OT>[] inputSplits) Return the input split asigner

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

filePath

protected Path filePath

config

protected Config config

stream

protected FSDataInputStream stream

Constructor Detail

FixedInputPartitioner

public FixedInputPartitioner(Path filePath)

FixedInputPartitioner

public FixedInputPartitioner(Path filePath,
                             Config cfg)

FixedInputPartitioner

public FixedInputPartitioner(Path filePath,
                             Config cfg,
                             int datasize)

Method Detail

configure

public void configure(Config parameters)

Description copied from interface: InputPartitioner

Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated input format.

Specified by:

configure in interface InputPartitioner<OT,FileInputSplit<OT>>

Parameters:

parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).

createInputSplits

public FileInputSplit<OT>[] createInputSplits(int minNumSplits)
                                       throws java.io.IOException

This method create the input splits which is based on the number of lines in the input and the parallelism value.

Specified by:

createInputSplits in interface InputPartitioner<OT,FileInputSplit<OT>>

Parameters:

minNumSplits - Number of minimal input splits, as a hint.

Returns:

set of input splits

Throws:

java.io.IOException

createSplit

protected abstract FileInputSplit createSplit(int num,
                                              Path file,
                                              long start,
                                              long length,
                                              java.lang.String[] hosts)

getInputSplitAssigner

public InputSplitAssigner<OT> getInputSplitAssigner(FileInputSplit<OT>[] inputSplits)

Description copied from interface: InputPartitioner

Return the input split asigner

Specified by:

getInputSplitAssigner in interface InputPartitioner<OT,FileInputSplit<OT>>

Parameters:

inputSplits - the input splits to be assigned

Returns:

assigner