edu.iu.dsc.tws.data.api

Interface InputPartitioner<OT,T extends InputSplit<OT>>

Type Parameters:

OT - The type of the produced records.

T - The type of input split.

All Superinterfaces:

java.io.Serializable

All Known Implementing Classes:

BinaryInputPartitioner, CompleteArrowInputPartitioner, CompleteCSVInputPartitioner, CompleteFileInputPartitioner, CSVInputPartitioner, FileInputPartitioner, FixedInputPartitioner, LocalBinaryInputPartitioner, LocalCompleteArrowInputPartitioner, LocalCompleteCSVInputPartitioner, LocalCompleteTextInputPartitioner, LocalCSVInputPartitioner, LocalFixedInputPartitioner, LocalTextInputPartitioner, LocalTextInputPartitioner1, SharedTextInputPartitioner

public interface InputPartitioner<OT,T extends InputSplit<OT>>
extends java.io.Serializable

The base interface for data sources that produces records.

The input format handles the following:

• It describes how the input is split into splits that can be processed in parallel.
• It describes how to read records from the input split.
• It describes how to gather basic statistics from the input.

The life cycle of an input format is the following:

1. After being instantiated (parameterless), it is configured with a Configuration object. Basic fields are read from the configuration, such as for example a file path, if the format describes files as input.
2. Optionally: It is called by the compiler to produce basic statistics about the input.
3. It is called to create the input splits.
4. Each parallel input task creates an instance, configures it and opens it for a specific split.
5. All records are read from the input
6. The input format is closed

IMPORTANT NOTE: Input formats must be written such that an instance can be opened again after it was closed. That is due to the fact that the input format is used for potentially multiple splits. After a split is done, the format's close function is invoked and, if another split is available, the open function is invoked afterwards for the next split.

See Also:

//@see BaseStatistics

Method Summary

Modifier and Type	Method and Description
void	configure(Config parameters) Configures this input format.
T[]	createInputSplits(int minNumSplits) Create the input splits
InputSplitAssigner<OT>	getInputSplitAssigner(T[] inputSplits) Return the input split asigner

Method Detail

configure

void configure(Config parameters)

Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated input format.

Parameters:

parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).

createInputSplits

T[] createInputSplits(int minNumSplits)
               throws java.lang.Exception

Create the input splits

Parameters:

minNumSplits - Number of minimal input splits, as a hint.

Returns:

set of input splits

Throws:

java.lang.Exception - if an error occurred

getInputSplitAssigner

InputSplitAssigner<OT> getInputSplitAssigner(T[] inputSplits)

Return the input split asigner

Parameters:

inputSplits - the input splits to be assigned

Returns:

assigner