edu.iu.dsc.tws.data.api.formatters

Class FileInputPartitioner<OT>

java.lang.Object

edu.iu.dsc.tws.data.api.formatters.FileInputPartitioner<OT>

All Implemented Interfaces:

InputPartitioner<OT,FileInputSplit<OT>>, java.io.Serializable

Direct Known Subclasses:

BinaryInputPartitioner, LocalTextInputPartitioner, LocalTextInputPartitioner1, SharedTextInputPartitioner

public abstract class FileInputPartitioner<OT>
extends java.lang.Object
implements InputPartitioner<OT,FileInputSplit<OT>>

Base class for File input formats for specific file types the methods

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Config	config
protected Path	filePath The path to the file that contains the input.
protected int	numSplits The desired number of splits, as set by the configure() method.
static long	READ_WHOLE_SPLIT_FLAG The splitLength is set to -1L for reading the whole split.
protected FSDataInputStream	stream The input data stream

Constructor Summary

Constructors

Constructor and Description
FileInputPartitioner(Path filePath)
FileInputPartitioner(Path filePath, Config cfg)

Method Summary

Modifier and Type	Method and Description
void	configure(Config parameters) Configures this input format.
FileInputSplit<OT>[]	createInputSplits(int minNumSplits) Computes the input splits for the file.
protected abstract FileInputSplit	createSplit(int num, Path file, long start, long length, java.lang.String[] hosts)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.iu.dsc.tws.data.api.InputPartitioner

getInputSplitAssigner

Field Detail

numSplits

protected int numSplits

The desired number of splits, as set by the configure() method.

READ_WHOLE_SPLIT_FLAG

public static final long READ_WHOLE_SPLIT_FLAG

The splitLength is set to -1L for reading the whole split.

See Also:

Constant Field Values

filePath

protected Path filePath

The path to the file that contains the input.

config

protected Config config

stream

protected FSDataInputStream stream

The input data stream

Constructor Detail

FileInputPartitioner

public FileInputPartitioner(Path filePath)

FileInputPartitioner

public FileInputPartitioner(Path filePath,
                            Config cfg)

Method Detail

configure

public void configure(Config parameters)

Description copied from interface: InputPartitioner

Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated input format.

Specified by:

configure in interface InputPartitioner<OT,FileInputSplit<OT>>

Parameters:

parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).

createInputSplits

public FileInputSplit<OT>[] createInputSplits(int minNumSplits)
                                       throws java.io.IOException

Computes the input splits for the file. By default, one file block is one split. If more splits are requested than blocks are available, then a split may be a fraction of a block and splits may cross block boundaries.

Specified by:

createInputSplits in interface InputPartitioner<OT,FileInputSplit<OT>>

Parameters:

minNumSplits - The minimum desired number of file splits.

Returns:

The computed file splits.

Throws:

java.io.IOException

createSplit

protected abstract FileInputSplit createSplit(int num,
                                              Path file,
                                              long start,
                                              long length,
                                              java.lang.String[] hosts)