edu.iu.dsc.tws.data.api.formatters

Class CompleteFileInputPartitioner<OT>

java.lang.Object

edu.iu.dsc.tws.data.api.formatters.CompleteFileInputPartitioner<OT>

All Implemented Interfaces:

InputPartitioner<OT,FileInputSplit<OT>>, java.io.Serializable

Direct Known Subclasses:

LocalCompleteTextInputPartitioner

public abstract class CompleteFileInputPartitioner<OT>
extends java.lang.Object
implements InputPartitioner<OT,FileInputSplit<OT>>

Base class for File input formats for specific file types the methods

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Config	config Config file required for Hdfs connection
protected Path	filePath The path to the file that contains the input.
protected int	numSplits The desired number of splits, as set by the configure() method.
static long	READ_WHOLE_SPLIT_FLAG The splitLength is set to -1L for reading the whole split.
protected FSDataInputStream	stream The input data stream

Constructor Summary

Constructors

Constructor and Description
CompleteFileInputPartitioner(Path filePath)
CompleteFileInputPartitioner(Path filePath, Config cfg)

Method Summary

Modifier and Type	Method and Description
void	configure(Config parameters) Configures this input format.
FileInputSplit<OT>[]	createInputSplits(int minNumSplits) It creates the split for the complete file.
protected abstract FileInputSplit	createSplit(int num, Path file, long start, long length, java.lang.String[] hosts)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.iu.dsc.tws.data.api.InputPartitioner

getInputSplitAssigner

Field Detail

numSplits

protected int numSplits

The desired number of splits, as set by the configure() method.

READ_WHOLE_SPLIT_FLAG

public static final long READ_WHOLE_SPLIT_FLAG

The splitLength is set to -1L for reading the whole split.

See Also:

Constant Field Values

filePath

protected Path filePath

The path to the file that contains the input.

config

protected Config config

Config file required for Hdfs connection

stream

protected FSDataInputStream stream

The input data stream

Constructor Detail

CompleteFileInputPartitioner

public CompleteFileInputPartitioner(Path filePath)

CompleteFileInputPartitioner

public CompleteFileInputPartitioner(Path filePath,
                                    Config cfg)

Method Detail

configure

public void configure(Config parameters)

Description copied from interface: InputPartitioner

Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated input format.

Specified by:

configure in interface InputPartitioner<OT,FileInputSplit<OT>>

Parameters:

parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).

createInputSplits

public FileInputSplit<OT>[] createInputSplits(int minNumSplits)
                                       throws java.io.IOException

It creates the split for the complete file.

Specified by:

createInputSplits in interface InputPartitioner<OT,FileInputSplit<OT>>

Parameters:

minNumSplits - Number of minimal input splits, as a hint.

Returns:

set of input splits

Throws:

java.io.IOException

createSplit

protected abstract FileInputSplit createSplit(int num,
                                              Path file,
                                              long start,
                                              long length,
                                              java.lang.String[] hosts)