edu.iu.dsc.tws.data.api.formatters

Class BinaryInputPartitioner

java.lang.Object

edu.iu.dsc.tws.data.api.formatters.FileInputPartitioner<byte[]>

edu.iu.dsc.tws.data.api.formatters.BinaryInputPartitioner

All Implemented Interfaces:

InputPartitioner<byte[],FileInputSplit<byte[]>>, java.io.Serializable

Direct Known Subclasses:

LocalBinaryInputPartitioner

public class BinaryInputPartitioner
extends FileInputPartitioner<byte[]>

Input formatter class that reads binary files

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected int	numSplits
protected int	recordLength The length of a single record in the given binary file.

Fields inherited from class edu.iu.dsc.tws.data.api.formatters.FileInputPartitioner

config, filePath, READ_WHOLE_SPLIT_FLAG, stream

Constructor Summary

Constructors

Constructor and Description
BinaryInputPartitioner(Path filePath, int recordLen)
BinaryInputPartitioner(Path filePath, int recordLen, Config cfg)
BinaryInputPartitioner(Path filePath, int recordLen, int numberOfTasks)
BinaryInputPartitioner(Path filePath, int recordLen, int numberOfTasks, Config config)

Method Summary

Modifier and Type	Method and Description
void	configure(Config parameters) Configures this input format.
FileInputSplit[]	createInputSplits(int minNumSplits) Computes the input splits for the file.
protected FileInputSplit	createSplit(int num, Path file, long start, long length, java.lang.String[] hosts)
InputSplitAssigner<byte[]>	getInputSplitAssigner(FileInputSplit<byte[]>[] inputSplits) Return the input split asigner

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

recordLength

protected transient int recordLength

The length of a single record in the given binary file.

numSplits

protected transient int numSplits

Constructor Detail

BinaryInputPartitioner

public BinaryInputPartitioner(Path filePath,
                              int recordLen)

BinaryInputPartitioner

public BinaryInputPartitioner(Path filePath,
                              int recordLen,
                              Config cfg)

BinaryInputPartitioner

public BinaryInputPartitioner(Path filePath,
                              int recordLen,
                              int numberOfTasks)

BinaryInputPartitioner

public BinaryInputPartitioner(Path filePath,
                              int recordLen,
                              int numberOfTasks,
                              Config config)

Method Detail

configure

public void configure(Config parameters)

Description copied from interface: InputPartitioner

Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated input format.

Specified by:

configure in interface InputPartitioner<byte[],FileInputSplit<byte[]>>

Overrides:

configure in class FileInputPartitioner<byte[]>

Parameters:

parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).

createInputSplits

public FileInputSplit[] createInputSplits(int minNumSplits)
                                   throws java.io.IOException

Computes the input splits for the file. By default, one file block is one split. If more splits are requested than blocks are available, then a split may be a fraction of a block and splits may cross block boundaries.

Specified by:

createInputSplits in interface InputPartitioner<byte[],FileInputSplit<byte[]>>

Overrides:

createInputSplits in class FileInputPartitioner<byte[]>

Parameters:

minNumSplits - The minimum desired number of file splits.

Returns:

The computed file splits.

Throws:

java.io.IOException

getInputSplitAssigner

public InputSplitAssigner<byte[]> getInputSplitAssigner(FileInputSplit<byte[]>[] inputSplits)

Description copied from interface: InputPartitioner

Return the input split asigner

Parameters:

inputSplits - the input splits to be assigned

Returns:

assigner

createSplit

protected FileInputSplit createSplit(int num,
                                     Path file,
                                     long start,
                                     long length,
                                     java.lang.String[] hosts)

Specified by:

createSplit in class FileInputPartitioner<byte[]>