edu.iu.dsc.tws.data.api.formatters

Class LocalCompleteArrowInputPartitioner

java.lang.Object

edu.iu.dsc.tws.data.api.formatters.CompleteArrowInputPartitioner<java.lang.String>

edu.iu.dsc.tws.data.api.formatters.LocalCompleteArrowInputPartitioner

All Implemented Interfaces:

InputPartitioner<java.lang.String,FileInputSplit<java.lang.String>>, java.io.Serializable

public class LocalCompleteArrowInputPartitioner
extends CompleteArrowInputPartitioner<java.lang.String>

See Also:

Serialized Form

Field Summary

Fields inherited from class edu.iu.dsc.tws.data.api.formatters.CompleteArrowInputPartitioner

config, filePath, numSplits, READ_WHOLE_SPLIT_FLAG, stream

Constructor Summary

Constructors

Constructor and Description
LocalCompleteArrowInputPartitioner(Path filePath, int numTasks, Config config)
LocalCompleteArrowInputPartitioner(Path filePath, int numTasks, int datasize, Config config)

Method Summary

Modifier and Type	Method and Description
void	configure(Config parameters) Configures this input format.
FileInputSplit<java.lang.String>[]	createInputSplits(int minNumSplits) It creates the split for the complete file.
protected ArrowInputSplit	createSplit(int num, Path file, long start, long length, java.lang.String[] hosts)
InputSplitAssigner	getInputSplitAssigner(FileInputSplit[] inputSplits) Return the input split asigner

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LocalCompleteArrowInputPartitioner

public LocalCompleteArrowInputPartitioner(Path filePath,
                                          int numTasks,
                                          Config config)

LocalCompleteArrowInputPartitioner

public LocalCompleteArrowInputPartitioner(Path filePath,
                                          int numTasks,
                                          int datasize,
                                          Config config)

Method Detail

createSplit

protected ArrowInputSplit createSplit(int num,
                                      Path file,
                                      long start,
                                      long length,
                                      java.lang.String[] hosts)

Specified by:

createSplit in class CompleteArrowInputPartitioner<java.lang.String>

getInputSplitAssigner

public InputSplitAssigner getInputSplitAssigner(FileInputSplit[] inputSplits)

Description copied from interface: InputPartitioner

Return the input split asigner

Parameters:

inputSplits - the input splits to be assigned

Returns:

assigner

configure

public void configure(Config parameters)

Description copied from interface: InputPartitioner

Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated input format.

Specified by:

configure in interface InputPartitioner<java.lang.String,FileInputSplit<java.lang.String>>

Overrides:

configure in class CompleteArrowInputPartitioner<java.lang.String>

Parameters:

parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).

createInputSplits

public FileInputSplit<java.lang.String>[] createInputSplits(int minNumSplits)

Description copied from class: CompleteArrowInputPartitioner

It creates the split for the complete file.

Specified by:

createInputSplits in interface InputPartitioner<java.lang.String,FileInputSplit<java.lang.String>>

Overrides:

createInputSplits in class CompleteArrowInputPartitioner<java.lang.String>

Parameters:

minNumSplits - Number of minimal input splits, as a hint.

Returns:

set of input splits