edu.iu.dsc.tws.tset.sources

Class HadoopSourceWithMap<K,V,F extends <any>,I>

java.lang.Object

edu.iu.dsc.tws.tset.sources.HadoopSourceWithMap<K,V,F,I>

Type Parameters:

K - Key

V - Value

F - InputFormat

I - Output

All Implemented Interfaces:

SourceFunc<I>, TFunction<I,I>, java.io.Serializable

public class HadoopSourceWithMap<K,V,F extends <any>,I>
extends java.lang.Object
implements SourceFunc<I>

A Hadoop source for reading the values. The source is attached to a map function. Immediately after the record reader reads values, it passes them to map. This allows twister2 to use Hadoop built in record readers without issue.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
HadoopSourceWithMap(Configuration conf, java.lang.Class<F> inputClazz, MapFunc<I,Tuple<K,V>> mapFunc)

Method Summary

Modifier and Type	Method and Description
boolean	hasNext() Weather we have more data
I	next() Get the next value
void	prepare(TSetContext ctx) Prepare the function.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.iu.dsc.tws.api.tset.fn.SourceFunc

close

Methods inherited from interface edu.iu.dsc.tws.api.tset.fn.TFunction

end

Constructor Detail

HadoopSourceWithMap

public HadoopSourceWithMap(Configuration conf,
                           java.lang.Class<F> inputClazz,
                           MapFunc<I,Tuple<K,V>> mapFunc)

Method Detail

prepare

public void prepare(TSetContext ctx)

Description copied from interface: TFunction

Prepare the function. Use this context in case you need to add/get inputs

Specified by:

prepare in interface TFunction<I,I>

Parameters:

ctx - CONTEXT

hasNext

public boolean hasNext()

Description copied from interface: SourceFunc

Weather we have more data

Specified by:

hasNext in interface SourceFunc<I>

Returns:

true if there is more data to be read

next

public I next()

Description copied from interface: SourceFunc

Get the next value

Specified by:

next in interface SourceFunc<I>

Returns:

the next value