edu.iu.dsc.tws.connectors.kafka

Class KafkaSource<K,V>

java.lang.Object

edu.iu.dsc.tws.connectors.kafka.KafkaSource<K,V>

All Implemented Interfaces:

INode, ISource, CheckpointableTask, java.io.Serializable

Direct Known Subclasses:

KafkaExample.KSource

public abstract class KafkaSource<K,V>
extends java.lang.Object
implements CheckpointableTask, ISource

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Config	cfg
protected TaskContext	context

Constructor Summary

Constructors

Constructor and Description
KafkaSource()

Method Summary

Modifier and Type	Method and Description
void	execute() Execute without an incoming message
java.lang.String	getConsumerGroup(TaskContext ctx) This method returns the consumer group for this consumer.
abstract java.util.Properties	getConsumerProperties()
<any>	getKafkaConsumer() This method can be used to get the kafka consumer instance, which is created in prepare(Config, TaskContext) method.
abstract java.time.Duration	getPollingTimeout()
abstract java.util.Set<java.lang.String>	getTopics()
void	initSnapshot(Snapshot snapshot) This method could be used to predefine packers
void	onCheckpointPropagated(Snapshot snapshot) This method will be called immediately after task report it's version to the checkpoint master
void	prepare(Config config, TaskContext ctx) Prepare the task to be executed
void	restoreSnapshot(Snapshot snapshot) This method will be called when task is initializing
void	takeSnapshot(Snapshot snapshot) Snapshot should be updated in this method with new values
abstract void	writeRecord(<any> kafkaRecord)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.iu.dsc.tws.checkpointing.task.CheckpointableTask

onSnapshotPersisted

Field Detail

cfg

protected Config cfg

context

protected TaskContext context

Constructor Detail

KafkaSource

public KafkaSource()

Method Detail

prepare

public void prepare(Config config,
                    TaskContext ctx)

Description copied from interface: INode

Prepare the task to be executed

Specified by:

prepare in interface INode

Parameters:

config - the configuration

ctx - the task context, this has runtime information about the task

getConsumerGroup

public java.lang.String getConsumerGroup(TaskContext ctx)

This method returns the consumer group for this consumer. By default it returns the task name, which will effectively load balance the data from kafka across the tasks of this type. Users can override this method to change this behaviour

Parameters:

ctx - Instance of TaskContext

getConsumerProperties

public abstract java.util.Properties getConsumerProperties()

getTopics

public abstract java.util.Set<java.lang.String> getTopics()

writeRecord

public abstract void writeRecord(<any> kafkaRecord)

getPollingTimeout

public abstract java.time.Duration getPollingTimeout()

getKafkaConsumer

public <any> getKafkaConsumer()

This method can be used to get the kafka consumer instance, which is created in prepare(Config, TaskContext) method.

execute

public void execute()

Description copied from interface: ISource

Execute without an incoming message

Specified by:

execute in interface ISource

restoreSnapshot

public void restoreSnapshot(Snapshot snapshot)

Description copied from interface: CheckpointableTask

This method will be called when task is initializing

Specified by:

restoreSnapshot in interface CheckpointableTask

takeSnapshot

public void takeSnapshot(Snapshot snapshot)

Description copied from interface: CheckpointableTask

Snapshot should be updated in this method with new values

Specified by:

takeSnapshot in interface CheckpointableTask

onCheckpointPropagated

public void onCheckpointPropagated(Snapshot snapshot)

Description copied from interface: CheckpointableTask

This method will be called immediately after task report it's version to the checkpoint master

Specified by:

onCheckpointPropagated in interface CheckpointableTask

initSnapshot

public void initSnapshot(Snapshot snapshot)

Description copied from interface: CheckpointableTask

This method could be used to predefine packers

Specified by:

initSnapshot in interface CheckpointableTask