edu.iu.dsc.tws.api.comms

Class CommunicationContext

java.lang.Object

edu.iu.dsc.tws.api.config.Context

edu.iu.dsc.tws.api.comms.CommunicationContext

public class CommunicationContext
extends Context

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CommunicationContext.JoinAlgorithm
static class	CommunicationContext.JoinType

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	ALLGATHER
static java.lang.String	ALLREDUCE
static java.lang.String	ALLTOALL_ALGO_KEY
static java.lang.String	ALLTOALL_ALGO_RING
static java.lang.String	ALLTOALL_ALGO_SIMPLE
static java.lang.String	BROADCAST
static java.lang.String	BUFFER_SIZE
static java.lang.String	CHANNEL_PENDING_SIZE
static java.nio.ByteOrder	DEFAULT_BYTEORDER
static int	DEFAULT_DESTINATION
static java.lang.String	DIRECT
static java.util.List<java.lang.Object>	EMPTY_OBJECT
static java.lang.String	GATHER
static java.lang.String	GROUP_BY_KEY
static java.lang.String	INTER_NODE_DEGREE
static java.lang.String	INTRA_NODE_DEGREE
static java.lang.String	JOIN
static java.lang.String	JOIN_ALGORITHM
static java.lang.String	JOIN_TYPE Type of the join algorithm to apply with join operation
static java.lang.String	KEY_COMPARATOR Twister2 keyed operations sometimes requires to compare keys.
static java.lang.String	KEYED_GATHER
static java.lang.String	KEYED_PARTITION
static java.lang.String	KEYED_REDUCE
static java.lang.String	MESSAGE_GROUP_HIGH_WATERMARK
static java.lang.String	MESSAGE_GROUP_LOW_WATERMARK
static java.lang.String	MESSAGE_GROUPING_SIZE
static java.lang.String	PARTITION
static java.lang.String	PERSISTENT_DIRECTORIES
static java.lang.String	PERSISTENT_DIRECTORY_DEFAULT_VALUE
static java.lang.String	RECEIVE_BUFFERS_COUNT
static java.lang.String	REDUCE
static java.lang.String	RING_GROUPING_WORKER_PER_GROUPS
static java.lang.String	ROW_SCHEMA
static java.lang.String	SEND_BUFFERS_COUNT
static java.lang.String	SEND_PENDING_MAX
static java.lang.String	SHUFFLE_MAX_BYTES_IN_MEMORY
static java.lang.String	SHUFFLE_MAX_FILE_SIZE
static java.lang.String	SHUFFLE_MAX_RECORDS_IN_MEMORY
static java.lang.String	SHUFFLE_PARALLEL_IO
static java.lang.String	SORT_BY_KEY Some keyed communication operations are capable of sorting the outputs by key.
static java.lang.String	TABLE_PARTITION
static java.lang.String	TABLE_PIPE
static java.lang.String	USE_DISK Twister2 has the support to use disk for some operations.

Fields inherited from class edu.iu.dsc.tws.api.config.Context

AURORA_SCRIPT, CHECKPOINT_YAML, CLUSTER, CLUSTER_CONF, CLUSTER_HOME, CORE_YAML, DATA_YAML, defaults, GRAPH_TYPE, HOME, JAVA_HOME, JOB, JOB_ARCHIVE_DIRECTORY, JOB_ID, JOB_NAME, JOB_OBJECT, NETWORK_YAML, OPERATION_NAME, OVERRIDE_YAML, RESOURCE_SCHEDULER_YAML, STREAMING, substitutions, TASK_YAML, TWISTER2_BANDWIDTH, TWISTER2_BIN, TWISTER2_CLUSTER_TYPE, TWISTER2_COMMON_CONF_DIR, TWISTER2_CONF, TWISTER2_CONTAINER_ID, TWISTER2_DIRECT_EDGE, TWISTER2_DIST, TWISTER2_GRAPH_JOB, TWISTER2_HOME, TWISTER2_LATENCY, TWISTER2_LIB, TWISTER2_MAX_TASK_INSTANCES_PER_WORKER, TWISTER2_TASK_INSTANCE_ODD_PARALLELISM, TWISTER2_TSET_JOB, TWISTER2_WORKER_INSTANCES, TWISTER2_WORKER_INSTANCES_DEFAULT, USER_NAME, VERBOSE

Constructor Summary

Constructors

Constructor and Description
CommunicationContext()

Method Summary

Modifier and Type	Method and Description
static int	bufferSize(Config cfg)
static int	getNetworkPartitionBatchGroupingSize(Config cfg)
static int	getNetworkPartitionMessageGroupHighWaterMark(Config cfg)
static int	getNetworkPartitionMessageGroupLowWaterMark(Config cfg)
static int	getParallelIOAllowance(Config cfg)
static int	getRingWorkersPerGroup(Config cfg)
static long	getShuffleFileSize(Config cfg)
static long	getShuffleMaxBytesInMemory(Config cfg)
static long	getShuffleMaxRecordsInMemory(Config cfg)
static int	interNodeDegree(Config cfg, int defaultValue)
static int	intraNodeDegree(Config cfg, int defaultValue)
static int	networkChannelPendingSize(Config cfg)
static java.lang.String	partitionAlgorithm(Config cfg)
static java.util.List<java.lang.String>	persistentDirectory(Config cfg)
static int	receiveBufferCount(Config cfg)
static int	sendBuffersCount(Config cfg)
static int	sendPendingMax(Config cfg)

Methods inherited from class edu.iu.dsc.tws.api.config.Context

auroraScript, checkpointCofigurationFile, checkpointConfigurationFile, clusterType, commonConfDir, conf, containerId, dataConfigurationFile, distDirectory, getDefaults, getDoublePropertyValue, getIntPropertyValue, getLongPropertyValue, getStringPropertyValue, isKubernetesCluster, jobId, jobName, libDirectory, networkConfigurationFile, resourceSchedulerConfigurationFile, systemConfigurationFile, taskConfigurationFile, twister2Home, userName, verbose, workerInstances

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

REDUCE

public static final java.lang.String REDUCE

See Also:

Constant Field Values

GATHER

public static final java.lang.String GATHER

See Also:

Constant Field Values

PARTITION

public static final java.lang.String PARTITION

See Also:

Constant Field Values

ALLGATHER

public static final java.lang.String ALLGATHER

See Also:

Constant Field Values

ALLREDUCE

public static final java.lang.String ALLREDUCE

See Also:

Constant Field Values

BROADCAST

public static final java.lang.String BROADCAST

See Also:

Constant Field Values

KEYED_PARTITION

public static final java.lang.String KEYED_PARTITION

See Also:

Constant Field Values

KEYED_REDUCE

public static final java.lang.String KEYED_REDUCE

See Also:

Constant Field Values

KEYED_GATHER

public static final java.lang.String KEYED_GATHER

See Also:

Constant Field Values

DIRECT

public static final java.lang.String DIRECT

See Also:

Constant Field Values

JOIN

public static final java.lang.String JOIN

See Also:

Constant Field Values

TABLE_PARTITION

public static final java.lang.String TABLE_PARTITION

See Also:

Constant Field Values

TABLE_PIPE

public static final java.lang.String TABLE_PIPE

See Also:

Constant Field Values

INTER_NODE_DEGREE

public static final java.lang.String INTER_NODE_DEGREE

See Also:

Constant Field Values

INTRA_NODE_DEGREE

public static final java.lang.String INTRA_NODE_DEGREE

See Also:

Constant Field Values

DEFAULT_BYTEORDER

public static final java.nio.ByteOrder DEFAULT_BYTEORDER

PERSISTENT_DIRECTORIES

public static final java.lang.String PERSISTENT_DIRECTORIES

See Also:

Constant Field Values

PERSISTENT_DIRECTORY_DEFAULT_VALUE

public static final java.lang.String PERSISTENT_DIRECTORY_DEFAULT_VALUE

See Also:

Constant Field Values

ALLTOALL_ALGO_KEY

public static final java.lang.String ALLTOALL_ALGO_KEY

See Also:

Constant Field Values

ALLTOALL_ALGO_SIMPLE

public static final java.lang.String ALLTOALL_ALGO_SIMPLE

See Also:

Constant Field Values

ALLTOALL_ALGO_RING

public static final java.lang.String ALLTOALL_ALGO_RING

See Also:

Constant Field Values

BUFFER_SIZE

public static final java.lang.String BUFFER_SIZE

See Also:

Constant Field Values

SEND_BUFFERS_COUNT

public static final java.lang.String SEND_BUFFERS_COUNT

See Also:

Constant Field Values

RECEIVE_BUFFERS_COUNT

public static final java.lang.String RECEIVE_BUFFERS_COUNT

See Also:

Constant Field Values

SEND_PENDING_MAX

public static final java.lang.String SEND_PENDING_MAX

See Also:

Constant Field Values

CHANNEL_PENDING_SIZE

public static final java.lang.String CHANNEL_PENDING_SIZE

See Also:

Constant Field Values

MESSAGE_GROUP_LOW_WATERMARK

public static final java.lang.String MESSAGE_GROUP_LOW_WATERMARK

See Also:

Constant Field Values

MESSAGE_GROUP_HIGH_WATERMARK

public static final java.lang.String MESSAGE_GROUP_HIGH_WATERMARK

See Also:

Constant Field Values

MESSAGE_GROUPING_SIZE

public static final java.lang.String MESSAGE_GROUPING_SIZE

See Also:

Constant Field Values

SHUFFLE_MAX_BYTES_IN_MEMORY

public static final java.lang.String SHUFFLE_MAX_BYTES_IN_MEMORY

See Also:

Constant Field Values

SHUFFLE_MAX_RECORDS_IN_MEMORY

public static final java.lang.String SHUFFLE_MAX_RECORDS_IN_MEMORY

See Also:

Constant Field Values

SHUFFLE_MAX_FILE_SIZE

public static final java.lang.String SHUFFLE_MAX_FILE_SIZE

See Also:

Constant Field Values

SHUFFLE_PARALLEL_IO

public static final java.lang.String SHUFFLE_PARALLEL_IO

See Also:

Constant Field Values

RING_GROUPING_WORKER_PER_GROUPS

public static final java.lang.String RING_GROUPING_WORKER_PER_GROUPS

See Also:

Constant Field Values

DEFAULT_DESTINATION

public static final int DEFAULT_DESTINATION

See Also:

Constant Field Values

EMPTY_OBJECT

public static final java.util.List<java.lang.Object> EMPTY_OBJECT

USE_DISK

public static final java.lang.String USE_DISK

Twister2 has the support to use disk for some operations. For example, keyed gather in batch mode can disk to handle tera bytes of data. This flag can be used to signal engine to use disk whenever possible. Please note that, this option is not applicable for all the operations and not intended to use directly.

See Also:

Constant Field Values

ROW_SCHEMA

public static final java.lang.String ROW_SCHEMA

See Also:

Constant Field Values

KEY_COMPARATOR

public static final java.lang.String KEY_COMPARATOR

Twister2 keyed operations sometimes requires to compare keys. This property can be set to define key comparator for such operations.

See Also:

Constant Field Values

JOIN_TYPE

public static final java.lang.String JOIN_TYPE

Type of the join algorithm to apply with join operation

See Also:

Constant Field Values

JOIN_ALGORITHM

public static final java.lang.String JOIN_ALGORITHM

See Also:

Constant Field Values

SORT_BY_KEY

public static final java.lang.String SORT_BY_KEY

Some keyed communication operations are capable of sorting the outputs by key. This property can be used to configure such operators.

See Also:

Constant Field Values

GROUP_BY_KEY

public static final java.lang.String GROUP_BY_KEY

See Also:

Constant Field Values

Constructor Detail

CommunicationContext

public CommunicationContext()

Method Detail

interNodeDegree

public static int interNodeDegree(Config cfg,
                                  int defaultValue)

intraNodeDegree

public static int intraNodeDegree(Config cfg,
                                  int defaultValue)

persistentDirectory

public static java.util.List<java.lang.String> persistentDirectory(Config cfg)

partitionAlgorithm

public static java.lang.String partitionAlgorithm(Config cfg)

bufferSize

public static int bufferSize(Config cfg)

sendBuffersCount

public static int sendBuffersCount(Config cfg)

receiveBufferCount

public static int receiveBufferCount(Config cfg)

sendPendingMax

public static int sendPendingMax(Config cfg)

networkChannelPendingSize

public static int networkChannelPendingSize(Config cfg)

getNetworkPartitionMessageGroupLowWaterMark

public static int getNetworkPartitionMessageGroupLowWaterMark(Config cfg)

getNetworkPartitionMessageGroupHighWaterMark

public static int getNetworkPartitionMessageGroupHighWaterMark(Config cfg)

getShuffleMaxRecordsInMemory

public static long getShuffleMaxRecordsInMemory(Config cfg)

getShuffleMaxBytesInMemory

public static long getShuffleMaxBytesInMemory(Config cfg)

getNetworkPartitionBatchGroupingSize

public static int getNetworkPartitionBatchGroupingSize(Config cfg)

getShuffleFileSize

public static long getShuffleFileSize(Config cfg)

getParallelIOAllowance

public static int getParallelIOAllowance(Config cfg)

getRingWorkersPerGroup

public static int getRingWorkersPerGroup(Config cfg)