edu.iu.dsc.tws.task.impl

Class TaskExecutor

java.lang.Object

edu.iu.dsc.tws.task.impl.TaskExecutor

All Implemented Interfaces:

FaultAcceptable

public class TaskExecutor
extends java.lang.Object
implements FaultAcceptable

The task executor API, this class can be used to create an execution plan and execute

Constructor Summary

Constructors

Constructor and Description
TaskExecutor(Config cfg, int wId, java.util.List<JobMasterAPI.WorkerInfo> workerInfoList, Communicator net, CheckpointingClient checkpointingClient) Creates a task executor.
TaskExecutor(WorkerEnvironment workerEnv)

Method Summary

Modifier and Type	Method and Description
void	addInput(ComputeGraph graph, ExecutionPlan plan, java.lang.String taskName, java.lang.String inputKey, DataObject<?> input) Deprecated. Inputs are automatically handled now
void	addInput(IExecutor ex, java.lang.String taskName, java.lang.String inputKey, DataObject<?> input) Deprecated. Inputs are automatically handled now
void	addInput(java.lang.String name, DataObject dataObject)
void	addSourceInput(ComputeGraph graph, ExecutionPlan plan, java.lang.String inputKey, DataObject<java.lang.Object> input) Deprecated. Inputs are handled automatically now
void	addSourceInput(IExecutor ex, java.lang.String inputKey, DataObject<java.lang.Object> input) Deprecated. Inputs are handled automatically now
void	clearData(java.lang.String var) This method can be used to clear DataPartitions collected from previous task graphs, which are no longer required
void	close()
static void	collectData(Config cfg, ExecutionPlan executionPlan, java.util.Map<java.lang.String,DataObject> dataMap) This method collects all the output from the provided ExecutionPlan.
IExecutor	createExecution(ComputeGraph graph) Execute a plan and a graph.
IExecutor	createExecution(ComputeGraph graph, ExecutionPlan plan) Execute a plan and a graph.
static void	distributeData(ExecutionPlan executionPlan, java.util.Map<java.lang.String,DataObject> dataMap) This method distributes collected DataPartitions to the intended Receptors
void	execute(ComputeGraph graph) Execute a plan and a graph.
void	execute(ComputeGraph graph, ExecutionPlan plan) Execute a plan and a graph.
void	execute(Config taskConfig, ComputeGraph graph, ExecutionPlan plan) Execute a plan and a graph.
ExecutionPlan	executionPlan(ComputeGraph graph, TaskSchedulePlan taskSchedulePlan) Create a execution plan
<T> DataObject<T>	getOutput(ComputeGraph graph, ExecutionPlan plan, java.lang.String taskName) Deprecated. There is no need of using this method anymore. Input - Output is handled automatically
<T> DataObject<T>	getOutput(ComputeGraph graph, ExecutionPlan plan, java.lang.String taskName, java.lang.String dataName) Deprecated. There is no need of using this method anymore. Input - Output is handled automatically
<T> DataObject<T>	getOutput(java.lang.String varName)
boolean	isOutputAvailable(java.lang.String name)
void	onFault(Fault fault) A fault has occurred
java.util.Map<java.lang.String,ExecutionPlan>	plan(ComputeGraph... graph) Create execution plans for each graph.
ExecutionPlan	plan(ComputeGraph graph) Create an execution plan from the given graph

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TaskExecutor

public TaskExecutor(Config cfg,
                    int wId,
                    java.util.List<JobMasterAPI.WorkerInfo> workerInfoList,
                    Communicator net,
                    CheckpointingClient checkpointingClient)

Creates a task executor.

Parameters:

cfg - the configuration

wId - the worker id

net - communicator

TaskExecutor

public TaskExecutor(WorkerEnvironment workerEnv)

Method Detail

plan

public ExecutionPlan plan(ComputeGraph graph)

Create an execution plan from the given graph

Parameters:

graph - task graph

Returns:

the data set

plan

public java.util.Map<java.lang.String,ExecutionPlan> plan(ComputeGraph... graph)

Create execution plans for each graph.

Parameters:

graph - list of graphs

Returns:

graph name -> execution plan map

executionPlan

public ExecutionPlan executionPlan(ComputeGraph graph,
                                   TaskSchedulePlan taskSchedulePlan)

Create a execution plan

Parameters:

graph - the graph

taskSchedulePlan - task schedule

Returns:

the execution plan

execute

public void execute(Config taskConfig,
                    ComputeGraph graph,
                    ExecutionPlan plan)

Execute a plan and a graph. This call blocks until the execution finishes. In case of streaming, this call doesn't return while for batch computations it returns after the execution is done.

Parameters:

taskConfig - the user configuration to be passed to the task instances

graph - the dataflow graph

plan - the execution plan

execute

public void execute(ComputeGraph graph,
                    ExecutionPlan plan)

Execute a plan and a graph. This call blocks until the execution finishes. In case of streaming, this call doesn't return while for batch computations it returns after the execution is done.

Parameters:

graph - the dataflow graph

plan - the execution plan

execute

public void execute(ComputeGraph graph)

Execute a plan and a graph. This call blocks until the execution finishes. In case of streaming, this call doesn't return while for batch computations it returns after the execution is done.

Parameters:

graph - the dataflow graph

createExecution

public IExecutor createExecution(ComputeGraph graph,
                                 ExecutionPlan plan)

Execute a plan and a graph. This call blocks until the execution finishes. In case of streaming, this call doesn't return while for batch computations it returns after the execution is done.

Parameters:

graph - the dataflow graph

plan - the execution plan

createExecution

public IExecutor createExecution(ComputeGraph graph)

Execute a plan and a graph. This call blocks until the execution finishes. In case of streaming, this call doesn't return while for batch computations it returns after the execution is done.

Parameters:

graph - the dataflow graph

addInput

@Deprecated
public void addInput(ComputeGraph graph,
                                 ExecutionPlan plan,
                                 java.lang.String taskName,
                                 java.lang.String inputKey,
                                 DataObject<?> input)

Deprecated. Inputs are automatically handled now

Add input to the the task instances

Parameters:

graph - task graph

plan - execution plan

taskName - task name

inputKey - inputkey

input - input

addInput

@Deprecated
public void addInput(IExecutor ex,
                                 java.lang.String taskName,
                                 java.lang.String inputKey,
                                 DataObject<?> input)

Deprecated. Inputs are automatically handled now

Add input to the the task instances

Parameters:

ex - execution plan

taskName - task name

inputKey - inputkey

input - input

addSourceInput

@Deprecated
public void addSourceInput(ComputeGraph graph,
                                       ExecutionPlan plan,
                                       java.lang.String inputKey,
                                       DataObject<java.lang.Object> input)

Deprecated. Inputs are handled automatically now

Add input to the the task instances

Parameters:

graph - task graph

plan - execution plan

inputKey - inputkey

input - input

addSourceInput

@Deprecated
public void addSourceInput(IExecutor ex,
                                       java.lang.String inputKey,
                                       DataObject<java.lang.Object> input)

Deprecated. Inputs are handled automatically now

Add input to the the task instances

Parameters:

ex - execution plan

inputKey - inputkey

input - input

getOutput

@Deprecated
public <T> DataObject<T> getOutput(ComputeGraph graph,
                                               ExecutionPlan plan,
                                               java.lang.String taskName)

Deprecated. There is no need of using this method anymore. Input - Output is handled automatically

Extract output from a task graph

Parameters:

graph - the graph

plan - plan created from the graph

taskName - name of the output to retrieve

Returns:

a DataObjectImpl with set of partitions from each task in this executor

getOutput

@Deprecated
public <T> DataObject<T> getOutput(ComputeGraph graph,
                                               ExecutionPlan plan,
                                               java.lang.String taskName,
                                               java.lang.String dataName)

Deprecated. There is no need of using this method anymore. Input - Output is handled automatically

Extract output from a task graph

Parameters:

graph - the graph

plan - plan created from the graph

taskName - name of the output to retrieve

dataName - name of the data set

Returns:

a DataObjectImpl with set of partitions from each task in this executor

getOutput

public <T> DataObject<T> getOutput(java.lang.String varName)

isOutputAvailable

public boolean isOutputAvailable(java.lang.String name)

addInput

public void addInput(java.lang.String name,
                     DataObject dataObject)

collectData

public static void collectData(Config cfg,
                               ExecutionPlan executionPlan,
                               java.util.Map<java.lang.String,DataObject> dataMap)

This method collects all the output from the provided ExecutionPlan. The partition IDs will be assigned just before adding the partitions to the DataObject

distributeData

public static void distributeData(ExecutionPlan executionPlan,
                                  java.util.Map<java.lang.String,DataObject> dataMap)

This method distributes collected DataPartitions to the intended Receptors

clearData

public void clearData(java.lang.String var)

This method can be used to clear DataPartitions collected from previous task graphs, which are no longer required

close

public void close()

onFault

public void onFault(Fault fault)

Description copied from interface: FaultAcceptable

A fault has occurred

Specified by:

onFault in interface FaultAcceptable

Parameters:

fault - the fault