edu.iu.dsc.tws.master.server

Class JobMaster

java.lang.Object

edu.iu.dsc.tws.master.server.JobMaster

public class JobMaster
extends java.lang.Object

JobMaster class It is started for each Twister2 job It provides: worker discovery barrier method Ping service

It can be started in two different modes: Threaded and Blocking

If the user calls: startJobMasterThreaded() It starts as a Thread and the call to this method returns

If the user calls: startJobMasterBlocking() It uses the calling thread and this call does not return unless the JobMaster completes

JobMaster to Dashboard messaging JobMaster reports to Dashboard server when dashboard address is provided in the config If dashboard host address is not provided, it does not try to connect to dashboard server

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	JobMaster.ServerConnectHandler

Field Summary

Fields

Modifier and Type	Field and Description
static int	JOB_MASTER_ID an id to be used when comminicating with workers and the client

Constructor Summary

Constructors

Constructor and Description
JobMaster(Config config, java.lang.String jmAddress, IJobTerminator jobTerminator, JobAPI.Job job, JobMasterAPI.NodeInfo nodeInfo, IScalerPerCluster clusterScaler, JobMasterAPI.JobMasterState initialState) JobMaster constructor to create a job master, the port of job master is read from config file
JobMaster(Config config, java.lang.String jmAddress, int port, IJobTerminator jobTerminator, JobAPI.Job job, JobMasterAPI.NodeInfo nodeInfo, IScalerPerCluster clusterScaler, JobMasterAPI.JobMasterState initialState) JobMaster constructor

Method Summary

Modifier and Type	Method and Description
void	addShutdownHook(boolean clearK8sJobResourcesOnKill) A job can be terminated in two ways: a) workers end a job: workers either successfully complete or fail all workers complete their work and send COMPLETED messages to the Job Master.
void	endJob(JobAPI.JobState finalState1) this method finishes the job It is executed when the worker completed message received from all workers
IDriver	getDriver()
JMWorkerHandler	getWorkerHandler()
ZKMasterController	getZkMasterController()
void	jmFailed() this method is called when JobMaster fails because of some uncaught exception ıt is called by the JobMasterStarter program It closes all threads started by JM It marks its state at ZK persistent storage as FAILED
java.lang.Thread	startDriverThread() start Driver in a Thread
void	startJobMasterBlocking() start the Job Master in a blocking call
java.lang.Thread	startJobMasterThreaded() start the Job Master in a Thread

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

JOB_MASTER_ID

public static final int JOB_MASTER_ID

an id to be used when comminicating with workers and the client

See Also:

Constant Field Values

Constructor Detail

JobMaster

public JobMaster(Config config,
                 java.lang.String jmAddress,
                 int port,
                 IJobTerminator jobTerminator,
                 JobAPI.Job job,
                 JobMasterAPI.NodeInfo nodeInfo,
                 IScalerPerCluster clusterScaler,
                 JobMasterAPI.JobMasterState initialState)

JobMaster constructor

Parameters:

config - configuration

jmAddress - master host

port - the port number

jobTerminator - terminator

job - the job in proto format

nodeInfo - node info of master

JobMaster

public JobMaster(Config config,
                 java.lang.String jmAddress,
                 IJobTerminator jobTerminator,
                 JobAPI.Job job,
                 JobMasterAPI.NodeInfo nodeInfo,
                 IScalerPerCluster clusterScaler,
                 JobMasterAPI.JobMasterState initialState)

JobMaster constructor to create a job master, the port of job master is read from config file

Parameters:

config - configuration

jmAddress - master host

jobTerminator - terminator

job - the job in proto format

nodeInfo - node info of master

Method Detail

startJobMasterThreaded

public java.lang.Thread startJobMasterThreaded()
                                        throws Twister2Exception

start the Job Master in a Thread

Throws:

Twister2Exception

startJobMasterBlocking

public void startJobMasterBlocking()
                            throws Twister2Exception

start the Job Master in a blocking call

Throws:

Twister2Exception

startDriverThread

public java.lang.Thread startDriverThread()

start Driver in a Thread

getZkMasterController

public ZKMasterController getZkMasterController()

getWorkerHandler

public JMWorkerHandler getWorkerHandler()

jmFailed

public void jmFailed()

this method is called when JobMaster fails because of some uncaught exception ıt is called by the JobMasterStarter program It closes all threads started by JM It marks its state at ZK persistent storage as FAILED

endJob

public void endJob(JobAPI.JobState finalState1)

this method finishes the job It is executed when the worker completed message received from all workers

addShutdownHook

public void addShutdownHook(boolean clearK8sJobResourcesOnKill)

A job can be terminated in two ways: a) workers end a job: workers either successfully complete or fail all workers complete their work and send COMPLETED messages to the Job Master. some workers send FULLY_FAILED message to JM Job master clears all job resources from the cluster and informs Dashboard. This is handled in the method: endJob() b) client kills the job: users kill the job explicitly by either executing job kill command or killing the client process if JM is running on the client. JobMaster endJob() method is not called. JM ShutDown Hook gets executed.

In this case, it can either clear job resources on k8s or does not based on the parameter that is provided when the shut down hook is registered. If the job master runs in the client, it should clear k8s resources, because the job is ended by killing JM process. when the job master runs in the cluster, it should not clear resources. The resources should be cleared by the twister2 client.

getDriver

public IDriver getDriver()