edu.iu.dsc.tws.common.zk

Class ZKPersStateManager

java.lang.Object

edu.iu.dsc.tws.common.zk.ZKPersStateManager

public final class ZKPersStateManager
extends java.lang.Object

This class has methods to keep persistent status of a job in ZooKeeper servers Parent znode (directory) has the job object as its body One persistent child znode is created for each worker with workerID as its name Each worker znode has WorkerInfo and the last WorkerState as its body A separate persistent znode is created for job master. JM znode is not in workers directory. It is in the upper directory in the main job directory.

When a worker starts, it needs to know whether it is starting for the first time or it is restarting from failure. When starting, each worker checks whether there exists a znode for itself. If the znode exists, it means that this worker has started before.

When the job is scaled down, we delete the znodes of killed workers. This is handled by the scaler in Job Master.

Field Summary

Fields

Modifier and Type	Field and Description
static java.util.logging.Logger	LOG

Method Summary

Modifier and Type	Method and Description
static void	createJobMasterPersState(CuratorFramework client, java.lang.String rootPath, java.lang.String jobID, java.lang.String jmAddress) create job master persistent state at ZooKeeper server this method must be called if isJobMasterRestarting returns false
static void	createPersStateDir(CuratorFramework client, java.lang.String rootPath, java.lang.String jobID) Create job znode for persistent states Assumes that there is no znode exists in the ZooKeeper This method should be called by the submitting client
static void	createWorkerPersState(CuratorFramework client, java.lang.String rootPath, java.lang.String jobID, WorkerInfo workerInfo) Initialize worker persistent state at ZooKeeper server This method must be called after making sure that there is no pers state znode for this worker at zk
static java.util.LinkedList<WorkerWithState>	getWorkers(CuratorFramework client, java.lang.String rootPath, java.lang.String jobID) return all registered workers
static WorkerWithState	getWorkerWithState(CuratorFramework client, java.lang.String workerFullPath)
static WorkerWithState	getWorkerWithState(CuratorFramework client, java.lang.String rootPath, java.lang.String jobID, int workerID)
static int	initAndGetRestartCount(CuratorFramework client, java.lang.String rootPath, java.lang.String jobID, WorkerInfo workerInfo) Initialize worker persistent state at ZooKeeper server return restart count: 0 means first start If the worker is restarting, update restartCount and WorkerInfo
static void	removeScaledDownZNodes(CuratorFramework client, java.lang.String rootPath, java.lang.String jobID, int minID, int maxID) When a job is scaled down, we must delete the znodes of killed workers.
static void	updateJobMasterStatus(CuratorFramework client, java.lang.String rootPath, java.lang.String jobID, java.lang.String jmAddress, JobMasterState state) update jm status at ZK
static boolean	updateWorkerStatus(CuratorFramework client, java.lang.String rootPath, java.lang.String jobID, WorkerInfo workerInfo, int restartCount, WorkerState newStatus)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final java.util.logging.Logger LOG

Method Detail

createPersStateDir

public static void createPersStateDir(CuratorFramework client,
                                      java.lang.String rootPath,
                                      java.lang.String jobID)

Create job znode for persistent states Assumes that there is no znode exists in the ZooKeeper This method should be called by the submitting client

initAndGetRestartCount

public static int initAndGetRestartCount(CuratorFramework client,
                                         java.lang.String rootPath,
                                         java.lang.String jobID,
                                         WorkerInfo workerInfo)
                                  throws Twister2Exception

Initialize worker persistent state at ZooKeeper server return restart count: 0 means first start If the worker is restarting, update restartCount and WorkerInfo

A persistent znode is created/updated for this worker on ZooKeeper server Each worker must call this method exactly once when they start

Throws:

Twister2Exception

createWorkerPersState

public static void createWorkerPersState(CuratorFramework client,
                                         java.lang.String rootPath,
                                         java.lang.String jobID,
                                         WorkerInfo workerInfo)
                                  throws Twister2Exception

Initialize worker persistent state at ZooKeeper server This method must be called after making sure that there is no pers state znode for this worker at zk

Throws:

Twister2Exception

updateJobMasterStatus

public static void updateJobMasterStatus(CuratorFramework client,
                                         java.lang.String rootPath,
                                         java.lang.String jobID,
                                         java.lang.String jmAddress,
                                         JobMasterState state)

update jm status at ZK

createJobMasterPersState

public static void createJobMasterPersState(CuratorFramework client,
                                            java.lang.String rootPath,
                                            java.lang.String jobID,
                                            java.lang.String jmAddress)

create job master persistent state at ZooKeeper server this method must be called if isJobMasterRestarting returns false

removeScaledDownZNodes

public static void removeScaledDownZNodes(CuratorFramework client,
                                          java.lang.String rootPath,
                                          java.lang.String jobID,
                                          int minID,
                                          int maxID)
                                   throws Twister2Exception

When a job is scaled down, we must delete the znodes of killed workers. minID inclusive, maxID exclusive

Throws:

Twister2Exception

updateWorkerStatus

public static boolean updateWorkerStatus(CuratorFramework client,
                                         java.lang.String rootPath,
                                         java.lang.String jobID,
                                         WorkerInfo workerInfo,
                                         int restartCount,
                                         WorkerState newStatus)
                                  throws Twister2Exception

Throws:

Twister2Exception

getWorkerWithState

public static WorkerWithState getWorkerWithState(CuratorFramework client,
                                                 java.lang.String workerFullPath)
                                          throws Twister2Exception

Throws:

Twister2Exception

getWorkerWithState

public static WorkerWithState getWorkerWithState(CuratorFramework client,
                                                 java.lang.String rootPath,
                                                 java.lang.String jobID,
                                                 int workerID)
                                          throws Twister2Exception

Throws:

Twister2Exception

getWorkers

public static java.util.LinkedList<WorkerWithState> getWorkers(CuratorFramework client,
                                                               java.lang.String rootPath,
                                                               java.lang.String jobID)
                                                        throws Twister2Exception

return all registered workers

Throws:

Twister2Exception