org.apache.flink.runtime.util

Class ZooKeeperUtils

java.lang.Object

org.apache.flink.runtime.util.ZooKeeperUtils

public class ZooKeeperUtils
extends Object

Class containing helper functions to interact with ZooKeeper.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ZooKeeperUtils.SecureAclProvider Secure ACLProvider implementation.
static class	ZooKeeperUtils.ZkClientACLMode ZooKeeper client ACL mode enum.

Field Summary

Fields

Modifier and Type	Field and Description
static String	CONNECTION_INFO_NODE
static String	HA_STORAGE_COMPLETED_CHECKPOINT The prefix of the completed checkpoint file.
static String	HA_STORAGE_SUBMITTED_JOBGRAPH_PREFIX The prefix of the submitted job graph file.

Method Summary

Modifier and Type	Method and Description
static ZooKeeperCheckpointIDCounter	createCheckpointIDCounter(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client) Creates a ZooKeeperCheckpointIDCounter instance.
static CompletedCheckpointStore	createCompletedCheckpoints(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, org.apache.flink.configuration.Configuration configuration, int maxNumberOfCheckpointsToRetain, SharedStateRegistryFactory sharedStateRegistryFactory, Executor ioExecutor, Executor executor, RestoreMode restoreMode) Creates a DefaultCompletedCheckpointStore instance with ZooKeeperStateHandleStore.
static <T extends Serializable> FileSystemStateStorageHelper<T>	createFileSystemStateStorage(org.apache.flink.configuration.Configuration configuration, String prefix) Creates a FileSystemStateStorageHelper instance.
static JobGraphStore	createJobGraphs(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, org.apache.flink.configuration.Configuration configuration) Creates a DefaultJobGraphStore instance with ZooKeeperStateHandleStore, ZooKeeperJobGraphStoreWatcher and ZooKeeperJobGraphStoreUtil.
static ZooKeeperLeaderElectionDriverFactory	createLeaderElectionDriverFactory(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client) Creates a LeaderElectionDriverFactory implemented by ZooKeeper.
static ZooKeeperLeaderElectionDriverFactory	createLeaderElectionDriverFactory(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String path) Creates a LeaderElectionDriverFactory implemented by ZooKeeper.
static DefaultLeaderElectionService	createLeaderElectionService(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client) Creates a DefaultLeaderElectionService instance with ZooKeeperLeaderElectionDriver.
static DefaultLeaderElectionService	createLeaderElectionService(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String path) Creates a DefaultLeaderElectionService instance with ZooKeeperLeaderElectionDriver.
static ZooKeeperLeaderRetrievalDriverFactory	createLeaderRetrievalDriverFactory(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client) Creates a LeaderRetrievalDriverFactory implemented by ZooKeeper.
static ZooKeeperLeaderRetrievalDriverFactory	createLeaderRetrievalDriverFactory(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String path, org.apache.flink.configuration.Configuration configuration) Creates a LeaderRetrievalDriverFactory implemented by ZooKeeper.
static DefaultLeaderRetrievalService	createLeaderRetrievalService(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client) Creates a DefaultLeaderRetrievalService instance with ZooKeeperLeaderRetrievalDriver.
static DefaultLeaderRetrievalService	createLeaderRetrievalService(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String path, org.apache.flink.configuration.Configuration configuration) Creates a DefaultLeaderRetrievalService instance with ZooKeeperLeaderRetrievalDriver.
static org.apache.flink.shaded.curator5.org.apache.curator.framework.recipes.cache.TreeCache	createTreeCache(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String pathToNode, org.apache.flink.util.function.RunnableWithException nodeChangeCallback) Creates a TreeCache that only observes a specific node.
static <T extends Serializable> ZooKeeperStateHandleStore<T>	createZooKeeperStateHandleStore(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String path, RetrievableStateStorageHelper<T> stateStorage) Creates an instance of ZooKeeperStateHandleStore.
static void	deleteZNode(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework curatorFramework, String path)
static String	generateConnectionInformationPath(String path)
static String	generateLeaderLatchPath(String path)
static String	generateZookeeperPath(String... paths) Creates a ZooKeeper path of the form "/a/b/.../z".
static String	getCheckpointIdCounterPath()
static String	getDispatcherNode()
static String	getJobsPath()
static String	getLeaderLatchPath()
static String	getLeaderPath()
static String	getLeaderPathForDispatcher()
static String	getLeaderPathForJob(org.apache.flink.api.common.JobID jobId)
static String	getLeaderPathForJobManager(org.apache.flink.api.common.JobID jobId)
static String	getLeaderPathForResourceManager()
static String	getLeaderPathForRestServer()
static String	getPathForJob(org.apache.flink.api.common.JobID jobId) Returns the JobID as a String (with leading slash).
static String	getResourceManagerNode()
static String	getRestServerNode()
static String	getSingleLeaderElectionPathForJobManager(org.apache.flink.api.common.JobID jobID)
static String	getZooKeeperEnsemble(org.apache.flink.configuration.Configuration flinkConf) Returns the configured ZooKeeper quorum (and removes whitespace, because ZooKeeper does not tolerate it).
static boolean	isZooKeeperRecoveryMode(org.apache.flink.configuration.Configuration flinkConf) Returns whether HighAvailabilityMode.ZOOKEEPER is configured.
static LeaderInformation	readLeaderInformation(byte[] data)
static String[]	splitZooKeeperPath(String path) Splits the given ZooKeeper path into its parts.
static CuratorFrameworkWithUnhandledErrorListener	startCuratorFramework(org.apache.flink.configuration.Configuration configuration, org.apache.flink.runtime.rpc.FatalErrorHandler fatalErrorHandler) Starts a CuratorFramework instance and connects it to the given ZooKeeper quorum.
static CuratorFrameworkWithUnhandledErrorListener	startCuratorFramework(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFrameworkFactory.Builder builder, org.apache.flink.runtime.rpc.FatalErrorHandler fatalErrorHandler) Starts a CuratorFramework instance and connects it to the given ZooKeeper quorum from a builder.
static String	trimStartingSlash(String path)
static org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework	useNamespaceAndEnsurePath(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client, String path) Returns a facade of the client that uses the specified namespace, and ensures that all nodes in the path exist.
static void	writeLeaderInformationToZooKeeper(LeaderInformation leaderInformation, org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework curatorFramework, java.util.function.BooleanSupplier hasLeadershipCheck, String connectionInformationPath)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HA_STORAGE_SUBMITTED_JOBGRAPH_PREFIX

public static final String HA_STORAGE_SUBMITTED_JOBGRAPH_PREFIX

The prefix of the submitted job graph file.

See Also:

Constant Field Values

HA_STORAGE_COMPLETED_CHECKPOINT

public static final String HA_STORAGE_COMPLETED_CHECKPOINT

The prefix of the completed checkpoint file.

See Also:

Constant Field Values

CONNECTION_INFO_NODE

public static final String CONNECTION_INFO_NODE

See Also:

Constant Field Values

Method Detail

getLeaderPathForResourceManager

public static String getLeaderPathForResourceManager()

getLeaderPathForDispatcher

public static String getLeaderPathForDispatcher()

getLeaderPathForRestServer

public static String getLeaderPathForRestServer()

getLeaderPathForJobManager

public static String getLeaderPathForJobManager(org.apache.flink.api.common.JobID jobId)

getSingleLeaderElectionPathForJobManager

public static String getSingleLeaderElectionPathForJobManager(org.apache.flink.api.common.JobID jobID)

getLeaderPathForJob

@Nonnull
public static String getLeaderPathForJob(org.apache.flink.api.common.JobID jobId)

getJobsPath

public static String getJobsPath()

getCheckpointIdCounterPath

public static String getCheckpointIdCounterPath()

getLeaderPath

public static String getLeaderPath()

getDispatcherNode

public static String getDispatcherNode()

getResourceManagerNode

public static String getResourceManagerNode()

getRestServerNode

public static String getRestServerNode()

getLeaderLatchPath

public static String getLeaderLatchPath()

generateConnectionInformationPath

@Nonnull
public static String generateConnectionInformationPath(String path)

generateLeaderLatchPath

public static String generateLeaderLatchPath(String path)

startCuratorFramework

public static CuratorFrameworkWithUnhandledErrorListener startCuratorFramework(org.apache.flink.configuration.Configuration configuration,
                                                                               org.apache.flink.runtime.rpc.FatalErrorHandler fatalErrorHandler)

Starts a CuratorFramework instance and connects it to the given ZooKeeper quorum.

Parameters:

configuration - Configuration object containing the configuration values

fatalErrorHandler - FatalErrorHandler fatalErrorHandler to handle unexpected errors of CuratorFramework

Returns:

CuratorFrameworkWithUnhandledErrorListener instance

startCuratorFramework

@VisibleForTesting
public static CuratorFrameworkWithUnhandledErrorListener startCuratorFramework(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFrameworkFactory.Builder builder,
                                                                                                  org.apache.flink.runtime.rpc.FatalErrorHandler fatalErrorHandler)

Starts a CuratorFramework instance and connects it to the given ZooKeeper quorum from a builder.

Parameters:

builder - CuratorFrameworkFactory.Builder A builder for curatorFramework.

fatalErrorHandler - FatalErrorHandler fatalErrorHandler to handle unexpected errors of CuratorFramework

Returns:

CuratorFrameworkWithUnhandledErrorListener instance

isZooKeeperRecoveryMode

public static boolean isZooKeeperRecoveryMode(org.apache.flink.configuration.Configuration flinkConf)

Returns whether HighAvailabilityMode.ZOOKEEPER is configured.

getZooKeeperEnsemble

public static String getZooKeeperEnsemble(org.apache.flink.configuration.Configuration flinkConf)
                                   throws org.apache.flink.configuration.IllegalConfigurationException

Returns the configured ZooKeeper quorum (and removes whitespace, because ZooKeeper does not tolerate it).

Throws:

org.apache.flink.configuration.IllegalConfigurationException

createLeaderRetrievalService

public static DefaultLeaderRetrievalService createLeaderRetrievalService(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client)

Creates a DefaultLeaderRetrievalService instance with ZooKeeperLeaderRetrievalDriver.

Parameters:

client - The CuratorFramework ZooKeeper client to use

Returns:

DefaultLeaderRetrievalService instance.

createLeaderRetrievalService

public static DefaultLeaderRetrievalService createLeaderRetrievalService(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client,
                                                                         String path,
                                                                         org.apache.flink.configuration.Configuration configuration)

Creates a DefaultLeaderRetrievalService instance with ZooKeeperLeaderRetrievalDriver.

Parameters:

client - The CuratorFramework ZooKeeper client to use

path - The path for the leader retrieval

configuration - configuration for further config options

Returns:

DefaultLeaderRetrievalService instance.

createLeaderRetrievalDriverFactory

public static ZooKeeperLeaderRetrievalDriverFactory createLeaderRetrievalDriverFactory(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client)

Creates a LeaderRetrievalDriverFactory implemented by ZooKeeper.

Parameters:

client - The CuratorFramework ZooKeeper client to use

Returns:

LeaderRetrievalDriverFactory instance.

createLeaderRetrievalDriverFactory

public static ZooKeeperLeaderRetrievalDriverFactory createLeaderRetrievalDriverFactory(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client,
                                                                                       String path,
                                                                                       org.apache.flink.configuration.Configuration configuration)

Creates a LeaderRetrievalDriverFactory implemented by ZooKeeper.

Parameters:

client - The CuratorFramework ZooKeeper client to use

path - The path for the leader zNode

configuration - configuration for further config options

Returns:

LeaderRetrievalDriverFactory instance.

createLeaderElectionService

public static DefaultLeaderElectionService createLeaderElectionService(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client)

Creates a DefaultLeaderElectionService instance with ZooKeeperLeaderElectionDriver.

Parameters:

client - The CuratorFramework ZooKeeper client to use

Returns:

DefaultLeaderElectionService instance.

createLeaderElectionService

public static DefaultLeaderElectionService createLeaderElectionService(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client,
                                                                       String path)

Creates a DefaultLeaderElectionService instance with ZooKeeperLeaderElectionDriver.

Parameters:

client - The CuratorFramework ZooKeeper client to use

path - The path for the leader election

Returns:

DefaultLeaderElectionService instance.

createLeaderElectionDriverFactory

public static ZooKeeperLeaderElectionDriverFactory createLeaderElectionDriverFactory(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client)

Creates a LeaderElectionDriverFactory implemented by ZooKeeper.

Parameters:

client - The CuratorFramework ZooKeeper client to use

Returns:

LeaderElectionDriverFactory instance.

createLeaderElectionDriverFactory

public static ZooKeeperLeaderElectionDriverFactory createLeaderElectionDriverFactory(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client,
                                                                                     String path)

Creates a LeaderElectionDriverFactory implemented by ZooKeeper.

Parameters:

client - The CuratorFramework ZooKeeper client to use

path - The path suffix which we want to append

Returns:

LeaderElectionDriverFactory instance.

writeLeaderInformationToZooKeeper

public static void writeLeaderInformationToZooKeeper(LeaderInformation leaderInformation,
                                                     org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework curatorFramework,
                                                     java.util.function.BooleanSupplier hasLeadershipCheck,
                                                     String connectionInformationPath)
                                              throws Exception

Throws:

Exception

readLeaderInformation

public static LeaderInformation readLeaderInformation(byte[] data)
                                               throws IOException,
                                                      ClassNotFoundException

Throws:

IOException

ClassNotFoundException

createJobGraphs

public static JobGraphStore createJobGraphs(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client,
                                            org.apache.flink.configuration.Configuration configuration)
                                     throws Exception

Creates a DefaultJobGraphStore instance with ZooKeeperStateHandleStore, ZooKeeperJobGraphStoreWatcher and ZooKeeperJobGraphStoreUtil.

Parameters:

client - The CuratorFramework ZooKeeper client to use

configuration - Configuration object

Returns:

DefaultJobGraphStore instance

Throws:

Exception - if the submitted job graph store cannot be created

createCompletedCheckpoints

public static CompletedCheckpointStore createCompletedCheckpoints(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client,
                                                                  org.apache.flink.configuration.Configuration configuration,
                                                                  int maxNumberOfCheckpointsToRetain,
                                                                  SharedStateRegistryFactory sharedStateRegistryFactory,
                                                                  Executor ioExecutor,
                                                                  Executor executor,
                                                                  RestoreMode restoreMode)
                                                           throws Exception

Creates a DefaultCompletedCheckpointStore instance with ZooKeeperStateHandleStore.

Parameters:

client - The CuratorFramework ZooKeeper client to use

configuration - Configuration object

maxNumberOfCheckpointsToRetain - The maximum number of checkpoints to retain

executor - to run ZooKeeper callbacks

restoreMode - the mode in which the job is being restored

Returns:

DefaultCompletedCheckpointStore instance

Throws:

Exception - if the completed checkpoint store cannot be created

getPathForJob

public static String getPathForJob(org.apache.flink.api.common.JobID jobId)

Returns the JobID as a String (with leading slash).

createZooKeeperStateHandleStore

public static <T extends Serializable> ZooKeeperStateHandleStore<T> createZooKeeperStateHandleStore(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client,
                                                                                                    String path,
                                                                                                    RetrievableStateStorageHelper<T> stateStorage)
                                                                                             throws Exception

Creates an instance of ZooKeeperStateHandleStore.

Type Parameters:

T - Type of state

Parameters:

client - ZK client

path - Path to use for the client namespace

stateStorage - RetrievableStateStorageHelper that persist the actual state and whose returned state handle is then written to ZooKeeper

Returns:

ZooKeeperStateHandleStore instance

Throws:

Exception - ZK errors

createCheckpointIDCounter

public static ZooKeeperCheckpointIDCounter createCheckpointIDCounter(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client)

Creates a ZooKeeperCheckpointIDCounter instance.

Parameters:

client - The CuratorFramework ZooKeeper client to use

Returns:

ZooKeeperCheckpointIDCounter instance

createFileSystemStateStorage

public static <T extends Serializable> FileSystemStateStorageHelper<T> createFileSystemStateStorage(org.apache.flink.configuration.Configuration configuration,
                                                                                                    String prefix)
                                                                                             throws IOException

Creates a FileSystemStateStorageHelper instance.

Type Parameters:

T - Type of the state objects

Parameters:

configuration - Configuration object

prefix - Prefix for the created files

Returns:

FileSystemStateStorageHelper instance

Throws:

IOException - if file system state storage cannot be created

generateZookeeperPath

public static String generateZookeeperPath(String... paths)

Creates a ZooKeeper path of the form "/a/b/.../z".

splitZooKeeperPath

public static String[] splitZooKeeperPath(String path)

Splits the given ZooKeeper path into its parts.

Parameters:

path - path to split

Returns:

splited path

trimStartingSlash

public static String trimStartingSlash(String path)

useNamespaceAndEnsurePath

public static org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework useNamespaceAndEnsurePath(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client,
                                                                                                                       String path)
                                                                                                                throws Exception

Returns a facade of the client that uses the specified namespace, and ensures that all nodes in the path exist.

Parameters:

client - ZK client

path - the new namespace

Returns:

ZK Client that uses the new namespace

Throws:

Exception - ZK errors

createTreeCache

public static org.apache.flink.shaded.curator5.org.apache.curator.framework.recipes.cache.TreeCache createTreeCache(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework client,
                                                                                                                    String pathToNode,
                                                                                                                    org.apache.flink.util.function.RunnableWithException nodeChangeCallback)

Creates a TreeCache that only observes a specific node.

Parameters:

client - ZK client

pathToNode - full path of the node to observe

nodeChangeCallback - callback to run if the node has changed

Returns:

tree cache

deleteZNode

public static void deleteZNode(org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFramework curatorFramework,
                               String path)
                        throws Exception

Throws:

Exception