# README
Structure of Master
Structure
The master module consists of several components:
- Master Server.
- provide Grpc/http api, including
- operate cluster
- register/delete executor server
- handle heartbeat (keep alive)
- user interface
- submit job
- query job(s)
- cancel job
- schedule and failover jobs
- operate cluster
- provide Grpc/http api, including
- ExecutorManager
- Handle Heartbeat
- Notify Resource Manager updates the status infos
- Maintain the aliveness of Executors.
- Check liveness
- Executor Manager check every executor whether the heartbeat has been timeout.
- Once an executor is offline by
heartbeat timeout
, it should notify the job manager to reschedule all the tasks on it.
- Check liveness
- Handle Heartbeat
- Executor Client
- Embeds two independent interfaces
ExecutorServiceClient
, which is used to dispatch task to executor.BrokerServiceClient
, which is used to manage resource belongs to an executor.
- Embeds two independent interfaces
- JobManager
- Receive SubmitJob Request, Check the type of Job, Create the JobMaster.
- JobMaster (per job)
- Generate the Tasks for the job
- Schedule and Dispatch Tasks
# Packages
No description provided by the author
No description provided by the author
No description provided by the author
# Functions
GetDefaultMasterConfig returns a default master config.
NewDefaultBusinessMetaConfig return the default business metastore config.
NewExecutorManagerImpl creates a new ExecutorManagerImpl instance.
NewJobFsm creates a new job fsm.
NewJobManagerImpl creates a new JobManagerImpl instance.
NewMetaStoreManager creates a new metaStoreManagerImpl instance.
NewServer creates a new master-server.
# Constants
DefaultBusinessMetaID is the ID for default business metastore.
FrameMetaID is the ID for frame metastore.
# Structs
Config is the configuration for server-master.
Executor records the status of an executor instance.
ExecutorManagerImpl holds all the executors' info, including liveness, status, resource usage.
JobFsm manages state of all job masters, job master state forms a finite-state machine.
JobHolder holds job meta and worker handle for a job.
JobManagerImpl is a special job master that manages all the job masters, and notify the offline executor to them.
Server handles PRC requests for df master.
# Interfaces
ExecutorManager defines an interface to manager all executors.
JobManager defines manager of job master.
JobStats defines a statistics interface for JobFsm.
MetaStoreManager defines an interface to manage metastore.