Notify Protocol

The Notify protocol is used to distribute and sync information between the Edge-Cloud services of the Controller, CRM, DME, and others.

Protobuf annotations are used to indicate which objects can be sent/received via the notify protocol. There are two types of objects, notify_cache objects which mirror data (state) from one node to another, and notify_message objects, which send one-off messages.

Object specific code for send and receive are auto-generated by protoc-gen-notify. These generated objects have generic interfaces which are used by the hand-written notify code. This allows the notify code to be agnostic to the specific object type (like App or AppInst), with the exception of filtering based on Cloudlet keys.

Connectivity

Connectivity between services is organized as a tree, with Controllers at the top, and other nodes below. It is intended to be able to support multiple levels of hierarchy, but currently only two are implemented. Connectivity is established bottom-up, so CRM/DMEs have a list (or subset list) of Controllers above them, and connect to one of them. A node may have a single node it connects to above it, and/or may have multiple nodes connecting it from below in a multi-level hierarchy. Nodes never connect horizontally, i.e. CRM will never connect to another CRM.

This connectivity is established by the notify protocol. Once connected and an initial negotation is done, the connection is fully bidirectional and both directions are independent.

The notify protocol itself does not care about the number of levels in the tree, nor the radix.

Filtering

The notify protocol tells the remote node what types of objects (cache/message) it wants. It knows which ones it wants to receive based on which types have been registered for receive with it. The remote side will only send object types that have been requested. The local side will only send objects that have been registered for send, and are "wanted" by the remote.

Additionally, there is custom filtering based on Cloudlet key. Since this is a special case, notify_custom_update and notify_filter_cloudlet_key protobuf options are used to introduce hooks into the auto-generated code. Additionally the client/server itself must be told to ask to the remote side to filter based on cloudlet key.

Data Mirroring

The server side (upstream) pushes objects from the persistent (etcd) database down to clients. The client pushes objects containing dynamic state (cloudlet resources, etc) up to upstream server nodes.

In effect, both directions are mirroring their local state onto the remote node. The goal is to keep the remote state in sync with the local state. Pushes are triggered by node-specific code calling into the notify code to tell it about the key of an object that has been changed/deleted. The notify code will then look up a copy of that data and then push it to the remote node.

There is a difference in behavior between upstream/downstream when handling disconnects. When a client node loses a server, it tries to reconnect to a new server. In the meantime, it does not change its copy of the upstream data. When it manages to reconnect, the server will resend it the full data. Once the initial send is done, the client removes any data that was not received (because that data had been removed while it was disconnected). This minimizes changes to the local mirrored data in the face of server disconnects. However, when a server loses its client, it flushes all data related to that client. This is because the client may reconnect to a different server, or may just be gone forever. So the notify code tracks downstream data on a per-client basis using a NotifyId.