containerd services

Mental Model

Most containerd services follow a client-server/service model. And the service is usually implemented as a grpc service where the underlying storage is a bolt DB. So the workflow of a containerd request is usually:

user -> containerd client -> service client -> grpc service client -> grp service server -> server -> bolt DB.

On data representation side, instead of exposing metadata directly, containerd only exposes interfaces to users, which defines the set of operations a user is expected to call for an object (e.g. container/image). Then it wraps the metadata object in an internal client-side struct and implements the user-facing interface. This approach has the benefits of (1) abstracting away some implementation details (e.g., inter-service communication); (2) providing better instructions compared to giving all data to user directly.

This usually leads to class/struct changes along a containerd request:

user -> client struct/interface -> metadata struct -> grpc-generated struct -> metadata struct.

Container Service

Container Service in containerd is responsible for managing and storing container metadata. The workflow from containerd client to the underlying DB involves multiple components and different (but similar) container structs. This section gives a bottom-up view of the container service and related code path.

metadata.containerStore is the underlying container storage implemented using a bolt DB(same as other containerd storage). It implements the CRUD operations for containers, which are defined as an interface containers.Store.

Going up, how is a container CRUD operation called (and executed by metadata.containerStore)? The answer is through Container Service which is a grpc service defined and auto-generated in this folder.

I’m also new to proto/grpc, so I won’t go into details here. :) I’ll revisit here when I have a better understanding of the grpc services in containerd.

Now that we know how a container is stored (in a bolt DB) and how a container CRUD operation is delivered to the container storage (via a grpc service). Next is to define what data we want to store as a container and what CRUD operation we want to support. They are located in the same containerd/containers.go file:

1. containers.Container struct defines the metadata for a container to be stored in metadata.containerStore. It has a proto-equivalent Message definition used by grpc.
2. containers.Store interface defines the CRUD operations for containers (implemented by metadata.containerStore). It also has a proto-equivalent Service definition used by grpc.

All previous steps and for containerd internals: (container) data definition, data storage, service communication. Next question is how a container CRUD request comes from users and is sent to/processed by metadata.containerStore.

The answer to the 1st question is, same as other containerd services, via containerd client which has (grpc) clients for all containerd services. As an example, this is the entry to ContainerService. It first creates a grpc client to the containerStore, which is wrapped into containerd.NewRemoteContainerStore. containerd.NewRemoteContainerStore implements the same containers.Store interface and acts as a wrapper abstracting the fact that the data is stored in an underlying bolt DB (metadata.containerStore) and accessed through a grpc service client (containersClient).

The last question is how to make a container (metadata) object containers.Container visible to users. Instead of returning the metadata Container struct directly, containerd returns a containerd.Container interface. Based on my understanding, it has the following benefits:

1. Clearly defines the set of container operations/data that containerd wants to expose to users;
2. Containerd provides the implementation so that users don’t need to write much process code based on the metadata containers.Container object.
3. Containerd handles the coordinations between multiple services. For example, users can call container.Image(ctx) directly to get the container’s image instead of calling ImageService.

Under the hook, containerd implements the interface as a non-export struct containerd.container which includes a containerd client, the underlying metadata container (containers.Container), and the container id.

Let’s end with an end-to-end digram showing the above steps for container service in containerd.

Image Service

Personally I feel image service is more complicated than container service because:

1. It has more functionalities other than simply (image) metadata management: image conversion between different formats, image import/export, image (un)compression, etc.
2. It handles different specs: OCI, docker (v1.1, v1.2).

I haven’t looked into all features of image service, so this section only focues on the code path for image service itself other than other features related to images.

Service

Image service also has an entry in containerd client, containerd.Client.ImageService(), where you can get the grpc client wrapper (containerd.NewImageStoreFromClient) for image service grpc client (ImagesClient).

The grpc wrapper handles 2 things: (1) convert user requests to protobuf request messages and send to grpc client; (2) convert grpc responses (from grpc client) back to corresponding structs (e.g., image.Image) and return back to the caller.

The image service grpc client, similar to other grpc clients, is auto-generated based on its grpc/proto definition (images.proto) and responsible for interacting with the image service backend service (which is basically a image.Image metadata store).

Similarly, the grpc service binding for image service (bind the grpc client and backend service) happens in services/images.

Finally, the image metadata store, metadata.NewImageStore, is the backend of image service and is also where a grpc request is processed.

Struct

Image service has 3 main structs:

1. containerd.Image: client interface (and implementation) that wraps image metadata and is returned by containerd client.
2. images.Image: metadata struct that contains image metadata and is stored by image metadata store.
3. Image protobuf message and its auto-generated Image struct: this is mainly used for grpc and should be identical with images.Image (data fields).

Additionally, image service also has the data store interface, images.Store, that defines its main features (image CRUD). It is implemented both by the grpc client wrapper (to send a request) and by the image metadata store (to receive and process a request).

updated_at 28-11-2022