OCI Image Specification

• descriptor
  • digest
• (optional) image index
• image manifest
• (filesystem) layer
• image configuration
• manifest v.s. artifact v.s. descriptor v.s. digest
• image layout
• An image layout example

From the image-spec page:

This specification defines an OCI Image, consisting of an image manifest, an image index (optional), a set of filesystem layers, and a configuration.

It defines both the content and the layout of these objects. An image is called OCI image (v.s. a docker image) if it satisfies this spec.

Based on my own understanding, the image-spec defines the following:

1. Objects/components (JSON, gzip, tar, etc) that are composed of an OCI image: image index, image manifest, image config, layers;
2. How different objects reference each other in a DAG format (e.g., image index -> image manifest -> layer): descriptor;
3. How to identify (and verify) an object: digest;
4. How to layout these objects within an image tar file.

Media types: every object defined in the spec can be identified by its media type.

Now let’s look at different parts of the image-spec. We’ll first look at descriptor and digest since they’re used by many other objects. Then we’ll follow a top-down approach to look at image index, image manifest, image layers and image config. We’ll also look at the image layout defined in the spec with an example image tar file.

descriptor

application/vnd.oci.descriptor.v1+json media type. A JSON object put in one object (e.g., image manifest) to reference another object (e.g., image config, layers).

An OCI image has different components such as image manifest, layers, etc, which are arranged in a DAG (Merkle tree). The reference between components (e.g., image index -> image manifest -> layer) are described as (Content) Descriptor.

A descriptor specifies the type/identifier/size of the referenced content. These fields are worth mentioning:

1. mediaType: unlike in other cases, the mediaType within a descriptor refers to the media type of the referenced content, not the object itself (i.e. the descriptor).
2. digest: the digest of the targeted object/content. For example, the digest within a descriptor in .layers of an image manifest means the the digest of the referenced layer.
3. size: the size of the targeted content.

digest

Digest itself is a broader concept that enables content addressability. It’s also a field of a descriptor.

Digest is a string that uniquely identifies content/bytes (e.g., a blob file, a string). It’s a concatenation of the hash algorithm name and the hash value calculated from the content using the hash algorithm (e.g. sha256:6c3c624b58dbbcd3c0dd82b4c53f04194d1247c6eebdaab7c610cf7d66709b3b).

It can also verify that the content (e.g., an image layer tar file) downloaded from a source (e.g. a registry) is the same bytes generated by the image builder.

(optional) image index

application/vnd.oci.image.index.v1+json media type.

Image index is a JSON object that simply references image manifests or other image indexes (nested index). It acts as a single entrypoint for multiple images (for example, a multi-platform image). A runtime can then choose one of the image manifests referenced by the image index, based on the host environment.

Image index defines the list of image manifests/indexes referenced by it in the manifests field, which is a descriptor array. Each descriptor also has information and metadata such as platform that can be used by runtime to select a specific image manifest/index.

image manifest

application/vnd.oci.image.manifest.v1+json media type. A JSON object describing an image or an artifact.

From the image manifest spec, it has 3 goals:

1. content-addressable image: image configuration content <-> unique ID;
2. multi-architecture images: use image index as a fat manifest;
3. translatable to OCI runtime-spec.

The image manifest spec looks complicated because it has generalized to a broader manifest spec which covers the main image manifest AND the general artifact manifest.

However, the artifact part hasn’t been released and expected to be released as part of OCI image-spec 1.1. Some of the artifact related content are based on the main branch of the spec, which might be changed or even removed. Below we still use image manifest to refer the whole spec, and only use artifact when necessary.

Image manifest defines an specific image (or artifact) that is applicable to a specific arch+OS. The following fields in the spec worth mentioning:

1. config: a descriptor that references an image configuration object.
2. layers: a descriptor array.
3. subject: a descriptor referencing another manifest. This is used by the referrers API to indicate its relationship to the referenced manifest (imagine it as a pointer). For example, an image signature artifact manifest uses subject to reference the image manifest it signs.
4. artifactType: a string indicating the type of artifact, when it’s used as an artifact manifest instead of image manifest.

Another confusing point in the image manifest spec is that it has many fields that define the media type of the specific component in the manifest, such as:

1. .mediaType: the media type of the manifest itself. In most case, it should be application/vnd.oci.image.manifest.v1+json for both image and artifact manifest.
2. .artifactType: the artifact type of the manifest, if not an image.
3. .config.mediaType: for image manifest, it should be application/vnd.oci.image.config.v1+json; for artifact manifest, it should be a self-defined type (e.g., application/vnd.example.config.v1+json) or application/vnd.oci.scratch.v1+json.
4. .layers.[].mediaType: for image, it should be a layer media type such as application/vnd.oci.image.layer.v1.tar+gzip; for artifact, it should be a self-defined type that describe the data type of the artifact.
5. .subject.mediaType: the mediaType of the referenced manifest.

It becomes more complicated when describing an artifact. See the spec for more details.

(filesystem) layer

application/vnd.oci.image.layer.v1.tar, application/vnd.oci.image.layer.v1.tar+gzip, application/vnd.oci.image.layer.v1.tar+zstd, etc media types.

A layer describe a file system diff (add/remove/modify files), which can be applied on top of each other to construct a complete container file system.

Each layer is represented as a tar file (w, or w/o compression), which is also called a changeset. For a base layer, it contains the initial file system content.

For child layers, it only contains file entries that are different (add/remove/modify) from its parent layer tar file. Since layers are just tar files with regular file entries, we need to figure out how to represent file diffs that can be applied to parent layers:

1. Add a file/directory: simply add the file/directory to the specified location;
2. Modify a file/directory: similarly, overwrite the same file/directory with the new content;
3. Delete a file: use a special whiteout marker file with a .wh. prefix. For example, a tar entry named ./foo/.wh.bar.txt means deleting the ./foo/bar.txt file.
4. Delete a directory: use a opaque whiteout marker file named .wh..wh.opq. For example, a tar entry named ./foo/.wh..wh.opq means deleting the ./foo/ directory.

By applying each tar file from base layer to the last child layer based on 1-4, we can get a complete file system specified by an image.

image configuration

application/vnd.oci.image.config.v1+json media type.

With an image manifest, we can identify a single image (layers + config) specific to an arch+OS; with the layers, we can construct a complete file system from the image that can be used by a container. However, we still miss metadata that is necessary to launch a container, such as environment variables or default arguments (uid, volumes, etc).

Image configuration (also called Image JSON in the spec) is a JSON object that contains all these metadata for an image. It also defines the order of layers used to construct the file system.

First let’s explain some concepts that are useful to understand the configuration:

Image ID: the digest of an image configuration JSON object. The image ID makes images content-addressable, because the configuration JSON object contains the digests (actually DiffID) of each image layer.

Layer DiffID: the digest of a layer’s uncompressed tar archive. It’s different from layer digest, which is the digest of a layer’s compressed or uncompressed content depending the layer’s format (tar, gzip, zstd).

Layer ChainID: the digest of a stack of ordered/chained layers. It’s calculated as follows:

ChainID(L_0) = DiffID(L_0)
ChainID(L_i) = Digest(ChainID(L_i-1) + " " + DiffID(L_i))

Why we need both ChainID and DiffID? Because DiffID only identifies a single layer/changeset, with no information about it’s parent/child layers. If we use DiffID only, an attacker can replace a layer’s parent layer with a malicious layer without changing the DiffID. ChainID can resolve this because it’s a hash of a unique stack of layers applied on top of each other.

Now let’s look at the configuration JSON object. The following fields worth mentioning:

1. architecture and os: both are strings specifying the arch+OS of the image;
2. config an optional JSON object (don’t confused with the whole configuration JSON object) containing the base execution parameters when running a container. Some example fields: a. User string: username/UID used to run the container; b. Env string array: environment variables; c. Labels object: arbitrary metadata for the container;
3. rootfs object: rootfs.diff_ids contains an array of layer DiffIDs from base layer to the top layer, making the image config hash depend on the file system / layers;

manifest v.s. artifact v.s. descriptor v.s. digest

Being a non English-native speaker, I previously found these terms confusing (mostly due to my weak vocubulary 🥲). So I just quickly summarize them here, mainly for my own reference :)

1. manifest: a JSON object that describes an image or artifact; in most cases, it refers an image manifest.
2. artifact: an arbitrary data blob that can be referenced by a manifest, and is content-addressable. We can say an image is a special type of artifact. Other types of artifacts include image signature, SBOM, helm chart, etc.
3. descriptor: a JSON object describing the reference relationship between components (e.g. image manifest -> config, image manifest -> layer, etc). It includes information to identify the referenced component, such as media type, digest, size, etc.
4. digest: a string (hash_algorithm:hash_value) that uniquely identifies a specific content/bytes, calcuated from the content using the given hash algorithm.

image layout

Image layout defines the the layout (i.e., the directory extracted from an OCI image tar file) of an OCI image. Given an image layout and an image ref (e.g. an image id), an OCI runtime bundle can be created following:

1. Follow the ref to find a manifest;
2. Apply the layers in the specified order (by rootfs.diff_ids in image-spec config) to get the file system;
3. Convert image-spec config into an runtime-spec bundle config.json.

The image layout must contains:

1. blobs directory: contains all content-addressable blobs. Each hash algorithm has its own sub-directory, within which are the blobs. Each blob file is named to its content hash value. So given a digest <alg>:<hash>, we can find the blob file at blobs/<alg>/<hash>.
2. oci-layout file: a JSON object (application/vnd.oci.layout.header.v1+json media type). It marks and versions an OCI layout.
3. index.json file: an image index acting as an entry point to the image. For example, given an image id (ref), we can find its image manifest from index.json’s .manifests field.

An image layout example

Let’s look at a real image and its layout. I’ll use the ubuntu:latest image and docker save for convenience to get its content since I’m using a macbook.

First let’s pull and save the image, and extract the image tar file to a directory:

$ docker pull ubuntu
$ docker save ubuntu -o ubuntu.tar
$ tar -C ./ubuntu -xvf ubuntu.tar
$ cd ubuntu

Then let’s check all the files in the image directory:

$ find . -type f
./oci-layout
./blobs/sha256/537da24818633b45fcb65e5285a68c3ec1f3db25f5ae5476a7757bc8dfae92a3
./blobs/sha256/bab8ce5c00ca3ef91e0d3eb4c6e6d6ec7cffa9574c447fd8d54a8d96e7c1c80e
./blobs/sha256/cd741b12a7eaa64357041c2d3f4590c898313a7f8f65cd1577594e6ee03a8c38
./blobs/sha256/f2ef2e10c68c0d0c7a97d0c45076cd9549ac25cef6647b3a28487131066c406c
./manifest.json
./index.json

From the output, we can validate that the image layout contains required blobs directory and index.json and oci-layout files. And the blobs directory is organized by hash algorithm as sub-directories and hash value as file names.

There is an extra file, manifest.json, which based on my understanding is not part of OCI image spec and only used by docker (i.e., it’s part of docker image spec instead).

Now let’s check the oci-layout and index.json files:

# oci-layout only defines layout version
$ cat oci-layout | jq
{
  "imageLayoutVersion": "1.0.0"
}
$ cat index.json | jq
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "digest": "sha256:f2ef2e10c68c0d0c7a97d0c45076cd9549ac25cef6647b3a28487131066c406c",
      "size": 304,
      "annotations": {
        "io.containerd.image.name": "docker.io/library/ubuntu:latest",
        "org.opencontainers.image.ref.name": "latest"
      }
    }
  ]
}

The index.json contains only one manifest which is an image index. Let’s find the index by following the blobs/<alg>/<hash> path based its digest:

$ cat blobs/sha256/f2ef2e10c68c0d0c7a97d0c45076cd9549ac25cef6647b3a28487131066c406c | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:537da24818633b45fcb65e5285a68c3ec1f3db25f5ae5476a7757bc8dfae92a3",
      "size": 424,
      "platform": {
        "architecture": "arm64",
        "os": "linux",
        "variant": "v8"
      }
    }
  ]
}

This image index, similar to index.json, points to a single image manifest. Doing the same, we have:

$ cat blobs/sha256/537da24818633b45fcb65e5285a68c3ec1f3db25f5ae5476a7757bc8dfae92a3 | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "size": 2316,
    "digest": "sha256:bab8ce5c00ca3ef91e0d3eb4c6e6d6ec7cffa9574c447fd8d54a8d96e7c1c80e"
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 27347481,
      "digest": "sha256:cd741b12a7eaa64357041c2d3f4590c898313a7f8f65cd1577594e6ee03a8c38"
    }
  ]
}

As expected, the image manifest contains a config descriptor and a list of layer descriptors. Let’s check the config JSON object:

# I removed many details in the output for simplicity
$ cat blobs/sha256/bab8ce5c00ca3ef91e0d3eb4c6e6d6ec7cffa9574c447fd8d54a8d96e7c1c80e | jq
{
  "architecture": "arm64",
  "config": {
    "User": "",
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "/bin/bash"
    ],
    "Image": "sha256:e7deb3a6ff503af01d302e3ec352370e33ecb2cc064f08d2ca40c87ec02aa227",
  },
  "history": [
    {
      "created": "2023-03-08T04:32:38.832581437Z",
      "created_by": "/bin/sh -c #(nop)  ARG RELEASE",
      "empty_layer": true
    },
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:874b048c963ab55b06939c39d59303fb975d323822a4ea48a02ac8dc635ea371"
    ]
  },
  "variant": "v8"
}

From the output, we can see there is one layer DiffID in .rootfs.diff_ids which matches length of .layers in the image manifest. Notice for this layer, the layer digest is sha256:cd741b12a7eaa64357041c2d3f4590c898313a7f8f65cd1577594e6ee03a8c38 and the DiffID is sha256:874b048c963ab55b06939c39d59303fb975d323822a4ea48a02ac8dc635ea371.

Now let’s take a look at the layer and validate that the layer digest indeed matches its content (the same applies to other objects as well), and the layer DiffID matches the content of the uncompressed layer tar file:

# layer digest matches layer blob file
$ shasum -a 256 ./blobs/sha256/cd741b12a7eaa64357041c2d3f4590c898313a7f8f65cd1577594e6ee03a8c38
cd741b12a7eaa64357041c2d3f4590c898313a7f8f65cd1577594e6ee03a8c38  ./blobs/sha256/cd741b12a7eaa64357041c2d3f4590c898313a7f8f65cd1577594e6ee03a8c38
# layer DiffID matches the uncompressed layer tar file
$ gunzip -c ./blobs/sha256/cd741b12a7eaa64357041c2d3f4590c898313a7f8f65cd1577594e6ee03a8c38 | shasum -a 256
874b048c963ab55b06939c39d59303fb975d323822a4ea48a02ac8dc635ea371  -

So far, we checked 4 blobs: image index, image manifest, image config, and layer. This is all blob files in blobs/sha256 directory. In some cases, blobs may contain blob file not referenced by any descriptor.

$ find ./blobs/sha256 -type f
./blobs/sha256/537da24818633b45fcb65e5285a68c3ec1f3db25f5ae5476a7757bc8dfae92a3
./blobs/sha256/bab8ce5c00ca3ef91e0d3eb4c6e6d6ec7cffa9574c447fd8d54a8d96e7c1c80e
./blobs/sha256/cd741b12a7eaa64357041c2d3f4590c898313a7f8f65cd1577594e6ee03a8c38
./blobs/sha256/f2ef2e10c68c0d0c7a97d0c45076cd9549ac25cef6647b3a28487131066c406c

From this example, we can see how we can find all the blobs (image manifest, config, layer, etc) from a given image ref and how we can validate the integrity of each blob based on its digest.