* Update protobuf generated files and improve algorithm handling in runner service - Bump protoc version from v7.35.0 to v7.35.1 in generated protobuf files. - Refactor RunRequest message in runner.proto to use string paths for algorithm and requirements instead of byte arrays. - Update runner service to handle algorithm and requirements paths, removing the need for temporary file creation. - Enhance error handling for missing algorithm paths in the runner service. - Modify tests to align with the new RunRequest structure and ensure proper file handling. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * chore: update protoc version to 35.1 in CI workflow Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: update systemd service configurations and dependencies for improved service management Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix: adjust AlgoWorkingDir handling in tests for Docker algorithm Signed-off-by: Sammy Oina <sammyoina@gmail.com> --------- Signed-off-by: Sammy Oina <sammyoina@gmail.com>
Disk Image Workflow
This directory is the Buildroot external tree for the current Cocos disk test VM image and its runtime configuration.
Layout
- configs/cocos_defconfig: Buildroot configuration for the bootable image.
- board/rootfs-overlay/init:
early initramfs script that provisions
/cocos, mounts the real root, and switches into the installed system. - board/cocos/genimage.cfg:
GPT disk layout for the final
disk.img. - board/cocos/post-image.sh:
builds the minimal initramfs, stages EFI files, signs boot artifacts, and
assembles
disk.img. - external.desc: Buildroot external tree descriptor.
- external.mk: includes package makefiles from
package/*.
Current Buildroot Image
The current Buildroot flow produces a bootable GPT disk image:
efipartition: FAT EFI system partition with GRUB, kernel, and initramfsrootpartition: ext4 root filesystem protected by dm-verityveritypartition: dm-verity hash tree for the root filesystemcocospartition: blank partition provisioned at boot as an encrypted ext4 filesystem mounted at/cocos
The final image is written to:
output/images/disk.img
The root filesystem image is also available separately as:
output/images/rootfs.ext4
Current Boot Flow
At boot, GRUB loads:
bzImageinitrd.cpio.gz
The initramfs script in board/rootfs-overlay/init then:
- mounts
/proc,/sys,devtmpfs, anddevpts - assumes the boot disk is
/dev/sda - opens a dm-verity mapping for the root filesystem using:
/dev/sda2as the data partition/dev/sda3as the verity hash partitionroothash=from the kernel command line
- mounts
/dev/mapper/root_verityread-only at/root - generates a fresh ephemeral key
- formats
/dev/sda4as LUKS2 - opens it as
/dev/mapper/cocos_crypt - formats that mapper as ext4 and mounts it at
/root/cocos - creates working directories on
/cocos, including:/cocos/.cache/oci/cocos/datasets/cocos/docker/cocos/cocos_init
- mounts
tmpfson/tmpand/varbecause the root filesystem is intentionally read-only - bind-mounts
/cocos/dockeronto/var/lib/docker - bind-mounts
/cocos/cocos_initonto/cocos_init - rewrites
/etc/fstabin the mounted root to describe the live runtime - preserves or adds 9P mounts for:
certs_share->/etc/certsenv_share->/etc/cocos
- securely wipes the temporary LUKS key file
- runs
switch_root /root /sbin/init
Important details:
- the root filesystem is verified through dm-verity before it is mounted
/cocosis encrypted with an ephemeral per-boot key- that key is not persisted, so
/cocosis provisioned fresh on each boot
Runtime Filesystem Model
The running system is split into:
- read-only root on
/ - encrypted writable storage on
/cocos tmpfson/tmptmpfson/var
Service state that must survive within a boot session is redirected away from the read-only root:
- Docker data lives on
/cocos/docker - agent setup scripts work through
/cocos_init, which is backed by/cocos/cocos_init - algorithm datasets and results live under
/cocos
This means services can use /cocos like a regular directory tree after boot,
even though it is backed by an encrypted mapper created in early userspace.
systemd Runtime Expectations
Several services depend on files mounted from 9P shares under /etc/certs and
/etc/cocos. To avoid boot-order races, the rootfs overlay includes systemd
drop-ins under:
board/rootfs-overlay/usr/lib/systemd/system/*service.d/
These drop-ins require the relevant mount points before starting services such as:
egress-proxy.servicelog-forwarder.servicecomputation-runner.servicecocos-agent.service
The overlay also ships tmpfiles rules in board/rootfs-overlay/usr/lib/tmpfiles.d/cocos.conf to create:
/var/log/cocos/run/cocos
Agent Packaging In Buildroot
The Buildroot agent package is wired to build the binary from the local Cocos
checkout, not only from a downloaded release snapshot. The package definition is
in package/agent/agent.mk.
That package currently:
- builds
cocos-agentfrom the local source tree - installs the local cocos-agent.service
- installs the local agent_setup.sh
- installs the local agent_start_script.sh
So changes under:
cocos/agent/...cocos/init/systemd/...
are intended to be picked up by the next Buildroot rebuild.
Buildroot Packages And Tools
The current cocos_defconfig includes the components needed by the boot flow
and runtime image, including:
- systemd
- DHCP client
cryptsetupeudeve2fsprogs- Docker, containerd, and runc
skopeo- TPM2 tools
- 9P filesystem support
- GRUB2 EFI boot support
- host
genimage
The initramfs built in post-image.sh is intentionally minimal and contains
only the binaries needed for early boot, dm-verity root verification, and
/cocos provisioning.
Secure Boot Notes
During post-image.sh:
- GRUB is rebuilt with
--disable-shim-lock bootx64.efiandbzImageare signed with the configured Secure Boot keys when those keys are present
This flow is designed for booting directly through OVMF with your own enrolled
keys. It does not currently rely on booting through shim.
Rebuilding
This directory is meant to be used as a Buildroot external tree. From this directory, configure a Buildroot checkout with:
make -C /path/to/buildroot BR2_EXTERNAL=$PWD cocos_defconfig
Then build with:
make -C /path/to/buildroot BR2_EXTERNAL=$PWD -j$(nproc)
The resulting boot image is:
/path/to/buildroot/output/images/disk.img
Additional generated artifacts include:
/path/to/buildroot/output/images/rootfs.ext4
/path/to/buildroot/output/images/rootfs.verity
/path/to/buildroot/output/images/rootfs.roothash