cocos/agent/TESTING_REMOTE_RESOURCES.md

Testing Remote Resources with CoCo Key Provider

This guide explains how to test Cocos with encrypted remote resources using the Confidential Containers Key Provider ecosystem.

Architecture Overview

┌────────────────────────────────────────────────────────────┐
│                         CVM (Agent)                        │
│                                                            │
│  ┌──────────┐    ┌────────────────┐    ┌─────────────────┐ │
│  │  Agent   │──▶│    Skopeo      │──▶│ CoCo Keyprovider│ │
│  │          │    │  (ocicrypt)    │    │   (gRPC:50011)  │ │
│  │          │    └───────┬────────┘    └────────┬────────┘ │
│  │          │            │                      │          │
│  │          │    ┌───────▼────────┐    ┌────────▼────────┐ │
│  │          │──▶│  S3/HTTP       │    │ Attestation     │ │
│  │          │    │  Downloader    │    │ Agent (50002)   │ │
│  └────┬─────┘    └───────┬────────┘    └────────┬────────┘ │
│       │                  │                      │          │
│       └──────────────────┼──────────────────────┘          │
└────────┬─────────────────┼──────────────────────┬──────────┘
         │ (Resource)      │ (Resource)           │ (Attest)
         ▼                 ▼                      ▼
    OCI Registry       S3 / HTTP / GCS           KBS
                                             (Key Broker)

Prerequisites

1. Install Skopeo (Host Machine)

# Ubuntu/Debian
sudo apt-get install skopeo

# macOS
brew install skopeo

# Or build from source
git clone https://github.com/containers/skopeo
cd skopeo
make bin/skopeo
sudo make install

2. Start KBS Server (Host Machine)

# Clone and build KBS
git clone https://github.com/confidential-containers/trustee
cd trustee/kbs
# Patch Cargo.toml to disable SGX requirement (for testing only)
sed -i 's/"all-verifier",//g' Cargo.toml

make
make cli

# Generate admin keys
openssl genpkey -algorithm ed25519 -out kbs-admin.key
openssl pkey -in kbs-admin.key -pubout -out kbs-admin.pub

# Create KBS configuration file
cat > kbs-config.toml << 'EOF'
[http_server]
sockets = ["0.0.0.0:8080"]
insecure_http = true

[admin]
type = "Simple"
[[admin.personas]]
id = "admin"
public_key_path = "kbs-admin.pub"

[attestation_service]
type = "coco_as_builtin"
work_dir = "kbs-data/as"

[attestation_service.rvps_config]
type = "BuiltIn"

[attestation_service.rvps_config.storage]
type = "LocalFs"
file_path = "kbs-data/rvps-values"

[[plugins]]
name = "resource"
type = "LocalFs"
dir_path = "kbs-data/repository"
EOF

# Create configuration directories
mkdir -p kbs-data/as kbs-data/rvps kbs-data/repository

# Start KBS
sudo ../target/release/kbs --config-file kbs-config.toml

KBS will listen on http://localhost:8080

3. Setup Local OCI Registry (Optional)

For testing, you can use a local registry:

docker run -d -p 5000:5000 --name registry registry:2

Creating Encrypted Resources

Encrypt an Algorithm (Python Script)

# 1. Create a simple algorithm
cat > lin_reg.py << 'EOF'
import pandas as pd
from sklearn.linear_model import LinearRegression
import sys
import os

# Load dataset
data = pd.read_csv(sys.argv[1])
X = data[['feature1', 'feature2']]
y = data['target']

# Train model
model = LinearRegression()
model.fit(X, y)

# Save results
os.makedirs("results", exist_ok=True)
with open("results/output.txt", "w") as f:
    f.write(f"Coefficients: {model.coef_}\n")
    f.write(f"Intercept: {model.intercept_}\n")

print(f"Coefficients: {model.coef_}")
print(f"Intercept: {model.intercept_}")
EOF

# 2. Create requirements.txt
cat > requirements.txt << 'EOF'
pandas
scikit-learn
EOF

# 3. Create a Dockerfile
cat > Dockerfile << 'EOF'
FROM python:3.9-slim
RUN pip install pandas scikit-learn
COPY lin_reg.py /app/algorithm.py
COPY requirements.txt /app/requirements.txt
WORKDIR /app
ENTRYPOINT ["python", "algorithm.py"]
EOF

# 4. Build the image
docker build -t localhost:5000/lin-reg-algo:v1.0 .
docker push localhost:5000/lin-reg-algo:v1.0

# 5. Generate and store key
openssl rand -out algo.key 32

# 6. Store key in KBS using kbs-client
../target/release/kbs-client --url http://localhost:8080 config \
  --auth-private-key kbs-admin.key \
  set-resource \
  --path default/key/algo-key \
  --resource-file algo.key

# 7. Encrypt the image using Host Skopeo + Docker Keyprovider
# Start Keyprovider in background
docker run -d --rm --name keyprovider --network host \
  -v "$PWD:/work" -w /work \
  ghcr.io/confidential-containers/staged-images/coco-keyprovider:latest \
  coco_keyprovider --socket 127.0.0.1:50000

# Configure Ocicrypt to use local Keyprovider
cat <<EOF > ocicrypt.conf
{
  "key-providers": {
    "attestation-agent": {
      "grpc": "127.0.0.1:50000"
    }
  }
}
EOF
export OCICRYPT_KEYPROVIDER_CONFIG=$(pwd)/ocicrypt.conf

# Encrypt Algo
skopeo copy \
  --src-tls-verify=false \
  --dest-tls-verify=false \
  --encryption-key "provider:attestation-agent:keypath=/work/algo.key::keyid=kbs:///default/key/algo-key::algorithm=A256GCM" \
  docker://localhost:5000/lin-reg-algo:v1.0 \
  docker://localhost:5000/encrypted-lin-reg:v1.0

# Stop Keyprovider
docker stop keyprovider

Encrypt a Dataset (CSV in OCI Image)

# 1. Create dataset
cat > iris.csv << 'EOF'
feature1,feature2,target
5.1,3.5,0
4.9,3.0,0
6.2,3.4,1
5.9,3.0,1
EOF

# 2. Create Dockerfile for dataset
cat > Dockerfile.dataset << 'EOF'
FROM scratch
COPY iris.csv /data/iris.csv
EOF

# 3. Build and push
docker build -f Dockerfile.dataset -t localhost:5000/iris-dataset:v1.0 .
docker push localhost:5000/iris-dataset:v1.0

# 4. Generate and store key
# 4. Generate and store key
openssl rand -out dataset.key 32
../target/release/kbs-client --url http://localhost:8080 config \
  --auth-private-key kbs-admin.key \
  set-resource \
  --path default/key/dataset-key \
  --resource-file dataset.key

# 5. Encrypt dataset image using Host Skopeo + Docker Keyprovider
# Start Keyprovider in background
docker run -d --rm --name keyprovider --network host \
  -v "$PWD:/work" -w /work \
  ghcr.io/confidential-containers/staged-images/coco-keyprovider:latest \
  coco_keyprovider --socket 127.0.0.1:50000

# Configure Ocicrypt (if not already done)
export OCICRYPT_KEYPROVIDER_CONFIG=$(pwd)/ocicrypt.conf

# Encrypt Dataset
skopeo copy \
  --src-tls-verify=false \
  --dest-tls-verify=false \
  --encryption-key "provider:attestation-agent:keypath=/work/dataset.key::keyid=kbs:///default/key/dataset-key::algorithm=A256GCM" \
  docker://localhost:5000/iris-dataset:v1.0 \
  docker://localhost:5000/encrypted-iris:v1.0

# Stop Keyprovider
docker stop keyprovider

Running a Computation

1. Start Manager (Host)

cd /path/to/cocos-ai
./build/cocos-manager

2. Start CVMS Test Server (Host)

Get your host IP:

HOST_IP=$(ip -4 addr show | grep -oP '(?<=inet\s)\d+(\.\d+){3}' | grep -v 127.0.0.1 | head -n1)

Start CVMS server:

# Calculate SHA3-256 of decrypted files using cocos-cli or cvms-test
# NOTE: We use the hash of the original plaintext files, as the Agent validates the decrypted content.
# For single files, use the file hash. For directories, use the hash of the directory (which the tools zip deterministically).

ALGO_HASH=$(./build/cocos-cli checksum lin_reg.py 2>&1 | awk '{print $NF}')

DATASET_HASH=$(./build/cocos-cli checksum iris.csv 2>&1 | awk '{print $NF}')

go build -o build/cvms-test ./test/cvms/main.go
HOST=$HOST_IP PORT=7001 ./build/cvms-test \
  -public-key-path ./public.pem \
  -attested-tls-bool false \
  -algo-type python \
  -algo-source-url docker://$HOST_IP:5000/encrypted-lin-reg:v1.0 \
  -algo-kbs-path default/key/algo-key \
  -algo-kbs-url http://$HOST_IP:8080 \
  -algo-hash $ALGO_HASH \
  -algo-args datasets/dataset_0.csv \
  -dataset-source-urls docker://$HOST_IP:5000/encrypted-iris:v1.0 \
  -dataset-kbs-paths default/key/dataset-key \
  -dataset-kbs-urls http://$HOST_IP:8080 \
  -dataset-hash $DATASET_HASH

Note

You must specify the KBS URL for each encrypted resource using -algo-kbs-url and -dataset-kbs-urls. A global KBS is no longer supported.

3. Create VM via CLI (Host)

export MANAGER_GRPC_URL=localhost:7002
./build/cocos-cli create-vm \
  --server-url $HOST_IP:7001 \
  --log-level debug

The agent will:

1. Receive computation manifest from CVMS
2. Use Skopeo to download encrypted OCI images
3. Skopeo invokes CoCo Keyprovider via ocicrypt
4. CoCo Keyprovider requests decryption key from KBS
5. Attestation Agent generates TEE evidence for KBS
6. KBS validates evidence and returns decryption key
7. Image layers are decrypted and extracted
8. Computation executes with decrypted algorithm and dataset

Verifying the Setup

Check CoCo Keyprovider Status (Inside CVM)

# SSH into CVM or use console
systemctl status coco-keyprovider
journalctl -u coco-keyprovider -f

Check Attestation Agent Status

systemctl status attestation-agent
journalctl -u attestation-agent -f

Test Skopeo Decryption Manually

# Inside CVM
export OCICRYPT_KEYPROVIDER_CONFIG=/etc/ocicrypt_keyprovider.conf

skopeo copy \
  --src-tls-verify=false \
  --dest-tls-verify=false \
  --decryption-key provider:attestation-agent:cc_kbc::null \
  docker://localhost:5000/encrypted-lin-reg:v1.0 \
  oci:/tmp/decrypted-algo

# Verify decryption
skopeo inspect oci:/tmp/decrypted-algo | jq -r '.LayersData[].MIMEType'
# Should show: application/vnd.oci.image.layer.v1.tar+gzip

Computation Manifest Format

The CVMS server sends this manifest to the agent:

{
  "computation_id": "1",
  "algorithm": {
    "type": "oci-image",
    "uri": "docker://localhost:5000/encrypted-lin-reg:v1.0",
    "encrypted": true,
    "kbs_resource_path": "default/key/algo-key",
    "kbs": {
      "url": "http://192.168.100.15:8080",
      "enabled": true
    }
  },
  "datasets": [
    {
      "filename": "iris.csv",
      "source": {
        "type": "oci-image",
        "url": "docker://localhost:5000/encrypted-iris:v1.0",
        "encrypted": true,
        "kbs_resource_path": "default/key/dataset-key"
      },
      "kbs": {
        "url": "http://192.168.100.20:8080",
        "enabled": true
      }
    }
  ],
  "kbs": {
    "url": "http://192.168.100.15:8080",
    "enabled": true
  }
}

Troubleshooting

CoCo Keyprovider Not Starting

# Check logs
journalctl -u coco-keyprovider -n 50

# Verify socket is listening
ss -tlnp | grep 50011

# Check environment
cat /etc/default/coco-keyprovider

Skopeo Decryption Fails

# Verify ocicrypt config
cat /etc/ocicrypt_keyprovider.conf

# Test keyprovider connection
grpcurl -plaintext 127.0.0.1:50011 list

# Check KBS connectivity from CVM
curl http://HOST_IP:8080/kbs/v0/auth

KBS Returns 401

# Check KBS logs on host
# Verify attestation evidence format
# Ensure KBS is configured for sample attestation

4. Testing with Non-OCI Sources (S3, HTTP, GCS)

The cvms test utility also supports testing remote encrypted resources hosted in more traditional environments like S3-compatible storage or simple web servers, bypassing the need for container registries and OCI images.

Supported Flags

The following flags define how resources should be fetched:

• --algo-source-url: The URL of the algorithm (e.g. s3://bucket/algo.bin, https://server/algo.bin)
• --algo-source-type: The type of remote endpoint (s3, gcs, https, http). If omitted, it will automatically be inferred from the URL scheme.
• --algo-kbs-path: The KBS path to retrieve the AES-256-GCM key from. If present, the agent will attempt decryption.
• --dataset-source-urls and --dataset-source-type: Defines the locations and protocols for datasets.

Encryption Format for Non-OCI Sources

Unlike OCI images where ocicrypt wraps the dataset, resources hosted on HTTP/S3 must be straightforwardly encrypted using AES-256-GCM.

The expected format is exactly as produced by standard Go AES-GCM: nonce (12 bytes) || ciphertext || tag

Test Example

If you had a Python script encrypted using a key hosted at KBS path default/my-keys/python-script and uploaded to s3://my-secure-bucket/script.enc, you could run:

cd test
go run cvms/main.go --algo-source-url="s3://my-secure-bucket/script.enc" \
                    --algo-source-type="s3" \
                    --algo-kbs-path="default/my-keys/python-script" \
                    --algo-type="python" \
                    --public-key-path=./test-data/public-key.pem

The system will:

1. Connect via attestation-agent to the KBS to retrieve the symmetric key
2. Use Google Cloud Storage client library methods (support for generic S3 via environment variables is standard) to fetch the resource
3. Decrypt using AES-256-GCM
4. Run the code normally

---