opensource/cocos
1
0
Fork 0
mirror of https://github.com/ultravioletrs/cocos.git synced 2026-06-23 04:10:25 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
597 Commits 18 Branches 17 Tags
da31d76c94809a7296558a112d900b10aeefdfd9
Commit Graph

39 Commits

Author SHA1 Message Date
Sammy Kerata Oina da31d76c94 NOISSUE - Agent Pull mode for remote resources (#575)
CI / checkproto (push) Has been cancelled
Details
CI / lint (push) Has been cancelled
Details
Rust CI Pipeline / rust-check (push) Has been cancelled
Details
CI / test (agent) (push) Has been cancelled
Details
CI / test (cli) (push) Has been cancelled
Details
CI / test (cmd) (push) Has been cancelled
Details
CI / test (internal) (push) Has been cancelled
Details
CI / test (manager, true) (push) Has been cancelled
Details
CI / test (pkg) (push) Has been cancelled
Details
CI / upload-coverage (push) Has been cancelled
Details 
* feat(kbs): implement KBS client for attestation and resource retrieval

- Added KBS client implementation in pkg/kbs/client.go with methods for attestation and resource retrieval.
- Introduced necessary data structures for requests and responses.
- Implemented error handling for various scenarios.

test(kbs): add unit tests for KBS client

- Created comprehensive tests for the KBS client in pkg/kbs/client_test.go.
- Included tests for attestation success and failure cases, as well as resource retrieval.

feat(registry): introduce HTTP and S3 registry implementations

- Added HTTPRegistry for downloading resources over HTTP/HTTPS with retry logic in pkg/registry/http.go.
- Implemented S3Registry for downloading resources from AWS S3 and S3-compatible services in pkg/registry/s3.go.
- Included error handling and configuration options for both registries.

chore(registry): define registry interface and configuration

- Created registry interface and configuration struct in pkg/registry/registry.go.
- Added default configuration settings for registry clients.

docs(cvms): update README for CVMS server configuration and usage

- Enhanced documentation for CVMS server with detailed command-line flags and usage examples.
- Clarified direct upload and remote resource modes, including KBS integration.

fix(cvms): integrate KBS for remote resource handling in main.go

- Updated main.go to support remote datasets and algorithms using KBS.
- Added validation for command-line flags to ensure proper configuration.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fix: Move ifeq conditional outside define block in attestation-service.mk

Make conditionals cannot be evaluated inside define...endef blocks
when used as recipe bodies. Restructured to define the
ATTESTATION_SERVICE_INSTALL_INIT_SYSTEMD block conditionally based
on BR2_PACKAGE_CC_ATTESTATION_AGENT configuration.

* feat: Implement remote resource downloading for algorithms and datasets using AWS S3/MinIO credentials.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: Add comprehensive documentation and agent support for testing remote resource download with KBS attestation.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: Improve agent logging for remote resource configuration and KBS status, and add a testing guide for remote resource downloads with KBS attestation.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: Add a comprehensive guide for testing remote resource download with KBS attestation and update multiple package versions to a specific commit.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: Add failure transitions for resource reception states and a comprehensive guide for testing remote resource downloads with KBS attestation.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: Implement remote resource download with KBS attestation in the agent and add a comprehensive testing guide.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* test: Add comprehensive guide for testing remote resource download with KBS attestation and include a debug log in the attestation client.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: Delegate KBS attestation and token retrieval to a new attestation-agent service and document remote resource testing.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* client fixes

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* raw evidence

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fix: Build all Go files in cmd directories, not just main.go

This fixes the issue where fetch_raw_evidence.go wasn't being included
in the attestation-service build.

* fix: Wrap binary evidence in JSON for KBS compatibility

Fixes 'invalid character' error by wrapping raw binary evidence
in a JSON structure with base64 encoding, as expected by KBS.

* chore: Update buildroot packages to c28cefae

Includes fixes for:
1. attestation-service build (including fetch_raw_evidence.go)
2. Agent KBS evidence format (wrapping binary in JSON)

* fix: Implement KBS RCAR handshake with cookies

Fixes 'cookie not found' error (401) from KBS by:
1. Adding CookieJar support to KBS client
2. Implementing GetChallenge() to perform /auth handshake and capture session cookie
3. Updating Agent to get challenge, decode nonce, and use it for evidence generation
4. Regenerating mocks

* chore: Update buildroot packages to f6981ac5

Includes KBS RCAR handshake fix (cookie support + GetChallenge loop)

* fix: Update KBS client JSON tags to kebab-case

Fixes deserialization error (401) from KBS by:
1. Using kebab-case (e.g. extra-params) for JSON tags as per protocol.
2. Initializing ExtraParams as empty object {} instead of null/omitted.

* fix: Wrap attestation evidence in primary_evidence format

Updates Agent to construct 'tee-evidence' payload with:
- primary_evidence: containing the actual quote/data
- additional_evidence: empty JSON object

This matches the Confidential Containers KBS Attestation Protocol requirements.

* fix: Update KBS protocol version to 0.4.0

KBS rejected 0.1.0 with a version mismatch error. Bumping to 0.4.0 to match server expectation.

* fix: Generate ephemeral key for KBS RuntimeData

Updates RuntimeData to include a valid ephemeral EC P-256 public key in JWK format, as required by the KBS RCAR protocol.
Also fixes the KBS client struct to support TEEPubKey as an object.

* fix: Update sample attestation quote to valid JSON

The default attestation.bin was binary, but the KBS Sample Verifier expects a valid JSON quote containing 'svn' and 'report_data'.
Updated the embedded bin file to contain this JSON structure.

* fix: Generate dynamic JSON quote for Sample TEE in FetchRawEvidence

The KBS Sample Verifier expects a JSON object with 'svn' and 'report_data'.
Previously, we were returning raw binary data (reportData+nonce).
This commit updates FetchRawEvidence to return a marshaled JSON structure with:
- svn: "1"
- report_data: base64(req.ReportData)

* refactor: Delegate Sample Attestation to Provider

Refactored sample attestation logic:
- Moved JSON Quote generation into EmptyProvider (standalone mode).
- Updated FetchRawEvidence to call provider.TeeAttestation instead of manual generation.
This enables using the real CC Attestation Agent for UNSPECIFIED platform if configured.

* feat: Add comprehensive debug logging and enforce CC AA usage

Changes:
- Updated EmptyProvider to return error instead of generating mock data
  This forces proper use of CC Attestation Agent's sample attester
- Added detailed logging to attestation-service FetchRawEvidence:
  * Hex dump of evidence (first 200 bytes)
  * String preview of evidence
  * Total evidence length
- Added detailed logging to agent service:
  * Raw evidence hex and string previews
  * KBS evidence JSON preview (first 500 bytes)
  * Evidence lengths at each transformation step

This logging will help diagnose why KBS Sample Verifier is rejecting evidence.

* fix: Enable CC AA by default and add attestation-service log forwarding

Changes:
- Set USE_CC_ATTESTATION_AGENT=true by default in systemd service
- Added StandardOutput/StandardError to forward logs to /var/log/cocos/
- Updated HAL makefile to handle new default value
- This ensures attestation-service uses CC AA's sample attester
- Logs will now be visible in CVMS output for debugging

* feat: Add gRPC log forwarding to attestation-service

Implemented the same log forwarding mechanism used by the agent:
- Added ProtoHandler to write logs to both stdout and logQueue
- Connected to log client (/run/cocos/log.sock) for gRPC forwarding
- Added goroutine to forward logs to CVMS via log client
- Logs will now appear in CVMS output during computation runs

This enables visibility into attestation-service debug output including:
- CC AA connection status
- Evidence generation details (hex dumps, string previews)
- Any errors from providers

* fix: Parse sample evidence JSON instead of base64-encoding it

The attestation-service returns sample evidence as JSON:
{"svn":"1","report_data":"base64..."}

The agent was incorrectly base64-encoding this JSON string again.
KBS Sample Verifier expects the parsed JSON object directly.

Fixed by:
- Parsing the JSON evidence from attestation-service
- Passing the parsed object directly in primary_evidence.evidence
- This matches what KBS Sample Verifier expects

* debug: Increase KBS evidence logging preview to 1000 bytes

Show the complete JSON structure being sent to KBS to debug
the attestation failure.

* debug: Add comprehensive CC AA configuration logging

Added debug logs to show:
- Whether CC AA is enabled in config
- CC AA address being used
- Connection success/failure
- Which provider is ultimately selected
- Warning when falling back to EmptyProvider

This will help diagnose why EmptyProvider is being used
instead of CC Attestation Agent.

* debug: Add startup logging for log client connection

Added log message to show if log client connection succeeds
at attestation-service startup. This will help diagnose why
logs aren't appearing in CVMS output.

* feat: Add retry logic with exponential backoff to log client

Added simple retry mechanism to handle concurrent log requests:
- 3 retry attempts with exponential backoff (10ms, 20ms, 40ms)
- Applies to both SendLog and SendEvent methods
- Centralized in log client so all services benefit
- Should eliminate 'failed to send log' errors from concurrent requests

This fixes the issue where attestation-service logs weren't
appearing in CVMS output due to dropped messages.

* fix: Flatten sample evidence fields in primary_evidence for KBS

KBS Sample Verifier expects svn and report_data at the top level
of primary_evidence, not nested under an 'evidence' key.

Changed structure from:
{"primary_evidence": {"tee": "sample", "evidence": {"svn": "1", ...}}}

To:
{"primary_evidence": {"tee": "sample", "svn": "1", "report_data": "...", ...}}

This matches what KBS expects when deserializing the Quote structure.

* fix: Use sample quote directly as primary_evidence per KBS protocol

According to KBS attestation protocol spec, for sample TEE type,
primary_evidence should be the sample quote JSON directly:
{"svn": "1", "report_data": "..."}

Removed extra 'tee' and 'platform' fields that were causing KBS
to fail deserializing the Quote structure. The 'tee' field is
already sent in the Request payload during RCAR handshake.

Refs:
- https://github.com/confidential-containers/trustee/blob/main/kbs/docs/kbs_attestation_protocol.md
- https://github.com/confidential-containers/guest-components/blob/main/attestation-agent/attester/src/sample/mod.rs

* fix: Make CC AA required for sample attestation when configured

When USE_CC_ATTESTATION_AGENT=true, attestation-service now
requires AA to be available for NoCC/sample platform. This ensures
sample evidence always comes from AA with the correct KBS format.

Changes:
- Error out if AA connection fails for NoCC platform when AA is configured
- Only use EmptyProvider if AA is explicitly NOT configured
- Prevents incorrect sample evidence format from EmptyProvider

This ensures attestation-service delegates to AA for sample evidence
generation instead of creating it itself.

* fix: Implement proper RCAR protocol with tee-pubkey and runtime-data hash

Fixed KBS attestation error 'REPORT_DATA is different from that in Sample Quote'

Changes:
1. Generate ephemeral EC key pair BEFORE getting evidence from AA
2. Create runtime-data with nonce + tee-pubkey (JWK format)
3. Hash runtime-data (SHA-256) and use as report_data for AA
4. This binds the tee-pubkey to the TEE evidence per RCAR protocol

The report_data in the evidence now matches what KBS expects:
hash(runtime-data) instead of computation ID.

This completes the full RCAR protocol implementation:
- Request → Challenge → Attestation (with bound tee-pubkey) → Response

* fix(agent): use simple nonce for Sample attestation report_data

For Sample/NoCC attestation, use the raw nonce bytes directly as
report_data instead of hashing runtime-data. This avoids JSON
serialization mismatches with the KBS Sample verifier.

Real TEEs (TDX/SNP) still use runtime-data hash binding to
cryptographically bind the ephemeral tee-pubkey to the evidence.

* fix(agent): use RFC 8785 canonical JSON for runtime-data hashing

The KBS Sample attestation verifier (and likely others) expects the
report_data to be the SHA-256 hash of the *canonical* JSON serialization
(RFC 8785) of the runtime-data. Standard Go JSON marshaling does not
guarantee key ordering, leading to hash mismatches.

This change uses github.com/gowebpki/jcs to canonicalize the runtime-data
before hashing, ensuring compatibility with the KBS RCAR implementation.
Also reverted the temporary 'simple nonce' workaround.

* feat(hal): add CoCo Keyprovider and Skopeo packages

- Add coco-keyprovider buildroot package with systemd service
- Add skopeo buildroot package for OCI image handling
- Add ocicrypt_keyprovider.conf for encrypted image decryption
- Update Config.in to include new packages

This enables standard CoCo ecosystem integration for encrypted
OCI images instead of custom S3/HTTP registry clients.

* feat(oci): add OCI image handling package with Skopeo integration

- Add pkg/oci/types.go with ResourceSource and ImageManifest types
- Add pkg/oci/skopeo.go with Skopeo wrapper for pull/decrypt
- Add pkg/oci/extract.go for extracting algorithms and datasets from layers

This package provides OCI image handling using Skopeo and CoCo
Keyprovider for encrypted image decryption, replacing custom
S3/HTTP registry clients.

* chore: regenerate protobuf files for updated cvms.proto

* refactor(agent): replace S3/HTTP/KBS with OCI package

- Remove pkg/kbs and pkg/registry imports
- Add pkg/oci import for OCI image handling
- Replace downloadAndDecryptResource with OCI-based implementation
- Use Skopeo + CoCo Keyprovider for automatic decryption
- Reduce code from ~240 lines to ~70 lines

This eliminates custom KBS RCAR handshake, S3/HTTP registry clients,
and manual decryption logic. CoCo Keyprovider handles all decryption
automatically via ocicrypt protocol.

* chore: remove obsolete pkg/kbs and pkg/registry packages

- Delete pkg/kbs/ (custom KBS client, ~300 lines)
- Delete pkg/registry/ (S3/HTTP registry clients, ~400 lines)
- Remove unused imports from agent/service.go
- Run go mod tidy to clean up dependencies

These packages have been replaced by pkg/oci with Skopeo and
CoCo Keyprovider for standard CoCo ecosystem integration.

* fix(agent): update ResourceSource struct to include type and encryption fields

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fix(hal): update CoCo Keyprovider to v0.16.0 and fix build path

- Update version from v0.11.0 to v0.16.0 (matches attestation agent)
- Fix install path: target is at repo root, not in coco_keyprovider subdir
- This fixes the build error where coco_keyprovider binary wasn't found

The cargo workspace in guest-components builds to a shared target/
directory at the repository root, not within each crate's subdirectory.

* feat: Update remote resources testing guide to use kbs-client and coco-keyprovider for key management and encryption, enable insecure TLS for Skopeo, and enhance CVMS with

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: Update component versions, revise image encryption documentation, and sanitize OCI image paths for Skopeo compatibility.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: Add `decompress` option to Dataset and `algo_type`/`algo_args` to Algorithm protobuf messages, updating client, test, and build configurations.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Update multiple package versions and enhance OCI image extraction error reporting for missing algorithm files.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* chore: Bump package versions, improve OCI image extraction debugging by returning seen files, and remove unused dataset type parsing from test code.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* refactor: Migrate OCI extraction to use structured logging with `slog` and `context`, and update package versions.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: Bump multiple component versions, add encrypted status for computation inputs and algorithms, and refine OCI layer extraction warnings.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* logging

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: Add `Encrypted` field to algorithm and dataset resource sources and update all component versions.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: update component versions, integrate coco-keyprovider service, and configure ocicrypt key provider.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: add support for KBS parameters and dataset/algorithm hash calculations in CVMS

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: update resource download and extraction logic to support requirements.txt and improve hash verification

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* chore: Update dependencies, improve code style, and add GetRawEvidence to attestation client mocks.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Refactor code structure for improved readability and maintainability

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fix: update golangci configuration to include errcheck for build path and remove unnecessary exclusions

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fix: streamline kernel command line handling in QEMU args construction

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* feat: add attestation binary and update checksum tests and policy structure

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Add unit tests for attestation agent, attestation, log, crypto, OCI, and Skopeo clients

- Implement tests for the attestation agent client including Unix socket and TCP address handling, token retrieval, and error scenarios.
- Enhance attestation client tests to cover fetching raw evidence for various platforms (SNP, TDX, VTPM, SNPvTPM) and validate error handling.
- Introduce log client tests to verify retry behavior for sending logs and events.
- Create comprehensive tests for crypto package focusing on AES-GCM decryption, encrypted resource parsing, and key unwrapping.
- Add tests for OCI package to validate algorithm and dataset extraction, including JSON serialization of OCILayout.
- Implement Skopeo client tests to ensure proper functionality for image pulling, inspecting, and resource source handling.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fix: handle JSON marshal errors in test cases for decrypt and extract functions

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* test: add comprehensive tests for algorithm and dataset extraction with various scenarios

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* refactor: replace hardcoded Python script content with constant variable

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fix: remove redundant mock expectation for SendAgentConfig in TestCreateVMWithAaKbsParams

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* test: add tests for event sending failure, dataset extraction with path traversal, and Skopeo client behavior

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* test: add tests for download and decryption of resources with various URL formats

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* refactor: Introduce OCIClient interface for agent service to improve testability of OCI image operations and enhance related tests.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* refactor: Change `get_uint64_from_tcb` to accept `TcbVersion` by value and use `u64::from` for type conversions.

---------

Signed-off-by: Sammy Oina <sammyoina@gmail.com>
 2026-03-16 14:48:55 +01:00
Sammy Kerata Oina 8eb1fac9ad NOISSUE - Refactor and update dependencies in the project (#491) 
* Refactor and update dependencies in the project

- Updated go.sum to replace `github.com/absmach/magistrala` with `github.com/absmach/supermq` across various modules.
- Removed VSock configuration from environment variables and QEMU arguments.
- Updated QEMU configuration and related tests to remove references to guest CID and VSock.
- Added new HTTP transport layer for API endpoints in the manager.
- Introduced Prometheus monitoring configuration with alert rules and Alertmanager setup.
- Updated service and VM interfaces to remove unused methods and references.
- Refactored tests to align with the new structure and dependencies.

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Add MaxVMs configuration and enforce limit on VM creation

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Add comprehensive tests for HTTP transport handlers and endpoints

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Add test case for exceeding maximum number of VMs in TestRun

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Improve error handling in TestHandlerWithCustomRouter to ensure response writing is checked

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Update dependencies to latest versions

- Upgrade cel.dev/expr from v0.23.0 to v0.24.0
- Upgrade github.com/absmach/supermq from v0.16.0 to v0.17.0
- Upgrade github.com/cenkalti/backoff from v4.3.0 to v5.0.2
- Upgrade github.com/cncf/xds/go to v0.0.0-20250501225837-2ac532fd4443
- Upgrade github.com/go-chi/chi/v5 from v5.2.1 to v5.2.2
- Upgrade github.com/go-jose/go-jose/v3 from v3.0.3 to v3.0.4
- Upgrade github.com/gofrs/uuid/v5 from v5.3.0 to v5.3.2
- Upgrade github.com/prometheus/client_golang from v1.22.0 to v1.23.0
- Upgrade github.com/prometheus/client_model from v0.6.1 to v0.6.2
- Upgrade github.com/prometheus/common from v0.62.0 to v0.65.0
- Upgrade github.com/prometheus/procfs from v0.15.1 to v0.16.1
- Upgrade go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp from v0.60.0 to v0.62.0
- Upgrade go.opentelemetry.io/otel/exporters/otlp/otlptrace from v1.36.0 to v1.37.0
- Upgrade golang.org/x/crypto from v0.39.0 to v0.40.0
- Upgrade golang.org/x/sys from v0.33.0 to v0.34.0
- Upgrade golang.org/x/text from v0.26.0 to v0.27.0
- Upgrade golang.org/x/time from v0.11.0 to v0.12.0
- Upgrade google.golang.org/grpc from v1.73.0 to v1.74.2

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

---------

Signed-off-by: Sammy Oina <sammyoina@gmail.com>
 2025-08-05 11:22:02 +02:00
dorcaslitunya f543cb4363 COCOS-456 Remove SEV support from repo (#472) 
* Remove SEV support from repo

* Remove SEV references
 2025-07-10 10:58:36 +02:00
dorcaslitunya 77325753f8 NOISSUE - Add TDX support to Manager (#446)
CI / ci (push) Has been cancelled
Details 
* Add TDX support on manager

* Add functions to check platform

* Search for tdx in kernel parameters

* Modify based on comments
 2025-06-17 11:34:30 +02:00
Sammy Kerata Oina 7e63921896 NOISSUE - Simplify local agent running in non sev-snp environment (#411) 
* Add vtpm attestation support to agent service and server

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Update mockery version to v2.53.2 and refactor VM factory to include logger

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Send event notification when computation is stopped in agentService

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Remove redundant assignment of Stderr in qemuVM Start method

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Rename SVM references to CVM in tracing, logging, metrics, and service layers

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

---------

Signed-off-by: Sammy Oina <sammyoina@gmail.com>
 2025-04-01 21:59:11 +02:00
Danko Miladinovic ebc8f1bba4 NOISSUE - Update documentation for vTPM changes (#408) 
* change readme according to vTPM changes

* rebase
 2025-03-19 09:38:48 +01:00
Danko Miladinovic 293c65a3aa NOISSUE - Add path to expected PCR values (#398)
CI / ci (push) Has been cancelled
Details
Rust CI Pipeline / rust-check (push) Has been cancelled
Details 
* add path to expected PCR values

* change rust scrtip for attestation policy to print policy to stdout

* fix cli test

* remove stdout from cmd config struct

* fix manager test

* fix manager readme
 2025-03-17 12:25:42 +01:00
Jovan Djukic 33744a12a8 COCOS-390 - Add IGVM measurement on manager (#404)
CI / ci (push) Has been cancelled
Details 
* resolved issue 390

* updated readme.md for issue 390

* resolved issue 390

* updated readme.md for issue 390

* implemented suggested changes

* refactored code so it passes the linter test

* change the Run fn so it prints the meassurement t a buffer

* refactored code so it passes the linter test

* fixed the test Run_-_Failure_Execution

* changed recipe so it builds igvmmeasure binary when building manager
 2025-03-12 13:43:26 +01:00
Danko Miladinovic 67f939fc66 COCOS-326 - Add vTPM support to CoCoS (#376)
CI / checkproto (push) Has been cancelled
Details
CI / ci (push) Has been cancelled
Details
Rust CI Pipeline / rust-check (push) Has been cancelled
Details 
* manager, cli and agent vtpm support

* rebase and changed atls for vtpm

* deleted unused code

* changed chekproto.yaml script so it find the manager proto file correctly

* fixe manager proto version

* fix agent tests

* fix server agent test

* fix attestation test

* fix attestation test gofumpt

* created dummy RWC for TPM

* fix comment

* add default PCR values

* rebase main

* fix rust ci and missing header

* changed embedded  attestation to VMPL 2

* fix unused impot

* fix pkg test

* address attestation type

* fix agent attestation test

* add prc15 check

* fix comments

* fix cli tests

* add doc

* add mock for LeveledQuoteProvider when SEV-SNP device is not found

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fix manager reading attestation policy

* refactor PCR value checks and update attestation policy values

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fix tests for sev and grpc

---------

Signed-off-by: Sammy Oina <sammyoina@gmail.com>
Co-authored-by: Sammy Oina <sammyoina@gmail.com>
 2025-03-07 16:36:47 +01:00
Sammy Kerata Oina cef9cbbecd COCOS-364 - Make agent more resilient to gRPC disconnection on cvms cloud server (#375)
CI / ci (push) Has been cancelled
Details 
* Refactor AgentServer interface and update related implementations; adjust dependency versions in go.mod

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Implement State method for agent.Service and enhance metrics and logging middleware to track state changes

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Update file permission modes to use octal notation in CVMS client and agent main

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Refactor CVMS client and agent main to improve function signatures and variable naming for clarity

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Remove unnecessary fmt import and logging statement in CVMSClient Process method

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Add file-based storage implementation for message persistence in CVMSClient

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Update README and main.go for CVMS service: correct references and improve clarity

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Update README to clarify gRPC host and port descriptions for CVMS server

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Rename sendMessageWithRetry to sendStreamMessage for clarity and consistency in CVMSClient

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* Update README to add detailed descriptions for algorithm and dataset paths

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

---------

Signed-off-by: Sammy Oina <sammyoina@gmail.com>
 2025-02-16 18:02:17 +01:00
Sammy Kerata Oina 760c9bb580 NOISSUE - Rename backend info to attestation policy (#314) 
* attestation policy field

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fix tests

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fmt

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

---------

Signed-off-by: Sammy Oina <sammyoina@gmail.com>
 2024-11-21 20:56:03 +01:00
Danko Miladinovic d5941edb56 NOISSUE - Add information on OVMF verison, CPU type, CPU number and EOS version (#307) 
* add manager OVMF verison information

* add Manager documentation

* add kernel cmd

* add SVMInfo test
 2024-11-18 17:16:33 +01:00
Sammy Kerata Oina f906593492 remove tmp directory (#204) 
Signed-off-by: Sammy Oina <sammyoina@gmail.com>
 2024-08-21 11:55:36 +02:00
Sammy Kerata Oina 8f2cbd349d bump port range (#206) 
Signed-off-by: Sammy Oina <sammyoina@gmail.com>
 2024-08-20 23:56:34 +02:00
Sammy Kerata Oina 971e00aa68 NOISSUE - Select from port range (#196) 
* select from port range

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fix failing test

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* remove whitespace

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

---------

Signed-off-by: Sammy Oina <sammyoina@gmail.com>
 2024-08-08 15:48:12 +02:00
Smith Jilks 88707c4668 NOISSUE - Fix Jaeger tracer (#177) 2024-07-22 14:49:42 +02:00
Sammy Kerata Oina 64f7e7f7fd NOISSUE - Refactor single algorithm processing (#117) 
* Refactor single algorithm processing

Simplified the agent service's algorithm handling logic to process a single algorithm instead of multiple. This change:
- Removed the `Algorithms` type and associated stringer implementation.
- Updated the state machine and service logic to expect a singular algorithm, aligning the agent's internal state transitions with the new model.
- Adjusted the manager service and computations test server to mirror these changes in their respective payload structures, ensuring API and test consistency.
- Altered README files to reflect the simplified interaction model and removed outdated descriptions.
- Reverted the protoc-gen-go version used for generating protobuf files to maintain compatibility with the rest of the codebase.

The single-algorithm approach streamlines the computation running process, reducing complexity and potential error conditions. It directly impacts how external services will construct and send computation requests.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update protoc-gen-go version to v1.33.0

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor variable name in computations.go and grpc.go

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
 2024-04-17 13:34:54 +02:00
Danko Miladinovic 3a14896555 NOISSUE - Attested TLS (#99) 
* added initial code for attested TLS

* added client validation and verification

* fixed bugs for attested TLS

* updated README for manual testing

* fixed CI errors

* removed SNP pollicy from agent config

* added attested TLS config param to AgentConfig

* generated manager.pb.go for protoc v25.2

* updated proto-gen-go version on CI

* generated agent.pb.go to match newest proto gen version

* define errors for error handling

* fixed comments
 2024-04-16 15:19:19 +02:00
Sammy Kerata Oina 2b760ec207 NOISSUE - Streamline client identification (#105) 
* Update protoc to v4.25.3 and streamline client identification

Protocol Buffer compiler `protoc` has been upgraded to version 4.25.3. This update unifies the version across multiple generated files to ensure compatibility and take advantage of any bug fixes and performance improvements in the new release.

Additionally, the client identification process has been refined. The redundant `WhoAmI` message and the corresponding checks have been removed from the gRPC server implementation. Clients are now identified via their address from the `Process` stream's context as soon as a connection is established, simplifying the code and potentially reducing handshake time.

This change sets the foundation for a leaner communication protocol between manager and agents, and could contribute to lower latencies in client-server interactions.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix goroutine bug in grpc server

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update PROTOC_VERSION to 25.3

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor server reference in readme

Updated the deployment instructions in the README to point to the correct server location after consolidating test server documentation. Removed obsolete `manager-server` module and associated main entry point to align with the new architecture and streamline workflows.

Resolves issues with outdated links and cluttered repository structure.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
 2024-04-15 22:08:37 +02:00
Filip Bugarski 35c2a9e92a Update cocos documentation (#89) 
* Update cocos documentation

Signed-off-by: fbugarski <filipbugarski@gmail.com>

* inserted go run commands into separate lines

* updated syntax higlighting to bash

---------

Signed-off-by: fbugarski <filipbugarski@gmail.com>
Co-authored-by: Danko Miladinovic <danko@etf.rs>
 2024-04-13 18:33:10 +02:00
Smith Jilks b6f116474d Cocos-91 - Update Makefile and cocos documentation (#93) 
* Update Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update cocos documentation

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update cocos documentation

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update cocos documentation

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Remove install rule from Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update agent README.md

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update agent README.md

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update README.md files

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Remove INSTALL_DIR from Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Fix comments in README.md files

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

---------

Signed-off-by: Jilks Smith <smithjilks@gmail.com>
 2024-04-08 15:59:38 +02:00
Danko Miladinovic 4b38ac09dd NOISSUE - Added SEV-SNP support (#98) 
* added SEV-SNP support

* updated documentation for SEV-SNP

* changed variable names to adhere to the GO naming convention

* changed the variable name from V-sock to Vsock
 2024-03-22 10:55:09 +01:00
Sammy Kerata Oina 938dd6cb78 NOISSUE - mTLS support across services (#71) 
* Implemented mTLS support across services

Extended gRPC configuration to support mutual TLS (mTLS) in agent and manager components for enhanced security. This includes the loading of Certificate Authority (CA) certificates, server, and client certificates, and keys. Updated README documentation to reflect the new environment variables required for mTLS configuration. Additionally, streamlined secure gRPC client connection setup and logging messages to indicate whether a service is running with TLS, mTLS, or without TLS.

The change ensures secure communication between services by verifying both client and server identities, thus addressing potential security concerns in network-level interactions.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance agent cert handling and update copyright

- Implement function to create certificate files for the agent configuration dynamically, ensuring file paths are updated to reflect newly created files. This improves the agent's setup process by automating the certificate handling.
- Update copyright clause to reflect the new owning entity, Ultraviolet, affirming correct attribution and compliance with legal requirements.
- Refactor gRPC client connection code to remove redundant package alias, streamlining the codebase and improving readability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor cert loading with fallbacks

Removed redundant certificate file creation logic in the agent module and introduced a more robust loading mechanism in the gRPC server module to support direct byte content aside from file paths. This change simplifies the initial setup process for the agent by removing the need to create certificate files preemptively, thereby streamlining deployment in environments with varying filesystem access. It supports using certificate contents directly, enhancing compatibility with in-memory configurations or environments where file storage may not be ideal.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
 2024-02-08 10:07:51 +01:00
Sammy Kerata Oina e2fb7ea88d NOISSUE - Add test server (#80) 
* add test server

Signed-off-by: SammyOina <sammyoina@gmail.com>

* clean up and update docs

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update docs

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
 2024-02-07 16:16:27 +01:00
Sammy Kerata Oina 8975e28437 COCOS-77 - Stream agent logs and events over gRPC (#78) 
* Refactor GRPC manager service and client

The manager service and client have been restructured for stream communication, facilitating real-time agent events, logs, and run responses. The `Run` RPC is replaced by the `Process` stream RPC, enabling bidirectional streaming between clients and the manager service. This allows continuous interchange of different message types including `WhoAmIRequest`, `AgentLog`, `AgentEvent`, and `RunResponse`.

Several message types have been adjusted and new fields introduced, like `AgentPort` in `RunResponse` and various agent-config attributes including CA files and instance IDs, to support TLS client authentication and distinguish between agent instances.

We've also incorporated `google.protobuf.Timestamp` in `AgentEvent` for precise event logging. The client code reflects these modifications with updated method calls and stream handling logic for ongoing communication. Moreover, the updates necessitate corresponding changes throughout service, grpc, and sdk layers to interoperate with the new streaming approach.

The transition to streaming paves the way for a more interactive, flexible communication system that can accommodate future expansion and real-time monitoring features.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update GitHub Actions to Latest Versions

Upgraded GitHub Actions 'checkout' to version 4 and 'setup-go' to version 5 across various workflow files to leverage the latest features and improvements for better performance and reliability. This also ensures compatibility with Go version 1.21.x which is specified in the workflows.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor event handling and logging

Reworked event and log processing to use channels instead of direct HTTP calls. Removed obsolete events package and consolidated event structures, leading to cleaner and more maintainable code. Updated agent events to use channels, enhanced error handling in log forwarding, and simplified manager `New` function signature to accept an event channel directly.

- Removed `events` and `agentevents` packages to reduce complexity.
- Replaced direct event server communication with internal channel usage.
- Introduced `AgentEvent` struct in events.go for standardized event objects.
- Adapted `managerService` to dispatch events and logs through channels.
- Streamlined manager construction by removing the now-unnecessary event service and host IP parameters.

This change results in a more robust and easier to extend event and log management system within the agent-manager interaction.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix ci

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove unused code

Signed-off-by: SammyOina <sammyoina@gmail.com>

* add comments

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
 2024-02-07 12:08:39 +01:00
Sammy Kerata Oina 046b549079 Remove HTTP server support and streamline config (#75) 
The HTTP server-related code, documentation, and configurations have been removed as part of a shift towards prioritizing gRPC for service communication. This update includes deletions of HTTP host and port configs across various components, the manager HTTP API alongside its Swagger definition, and the removal of related scaffolding and utility code. This change simplifies the overall architecture and eliminates redundant HTTP support, focusing on optimizing gRPC performance and security features.

Signed-off-by: SammyOina <sammyoina@gmail.com>
 2024-02-07 11:03:46 +01:00
Sammy Kerata Oina f2567830b3 NOISSUE - Remove docker (#67) 
* Remove Docker-related build functionality

Removed the Docker build scripts, Dockerfile, and docker-compose definitions from the Makefile and relevant directories. Updated documentation by stripping out references to building and running Docker containers. This change likely reflects a shift towards a different deployment strategy or a move away from Docker as a dependency for builds and runtime. This simplification could lead to less complexity in the build process and reduce maintenance overhead associated with Docker configurations.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Removed Docker image build and publish workflow

The workflow for building and publishing a Docker image on pushes to the main branch has been removed. This step likely reflects a change in deployment strategy or a move to a different CI/CD pipeline. Continuous deployment might now be managed by another service or process, adhering to updated operational requirements or infrastructure changes.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
 2024-01-25 15:29:16 +01:00
Sammy Kerata Oina 55afe4c038 COCOS-49 - Pass agent configuration and computation via vsock (#57) 
* Optimize QEMU launch and add V-sock support

Refactored QEMU argument construction and launching logic by removing the dependency on 'agent.Computation'. This simplification makes the VM creation process more streamlined. Additionally, introduced V-sock capabilities in the QEMU configuration to facilitate improved guest-host communication. Updated the README to include kernel module setup instructions for the new V-sock feature.

The V-sock implementation enables VMs to use a consistent communication channel that is not affected by network configuration changes, enhancing reliability and potential interoperability with host services. It's important to ensure that the necessary kernel modules are loaded as part of the setup process, as documented.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add vsock-based communication to manager

Introduced virtual socket (vsock) communication abilities in the manager package by implementing a new socket service. This includes establishing a vsock listener and stub methods for sending computation results and cleaning up resources. The addition provides the groundwork for interprocess communication between guest and host in virtualized environments.

- Integrated the `mdlayher/vsock` library for handling virtual socket operations.
- Created a new `sockService` struct to encapsulate vsock listener handling.
- Implemented `NewVsock` constructor to initialize the listener with domain value `3`.
- Added placeholder methods for future computation sending and service closing logic.

This enhancement targets scenarios where efficient VM-to-host communication is required.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove env

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor agent config and use vsock

Introduce `AgentConfig` struct to group agent-related configurations, and update `Computation` struct to include the new `AgentConfig` field. Replace command-line computation extraction with vsock-based config retrieval for robustness and decoupling. The agent configuration is now read from a vsock connection during runtime, allowing for more dynamic and flexible deployments. Adjusted the main agent application logic to support these configuration changes, and corresponding changes have been made in the manager to facilitate vsock communication.

This approach aligns with modern practices for microservices by streamlining configuration management and reducing reliance on static command-line parameters. Moreover, it enhances the scalability of the agent service by allowing configuration to be managed externally.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor agent config and remove deprecated code

Consolidated agent configuration management into a single `AgentConfig` message and pruned deprecated Protobuf `ComputationReq`, `DatasetReq`, and `AlgorithmReq` messages. Adapted corresponding manager service logic to the new configuration structure. These modifications align with updated manager API schema, facilitate clearer configuration handling, and improve maintainability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* send configuration

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Switch agent to listen mode for manager connections

Previously, the agent established a connection to the manager using a direct dial. This change shifts the setup to where the agent listens on a specified port and accepts incoming connections. It ensures that the agent properly handles incoming requests by initiating a listening socket and waiting for the manager to connect, enhancing the system's flexibility in connection management. This adjustment also includes graceful closure of the listening socket.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
 2024-01-19 17:24:51 +01:00
Sammy Kerata Oina 1167aeb53f NOISSUE - Update documentation (#20) 
* Fix bug in agent service and state

The commit fixes a bug in the agent service and state files.
Previously, the condition to check the state in the agent service
was incorrect. It was checking the state directly instead of
using the GetState() method. This has been fixed by using the
GetState() method to check the state.

Additionally, a new GetState() method has been added to the
StateMachine struct in the state file. This method retrieves
the current state by acquiring a lock and returning the state
value.

The changes have been tested and verified to resolve the bug
and improve the accuracy of state checking in the agent service.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix bug in agent state machine

The bug in the agent state machine caused an error when attempting an invalid transition. This commit fixes the bug by properly locking and unlocking the state machine before and after transitioning to the next state. Additionally, the logger now correctly logs the current and next state during a valid transition.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix race condition in state machine

The commit fixes a race condition in the state machine implementation in the `Start` method. The race condition occurs when multiple goroutines try to access and modify the state concurrently. To fix this, a mutex lock and unlock are added around the critical sections of code to ensure exclusive access to the state variable. This prevents race conditions and ensures the state transitions are executed correctly.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix race condition in StateMachine.Start()

The StateMachine.Start() method was experiencing a race condition
when multiple events were being processed concurrently. This was
caused by not properly locking and unlocking the state machine
before and after updating the state. This commit fixes the issue
by adding proper locking and unlocking around the state update
operation. Additionally, the logging statement has been updated
to include the previous and next states for better debugging.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* add magistrala dep

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove mainflux

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger.

The commit message should be:

"Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger"

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix notification topic in agent service and update NATS ports in Docker environment variables

The agent service's notification topic was incorrectly set to "channels.manager" instead of "agent". This commit fixes the issue by updating the notification topic.

Additionally, the NATS ports in the Docker environment variables were incorrect. The COCOS_NATS_PORT and COCOS_NATS_HTTP_PORT have been updated to the correct values.

These changes ensure that the agent service uses the correct notification topic and the NATS ports are properly configured.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor Makefile and add new targets for building Docker images

The Makefile has been refactored to include new targets for building Docker images. The `make_docker` and `make_docker_dev` functions have been defined to handle the Docker build process. The `dockers` and `dockers_dev` targets have been added to build the Docker images for all services and development environments respectively.

This commit introduces changes to the Makefile to improve the build process and provide support for Docker images.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update readme

Signed-off-by: SammyOina <sammyoina@gmail.com>

* wrap env vars

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor main.go in cmd/cli

The main.go file in cmd/cli has been refactored to improve code readability and maintainability. The defURL constant has been removed as it is no longer needed. Additionally, unnecessary whitespace has been removed.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix linting

Signed-off-by: sammy <sammyoina@gmail.com>

* rename cocos-ai to cocos

Signed-off-by: sammy <sammyoina@gmail.com>

* Updated README with NATS setup instructions and correct systemd path

Expanded the project's README to include setup instructions for the NATS server, which is now a necessary component for agent and manager communication. Additionally, the systemd service file path has been corrected from 'systemd' to 'init/systemd' ensuring the agent's proper installation and configuration as a daemon. This facilitates a smoother setup experience and reflects the dependency on NATS for push notifications.

Ref: Agent and manager communication enhancement
Signed-off-by: SammyOina <sammyoina@gmail.com>

* Added message broker URL to agent service config

Included the environment variable for the message broker URL in the systemd service configuration to facilitate agent communication with the messaging system. This ensures the agent can connect to the designated message broker for event publishing and subscription.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update docs to hal

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Remove TLS and timeout config from gRPC client

Refactored gRPC client and associated API code by removing unnecessary TLS configuration and timeout settings. Simplified the communication setup by trusting the environment to enforce security policies and handle operation durations, instead of hardcoding these within the application.

This brings about a cleaner, more maintainable codebase and shifts responsibility for security configurations out of the code, aligning with infrastructure-as-code practices and enabling easier scalability and environment-specific adjustments.

This change also affects the API documentation and example commands, which have been updated accordingly to reflect the simplification and to guide users with the streamlined setup process.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
Signed-off-by: sammy <sammyoina@gmail.com>
 2024-01-11 15:19:34 +01:00
Sammy Kerata Oina a5e6cae92c COCOS-44 - Add agent address to run responses (#45) 
* Add agent address to run responses

The manager service's Run method now retrieves the agent address upon successful computation execution, providing more informative responses across gRPC, HTTP, and logging endpoints. This change improves service transparency by returning the agent's address to be used by client services, making the manager service's external communication more comprehensive. Updated the `RunResponse` structure in the corresponding protocol buffers definition and response handling in gRPC and HTTP APIs, ensuring that agent address information is serialized appropriately. It also necessitates a slight adjustment in the QEMU configuration to manage port forwarding rules more dynamically, simplifying the process as only agent-relevant ports are incremented with each new computation. This extra detail in responses aids in debugging and offers better integration capabilities for clients.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Allocate dynamic ports for VM guests and expand error handling

Refactored service initialization to accept host IP and incorporated dynamic port allocation for VM guests, replacing the prior static increment method. Introduced a new error type 'ErrFailedToAllocatePort' to capture instances where the system is unable to find a free port. Integrated a third-party error package for improved error wrapping and context. These changes prevent port conflicts between VM guests and enhance error diagnostics for service operations.

Resolves issue with static port allocation leading to conflicts.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add HOST_IP to service configuration

Extend service configuration to include the host machine's IP address, allowing instances to be aware of their deployment environment. This update passes the new HostIP field to the service constructor, ensuring the service can now operate with host-specific logic.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Populate AgentAddress in gRPC Responses

Enhanced the gRPC encode/decode functions to properly populate the 'AgentAddress' field in 'RunResponse' objects. This ensures that consumers of the gRPC interface receive complete response data, which previously omitted the important 'AgentAddress' information. The change impacts both server-side response encoding and client-side response decoding, aligning the implementation with the expected interface contract.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix ci

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
 2024-01-11 15:08:35 +01:00
Sammy Kerata Oina c29ef354fa COCOS-24 - Add events service via http (#27) 
* add status endpoint

Signed-off-by: sammy <sammyoina@gmail.com>

* feat: Update code generation tools to latest versions

Update the code generation tools, including protoc-gen-go and
protoc-gen-go-grpc, to their latest versions (v1.31.0 and v1.3.0,
respectively). This ensures compatibility with the latest features and
improvements. The updated tools also require gRPC-Go v1.32.0 or later.

The new versions bring important updates and bug fixes, enhancing the
performance and stability of the generated code. By staying up-to-date
with the latest tooling, we can take advantage of the latest
functionality and ensure a smooth development experience.

No code changes are included in this commit. These updates only impact
the code generation process.

Signed-off-by: sammy <sammyoina@gmail.com>

* Update Go version to 1.21.x

Signed-off-by: sammy <sammyoina@gmail.com>

* Refactor agent and manager services to publish event notifications

The refactoring includes changes to the agent and manager services to incorporate event notifications. By publishing events, the services can inform subscribers about the current state of the computation or any updates. Specifically, the `agentService` now includes a `cmpHash` field to store the SHA-256 hash of the computation, which is subsequently used when publishing events. The `agentService` and `managerService` now use the `publisher` interface to publish events to the topic "manager". Notably, the removed `pubsub.go` file is no longer necessary.

This commit improves the service architecture by allowing subscribers to receive relevant updates and monitor the progress of computations. It enhances the overall system by providing more transparency and enabling better coordination between the agent and manager services.

Signed-off-by: sammy <sammyoina@gmail.com>

* Improve generated Go file comparison in checkproto workflow

Refactor the file comparison logic in the checkproto workflow to use the `-p` flag instead of `-s` for improved accuracy. This change ensures that the generated Go files are thoroughly compared with the original ones, detecting any discrepancies and preventing out-of-sync files from passing the validation. By using the `-p` flag, we now check both the contents and the metadata of the files, providing more robust synchronization checks. This update enhances the reliability of the checkproto workflow and helps maintain consistency between the proto files and their corresponding generated Go files.

Signed-off-by: sammy <sammyoina@gmail.com>

* Update file comparison command to detect differences line by line

The code change updates the file comparison command used in the CI workflow to detect differences line by line instead of only reporting the first difference encountered. This change improves the accuracy of detecting inconsistencies between the original protobuf files and the generated Go files. Previously, only the first difference was reported, leading to potential missed issues. By comparing the files line by line, we can now detect and report all differences accurately. This change enhances the reliability of our CI pipeline and ensures that the generated Go files stay in sync with the protobuf files.

Signed-off-by: sammy <sammyoina@gmail.com>

* add event exporting to external server

Signed-off-by: sammy <sammyoina@gmail.com>

* feat: Add support for notification server URL

The commit adds a new environment variable, `COCOS_NOTIFICATION_SERVER_URL`, which allows specifying the server to receive notification events from the agent. This addition provides flexibility to configure the notification server URL based on the deployment environment. This change enables seamless integration with different notification server instances and enhances the extensibility of the system. It resolves the need to modify the code directly when changing the server URL.

Signed-off-by: sammy <sammyoina@gmail.com>

* Refactor gRPC client and server, remove unused handlers

The commit refactors the gRPC client and server code by removing the unused `nopDecoder` and `status` handlers from the client and server, respectively. This cleanup reduces code clutter and improves maintenance. No significant consequences are expected.

Signed-off-by: sammy <sammyoina@gmail.com>

* Ensure generated Go files stay in sync with proto files during the CI workflow

Fixes an issue in the CI workflow where proto files and their corresponding generated Go files were not being properly compared for synchronization. Previously, the `cmp -l` command was used, which only printed differing byte positions, leading to false negatives. This has been corrected by using `cmp -s` instead, which outputs nothing if the files are identical. This change ensures that any differences between the proto files and their generated Go files will be detected, helping to maintain consistency and accuracy in the codebase.

Signed-off-by: sammy <sammyoina@gmail.com>

* Enhance notification payload and endpoint

Extended the notification system to include 'status' and 'details' in the payload, improving traceability and debugging. Adapted the serialized JSON structure for clarity and added an 'originator' field to track the source service. Transitioned to a generalized event endpoint, facilitating a more streamlined event handling process.

Refactors POST request to a more appropriate endpoint and updates the notification service interface to reflect new payload requirements.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor event notification logic

Removed the legacy notifications package and consolidated event notification functionality using the new internal events service. Modified agent, manager, and main application code to use this service for consistent event reporting and error handling workflows across services. This change simplifies event management, improves error visibility, and allows for more maintainable code by centralizing event-related logic. The substitution of verbose state-specific publishEvent calls with generic status reporting aligns with the new service's capabilities.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Expand agent protobuf message types and improve error handling

The protobuf definition for agent messages has been updated to include an additional message type, facilitating future data structure expansions. Additionally, error handling for event sending in the main agent execution has been enhanced to log errors when sending 'init' events fail, ensuring issues are properly tracked. The unused `notificationTopic` constant in the manager service has been removed for cleaner code maintenance.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Optimize JSON parsing and fix header omission

Removed unnecessary unquoting of a JSON string before unmarshaling, streamlining the computation value extraction process. Also corrected a missing Content-Type header in the event sending function, ensuring proper handling of JSON requests by recipients. These changes improve performance and communication reliability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* align vars

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Remove computation request timeout feature

The timeout feature for computation requests has been removed to simplify the computation execution flow. This involved changes across multiple files, including protobuf definitions, HTTP endpoint handling, and the internal computation logic. We eliminated the timeout field, associated logic, and error handling to ensure the system no longer supports timeouts for computations, mitigating any unintended timeout impacts on long-running processes.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: sammy <sammyoina@gmail.com>
Signed-off-by: SammyOina <sammyoina@gmail.com>
 2024-01-11 14:37:14 +01:00
Sammy Kerata Oina 46a91ccbc3 remove nats (#41) 
Signed-off-by: SammyOina <sammyoina@gmail.com>
 2024-01-09 19:56:31 +01:00
Sammy Kerata Oina a3c5c765b8 NOISSUE - Connect to agent when vm is created (#33) 
* Add gRPC TLS config and update protoc versions

Enhanced manager service with TLS configuration options and timeout settings for gRPC communication with agent services. Updated corresponding protobuf definitions to include new fields for CA certificates, TLS status, and timeouts. The added TLS support ensures secure inter-service communication while flexible timeout configurations improve the robustness of network interactions.

The following adjustments and additions were made:
- Included new gRPC configuration fields (ca_certs, client_tls, timeout) for agent communication in protobuf definitions.
- Injected agent gRPC configuration into service endpoints, ensuring secure TLS setup and compliance with provided settings.
- Revised main service function signatures to accept the gRPC configuration object.
- Incremented port forwarding counters post-computation to avoid port conflicts.
- Conducted compatibility update of protobuf version comments to reflect minor version bump (v4.25.0 to v4.25.1).

This change impacts service deployment that requires proper configuration of TLS credentials and mindful determination of timeouts for efficient network use.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update proto

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Set default timeout for agent configuration

Introduced a default timeout of 60 seconds in both gRPC and HTTP endpoints for agent configuration when none is specified. This change ensures that operations do not hang indefinitely and provides a reasonable default for client interactions. Additionally, the instantiation of a new agent client is now outside the retry loop to avoid repeated setup on transient failures.

Refactors service logic to optimize client connection handling by moving the agent client setup to occur before attempting retries, which should reduce overhead and improve clarity in error situations.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Subject: Move agent client creation into retry loop

Body:
Refactored the service manager's Run method to initialize the agent gRPC client inside the exponential backoff retry loop. This change addresses intermittent connection issues by reattempting client creation on temporary network failures, ensuring a robust setup before calling the Run method on the client.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
 2024-01-04 12:34:57 +01:00
Sammy Kerata Oina 34c5472485 NOISSUE - Remove multiple brokers (#31) 
* remove multiple brokers

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Standardize build process on NATS messaging

Removed conditional build flags for different message broker types, making NATS the default and only option. Simplified the Makefile by eliminating the toggleable message broker configuration and related docker-based setup. Updated the NATS URL in docker `.env` and docker-compose service dependency to reflect this change. This adjustment streamlines the build process and reduces configuration complexity.

Resolves issues with inconsistent messaging setups across various environments.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor message broker configuration

Standardized the environment variable for the message broker URL across services and updated documentation to reflect this change. Renamed the environment variable from COCOS_NATS_URL to COCOS_MESSAGE_BROKER_URL to improve clarity and maintain consistency in service configuration. This alteration facilitates future support for different message broker backends beyond NATS without further variable name changes.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
 2023-12-24 19:45:04 +01:00
Sammy Kerata Oina 5adf0349a3 COCOS-9 - Dockerize manager (#18) 
* Fix bug in agent service and state

The commit fixes a bug in the agent service and state files.
Previously, the condition to check the state in the agent service
was incorrect. It was checking the state directly instead of
using the GetState() method. This has been fixed by using the
GetState() method to check the state.

Additionally, a new GetState() method has been added to the
StateMachine struct in the state file. This method retrieves
the current state by acquiring a lock and returning the state
value.

The changes have been tested and verified to resolve the bug
and improve the accuracy of state checking in the agent service.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix bug in agent state machine

The bug in the agent state machine caused an error when attempting an invalid transition. This commit fixes the bug by properly locking and unlocking the state machine before and after transitioning to the next state. Additionally, the logger now correctly logs the current and next state during a valid transition.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix race condition in state machine

The commit fixes a race condition in the state machine implementation in the `Start` method. The race condition occurs when multiple goroutines try to access and modify the state concurrently. To fix this, a mutex lock and unlock are added around the critical sections of code to ensure exclusive access to the state variable. This prevents race conditions and ensures the state transitions are executed correctly.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix race condition in StateMachine.Start()

The StateMachine.Start() method was experiencing a race condition
when multiple events were being processed concurrently. This was
caused by not properly locking and unlocking the state machine
before and after updating the state. This commit fixes the issue
by adding proper locking and unlocking around the state update
operation. Additionally, the logging statement has been updated
to include the previous and next states for better debugging.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* add magistrala dep

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove mainflux

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger.

The commit message should be:

"Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger"

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix Makefile to properly set DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE

The Makefile has been updated to fix an issue with setting the DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE variables. The USER_REPO variable is now used to generate the DOCKER_PROJECT name following the Docker Compose guidelines. Additionally, the COCOS_MESSAGE_BROKER_TYPE variable is now properly set to "nats" if it is empty. This ensures that the correct values are used when compiling and installing the service.

Summary:
Fix Makefile to properly set DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE

Details:
- Update USER_REPO variable to generate DOCKER_PROJECT name
- Set COCOS_MESSAGE_BROKER_TYPE to "nats" if empty

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix notification topic in agent service and update NATS ports in Docker environment variables

The agent service's notification topic was incorrectly set to "channels.manager" instead of "agent". This commit fixes the issue by updating the notification topic.

Additionally, the NATS ports in the Docker environment variables were incorrect. The COCOS_NATS_PORT and COCOS_NATS_HTTP_PORT have been updated to the correct values.

These changes ensure that the agent service uses the correct notification topic and the NATS ports are properly configured.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add Dockerfiles for production and development environments

This commit adds two new Dockerfiles, one for the production environment and one for the development environment. The production Dockerfile sets up the necessary dependencies and builds the application, while the development Dockerfile simply copies the built application. Both Dockerfiles include the necessary SSL certificates for the application to work properly.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor Makefile and add new targets for building Docker images

The Makefile has been refactored to include new targets for building Docker images. The `make_docker` and `make_docker_dev` functions have been defined to handle the Docker build process. The `dockers` and `dockers_dev` targets have been added to build the Docker images for all services and development environments respectively.

This commit introduces changes to the Makefile to improve the build process and provide support for Docker images.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix makefile to generate Docker images for services

- Added DOCKERS and DOCKERS_DEV variables to generate Docker images for services
- Updated the dockers target to include DOCKERS
- Updated the dockers_dev target to include DOCKERS_DEV
- Updated the $(DOCKERS) and $(DOCKERS_DEV) targets to call the respective make_docker and make_docker_dev functions

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor Makefile and Dockerfile

The Makefile has been refactored to remove unnecessary sed commands and improve code readability. The changes include:

- Removed sed commands related to changing the broker in docker-compose.yml for nats and rabbitmq profiles.
- Removed the eer target from the Makefile.

The Dockerfile has been modified to include the WORKDIR directive.

These changes improve the maintainability and readability of the codebase.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix NATS URL in .env file and add Manager configuration in docker-compose.yml

The NATS URL in the .env file was updated to use the correct hostname for the broker. Additionally, the Manager configuration was added to the docker-compose.yml file.

This commit fixes the NATS URL in the .env file and adds the necessary Manager configuration to the docker-compose.yml file.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix incorrect host and port values in Makefile and .env

The commit updates the host and port values in the Makefile and .env files to fix incorrect values. The host and port values are updated to "cocos-manager" and "7003" respectively. This ensures that the correct host and port are used for the Manager HTTP and gRPC services.

This commit resolves the issue with the incorrect host and port values and ensures that the Manager services are configured correctly.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix Dockerfile and docker-compose.yml configurations

The Dockerfile and docker-compose.yml configurations have been fixed to address the following issues:

- Removed unnecessary COPY commands in the Dockerfile.
- Removed the "privileged" flag in the docker-compose.yml file.

These changes ensure that the Docker image is built correctly and the container is launched with the appropriate configurations.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove manual assignment

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update protoc CI

Signed-off-by: SammyOina <sammyoina@gmail.com>

* add paths

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove profiles

Signed-off-by: SammyOina <sammyoina@gmail.com>

* restore default config

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix Docker project name in Makefile

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
 2023-11-24 18:57:31 +01:00
Sammy Kerata Oina 298878c96d NOISSUE - Rename module to cocos (#22) 
* add stringer

Signed-off-by: SammyOina <sammyoina@gmail.com>

* rename module to cocos

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix fmt.Stringer implementation in computations.go

The fmt.Stringer implementation for Datasets and Algorithms in computations.go was fixed to correctly use pointers.

This commit addresses the issue where the String() method for Datasets and Algorithms in computations.go was not correctly implemented. The fix ensures that the String() method now correctly marshals the data to JSON and returns the string representation.

The changes made in this commit will improve the functionality and accuracy of the String() method for Datasets and Algorithms.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
 2023-11-22 13:15:19 +01:00
Sammy Kerata Oina f1f8f95653 NOISSUE - Fix dependencies (#13) 
* vendor

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Return agent changes

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Add missing import

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* remove vendor

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Fix formatting

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Formatting errors

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Update agent/api/grpc/client.go

Signed off: WashingtonKK

washingtonkigan@gmail.com

Co-authored-by: Sammy Kerata Oina <44265300+SammyOina@users.noreply.github.com>

* add linters and fix

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update ci

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove deprecated dependencies and
 use local agent and manager

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update mainflux

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix Jaeger URL in agent and manager main.go files

The Jaeger URL in the agent and manager main.go files was incorrect. This commit fixes the Jaeger URL by updating it to "http://localhost::4318/v1/traces".

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
Signed-off-by: SammyOina <sammyoina@gmail.com>
Co-authored-by: WashingtonKK <washingtonkigan@gmail.com>
 2023-10-27 22:12:53 +02:00
rodneyosodo f5f645db0a Remove MFXKIT Dep 
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
 2023-07-27 11:59:59 +03:00
Darko Draskovic 278331dc81 Add libvirt 
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
 2023-03-15 11:50:39 +01:00