Compare commits

..

70 Commits

Author SHA1 Message Date
Sammy Kerata Oina e376cf35a4 feat: Add support for GCC and libstdc++ packages in cocos_defconfig (#188)
Signed-off-by: Sammy Oina <sammyoina@gmail.com>
2024-07-31 12:43:40 +02:00
Danko Miladinovic 24a76a1685 NOISSUE - Fix attested TLS and attestation CLI (#184)
* update code to reflect updated libraries

* fix attestation CLI

* update manual test for attestation CLI
2024-07-30 16:05:49 +02:00
b1ackd0t 6d170e9ca4 NOISSUE - Fix failing tests (#186)
* test(events): failing tests

* fix: Add datapath as optional parameter

If datapath is not provided if should work as assume dataset is empty
If an invalid i.e non existent path is given it should return an error
2024-07-26 12:05:26 +02:00
b1ackd0t c9ec42deb2 fix(computation): Allow computation with empty dataset (#185) 2024-07-26 11:22:13 +02:00
Washington Kigani Kamadi 7498800d2d update attestation command example (#183)
Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2024-07-25 01:25:31 +02:00
Sammy Kerata Oina 46555a72ba NOISSUE - Error logs instead of failure on stderr (#182)
* warning logs

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* switch to error

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

---------

Signed-off-by: Sammy Oina <sammyoina@gmail.com>
2024-07-24 14:28:51 +02:00
Sammy Kerata Oina 67d01e39be COCOS-155 - Add python algo support (#178)
* * feat(algorithm.go): add support for algorithm type context
* feat(python.go): implement Python algorithm runtime
* fix(cocos_defconfig): add IPTABLES package

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update proto

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* small fixes

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* add metadata

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* debug

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* debug

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* chunk logger

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* debug logger

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* test lock

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* add req file

Signed-off-by: SammyOina <sammyoina@gmail.com>

* stream result

Signed-off-by: SammyOina <sammyoina@gmail.com>

* test with venv

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fix missing requirements file

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* result stream

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* modify test server

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* remove debugging and cleaning up

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* original repo

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* add missing header

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* downgrade protoc

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
Signed-off-by: Sammy Oina <sammyoina@gmail.com>
2024-07-23 16:38:03 +02:00
b1ackd0t 2f8109879c COCOS-168 - Allow running Computations without datasets (#175)
* feat(agent): Allow empty dataset

Allow running of algorithm with empty dataset since not all algorithms require datasets.
Allow state-machine transition from algo-received state to running state incase of no dataset provided

Fixes https://github.com/ultravioletrs/cocos/issues/168

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* chore(gitignore): Remove build artefacts

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* feat(algorithms): Add test algorithm for addition

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* refactor(addition): Modify addition algo to one file

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* fix(agent): move state transition to callback func

Move state transition from `receivingAlgorithm` to `running` to state call back function

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* feat(agent-event): Add `algoReceivedNoData` event

`algoReceivedNoData` is an event that is sent if we receive an algorithm and it should not have a dataset hence changes the state from `receivingAlgorithm` to `running`

* fix(agent-state): Change state depending on manifest

Change state from `receivingAlgorithm` to either `receivingData` if there is a dataset or `running` if there is no dataset provided

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

---------

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
2024-07-22 14:51:16 +02:00
Smith Jilks 88707c4668 NOISSUE - Fix Jaeger tracer (#177) 2024-07-22 14:49:42 +02:00
Washington Kigani Kamadi d76074ae41 COCOS-144 - Add Agent SDK Tests (#167)
* add tests and mocks

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix ci

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

update test

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix(agent/grpc): revert change

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix ci

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* refactor attestation and report tests

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

refactor tests

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

remove commented code

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

remove comment

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

remove comments

* add test cases

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

export agent errors

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

remove comm

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* fix tests

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

---------

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2024-07-11 20:53:31 +02:00
Sammy Kerata Oina 2ceb1c3562 COCOS-152 - Refactor algorithm execution and add visibility through logging and events (#176)
* * feat(agent): add support for binary algorithm execution
*
* feat(agent/algorithm): add Algorithm interface and binary implementation
*
* feat(agent/algorithm/binary): implement Run method for binary algorithm execution
*
* feat(agent/algorithm/logging): implement Stdout and Stderr writers for algorithm logging
*
* feat(agent/algorithm/logging_test): add tests for Stdout and Stderr writers
*
* feat(agent/events): add Service interface for sending events
*
* feat(agent/events/mocks): add mock implementation for Service interface
*
* refactor(agent/service): update runComputation method to use binary algorithm implementation

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * fix(logging.go): handle error when sending event in Write method of Stderr struct
* test(logging_test.go): add copyright header
* fix(backend_info.go): add missing type declaration in function signature
* fix(agent.go): rename progressbar variable to pb for clarity and consistency

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-07-11 12:24:19 +02:00
b1ackd0t dc16e8a997 NOISSUE - Use Magistrala Shared Packages (#174)
* refactor(env): remove internal env package

No need for this package since the parent package github.com/caarlos0/env does everything we need

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* refactor(jaeger): remove internal jaeger package

No need for this package since we can use magistrala exported package

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* refactor(metrics): remove internal metrics pkg

Use exported magistrala prometheus package

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* chore(dep): Update grpc and other dependencies

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* style(linter): remove enabled by default linters

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

---------

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
2024-07-10 15:48:37 +02:00
Sammy Kerata Oina f4e3e8e09c COCOS-157 - Provide abstractions for VM management (#171)
* abstract vm creation and allow stopping computation

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor QEMU configuration loading and execution in main.go

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * feat(agent-config): add support for sending agent configuration to manager

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * chore(checkproto.yaml): update protoc-gen and protoc-grpc versions

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * chore(auth): update mockery version to v2.43.2
* chore(main.go): update import path for vm package in agent
* chore(main.go): update import path for vm package in manager
* chore(go.mod): add github.com/google/logger v1.1.1 as a required dependency
* chore(manager_test.go): update import path for vm package in manager
* chore(logging.go): move logging.go to manager/qemu/vm package
* chore(logging_test.go): move logging_test.go to manager/qemu/vm package
* chore(vm_factory.go): rename vm_factory.go to provider.go in manager/qemu/vm/mocks package
* chore(vm.go): move vm.go to manager/qemu/vm package
* chore(vm.go): update import path for vm package in manager
* chore(vm_test.go): move vm_test.go to manager/qemu/vm package
* chore(vsock.go): move vsock.go to manager

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * fix(main.go): change import path for 'github.com/ultravioletrs/cocos/manager/qemu/vm' to 'github.com/ultravioletrs/cocos/manager/vm'
* fix(main.go): change vsock.Dial argument from 'vm.VsockConfigPort' to 'qemu.VsockConfigPort'
* fix(main.go): change import path for 'github.com/ultravioletrs/cocos/manager/qemu' to 'github.com/ultravioletrs/cocos/manager/qemu'

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-07-10 15:33:54 +02:00
Sammy Kerata Oina d1c9834d86 NOISSUE - Fix user role on RPC call (#173)
* * fix(interceptors.go): change role from auth.DataProviderRole to auth.ConsumerRole in AuthenticateUser call

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * test(interceptors_test.go): update role variable values in TestAuthUnaryInterceptor
* - Change role value from auth.DataProviderRole to auth.ConsumerRole in the "authorized result method" test case
* - Change role value from auth.DataProviderRole to auth.ConsumerRole in the "unauthorized result method" test case
* - Change role value from auth.DataProviderRole to auth.ConsumerRole in the "other method" test case

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-07-10 13:42:41 +02:00
Danko Miladinovic 654be60a80 COCOS-132 - Add progress bar for algo and data uploads (#162)
* add progress bar to CLI

* fix error handling

* fix comments errors

* add header

* add wraper for AlgoClient and DataClient

* add compile time check for wrapper structs

* refactor code
2024-07-09 09:35:40 +02:00
Danko Miladinovic 006897a57c COCOS-153 - Add host-data option (#163)
* add host_data option

* add CLI hostdata option and rename platform to backend

* move code for computation hash to a function

* rename getComputationHash to computationHash

* add default for backend information field switch
2024-07-08 15:32:13 +02:00
Sammy Kerata Oina 0e236bf2e8 * feat(hal/linux): add support for wasmtime package (#166)
* fix(hal/linux/Config.in): add source for wasmtime Config.in file

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-07-04 17:04:51 +02:00
Washington Kigani Kamadi ff5bade57d COCOS-145 : Support Other Private/Public Key Pair Algorithms (#147)
* feat: generate multiple key types using cli

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

pass cert to agent

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix ci

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

standardize key types

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

marshal public key

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

use openssl format for keys

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* update auth

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

update makefile

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

update agent

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

enhance parsing logic

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* pass key type as flag

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix ci

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

return key on successful signature verification

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix cli

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix ed25519 signing

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

ci

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* refactor code

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* fix(cli/result.go): dry code

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

---------

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2024-07-03 22:56:18 +02:00
Danko Miladinovic 7c090fd19f NOISSUE - Add CLI option to download ASK and ARK (#149)
* add cli option to download ask and ark

* add ARK and ASK to cert chain of the attestation report

* fix spelling mistake

* add explanation for aTLS testing

* remove commented code
2024-07-03 16:07:19 +02:00
Sammy Kerata Oina 30092c6a45 COCOS-135 - Refactor gRPC auth interceptors and update dependencies (#140)
* Implement gRPC auth interceptors and update dependencies

Introduced gRPC authentication interceptors for unary and stream calls in the agent's API to enhance security. These interceptors verify signatures based on roles before handling requests, rejecting unauthorized ones. The authentication logic was refactored, adding relevant error handling.

Added mocks for authentication interfaces to facilitate testing and replaced direct errors with wrapped ones to provide additional context.

The Makefile now includes a version pin for the Mockery tool, ensuring consistent mock generation across environments.

Updated the gRPC server setup in `grpc.go` to utilize the new interceptors. This change aims to reinforce the API's access control by authenticating users according to their roles and is expected to prevent unauthorized access to sensitive endpoints.

Lastly, dependencies in go.mod and go.sum were updated, most notably the inclusion of testify for improved testing capabilities.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix linting errors

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * chore(Makefile): remove unused variables and update 'mocks' target

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-07-01 21:39:35 +02:00
Sammy Kerata Oina 617b595559 Add Docker support to system configuration (#164)
Added support for Docker in the system configuration file. This includes packages like libseccomp, CA certificates, Docker CLI, Docker Compose, Docker Engine, containerd, and runc.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-07-01 21:37:58 +02:00
Sammy Kerata Oina 4c4161ccfb Fix capitalized Vnc field and increment Vnc on run (#139)
Corrected the struct field 'vnc' to 'Vnc' to match Go's public field naming conventions and to make it exportable. Each QEMU virtual machine's VNC display number now increments upon instantiation to avoid display conflicts.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-06-06 16:34:13 +02:00
Sammy Kerata Oina ff64c10ac3 NOISSUE - Add new key management command to CLI (#136)
Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-05-29 13:09:54 +02:00
Danko Miladinovic 0574abc228 NOISSUE - Add Rust script for fetching platform data (#133)
* add rust program for fetching platform data

* fix new line error

* add CLI options to add the measurement to platform_info.json file

* add documentation for platform info testing

* add explanation for sev-snp-measure

* delete excess space

* fix minor errors

* fix minor errors

* add file permision constant
2024-05-28 18:08:07 +02:00
Sammy Kerata Oina 2ce112cc1b COCOS-103 - User authN and AuthZ using digital signatures (#128)
* Update Go to 1.22 and enhance security features

- Upgraded the Go version in GitHub Actions workflows to 1.22.x for latest features and security patches.
- Added RSA public key field `UserKey` in `Dataset` and `Algorithm` to reinforce data integrity and encryption.
- Refactored `Result` method in `agentService` to use `containsID` for improved readability and potential performance benefits.
- Updated `grpcserver.New` and `internal/server/grpc` invocations to pass `agent.Service` by value in line with recommended Go practices.
- Introduced `grpc.StreamInterceptor` with no args in `Server.Start` which seems to be an initial step for future stream interceptor configuration.

These changes prepare for stronger data security measures, maintain compatibility with the latest Go features, and improve code quality regarding service struct usage. Potential follow-up is needed to configure the stream interceptor and to ensure the new RSA key field is appropriately utilized in data handling.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor auth system and protocol buffers

Enhanced the authentication system by adding context support and an improved user-role model. Implemented robust RSA public key verification for users and a restructured interceptor logic specific to stream types, streamlining the auth process. Updated protocol buffers and associated structures to accommodate user keys as byte slices, aligning with standard cryptographic practice. CLI commands for algorithms and datasets now require a private key file path argument for signing, strengthening security during interactions.

This comprehensive overhaul addresses security and efficiency considerations in the RPC framework and aligns with best practices for key handling. By streamlining and securing the user authentication process, the agent service's reliability is greatly improved, directly impacting the robustness of the entire computation pipeline.

- Refactored auth: added role-based user validation, context handling
- Reworked interceptors: separated stream types, fortified signature checks
- Updated protocol buffers: user public keys as byte slices for standard compatibility
- Enhanced CLI: introduced private key argument, ensuring secure algorithm and dataset submission
- Improved server and SDK contracts to align with auth changes

Related issues:
- Implements user roles and auth context [#103]
- CLI security enhancement for private key management

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Updated PEM decoding for key parsing in CLI and tests

Added `encoding/pem` to decode PEM blocks when parsing private and public keys across CLI commands and test computation scenarios, ensuring compatibility with key files. This enhances robustness in key handling by supporting PEM encoded keys. The update also includes registration of a new Keys command in the CLI.

Refactored code is now compliant with common key formats, addressing potential parsing issues.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix auth signature encoding and improve CLI usage example

The authentication system now decodes base64 strings before verifying signatures to align with the expected format. Additionally, the signature generation now encodes the output in base64, ensuring consistency across the auth process.

The CLI help message for the `result` command is enhanced by providing a usage example, making it more user-friendly and informative.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor containsID to handle dynamic fields

Updated the `containsID` function to accept a field name parameter, enabling dynamic field lookup within the reflection logic. This change facilitates the use of the function for various struct fields, improving code reusability and flexibility.

CLI command 'data' now requires an additional argument for the private key file path, outlined in the usage example update, reinforcing command clarity and user guidance.

Resolves issues with hardcoded field lookups and enhances CLI usability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Remove extraneous newline in key generation log output

A redundant newline after the success message in the key generation command was removed to clean up log output formatting. This change ensures a more consistent and professional appearance of the CLI tool's messages.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Implemented auth service in gRPC startup

Added authentication services to the gRPC server initialization to enforce security measures. The gRPC server's New function now includes an `authSvc` parameter, requiring instantiation of the auth service before starting the server. Failure to create the auth service results in a fatal error, halting the process to avoid running without protection. Tests have been updated to include `nil` values for the auth service parameter to maintain their functionality without authentication.

Refactored `grpcserver.New` to accept the new auth service, and updated the main agent startup logic to create and inject the auth service. Added the auth middleware interceptors to the server options, which ensures that each gRPC call will undergo authentication.

This change is a step towards secure communication, and affected components should now consider the authentication requirement.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor config read logic and update agent setup

Improved the configuration reading in `cmd/agent/main.go` to handle larger payloads by reading data in chunks and checking for EOF, ensuring that all config data is captured even if it exceeds the initial buffer size. Enhanced the `test/manual/agent-config/main.go` to require additional command-line arguments, improving the setup process by explicitly requiring paths for data, algorithm, and public key as well as a boolean for attested TLS. Also updated the hashing method to SHA3 for the algorithm and data files, and included the hash and public keys as part of the agent, dataset, and result consumer configurations. These changes will make the agent setup more robust and provide better integrity checks for the involved files.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor run method to agentService

Moved the run function into agentService for better encapsulation and maintainability. This refactoring includes capturing both stdout and stderr during algorithm execution, enabling more informative debugging through enhanced logging. Consequentially, the run method now references members through the service instance, aligning with object-oriented best practices and improving code coherence.

Resolves issue with insufficient execution details when computations fail.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor computation data handling to use filepaths

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor error logging and ensure consistency

Replaced usage of the standard log package with a custom logger for error reporting to standardize error logging throughout the application. Additionally, introduced graceful shutdown by returning from the main function rather than forcing exit when failing to create auth service, aligning the application's error handling strategy.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor auth initialization and key file handling

Improved the readability and maintainability of the authentication service initialization by adding line breaks for logical separation. Also, standardized key filenames in the CLI key generation by introducing constants, enhancing code clarity and reducing the likelihood of file-naming errors.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor auth verification logic for improved security

Removed an extraneous line in the `verifySignature` function that was not necessary for the signature verification process. This change simplifies the code and improves readability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor payload structures to simplify API

Removed the 'provider', 'id', 'consumer' fields from protocol buffers, gRPC services, and related functions across various files to streamline the data model and align with the new authentication system based on cryptographic verification rather than string identifiers. This results in more efficient data handling and a reduction in unnecessary payload data, while enhancing security by making entity validation strictly cryptographic.

The changes affect agent-SDK interactions, CLI tools, and related services, ensuring only the necessary data (algorithm/data bytes, user keys, and hashes) is transmitted and processed. Consequently, the core computation algorithm and dataset handlers now rely on indexes derived from context to associate data with respective manifest entries, thus maintaining the ability to link to specific computation manifests without relying on explicit IDs in the payload. Additionally, refactored authentication methods now enforce role-based security seamlessly through metadata.

This approach enhances privacy by avoiding transmission of potentially sensitive strings over the network and by ensuring that only internal indices, not globally interpretable identifiers, are used to process computations.

Aligned with the broader architectural goal of simplifying and securing the platform's core services, this change paves the way for upcoming revisions to the authentication scheme that will further consolidate role-based security and improve system integrity.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance CLI security with key paths

Removed the section on running computations from the CLI README as it may no longer be necessary or the functionality has been moved elsewhere. Required private key file paths for algorithm, dataset upload, and result retrieval commands to enhance security. This change associates each action with a specific identity, ensuring secure and traceable operations. Additionally, updated the manual test commands to reflect this new requirement.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix ci

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix fmt

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-05-28 14:10:13 +02:00
Sammy Kerata Oina aebe01a873 NOISSUE - Updated README with static binary instructions (#130)
Enhanced the manual algorithm testing documentation by including steps to install additional dependencies and guidance on creating static binaries. This ensures wider compatibility and ease of deployment for python programs tested with this framework.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-05-22 14:40:34 +02:00
Sammy Kerata Oina 4c681569f1 NOISSUE - Add sha3-256 file hashing command to CLI (#129)
Introduced a new command for the CLI to calculate the sha3-256 hash of a given file. The update enhances the utility's cryptography toolkit by allowing users to easily generate and verify file integrity directly through the command-line interface.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-05-20 16:56:42 +02:00
Sammy Kerata Oina c274521faf Refactor computation data handling to use filepaths (#126)
Changed the internal representation of algorithms and datasets within the service from byte slices to file paths, writing received data directly to temp files. This modification allows for handling potentially large data sets without the need to load them entirely into memory, improving the memory efficiency and scalability of the service. Additionally, it aligns the call signature of external algorithms with the new approach, updating documentation and examples accordingly. Updated the linear regression example for consistency with the new data handling process.

Resolves issues with memory bloat when processing large datasets.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-05-13 17:16:39 +02:00
Sammy Kerata Oina 226704cf0d COCOS-122 - Enable streaming RPCs for Algo and Data services (#123)
* Enable streaming RPCs for Algo and Data services

Modified the gRPC service definitions for Algo and Data methods to use stream processing, enabling the handling of larger datasets and algorithms without being limited by memory restrictions. This allows client and server to send chunks of data sequentially rather than requiring the entire payload to be loaded into memory at once.

Updated server implementations to accumulate data from multiple chunks, allowing for more efficient processing and communication when dealing with large files. Client implementations have been adjusted to segment and send data in a streaming fashion.

Removed previously existing synchronous client code as it became redundant with the new streaming approach, streamlining the client's communication patterns with the gRPC backend.

This change allows for better resource management, especially in systems with constraints on memory, improving overall scalability and performance of the data and algorithm processing pipeline.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor algorithm ID check logic

Simplify the algorithm validation logic in the agent service by replacing the previous containment check with direct ID comparison. This change streamlines the error handling for undeclared algorithms and hash mismatches, while also ensuring clear and direct provider validation. The modifications enhance the readability and maintainability of the code without altering functionality.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Updated README to build single-file executable with PyInstaller

Modified the PyInstaller command in the manual testing README to bundle the linear regression script into a single executable file. This simplifies distribution and execution of the script by eliminating the need for multiple dependency files.

Ref: Optimization of deployment process
Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-05-13 13:14:50 +02:00
Danko Miladinovic 479598a663 COCOS-116 -Separated networking setup from agent start up script (#118)
* added network setup script

* removed fixed name enp0s2 from code

* fixed small typo

* changed the capitalization of a letter
2024-05-01 17:03:28 +02:00
Sammy Kerata Oina 8d082567d7 COCOS-101 - Switch to self-contained algorithms as executables (#119)
* Switch to self-contained algorithms as executables

Transitioned from using Python scripts to self-contained binary executables for running algorithms, improving modularity and reducing dependencies. This change removes the reliance on a Python environment, as evident by the removal of Python setup and packages from the build configuration. The service now creates temporary executable files for algorithm runs, handling all permissions and cleanup, enhancing security and maintaining clean execution states. A warning is logged if computation fails, aiding in debugging. Additionally, updated manual tests to reflect these changes in the agent's handling of algorithms.

Refactors:
- Removed Python runtime const since it's no longer needed.
- Updated documentation and test commands to reflect the change from .py to .bin for algorithm files.

Build config:
- Removed Python and pip packages to reduce the build size and complexity.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update agent service.go file with new constants and file permission

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refine singular usage of 'algorithm' across modules

Standardized terminology throughout the project to refer to 'algorithm' in the singular form rather than plural. Streamlined various documentations, string constants, function names, and variable names to bring cohesiveness and eliminate ambiguity when handling algorithms across README files, CLI interfaces, and internal API representations.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix state names and indices in state_string.go

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-29 15:48:17 +02:00
Sammy Kerata Oina 4b5000d107 NOISSUE - Allow termination of manager (#95)
* Implement manager client heartbeat mechanism

Introduced a heartbeat system in the manager service to maintain an active link with client agents. The updates involve sending periodic heartbeat signals controlled via the new environment variable `MANAGER_HEARTBEAT_INTERVAL`, defaulting to 1 second. The protobuf definitions, service interfaces, and server logic have been updated accordingly to handle the incoming signals. This change allows better tracking of active client connections and could be used for features like auto-reconnect or resource cleanup for lost connections in the future.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance gRPC server to handle client termination

Introduce the capability for the gRPC server in the manager module to process termination requests from clients. Server and client proto buffers have been updated to include a Termination message, allowing clients to signal the server to stop processing. Handling for communication and termination flows involving context cancellation have been added across server implementation, manager client, and main execution logic to properly respond to termination signals. This ensures clean shutdowns and resource deallocation when a client requests termination, improving the robustness of the system.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor gRPC server context handling

Removed the redundant storage of context in grpcServer struct and modified the NewServer constructor to exclude the context parameter. Updated the Process method to derive context from the stream directly, ensuring better context scoping and adherence to proper gRPC patterns. This change simplifies the server's context management and ensures contexts are more accurately associated with their corresponding streams, which can improve debugging and request cancellation behavior.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update protoc to v4.25.3 and remove heartbeat system

Upgraded protoc version to 4.25.3, ensuring compatibility with the latest protocol buffer features and potential performance improvements. Additionally, removed the entire heartbeat system, involving both its gRPC server implementation and associated client-side logic. This elimination suggests a shift in the service health-check strategy, potentially to a more modern or efficient model.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Removed heartbeat functionality for manager service

The heartbeat feature and its related configuration have been removed across the application. This includes the removal of the Heartbeat method from the Service interface, the associated interval setting in the config structure, and related environment variable documentation. The change implies an architectural shift towards alternative methods of service availability checking or potentially a move to stateless service design. The cleanup removes unnecessary code, thus simplifying the overall service logic.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Introduce Terminate message and streamline gRPC server logic

- Added a new `Terminate` protobuf message to handle termination requests more explicitly.
- Removed redundant context cancels in gRPC server stream processing to rely on context propagation.
- Updated channel usage to match the altered stream message types; this aligns message handling with the updated protobuf definitions.
- Passed client authentication info to service run function to enhance the security during service execution.
- Utilized stream's context directly in error groups for improved clarity and error handling scope.
- Refactored message descriptors in protobuf to maintain correct ordering after introducing new message types.

These changes enhance the codebase's maintainability, simplify the logic, and align with the updated messaging protocol.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update agent configuration in main.go

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-29 14:08:42 +02:00
Sammy Kerata Oina 64f7e7f7fd NOISSUE - Refactor single algorithm processing (#117)
* Refactor single algorithm processing

Simplified the agent service's algorithm handling logic to process a single algorithm instead of multiple. This change:
- Removed the `Algorithms` type and associated stringer implementation.
- Updated the state machine and service logic to expect a singular algorithm, aligning the agent's internal state transitions with the new model.
- Adjusted the manager service and computations test server to mirror these changes in their respective payload structures, ensuring API and test consistency.
- Altered README files to reflect the simplified interaction model and removed outdated descriptions.
- Reverted the protoc-gen-go version used for generating protobuf files to maintain compatibility with the rest of the codebase.

The single-algorithm approach streamlines the computation running process, reducing complexity and potential error conditions. It directly impacts how external services will construct and send computation requests.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update protoc-gen-go version to v1.33.0

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor variable name in computations.go and grpc.go

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-17 13:34:54 +02:00
Danko Miladinovic 3a14896555 NOISSUE - Attested TLS (#99)
* added initial code for attested TLS

* added client validation and verification

* fixed bugs for attested TLS

* updated README for manual testing

* fixed CI errors

* removed SNP pollicy from agent config

* added attested TLS config param to AgentConfig

* generated manager.pb.go for protoc v25.2

* updated proto-gen-go version on CI

* generated agent.pb.go to match newest proto gen version

* define errors for error handling

* fixed comments
2024-04-16 15:19:19 +02:00
Drasko DRASKOVIC b57bafd310 Fix usage (#114)
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2024-04-16 00:42:47 +02:00
Drasko DRASKOVIC 9e6fb31d90 Fix links and add features (#113)
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2024-04-15 22:26:41 +02:00
Sammy Kerata Oina 2b760ec207 NOISSUE - Streamline client identification (#105)
* Update protoc to v4.25.3 and streamline client identification

Protocol Buffer compiler `protoc` has been upgraded to version 4.25.3. This update unifies the version across multiple generated files to ensure compatibility and take advantage of any bug fixes and performance improvements in the new release.

Additionally, the client identification process has been refined. The redundant `WhoAmI` message and the corresponding checks have been removed from the gRPC server implementation. Clients are now identified via their address from the `Process` stream's context as soon as a connection is established, simplifying the code and potentially reducing handshake time.

This change sets the foundation for a leaner communication protocol between manager and agents, and could contribute to lower latencies in client-server interactions.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix goroutine bug in grpc server

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update PROTOC_VERSION to 25.3

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor server reference in readme

Updated the deployment instructions in the README to point to the correct server location after consolidating test server documentation. Removed obsolete `manager-server` module and associated main entry point to align with the new architecture and streamline workflows.

Resolves issues with outdated links and cluttered repository structure.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-15 22:08:37 +02:00
Sammy Kerata Oina 2e612a2867 COCOS-108 - Enhanced state machine Synchronization (#112)
* Enhanced state machine synchronization

Introduced a wait group to the state machine to manage go-routine synchronization effectively. The Start method now ensures that the state machine's execution is tracked, preventing potential early termination of go-routines during state transitions. The test suite is updated to wait for the go-routine to complete before proceeding, ensuring more reliable test execution.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Remove unnecessary code and fix test case in agent package

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-15 18:55:58 +02:00
Drasko DRASKOVIC f2f57ac413 Fix license (#107)
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2024-04-14 18:05:16 +02:00
Drasko Draskovic d50a6fcfda Fix releases link
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2024-04-14 17:55:29 +02:00
Drasko DRASKOVIC d323fd3b7e Fix docs link (#106)
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2024-04-14 17:49:10 +02:00
Filip Bugarski 35c2a9e92a Update cocos documentation (#89)
* Update cocos documentation

Signed-off-by: fbugarski <filipbugarski@gmail.com>

* inserted go run commands into separate lines

* updated syntax higlighting to bash

---------

Signed-off-by: fbugarski <filipbugarski@gmail.com>
Co-authored-by: Danko Miladinovic <danko@etf.rs>
2024-04-13 18:33:10 +02:00
Sammy Kerata Oina 5c406743e0 NOISSUE - Add attesatation validation/verification options (#87)
* Standardize attestation report data size to 64 bytes

Redefined the report data for attestation requests from a variable byte slice to a fixed 64-byte array across various critical components. This change enforces a standardized data length, simplifying validation logic and ensuring consistency when handling the attestation data for both the server and client end-points, logging functionalities, and the CLI tool. Updated attestation data handling in the SDK to accommodate the new fixed-length constraint.

By strictly adhering to the 64-byte requirement, the modifications promote robust input checking and prevent potential issues related to dynamic data length processing.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor attestation validation logic

Streamlined attestation validation in the CLI with a new configuration parsing approach. Introduced a retry mechanism with timeout and backoff for HTTPS getters that facilitate remote trust validation. Clarified expected data lengths in proto comments across attestation and manager services, ensuring consistency and correctness of the cryptographic data fields.

This modification enhances maintainability by replacing a verbose struct with a configurable object model, thus simplifying parameter handling. Robustness is improved through failure-retrieval strategies during remote validation, and additional in-code documentation specifies cryptographic constraints for critical data elements.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add new attestation validation flags

Enhanced attestation validation in the CLI with additional flag support for FAMILY_ID, IMAGE_ID, REPORT_ID, REPORT_ID_MA, MEASUREMENT, CHIP_ID, and TCB-related fields. This update enables fine-grained control over attestation report criteria, ensuring stricter validation conforming to protocol requirements. Adjusted the default value for MinimumGuestSvn to align with the new policy specifications.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance attestation policy configuration

Introduced several new configuration flags to the CLI's attestation policy, including options for minimum AMD-SP firmware builds, revocation list checking, and network restrictions. Expanded trust criteria parameters with the support for trusted author and identity key hashes. The default VMPL setting is now explicitly initialized. These changes improve security controls and offer more detailed attestation verification settings.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Simplify attestation validation process

Refactored the attestation CLI command to improve usability by removing the need for explicitly provided report data as an argument. Default values for configuration parameters have been centralized into constants for maintainability. Additionally, integrated data parsing functions to streamline the validation checks with robust error handling for cases such as invalid hex strings and file I/O issues.

This enhancement reduces user error and the complexity of entering attestation data by parsing relevant information from within the environment, all while maintaining the same security standards.

The process of marking flags as required for CLI commands has been corrected to prevent runtime errors and improve command reliability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refined attestation validation logic

Enhanced attestation command handling by removing the hardcoded report data length and replacing it with the standard SHA-512 hash size, ensuring dynamic compatibility with hash lengths. Introduced size constants to improve code readability and enforce explicit length checks on attestation input data, raising errors when requirements are unmet. This preventative measure aims to avert potential runtime errors stemming from unexpected data sizes. The change also simplifies the minimum guest SVN description for clarity.

References to removed unnecessary whitespace maintain code cleanliness. These adjustments culminate in a more robust and maintainable attestation process, in accordance with the best practices for secure handling of cryptographic data and compliance with AMD's ABI format specifications.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhanced attestation documentation and command syntax

Updated CLI documentation to include detailed descriptions of the attestation retrieval and validation process, along with a comprehensive list of new flags for the validation command, providing users with extensive control over the attestation verification process. Updated command syntax with flags for specifying report data, improving command clarity and consistency. These changes ensure better user guidance and offer a more robust attestation handling experience.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enforce report data size validation uniformly

Refactored report data size validation to ensure it is performed consistently across the system. Removed the hard-coded length check from attestation request validation in favor of a centralized verification based on the sha512 digest size. Also, eliminated fallback mechanism to the SHA-512 hash of report data and made the length requirement explicit, leading to immediate failure if not met. This harmonizes the error handling logic, reinforces data integrity checks, and simplifies debugging by removing ambiguous length correction behavior. Adjusted related validation functions to use a common utility, enhancing maintainability.

Fix spelling errors in log messages for attestation validation commands to improve clarity in output.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Initialize attestation config with defaults

Enhanced the attestation command initialization by setting default values for the `Config` struct to prevent nil pointer exceptions and potentially streamline configuration handling. Moreover, corrected a flag declaration by removing shorthand.P support for 'permit_provisional_software', aligning it with other flags for consistency. This change should improve stability and user experience while configuring attestation policy.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* nil check prior

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix typos

Signed-off-by: SammyOina <sammyoina@gmail.com>

* use file for attestation report

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refine attestation validation with default policy values

Adjusted default values and constants related to the attestation configuration for enhanced clarity and consistency. Changed the default minimum guest SVN value to align with new policy requirements and introduced a default guest policy constant. Additionally, standardized byte array placeholders for attestation validation fields to prevent potential issues with uninitialized bytes and ensure robust input validation checks. These changes promote more intuitive and secure attestation validation procedures.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refine attestation defaults and flag requirements

Updated the default guest policy to use hexadecimal notation for clarity and added a default minimum version for the firmware API version flag. Made 'chip_id', 'measurement', 'report_id', and 'report_id_ma' flags mandatory to ensure critical attestation data is provided by the user, enhancing validation robustness. Added a check for 'measurement' field length for consistency in input validation.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Optimize attestation file format

Standardized attestation file format across the codebase by moving the 'attestationFilePath' constant from the function scope to a higher scope within the package. Switched the file extension from .txt to .bin to reflect binary data storage, improving consistency and potential performance benefits from binary file handling.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Optimize default values for attestation validation

Removed unnecessary empty initializers and replaced hardcoded empty values with nil for optional fields in attestation validation flags. Introduced a constant defaultReportIdMa for the REPORT_ID_MA field to provide a meaningful default instead of an empty one. Streamlined the command setup by removing the requirement flags for fields that are now optional and unchecked if unset, reducing burdens on the user for providing unnecessary input.

The change simplifies the command's usage and eliminates the need for users to specify values for certain fields that can be optional, improving the command's flexibility and user experience. Additionally, setting a specific default for REPORT_ID_MA ensures clarity in cases where this is not provided by the user.

This update enhances the maintainability and usability of the attestation validation functionality.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor hardcoded attestation report size

Introduced a constant `size64` to replace the previously hardcoded array size of `64` used across various Attestation methods. This change enhances code maintainability and readability by centralizing the size definition, making it easier to update in the future if necessary.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Updated protocol buffer dependencies and cleaned CLI flags

Protobuf compiler versions were updated to v4.25.2 across various RPC service definitions for agent and manager packages, ensuring compatibility with the latest features and fixes. In the CLI attestation code, unnecessary flags (`DisallowNetwork` and `PermitProvisionalSoftware`) were removed to streamline user options, reflecting a more secure and user-focused configuration interface. Added an input validation check in `attestation.go` for CA bundle presence when a product name is set, enhancing the reliability of attestation validation. Also rectified a bug in `grpc.go` by fixing an incorrect variable assignment for reading the key file content, thereby preventing potential TLS-related errors.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Updated protoc version and adjusted proto file paths

Bumped the protocol compiler version to 25.2 to align with our dependency updates and ensure compatibility. Additionally, revised the proto file paths under the continuous integration setup, moving from 'manager/' to 'pkg/manager/' to reflect the recent directory restructuring. The changes ensure that proto file checks and comparisons are conducted in the correct file locations, preventing potential build and sync issues in future developments.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance hash size consistency in gRPC attestation

Aligned the hard-coded byte array size for `ReportData` with `sha512.Size` constant to ensure consistency and maintainability in gRPC attestation requests and responses. This change mitigates the risk of future errors if the SHA512 hash size standard is altered.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Optimize agent-config listener loop

Removed the goroutine wrapping the listener loop in 'agent-config' manual test to streamline and simplify the connection handling logic. This change ensures the loop runs on the main goroutine, improving the readability and maintainability of the code.

Refactoring assists in avoiding potential race conditions and makes the server's flow easier to understand for future debugging and development.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor report data size checks to use constant

Centralize the report data size definition by replacing various size checks and array declarations with a constant `ReportDataSize`. This streamlines code maintenance and ensures consistency across all instances where report data size is validated or used. The changes remove direct references to the hash function output size, decoupling the report data size from the hash function's characteristics and allowing easier updates if the data size requirement changes.

These modifications bolster the codebase's flexibility for potential adjustments in security protocols or data handling specifications.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor config JSON handling in attestation CLI

Introduced an `exampleJSONConfig` constant containing a sample configuration to improve code readability and maintainability. Previously, the example JSON configuration for the attestation CLI was an inline string, making the code cluttered and less maintainable. With this change, the example configuration is now stored as a constant, resulting in cleaner command flag setup and enhanced clarity. This constant is used in the command flag description to guide users when providing their custom configuration. Additionally, refactored the command flags related to the configuration by aligning and sorting them for better code organization.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-09 10:33:26 +02:00
Smith Jilks b6f116474d Cocos-91 - Update Makefile and cocos documentation (#93)
* Update Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update cocos documentation

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update cocos documentation

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update cocos documentation

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Remove install rule from Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update agent README.md

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update agent README.md

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update README.md files

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Remove INSTALL_DIR from Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Fix comments in README.md files

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

---------

Signed-off-by: Jilks Smith <smithjilks@gmail.com>
2024-04-08 15:59:38 +02:00
Sammy Kerata Oina 3e3e02f132 Enhance security with SHA-3 and command-line args (#97)
Updated the hashing algorithm from SHA-256 to SHA-3 for improved security and reliability in the computation service. Modified the service to accept dataset and algorithm paths as command-line arguments, enhancing flexibility and user control. Increased robustness of the service's error handling by incorporating file read validations and updated usage instructions in both the README and the program's startup error messages.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-06 23:45:13 +02:00
Danko Miladinovic 4b38ac09dd NOISSUE - Added SEV-SNP support (#98)
* added SEV-SNP support

* updated documentation for SEV-SNP

* changed variable names to adhere to the GO naming convention

* changed the variable name from V-sock to Vsock
2024-03-22 10:55:09 +01:00
Danko Miladinovic 1a322f8ecb NOISSUE - Fix network interface problem for agent (#96)
* Added bus/addr for pcie net

* Remove SCSI parameters

* Remove bus parameter

* Removed blank lines
2024-03-15 16:37:41 +01:00
Sammy Kerata Oina 8c2caec395 NOISSUE - Use standard GitHub templates for issues and PRs (#92)
Implemented new GitHub templates to streamline the contribution process. Bug report and feature request templates guide users in creating detailed and actionable issues, while a universal pull request template ensures systematically structured submissions. These additions facilitate more efficient reviews, and prioritize issues effectively. Additionally, configured the repository to disallow blank issues, encouraging use of the templates for better project management.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-03-13 13:06:51 +01:00
Sammy Kerata Oina de60358c02 NOISSUE - Enforce 32-byte length check for hashes (#86)
* Enforce 32-byte length check for hashes

Introduced a validation step in the manager service to ensure both algorithm and dataset hashes are of byte length 32 before provisioning VMs, preventing runtime errors due to invalid hash lengths. The test manager-server now generates a valid 32-byte hash using SHA256 for mock data to align with the new validation requirement.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* use equal comparison

Signed-off-by: SammyOina <sammyoina@gmail.com>

* use a constant

Signed-off-by: SammyOina <sammyoina@gmail.com>

* use constants

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-24 22:22:25 +01:00
Sammy Kerata Oina 997fb3bf48 COCOS-83 - Add hash verification for datasets and algorithms (#84)
* Add hash verification for datasets and algorithms

Enhanced data integrity checking by incorporating hash fields in Dataset and Algorithm structures, and modified the corresponding service logic to validate these hashes during processing. The update includes SHA-3 for hash computation, replacing the former SHA-256 usage, and ensures the provided data matches the expected hash from the manifest to prevent processing malformed or corrupted data.

- Introduce `Hash` field to both Dataset and Algorithm structs to store the expected hash value.
- Implement SHA-3 hashing within service methods that process the data, ensuring consistency with newly added `Hash` fields.
- Add error handling for hash mismatches, preventing further processing and alerting to potential data integrity issues.
- Update Protocol Buffers serialization to accommodate the new hash fields for gRPC communication.
- Modify manager service's Run method to pass the hash information when creating agent configurations.

Go module dependencies were updated to include the new SHA-3 package and upgrade Go version to 1.21.6 for improved stability and compatibility.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Remove identifiers from protobuf and related code

The protobuf definitions and related service handling code have been revised to drop specific identifier fields (`AlgorithmID` and `DatasetID`) to simplify API responses and internal function signatures. These removals streamline the overall data flow between components, reduce unnecessary data transmission, and lead to an aligned server-client expectation where identifiers are no longer a part of the response payload. Consequently, these changes simplify the logic within various functions and client commands, reinforcing encapsulation by ensuring that internal identifiers do not need to be managed or exposed unnecessarily.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-19 15:31:35 +01:00
Sammy Kerata Oina 722b463b6a NOISSUE - Use a single listener for logs and events (#82)
* add handler

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor gRPC and Protobuf integration for manager service

- Shifted Protobuf message definitions to a separate package `pkg/manager`.
- Updated references throughout the codebase to import and use the new package for gRPC service definitions.
- Enhanced AgentLog message with additional fields `level` and `timestamp`.
- Removed direct dependencies on old Protobuf-generated types in favor of the new package.
- Deleted obsolete Protobuf-generated files as they are now superseded by the new `pkg/manager`.
- Streamlined event publishing and gRPC handling in the manager service to use the updated Protobuf messages.

This refactoring improves modularity by centralizing Protobuf message definitions and decouples internal representation from the gRPC interface, aligning with best practices for microservice architecture. Additionally, the enriched logging structure paves the way for more detailed and fine-grained log analysis.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor vsock event/log handling and config

Streamlined event and log services in the manager by moving vsock listening functions out of `managerService` initialization and into dedicated `RetrieveAgentEventsLogs` methods. This change decouples the manager service creation from the actual start of log listening, adding clarity and flexibility in service management. Also moved logging middleware invocation outside of network handling loops to avoid unnecessary overhead.

Additionally, the agent's vsock port configuration is now dynamically passed to the `New` function in the `events` package instead of relying on a hardcoded constant, allowing for greater configurability and testability.

Finally, updated message structures for event and log sending to conform with the `ClientStreamMessage` definitions. These modifications should improve parsing and handling consistency and prepare our system for future enhancements related to inter-process communication.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix linting errors

Signed-off-by: SammyOina <sammyoina@gmail.com>

* correct path to generated files

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix comments

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove uneccessary comments

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-15 17:02:05 +01:00
Sammy Kerata Oina 938dd6cb78 NOISSUE - mTLS support across services (#71)
* Implemented mTLS support across services

Extended gRPC configuration to support mutual TLS (mTLS) in agent and manager components for enhanced security. This includes the loading of Certificate Authority (CA) certificates, server, and client certificates, and keys. Updated README documentation to reflect the new environment variables required for mTLS configuration. Additionally, streamlined secure gRPC client connection setup and logging messages to indicate whether a service is running with TLS, mTLS, or without TLS.

The change ensures secure communication between services by verifying both client and server identities, thus addressing potential security concerns in network-level interactions.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance agent cert handling and update copyright

- Implement function to create certificate files for the agent configuration dynamically, ensuring file paths are updated to reflect newly created files. This improves the agent's setup process by automating the certificate handling.
- Update copyright clause to reflect the new owning entity, Ultraviolet, affirming correct attribution and compliance with legal requirements.
- Refactor gRPC client connection code to remove redundant package alias, streamlining the codebase and improving readability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor cert loading with fallbacks

Removed redundant certificate file creation logic in the agent module and introduced a more robust loading mechanism in the gRPC server module to support direct byte content aside from file paths. This change simplifies the initial setup process for the agent by removing the need to create certificate files preemptively, thereby streamlining deployment in environments with varying filesystem access. It supports using certificate contents directly, enhancing compatibility with in-memory configurations or environments where file storage may not be ideal.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-08 10:07:51 +01:00
Sammy Kerata Oina e86860b9ea NOISSUE - Enhanced gRPC server concurrency with errgroup (#81)
* Enhanced gRPC server concurrency with errgroup

Refactored the gRPC server's `Process` method to handle incoming messages and service execution concurrently using an `errgroup.Group`. Added a context to the `grpcServer` struct and propagated it to support graceful shutdowns and better request handling. Changed the `Service` interface's `Run` method to use a channel for transmitting `ComputationRunReq`, facilitating non-blocking operation and streamlining the request-response cycle.

Asynchronous handling of stream requests with `errgroup` allows simultaneous processing and error management, improving the server's responsiveness and fault tolerance. This update paves the way for more robust service implementations that require concurrent operations.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove trailing space

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-08 10:05:41 +01:00
Sammy Kerata Oina e2fb7ea88d NOISSUE - Add test server (#80)
* add test server

Signed-off-by: SammyOina <sammyoina@gmail.com>

* clean up and update docs

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update docs

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-07 16:16:27 +01:00
Sammy Kerata Oina 6d0100c096 COCOS-73 - Remove server (#76)
* Refactor GRPC manager service and client

The manager service and client have been restructured for stream communication, facilitating real-time agent events, logs, and run responses. The `Run` RPC is replaced by the `Process` stream RPC, enabling bidirectional streaming between clients and the manager service. This allows continuous interchange of different message types including `WhoAmIRequest`, `AgentLog`, `AgentEvent`, and `RunResponse`.

Several message types have been adjusted and new fields introduced, like `AgentPort` in `RunResponse` and various agent-config attributes including CA files and instance IDs, to support TLS client authentication and distinguish between agent instances.

We've also incorporated `google.protobuf.Timestamp` in `AgentEvent` for precise event logging. The client code reflects these modifications with updated method calls and stream handling logic for ongoing communication. Moreover, the updates necessitate corresponding changes throughout service, grpc, and sdk layers to interoperate with the new streaming approach.

The transition to streaming paves the way for a more interactive, flexible communication system that can accommodate future expansion and real-time monitoring features.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* add computation id to run response

Signed-off-by: SammyOina <sammyoina@gmail.com>

* rename request

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-07 15:32:35 +01:00
Sammy Kerata Oina 8975e28437 COCOS-77 - Stream agent logs and events over gRPC (#78)
* Refactor GRPC manager service and client

The manager service and client have been restructured for stream communication, facilitating real-time agent events, logs, and run responses. The `Run` RPC is replaced by the `Process` stream RPC, enabling bidirectional streaming between clients and the manager service. This allows continuous interchange of different message types including `WhoAmIRequest`, `AgentLog`, `AgentEvent`, and `RunResponse`.

Several message types have been adjusted and new fields introduced, like `AgentPort` in `RunResponse` and various agent-config attributes including CA files and instance IDs, to support TLS client authentication and distinguish between agent instances.

We've also incorporated `google.protobuf.Timestamp` in `AgentEvent` for precise event logging. The client code reflects these modifications with updated method calls and stream handling logic for ongoing communication. Moreover, the updates necessitate corresponding changes throughout service, grpc, and sdk layers to interoperate with the new streaming approach.

The transition to streaming paves the way for a more interactive, flexible communication system that can accommodate future expansion and real-time monitoring features.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update GitHub Actions to Latest Versions

Upgraded GitHub Actions 'checkout' to version 4 and 'setup-go' to version 5 across various workflow files to leverage the latest features and improvements for better performance and reliability. This also ensures compatibility with Go version 1.21.x which is specified in the workflows.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor event handling and logging

Reworked event and log processing to use channels instead of direct HTTP calls. Removed obsolete events package and consolidated event structures, leading to cleaner and more maintainable code. Updated agent events to use channels, enhanced error handling in log forwarding, and simplified manager `New` function signature to accept an event channel directly.

- Removed `events` and `agentevents` packages to reduce complexity.
- Replaced direct event server communication with internal channel usage.
- Introduced `AgentEvent` struct in events.go for standardized event objects.
- Adapted `managerService` to dispatch events and logs through channels.
- Streamlined manager construction by removing the now-unnecessary event service and host IP parameters.

This change results in a more robust and easier to extend event and log management system within the agent-manager interaction.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix ci

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove unused code

Signed-off-by: SammyOina <sammyoina@gmail.com>

* add comments

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-07 12:08:39 +01:00
Sammy Kerata Oina 3e2be03047 NOISSUE - Attest validation and verification (#74)
* Expand CLI with Manager Service and Enhanced Attestation Commands

The CLI has been updated to interact not only with the Agent service but also with the Manager service, reflecting an expanded scope of operations. As part of this update, 'manager' subcommands have now been incorporated, ensuring users can seamlessly initiate computations through the manager service using the CLI.

Furthermore, attestation functionality has greatly improved. In addition to retrieving attestations from the agent, users can now validate and verify them directly from the CLI, bolstering security measures. New subcommands under 'attestation' provide clear and specific actions for attestation management.

Additionally, the command structure has been refactored to categorize commands under 'agent' or 'manager' prefixes, promoting better command organization and a more intuitive user experience.

These enhancements are part of ongoing efforts to provide robust and secure tools for service interaction and computational task management in distributed networks.

Relevant documentation adjustments and example usage have been updated in README files to align with these functional improvements.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* capitalize comments

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-07 12:02:41 +01:00
Sammy Kerata Oina 046b549079 Remove HTTP server support and streamline config (#75)
The HTTP server-related code, documentation, and configurations have been removed as part of a shift towards prioritizing gRPC for service communication. This update includes deletions of HTTP host and port configs across various components, the manager HTTP API alongside its Swagger definition, and the removal of related scaffolding and utility code. This change simplifies the overall architecture and eliminates redundant HTTP support, focusing on optimizing gRPC performance and security features.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-07 11:03:46 +01:00
Sammy Kerata Oina 6f42135ec0 NOISSUE - Fetch attestation report (#66)
* Improve attestation and update deps

Enhanced the attestation process to require additional report data and a variable privacy level, catering to varying security requirements. The `Run` method's context parameter was removed to simplify invocation following API alignment. Dependencies got updated, notably adding `github.com/google/go-sev-guest` for attestation, and some were indirectly bumped. These updates ensure the agent service leverages the latest available features and improvements for security and stability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Extend AttestationRequest with data fields

Enhanced the AttestationRequest message structure by introducing 'report_data' and 'vpml' fields to encapsulate additional attestation context. Adjusted service and middleware layers to incorporate these new parameters, enabling richer attestation workflows. The GRPC endpoint, logging, and metrics layers have been updated to handle the additional data, catering to more secure and nuanced attestation use cases.

Refactors Attestation command in CLI to accept report data argument and include an optional 'vpml' flag for specifying virtual machine privilege level. This change empowers users to request attestation with specific context directly from the CLI.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor attestation to remove vpml dependency

Removed the `vpml` field from the AttestationRequest protobuf, and all associated handling code, streamlining the attestation process. Code adjustments were made in gRPC client, server, and service layers to ensure proper request construction and validation that focuses only on report data. This change simplifies the attestation API and its usage, reducing error risk and improving maintainability. It reflects a decision to decouple the protocol from specific vendor measurement levels, potentially due to a shift towards a more generalized attestation procedure.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance CLI usability and streamline SDK attestation

- Added a usage example to the CLI attestation command for better user guidance.
- Simplified the SDK attestation function by removing an unused parameter and directly populating the request structure with report data to align function parameters with actual usage requirements.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-06 17:55:42 +01:00
Sammy Kerata Oina e5c8243f5f COCOS-62 - Add EOS images to release (#72)
* Add build and release GitHub workflow for tagged commits

Introduced a new GitHub Actions workflow to automate building and releasing for tagged commits. The workflow checks out the required repositories, builds the project using Buildroot configurations, and creates a release with the resultant kernel and rootfs artifacts. This streamlines the release process, ensuring consistent and reproducible builds.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update GitHub Actions to Latest Versions

Upgraded GitHub Actions in the workflow to latest or more recent versions to leverage improvements and maintain compatibility. This includes using version 4 for checkout steps and switching to 'latest' for release creation and asset upload actions, ensuring the use of up-to-date features and potential bug fixes.

Ref: Internal upgrade protocol for CI/CD consistency.
Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix workflow

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Optimized checkout step in CI workflow

Ensured the checked-out repository's reference matches the current workflow's triggering reference. This alignment can result in more predictable builds, especially when the workflow is triggered by pushes to branches other than the default or by tagged commits.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* test repo

Signed-off-by: SammyOina <sammyoina@gmail.com>

* test modified module

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update and upgrade Ubuntu in GitHub actions workflow

Ensure the GitHub actions runner has the latest Ubuntu updates before checking out the repository. This change significantly reduces the chances of encountering bugs or security vulnerabilities stemming from outdated packages, creating a more reliable and secure CI/CD environment.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* restore repo

Signed-off-by: SammyOina <sammyoina@gmail.com>

* install golang

Signed-off-by: SammyOina <sammyoina@gmail.com>

* create tags only for release

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-31 00:08:01 +01:00
Sammy Kerata Oina 8be6129889 NOISSUE - Fix bug on receiving logs and events, update deprecated methods (#68)
* Ensure graceful shutdown and improve connection handling

Refined the network connection handling in both agent events and logs to continuously process incoming data in a loop, enabling the services to handle more than a single message per connection. Additionally, instituted a deferred close operation for the event service to guarantee resources are cleanly released upon the application's termination.

Resolves potential resource leakage and enhances log processing efficiency.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Improve robustness in event and log handling

Altered handling in agent event and log services to continue processing incoming data rather than returning early upon encountering errors. This ensures that a single erroneous data point does not prematurely halt the processing loop, improving the robustness and reliability of the services. Potential errors are now reported and logged, yet the system remains operational to handle subsequent data.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Upgrade gRPC telemetry to use StatsHandler

Switched from using gRPC's UnaryInterceptor for telemetry to the more comprehensive StatsHandler provided by otelgrpc. This enhances telemetry collection by allowing the capture of a wider range of RPC stats, leading to improved monitoring and observability of the gRPC server.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* use constants

Signed-off-by: sammy <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
Signed-off-by: sammy <sammyoina@gmail.com>
2024-01-25 15:30:27 +01:00
Sammy Kerata Oina f2567830b3 NOISSUE - Remove docker (#67)
* Remove Docker-related build functionality

Removed the Docker build scripts, Dockerfile, and docker-compose definitions from the Makefile and relevant directories. Updated documentation by stripping out references to building and running Docker containers. This change likely reflects a shift towards a different deployment strategy or a move away from Docker as a dependency for builds and runtime. This simplification could lead to less complexity in the build process and reduce maintenance overhead associated with Docker configurations.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Removed Docker image build and publish workflow

The workflow for building and publishing a Docker image on pushes to the main branch has been removed. This step likely reflects a change in deployment strategy or a move to a different CI/CD pipeline. Continuous deployment might now be managed by another service or process, adhering to updated operational requirements or infrastructure changes.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-25 15:29:16 +01:00
Sammy Kerata Oina 55c19ca6b5 NOISSUE - Refactor SDK and CLI (#69)
* Refactor CLI commands to use CLI struct

Introduced a CLI struct to encapsulate dependencies such as agentSDK and managerSDK previously passed to command constructors. This shift towards a more object-oriented approach simplifies command creation and enhances code readability. CLI-related commands are now methods on the CLI struct, accessing shared services directly, which streamlines the setup of CLI command hierarchies and reduces the need to pass dependencies around.

Removed the OpenAPI specification file as it was likely deemed obsolete or no longer necessary to be included with the CLI binary, suggesting a possible shift in how the API is managed or documented.

Lastly, the main application now sets up two distinct command groups for 'agent' and 'manager', with respective commands organized under them, making the CLI tool's structure more intuitive for users.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix ci

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-25 15:07:33 +01:00
Sammy Kerata Oina 0c77167be1 COCOS-56 - Pass agent computation events to manager (#63)
* Simplify event handling and config

Streamlined event service interface by consolidating `SendEvent` and introducing `SendRaw`. Removed `notification_server_url` and `instance_id` parameters from several event publication calls to leverage centralized event construction. This change not only cleans up redundancy in event-related code but also simplifies the configuration data flow across the system, making it easier to manage and less error-prone. Uniform event generation now improves consistency and maintainability.

Refactored configuration management in the agent and manager services. Removed notifications URL from the agent configuration, relying on a simplification that assumes a single source of events. Updated Manager Port to VsockConfigPort for clarity and consistency across vsock communication.

These modifications should facilitate easier integration and extension of event and configuration systems in the future.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor error handling in agent event forwarding

Introduced context and error channel handling to the agent event forwarding process. The logger now warns on errors during forward operations asynchronously, allowing for non-blocking error reporting. Additionally, reliance on the global logger was removed in favor of passing error information via channels, improving modularity and error flow control.

Resolves issue with silent forwarding failures by providing a means to alert system operators without halting the service. This enhancement makes the error reporting more robust and reactive while maintaining service continuity.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove unused field

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update package name

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-22 16:36:34 +01:00
Sammy Kerata Oina 34c3bbdbd8 COCOS-55 - Pass agent logs to manager via vsock (#64)
* Simplify event handling and config

Streamlined event service interface by consolidating `SendEvent` and introducing `SendRaw`. Removed `notification_server_url` and `instance_id` parameters from several event publication calls to leverage centralized event construction. This change not only cleans up redundancy in event-related code but also simplifies the configuration data flow across the system, making it easier to manage and less error-prone. Uniform event generation now improves consistency and maintainability.

Refactored configuration management in the agent and manager services. Removed notifications URL from the agent configuration, relying on a simplification that assumes a single source of events. Updated Manager Port to VsockConfigPort for clarity and consistency across vsock communication.

These modifications should facilitate easier integration and extension of event and configuration systems in the future.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor error handling in agent event forwarding

Introduced context and error channel handling to the agent event forwarding process. The logger now warns on errors during forward operations asynchronously, allowing for non-blocking error reporting. Additionally, reliance on the global logger was removed in favor of passing error information via channels, improving modularity and error flow control.

Resolves issue with silent forwarding failures by providing a means to alert system operators without halting the service. This enhancement makes the error reporting more robust and reactive while maintaining service continuity.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove unused field

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance agent logging via vsock connection

Redirected agent logging to use a vsock connection instead of standard output, improving the process isolation and enabling centralized log management. The change involved dialing to the specified vsock log port and initializing the logger with the vsock connection rather than stdout.

Additionally, the manager service now maintains a map of agent vsock cids to computation IDs, providing better tracking of computation resources. A routine to retrieve logs from agents was also initiated during the service setup to facilitate log collection.

Consequential to these changes is the removal of a redundant os package import in the agent's main.go, further cleaning up the dependencies.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fail gracefully

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Updated backoff strategy and VM configurations

- Added `github.com/cenkalti/backoff` to direct dependencies for robust retry logic in agent configuration sending.
- Modified the vsock logs port to align with the updated port range standards.
- Enclosed kernel console arguments in quotes to ensure proper parsing in QEMU configurations.
- Implemented exponential backoff when sending agent configurations to handle transient failures.

Refactors:
- Streamlined creation of `AgentConfig` within the computation setup to avoid unnecessary initializations when `c.AgentConfig` is not nil.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor command execution and improve argument construction

Consolidated the error handling in the command execution function for better readability. In the QEMU configuration, the argument assembly process is enhanced for clarity and correctness; the VNC parameter is now separate, and string quoting is handled properly for kernel parameters. These changes result in more maintainable code and prevent potential formatting issues during QEMU argument parsing.

Resolves issues with argument construction in QEMU config module.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refine default config handling and unpacking

Improved the agent configuration by dynamically setting default values for the log level and port if they are not specified in the incoming configuration. Also streamlined configuration unpacking in the endpoint and service layers, reducing redundancy and ensuring all required fields are correctly copied over to the Manager's configuration structure. This change ensures better fault tolerance and more maintainable code by handling edge cases where configuration values might be missing.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* rename dir

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Ensure runRes.Empty() reflects non-empty state

Changed the always-true return value of the `runRes.Empty()` method to `false` to accurately indicate the presence of a response body. This adjustment ensures downstream handling of API responses aligns with actual content state.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Replace mglog with slog across codebase

Updated various components to replace the `mglog` logger implementation with the `slog` logger. This change affects logging initialization and calls throughout the codebase including the agent, manager, and internal server components. Transitioning to `slog` is part of a broader shift to standardize the logging mechanism to improve maintainability and consistency.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-22 14:13:47 +01:00
Sammy Kerata Oina 55afe4c038 COCOS-49 - Pass agent configuration and computation via vsock (#57)
* Optimize QEMU launch and add V-sock support

Refactored QEMU argument construction and launching logic by removing the dependency on 'agent.Computation'. This simplification makes the VM creation process more streamlined. Additionally, introduced V-sock capabilities in the QEMU configuration to facilitate improved guest-host communication. Updated the README to include kernel module setup instructions for the new V-sock feature.

The V-sock implementation enables VMs to use a consistent communication channel that is not affected by network configuration changes, enhancing reliability and potential interoperability with host services. It's important to ensure that the necessary kernel modules are loaded as part of the setup process, as documented.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add vsock-based communication to manager

Introduced virtual socket (vsock) communication abilities in the manager package by implementing a new socket service. This includes establishing a vsock listener and stub methods for sending computation results and cleaning up resources. The addition provides the groundwork for interprocess communication between guest and host in virtualized environments.

- Integrated the `mdlayher/vsock` library for handling virtual socket operations.
- Created a new `sockService` struct to encapsulate vsock listener handling.
- Implemented `NewVsock` constructor to initialize the listener with domain value `3`.
- Added placeholder methods for future computation sending and service closing logic.

This enhancement targets scenarios where efficient VM-to-host communication is required.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove env

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor agent config and use vsock

Introduce `AgentConfig` struct to group agent-related configurations, and update `Computation` struct to include the new `AgentConfig` field. Replace command-line computation extraction with vsock-based config retrieval for robustness and decoupling. The agent configuration is now read from a vsock connection during runtime, allowing for more dynamic and flexible deployments. Adjusted the main agent application logic to support these configuration changes, and corresponding changes have been made in the manager to facilitate vsock communication.

This approach aligns with modern practices for microservices by streamlining configuration management and reducing reliance on static command-line parameters. Moreover, it enhances the scalability of the agent service by allowing configuration to be managed externally.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor agent config and remove deprecated code

Consolidated agent configuration management into a single `AgentConfig` message and pruned deprecated Protobuf `ComputationReq`, `DatasetReq`, and `AlgorithmReq` messages. Adapted corresponding manager service logic to the new configuration structure. These modifications align with updated manager API schema, facilitate clearer configuration handling, and improve maintainability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* send configuration

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Switch agent to listen mode for manager connections

Previously, the agent established a connection to the manager using a direct dial. This change shifts the setup to where the agent listens on a specified port and accepts incoming connections. It ensures that the agent properly handles incoming requests by initiating a listening socket and waiting for the manager to connect, enhancing the system's flexibility in connection management. This adjustment also includes graceful closure of the listening socket.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-19 17:24:51 +01:00
Danko Miladinovic 2bb3b5650a Add support for vSockets in guest kernel (#61)
* Add python pip to buildroot

* Use DHCP for agent

* Use systemd for naming network interfaces

* Removed eth0 dhcp setting

* Add ETH_IFACE variable

* Added explanation for ETH_IFACE name

* Further explain ETH_IFACE value

* Add guest kernel support for VSOCKETS
2024-01-18 14:41:05 +01:00
Sammy Kerata Oina b4a49fb56c COCOS-58 - Remove refernece to network.link file (#59)
* Optimize QEMU launch and add V-sock support

Refactored QEMU argument construction and launching logic by removing the dependency on 'agent.Computation'. This simplification makes the VM creation process more streamlined. Additionally, introduced V-sock capabilities in the QEMU configuration to facilitate improved guest-host communication. Updated the README to include kernel module setup instructions for the new V-sock feature.

The V-sock implementation enables VMs to use a consistent communication channel that is not affected by network configuration changes, enhancing reliability and potential interoperability with host services. It's important to ensure that the necessary kernel modules are loaded as part of the setup process, as documented.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add vsock-based communication to manager

Introduced virtual socket (vsock) communication abilities in the manager package by implementing a new socket service. This includes establishing a vsock listener and stub methods for sending computation results and cleaning up resources. The addition provides the groundwork for interprocess communication between guest and host in virtualized environments.

- Integrated the `mdlayher/vsock` library for handling virtual socket operations.
- Created a new `sockService` struct to encapsulate vsock listener handling.
- Implemented `NewVsock` constructor to initialize the listener with domain value `3`.
- Added placeholder methods for future computation sending and service closing logic.

This enhancement targets scenarios where efficient VM-to-host communication is required.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Revert "Optimize QEMU launch and add V-sock support"

This reverts commit 44557c39d9.

* Revert "Add vsock-based communication to manager"

This reverts commit d1d3c01f3b.

* Remove unnecessary network link file installation

Network configuration is being handled elsewhere, making the installation of the `00-network.link` file redundant. Eliminated the line from the agent Makefile to prevent potential conflicts and streamline the initialization process.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-17 15:14:53 +01:00
Danko Miladinovic c25cfec84c Use DHCP for VM networking (#54)
* Add python pip to buildroot

* Use DHCP for agent

* Use systemd for naming network interfaces

* Removed eth0 dhcp setting

* Add ETH_IFACE variable

* Added explanation for ETH_IFACE name

* Further explain ETH_IFACE value
2024-01-15 19:27:07 +01:00
Danko Miladinovic cca75fc5cc Add python pip to buildroot (#51) 2024-01-12 17:51:26 +01:00
142 changed files with 8780 additions and 3456 deletions
+5 -15
View File
@@ -1,28 +1,21 @@
# Docker: Environment variables in Compose
## Jaeger
COCOS_JAEGER_PORT=6831
COCOS_JAEGER_FRONTEND=16686
COCOS_JAEGER_COLLECTOR=14268
COCOS_JAEGER_CONFIGS=5778
COCOS_JAEGER_URL=http://jaeger:14268/api/traces
COCOS_JAEGER_URL=http://jaeger:4318
COCOS_JAEGER_TRACE_RATIO=1.0
## Notification Server
COCOS_NOTIFICATION_SERVER_URL="http://localhost:9000"
COCOS_JAEGER_COLLECTOR_OTLP_ENABLED=true
COCOS_JAEGER_OLTP_HTTP_PORT=4318
## Core Services
### Manager
MANAGER_HTTP_HOST="cocos-manager"
MANAGER_HTTP_PORT=9021
MANAGER_HTTP_SERVER_CERT=""
MANAGER_HTTP_SERVER_KEY=""
MANAGER_GRPC_HOST="cocos-manager"
MANAGER_GRPC_HOST=""
MANAGER_GRPC_PORT=7003
MANAGER_GRPC_SERVER_CERT=""
MANAGER_GRPC_SERVER_KEY=""
AGENT_GRPC_URL="192.168.100.4:7002"
AGENT_GRPC_URL="localhost:7002"
AGENT_GRPC_TIMEOUT=""
AGENT_GRPC_CA_CERTS=""
AGENT_GRPC_CLIENT_TLS=""
@@ -33,6 +26,3 @@ MANAGER_QEMU_ENABLE_SEV=false
MANAGER_QEMU_SEV_CBITPOS=51
MANAGER_QEMU_OVMF_CODE_FILE=/usr/share/OVMF/OVMF_CODE.fd
MANAGER_QEMU_OVMF_VARS_FILE=/usr/share/OVMF/OVMF_VARS.fd
# Docker image tag
COCOS_RELEASE_TAG=latest
+49
View File
@@ -0,0 +1,49 @@
name: Bug Report
description: File a bug/issue report. Make sure to search to see if an issue already exists for the bug you encountered.
title: "Bug: <title>"
labels: ["bug", "needs-review", "help wanted"]
body:
- type: textarea
attributes:
label: What were you trying to achieve?
description: A clear and concise description of what the bug is.
validations:
required: true
- type: textarea
attributes:
label: What are the expected results?
description: A concise description of what you expected to happen.
validations:
required: true
- type: textarea
attributes:
label: What are the received results?
description: A concise description of what you received.
validations:
required: true
- type: textarea
attributes:
label: Steps To Reproduce
description: What are the steps to reproduce the issue?
placeholder: |
1. In this environment...
2. With this config...
3. Run '...'
4. See error...
validations:
required: false
- type: textarea
attributes:
label: In what environment did you encounter the issue?
description: A concise description of the environment you encountered the issue in.
validations:
required: true
- type: textarea
attributes:
label: Additional information you deem important
description: |
Links? References? Anything that will give us more context about the issue you are encountering!
Tip: You can attach images or log files by clicking this area to highlight it and then dragging files in.
validations:
required: false
+1
View File
@@ -0,0 +1 @@
blank_issues_enabled: false
@@ -0,0 +1,36 @@
name: Feature Request
description: File a feature request. Make sure to search to see if a request already exists for the feature you are requesting.
title: "Feature: <title>"
labels: ["enchancement", "needs-review"]
body:
- type: textarea
attributes:
label: Is your feature request related to a problem? Please describe.
description: A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
validations:
required: true
- type: textarea
attributes:
label: Describe the feature you are requesting, as well as the possible use case(s) for it.
description: A clear and concise description of what you want to happen.
validations:
required: true
- type: dropdown
attributes:
label: Indicate the importance of this feature to you.
description: This will help us prioritize the feature request.
options:
- Must-have
- Should-have
- Nice-to-have
validations:
required: true
- type: textarea
attributes:
label: Anything else?
description: |
Links? References? Anything that will give us more context about the feature that you are requesting.
Tip: You can attach images or log files by clicking this area to highlight it and then dragging files in.
validations:
required: false
+66
View File
@@ -0,0 +1,66 @@
<!--
Pull request title should be `COCOS-XXX - description` or `NOISSUE - description` where XXX is ID of the issue that this PR relate to.
Please review the [CONTRIBUTING.md](https://github.com/ultravioletrs/cocos/blob/master/CONTRIBUTING.md) file for detailed contributing guidelines.
For Work In Progress Pull Requests, please use the Draft PR feature, see https://github.blog/2019-02-14-introducing-draft-pull-requests/ for further details.
For a timely review/response, please avoid force-pushing additional commits if your PR already received reviews or comments.
- Provide tests for your changes.
- Use descriptive commit messages.
- Comment your code where appropriate.
- Squash your commits
- Update any related documentation.
-->
# What type of PR is this?
<!--This represents the type of PR you are submitting.
For example:
This is a bug fix because it fixes the following issue: #1234
This is a feature because it adds the following functionality: ...
This is a refactor because it changes the following functionality: ...
This is a documentation update because it updates the following documentation: ...
This is a dependency update because it updates the following dependencies: ...
This is an optimization because it improves the following functionality: ...
-->
## What does this do?
<!--
Please provide a brief description of what this PR is intended to do.
Include List any changes that modify/break current functionality.
-->
## Which issue(s) does this PR fix/relate to?
<!--
For pull requests that relate or close an issue, please include them below. We like to follow [Github's guidance on linking issues to pull requests](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue).
For example having the text: "Resolves #1234" would connect the current pull request to issue 1234. And when we merge the pull request, Github will automatically close the issue.
-->
- Related Issue #
- Resolves #
## Have you included tests for your changes?
<!--If you have not included tests, please explain why.
For example:
Yes, I have included tests for my changes.
No, I have not included tests because I do not know how to.
-->
## Did you document any new/modified feature?
<!--If you have not included documentation, please explain why.
For example:
Yes, I have updated the documentation for the new feature.
No, I have not updated the documentation because I do not know how to.
-->
### Notes
<!--Please provide any additional information you feel is important.-->
-41
View File
@@ -1,41 +0,0 @@
# Copyright (c) Ultraviolet
# SPDX-License-Identifier: Apache-2.0
name: Create and publish a Docker image
on:
push:
branches: ["main"]
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
build-and-push-image:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Log in to the Container registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build Manager and push Docker image
uses: docker/build-push-action@v5
with:
context: .
file: ./docker/Dockerfile
push: true
build-args: |
SVC=manager
tags: ghcr.io/ultravioletrs/cocos/manager:latest
+10 -10
View File
@@ -6,7 +6,7 @@ on:
- main
paths:
- "manager/manager.proto"
- "manager/*.pb.go"
- "pkg/manager/*.pb.go"
- "agent/agent.proto"
- "agent/*.pb.go"
pull_request:
@@ -14,7 +14,7 @@ on:
- main
paths:
- "manager/manager.proto"
- "manager/*.pb.go"
- "pkg/manager/*.pb.go"
- "agent/agent.proto"
- "agent/*.pb.go"
@@ -24,18 +24,18 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v2
uses: actions/checkout@v4
- name: Install Go
uses: actions/setup-go@v3
uses: actions/setup-go@v5
with:
go-version: 1.21.x
go-version: 1.22.x
- name: Set up protoc
run: |
PROTOC_VERSION=25.1
PROTOC_GEN_VERSION=v1.31.0
PROTOC_GRPC_VERSION=v1.3.0
PROTOC_VERSION=25.3
PROTOC_GEN_VERSION=v1.34.2
PROTOC_GRPC_VERSION=v1.4.0
# Download and install protoc
PROTOC_ZIP=protoc-$PROTOC_VERSION-linux-x86_64.zip
@@ -55,7 +55,7 @@ jobs:
- name: Set up Cocos-AI
run: |
# Rename .pb.go files to .pb.go.tmp to prevent conflicts
for p in $(ls manager/*.pb.go); do
for p in $(ls pkg/manager/*.pb.go); do
mv $p $p.tmp
done
@@ -67,7 +67,7 @@ jobs:
make protoc
# Compare generated Go files with the original ones
for p in $(ls manager/*.pb.go); do
for p in $(ls pkg/manager/*.pb.go); do
if ! cmp -s $p $p.tmp; then
echo "Proto file and generated Go file $p are out of sync!"
exit 1
+73
View File
@@ -0,0 +1,73 @@
name: Build and Release
on:
push:
tags:
- '*'
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Update Ubuntu
run: |
sudo apt-get update
sudo apt-get upgrade -y
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: 1.22.x
cache-dependency-path: "go.sum"
- name: Checkout cocos
uses: actions/checkout@v4
with:
repository: 'ultravioletrs/cocos'
path: cocos
- name: Checkout buildroot
uses: actions/checkout@v4
with:
repository: 'buildroot/buildroot'
path: buildroot
- name: Build
run: |
cd buildroot
make BR2_EXTERNAL=../cocos/hal/linux cocos_defconfig
make
- name: Create Release
id: create_release
uses: actions/create-release@latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
tag_name: ${{ github.ref }}
release_name: Release ${{ github.ref }}
draft: false
prerelease: false
- name: Upload Release Asset
id: upload-release-kernel
uses: actions/upload-release-asset@latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: ./buildroot/output/images/bzImage
asset_name: bzImage
asset_content_type: application/octet-stream
- name: Upload Release Asset
id: upload-release-rootfs
uses: actions/upload-release-asset@latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: ./buildroot/output/images/rootfs.cpio.gz
asset_name: rootfs.cpio.gz
asset_content_type: application/gzip
+5 -5
View File
@@ -14,17 +14,17 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v2
uses: actions/checkout@v4
- name: Install Go
uses: actions/setup-go@v3
uses: actions/setup-go@v5
with:
go-version: 1.21.x
go-version: 1.22.x
- name: golangci-lint
uses: golangci/golangci-lint-action@v3
uses: golangci/golangci-lint-action@v6
with:
version: latest
version: v1.59.1
- name: Build
run: |
+6
View File
@@ -6,3 +6,9 @@ cmd/manager/iso
cmd/manager/tmp
.cov
*.pem
dist/
result.bin
*.spec
+2 -8
View File
@@ -1,6 +1,6 @@
run:
timeout: 3m
issues:
max-issues-per-linter: 10
max-same-issues: 10
@@ -15,10 +15,7 @@ linters-settings:
gocritic:
enabled-checks:
- captLocal
- dupImport
- singleCaseSwitch
- switchTrue
- importShadow
- httpNoBody
- paramTypeCombine
@@ -34,9 +31,6 @@ linters-settings:
- style
- experimental
- opinionated
misspell:
ignore-words:
- "mosquitto"
stylecheck:
checks: ["-ST1000", "-ST1003", "-ST1020", "-ST1021", "-ST1022"]
goheader:
@@ -83,4 +77,4 @@ linters:
- makezero
- mirror
- nakedret
- dupword
- dupword
+87
View File
@@ -0,0 +1,87 @@
# Contributing to CoCos
The following is a set of guidelines to contribute to CoCos and its libraries, which are
hosted on the [Ultraviolet Organization](https://github.com/ultravioletrs) on GitHub.
This project adheres to the [Contributor Covenant 1.2](http://contributor-covenant.org/version/1/2/0).
By participating, you are expected to uphold this code. Please report unacceptable behavior to
[abuse@ultraviolet.rs](mailto:abuse@ultraviolet.rs).
## Reporting issues
Reporting issues are a great way to contribute to the project. We are perpetually grateful about a well-written,
thorough bug report.
Before raising a new issue, check [our issue
list](https://github.com/ultravioletrs/cocos/issues) to determine if it already contains the
problem that you are facing.
A good bug report shouldn't leave others needing to chase you for more information. Please be as detailed as possible. The following questions might serve as a template for writing a detailed
report:
- What were you trying to achieve?
- What are the expected results?
- What are the received results?
- What are the steps to reproduce the issue?
- In what environment did you encounter the issue?
## Pull requests
Good pull requests (e.g. patches, improvements, new features) are a fantastic help. They should
remain focused in scope and avoid unrelated commits.
**Please ask first** before embarking on any significant pull request (e.g. implementing new features,
refactoring code etc.), otherwise you risk spending a lot of time working on something that the
maintainers might not want to merge into the project.
Please adhere to the coding conventions used throughout the project. If in doubt, consult the
[Effective Go](https://golang.org/doc/effective_go.html) style guide.
To contribute to the project, [fork](https://help.github.com/articles/fork-a-repo/) it,
clone your fork repository, and configure the remotes:
```
git clone https://github.com/<your-username>/cocos.git
cd cocos
git remote add upstream https://github.com/ultravioletrs/cocos.git
```
If your cloned repository is behind the upstream commits, then get the latest changes from upstream:
```
git checkout master
git pull --rebase upstream main
```
Create a new topic branch from `master` using the naming convention `COCOS-[issue-number]`
to help us keep track of your contribution scope:
```
git checkout -b COCOS-[issue-number]
```
Commit your changes in logical chunks. When you are ready to commit, make sure
to write a Good Commit Message™. Consult the [Erlang's contributing guide](https://github.com/erlang/otp/wiki/Writing-good-commit-messages)
if you're unsure of what constitutes a Good Commit Message™. Use [interactive rebase](https://help.github.com/articles/about-git-rebase)
to group your commits into logical units of work before making it public.
Note that every commit you make must be signed. By signing off your work you indicate that you
are accepting the [Developer Certificate of Origin](https://developercertificate.org/).
Use your real name (sorry, no pseudonyms or anonymous contributions). If you set your `user.name`
and `user.email` git configs, you can sign your commit automatically with `git commit -s`.
Locally merge (or rebase) the upstream development branch into your topic branch:
```
git pull --rebase upstream main
```
Push your topic branch up to your fork:
```
git push origin COCOS-[issue-number]
```
[Open a Pull Request](https://help.github.com/articles/using-pull-requests/) with a clear title
and detailed description.
View File
+7 -50
View File
@@ -1,19 +1,11 @@
BUILD_DIR = build
SERVICES = manager agent cli
PLATFORM_INFO = platform_info
CGO_ENABLED ?= 0
GOARCH ?= amd64
VERSION ?= $(shell git describe --abbrev=0 --tags --always)
COMMIT ?= $(shell git rev-parse HEAD)
TIME ?= $(shell date +%F_%T)
CLI_SOURCE = ./cmd/cli/main.go
CLI_BIN = ${BUILD_DIR}/cocos-cli
DOCKERS = $(addprefix docker_,$(SERVICES))
DOCKERS_DEV = $(addprefix docker_dev_,$(SERVICES))
COCOS_DOCKER_IMAGE_NAME_PREFIX=ghcr.io/ultravioletrs/cocos/
USER_REPO ?= $(shell git remote get-url origin | sed -e 's/.*\/\([^/]*\)\/\([^/]*\).*/\1_\2/' )
empty:=
space:= $(empty) $(empty)
define compile_service
CGO_ENABLED=$(CGO_ENABLED) GOOS=$(GOOS) GOARCH=$(GOARCH) GOARM=$(GOARM) \
@@ -24,54 +16,19 @@ define compile_service
-o ${BUILD_DIR}/cocos-$(1) cmd/$(1)/main.go
endef
define make_docker
$(eval svc=$(subst docker_,,$(1)))
docker build \
--no-cache \
--build-arg SVC=$(svc) \
--build-arg GOARCH=$(GOARCH) \
--build-arg GOARM=$(GOARM) \
--build-arg VERSION=$(VERSION) \
--build-arg COMMIT=$(COMMIT) \
--build-arg TIME=$(TIME) \
--tag=$(COCOS_DOCKER_IMAGE_NAME_PREFIX)$(svc) \
-f docker/Dockerfile .
endef
define make_docker_dev
$(eval svc=$(subst docker_dev_,,$(1)))
docker build \
--no-cache \
--build-arg SVC=$(svc) \
--tag=$(COCOS_DOCKER_IMAGE_NAME_PREFIX)$(svc) \
-f docker/Dockerfile.dev ./build
endef
.PHONY: all $(SERVICES) dockers dockers_dev
.PHONY: all $(SERVICES) $(PLATFORM_INFO)
all: $(SERVICES)
$(DOCKERS):
$(call make_docker,$(@),$(GOARCH))
$(DOCKERS_DEV):
$(call make_docker_dev,$(@))
dockers: $(DOCKERS)
dockers_dev: $(DOCKERS_DEV)
$(SERVICES):
$(call compile_service,$(@))
install-cli: cli
cp ${CLI_BIN} ~/.local/bin/cocos-cli
$(PLATFORM_INFO):
$(MAKE) -C ./scripts/platform_info
protoc:
protoc -I. --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative agent/agent.proto
protoc -I. --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative manager/manager.proto
protoc -I. --go_out=./pkg --go_opt=paths=source_relative --go-grpc_out=./pkg --go-grpc_opt=paths=source_relative manager/manager.proto
run:
docker compose -f docker/docker-compose.yml -p cocos up
mocks:
go generate ./...
+30 -34
View File
@@ -7,59 +7,55 @@ based on the [Confidential Computing][cc] and [Trusted Execution Environments (T
</p>
With Cocos AI it becomes possible to run AI/ML workloads on combined datasets from multiple organizations
while guaranteeing the privacy and security of the data and the algorithms.
while guaranteeing the privacy and security of the data and the algorithm.
Data is always encrypted, protected by hardware secure enclaves (Trusted Execution Environments),
attested via secure remote attestation protocols, and invisible to cloud processors or any other
3rd party to which computation is offloaded.
## Install
The following prerequisites are needed to run Cocos:
## Features
- [Docker](https://docs.docker.com/install/)
- [Docker compose](https://docs.docker.com/compose/install/)
Cocos AI is implementing the following features:
### Build Docker Images
Currenty, there is no Docker registry, so Cocos images must be built by hand:
```
make dockers
```
For this you might be needing to setup DNS servers in your `/etc/resolf.conf` as explained
[here](https://github.com/docker/cli/issues/2618) (i.e. add Google's `nameserver 8.8.8.8`).
### Run Composition
Once the images are built (`docker images` command should show you `ghcr.io/ultravioletrs/cocos/manager`),
composition can be run:
```bash
make run
```
This will bring up the Cocos docker services and interconnect them.
- TEE enablement, deployment and monitoring (secure VM manager)
- HAL for TEEs based on hardened Linux kernel, secure bootloader and custom-tailored embedded rootfs for minimal TCB
- In-enclave agent, netowrking controller and other system software
- Encrypted asynchronous data transfer and result delivery
- API for programmable platform manipulation
- HW and SW supported attestation with verification tools
- CLI for system interaction
## Usage
The quickest way to start using Cocos is via the CLI. The latest version can be downloaded from the [official releases page][rel].
It can also be built and used from the project's root directory:
Clone the repo and create binaries:
```bash
make cli
./build/cocos-cli version
git clone git@github.com:ultravioletrs/cocos.git
make
```
Additional details on using the CLI can be found in the [CLI documentation](https://docs.cocos.ai/cli).
This will create 3 binaries:
```bash
ls build/
# cocos-agent cocos-cli cocos-manager
```
- Manager can be deployed on the AMD SEV-SNP host
- Agent can be built into [EOS][eos]-based HAL
- CLI can be used to communicate to remote Agent.
## Documentation
Official documentation is hosted at [Cocos official docs page][docs]. Documentation is auto-generated, checkout the instructions on [official docs repository](https://github.com/ultravioletrs/docs).
Project documentation is hosted at [Cocos AI official docs page][docs].
Documentation is generated from the [docs repository](https://github.com/ultravioletrs/docs).
## License
Cocos AI is a proprietary product created by Ultraviolet company.
Cocos AI is published under permissive open-source [Apache-2.0](LICENSE) license.
[cc]: https://confidentialcomputing.io/white-papers-reports/
[cocos]: https://cocos.ai/
[rel]: https://github.com/ultraviolet/cocos/releases
[rel]: https://github.com/ultravioletrs/cocos/releases
[tee]: https://en.wikipedia.org/wiki/Trusted_execution_environment
[docs]: https://docs.cocos.ultraviolet.rs
[cli]: https://docs.cocos.ultraviolet.rs/cli
[eos]: https://github.com/ultravioletrs/eos
+7 -10
View File
@@ -9,15 +9,12 @@ The service is configured using the environment variables from the following tab
| Variable | Description | Default |
| ----------------------------- | ------------------------------------------------------ | ------------------------------ |
| AGENT_LOG_LEVEL | Log level for agent service (debug, info, warn, error) | info |
| AGENT_HTTP_HOST | Agent service HTTP host | "" |
| AGENT_HTTP_PORT | Agent service HTTP port | 9031 |
| AGENT_HTTP_SERVER_CERT | Path to HTTP server certificate in pem format | "" |
| AGENT_HTTP_SERVER_KEY | Path to HTTP server key in pem format | "" |
| AGENT_GRPC_HOST | Agent service gRPC host | "" |
| AGENT_GRPC_PORT | Agent service gRPC port | 7002 |
| AGENT_GRPC_SERVER_CERT | Path to gRPC server certificate in pem format | "" |
| AGENT_GRPC_SERVER_KEY | Path to gRPC server key in pem format | "" |
| COCOS_NOTIFICATION_SERVER_URL | Server to receive notification events from agent. | http:/localhost:9000 |
| AGENT_GRPC_SERVER_CA_CERTS | Path to gRPC server CA certificate | "" |
| AGENT_GRPC_CLIENT_CA_CERTS | Path to gRPC client CA certificate | "" |
## Deployment
@@ -25,15 +22,15 @@ The service is configured using the environment variables from the following tab
To start the service outside of the container, execute the following shell script:
```bash
# download the latest version of the service
go get github.com/ultravioletrs/cocos
# Download the latest version of the service
git clone git@github.com:ultravioletrs/cocos.git
cd $GOPATH/src/github.com/ultravioletrs/cocos
cd cocos
# compile the agent
# Compile the service
make agent
# set the environment variables and run the service
# Run the service
./build/cocos-agent
```
+74 -383
View File
@@ -3,8 +3,8 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.31.0
// protoc v4.25.1
// protoc-gen-go v1.34.2
// protoc v4.25.3
// source: agent/agent.proto
package agent
@@ -28,9 +28,8 @@ type AlgoRequest struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Algorithm []byte `protobuf:"bytes,1,opt,name=algorithm,proto3" json:"algorithm,omitempty"`
Provider string `protobuf:"bytes,2,opt,name=provider,proto3" json:"provider,omitempty"`
Id string `protobuf:"bytes,3,opt,name=id,proto3" json:"id,omitempty"`
Algorithm []byte `protobuf:"bytes,1,opt,name=algorithm,proto3" json:"algorithm,omitempty"`
Requirements []byte `protobuf:"bytes,2,opt,name=requirements,proto3" json:"requirements,omitempty"`
}
func (x *AlgoRequest) Reset() {
@@ -72,26 +71,17 @@ func (x *AlgoRequest) GetAlgorithm() []byte {
return nil
}
func (x *AlgoRequest) GetProvider() string {
func (x *AlgoRequest) GetRequirements() []byte {
if x != nil {
return x.Provider
return x.Requirements
}
return ""
}
func (x *AlgoRequest) GetId() string {
if x != nil {
return x.Id
}
return ""
return nil
}
type AlgoResponse struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
AlgorithmID string `protobuf:"bytes,1,opt,name=algorithmID,proto3" json:"algorithmID,omitempty"`
}
func (x *AlgoResponse) Reset() {
@@ -126,21 +116,12 @@ func (*AlgoResponse) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{1}
}
func (x *AlgoResponse) GetAlgorithmID() string {
if x != nil {
return x.AlgorithmID
}
return ""
}
type DataRequest struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Dataset []byte `protobuf:"bytes,1,opt,name=dataset,proto3" json:"dataset,omitempty"`
Provider string `protobuf:"bytes,2,opt,name=provider,proto3" json:"provider,omitempty"`
Id string `protobuf:"bytes,3,opt,name=id,proto3" json:"id,omitempty"`
Dataset []byte `protobuf:"bytes,1,opt,name=dataset,proto3" json:"dataset,omitempty"`
}
func (x *DataRequest) Reset() {
@@ -182,26 +163,10 @@ func (x *DataRequest) GetDataset() []byte {
return nil
}
func (x *DataRequest) GetProvider() string {
if x != nil {
return x.Provider
}
return ""
}
func (x *DataRequest) GetId() string {
if x != nil {
return x.Id
}
return ""
}
type DataResponse struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
DatasetID string `protobuf:"bytes,1,opt,name=datasetID,proto3" json:"datasetID,omitempty"`
}
func (x *DataResponse) Reset() {
@@ -236,19 +201,10 @@ func (*DataResponse) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{3}
}
func (x *DataResponse) GetDatasetID() string {
if x != nil {
return x.DatasetID
}
return ""
}
type ResultRequest struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Consumer string `protobuf:"bytes,1,opt,name=consumer,proto3" json:"consumer,omitempty"`
}
func (x *ResultRequest) Reset() {
@@ -283,13 +239,6 @@ func (*ResultRequest) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{4}
}
func (x *ResultRequest) GetConsumer() string {
if x != nil {
return x.Consumer
}
return ""
}
type ResultResponse struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
@@ -341,6 +290,8 @@ type AttestationRequest struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
ReportData []byte `protobuf:"bytes,1,opt,name=report_data,json=reportData,proto3" json:"report_data,omitempty"` // Should be of length 64.
}
func (x *AttestationRequest) Reset() {
@@ -375,6 +326,13 @@ func (*AttestationRequest) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{6}
}
func (x *AttestationRequest) GetReportData() []byte {
if x != nil {
return x.ReportData
}
return nil
}
type AttestationResponse struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
@@ -422,273 +380,47 @@ func (x *AttestationResponse) GetFile() []byte {
return nil
}
type ComputationReq struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"`
Description string `protobuf:"bytes,3,opt,name=description,proto3" json:"description,omitempty"`
Datasets []*DatasetReq `protobuf:"bytes,4,rep,name=datasets,proto3" json:"datasets,omitempty"`
Algorithms []*AlgorithmReq `protobuf:"bytes,5,rep,name=algorithms,proto3" json:"algorithms,omitempty"`
ResultConsumers []string `protobuf:"bytes,6,rep,name=result_consumers,json=resultConsumers,proto3" json:"result_consumers,omitempty"`
}
func (x *ComputationReq) Reset() {
*x = ComputationReq{}
if protoimpl.UnsafeEnabled {
mi := &file_agent_agent_proto_msgTypes[8]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *ComputationReq) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*ComputationReq) ProtoMessage() {}
func (x *ComputationReq) ProtoReflect() protoreflect.Message {
mi := &file_agent_agent_proto_msgTypes[8]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use ComputationReq.ProtoReflect.Descriptor instead.
func (*ComputationReq) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{8}
}
func (x *ComputationReq) GetId() string {
if x != nil {
return x.Id
}
return ""
}
func (x *ComputationReq) GetName() string {
if x != nil {
return x.Name
}
return ""
}
func (x *ComputationReq) GetDescription() string {
if x != nil {
return x.Description
}
return ""
}
func (x *ComputationReq) GetDatasets() []*DatasetReq {
if x != nil {
return x.Datasets
}
return nil
}
func (x *ComputationReq) GetAlgorithms() []*AlgorithmReq {
if x != nil {
return x.Algorithms
}
return nil
}
func (x *ComputationReq) GetResultConsumers() []string {
if x != nil {
return x.ResultConsumers
}
return nil
}
type DatasetReq struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Provider string `protobuf:"bytes,1,opt,name=provider,proto3" json:"provider,omitempty"`
Id string `protobuf:"bytes,2,opt,name=id,proto3" json:"id,omitempty"`
}
func (x *DatasetReq) Reset() {
*x = DatasetReq{}
if protoimpl.UnsafeEnabled {
mi := &file_agent_agent_proto_msgTypes[9]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *DatasetReq) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*DatasetReq) ProtoMessage() {}
func (x *DatasetReq) ProtoReflect() protoreflect.Message {
mi := &file_agent_agent_proto_msgTypes[9]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use DatasetReq.ProtoReflect.Descriptor instead.
func (*DatasetReq) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{9}
}
func (x *DatasetReq) GetProvider() string {
if x != nil {
return x.Provider
}
return ""
}
func (x *DatasetReq) GetId() string {
if x != nil {
return x.Id
}
return ""
}
type AlgorithmReq struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Provider string `protobuf:"bytes,1,opt,name=provider,proto3" json:"provider,omitempty"`
Id string `protobuf:"bytes,2,opt,name=id,proto3" json:"id,omitempty"`
}
func (x *AlgorithmReq) Reset() {
*x = AlgorithmReq{}
if protoimpl.UnsafeEnabled {
mi := &file_agent_agent_proto_msgTypes[10]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *AlgorithmReq) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*AlgorithmReq) ProtoMessage() {}
func (x *AlgorithmReq) ProtoReflect() protoreflect.Message {
mi := &file_agent_agent_proto_msgTypes[10]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use AlgorithmReq.ProtoReflect.Descriptor instead.
func (*AlgorithmReq) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{10}
}
func (x *AlgorithmReq) GetProvider() string {
if x != nil {
return x.Provider
}
return ""
}
func (x *AlgorithmReq) GetId() string {
if x != nil {
return x.Id
}
return ""
}
var File_agent_agent_proto protoreflect.FileDescriptor
var file_agent_agent_proto_rawDesc = []byte{
0x0a, 0x11, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2f, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x70, 0x72,
0x6f, 0x74, 0x6f, 0x12, 0x05, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x22, 0x57, 0x0a, 0x0b, 0x41, 0x6c,
0x6f, 0x74, 0x6f, 0x12, 0x05, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x22, 0x4f, 0x0a, 0x0b, 0x41, 0x6c,
0x67, 0x6f, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x1c, 0x0a, 0x09, 0x61, 0x6c, 0x67,
0x6f, 0x72, 0x69, 0x74, 0x68, 0x6d, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x09, 0x61, 0x6c,
0x67, 0x6f, 0x72, 0x69, 0x74, 0x68, 0x6d, 0x12, 0x1a, 0x0a, 0x08, 0x70, 0x72, 0x6f, 0x76, 0x69,
0x64, 0x65, 0x72, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x70, 0x72, 0x6f, 0x76, 0x69,
0x64, 0x65, 0x72, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52,
0x02, 0x69, 0x64, 0x22, 0x30, 0x0a, 0x0c, 0x41, 0x6c, 0x67, 0x6f, 0x52, 0x65, 0x73, 0x70, 0x6f,
0x6e, 0x73, 0x65, 0x12, 0x20, 0x0a, 0x0b, 0x61, 0x6c, 0x67, 0x6f, 0x72, 0x69, 0x74, 0x68, 0x6d,
0x49, 0x44, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0b, 0x61, 0x6c, 0x67, 0x6f, 0x72, 0x69,
0x74, 0x68, 0x6d, 0x49, 0x44, 0x22, 0x53, 0x0a, 0x0b, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71,
0x75, 0x65, 0x73, 0x74, 0x12, 0x18, 0x0a, 0x07, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x18,
0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x07, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x12, 0x1a,
0x0a, 0x08, 0x70, 0x72, 0x6f, 0x76, 0x69, 0x64, 0x65, 0x72, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09,
0x52, 0x08, 0x70, 0x72, 0x6f, 0x76, 0x69, 0x64, 0x65, 0x72, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64,
0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x22, 0x2c, 0x0a, 0x0c, 0x44, 0x61,
0x74, 0x61, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1c, 0x0a, 0x09, 0x64, 0x61,
0x74, 0x61, 0x73, 0x65, 0x74, 0x49, 0x44, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x64,
0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x49, 0x44, 0x22, 0x2b, 0x0a, 0x0d, 0x52, 0x65, 0x73, 0x75,
0x6c, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x1a, 0x0a, 0x08, 0x63, 0x6f, 0x6e,
0x73, 0x75, 0x6d, 0x65, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x63, 0x6f, 0x6e,
0x73, 0x75, 0x6d, 0x65, 0x72, 0x22, 0x24, 0x0a, 0x0e, 0x52, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x52,
0x67, 0x6f, 0x72, 0x69, 0x74, 0x68, 0x6d, 0x12, 0x22, 0x0a, 0x0c, 0x72, 0x65, 0x71, 0x75, 0x69,
0x72, 0x65, 0x6d, 0x65, 0x6e, 0x74, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x0c, 0x72,
0x65, 0x71, 0x75, 0x69, 0x72, 0x65, 0x6d, 0x65, 0x6e, 0x74, 0x73, 0x22, 0x0e, 0x0a, 0x0c, 0x41,
0x6c, 0x67, 0x6f, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x27, 0x0a, 0x0b, 0x44,
0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x18, 0x0a, 0x07, 0x64, 0x61,
0x74, 0x61, 0x73, 0x65, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x07, 0x64, 0x61, 0x74,
0x61, 0x73, 0x65, 0x74, 0x22, 0x0e, 0x0a, 0x0c, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x73, 0x70,
0x6f, 0x6e, 0x73, 0x65, 0x22, 0x0f, 0x0a, 0x0d, 0x52, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x52, 0x65,
0x71, 0x75, 0x65, 0x73, 0x74, 0x22, 0x24, 0x0a, 0x0e, 0x52, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x52,
0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x66, 0x69, 0x6c, 0x65, 0x18,
0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x04, 0x66, 0x69, 0x6c, 0x65, 0x22, 0x14, 0x0a, 0x12, 0x41,
0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x04, 0x66, 0x69, 0x6c, 0x65, 0x22, 0x35, 0x0a, 0x12, 0x41,
0x74, 0x74, 0x65, 0x73, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73,
0x74, 0x22, 0x29, 0x0a, 0x13, 0x41, 0x74, 0x74, 0x65, 0x73, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e,
0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x66, 0x69, 0x6c, 0x65,
0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x04, 0x66, 0x69, 0x6c, 0x65, 0x22, 0xe5, 0x01, 0x0a,
0x0e, 0x43, 0x6f, 0x6d, 0x70, 0x75, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x71, 0x12,
0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x12,
0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e,
0x61, 0x6d, 0x65, 0x12, 0x20, 0x0a, 0x0b, 0x64, 0x65, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, 0x69,
0x6f, 0x6e, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0b, 0x64, 0x65, 0x73, 0x63, 0x72, 0x69,
0x70, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x2d, 0x0a, 0x08, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74,
0x73, 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x11, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e,
0x44, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x52, 0x65, 0x71, 0x52, 0x08, 0x64, 0x61, 0x74, 0x61,
0x73, 0x65, 0x74, 0x73, 0x12, 0x33, 0x0a, 0x0a, 0x61, 0x6c, 0x67, 0x6f, 0x72, 0x69, 0x74, 0x68,
0x6d, 0x73, 0x18, 0x05, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x13, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74,
0x2e, 0x41, 0x6c, 0x67, 0x6f, 0x72, 0x69, 0x74, 0x68, 0x6d, 0x52, 0x65, 0x71, 0x52, 0x0a, 0x61,
0x6c, 0x67, 0x6f, 0x72, 0x69, 0x74, 0x68, 0x6d, 0x73, 0x12, 0x29, 0x0a, 0x10, 0x72, 0x65, 0x73,
0x75, 0x6c, 0x74, 0x5f, 0x63, 0x6f, 0x6e, 0x73, 0x75, 0x6d, 0x65, 0x72, 0x73, 0x18, 0x06, 0x20,
0x03, 0x28, 0x09, 0x52, 0x0f, 0x72, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x43, 0x6f, 0x6e, 0x73, 0x75,
0x6d, 0x65, 0x72, 0x73, 0x22, 0x38, 0x0a, 0x0a, 0x44, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x52,
0x65, 0x71, 0x12, 0x1a, 0x0a, 0x08, 0x70, 0x72, 0x6f, 0x76, 0x69, 0x64, 0x65, 0x72, 0x18, 0x01,
0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x70, 0x72, 0x6f, 0x76, 0x69, 0x64, 0x65, 0x72, 0x12, 0x0e,
0x0a, 0x02, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x22, 0x3a,
0x0a, 0x0c, 0x41, 0x6c, 0x67, 0x6f, 0x72, 0x69, 0x74, 0x68, 0x6d, 0x52, 0x65, 0x71, 0x12, 0x1a,
0x0a, 0x08, 0x70, 0x72, 0x6f, 0x76, 0x69, 0x64, 0x65, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09,
0x52, 0x08, 0x70, 0x72, 0x6f, 0x76, 0x69, 0x64, 0x65, 0x72, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64,
0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x32, 0xf5, 0x01, 0x0a, 0x0c, 0x41,
0x67, 0x65, 0x6e, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x31, 0x0a, 0x04, 0x41,
0x6c, 0x67, 0x6f, 0x12, 0x12, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x41, 0x6c, 0x67, 0x6f,
0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x13, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e,
0x41, 0x6c, 0x67, 0x6f, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x12, 0x31,
0x0a, 0x04, 0x44, 0x61, 0x74, 0x61, 0x12, 0x12, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x44,
0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x13, 0x2e, 0x61, 0x67, 0x65,
0x6e, 0x74, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22,
0x00, 0x12, 0x37, 0x0a, 0x06, 0x52, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x12, 0x14, 0x2e, 0x61, 0x67,
0x65, 0x6e, 0x74, 0x2e, 0x52, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73,
0x74, 0x1a, 0x15, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x52, 0x65, 0x73, 0x75, 0x6c, 0x74,
0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x12, 0x46, 0x0a, 0x0b, 0x41, 0x74,
0x74, 0x65, 0x73, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x19, 0x2e, 0x61, 0x67, 0x65, 0x6e,
0x74, 0x2e, 0x41, 0x74, 0x74, 0x65, 0x73, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x71,
0x75, 0x65, 0x73, 0x74, 0x1a, 0x1a, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x41, 0x74, 0x74,
0x65, 0x73, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65,
0x22, 0x00, 0x42, 0x09, 0x5a, 0x07, 0x2e, 0x2f, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x62, 0x06, 0x70,
0x72, 0x6f, 0x74, 0x6f, 0x33,
0x74, 0x12, 0x1f, 0x0a, 0x0b, 0x72, 0x65, 0x70, 0x6f, 0x72, 0x74, 0x5f, 0x64, 0x61, 0x74, 0x61,
0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x0a, 0x72, 0x65, 0x70, 0x6f, 0x72, 0x74, 0x44, 0x61,
0x74, 0x61, 0x22, 0x29, 0x0a, 0x13, 0x41, 0x74, 0x74, 0x65, 0x73, 0x74, 0x61, 0x74, 0x69, 0x6f,
0x6e, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x66, 0x69, 0x6c,
0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x04, 0x66, 0x69, 0x6c, 0x65, 0x32, 0xfb, 0x01,
0x0a, 0x0c, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x33,
0x0a, 0x04, 0x41, 0x6c, 0x67, 0x6f, 0x12, 0x12, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x41,
0x6c, 0x67, 0x6f, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x13, 0x2e, 0x61, 0x67, 0x65,
0x6e, 0x74, 0x2e, 0x41, 0x6c, 0x67, 0x6f, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22,
0x00, 0x28, 0x01, 0x12, 0x33, 0x0a, 0x04, 0x44, 0x61, 0x74, 0x61, 0x12, 0x12, 0x2e, 0x61, 0x67,
0x65, 0x6e, 0x74, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a,
0x13, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x73, 0x70,
0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x28, 0x01, 0x12, 0x39, 0x0a, 0x06, 0x52, 0x65, 0x73, 0x75,
0x6c, 0x74, 0x12, 0x14, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x52, 0x65, 0x73, 0x75, 0x6c,
0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x15, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74,
0x2e, 0x52, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22,
0x00, 0x30, 0x01, 0x12, 0x46, 0x0a, 0x0b, 0x41, 0x74, 0x74, 0x65, 0x73, 0x74, 0x61, 0x74, 0x69,
0x6f, 0x6e, 0x12, 0x19, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x41, 0x74, 0x74, 0x65, 0x73,
0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1a, 0x2e,
0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x41, 0x74, 0x74, 0x65, 0x73, 0x74, 0x61, 0x74, 0x69, 0x6f,
0x6e, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x42, 0x09, 0x5a, 0x07, 0x2e,
0x2f, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
}
var (
@@ -703,8 +435,8 @@ func file_agent_agent_proto_rawDescGZIP() []byte {
return file_agent_agent_proto_rawDescData
}
var file_agent_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 11)
var file_agent_agent_proto_goTypes = []interface{}{
var file_agent_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 8)
var file_agent_agent_proto_goTypes = []any{
(*AlgoRequest)(nil), // 0: agent.AlgoRequest
(*AlgoResponse)(nil), // 1: agent.AlgoResponse
(*DataRequest)(nil), // 2: agent.DataRequest
@@ -713,26 +445,21 @@ var file_agent_agent_proto_goTypes = []interface{}{
(*ResultResponse)(nil), // 5: agent.ResultResponse
(*AttestationRequest)(nil), // 6: agent.AttestationRequest
(*AttestationResponse)(nil), // 7: agent.AttestationResponse
(*ComputationReq)(nil), // 8: agent.ComputationReq
(*DatasetReq)(nil), // 9: agent.DatasetReq
(*AlgorithmReq)(nil), // 10: agent.AlgorithmReq
}
var file_agent_agent_proto_depIdxs = []int32{
9, // 0: agent.ComputationReq.datasets:type_name -> agent.DatasetReq
10, // 1: agent.ComputationReq.algorithms:type_name -> agent.AlgorithmReq
0, // 2: agent.AgentService.Algo:input_type -> agent.AlgoRequest
2, // 3: agent.AgentService.Data:input_type -> agent.DataRequest
4, // 4: agent.AgentService.Result:input_type -> agent.ResultRequest
6, // 5: agent.AgentService.Attestation:input_type -> agent.AttestationRequest
1, // 6: agent.AgentService.Algo:output_type -> agent.AlgoResponse
3, // 7: agent.AgentService.Data:output_type -> agent.DataResponse
5, // 8: agent.AgentService.Result:output_type -> agent.ResultResponse
7, // 9: agent.AgentService.Attestation:output_type -> agent.AttestationResponse
6, // [6:10] is the sub-list for method output_type
2, // [2:6] is the sub-list for method input_type
2, // [2:2] is the sub-list for extension type_name
2, // [2:2] is the sub-list for extension extendee
0, // [0:2] is the sub-list for field type_name
0, // 0: agent.AgentService.Algo:input_type -> agent.AlgoRequest
2, // 1: agent.AgentService.Data:input_type -> agent.DataRequest
4, // 2: agent.AgentService.Result:input_type -> agent.ResultRequest
6, // 3: agent.AgentService.Attestation:input_type -> agent.AttestationRequest
1, // 4: agent.AgentService.Algo:output_type -> agent.AlgoResponse
3, // 5: agent.AgentService.Data:output_type -> agent.DataResponse
5, // 6: agent.AgentService.Result:output_type -> agent.ResultResponse
7, // 7: agent.AgentService.Attestation:output_type -> agent.AttestationResponse
4, // [4:8] is the sub-list for method output_type
0, // [0:4] is the sub-list for method input_type
0, // [0:0] is the sub-list for extension type_name
0, // [0:0] is the sub-list for extension extendee
0, // [0:0] is the sub-list for field type_name
}
func init() { file_agent_agent_proto_init() }
@@ -741,7 +468,7 @@ func file_agent_agent_proto_init() {
return
}
if !protoimpl.UnsafeEnabled {
file_agent_agent_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
file_agent_agent_proto_msgTypes[0].Exporter = func(v any, i int) any {
switch v := v.(*AlgoRequest); i {
case 0:
return &v.state
@@ -753,7 +480,7 @@ func file_agent_agent_proto_init() {
return nil
}
}
file_agent_agent_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} {
file_agent_agent_proto_msgTypes[1].Exporter = func(v any, i int) any {
switch v := v.(*AlgoResponse); i {
case 0:
return &v.state
@@ -765,7 +492,7 @@ func file_agent_agent_proto_init() {
return nil
}
}
file_agent_agent_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} {
file_agent_agent_proto_msgTypes[2].Exporter = func(v any, i int) any {
switch v := v.(*DataRequest); i {
case 0:
return &v.state
@@ -777,7 +504,7 @@ func file_agent_agent_proto_init() {
return nil
}
}
file_agent_agent_proto_msgTypes[3].Exporter = func(v interface{}, i int) interface{} {
file_agent_agent_proto_msgTypes[3].Exporter = func(v any, i int) any {
switch v := v.(*DataResponse); i {
case 0:
return &v.state
@@ -789,7 +516,7 @@ func file_agent_agent_proto_init() {
return nil
}
}
file_agent_agent_proto_msgTypes[4].Exporter = func(v interface{}, i int) interface{} {
file_agent_agent_proto_msgTypes[4].Exporter = func(v any, i int) any {
switch v := v.(*ResultRequest); i {
case 0:
return &v.state
@@ -801,7 +528,7 @@ func file_agent_agent_proto_init() {
return nil
}
}
file_agent_agent_proto_msgTypes[5].Exporter = func(v interface{}, i int) interface{} {
file_agent_agent_proto_msgTypes[5].Exporter = func(v any, i int) any {
switch v := v.(*ResultResponse); i {
case 0:
return &v.state
@@ -813,7 +540,7 @@ func file_agent_agent_proto_init() {
return nil
}
}
file_agent_agent_proto_msgTypes[6].Exporter = func(v interface{}, i int) interface{} {
file_agent_agent_proto_msgTypes[6].Exporter = func(v any, i int) any {
switch v := v.(*AttestationRequest); i {
case 0:
return &v.state
@@ -825,7 +552,7 @@ func file_agent_agent_proto_init() {
return nil
}
}
file_agent_agent_proto_msgTypes[7].Exporter = func(v interface{}, i int) interface{} {
file_agent_agent_proto_msgTypes[7].Exporter = func(v any, i int) any {
switch v := v.(*AttestationResponse); i {
case 0:
return &v.state
@@ -837,42 +564,6 @@ func file_agent_agent_proto_init() {
return nil
}
}
file_agent_agent_proto_msgTypes[8].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*ComputationReq); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_agent_agent_proto_msgTypes[9].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*DatasetReq); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_agent_agent_proto_msgTypes[10].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*AlgorithmReq); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
}
type x struct{}
out := protoimpl.TypeBuilder{
@@ -880,7 +571,7 @@ func file_agent_agent_proto_init() {
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: file_agent_agent_proto_rawDesc,
NumEnums: 0,
NumMessages: 11,
NumMessages: 8,
NumExtensions: 0,
NumServices: 1,
},
+12 -29
View File
@@ -8,53 +8,36 @@ package agent;
option go_package = "./agent";
service AgentService {
rpc Algo(AlgoRequest) returns (AlgoResponse) {}
rpc Data(DataRequest) returns (DataResponse) {}
rpc Result(ResultRequest) returns (ResultResponse) {}
rpc Algo(stream AlgoRequest) returns (AlgoResponse) {}
rpc Data(stream DataRequest) returns (DataResponse) {}
rpc Result(ResultRequest) returns (stream ResultResponse) {}
rpc Attestation(AttestationRequest) returns (AttestationResponse) {}
}
message AlgoRequest {
bytes algorithm = 1;
string provider = 2;
string id = 3;
bytes requirements = 2;
}
message AlgoResponse { string algorithmID = 1; }
message AlgoResponse {}
message DataRequest {
bytes dataset = 1;
string provider = 2;
string id = 3;
}
message DataResponse { string datasetID = 1; }
message DataResponse {}
message ResultRequest {
string consumer = 1;
}
message ResultResponse { bytes file = 1; }
message AttestationRequest { }
message AttestationResponse { bytes file = 1; }
message ComputationReq {
string id = 1;
string name = 2;
string description = 3;
repeated DatasetReq datasets = 4;
repeated AlgorithmReq algorithms = 5;
repeated string result_consumers = 6;
message ResultResponse {
bytes file = 1;
}
message DatasetReq {
string provider = 1;
string id = 2;
message AttestationRequest {
bytes report_data = 1; // Should be of length 64.
}
message AlgorithmReq {
string provider = 1;
string id = 2;
message AttestationResponse {
bytes file = 1;
}
+189 -89
View File
@@ -3,8 +3,8 @@
// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
// versions:
// - protoc-gen-go-grpc v1.3.0
// - protoc v4.25.1
// - protoc-gen-go-grpc v1.4.0
// - protoc v4.25.3
// source: agent/agent.proto
package agent
@@ -18,8 +18,8 @@ import (
// This is a compile-time assertion to ensure that this generated file
// is compatible with the grpc package it is being compiled against.
// Requires gRPC-Go v1.32.0 or later.
const _ = grpc.SupportPackageIsVersion7
// Requires gRPC-Go v1.62.0 or later.
const _ = grpc.SupportPackageIsVersion8
const (
AgentService_Algo_FullMethodName = "/agent.AgentService/Algo"
@@ -32,9 +32,9 @@ const (
//
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
type AgentServiceClient interface {
Algo(ctx context.Context, in *AlgoRequest, opts ...grpc.CallOption) (*AlgoResponse, error)
Data(ctx context.Context, in *DataRequest, opts ...grpc.CallOption) (*DataResponse, error)
Result(ctx context.Context, in *ResultRequest, opts ...grpc.CallOption) (*ResultResponse, error)
Algo(ctx context.Context, opts ...grpc.CallOption) (AgentService_AlgoClient, error)
Data(ctx context.Context, opts ...grpc.CallOption) (AgentService_DataClient, error)
Result(ctx context.Context, in *ResultRequest, opts ...grpc.CallOption) (AgentService_ResultClient, error)
Attestation(ctx context.Context, in *AttestationRequest, opts ...grpc.CallOption) (*AttestationResponse, error)
}
@@ -46,36 +46,113 @@ func NewAgentServiceClient(cc grpc.ClientConnInterface) AgentServiceClient {
return &agentServiceClient{cc}
}
func (c *agentServiceClient) Algo(ctx context.Context, in *AlgoRequest, opts ...grpc.CallOption) (*AlgoResponse, error) {
out := new(AlgoResponse)
err := c.cc.Invoke(ctx, AgentService_Algo_FullMethodName, in, out, opts...)
func (c *agentServiceClient) Algo(ctx context.Context, opts ...grpc.CallOption) (AgentService_AlgoClient, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
stream, err := c.cc.NewStream(ctx, &AgentService_ServiceDesc.Streams[0], AgentService_Algo_FullMethodName, cOpts...)
if err != nil {
return nil, err
}
return out, nil
x := &agentServiceAlgoClient{ClientStream: stream}
return x, nil
}
func (c *agentServiceClient) Data(ctx context.Context, in *DataRequest, opts ...grpc.CallOption) (*DataResponse, error) {
out := new(DataResponse)
err := c.cc.Invoke(ctx, AgentService_Data_FullMethodName, in, out, opts...)
if err != nil {
return nil, err
}
return out, nil
type AgentService_AlgoClient interface {
Send(*AlgoRequest) error
CloseAndRecv() (*AlgoResponse, error)
grpc.ClientStream
}
func (c *agentServiceClient) Result(ctx context.Context, in *ResultRequest, opts ...grpc.CallOption) (*ResultResponse, error) {
out := new(ResultResponse)
err := c.cc.Invoke(ctx, AgentService_Result_FullMethodName, in, out, opts...)
type agentServiceAlgoClient struct {
grpc.ClientStream
}
func (x *agentServiceAlgoClient) Send(m *AlgoRequest) error {
return x.ClientStream.SendMsg(m)
}
func (x *agentServiceAlgoClient) CloseAndRecv() (*AlgoResponse, error) {
if err := x.ClientStream.CloseSend(); err != nil {
return nil, err
}
m := new(AlgoResponse)
if err := x.ClientStream.RecvMsg(m); err != nil {
return nil, err
}
return m, nil
}
func (c *agentServiceClient) Data(ctx context.Context, opts ...grpc.CallOption) (AgentService_DataClient, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
stream, err := c.cc.NewStream(ctx, &AgentService_ServiceDesc.Streams[1], AgentService_Data_FullMethodName, cOpts...)
if err != nil {
return nil, err
}
return out, nil
x := &agentServiceDataClient{ClientStream: stream}
return x, nil
}
type AgentService_DataClient interface {
Send(*DataRequest) error
CloseAndRecv() (*DataResponse, error)
grpc.ClientStream
}
type agentServiceDataClient struct {
grpc.ClientStream
}
func (x *agentServiceDataClient) Send(m *DataRequest) error {
return x.ClientStream.SendMsg(m)
}
func (x *agentServiceDataClient) CloseAndRecv() (*DataResponse, error) {
if err := x.ClientStream.CloseSend(); err != nil {
return nil, err
}
m := new(DataResponse)
if err := x.ClientStream.RecvMsg(m); err != nil {
return nil, err
}
return m, nil
}
func (c *agentServiceClient) Result(ctx context.Context, in *ResultRequest, opts ...grpc.CallOption) (AgentService_ResultClient, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
stream, err := c.cc.NewStream(ctx, &AgentService_ServiceDesc.Streams[2], AgentService_Result_FullMethodName, cOpts...)
if err != nil {
return nil, err
}
x := &agentServiceResultClient{ClientStream: stream}
if err := x.ClientStream.SendMsg(in); err != nil {
return nil, err
}
if err := x.ClientStream.CloseSend(); err != nil {
return nil, err
}
return x, nil
}
type AgentService_ResultClient interface {
Recv() (*ResultResponse, error)
grpc.ClientStream
}
type agentServiceResultClient struct {
grpc.ClientStream
}
func (x *agentServiceResultClient) Recv() (*ResultResponse, error) {
m := new(ResultResponse)
if err := x.ClientStream.RecvMsg(m); err != nil {
return nil, err
}
return m, nil
}
func (c *agentServiceClient) Attestation(ctx context.Context, in *AttestationRequest, opts ...grpc.CallOption) (*AttestationResponse, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
out := new(AttestationResponse)
err := c.cc.Invoke(ctx, AgentService_Attestation_FullMethodName, in, out, opts...)
err := c.cc.Invoke(ctx, AgentService_Attestation_FullMethodName, in, out, cOpts...)
if err != nil {
return nil, err
}
@@ -86,9 +163,9 @@ func (c *agentServiceClient) Attestation(ctx context.Context, in *AttestationReq
// All implementations must embed UnimplementedAgentServiceServer
// for forward compatibility
type AgentServiceServer interface {
Algo(context.Context, *AlgoRequest) (*AlgoResponse, error)
Data(context.Context, *DataRequest) (*DataResponse, error)
Result(context.Context, *ResultRequest) (*ResultResponse, error)
Algo(AgentService_AlgoServer) error
Data(AgentService_DataServer) error
Result(*ResultRequest, AgentService_ResultServer) error
Attestation(context.Context, *AttestationRequest) (*AttestationResponse, error)
mustEmbedUnimplementedAgentServiceServer()
}
@@ -97,14 +174,14 @@ type AgentServiceServer interface {
type UnimplementedAgentServiceServer struct {
}
func (UnimplementedAgentServiceServer) Algo(context.Context, *AlgoRequest) (*AlgoResponse, error) {
return nil, status.Errorf(codes.Unimplemented, "method Algo not implemented")
func (UnimplementedAgentServiceServer) Algo(AgentService_AlgoServer) error {
return status.Errorf(codes.Unimplemented, "method Algo not implemented")
}
func (UnimplementedAgentServiceServer) Data(context.Context, *DataRequest) (*DataResponse, error) {
return nil, status.Errorf(codes.Unimplemented, "method Data not implemented")
func (UnimplementedAgentServiceServer) Data(AgentService_DataServer) error {
return status.Errorf(codes.Unimplemented, "method Data not implemented")
}
func (UnimplementedAgentServiceServer) Result(context.Context, *ResultRequest) (*ResultResponse, error) {
return nil, status.Errorf(codes.Unimplemented, "method Result not implemented")
func (UnimplementedAgentServiceServer) Result(*ResultRequest, AgentService_ResultServer) error {
return status.Errorf(codes.Unimplemented, "method Result not implemented")
}
func (UnimplementedAgentServiceServer) Attestation(context.Context, *AttestationRequest) (*AttestationResponse, error) {
return nil, status.Errorf(codes.Unimplemented, "method Attestation not implemented")
@@ -122,58 +199,77 @@ func RegisterAgentServiceServer(s grpc.ServiceRegistrar, srv AgentServiceServer)
s.RegisterService(&AgentService_ServiceDesc, srv)
}
func _AgentService_Algo_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(AlgoRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(AgentServiceServer).Algo(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: AgentService_Algo_FullMethodName,
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(AgentServiceServer).Algo(ctx, req.(*AlgoRequest))
}
return interceptor(ctx, in, info, handler)
func _AgentService_Algo_Handler(srv interface{}, stream grpc.ServerStream) error {
return srv.(AgentServiceServer).Algo(&agentServiceAlgoServer{ServerStream: stream})
}
func _AgentService_Data_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(DataRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(AgentServiceServer).Data(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: AgentService_Data_FullMethodName,
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(AgentServiceServer).Data(ctx, req.(*DataRequest))
}
return interceptor(ctx, in, info, handler)
type AgentService_AlgoServer interface {
SendAndClose(*AlgoResponse) error
Recv() (*AlgoRequest, error)
grpc.ServerStream
}
func _AgentService_Result_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(ResultRequest)
if err := dec(in); err != nil {
type agentServiceAlgoServer struct {
grpc.ServerStream
}
func (x *agentServiceAlgoServer) SendAndClose(m *AlgoResponse) error {
return x.ServerStream.SendMsg(m)
}
func (x *agentServiceAlgoServer) Recv() (*AlgoRequest, error) {
m := new(AlgoRequest)
if err := x.ServerStream.RecvMsg(m); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(AgentServiceServer).Result(ctx, in)
return m, nil
}
func _AgentService_Data_Handler(srv interface{}, stream grpc.ServerStream) error {
return srv.(AgentServiceServer).Data(&agentServiceDataServer{ServerStream: stream})
}
type AgentService_DataServer interface {
SendAndClose(*DataResponse) error
Recv() (*DataRequest, error)
grpc.ServerStream
}
type agentServiceDataServer struct {
grpc.ServerStream
}
func (x *agentServiceDataServer) SendAndClose(m *DataResponse) error {
return x.ServerStream.SendMsg(m)
}
func (x *agentServiceDataServer) Recv() (*DataRequest, error) {
m := new(DataRequest)
if err := x.ServerStream.RecvMsg(m); err != nil {
return nil, err
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: AgentService_Result_FullMethodName,
return m, nil
}
func _AgentService_Result_Handler(srv interface{}, stream grpc.ServerStream) error {
m := new(ResultRequest)
if err := stream.RecvMsg(m); err != nil {
return err
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(AgentServiceServer).Result(ctx, req.(*ResultRequest))
}
return interceptor(ctx, in, info, handler)
return srv.(AgentServiceServer).Result(m, &agentServiceResultServer{ServerStream: stream})
}
type AgentService_ResultServer interface {
Send(*ResultResponse) error
grpc.ServerStream
}
type agentServiceResultServer struct {
grpc.ServerStream
}
func (x *agentServiceResultServer) Send(m *ResultResponse) error {
return x.ServerStream.SendMsg(m)
}
func _AgentService_Attestation_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
@@ -201,23 +297,27 @@ var AgentService_ServiceDesc = grpc.ServiceDesc{
ServiceName: "agent.AgentService",
HandlerType: (*AgentServiceServer)(nil),
Methods: []grpc.MethodDesc{
{
MethodName: "Algo",
Handler: _AgentService_Algo_Handler,
},
{
MethodName: "Data",
Handler: _AgentService_Data_Handler,
},
{
MethodName: "Result",
Handler: _AgentService_Result_Handler,
},
{
MethodName: "Attestation",
Handler: _AgentService_Attestation_Handler,
},
},
Streams: []grpc.StreamDesc{},
Streams: []grpc.StreamDesc{
{
StreamName: "Algo",
Handler: _AgentService_Algo_Handler,
ClientStreams: true,
},
{
StreamName: "Data",
Handler: _AgentService_Data_Handler,
ClientStreams: true,
},
{
StreamName: "Result",
Handler: _AgentService_Result_Handler,
ServerStreams: true,
},
},
Metadata: "agent/agent.proto",
}
+34
View File
@@ -0,0 +1,34 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package algorithm
import (
"context"
"google.golang.org/grpc/metadata"
)
type AlgorithType string
const (
AlgoTypeBin AlgorithType = "bin"
AlgoTypePython AlgorithType = "python"
AlgoTypeKey = "algo_type"
)
func AlgorithmTypeToContext(ctx context.Context, algoType string) context.Context {
return metadata.AppendToOutgoingContext(ctx, AlgoTypeKey, algoType)
}
func AlgorithmTypeFromContext(ctx context.Context) string {
return metadata.ValueFromIncomingContext(ctx, AlgoTypeKey)[0]
}
// Algorithm is an interface that specifies the API for an algorithm.
type Algorithm interface {
// Run executes the algorithm and returns the result.
Run() ([]byte, error)
// Add dataset to algorithm.
AddDataset(dataset string)
}
+82
View File
@@ -0,0 +1,82 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package binary
import (
"fmt"
"io"
"log/slog"
"os"
"os/exec"
"github.com/ultravioletrs/cocos/agent/algorithm"
"github.com/ultravioletrs/cocos/agent/events"
"github.com/ultravioletrs/cocos/pkg/socket"
)
const socketPath = "unix_socket"
var _ algorithm.Algorithm = (*binary)(nil)
type binary struct {
algoFile string
datasets []string
logger *slog.Logger
stderr io.Writer
stdout io.Writer
}
func New(logger *slog.Logger, eventsSvc events.Service, algoFile string) algorithm.Algorithm {
return &binary{
algoFile: algoFile,
logger: logger,
stderr: &algorithm.Stderr{Logger: logger, EventSvc: eventsSvc},
stdout: &algorithm.Stdout{Logger: logger},
}
}
func (b *binary) AddDataset(dataset string) {
b.datasets = append(b.datasets, dataset)
}
func (b *binary) Run() ([]byte, error) {
defer os.Remove(b.algoFile)
defer func() {
for _, file := range b.datasets {
os.Remove(file)
}
}()
listener, err := socket.StartUnixSocketServer(socketPath)
if err != nil {
return nil, fmt.Errorf("error creating stdout pipe: %v", err)
}
defer listener.Close()
// Create channels for received data and errors
dataChannel := make(chan []byte)
errorChannel := make(chan error)
var result []byte
go socket.AcceptConnection(listener, dataChannel, errorChannel)
args := append([]string{socketPath}, b.datasets...)
cmd := exec.Command(b.algoFile, args...)
cmd.Stderr = b.stderr
cmd.Stdout = b.stdout
if err := cmd.Start(); err != nil {
return nil, fmt.Errorf("error starting algorithm: %v", err)
}
if err := cmd.Wait(); err != nil {
return nil, fmt.Errorf("algorithm execution error: %v", err)
}
select {
case result = <-dataChannel:
return result, nil
case err = <-errorChannel:
return nil, fmt.Errorf("error receiving data: %v", err)
}
}
+73
View File
@@ -0,0 +1,73 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package algorithm
import (
"bytes"
"io"
"log/slog"
"github.com/ultravioletrs/cocos/agent/events"
)
var (
_ io.Writer = &Stdout{}
_ io.Writer = &Stderr{}
)
const bufSize = 1024
type Stdout struct {
Logger *slog.Logger
}
// Write implements io.Writer.
func (s *Stdout) Write(p []byte) (n int, err error) {
inBuf := bytes.NewBuffer(p)
buf := make([]byte, bufSize)
for {
n, err := inBuf.Read(buf)
if err != nil {
if err == io.EOF {
break
}
return len(p) - inBuf.Len(), err
}
s.Logger.Debug(string(buf[:n]))
}
return len(p), nil
}
type Stderr struct {
Logger *slog.Logger
EventSvc events.Service
}
// Write implements io.Writer.
func (s *Stderr) Write(p []byte) (n int, err error) {
inBuf := bytes.NewBuffer(p)
buf := make([]byte, bufSize)
for {
n, err := inBuf.Read(buf)
if err != nil {
if err == io.EOF {
break
}
return len(p) - inBuf.Len(), err
}
s.Logger.Error(string(buf[:n]))
}
if err := s.EventSvc.SendEvent("algorithm-run", "error", nil); err != nil {
return len(p), err
}
return len(p), nil
}
+85
View File
@@ -0,0 +1,85 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package algorithm
import (
"strings"
"testing"
mglog "github.com/absmach/magistrala/logger"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"github.com/ultravioletrs/cocos/agent/events/mocks"
)
func TestStdoutWrite(t *testing.T) {
tests := []struct {
name string
input string
expected []string
}{
{
name: "Single line",
input: "Hello, World!",
expected: []string{"Hello, World!"},
},
{
name: "Multiple lines",
input: "Line 1\nLine 2\nLine 3",
expected: []string{"Line 1\nLine 2\nLine 3"},
},
{
name: "Long input",
input: strings.Repeat("a", bufSize+100),
expected: []string{strings.Repeat("a", bufSize), strings.Repeat("a", 100)},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
stdout := &Stdout{Logger: mglog.NewMock()}
n, err := stdout.Write([]byte(tt.input))
assert.NoError(t, err)
assert.Equal(t, len(tt.input), n)
})
}
}
func TestStderrWrite(t *testing.T) {
tests := []struct {
name string
input string
expected []string
}{
{
name: "Single line",
input: "Error: Something went wrong",
expected: []string{"Error: Something went wrong"},
},
{
name: "Multiple lines",
input: "Error 1\nError 2\nError 3",
expected: []string{"Error 1\nError 2\nError 3"},
},
{
name: "Long input",
input: strings.Repeat("e", bufSize+100),
expected: []string{strings.Repeat("e", bufSize), strings.Repeat("e", 100)},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
mockEventService := mocks.NewService(t)
mockEventService.On("SendEvent", "algorithm-run", "error", mock.Anything).Return(nil)
stderr := &Stderr{Logger: mglog.NewMock(), EventSvc: mockEventService}
n, err := stderr.Write([]byte(tt.input))
assert.NoError(t, err)
assert.Equal(t, len(tt.input), n)
mockEventService.AssertExpectations(t)
})
}
}
+126
View File
@@ -0,0 +1,126 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package python
import (
"context"
"fmt"
"io"
"log/slog"
"os"
"os/exec"
"path/filepath"
"github.com/ultravioletrs/cocos/agent/algorithm"
"github.com/ultravioletrs/cocos/agent/events"
"github.com/ultravioletrs/cocos/pkg/socket"
"google.golang.org/grpc/metadata"
)
const (
socketPath = "unix_socket"
PyRuntime = "python3"
pyRuntimeKey = "python_runtime"
)
func PythonRunTimeToContext(ctx context.Context, runtime string) context.Context {
return metadata.AppendToOutgoingContext(ctx, pyRuntimeKey, runtime)
}
func PythonRunTimeFromContext(ctx context.Context) string {
return metadata.ValueFromIncomingContext(ctx, pyRuntimeKey)[0]
}
var _ algorithm.Algorithm = (*python)(nil)
type python struct {
algoFile string
datasets []string
logger *slog.Logger
stderr io.Writer
stdout io.Writer
runtime string
requirementsFile string
}
func New(logger *slog.Logger, eventsSvc events.Service, runtime, requirementsFile, algoFile string) algorithm.Algorithm {
p := &python{
algoFile: algoFile,
logger: logger,
stderr: &algorithm.Stderr{Logger: logger, EventSvc: eventsSvc},
stdout: &algorithm.Stdout{Logger: logger},
requirementsFile: requirementsFile,
}
if runtime != "" {
p.runtime = runtime
} else {
p.runtime = PyRuntime
}
return p
}
func (p *python) AddDataset(dataset string) {
p.datasets = append(p.datasets, dataset)
}
func (p *python) Run() ([]byte, error) {
venvPath := "venv"
createVenvCmd := exec.Command(p.runtime, "-m", "venv", venvPath)
createVenvCmd.Stderr = p.stderr
createVenvCmd.Stdout = p.stdout
if err := createVenvCmd.Run(); err != nil {
return nil, fmt.Errorf("error creating virtual environment: %v", err)
}
pythonPath := filepath.Join(venvPath, "bin", "python")
if p.requirementsFile != "" {
rcmd := exec.Command(pythonPath, "-m", "pip", "install", "-r", p.requirementsFile)
rcmd.Stderr = p.stderr
rcmd.Stdout = p.stdout
if err := rcmd.Run(); err != nil {
return nil, fmt.Errorf("error installing requirements: %v", err)
}
}
defer os.Remove(p.algoFile)
defer func() {
for _, file := range p.datasets {
os.Remove(file)
}
}()
defer os.RemoveAll(venvPath)
listener, err := socket.StartUnixSocketServer(socketPath)
if err != nil {
return nil, fmt.Errorf("error creating stdout pipe: %v", err)
}
defer listener.Close()
dataChannel := make(chan []byte)
errorChannel := make(chan error)
var result []byte
go socket.AcceptConnection(listener, dataChannel, errorChannel)
args := append([]string{p.algoFile, socketPath}, p.datasets...)
cmd := exec.Command(pythonPath, args...)
cmd.Stderr = p.stderr
cmd.Stdout = p.stdout
if err := cmd.Start(); err != nil {
return nil, fmt.Errorf("error starting algorithm: %v", err)
}
if err := cmd.Wait(); err != nil {
return nil, fmt.Errorf("algorithm execution error: %v", err)
}
select {
case result = <-dataChannel:
return result, nil
case err = <-errorChannel:
return nil, fmt.Errorf("error receiving data: %v", err)
}
}
-221
View File
@@ -1,221 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package grpc
import (
"context"
"fmt"
"time"
"github.com/go-kit/kit/endpoint"
kitgrpc "github.com/go-kit/kit/transport/grpc"
"github.com/ultravioletrs/cocos/agent"
"google.golang.org/grpc"
)
const svcName = "agent.AgentService"
type grpcClient struct {
algo endpoint.Endpoint
data endpoint.Endpoint
result endpoint.Endpoint
attestation endpoint.Endpoint
timeout time.Duration
}
// NewClient returns new gRPC client instance.
func NewClient(conn *grpc.ClientConn, timeout time.Duration) agent.AgentServiceClient {
return &grpcClient{
algo: kitgrpc.NewClient(
conn,
svcName,
"Algo",
encodeAlgoRequest,
decodeAlgoResponse,
agent.AlgoResponse{},
).Endpoint(),
data: kitgrpc.NewClient(
conn,
svcName,
"Data",
encodeDataRequest,
decodeDataResponse,
agent.DataResponse{},
).Endpoint(),
result: kitgrpc.NewClient(
conn,
svcName,
"Result",
encodeResultRequest,
decodeResultResponse,
agent.ResultResponse{},
).Endpoint(),
attestation: kitgrpc.NewClient(
conn,
svcName,
"Attestation",
encodeAttestationRequest,
decodeAttestationResponse,
agent.AttestationResponse{},
).Endpoint(),
timeout: timeout,
}
}
// encodeAlgoRequest is a transport/grpc.EncodeRequestFunc that
// converts a user-domain algoReq to a gRPC request.
func encodeAlgoRequest(_ context.Context, request interface{}) (interface{}, error) {
req, ok := request.(*algoReq)
if !ok {
return nil, fmt.Errorf("invalid request type: %T", request)
}
return &agent.AlgoRequest{
Algorithm: req.Algorithm,
Provider: req.Provider,
Id: req.Id,
}, nil
}
// decodeAlgoResponse is a transport/grpc.DecodeResponseFunc that
// converts a gRPC AlgoResponse to a user-domain response.
func decodeAlgoResponse(_ context.Context, grpcResponse interface{}) (interface{}, error) {
response, ok := grpcResponse.(*agent.AlgoResponse)
if !ok {
return nil, fmt.Errorf("invalid response type: %T", grpcResponse)
}
return algoRes{
AlgorithmID: response.AlgorithmID,
}, nil
}
// encodeDataRequest is a transport/grpc.EncodeRequestFunc that
// converts a user-domain dataReq to a gRPC request.
func encodeDataRequest(_ context.Context, request interface{}) (interface{}, error) {
req, ok := request.(*dataReq)
if !ok {
return nil, fmt.Errorf("invalid request type: %T", request)
}
return &agent.DataRequest{
Dataset: req.Dataset,
Provider: req.Provider,
Id: req.Id,
}, nil
}
// decodeDataResponse is a transport/grpc.DecodeResponseFunc that
// converts a gRPC DataResponse to a user-domain response.
func decodeDataResponse(_ context.Context, grpcResponse interface{}) (interface{}, error) {
response, ok := grpcResponse.(*agent.DataResponse)
if !ok {
return nil, fmt.Errorf("invalid response type: %T", grpcResponse)
}
return dataRes{
DatasetID: response.DatasetID,
}, nil
}
// encodeResultRequest is a transport/grpc.EncodeRequestFunc that
// converts a user-domain resultReq to a gRPC request.
func encodeResultRequest(_ context.Context, request interface{}) (interface{}, error) {
req, ok := request.(*resultReq)
if !ok {
return nil, fmt.Errorf("invalid request type: %T", request)
}
return &agent.ResultRequest{
Consumer: req.Consumer,
}, nil
}
// decodeResultResponse is a transport/grpc.DecodeResponseFunc that
// converts a gRPC ResultResponse to a user-domain response.
func decodeResultResponse(_ context.Context, grpcResponse interface{}) (interface{}, error) {
response, ok := grpcResponse.(*agent.ResultResponse)
if !ok {
return nil, fmt.Errorf("invalid response type: %T", grpcResponse)
}
return resultRes{
File: response.File,
}, nil
}
// encodeAttestationRequest is a transport/grpc.EncodeRequestFunc that
// converts a user-domain attestationReq to a gRPC request.
func encodeAttestationRequest(_ context.Context, request interface{}) (interface{}, error) {
// No request parameters needed for retrieving computation result file.
return &agent.AttestationRequest{}, nil
}
// decodeAttestationResponse is a transport/grpc.DecodeResponseFunc that
// converts a gRPC AttestationResponse to a user-domain response.
func decodeAttestationResponse(_ context.Context, grpcResponse interface{}) (interface{}, error) {
response, ok := grpcResponse.(*agent.AttestationResponse)
if !ok {
return nil, fmt.Errorf("invalid response type: %T", grpcResponse)
}
return attestationRes{
File: response.File,
}, nil
}
// Algo implements the Algo method of the agent.AgentServiceClient interface.
func (c grpcClient) Algo(ctx context.Context, request *agent.AlgoRequest, _ ...grpc.CallOption) (*agent.AlgoResponse, error) {
ctx, cancel := context.WithTimeout(ctx, c.timeout)
defer cancel()
res, err := c.algo(ctx, &algoReq{Algorithm: request.Algorithm, Provider: request.Provider, Id: request.Id})
if err != nil {
return nil, err
}
algoRes := res.(algoRes)
return &agent.AlgoResponse{AlgorithmID: algoRes.AlgorithmID}, nil
}
// Data implements the Data method of the agent.AgentServiceClient interface.
func (c grpcClient) Data(ctx context.Context, request *agent.DataRequest, _ ...grpc.CallOption) (*agent.DataResponse, error) {
ctx, cancel := context.WithTimeout(ctx, c.timeout)
defer cancel()
res, err := c.data(ctx, &dataReq{Dataset: request.Dataset, Provider: request.Provider, Id: request.Id})
if err != nil {
return nil, err
}
dataRes := res.(dataRes)
return &agent.DataResponse{DatasetID: dataRes.DatasetID}, nil
}
// Result implements the Result method of the agent.AgentServiceClient interface.
func (c grpcClient) Result(ctx context.Context, request *agent.ResultRequest, _ ...grpc.CallOption) (*agent.ResultResponse, error) {
ctx, cancel := context.WithTimeout(ctx, c.timeout)
defer cancel()
res, err := c.result(ctx, &resultReq{Consumer: request.Consumer})
if err != nil {
return nil, err
}
resultRes := res.(resultRes)
return &agent.ResultResponse{File: resultRes.File}, nil
}
// Result implements the Result method of the agent.AgentServiceClient interface.
func (c grpcClient) Attestation(ctx context.Context, request *agent.AttestationRequest, _ ...grpc.CallOption) (*agent.AttestationResponse, error) {
ctx, cancel := context.WithTimeout(ctx, c.timeout)
defer cancel()
res, err := c.attestation(ctx, &attestationReq{})
if err != nil {
return nil, err
}
attestationRes := res.(attestationRes)
return &agent.AttestationResponse{File: attestationRes.File}, nil
}
+8 -8
View File
@@ -17,14 +17,14 @@ func algoEndpoint(svc agent.Service) endpoint.Endpoint {
return algoRes{}, err
}
algo := agent.Algorithm{Algorithm: req.Algorithm, Provider: req.Provider, ID: req.Id}
algo := agent.Algorithm{Algorithm: req.Algorithm, Requirements: req.Requirements}
algorithmID, err := svc.Algo(ctx, algo)
err := svc.Algo(ctx, algo)
if err != nil {
return algoRes{}, err
}
return algoRes{AlgorithmID: algorithmID}, nil
return algoRes{}, nil
}
}
@@ -36,14 +36,14 @@ func dataEndpoint(svc agent.Service) endpoint.Endpoint {
return dataRes{}, err
}
dataset := agent.Dataset{Dataset: req.Dataset, Provider: req.Provider, ID: req.Id}
dataset := agent.Dataset{Dataset: req.Dataset}
datasetID, err := svc.Data(ctx, dataset)
err := svc.Data(ctx, dataset)
if err != nil {
return dataRes{}, err
}
return dataRes{DatasetID: datasetID}, nil
return dataRes{}, nil
}
}
@@ -54,7 +54,7 @@ func resultEndpoint(svc agent.Service) endpoint.Endpoint {
if err := req.validate(); err != nil {
return resultRes{}, err
}
file, err := svc.Result(ctx, req.Consumer)
file, err := svc.Result(ctx)
if err != nil {
return resultRes{}, err
}
@@ -70,7 +70,7 @@ func attestationEndpoint(svc agent.Service) endpoint.Endpoint {
if err := req.validate(); err != nil {
return attestationRes{}, err
}
file, err := svc.Attestation(ctx)
file, err := svc.Attestation(ctx, req.ReportData)
if err != nil {
return attestationRes{}, err
}
+74
View File
@@ -0,0 +1,74 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package grpc
import (
"context"
"github.com/ultravioletrs/cocos/agent"
"github.com/ultravioletrs/cocos/agent/auth"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
type authInterceptor struct {
auth auth.Authenticator
}
type wrappedServerStream struct {
grpc.ServerStream
ctx context.Context
}
func (s *wrappedServerStream) Context() context.Context {
return s.ctx
}
func NewAuthInterceptor(authSvc auth.Authenticator) (grpc.UnaryServerInterceptor, grpc.StreamServerInterceptor) {
ai := &authInterceptor{auth: authSvc}
return ai.AuthUnaryInterceptor(), ai.AuthStreamInterceptor()
}
func (s *authInterceptor) AuthStreamInterceptor() grpc.StreamServerInterceptor {
return func(srv interface{}, stream grpc.ServerStream, info *grpc.StreamServerInfo, handler grpc.StreamHandler) error {
switch info.FullMethod {
case agent.AgentService_Algo_FullMethodName:
if _, err := s.auth.AuthenticateUser(stream.Context(), auth.AlgorithmProviderRole); err != nil {
return status.Errorf(codes.Unauthenticated, err.Error())
}
return handler(srv, stream)
case agent.AgentService_Data_FullMethodName:
ctx, err := s.auth.AuthenticateUser(stream.Context(), auth.DataProviderRole)
if err != nil {
return status.Errorf(codes.Unauthenticated, err.Error())
}
wrapped := &wrappedServerStream{ServerStream: stream, ctx: ctx}
return handler(srv, wrapped)
case agent.AgentService_Result_FullMethodName:
ctx, err := s.auth.AuthenticateUser(stream.Context(), auth.ConsumerRole)
if err != nil {
return status.Errorf(codes.Unauthenticated, err.Error())
}
wrapped := &wrappedServerStream{ServerStream: stream, ctx: ctx}
return handler(srv, wrapped)
default:
return handler(srv, stream)
}
}
}
func (s *authInterceptor) AuthUnaryInterceptor() grpc.UnaryServerInterceptor {
return func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
switch info.FullMethod {
case agent.AgentService_Result_FullMethodName:
ctx, err := s.auth.AuthenticateUser(ctx, auth.ConsumerRole)
if err != nil {
return nil, status.Errorf(codes.Unauthenticated, err.Error())
}
return handler(ctx, req)
default:
return handler(ctx, req)
}
}
}
+152
View File
@@ -0,0 +1,152 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package grpc
import (
"context"
"testing"
"github.com/stretchr/testify/mock"
"github.com/ultravioletrs/cocos/agent"
"github.com/ultravioletrs/cocos/agent/auth"
"github.com/ultravioletrs/cocos/agent/mocks"
"google.golang.org/grpc"
"google.golang.org/grpc/metadata"
)
func TestAuthUnaryInterceptor(t *testing.T) {
authmock := new(mocks.Authenticator)
tests := []struct {
name string
authorized bool
method string
role auth.UserRole
wantErr bool
}{
{
name: "authorized result method",
authorized: true,
method: agent.AgentService_Result_FullMethodName,
role: auth.ConsumerRole,
wantErr: false,
},
{
name: "unauthorized result method",
authorized: false,
method: agent.AgentService_Result_FullMethodName,
role: auth.ConsumerRole,
wantErr: true,
},
{
name: "other method",
authorized: false,
method: "OtherMethod",
role: auth.ConsumerRole,
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
switch tt.authorized {
case true:
mockCall := authmock.On("AuthenticateUser", context.Background(), tt.role).Return(context.Background(), nil)
mockCall.Once()
case false:
mockCall := authmock.On("AuthenticateUser", context.Background(), tt.role).Return(context.Background(), auth.ErrMissingMetadata)
mockCall.Once()
}
unaryInt, _ := NewAuthInterceptor(authmock)
_, err := unaryInt(context.Background(), nil, &grpc.UnaryServerInfo{FullMethod: tt.method}, func(ctx context.Context, req interface{}) (interface{}, error) {
return nil, nil
})
if tt.wantErr && err == nil {
t.Errorf("expected error, got none")
} else if !tt.wantErr && err != nil {
t.Errorf("unexpected error: %v", err)
}
})
}
}
func TestAuthStreamInterceptor(t *testing.T) {
authmock := new(mocks.Authenticator)
tests := []struct {
name string
authorized bool
method string
role auth.UserRole
wantErr bool
}{
{
name: "authorized algo method",
authorized: true,
method: agent.AgentService_Algo_FullMethodName,
role: auth.AlgorithmProviderRole,
wantErr: false,
},
{
name: "unauthorized algo method",
authorized: false,
method: agent.AgentService_Algo_FullMethodName,
role: auth.AlgorithmProviderRole,
wantErr: true,
},
{
name: "authorized data method",
authorized: true,
method: agent.AgentService_Data_FullMethodName,
role: auth.DataProviderRole,
wantErr: false,
},
{
name: "unauthorized data method",
authorized: false,
method: agent.AgentService_Data_FullMethodName,
role: auth.DataProviderRole,
wantErr: true,
},
{
name: "other method",
authorized: false,
method: "OtherMethod",
role: auth.DataProviderRole,
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
switch tt.authorized {
case true:
mockCall := authmock.On("AuthenticateUser", mock.Anything, tt.role).Return(context.Background(), nil)
mockCall.Once()
case false:
mockCall := authmock.On("AuthenticateUser", mock.Anything, tt.role).Return(context.Background(), auth.ErrMissingMetadata)
mockCall.Once()
}
_, streamInt := NewAuthInterceptor(authmock)
err := streamInt(nil, &mockServerStream{ctx: metadata.NewIncomingContext(context.Background(), metadata.Pairs())}, &grpc.StreamServerInfo{FullMethod: tt.method}, func(srv interface{}, stream grpc.ServerStream) error {
return nil
})
if tt.wantErr && err == nil {
t.Errorf("expected error, got none")
} else if !tt.wantErr && err != nil {
t.Errorf("unexpected error: %v", err)
}
})
}
}
type mockServerStream struct {
grpc.ServerStream
ctx context.Context
}
func (m *mockServerStream) Context() context.Context {
return m.ctx
}
+5 -23
View File
@@ -7,46 +7,29 @@ import (
)
type algoReq struct {
Algorithm []byte `protobuf:"bytes,1,opt,name=algorithm,proto3" json:"algorithm,omitempty"`
Provider string `protobuf:"bytes,2,opt,name=provider,proto3" json:"provider,omitempty"`
Id string `protobuf:"bytes,3,opt,name=id,proto3" json:"id,omitempty"`
Algorithm []byte `protobuf:"bytes,1,opt,name=algorithm,proto3" json:"algorithm,omitempty"`
Requirements []byte
}
func (req algoReq) validate() error {
if len(req.Algorithm) == 0 {
return errors.New("algorithm binary is required")
}
if req.Id == "" {
return errors.New("malformed entity")
}
if req.Provider == "" {
return errors.New("malformed entity")
}
return nil
}
type dataReq struct {
Dataset []byte `protobuf:"bytes,1,opt,name=dataset,proto3" json:"dataset,omitempty"`
Provider string `protobuf:"bytes,2,opt,name=provider,proto3" json:"provider,omitempty"`
Id string `protobuf:"bytes,3,opt,name=id,proto3" json:"id,omitempty"`
Dataset []byte `protobuf:"bytes,1,opt,name=dataset,proto3" json:"dataset,omitempty"`
}
func (req dataReq) validate() error {
if len(req.Dataset) == 0 {
return errors.New("dataset CSV file is required")
}
if req.Id == "" {
return errors.New("malformed entity")
}
if req.Provider == "" {
return errors.New("malformed entity")
}
return nil
}
type resultReq struct {
Consumer string `protobuf:"bytes,1,opt,name=consumer,proto3" json:"consumer,omitempty"`
}
type resultReq struct{}
func (req resultReq) validate() error {
// No request parameters to validate, so no validation logic needed
@@ -54,10 +37,9 @@ func (req resultReq) validate() error {
}
type attestationReq struct {
// No request parameters needed for retrieving attestation output
ReportData [64]byte
}
func (req attestationReq) validate() error {
// No request parameters to validate, so no validation logic needed
return nil
}
+2 -6
View File
@@ -2,13 +2,9 @@
// SPDX-License-Identifier: Apache-2.0
package grpc
type algoRes struct {
AlgorithmID string `json:"algorithmId,omitempty"`
}
type algoRes struct{}
type dataRes struct {
DatasetID string `json:"datasetId,omitempty"`
}
type dataRes struct{}
type resultRes struct {
File []byte `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
+78 -32
View File
@@ -3,12 +3,21 @@
package grpc
import (
"bytes"
"context"
"errors"
"io"
"github.com/go-kit/kit/transport/grpc"
"github.com/ultravioletrs/cocos/agent"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
const bufferSize = 1024 * 1024
var _ agent.AgentServiceServer = (*grpcServer)(nil)
type grpcServer struct {
algo grpc.Handler
data grpc.Handler
@@ -47,39 +56,29 @@ func decodeAlgoRequest(_ context.Context, grpcReq interface{}) (interface{}, err
req := grpcReq.(*agent.AlgoRequest)
return algoReq{
Algorithm: req.Algorithm,
Provider: req.Provider,
Id: req.Id,
Algorithm: req.Algorithm,
Requirements: req.Requirements,
}, nil
}
func encodeAlgoResponse(_ context.Context, response interface{}) (interface{}, error) {
res := response.(algoRes)
return &agent.AlgoResponse{
AlgorithmID: res.AlgorithmID,
}, nil
return &agent.AlgoResponse{}, nil
}
func decodeDataRequest(_ context.Context, grpcReq interface{}) (interface{}, error) {
req := grpcReq.(*agent.DataRequest)
return dataReq{
Dataset: req.Dataset,
Provider: req.Provider,
Id: req.Id,
Dataset: req.Dataset,
}, nil
}
func encodeDataResponse(_ context.Context, response interface{}) (interface{}, error) {
res := response.(dataRes)
return &agent.DataResponse{
DatasetID: res.DatasetID,
}, nil
return &agent.DataResponse{}, nil
}
func decodeResultRequest(_ context.Context, grpcReq interface{}) (interface{}, error) {
req := grpcReq.(*agent.ResultRequest)
return resultReq{Consumer: req.Consumer}, nil
return resultReq{}, nil
}
func encodeResultResponse(_ context.Context, response interface{}) (interface{}, error) {
@@ -90,8 +89,11 @@ func encodeResultResponse(_ context.Context, response interface{}) (interface{},
}
func decodeAttestationRequest(_ context.Context, grpcReq interface{}) (interface{}, error) {
// No fields to extract from gRPC request, so returning an empty struct
return attestationReq{}, nil
req := grpcReq.(*agent.AttestationRequest)
if len(req.ReportData) != agent.ReportDataSize {
return nil, errors.New("malformed report data, expect 64 bytes")
}
return attestationReq{ReportData: [agent.ReportDataSize]byte(req.ReportData)}, nil
}
func encodeAttestationResponse(_ context.Context, response interface{}) (interface{}, error) {
@@ -101,31 +103,75 @@ func encodeAttestationResponse(_ context.Context, response interface{}) (interfa
}, nil
}
func (s *grpcServer) Algo(ctx context.Context, req *agent.AlgoRequest) (*agent.AlgoResponse, error) {
_, res, err := s.algo.ServeGRPC(ctx, req)
// Algo implements agent.AgentServiceServer.
func (s *grpcServer) Algo(stream agent.AgentService_AlgoServer) error {
var algoFile, reqFile []byte
for {
algoChunk, err := stream.Recv()
if err == io.EOF {
break
}
if err != nil {
return status.Error(codes.Internal, err.Error())
}
algoFile = append(algoFile, algoChunk.Algorithm...)
reqFile = append(reqFile, algoChunk.Requirements...)
}
_, res, err := s.algo.ServeGRPC(stream.Context(), &agent.AlgoRequest{Algorithm: algoFile, Requirements: reqFile})
if err != nil {
return nil, err
return err
}
ar := res.(*agent.AlgoResponse)
return ar, nil
return stream.SendAndClose(ar)
}
func (s *grpcServer) Data(ctx context.Context, req *agent.DataRequest) (*agent.DataResponse, error) {
_, res, err := s.data.ServeGRPC(ctx, req)
if err != nil {
return nil, err
// Data implements agent.AgentServiceServer.
func (s *grpcServer) Data(stream agent.AgentService_DataServer) error {
var dataFile []byte
for {
dataChunk, err := stream.Recv()
if err == io.EOF {
break
}
if err != nil {
return status.Error(codes.Internal, err.Error())
}
dataFile = append(dataFile, dataChunk.Dataset...)
}
dr := res.(*agent.DataResponse)
return dr, nil
_, res, err := s.data.ServeGRPC(stream.Context(), &agent.DataRequest{Dataset: dataFile})
if err != nil {
return err
}
ar := res.(*agent.DataResponse)
return stream.SendAndClose(ar)
}
func (s *grpcServer) Result(ctx context.Context, req *agent.ResultRequest) (*agent.ResultResponse, error) {
_, res, err := s.result.ServeGRPC(ctx, req)
func (s *grpcServer) Result(req *agent.ResultRequest, stream agent.AgentService_ResultServer) error {
_, res, err := s.result.ServeGRPC(stream.Context(), req)
if err != nil {
return nil, err
return err
}
rr := res.(*agent.ResultResponse)
return rr, nil
reusltBuffer := bytes.NewBuffer(rr.File)
buf := make([]byte, bufferSize)
for {
n, err := reusltBuffer.Read(buf)
if err == io.EOF {
break
}
if err != nil {
return status.Error(codes.Internal, err.Error())
}
if err := stream.Send(&agent.ResultResponse{File: buf[:n]}); err != nil {
return status.Error(codes.Internal, err.Error())
}
}
return nil
}
func (s *grpcServer) Attestation(ctx context.Context, req *agent.AttestationRequest) (*agent.AttestationResponse, error) {
+9 -22
View File
@@ -9,38 +9,25 @@ package api
import (
"context"
"fmt"
"log/slog"
"time"
mglog "github.com/absmach/magistrala/logger"
"github.com/ultravioletrs/cocos/agent"
)
var _ agent.Service = (*loggingMiddleware)(nil)
type loggingMiddleware struct {
logger mglog.Logger
logger *slog.Logger
svc agent.Service
}
// LoggingMiddleware adds logging facilities to the core service.
func LoggingMiddleware(svc agent.Service, logger mglog.Logger) agent.Service {
func LoggingMiddleware(svc agent.Service, logger *slog.Logger) agent.Service {
return &loggingMiddleware{logger, svc}
}
func (lm *loggingMiddleware) Run(ctx context.Context, ac agent.Computation) (response string, err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method Run for computation %s took %s to complete", ac.ID, time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s.", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors.", message))
}(time.Now())
return lm.svc.Run(ctx, ac)
}
func (lm *loggingMiddleware) Algo(ctx context.Context, algorithm agent.Algorithm) (response string, err error) {
func (lm *loggingMiddleware) Algo(ctx context.Context, algorithm agent.Algorithm) (err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method Algo took %s to complete", time.Since(begin))
if err != nil {
@@ -53,7 +40,7 @@ func (lm *loggingMiddleware) Algo(ctx context.Context, algorithm agent.Algorithm
return lm.svc.Algo(ctx, algorithm)
}
func (lm *loggingMiddleware) Data(ctx context.Context, dataset agent.Dataset) (response string, err error) {
func (lm *loggingMiddleware) Data(ctx context.Context, dataset agent.Dataset) (err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method Data took %s to complete", time.Since(begin))
if err != nil {
@@ -66,7 +53,7 @@ func (lm *loggingMiddleware) Data(ctx context.Context, dataset agent.Dataset) (r
return lm.svc.Data(ctx, dataset)
}
func (lm *loggingMiddleware) Result(ctx context.Context, consumer string) (response []byte, err error) {
func (lm *loggingMiddleware) Result(ctx context.Context) (response []byte, err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method Result took %s to complete", time.Since(begin))
if err != nil {
@@ -76,10 +63,10 @@ func (lm *loggingMiddleware) Result(ctx context.Context, consumer string) (respo
lm.logger.Info(fmt.Sprintf("%s without errors", message))
}(time.Now())
return lm.svc.Result(ctx, consumer)
return lm.svc.Result(ctx)
}
func (lm *loggingMiddleware) Attestation(ctx context.Context) (response []byte, err error) {
func (lm *loggingMiddleware) Attestation(ctx context.Context, reportData [agent.ReportDataSize]byte) (response []byte, err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method Attestation took %s to complete", time.Since(begin))
if err != nil {
@@ -89,5 +76,5 @@ func (lm *loggingMiddleware) Attestation(ctx context.Context) (response []byte,
lm.logger.Info(fmt.Sprintf("%s without errors", message))
}(time.Now())
return lm.svc.Attestation(ctx)
return lm.svc.Attestation(ctx, reportData)
}
+6 -15
View File
@@ -32,16 +32,7 @@ func MetricsMiddleware(svc agent.Service, counter metrics.Counter, latency metri
}
}
func (ms *metricsMiddleware) Run(ctx context.Context, ac agent.Computation) (string, error) {
defer func(begin time.Time) {
ms.counter.With("method", "run").Add(1)
ms.latency.With("method", "run").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.Run(ctx, ac)
}
func (ms *metricsMiddleware) Algo(ctx context.Context, algorithm agent.Algorithm) (string, error) {
func (ms *metricsMiddleware) Algo(ctx context.Context, algorithm agent.Algorithm) error {
defer func(begin time.Time) {
ms.counter.With("method", "algo").Add(1)
ms.latency.With("method", "algo").Observe(time.Since(begin).Seconds())
@@ -50,7 +41,7 @@ func (ms *metricsMiddleware) Algo(ctx context.Context, algorithm agent.Algorithm
return ms.svc.Algo(ctx, algorithm)
}
func (ms *metricsMiddleware) Data(ctx context.Context, dataset agent.Dataset) (string, error) {
func (ms *metricsMiddleware) Data(ctx context.Context, dataset agent.Dataset) error {
defer func(begin time.Time) {
ms.counter.With("method", "data").Add(1)
ms.latency.With("method", "data").Observe(time.Since(begin).Seconds())
@@ -59,20 +50,20 @@ func (ms *metricsMiddleware) Data(ctx context.Context, dataset agent.Dataset) (s
return ms.svc.Data(ctx, dataset)
}
func (ms *metricsMiddleware) Result(ctx context.Context, consumer string) ([]byte, error) {
func (ms *metricsMiddleware) Result(ctx context.Context) ([]byte, error) {
defer func(begin time.Time) {
ms.counter.With("method", "result").Add(1)
ms.latency.With("method", "result").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.Result(ctx, consumer)
return ms.svc.Result(ctx)
}
func (ms *metricsMiddleware) Attestation(ctx context.Context) ([]byte, error) {
func (ms *metricsMiddleware) Attestation(ctx context.Context, reportData [agent.ReportDataSize]byte) ([]byte, error) {
defer func(begin time.Time) {
ms.counter.With("method", "attestation").Add(1)
ms.latency.With("method", "attestation").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.Attestation(ctx)
return ms.svc.Attestation(ctx, reportData)
}
+174
View File
@@ -0,0 +1,174 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package auth
import (
"context"
"crypto"
"crypto/ecdsa"
"crypto/ed25519"
"crypto/rsa"
"crypto/sha256"
"crypto/x509"
"encoding/base64"
"github.com/absmach/magistrala/pkg/errors"
"github.com/ultravioletrs/cocos/agent"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/metadata"
"google.golang.org/grpc/status"
)
type UserRole string
const (
UserMetadataKey = "user-id"
SignatureMetadataKey = "signature"
ConsumerRole UserRole = "consumer"
DataProviderRole UserRole = "data-provider"
AlgorithmProviderRole UserRole = "algorithm-provider"
)
var (
ErrMissingMetadata = errors.New("missing metadata")
ErrInvalidMetadata = errors.New("invalid metadata")
ErrSignatureVerificationFailed = errors.New("signature verification failed")
)
//go:generate mockery --name Authenticator --output=../mocks --filename auth.go --quiet --note "Copyright (c) Ultraviolet \n // SPDX-License-Identifier: Apache-2.0"
type Authenticator interface {
AuthenticateUser(ctx context.Context, role UserRole) (context.Context, error)
}
type service struct {
resultConsumers []interface{}
datasetProviders []interface{}
algorithmProvider interface{}
}
func New(manifest agent.Computation) (Authenticator, error) {
s := &service{}
for _, rc := range manifest.ResultConsumers {
pubKey, err := x509.ParsePKIXPublicKey(rc.UserKey)
if err != nil {
return nil, err
}
pKey, err := decodePublicKey(pubKey)
if err != nil {
return nil, err
}
s.resultConsumers = append(s.resultConsumers, pKey)
}
for _, dp := range manifest.Datasets {
pubKey, err := x509.ParsePKIXPublicKey(dp.UserKey)
if err != nil {
return nil, err
}
pKey, err := decodePublicKey(pubKey)
if err != nil {
return nil, err
}
s.datasetProviders = append(s.datasetProviders, pKey)
}
pubKey, err := x509.ParsePKIXPublicKey(manifest.Algorithm.UserKey)
if err != nil {
return nil, err
}
pKey, err := decodePublicKey(pubKey)
if err != nil {
return nil, err
}
s.algorithmProvider = pKey
return s, nil
}
func extractSignature(md metadata.MD) (string, error) {
signature := md.Get(SignatureMetadataKey)
if len(signature) != 1 {
return "", status.Errorf(codes.Unauthenticated, "invalid metadata")
}
return signature[0], nil
}
func verifySignature(role UserRole, signature string, publicKey any) error {
hash := sha256.Sum256([]byte(role))
sigByte, err := base64.StdEncoding.DecodeString(signature)
if err != nil {
return err
}
var ok bool
switch publicKey := publicKey.(type) {
case *rsa.PublicKey:
if err = rsa.VerifyPKCS1v15(publicKey, crypto.SHA256, hash[:], sigByte); err != nil {
return err
}
return nil
case *ecdsa.PublicKey:
ok = ecdsa.VerifyASN1(publicKey, hash[:], sigByte)
case ed25519.PublicKey:
ok = ed25519.Verify(publicKey, []byte(role), sigByte)
}
if !ok {
return ErrSignatureVerificationFailed
}
return nil
}
func (s *service) AuthenticateUser(ctx context.Context, role UserRole) (context.Context, error) {
md, ok := metadata.FromIncomingContext(ctx)
if !ok {
return nil, ErrMissingMetadata
}
signature, err := extractSignature(md)
if err != nil {
return nil, errors.Wrap(err, ErrInvalidMetadata)
}
switch role {
case ConsumerRole:
for i, rc := range s.resultConsumers {
if err := verifySignature(role, signature, rc); err == nil {
return agent.IndexToContext(ctx, i), nil
}
}
case DataProviderRole:
for i, dp := range s.datasetProviders {
if err := verifySignature(role, signature, dp); err == nil {
return agent.IndexToContext(ctx, i), nil
}
}
case AlgorithmProviderRole:
if err := verifySignature(role, signature, s.algorithmProvider); err == nil {
return ctx, nil
}
}
return ctx, ErrSignatureVerificationFailed
}
func decodePublicKey(key any) (pubKey any, err error) {
switch key := key.(type) {
case *rsa.PublicKey:
return key, nil
case *ecdsa.PublicKey:
return key, nil
case ed25519.PublicKey:
return key, nil
default:
return nil, errors.New("unsupported public key type")
}
}
+158
View File
@@ -0,0 +1,158 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package auth
import (
"context"
"crypto"
"crypto/ecdsa"
"crypto/ed25519"
"crypto/elliptic"
"crypto/rand"
"crypto/rsa"
"crypto/sha256"
"crypto/x509"
"encoding/base64"
"testing"
"github.com/absmach/magistrala/pkg/errors"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/ultravioletrs/cocos/agent"
"google.golang.org/grpc/metadata"
)
func TestAuthenticateUser(t *testing.T) {
resultConsumerKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
require.NoError(t, err)
dataProviderKey, err := rsa.GenerateKey(rand.Reader, 2048)
require.NoError(t, err)
pubEd25519Key, algorithmProviderKey, err := ed25519.GenerateKey(rand.Reader)
require.NoError(t, err)
resultConsumerPubKey, err := x509.MarshalPKIXPublicKey(&resultConsumerKey.PublicKey)
require.NoError(t, err)
dataProviderPubKey, err := x509.MarshalPKIXPublicKey(&dataProviderKey.PublicKey)
require.NoError(t, err)
algorithmProviderPubKey, err := x509.MarshalPKIXPublicKey(pubEd25519Key)
require.NoError(t, err)
manifest := agent.Computation{
ResultConsumers: []agent.ResultConsumer{{UserKey: resultConsumerPubKey}},
Datasets: []agent.Dataset{{UserKey: dataProviderPubKey}},
Algorithm: agent.Algorithm{UserKey: algorithmProviderPubKey},
}
auth, err := New(manifest)
if err != nil {
t.Fatalf("failed to create authenticator: %v", err)
}
testCases := []struct {
name string
role UserRole
key any
expectedErr error
}{
{
name: "valid result consumer",
role: ConsumerRole,
key: resultConsumerKey,
expectedErr: nil,
},
{
name: "valid data provider",
role: DataProviderRole,
key: dataProviderKey,
expectedErr: nil,
},
{
name: "valid algorithm provider",
role: AlgorithmProviderRole,
key: algorithmProviderKey,
expectedErr: nil,
},
{
name: "invalid role",
role: "invalid-role",
key: resultConsumerKey,
expectedErr: ErrSignatureVerificationFailed,
},
{
name: "invalid key",
role: ConsumerRole,
key: dataProviderKey,
expectedErr: ErrSignatureVerificationFailed,
},
{
name: "missing signature",
role: ConsumerRole,
key: resultConsumerKey,
expectedErr: ErrInvalidMetadata,
},
{
name: "missing metadata",
role: ConsumerRole,
key: resultConsumerKey,
expectedErr: ErrMissingMetadata,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
signature, err := signRole(tc.role, tc.key)
if err != nil {
t.Fatalf("failed to sign role: %v", err)
}
ctx := context.Background()
switch tc.name {
case "missing signature":
ctx = metadata.NewIncomingContext(ctx, metadata.Pairs())
case "missing metadata":
default:
ctx = metadata.NewIncomingContext(ctx, metadata.Pairs(SignatureMetadataKey, signature))
}
ctx, err = auth.AuthenticateUser(ctx, tc.role)
assert.True(t, errors.Contains(err, tc.expectedErr), "expected error %v, got %v", tc.expectedErr, err)
if err == nil {
switch id, ok := agent.IndexFromContext(ctx); {
case tc.role == ConsumerRole, tc.role == DataProviderRole:
assert.True(t, ok, "expected index in context")
assert.Equal(t, 0, id, "expected index 0 in context")
default:
assert.False(t, ok, "expected no index in context")
}
}
})
}
}
func signRole(role UserRole, key crypto.PrivateKey) (string, error) {
var signature []byte
var err error
switch k := key.(type) {
case ed25519.PrivateKey:
signature, err = k.Sign(rand.Reader, []byte(role), crypto.Hash(0))
case *rsa.PrivateKey, *ecdsa.PrivateKey:
hash := sha256.Sum256([]byte(role))
signer := key.(crypto.Signer)
signature, err = signer.Sign(rand.Reader, hash[:], crypto.SHA256)
default:
return "", errors.New("unsupported key type")
}
if err != nil {
return "", err
}
return base64.StdEncoding.EncodeToString(signature), nil
}
+39 -38
View File
@@ -3,23 +3,36 @@
package agent
import (
"context"
"encoding/json"
"fmt"
"reflect"
)
var (
_ fmt.Stringer = (*Datasets)(nil)
_ fmt.Stringer = (*Algorithms)(nil)
)
var _ fmt.Stringer = (*Datasets)(nil)
type AgentConfig struct {
LogLevel string `json:"log_level"`
Host string `json:"host"`
Port string `json:"port"`
CertFile string `json:"cert_file"`
KeyFile string `json:"server_key"`
ServerCAFile string `json:"server_ca_file"`
ClientCAFile string `json:"client_ca_file"`
AttestedTls bool `json:"attested_tls"`
}
type Computation struct {
ID string `json:"id,omitempty"`
Name string `json:"name,omitempty"`
Description string `json:"description,omitempty"`
Datasets Datasets `json:"datasets,omitempty"`
Algorithms Algorithms `json:"algorithms,omitempty"`
ResultConsumers []string `json:"result_consumers,omitempty"`
ID string `json:"id,omitempty"`
Name string `json:"name,omitempty"`
Description string `json:"description,omitempty"`
Datasets Datasets `json:"datasets,omitempty"`
Algorithm Algorithm `json:"algorithm,omitempty"`
ResultConsumers []ResultConsumer `json:"result_consumers,omitempty"`
AgentConfig AgentConfig `json:"agent_config,omitempty"`
}
type ResultConsumer struct {
UserKey []byte `json:"user_key,omitempty"`
}
func (d *Datasets) String() string {
@@ -30,40 +43,28 @@ func (d *Datasets) String() string {
return string(dat)
}
func (a *Algorithms) String() string {
dat, err := json.Marshal(a)
if err != nil {
return ""
}
return string(dat)
}
type Dataset struct {
Dataset []byte `json:"-"`
Provider string `json:"provider,omitempty"`
ID string `json:"id,omitempty"`
Dataset []byte `json:"-"`
Hash [32]byte `json:"hash,omitempty"`
UserKey []byte `json:"user_key,omitempty"`
}
type Datasets []Dataset
type Algorithm struct {
Algorithm []byte `json:"-"`
Provider string `json:"provider,omitempty"`
ID string `json:"id,omitempty"`
Algorithm []byte `json:"-"`
Hash [32]byte `json:"hash,omitempty"`
UserKey []byte `json:"user_key,omitempty"`
Requirements []byte `json:"-"`
}
type Algorithms []Algorithm
type ManifestIndexKey struct{}
func containsID(slice interface{}, id string) int {
rangeOnMe := reflect.ValueOf(slice)
for i := 0; i < rangeOnMe.Len(); i++ {
s := rangeOnMe.Index(i)
f := s.FieldByName("ID")
if f.IsValid() {
if f.Interface() == id {
return i
}
}
}
return -1
func IndexToContext(ctx context.Context, index int) context.Context {
return context.WithValue(ctx, ManifestIndexKey{}, index)
}
func IndexFromContext(ctx context.Context) (int, bool) {
index, ok := ctx.Value(ManifestIndexKey{}).(int)
return index, ok
}
+69
View File
@@ -0,0 +1,69 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package events
import (
"encoding/json"
"time"
"github.com/mdlayher/vsock"
"github.com/ultravioletrs/cocos/pkg/manager"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/timestamppb"
)
type service struct {
service string
computationID string
conn *vsock.Conn
}
type AgentEvent struct {
EventType string `json:"event_type"`
Timestamp time.Time `json:"timestamp"`
ComputationID string `json:"computation_id,omitempty"`
Details json.RawMessage `json:"details,omitempty"`
Originator string `json:"originator"`
Status string `json:"status,omitempty"`
}
//go:generate mockery --name Service --output=./mocks --filename events.go --quiet --note "Copyright (c) Ultraviolet \n // SPDX-License-Identifier: Apache-2.0"
type Service interface {
SendEvent(event, status string, details json.RawMessage) error
Close() error
}
func New(svc, computationID string, sockPort uint32) (Service, error) {
conn, err := vsock.Dial(vsock.Host, sockPort, nil)
if err != nil {
return nil, err
}
return &service{
service: svc,
computationID: computationID,
conn: conn,
}, nil
}
func (s *service) SendEvent(event, status string, details json.RawMessage) error {
body := manager.ClientStreamMessage{Message: &manager.ClientStreamMessage_AgentEvent{AgentEvent: &manager.AgentEvent{
EventType: event,
Timestamp: timestamppb.Now(),
ComputationId: s.computationID,
Originator: s.service,
Status: status,
Details: details,
}}}
protoBody, err := proto.Marshal(&body)
if err != nil {
return err
}
if _, err := s.conn.Write(protoBody); err != nil {
return err
}
return nil
}
func (s *service) Close() error {
return s.conn.Close()
}
+67
View File
@@ -0,0 +1,67 @@
// Code generated by mockery v2.43.2. DO NOT EDIT.
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package mocks
import (
json "encoding/json"
mock "github.com/stretchr/testify/mock"
)
// Service is an autogenerated mock type for the Service type
type Service struct {
mock.Mock
}
// Close provides a mock function with given fields:
func (_m *Service) Close() error {
ret := _m.Called()
if len(ret) == 0 {
panic("no return value specified for Close")
}
var r0 error
if rf, ok := ret.Get(0).(func() error); ok {
r0 = rf()
} else {
r0 = ret.Error(0)
}
return r0
}
// SendEvent provides a mock function with given fields: event, status, details
func (_m *Service) SendEvent(event string, status string, details json.RawMessage) error {
ret := _m.Called(event, status, details)
if len(ret) == 0 {
panic("no return value specified for SendEvent")
}
var r0 error
if rf, ok := ret.Get(0).(func(string, string, json.RawMessage) error); ok {
r0 = rf(event, status, details)
} else {
r0 = ret.Error(0)
}
return r0
}
// NewService creates a new instance of Service. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
// The first argument is typically a *testing.T value.
func NewService(t interface {
mock.TestingT
Cleanup(func())
}) *Service {
mock := &Service{}
mock.Mock.Test(t)
t.Cleanup(func() { mock.AssertExpectations(t) })
return mock
}
+129
View File
@@ -0,0 +1,129 @@
// Code generated by mockery v2.42.3. DO NOT EDIT.
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package mocks
import (
context "context"
agent "github.com/ultravioletrs/cocos/agent"
mock "github.com/stretchr/testify/mock"
)
// Service is an autogenerated mock type for the Service type
type Service struct {
mock.Mock
}
// Algo provides a mock function with given fields: ctx, algorithm
func (_m *Service) Algo(ctx context.Context, algorithm agent.Algorithm) error {
ret := _m.Called(ctx, algorithm)
if len(ret) == 0 {
panic("no return value specified for Algo")
}
var r0 error
if rf, ok := ret.Get(0).(func(context.Context, agent.Algorithm) error); ok {
r0 = rf(ctx, algorithm)
} else {
r0 = ret.Error(0)
}
return r0
}
// Attestation provides a mock function with given fields: ctx, reportData
func (_m *Service) Attestation(ctx context.Context, reportData [64]byte) ([]byte, error) {
ret := _m.Called(ctx, reportData)
if len(ret) == 0 {
panic("no return value specified for Attestation")
}
var r0 []byte
var r1 error
if rf, ok := ret.Get(0).(func(context.Context, [64]byte) ([]byte, error)); ok {
return rf(ctx, reportData)
}
if rf, ok := ret.Get(0).(func(context.Context, [64]byte) []byte); ok {
r0 = rf(ctx, reportData)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).([]byte)
}
}
if rf, ok := ret.Get(1).(func(context.Context, [64]byte) error); ok {
r1 = rf(ctx, reportData)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// Data provides a mock function with given fields: ctx, dataset
func (_m *Service) Data(ctx context.Context, dataset agent.Dataset) error {
ret := _m.Called(ctx, dataset)
if len(ret) == 0 {
panic("no return value specified for Data")
}
var r0 error
if rf, ok := ret.Get(0).(func(context.Context, agent.Dataset) error); ok {
r0 = rf(ctx, dataset)
} else {
r0 = ret.Error(0)
}
return r0
}
// Result provides a mock function with given fields: ctx
func (_m *Service) Result(ctx context.Context) ([]byte, error) {
ret := _m.Called(ctx)
if len(ret) == 0 {
panic("no return value specified for Result")
}
var r0 []byte
var r1 error
if rf, ok := ret.Get(0).(func(context.Context) ([]byte, error)); ok {
return rf(ctx)
}
if rf, ok := ret.Get(0).(func(context.Context) []byte); ok {
r0 = rf(ctx)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).([]byte)
}
}
if rf, ok := ret.Get(1).(func(context.Context) error); ok {
r1 = rf(ctx)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// NewService creates a new instance of Service. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
// The first argument is typically a *testing.T value.
func NewService(t interface {
mock.TestingT
Cleanup(func())
}) *Service {
mock := &Service{}
mock.Mock.Test(t)
t.Cleanup(func() { mock.AssertExpectations(t) })
return mock
}
+63
View File
@@ -0,0 +1,63 @@
// Code generated by mockery v2.42.3. DO NOT EDIT.
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package mocks
import (
context "context"
auth "github.com/ultravioletrs/cocos/agent/auth"
mock "github.com/stretchr/testify/mock"
)
// Authenticator is an autogenerated mock type for the Authenticator type
type Authenticator struct {
mock.Mock
}
// AuthenticateUser provides a mock function with given fields: ctx, role
func (_m *Authenticator) AuthenticateUser(ctx context.Context, role auth.UserRole) (context.Context, error) {
ret := _m.Called(ctx, role)
if len(ret) == 0 {
panic("no return value specified for AuthenticateUser")
}
var r0 context.Context
var r1 error
if rf, ok := ret.Get(0).(func(context.Context, auth.UserRole) (context.Context, error)); ok {
return rf(ctx, role)
}
if rf, ok := ret.Get(0).(func(context.Context, auth.UserRole) context.Context); ok {
r0 = rf(ctx, role)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(context.Context)
}
}
if rf, ok := ret.Get(1).(func(context.Context, auth.UserRole) error); ok {
r1 = rf(ctx, role)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// NewAuthenticator creates a new instance of Authenticator. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
// The first argument is typically a *testing.T value.
func NewAuthenticator(t interface {
mock.TestingT
Cleanup(func())
}) *Authenticator {
mock := &Authenticator{}
mock.Mock.Test(t)
t.Cleanup(func() { mock.AssertExpectations(t) })
return mock
}
+139 -150
View File
@@ -5,21 +5,29 @@ package agent
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os/exec"
"log/slog"
"os"
"slices"
mglog "github.com/absmach/magistrala/logger"
"github.com/ultravioletrs/cocos/internal/events"
"github.com/ultravioletrs/cocos/pkg/socket"
"github.com/google/go-sev-guest/client"
"github.com/ultravioletrs/cocos/agent/algorithm"
"github.com/ultravioletrs/cocos/agent/algorithm/binary"
"github.com/ultravioletrs/cocos/agent/algorithm/python"
"github.com/ultravioletrs/cocos/agent/events"
"golang.org/x/crypto/sha3"
)
var _ Service = (*agentService)(nil)
const (
// ReportDataSize is the size of the report data expected by the attestation service.
ReportDataSize = 64
algoFilePermission = 0o700
)
var (
// ErrMalformedEntity indicates malformed entity specification (e.g.
// invalid username or password).
@@ -28,165 +36,178 @@ var (
// when accessing a protected resource.
ErrUnauthorizedAccess = errors.New("missing or invalid credentials provided")
// errUndeclaredAlgorithm indicates algorithm was not declared in computation manifest.
errUndeclaredAlgorithm = errors.New("algorithm not declared in computation manifest")
// errUndeclaredAlgorithm indicates algorithm was not declared in computation manifest.
errUndeclaredDataset = errors.New("dataset not declared in computation manifest")
// errProviderMissmatch algorithm/dataset provider does not match computation manifest.
errProviderMissmatch = errors.New("provider does not match declaration on manifest")
ErrUndeclaredDataset = errors.New("dataset not declared in computation manifest")
// errAllManifestItemsReceived indicates no new computation manifest items expected.
errAllManifestItemsReceived = errors.New("all expected manifest Items have been received")
ErrAllManifestItemsReceived = errors.New("all expected manifest Items have been received")
// errUndeclaredConsumer indicates the consumer requesting results in not declared in computation manifest.
errUndeclaredConsumer = errors.New("result consumer is undeclared in computation manifest")
ErrUndeclaredConsumer = errors.New("result consumer is undeclared in computation manifest")
// errResultsNotReady indicates the computation results are not ready.
errResultsNotReady = errors.New("computation results are not yet ready")
ErrResultsNotReady = errors.New("computation results are not yet ready")
// errStateNotReady agent received a request in the wrong state.
errStateNotReady = errors.New("agent not expecting this operation in the current state")
ErrStateNotReady = errors.New("agent not expecting this operation in the current state")
// errHashMismatch provided algorithm/dataset does not match hash in manifest.
ErrHashMismatch = errors.New("malformed data, hash does not match manifest")
)
// Service specifies an API that must be fullfiled by the domain service
// implementation, and all of its decorators (e.g. logging & metrics).
//
//go:generate mockery --name Service --output=mocks --filename agent.go --quiet --note "Copyright (c) Ultraviolet \n // SPDX-License-Identifier: Apache-2.0"
type Service interface {
Run(ctx context.Context, c Computation) (string, error)
Algo(ctx context.Context, algorithm Algorithm) (string, error)
Data(ctx context.Context, dataset Dataset) (string, error)
Result(ctx context.Context, consumer string) ([]byte, error)
Attestation(ctx context.Context) ([]byte, error)
Algo(ctx context.Context, algorithm Algorithm) error
Data(ctx context.Context, dataset Dataset) error
Result(ctx context.Context) ([]byte, error)
Attestation(ctx context.Context, reportData [ReportDataSize]byte) ([]byte, error)
}
type agentService struct {
computation Computation
algorithms [][]byte
datasets [][]byte
result []byte
attestation []byte
sm *StateMachine
runError error
eventSvc events.Service
computation Computation // Holds the current computation request details.
algorithm algorithm.Algorithm // Filepath to the algorithm received for the computation.
result []byte // Stores the result of the computation.
sm *StateMachine // Manages the state transitions of the agent service.
runError error // Stores any error encountered during the computation run.
eventSvc events.Service // Service for publishing events related to computation.
}
const (
socketPath = "unix_socket"
pyRuntime = "python3"
notificationTopic = "agent"
)
var _ Service = (*agentService)(nil)
// New instantiates the agent service implementation.
func New(ctx context.Context, logger mglog.Logger, eventSvc events.Service) Service {
func New(ctx context.Context, logger *slog.Logger, eventSvc events.Service, cmp Computation) Service {
svc := &agentService{
sm: NewStateMachine(logger),
sm: NewStateMachine(logger, cmp),
eventSvc: eventSvc,
}
go svc.sm.Start(ctx)
svc.sm.SendEvent(start)
svc.sm.StateFunctions[idle] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[receivingManifests] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[receivingAlgorithms] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[receivingManifest] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[receivingAlgorithm] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[receivingData] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[resultsReady] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[complete] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[running] = svc.runComputation
svc.computation = cmp
svc.sm.SendEvent(manifestReceived)
return svc
}
func (as *agentService) Run(ctx context.Context, c Computation) (string, error) {
if as.sm.GetState() != receivingManifests {
return "", errStateNotReady
func (as *agentService) Algo(ctx context.Context, algo Algorithm) error {
if as.sm.GetState() != receivingAlgorithm {
return ErrStateNotReady
}
cmpJSON, err := json.Marshal(c)
if as.algorithm != nil {
return ErrAllManifestItemsReceived
}
hash := sha3.Sum256(algo.Algorithm)
if hash != as.computation.Algorithm.Hash {
return ErrHashMismatch
}
f, err := os.CreateTemp("", "algorithm")
if err != nil {
return "", err
return fmt.Errorf("error creating algorithm file: %v", err)
}
as.computation = c
as.sm.SendEvent(manifestsReceived)
// Calculate the SHA-256 hash of the algorithm
hash := sha256.Sum256(cmpJSON)
cmpHash := hex.EncodeToString(hash[:])
return cmpHash, nil // return computation hash.
}
func (as *agentService) Algo(ctx context.Context, algorithm Algorithm) (string, error) {
if as.sm.GetState() != receivingAlgorithms {
return "", errStateNotReady
if _, err := f.Write(algo.Algorithm); err != nil {
return fmt.Errorf("error writing algorithm to file: %v", err)
}
if len(as.computation.Algorithms) == 0 {
return "", errAllManifestItemsReceived
if err := os.Chmod(f.Name(), algoFilePermission); err != nil {
return fmt.Errorf("error changing file permissions: %v", err)
}
index := containsID(as.computation.Algorithms, algorithm.ID)
switch index {
case -1:
return "", errUndeclaredAlgorithm
default:
if as.computation.Algorithms[index].Provider != algorithm.Provider {
return "", errProviderMissmatch
if err := f.Close(); err != nil {
return fmt.Errorf("error closing file: %v", err)
}
algoType := algorithm.AlgorithmTypeFromContext(ctx)
if algoType == "" {
algoType = string(algorithm.AlgoTypeBin)
}
switch algoType {
case string(algorithm.AlgoTypeBin):
as.algorithm = binary.New(as.sm.logger, as.eventSvc, f.Name())
case string(algorithm.AlgoTypePython):
fr, err := os.CreateTemp("", "requirements.txt")
if err != nil {
return fmt.Errorf("error creating requirments file: %v", err)
}
as.computation.Algorithms = slices.Delete(as.computation.Algorithms, index, index+1)
if _, err := fr.Write(algo.Requirements); err != nil {
return fmt.Errorf("error writing requirements to file: %v", err)
}
if err := fr.Close(); err != nil {
return fmt.Errorf("error closing file: %v", err)
}
runtime := python.PythonRunTimeFromContext(ctx)
as.algorithm = python.New(as.sm.logger, as.eventSvc, runtime, fr.Name(), f.Name())
}
as.algorithms = append(as.algorithms, algorithm.Algorithm)
if len(as.computation.Algorithms) == 0 {
as.sm.SendEvent(algorithmsReceived)
if as.algorithm != nil {
as.sm.SendEvent(algorithmReceived)
}
// Calculate the SHA-256 hash of the algorithm.
hash := sha256.Sum256(algorithm.Algorithm)
algorithmHash := hex.EncodeToString(hash[:])
// Return the algorithm hash or an error.
return algorithmHash, nil
return nil
}
func (as *agentService) Data(ctx context.Context, dataset Dataset) (string, error) {
func (as *agentService) Data(ctx context.Context, dataset Dataset) error {
if as.sm.GetState() != receivingData {
return "", errStateNotReady
return ErrStateNotReady
}
if len(as.computation.Datasets) == 0 {
return "", errAllManifestItemsReceived
}
index := containsID(as.computation.Datasets, dataset.ID)
switch index {
case -1:
return "", errUndeclaredDataset
default:
if as.computation.Datasets[index].Provider != dataset.Provider {
return "", errProviderMissmatch
}
as.computation.Datasets = slices.Delete(as.computation.Datasets, index, index+1)
return ErrAllManifestItemsReceived
}
as.datasets = append(as.datasets, dataset.Dataset)
hash := sha3.Sum256(dataset.Dataset)
index, ok := IndexFromContext(ctx)
if !ok {
return ErrUndeclaredDataset
}
if hash != as.computation.Datasets[index].Hash {
return ErrHashMismatch
}
as.computation.Datasets = slices.Delete(as.computation.Datasets, index, index+1)
f, err := os.CreateTemp("", fmt.Sprintf("dataset-%d", index))
if err != nil {
return fmt.Errorf("error creating dataset file: %v", err)
}
if _, err := f.Write(dataset.Dataset); err != nil {
return fmt.Errorf("error writing dataset to file: %v", err)
}
if err := f.Close(); err != nil {
return fmt.Errorf("error closing file: %v", err)
}
as.algorithm.AddDataset(f.Name())
if len(as.computation.Datasets) == 0 {
as.sm.SendEvent(dataReceived)
}
// Calculate the SHA-256 hash of the dataset.
hash := sha256.Sum256(dataset.Dataset)
datasetHash := hex.EncodeToString(hash[:])
// Return the dataset hash or an error.
return datasetHash, nil
return nil
}
func (as *agentService) Result(ctx context.Context, consumer string) ([]byte, error) {
func (as *agentService) Result(ctx context.Context) ([]byte, error) {
if as.sm.GetState() != resultsReady {
return []byte{}, errResultsNotReady
return []byte{}, ErrResultsNotReady
}
if len(as.computation.ResultConsumers) == 0 {
return []byte{}, errAllManifestItemsReceived
return []byte{}, ErrAllManifestItemsReceived
}
index := slices.Index(as.computation.ResultConsumers, consumer)
switch index {
case -1:
return []byte{}, errUndeclaredConsumer
default:
as.computation.ResultConsumers = slices.Delete(as.computation.ResultConsumers, index, index+1)
index, ok := IndexFromContext(ctx)
if !ok {
return []byte{}, ErrUndeclaredConsumer
}
as.computation.ResultConsumers = slices.Delete(as.computation.ResultConsumers, index, index+1)
if len(as.computation.ResultConsumers) == 0 {
as.sm.SendEvent(resultsConsumed)
@@ -195,14 +216,17 @@ func (as *agentService) Result(ctx context.Context, consumer string) ([]byte, er
return as.result, as.runError
}
func (as *agentService) Attestation(ctx context.Context) ([]byte, error) {
// Implement the logic for the Attestation method here
// Use the provided ctx parameter as needed
var attestation []byte
func (as *agentService) Attestation(ctx context.Context, reportData [ReportDataSize]byte) ([]byte, error) {
provider, err := client.GetQuoteProvider()
if err != nil {
return []byte{}, err
}
rawQuote, err := provider.GetRawQuote(reportData)
if err != nil {
return []byte{}, err
}
as.attestation = attestation
return as.attestation, nil
return rawQuote, nil
}
func (as *agentService) runComputation() {
@@ -210,9 +234,10 @@ func (as *agentService) runComputation() {
as.sm.logger.Debug("computation run started")
defer as.sm.SendEvent(runComplete)
as.publishEvent("in-progress", json.RawMessage{})()
result, err := run(as.algorithms[0], as.datasets[0])
result, err := as.algorithm.Run()
if err != nil {
as.runError = err
as.sm.logger.Warn(fmt.Sprintf("computation failed with error: %s", err.Error()))
as.publishEvent("failed", json.RawMessage{})()
return
}
@@ -222,44 +247,8 @@ func (as *agentService) runComputation() {
func (as *agentService) publishEvent(status string, details json.RawMessage) func() {
return func() {
if err := as.eventSvc.SendEvent(as.sm.State.String(), as.computation.ID, status, details); err != nil {
if err := as.eventSvc.SendEvent(as.sm.State.String(), status, details); err != nil {
as.sm.logger.Warn(err.Error())
}
}
}
func run(algoContent, dataContent []byte) ([]byte, error) {
listener, err := socket.StartUnixSocketServer(socketPath)
if err != nil {
return nil, fmt.Errorf("error creating stdout pipe: %v", err)
}
defer listener.Close()
// Create channels for received data and errors
dataChannel := make(chan []byte)
errorChannel := make(chan error)
var result []byte
go socket.AcceptConnection(listener, dataChannel, errorChannel)
// Construct the Python script content with CSV data as a command-line argument
script := string(algoContent)
data := string(dataContent)
cmd := exec.Command(pyRuntime, "-c", script, data, socketPath)
if err := cmd.Start(); err != nil {
return nil, fmt.Errorf("error starting Python script: %v", err)
}
if err := cmd.Wait(); err != nil {
return nil, fmt.Errorf("python script execution error: %v", err)
}
select {
case result = <-dataChannel:
return result, nil
case err = <-errorChannel:
return nil, fmt.Errorf("error receiving data: %v", err)
}
}
+35 -25
View File
@@ -5,30 +5,29 @@ package agent
import (
"context"
"fmt"
"log/slog"
"sync"
mglog "github.com/absmach/magistrala/logger"
)
//go:generate stringer -type=state
type state int
type state uint8
const (
idle state = iota
receivingManifests
receivingAlgorithms
receivingManifest
receivingAlgorithm
receivingData
running
resultsReady
complete
)
type event int
type event uint8
const (
start event = iota
manifestsReceived
algorithmsReceived
manifestReceived
algorithmReceived
dataReceived
runComplete
resultsConsumed
@@ -41,27 +40,34 @@ type StateMachine struct {
EventChan chan event
Transitions map[state]map[event]state
StateFunctions map[state]func()
logger mglog.Logger
logger *slog.Logger
wg *sync.WaitGroup
}
// NewStateMachine creates a new StateMachine.
func NewStateMachine(logger mglog.Logger) *StateMachine {
func NewStateMachine(logger *slog.Logger, cmp Computation) *StateMachine {
sm := &StateMachine{
State: idle,
EventChan: make(chan event),
Transitions: make(map[state]map[event]state),
StateFunctions: make(map[state]func()),
logger: logger,
wg: &sync.WaitGroup{},
}
sm.Transitions[idle] = make(map[event]state)
sm.Transitions[idle][start] = receivingManifests
sm.Transitions[idle][start] = receivingManifest
sm.Transitions[receivingManifests] = make(map[event]state)
sm.Transitions[receivingManifests][manifestsReceived] = receivingAlgorithms
sm.Transitions[receivingManifest] = make(map[event]state)
sm.Transitions[receivingManifest][manifestReceived] = receivingAlgorithm
sm.Transitions[receivingAlgorithms] = make(map[event]state)
sm.Transitions[receivingAlgorithms][algorithmsReceived] = receivingData
sm.Transitions[receivingAlgorithm] = make(map[event]state)
switch len(cmp.Datasets) {
case 0:
sm.Transitions[receivingAlgorithm][algorithmReceived] = running
default:
sm.Transitions[receivingAlgorithm][algorithmReceived] = receivingData
}
sm.Transitions[receivingData] = make(map[event]state)
sm.Transitions[receivingData][dataReceived] = running
@@ -77,19 +83,24 @@ func NewStateMachine(logger mglog.Logger) *StateMachine {
// Start the state machine.
func (sm *StateMachine) Start(ctx context.Context) {
sm.wg.Add(1)
defer sm.wg.Done()
for {
select {
case event := <-sm.EventChan:
nextState, valid := sm.Transitions[sm.GetState()][event]
sm.mu.Lock()
nextState, valid := sm.Transitions[sm.State][event]
if valid {
sm.mu.Lock()
sm.State = nextState
sm.mu.Unlock()
sm.logger.Debug(fmt.Sprintf("Transition: %v -> %v\n", sm.GetState(), nextState))
sm.logger.Debug(fmt.Sprintf("Transition: %v -> %v\n", sm.State, nextState))
} else {
sm.logger.Error(fmt.Sprintf("Invalid transition: %v -> ???\n", sm.GetState()))
sm.logger.Error(fmt.Sprintf("Invalid transition: %v -> ???\n", sm.State))
}
stateFunc, exists := sm.StateFunctions[sm.GetState()]
sm.mu.Unlock()
sm.mu.Lock()
stateFunc, exists := sm.StateFunctions[sm.State]
sm.mu.Unlock()
if exists {
go stateFunc()
}
@@ -106,13 +117,12 @@ func (sm *StateMachine) SendEvent(event event) {
func (sm *StateMachine) GetState() state {
sm.mu.Lock()
state := sm.State
sm.mu.Unlock()
return state
defer sm.mu.Unlock()
return sm.State
}
func (sm *StateMachine) SetState(state state) {
sm.mu.Lock()
defer sm.mu.Unlock()
sm.State = state
sm.mu.Unlock()
}
+4 -4
View File
@@ -9,17 +9,17 @@ func _() {
// Re-run the stringer command to generate them again.
var x [1]struct{}
_ = x[idle-0]
_ = x[receivingManifests-1]
_ = x[receivingAlgorithms-2]
_ = x[receivingManifest-1]
_ = x[receivingAlgorithm-2]
_ = x[receivingData-3]
_ = x[running-4]
_ = x[resultsReady-5]
_ = x[complete-6]
}
const _state_name = "idlereceivingManifestsreceivingAlgorithmsreceivingDatarunningresultsReadycomplete"
const _state_name = "idlereceivingManifestreceivingAlgorithmreceivingDatarunningresultsReadycomplete"
var _state_index = [...]uint8{0, 4, 22, 41, 54, 61, 73, 81}
var _state_index = [...]uint8{0, 4, 21, 39, 52, 59, 71, 79}
func (i state) String() string {
if i < 0 || i >= state(len(_state_index)-1) {
+27 -18
View File
@@ -10,45 +10,54 @@ import (
mglog "github.com/absmach/magistrala/logger"
)
var cmp = Computation{
Datasets: []Dataset{
{
Dataset: []byte("test"),
UserKey: []byte("test"),
},
},
}
func TestStateMachineTransitions(t *testing.T) {
testCases := []struct {
cases := []struct {
fromState state
event event
expected state
cmp Computation
}{
{idle, start, receivingManifests},
{receivingManifests, manifestsReceived, receivingAlgorithms},
{receivingAlgorithms, algorithmsReceived, receivingData},
{receivingData, dataReceived, running},
{running, runComplete, resultsReady},
{resultsReady, resultsConsumed, complete},
{idle, start, receivingManifest, cmp},
{receivingManifest, manifestReceived, receivingAlgorithm, cmp},
{receivingAlgorithm, algorithmReceived, receivingData, cmp},
{receivingAlgorithm, algorithmReceived, running, Computation{}},
{receivingData, dataReceived, running, cmp},
{running, runComplete, resultsReady, cmp},
{resultsReady, resultsConsumed, complete, cmp},
}
for _, testCase := range testCases {
t.Run(fmt.Sprintf("Transition from %v to %v", testCase.fromState, testCase.expected), func(t *testing.T) {
sm := NewStateMachine(mglog.NewMock())
done := make(chan struct{})
for _, tc := range cases {
t.Run(fmt.Sprintf("Transition from %v to %v", tc.fromState, tc.expected), func(t *testing.T) {
sm := NewStateMachine(mglog.NewMock(), tc.cmp)
ctx, cancel := context.WithCancel(context.Background())
go func() {
sm.Start(ctx)
close(done)
}()
sm.SetState(testCase.fromState)
sm.wg.Wait()
sm.SetState(tc.fromState)
sm.SendEvent(testCase.event)
sm.SendEvent(tc.event)
if sm.GetState() != testCase.expected {
t.Errorf("Expected state %v after the event, but got %v", testCase.expected, sm.GetState())
if sm.GetState() != tc.expected {
t.Errorf("Expected state %v after the event, but got %v", tc.expected, sm.GetState())
}
close(sm.EventChan)
cancel()
<-done
})
}
}
func TestStateMachineInvalidTransition(t *testing.T) {
sm := NewStateMachine(mglog.NewMock())
sm := NewStateMachine(mglog.NewMock(), cmp)
ctx, cancel := context.WithCancel(context.Background())
go sm.Start(ctx)
-124
View File
@@ -1,124 +0,0 @@
openapi: 3.0.1
info:
title: CoCos-AI Manager
description: |
The manager service creates a VM with CoCos-AI agent:
- [The CoCos-AI repository](https://github.com/ultravioletrs/cocos-ai)
contact:
email: info@ultraviolet.com
license:
name: Apache 2.0
url: https://github.com/ultravioletrs/cocos-ai/blob/main/LICENCE.md
version: "1"
servers:
- url: http://localhost:9021
- url: https://localhost:9021
paths:
/run:
post:
summary: Run computation on VM.
description: |
Send computation to agent. This creates agent virtual machine and sends
computation manifest.
requestBody:
$ref: "#/components/requestBodies/Run"
responses:
"200":
description: Computation manifest created successfully.
"400":
description: Request has malformed content.
'500':
$ref: "#/components/responses/ServiceError"
/health:
get:
summary: Retrieves service health check info.
tags:
- health
responses:
'200':
$ref: "#/components/responses/HealthRes"
'500':
$ref: "#/components/responses/ServiceError"
components:
schemas:
HealthRes:
type: object
properties:
status:
type: string
description: Service status.
enum:
- pass
version:
type: string
description: Service version.
example: 0.0.1
commit:
type: string
description: Service commit hash.
example: 7d6f4dc4f7f0c1fa3dc24eddfb18bb5073ff4f62
description:
type: string
description: Service description.
example: <service_name> service
build_time:
type: string
description: Service build time.
example: 1970-01-01_00:00:00
requestBodies:
Run:
required: true
content:
application/json:
schema:
type: object
properties:
computation:
description: byte array of computation request as defined in ComputationReq
type: object
properties:
id:
type: string
name:
type: string
description:
type: string
datasets:
type: array
items:
type: object
properties:
provider:
type: string
id:
type: string
algorithms:
type: array
items:
type: object
properties:
provider:
type: string
id:
type: string
results_consumers:
type: array
items:
type: string
timeout:
type: string
example: "2s"
responses:
ServiceError:
description: Unexpected server-side error occurred.
HealthRes:
description: Service Health Check.
content:
application/json:
schema:
$ref: "#/components/schemas/HealthRes"
+47 -22
View File
@@ -1,6 +1,6 @@
# Agent CLI
This repository contains the command-line interface (CLI) tool for interacting with the Agent service. The CLI allows you to perform various tasks such as running computations, uploading algorithms and datasets, and retrieving results.
This repository contains the command-line interface (CLI) tool for interacting with the Agent and manager service. The CLI allows you to perform various tasks such as running computations, uploading algorithm and datasets, and retrieving results.
## Build
@@ -12,20 +12,57 @@ make cli
## Usage
#### Run Computation
To run a computation, use the following command:
#### Get attestation
Retrieves attestation information from the SEV guest and saves it to a file.
To retrieve attestation from agent, use the following command:
```bash
./build/cocos-cli run --computation '{"name": "my-computation"}'
./build/cocos-cli agent attestation get '<report_data>'
```
#### Validate attestation
Validates the retrieved attestation information against a specified policy and checks its authenticity.
To validate and verify attestation from agent, use the following command:
```bash
./build/cocos-cli agent attestation validate '<attestation>' --report_data '<report_data>'
```
##### Flags
- --config: Path to a JSON file containing the validation configuration. This can be used to override individual flags.
- --report_data: Hex-encoded string representing the attestation report data. (Required for the validate command)
- --host_data: Hex-encoded string representing the expected HOST_DATA field.
- --family_id: Hex-encoded string representing the expected FAMILY_ID field.
- --image_id: Hex-encoded string representing the expected IMAGE_ID field.
- --report_id: Hex-encoded string representing the expected REPORT_ID field.
- --report_id_ma: Hex-encoded string representing the expected REPORT_ID_MA field.
- --measurement: Hex-encoded string representing the expected MEASUREMENT field.
- --chip_id: Hex-encoded string representing the expected CHIP_ID field.
- --minimum_tcb: Minimum acceptable value for CURRENT_TCB, COMMITTED_TCB, and REPORTED_TCB.
- --minimum_launch_tcb: Minimum acceptable value for LAUNCH_TCB.
- --minimum_guest_svn: Minimum acceptable SnpPolicy.
- --minimum_build: Minimum 8-bit build number for AMD-SP firmware.
- --check_crl: Download and check the CRL for revoked certificates (default: false).
- --disallow_network: If true, disallows network downloads for verification (default: false).
- --timeout: Duration to continue retrying failed HTTP requests (default: 2 minutes).
- --max_retry_delay: Maximum duration to wait between HTTP request retries (default: 30 seconds).
- --require_author_key: Require that AUTHOR_KEY_EN is set to 1 (default: false).
- --require_id_block: Require that the VM was launched with an ID_BLOCK signed by a trusted key (default: false).
- --permit_provisional_software: Allow provisional firmware (default: false).
- --platform_info: Maximum acceptable PLATFORM_INFO field bit-wise (optional).
- --minimum_version: Minimum AMD-SP firmware API version (major.minor) (optional).
- --trusted_author_keys: Paths to x.509 certificates of trusted author keys (optional).
- --trusted_author_key_hashes: Hex-encoded SHA-384 hashes of trusted author keys (optional).
- --trusted_id_keys: Paths to x.509 certificates of trusted identity keys (optional).
- --trusted_id_key_hashes: Hex-encoded SHA-384 hashes of trusted identity keys (optional).
- --product: AMD product name for the chip that generated the attestation report (optional).
- --stepping: Machine stepping for the chip that generated the attestation report (optional).
- --CA_bundles_paths: Paths to CA bundles for the AMD product (optional).
- --CA_bundles: PEM format CA bundles for the AMD product (optional).
#### Upload Algorithm
To upload an algorithm, use the following command:
```bash
./build/cocos-cli algo /path/to/algorithm
./build/cocos-cli agent algo /path/to/algorithm <private_key_file_path>
```
#### Upload Dataset
@@ -33,7 +70,7 @@ To upload an algorithm, use the following command:
To upload a dataset, use the following command:
```bash
./build/cocos-cli data /path/to/dataset.csv
./build/cocos-cli agent data /path/to/dataset.csv <private_key_file_path>
```
#### Retrieve result
@@ -41,17 +78,5 @@ To upload a dataset, use the following command:
To retrieve the computation result, use the following command:
```bash
./build/cocos-cli result
```
## Installtion
To use the CLI, you have the option to install it globally on your system. Here's how:
### Build the CLI:
Navigate to the project root and run the following command to build the CLI binary:
```bash
make install-cli
```
./build/cocos-cli agent result <private_key_file_path>
```
+50 -10
View File
@@ -3,19 +3,30 @@
package cli
import (
"context"
"encoding/pem"
"log"
"os"
"github.com/spf13/cobra"
"github.com/ultravioletrs/cocos/agent"
"github.com/ultravioletrs/cocos/agent/algorithm"
"github.com/ultravioletrs/cocos/agent/algorithm/python"
"google.golang.org/grpc/metadata"
)
func NewAlgorithmsCmd(sdk agent.Service) *cobra.Command {
return &cobra.Command{
var (
pythonRuntime string
algoType string
requirementsFile string
)
func (cli *CLI) NewAlgorithmCmd() *cobra.Command {
cmd := &cobra.Command{
Use: "algo",
Short: "Upload an algorithm binary",
Example: "algo <algo_file> <id> <provider>",
Args: cobra.ExactArgs(3),
Example: "algo <algo_file> <private_key_file_path>",
Args: cobra.ExactArgs(2),
Run: func(cmd *cobra.Command, args []string) {
algorithmFile := args[0]
@@ -26,18 +37,47 @@ func NewAlgorithmsCmd(sdk agent.Service) *cobra.Command {
log.Fatalf("Error reading algorithm file: %v", err)
}
var req []byte
if requirementsFile != "" {
req, err = os.ReadFile(requirementsFile)
if err != nil {
log.Fatalf("Error reading requirments file: %v", err)
}
}
algoReq := agent.Algorithm{
Algorithm: algorithm,
ID: args[1],
Provider: args[2],
Algorithm: algorithm,
Requirements: req,
}
response, err := sdk.Algo(cmd.Context(), algoReq)
privKeyFile, err := os.ReadFile(args[1])
if err != nil {
log.Fatalf("Error uploading algorithm with ID %s and provider %s: %v", algoReq.ID, algoReq.Provider, err)
log.Fatalf("Error reading private key file: %v", err)
}
log.Println("Successfully uploaded algorithm:", response)
pemBlock, _ := pem.Decode(privKeyFile)
privKey := decodeKey(pemBlock)
ctx := metadata.NewOutgoingContext(cmd.Context(), metadata.New(make(map[string]string)))
if err := cli.agentSDK.Algo(addAlgoMetadata(ctx), algoReq, privKey); err != nil {
log.Fatalf("Error uploading algorithm with error: %v", err)
}
log.Println("Successfully uploaded algorithm")
},
}
cmd.Flags().StringVarP(&algoType, "algorithm", "a", string(algorithm.AlgoTypeBin), "Algorithm type to run")
cmd.Flags().StringVar(&pythonRuntime, "python-runtime", python.PyRuntime, "Python runtime to use")
cmd.Flags().StringVarP(&requirementsFile, "requirements", "r", "", "Python requirements file")
return cmd
}
func addAlgoMetadata(ctx context.Context) context.Context {
ctx = algorithm.AlgorithmTypeToContext(ctx, algoType)
ctx = python.PythonRunTimeToContext(ctx, pythonRuntime)
return ctx
}
+635 -6
View File
@@ -3,23 +3,168 @@
package cli
import (
"encoding/hex"
"fmt"
"log"
"os"
"strconv"
"strings"
"time"
"github.com/google/go-sev-guest/abi"
"github.com/google/go-sev-guest/proto/check"
"github.com/google/go-sev-guest/proto/sevsnp"
"github.com/google/go-sev-guest/validate"
"github.com/google/go-sev-guest/verify"
"github.com/google/go-sev-guest/verify/trust"
"github.com/spf13/cobra"
"github.com/spf13/pflag"
"github.com/ultravioletrs/cocos/agent"
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/known/wrapperspb"
)
const attestationFilePath = "attestation.txt"
const (
defaultMinimumTcb = 0
defaultMinimumLaunchTcb = 0
defaultMinimumGuestSvn = 0
defaultGuestPolicy = 0x0000000000030000
defaultMinimumBuild = 0
defaultCheckCrl = false
defaultTimeout = 2 * time.Minute
defaultMaxRetryDelay = 30 * time.Second
defaultRequireAuthor = false
defaultRequireIdBlock = false
defaultMinVersion = "0.0"
size16 = 16
size32 = 32
size48 = 48
size64 = 64
attestationFilePath = "attestation.bin"
sevProductNameMilan = "Milan"
sevProductNameGenoa = "Genoa"
exampleJSONConfig = `
{
"rootOfTrust":{
"product":"test_product",
"cabundlePaths":[
"test_cabundlePaths"
],
"cabundles":[
"test_Cabundles"
],
"checkCrl":true,
"disallowNetwork":true
},
"policy":{
"minimumGuestSvn":1,
"policy":"1",
"familyId":"AQIDBAUGBwgJCgsMDQ4PEA==",
"imageId":"AQIDBAUGBwgJCgsMDQ4PEA==",
"vmpl":0,
"minimumTcb":"1",
"minimumLaunchTcb":"1",
"platformInfo":"1",
"requireAuthorKey":true,
"reportData":"J+60aXs8btm8VcGgaJYURGeNCu0FIyWMFXQ7ZUlJDC0FJGJizJsOzDIXgQ75UtPC+Zqe0A3dvnnf5VEeQ61RTg==",
"measurement":"8s78ewoX7Xkfy1qsgVnkZwLDotD768Nqt6qTL5wtQOxHsLczipKM6bhDmWiHLdP4",
"hostData":"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw=",
"reportId":"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw=",
"reportIdMa":"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw=",
"chipId":"J+60aXs8btm8VcGgaJYURGeNCu0FIyWMFXQ7ZUlJDC0FJGJizJsOzDIXgQ75UtPC+Zqe0A3dvnnf5VEeQ61RTg==",
"minimumBuild":1,
"minimumVersion":"0.90",
"permitProvisionalFirmware":true,
"requireIdBlock":true,
"trustedAuthorKeys":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"trustedAuthorKeyHashes":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"trustedIdKeys":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"trustedIdKeyHashes":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"product":{
"name":"SEV_PRODUCT_MILAN",
"stepping":1,
"machineStepping":1
}
}
}
`
)
func NewAttestationCmd(sdk agent.Service) *cobra.Command {
var (
cfg = check.Config{Policy: &check.Policy{}, RootOfTrust: &check.RootOfTrust{}}
cfgString string
timeout time.Duration
maxRetryDelay time.Duration
platformInfo string
stepping string
trustedAuthorKeys []string
trustedAuthorHashes []string
trustedIdKeys []string
trustedIdKeyHashes []string
attestationFile string
attestation []byte
empty16 = [size16]byte{}
empty32 = [size32]byte{}
empty64 = [size64]byte{}
defaultReportIdMa = []byte{255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255}
)
func (cli *CLI) NewAttestationCmd() *cobra.Command {
return &cobra.Command{
Use: "attestation",
Short: "Retrieve attestation information",
Use: "attestation [command]",
Short: "Get and validate attestations",
Run: func(cmd *cobra.Command, args []string) {
log.Println("Checking attestation")
fmt.Printf("Get and validate attestations\n\n")
fmt.Printf("Usage:\n %s [command]\n\n", cmd.CommandPath())
fmt.Printf("Available Commands:\n")
result, err := sdk.Attestation(cmd.Context())
// Filter out "completion" command
availableCommands := make([]*cobra.Command, 0)
for _, subCmd := range cmd.Commands() {
if subCmd.Name() != "completion" {
availableCommands = append(availableCommands, subCmd)
}
}
for _, subCmd := range availableCommands {
fmt.Printf(" %-15s%s\n", subCmd.Name(), subCmd.Short)
}
fmt.Printf("\nFlags:\n")
cmd.Flags().VisitAll(func(flag *pflag.Flag) {
fmt.Printf(" -%s, --%s %s\n", flag.Shorthand, flag.Name, flag.Usage)
})
fmt.Printf("\nUse \"%s [command] --help\" for more information about a command.\n", cmd.CommandPath())
},
}
}
func (cli *CLI) NewGetAttestationCmd() *cobra.Command {
return &cobra.Command{
Use: "get",
Short: "Retrieve attestation information from agent. Report data expected in hex enoded string of length 64 bytes.",
Example: "get <report_data>",
Args: cobra.ExactArgs(1),
Run: func(cmd *cobra.Command, args []string) {
log.Println("Getting attestation")
reportData, err := hex.DecodeString(args[0])
if err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
if len(reportData) != agent.ReportDataSize {
log.Fatalf("report data must be a hex encoded string of length %d bytes", agent.ReportDataSize)
}
result, err := cli.agentSDK.Attestation(cmd.Context(), [agent.ReportDataSize]byte(reportData))
if err != nil {
log.Fatalf("Error retrieving attestation: %v", err)
}
@@ -32,3 +177,487 @@ func NewAttestationCmd(sdk agent.Service) *cobra.Command {
},
}
}
func (cli *CLI) NewValidateAttestationValidationCmd() *cobra.Command {
cmd := &cobra.Command{
Use: "validate",
Short: "Validate and verify attestation information. The report is provided as a file path.",
Example: "validate <attestation_report_file_path>",
Args: cobra.ExactArgs(1),
Run: func(cmd *cobra.Command, args []string) {
log.Println("Checking attestation")
attestationFile = string(args[0])
if err := parseConfig(); err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
if err := parseHashes(); err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
if err := parseFiles(); err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
// This format is the attestation report in AMD's specified ABI format, immediately
// followed by the certificate table bytes.
if len(attestation) < abi.ReportSize {
log.Fatalf("attestation contents too small (0x%x bytes). Want at least 0x%x bytes", len(attestation), abi.ReportSize)
}
if err := parseUints(); err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
cfg.Policy.Vmpl = wrapperspb.UInt32(0)
if err := validateInput(); err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
if err := verifyAndValidateAttestation(attestation); err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
log.Println("Attestation validation and verification is successful!")
},
}
cmd.Flags().StringVar(
&cfgString,
"config",
"",
"Serialized json check.Config protobuf. This will overwrite individual flags. Unmarshalled as json. Example: "+exampleJSONConfig,
)
cmd.Flags().BytesHexVar(
&cfg.Policy.HostData,
"host_data",
empty32[:],
"The expected HOST_DATA field as a hex string. Must encode 32 bytes. Unchecked if unset.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.ReportData,
"report_data",
empty64[:],
"The expected REPORT_DATA field as a hex string. Must encode 64 bytes. Must be set.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.FamilyId,
"family_id",
empty16[:],
"The expected FAMILY_ID field as a hex string. Must encode 16 bytes. Unchecked if unset.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.ImageId,
"image_id",
empty16[:],
"The expected IMAGE_ID field as a hex string. Must encode 16 bytes. Unchecked if unset.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.ReportId,
"report_id",
nil,
"The expected REPORT_ID field as a hex string. Must encode 32 bytes. Unchecked if unset.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.ReportIdMa,
"report_id_ma",
defaultReportIdMa,
"The expected REPORT_ID_MA field as a hex string. Must encode 32 bytes. Unchecked if unset.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.Measurement,
"measurement",
nil,
"The expected MEASUREMENT field as a hex string. Must encode 48 bytes. Unchecked if unset.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.ChipId,
"chip_id",
nil,
"The expected MEASUREMENT field as a hex string. Must encode 48 bytes. Unchecked if unset.",
)
cmd.Flags().Uint64Var(
&cfg.Policy.MinimumTcb,
"minimum_tcb",
defaultMinimumTcb,
"The minimum acceptable value for CURRENT_TCB, COMMITTED_TCB, and REPORTED_TCB.",
)
cmd.Flags().Uint64Var(
&cfg.Policy.MinimumLaunchTcb,
"minimum_lauch_tcb",
defaultMinimumLaunchTcb,
"The minimum acceptable value for LAUNCH_TCB.",
)
cmd.Flags().Uint64Var(
&cfg.Policy.Policy,
"guest_policy",
defaultGuestPolicy,
"The most acceptable guest SnpPolicy.",
)
cmd.Flags().Uint32Var(
&cfg.Policy.MinimumGuestSvn,
"minimum_guest_svn",
defaultMinimumGuestSvn,
"The most acceptable GUEST_SVN.",
)
cmd.Flags().Uint32Var(
&cfg.Policy.MinimumBuild,
"minimum_build",
defaultMinimumBuild,
"The 8-bit minimum build number for AMD-SP firmware",
)
cmd.Flags().BoolVar(
&cfg.RootOfTrust.CheckCrl,
"check_crl",
defaultCheckCrl,
"Download and check the CRL for revoked certificates.",
)
cmd.Flags().DurationVar(
&timeout,
"timeout",
defaultTimeout,
"Duration to continue to retry failed HTTP requests.",
)
cmd.Flags().DurationVar(
&maxRetryDelay,
"max_retry_delay",
defaultMaxRetryDelay,
"Maximum Duration to wait between HTTP request retries.",
)
cmd.Flags().BoolVar(
&cfg.Policy.RequireAuthorKey,
"require_author_key",
defaultRequireAuthor,
"Require that AUTHOR_KEY_EN is 1.",
)
cmd.Flags().BoolVar(
&cfg.Policy.RequireIdBlock,
"require_id_block",
defaultRequireIdBlock,
"Require that the VM was launch with an ID_BLOCK signed by a trusted id key or author key",
)
cmd.Flags().StringVar(
&platformInfo,
"platform_info",
"",
"The maximum acceptable PLATFORM_INFO field bit-wise. May be empty or a 64-bit unsigned integer",
)
cmd.Flags().StringVar(
&cfg.Policy.MinimumVersion,
"minimum_version",
defaultMinVersion,
"Minimum AMD-SP firmware API version (major.minor). Each number must be 8-bit non-negative.",
)
cmd.Flags().StringArrayVar(
&trustedAuthorKeys,
"trusted_author_keys",
[]string{},
"Paths to x.509 certificates of trusted author keys",
)
cmd.Flags().StringArrayVar(
&trustedAuthorHashes,
"trusted_author_key_hashes",
[]string{},
"Hex-encoded SHA-384 hash values of trusted author keys in AMD public key format",
)
cmd.Flags().StringArrayVar(
&trustedIdKeys,
"trusted_id_keys",
[]string{},
"Paths to x.509 certificates of trusted author keys",
)
cmd.Flags().StringArrayVar(
&trustedIdKeyHashes,
"trusted_id_key_hashes",
[]string{},
"Hex-encoded SHA-384 hash values of trusted identity keys in AMD public key format",
)
cmd.Flags().StringVar(
&cfg.RootOfTrust.ProductLine,
"product",
"",
"The AMD product name for the chip that generated the attestation report.",
)
cmd.Flags().StringVar(
&stepping,
"stepping",
"",
"The machine stepping for the chip that generated the attestation report. Default unchecked.",
)
cmd.Flags().StringArrayVar(
&cfg.RootOfTrust.CabundlePaths,
"CA_bundles_paths",
[]string{},
"Paths to CA bundles for the AMD product. Must be in PEM format, ASK, then ARK certificates. If unset, uses embedded root certificates.",
)
cmd.Flags().StringArrayVar(
&cfg.RootOfTrust.Cabundles,
"CA_bundles",
[]string{},
"PEM format CA bundles for the AMD product. Combined with contents of cabundle_paths.",
)
if err := cmd.MarkFlagRequired("report_data"); err != nil {
log.Fatalf("Failed to mark flag as required: %s", err)
}
if err := cmd.MarkFlagRequired("product"); err != nil {
log.Fatalf("Failed to mark flag as required: %s", err)
}
return cmd
}
func verifyAndValidateAttestation(attestation []byte) error {
sopts, err := verify.RootOfTrustToOptions(cfg.RootOfTrust)
if err != nil {
return err
}
if cfg.Policy.Product == nil {
productName := sevsnp.SevProduct_SEV_PRODUCT_UNKNOWN
switch cfg.RootOfTrust.ProductLine {
case sevProductNameMilan:
productName = sevsnp.SevProduct_SEV_PRODUCT_MILAN
case sevProductNameGenoa:
productName = sevsnp.SevProduct_SEV_PRODUCT_GENOA
default:
}
if productName == sevsnp.SevProduct_SEV_PRODUCT_UNKNOWN {
return fmt.Errorf("product name must be %s or %s", sevProductNameMilan, sevProductNameGenoa)
}
sopts.Product = &sevsnp.SevProduct{
Name: productName,
}
} else {
sopts.Product = cfg.Policy.Product
}
sopts.Getter = &trust.RetryHTTPSGetter{
Timeout: timeout,
MaxRetryDelay: maxRetryDelay,
Getter: &trust.SimpleHTTPSGetter{},
}
// Only take the attestation report and ignore everything else.
attestationPB, err := abi.ReportCertsToProto(attestation[:abi.ReportSize])
if err != nil {
return err
}
if err = verify.SnpAttestation(attestationPB, sopts); err != nil {
return err
}
opts, err := validate.PolicyToOptions(cfg.Policy)
if err != nil {
return err
}
if err = validate.SnpAttestation(attestationPB, opts); err != nil {
return err
}
return nil
}
// parseConfig decodes config passed as json for check.Config struct.
// example
/* {
"rootOfTrust":{
"product":"test_product",
"cabundlePaths":[
"test_cabundlePaths"
],
"cabundles":[
"test_Cabundles"
],
"checkCrl":true,
"disallowNetwork":true
},
"policy":{
"minimumGuestSvn":1,
"policy":"1",
"familyId":"AQIDBAUGBwgJCgsMDQ4PEA==",
"imageId":"AQIDBAUGBwgJCgsMDQ4PEA==",
"vmpl":0,
"minimumTcb":"1",
"minimumLaunchTcb":"1",
"platformInfo":"1",
"requireAuthorKey":true,
"reportData":"J+60aXs8btm8VcGgaJYURGeNCu0FIyWMFXQ7ZUlJDC0FJGJizJsOzDIXgQ75UtPC+Zqe0A3dvnnf5VEeQ61RTg==",
"measurement":"8s78ewoX7Xkfy1qsgVnkZwLDotD768Nqt6qTL5wtQOxHsLczipKM6bhDmWiHLdP4",
"hostData":"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw=",
"reportId":"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw=",
"reportIdMa":"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw=",
"chipId":"J+60aXs8btm8VcGgaJYURGeNCu0FIyWMFXQ7ZUlJDC0FJGJizJsOzDIXgQ75UtPC+Zqe0A3dvnnf5VEeQ61RTg==",
"minimumBuild":1,
"minimumVersion":"0.90",
"permitProvisionalFirmware":true,
"requireIdBlock":true,
"trustedAuthorKeys":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"trustedAuthorKeyHashes":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"trustedIdKeys":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"trustedIdKeyHashes":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"product":{
"name":"SEV_PRODUCT_MILAN",
"stepping":1,
"machineStepping":1
}
}
}*/
func parseConfig() error {
if cfgString == "" {
return nil
}
if err := protojson.Unmarshal([]byte(cfgString), &cfg); err != nil {
return err
}
// Populate fields that should not be nil
if cfg.RootOfTrust == nil {
cfg.RootOfTrust = &check.RootOfTrust{}
}
if cfg.Policy == nil {
cfg.Policy = &check.Policy{}
}
return nil
}
func parseHashes() error {
for _, hash := range trustedAuthorHashes {
hashBytes, err := hex.DecodeString(hash)
if err != nil {
return err
}
cfg.Policy.TrustedAuthorKeyHashes = append(cfg.Policy.TrustedAuthorKeyHashes, hashBytes)
}
for _, hash := range trustedIdKeyHashes {
hashBytes, err := hex.DecodeString(hash)
if err != nil {
return err
}
cfg.Policy.TrustedIdKeyHashes = append(cfg.Policy.TrustedIdKeyHashes, hashBytes)
}
return nil
}
func parseFiles() error {
file, err := os.ReadFile(attestationFile)
if err != nil {
return err
}
attestation = file
for _, path := range trustedAuthorKeys {
file, err := os.ReadFile(path)
if err != nil {
return err
}
cfg.Policy.TrustedAuthorKeys = append(cfg.Policy.TrustedAuthorKeys, file)
}
for _, path := range trustedIdKeys {
file, err := os.ReadFile(path)
if err != nil {
return err
}
cfg.Policy.TrustedIdKeys = append(cfg.Policy.TrustedIdKeys, file)
}
return nil
}
func parseUints() error {
if stepping != "" {
if base := getBase(stepping); base == 10 {
num, err := strconv.ParseUint(stepping, getBase(stepping), 8)
if err != nil {
return err
}
cfg.Policy.Product.MachineStepping = wrapperspb.UInt32(uint32(num))
} else {
num, err := strconv.ParseUint(stepping[2:], base, 8)
if err != nil {
return err
}
cfg.Policy.Product.MachineStepping = wrapperspb.UInt32(uint32(num))
}
}
if platformInfo != "" {
if base := getBase(platformInfo); base == 10 {
num, err := strconv.ParseUint(platformInfo, getBase(platformInfo), 8)
if err != nil {
return err
}
cfg.Policy.PlatformInfo = wrapperspb.UInt64(num)
} else {
num, err := strconv.ParseUint(platformInfo[2:], base, 8)
if err != nil {
return err
}
cfg.Policy.PlatformInfo = wrapperspb.UInt64(num)
}
}
return nil
}
func getBase(val string) int {
switch {
case strings.HasPrefix(val, "0x"):
return 16
case strings.HasPrefix(val, "0o"):
return 8
case strings.HasPrefix(val, "0b"):
return 2
default:
return 10
}
}
func validateInput() error {
if len(cfg.RootOfTrust.CabundlePaths) != 0 || len(cfg.RootOfTrust.Cabundles) != 0 && cfg.RootOfTrust.Product == "" {
return fmt.Errorf("product name must be set if CA bundles are provided")
}
if err := validateFieldLength("report_data", cfg.Policy.ReportData, size64); err != nil {
return err
}
if err := validateFieldLength("host_data", cfg.Policy.HostData, size32); err != nil {
return err
}
if err := validateFieldLength("family_id", cfg.Policy.FamilyId, size16); err != nil {
return err
}
if err := validateFieldLength("image_id", cfg.Policy.ImageId, size16); err != nil {
return err
}
if err := validateFieldLength("report_id", cfg.Policy.ReportId, size32); err != nil {
return err
}
if err := validateFieldLength("report_id_ma", cfg.Policy.ReportIdMa, size32); err != nil {
return err
}
if err := validateFieldLength("measurement", cfg.Policy.Measurement, size48); err != nil {
return err
}
if err := validateFieldLength("chip_id", cfg.Policy.ChipId, size64); err != nil {
return err
}
for _, hash := range cfg.Policy.TrustedAuthorKeyHashes {
if err := validateFieldLength("trusted_author_key_hash", hash, size48); err != nil {
return err
}
}
for _, hash := range cfg.Policy.TrustedIdKeyHashes {
if err := validateFieldLength("trusted_id_key_hash", hash, size48); err != nil {
return err
}
}
return nil
}
func validateFieldLength(fieldName string, field []byte, expectedLength int) error {
if field != nil && len(field) != expectedLength {
return fmt.Errorf("%s length should be at least %d bytes long", fieldName, expectedLength)
}
return nil
}
+144
View File
@@ -0,0 +1,144 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package cli
import (
"encoding/base64"
"encoding/json"
"fmt"
"log"
"os"
"github.com/absmach/magistrala/pkg/errors"
"github.com/google/go-sev-guest/proto/check"
"github.com/spf13/cobra"
"github.com/spf13/pflag"
)
type fieldType int
const (
measurementField fieldType = iota
hostDataField
)
const (
// 0o744 file permission gives RWX permission to the user and only the R permission to others.
filePermission = 0o744
// Length of the expected host data and measurement field in bytes.
hostDataLength = 32
measurementLength = 48
)
var (
errDecode = errors.New("base64 string could not be decoded")
errDataLength = errors.New("data does not have an adequate length")
errReadingBackendInfoFile = errors.New("error while reading the backend information file")
errUnmarshalJSON = errors.New("failed to unmarshal json")
errMarshalJSON = errors.New("failed to marshal json")
errWriteFile = errors.New("failed to write to file")
errBackendField = errors.New("the specified field type does not exist in the backend information")
)
type AttestationConfiguration struct {
SNPPolicy *check.Policy `json:"snp_policy,omitempty"`
RootOfTrust *check.RootOfTrust `json:"root_of_trust,omitempty"`
}
func (cli *CLI) NewBackendCmd() *cobra.Command {
return &cobra.Command{
Use: "backend [command]",
Short: "Change backend information",
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("Change backend information\n\n")
fmt.Printf("Usage:\n %s [command]\n\n", cmd.CommandPath())
fmt.Printf("Available Commands:\n")
// Filter out "completion" command
availableCommands := make([]*cobra.Command, 0)
for _, subCmd := range cmd.Commands() {
if subCmd.Name() != "completion" {
availableCommands = append(availableCommands, subCmd)
}
}
for _, subCmd := range availableCommands {
fmt.Printf(" %-15s%s\n", subCmd.Name(), subCmd.Short)
}
fmt.Printf("\nFlags:\n")
cmd.Flags().VisitAll(func(flag *pflag.Flag) {
fmt.Printf(" -%s, --%s %s\n", flag.Shorthand, flag.Name, flag.Usage)
})
fmt.Printf("\nUse \"%s [command] --help\" for more information about a command.\n", cmd.CommandPath())
},
}
}
func (cli *CLI) NewAddMeasurementCmd() *cobra.Command {
return &cobra.Command{
Use: "measurement",
Short: "Add measurement to the backend info file. The value should be in base64. The second parameter is backend_info.json file",
Example: "measurement <measurement> <backend_info.json>",
Args: cobra.ExactArgs(2),
Run: func(cmd *cobra.Command, args []string) {
if err := changeAttestationConfiguration(args[1], args[0], measurementLength, measurementField); err != nil {
log.Fatalf("Error could not change measurement data %v", err)
}
},
}
}
func (cli *CLI) NewAddHostDataCmd() *cobra.Command {
return &cobra.Command{
Use: "hostdata",
Short: "Add host data to the backend info file. The value should be in base64. The second parameter is backend_info.json file",
Example: "hostdata <host-data> <backend_info.json>",
Args: cobra.ExactArgs(2),
Run: func(cmd *cobra.Command, args []string) {
if err := changeAttestationConfiguration(args[1], args[0], hostDataLength, hostDataField); err != nil {
log.Fatalf("Error could not change host data %v", err)
}
},
}
}
func changeAttestationConfiguration(fileName, base64Data string, expectedLength int, field fieldType) error {
data, err := base64.StdEncoding.DecodeString(base64Data)
if err != nil {
return errDecode
}
if len(data) != expectedLength {
return errDataLength
}
ac := AttestationConfiguration{}
backendInfo, err := os.ReadFile(fileName)
if err != nil {
return errors.Wrap(errReadingBackendInfoFile, err)
}
if err = json.Unmarshal(backendInfo, &ac); err != nil {
return errors.Wrap(errUnmarshalJSON, err)
}
switch field {
case measurementField:
ac.SNPPolicy.Measurement = data
case hostDataField:
ac.SNPPolicy.HostData = data
default:
return errBackendField
}
fileJson, err := json.MarshalIndent(ac, "", " ")
if err != nil {
return errors.Wrap(errMarshalJSON, err)
}
if err = os.WriteFile(fileName, fileJson, filePermission); err != nil {
return errors.Wrap(errWriteFile, err)
}
return nil
}
+69
View File
@@ -0,0 +1,69 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package cli
import (
"log"
"os"
"path"
"github.com/google/go-sev-guest/abi"
"github.com/google/go-sev-guest/kds"
"github.com/google/go-sev-guest/verify/trust"
"github.com/spf13/cobra"
"github.com/ultravioletrs/cocos/pkg/clients/grpc"
)
const (
caBundleName = "ask_ark.pem"
filePermisionKeys = 0o766
)
func (cli *CLI) NewCABundleCmd(fileSavePath string) *cobra.Command {
return &cobra.Command{
Use: "ca-bundle",
Short: "Fetch AMD SEV-SNPs CA Bundle (ASK and ARK)",
Example: "ca-bundle <path_to_platform_info_json>",
Args: cobra.ExactArgs(1),
Run: func(cmd *cobra.Command, args []string) {
attestationConfiguration := grpc.AttestationConfiguration{}
err := grpc.ReadManifest(args[0], &attestationConfiguration)
if err != nil {
log.Fatalf("Error while reading manifest: %v", err)
}
product := attestationConfiguration.RootOfTrust.Product
getter := trust.DefaultHTTPSGetter()
caURL := kds.ProductCertChainURL(abi.VcekReportSigner, product)
bundle, err := getter.Get(caURL)
if err != nil {
log.Fatalf("Error fetching ARK and ASK from AMD KDS for product: %s, error: %v", product, err)
}
err = os.MkdirAll(path.Join(fileSavePath, product), filePermisionKeys)
if err != nil {
log.Fatalf("Error while creating directory for product name %s, error: %v", product, err)
}
bundleFilePath := path.Join(fileSavePath, product, caBundleName)
if err = saveToFile(bundleFilePath, bundle); err != nil {
log.Fatalf("Error while saving ARK-ASK to file: %v", err)
}
},
}
}
func saveToFile(fileSavePath string, content []byte) error {
file, err := os.OpenFile(fileSavePath, os.O_CREATE|os.O_RDWR|os.O_TRUNC, filePermisionKeys)
if err != nil {
return err
}
if _, err := file.Write(content); err != nil {
return err
}
return nil
}
+39 -8
View File
@@ -3,6 +3,8 @@
package cli
import (
"crypto/x509"
"encoding/pem"
"log"
"os"
@@ -10,12 +12,12 @@ import (
"github.com/ultravioletrs/cocos/agent"
)
func NewDatasetsCmd(sdk agent.Service) *cobra.Command {
func (cli *CLI) NewDatasetsCmd() *cobra.Command {
return &cobra.Command{
Use: "data",
Short: "Upload a dataset CSV file",
Example: "data <dataset.csv> <id> <provider>",
Args: cobra.ExactArgs(3),
Example: "data <dataset.csv> <private_key_file_path>",
Args: cobra.ExactArgs(2),
Run: func(cmd *cobra.Command, args []string) {
datasetFile := args[0]
@@ -27,17 +29,46 @@ func NewDatasetsCmd(sdk agent.Service) *cobra.Command {
}
dataReq := agent.Dataset{
Dataset: dataset,
ID: args[1],
Provider: args[2],
Dataset: dataset,
}
response, err := sdk.Data(cmd.Context(), dataReq)
privKeyFile, err := os.ReadFile(args[1])
if err != nil {
log.Fatalf("Error reading private key file: %v", err)
}
pemBlock, _ := pem.Decode(privKeyFile)
privKey := decodeKey(pemBlock)
if err := cli.agentSDK.Data(cmd.Context(), dataReq, privKey); err != nil {
log.Fatalf("Error uploading dataset: %v", err)
}
log.Println("Response:", response)
log.Println("Successfully uploaded dataset")
},
}
}
func decodeKey(b *pem.Block) interface{} {
switch b.Type {
case rsaKeyType:
privKey, err := x509.ParsePKCS8PrivateKey(b.Bytes)
if err != nil {
privKey, err = x509.ParsePKCS1PrivateKey(b.Bytes)
if err != nil {
log.Fatalf("Error parsing private key: %v", err)
}
}
return privKey
case ecdsaKeyType:
privKey, err := x509.ParseECPrivateKey(b.Bytes)
if err != nil {
log.Fatalf("Error parsing private key: %v", err)
}
return privKey
default:
log.Fatalf("Error decoding key")
return nil
}
}
+35
View File
@@ -0,0 +1,35 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package cli
import (
"encoding/hex"
"log"
"os"
"github.com/spf13/cobra"
"golang.org/x/crypto/sha3"
)
func (cli *CLI) NewFileHashCmd() *cobra.Command {
return &cobra.Command{
Use: "file-hash",
Short: "Compute the sha3-256 hash of a file",
Example: "file-hash <file>",
Args: cobra.ExactArgs(1),
Run: func(cmd *cobra.Command, args []string) {
fileName := args[0]
file, err := os.ReadFile(fileName)
if err != nil {
log.Fatalf("Error reading dataset file: %v", err)
}
hashBytes := sha3.Sum256(file)
hash := hex.EncodeToString(hashBytes[:])
log.Println("Hash of file:", hash)
},
}
}
+128
View File
@@ -0,0 +1,128 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package cli
import (
"crypto/ecdsa"
"crypto/ed25519"
"crypto/elliptic"
"crypto/rand"
"crypto/rsa"
"crypto/x509"
"encoding/pem"
"log"
"os"
"reflect"
"github.com/spf13/cobra"
)
const (
keyBitSize = 4096
rsaKeyType = "PRIVATE KEY"
ecdsaKeyType = "EC PRIVATE KEY"
ed25519KeyType = "PRIVATE KEY"
publicKeyType = "PUBLIC KEY"
publicKeyFile = "public.pem"
privateKeyFile = "private.pem"
)
var KeyType string
func (cli *CLI) NewKeysCmd() *cobra.Command {
return &cobra.Command{
Use: "keys",
Short: "Generate a new public/private key pair",
Long: "Generates a new public/private key pair using an algorithm of the users choice.\n" +
"Supported algorithms are RSA, ecdsa, and ed25519.",
Example: "./build/cocos-cli keys -k rsa",
Args: cobra.ExactArgs(0),
Run: func(cmd *cobra.Command, args []string) {
switch KeyType {
case "ecdsa":
privEcdsaKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
log.Fatalf("Error generating keys: %v", err)
}
pubKeyBytes, err := x509.MarshalPKIXPublicKey(&privEcdsaKey.PublicKey)
if err != nil {
log.Fatalf("Error marshalling public key: %v", err)
}
generateAndWriteKeys(privEcdsaKey, pubKeyBytes, ecdsaKeyType)
case "ed25519":
pubEd25519Key, privEd25519Key, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
log.Fatalf("Error generating keys: %v", err)
}
pubKey, err := x509.MarshalPKIXPublicKey(pubEd25519Key)
if err != nil {
log.Fatalf("Error marshalling public key: %v", err)
}
generateAndWriteKeys(privEd25519Key, pubKey, ed25519KeyType)
// Default to RSA
default:
privKey, err := rsa.GenerateKey(rand.Reader, keyBitSize)
if err != nil {
log.Fatalf("Error generating keys: %v", err)
}
pubKeyBytes, err := x509.MarshalPKIXPublicKey(&privKey.PublicKey)
if err != nil {
log.Fatalf("Error marshalling public key: %v", err)
}
generateAndWriteKeys(privKey, pubKeyBytes, rsaKeyType)
}
},
}
}
func generateAndWriteKeys(privKey interface{}, pubKeyBytes []byte, keyType string) {
privFile, err := os.Create(privateKeyFile)
if err != nil {
log.Fatalf("Error creating private key file: %v", err)
}
defer privFile.Close()
var b []byte
switch privKey := privKey.(type) {
case *rsa.PrivateKey:
b = x509.MarshalPKCS1PrivateKey(privKey)
case *ecdsa.PrivateKey:
b, err = x509.MarshalECPrivateKey(privKey)
case ed25519.PrivateKey:
b, err = x509.MarshalPKCS8PrivateKey(privKey)
}
if err != nil {
log.Printf("Error marshalling private key: %v", err)
return
}
if err := pem.Encode(privFile, &pem.Block{
Type: keyType,
Bytes: b,
}); err != nil {
log.Printf("Error encoding private key: %v", err)
return
}
pubFile, err := os.Create(publicKeyFile)
if err != nil {
log.Printf("Error creating public key file: %v", err)
return
}
defer pubFile.Close()
if err := pem.Encode(pubFile, &pem.Block{
Type: publicKeyType,
Bytes: pubKeyBytes,
}); err != nil {
log.Printf("Error encoding public key: %v", err)
return
}
log.Printf("Successfully generated public/private key pair of type: %s", reflect.TypeOf(privKey).String())
}
-101
View File
@@ -1,101 +0,0 @@
openapi: 3.1.0
info:
title: Computation Service API
version: 1.0.0
servers:
- url: http://localhost:7002
paths:
/run:
post:
summary: Run a computation
requestBody:
required: true
content:
application/octet-stream:
schema:
type: string
format: binary
description: The computation binary file (Linux executable)
responses:
"200":
description: Computation started
content:
application/json:
schema:
type: object
properties:
computationId:
type: string
description: Identifier for the computation
/algo:
post:
summary: Upload algorithm binary
requestBody:
required: true
content:
application/octet-stream:
schema:
type: string
format: binary
description: The algorithm binary file (Linux executable)
responses:
"200":
description: Algorithm binary uploaded
content:
application/json:
schema:
type: object
properties:
algorithmId:
type: string
description: Identifier for the uploaded algorithm binary
/data:
post:
summary: Upload dataset CSV file
requestBody:
required: true
content:
application/octet-stream:
schema:
type: string
format: binary
description: The dataset CSV file as a binary
responses:
"200":
description: Dataset CSV uploaded
content:
application/json:
schema:
type: object
properties:
datasetId:
type: string
description: Identifier for the uploaded dataset CSV
/result:
get:
summary: Retrieve computation result file
responses:
"200":
description: Computation result file retrieved successfully
content:
application/octet-stream:
schema:
type: string
format: binary
description: The computation result file
/attestation:
get:
summary: Attestation endpoint
responses:
"200":
description: Attestation result file retrieved successfully
content:
application/octet-stream:
schema:
type: string
format: binary
description: The attestation result file
+17 -6
View File
@@ -3,24 +3,35 @@
package cli
import (
"encoding/pem"
"log"
"os"
"github.com/spf13/cobra"
"github.com/ultravioletrs/cocos/agent"
)
const resultFilePath = "result.bin"
func NewResultsCmd(sdk agent.Service) *cobra.Command {
func (cli *CLI) NewResultsCmd() *cobra.Command {
return &cobra.Command{
Use: "result",
Short: "Retrieve computation result file",
Args: cobra.ExactArgs(1),
Use: "result",
Short: "Retrieve computation result file",
Example: "result <private_key_file_path>",
Args: cobra.ExactArgs(1),
Run: func(cmd *cobra.Command, args []string) {
log.Println("Retrieving computation result file")
result, err := sdk.Result(cmd.Context(), args[0])
privKeyFile, err := os.ReadFile(args[0])
if err != nil {
log.Fatalf("Error reading private key file: %v", err)
}
pemBlock, _ := pem.Decode(privKeyFile)
var result []byte
privKey := decodeKey(pemBlock)
result, err = cli.agentSDK.Result(cmd.Context(), privKey)
if err != nil {
log.Fatalf("Error retrieving computation result: %v", err)
}
+9 -4
View File
@@ -2,9 +2,14 @@
// SPDX-License-Identifier: Apache-2.0
package cli
import (
"github.com/ultravioletrs/cocos/agent"
)
import "github.com/ultravioletrs/cocos/pkg/sdk"
func SetSDK(s agent.Service) {
type CLI struct {
agentSDK sdk.SDK
}
func New(agentSDK sdk.SDK) *CLI {
return &CLI{
agentSDK: agentSDK,
}
}
+82 -71
View File
@@ -5,22 +5,23 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"os"
"strings"
"log/slog"
mglog "github.com/absmach/magistrala/logger"
"github.com/absmach/magistrala/pkg/uuid"
"github.com/absmach/magistrala/pkg/prometheus"
"github.com/mdlayher/vsock"
"github.com/ultravioletrs/cocos/agent"
"github.com/ultravioletrs/cocos/agent/api"
agentgrpc "github.com/ultravioletrs/cocos/agent/api/grpc"
"github.com/ultravioletrs/cocos/internal"
"github.com/ultravioletrs/cocos/internal/env"
"github.com/ultravioletrs/cocos/internal/events"
"github.com/ultravioletrs/cocos/agent/auth"
"github.com/ultravioletrs/cocos/agent/events"
agentlogger "github.com/ultravioletrs/cocos/internal/logger"
"github.com/ultravioletrs/cocos/internal/server"
grpcserver "github.com/ultravioletrs/cocos/internal/server/grpc"
"github.com/ultravioletrs/cocos/manager"
"github.com/ultravioletrs/cocos/manager/qemu"
"golang.org/x/sync/errgroup"
"google.golang.org/grpc"
"google.golang.org/grpc/reflection"
@@ -28,65 +29,63 @@ import (
const (
svcName = "agent"
envPrefixHTTP = "AGENT_HTTP_"
envPrefixGRPC = "AGENT_GRPC_"
defSvcHTTPPort = "9031"
defSvcGRPCPort = "7002"
)
var errComputationNotFound = errors.New("computation not found in command line")
type config struct {
LogLevel string `env:"AGENT_LOG_LEVEL" envDefault:"info"`
InstanceID string `env:"AGENT_INSTANCE_ID" envDefault:""`
NotificationServerURL string `env:"COCOS_NOTIFICATION_SERVER_URL" envDefault:"http://localhost:9000"`
}
func main() {
ctx, cancel := context.WithCancel(context.Background())
g, ctx := errgroup.WithContext(ctx)
var cfg config
if err := env.Parse(&cfg); err != nil {
log.Fatalf("failed to load %s configuration : %s", svcName, err)
}
logger, err := mglog.New(os.Stdout, cfg.LogLevel)
cfg, err := readConfig()
if err != nil {
log.Fatalf(err.Error())
log.Fatalf("failed to read agent configuration from vsock %s", err.Error())
}
if cfg.InstanceID == "" {
cfg.InstanceID, err = uuid.New().ID()
if err != nil {
log.Fatalf("Failed to generate instanceID: %s", err)
}
}
eventSvc := events.New(svcName, cfg.NotificationServerURL)
svc := newService(ctx, logger, eventSvc)
ac, err := extractComputationValue()
conn, err := vsock.Dial(vsock.Host, manager.ManagerVsockPort, nil)
if err != nil {
logger.Fatal(fmt.Sprintf("computation not loaded from cmdline : %s", err.Error()))
}
if _, err := svc.Run(ctx, ac); err != nil {
if err := eventSvc.SendEvent("init", ac.ID, "failed", json.RawMessage{}); err != nil {
logger.Warn(err.Error())
}
logger.Fatal(fmt.Sprintf("failed to run computation with err: %s", err))
log.Fatal(err)
}
defer conn.Close()
grpcServerConfig := server.Config{Port: defSvcGRPCPort}
if err := env.Parse(&grpcServerConfig, env.Options{Prefix: envPrefixGRPC}); err != nil {
log.Printf("failed to load %s gRPC server configuration : %s", svcName, err.Error())
var level slog.Level
if err := level.UnmarshalText([]byte(cfg.AgentConfig.LogLevel)); err != nil {
log.Println(err)
return
}
handler := agentlogger.NewProtoHandler(conn, &slog.HandlerOptions{Level: level})
logger := slog.New(handler)
eventSvc, err := events.New(svcName, cfg.ID, manager.ManagerVsockPort)
if err != nil {
logger.Error(fmt.Sprintf("failed to create events service %s", err.Error()))
return
}
defer eventSvc.Close()
svc := newService(ctx, logger, eventSvc, cfg)
grpcServerConfig := server.Config{
Port: cfg.AgentConfig.Port,
Host: cfg.AgentConfig.Host,
CertFile: cfg.AgentConfig.CertFile,
KeyFile: cfg.AgentConfig.KeyFile,
ServerCAFile: cfg.AgentConfig.ServerCAFile,
ClientCAFile: cfg.AgentConfig.ClientCAFile,
AttestedTLS: cfg.AgentConfig.AttestedTls,
}
registerAgentServiceServer := func(srv *grpc.Server) {
reflection.Register(srv)
agent.RegisterAgentServiceServer(srv, agentgrpc.NewServer(svc))
}
gs := grpcserver.New(ctx, cancel, svcName, grpcServerConfig, registerAgentServiceServer, logger)
authSvc, err := auth.New(cfg)
if err != nil {
logger.Error(fmt.Sprintf("failed to create auth service %s", err.Error()))
return
}
gs := grpcserver.New(ctx, cancel, svcName, grpcServerConfig, registerAgentServiceServer, logger, svc, authSvc)
g.Go(func() error {
return gs.Start()
@@ -101,41 +100,53 @@ func main() {
}
}
func newService(ctx context.Context, logger mglog.Logger, eventSvc events.Service) agent.Service {
svc := agent.New(ctx, logger, eventSvc)
func newService(ctx context.Context, logger *slog.Logger, eventSvc events.Service, cmp agent.Computation) agent.Service {
svc := agent.New(ctx, logger, eventSvc, cmp)
svc = api.LoggingMiddleware(svc, logger)
counter, latency := internal.MakeMetrics(svcName, "api")
counter, latency := prometheus.MakeMetrics(svcName, "api")
svc = api.MetricsMiddleware(svc, counter, latency)
return svc
}
// extractComputationValue to extract computation value from the command line.
func extractComputationValue() (agent.Computation, error) {
cmdLineBytes, err := os.ReadFile("/proc/cmdline")
func readConfig() (agent.Computation, error) {
l, err := vsock.Listen(qemu.VsockConfigPort, nil)
if err != nil {
return agent.Computation{}, err
}
cmdLine := string(cmdLineBytes)
paramPrefix := "computation="
index := strings.Index(string(cmdLine), paramPrefix)
defer l.Close()
if index == -1 {
return agent.Computation{}, errComputationNotFound
}
start := index + len(paramPrefix)
end := strings.Index(cmdLine[start:], " ")
cmpUnescaped := cmdLine[start : start+end]
var ac agent.Computation
if end == -1 {
cmpUnescaped = cmdLine[start:]
}
if err := json.Unmarshal([]byte(cmpUnescaped), &ac); err != nil {
conn, err := l.Accept()
if err != nil {
return agent.Computation{}, err
}
defer conn.Close()
var buffer []byte
for {
chunk := make([]byte, 1024)
n, err := conn.Read(chunk)
if err != nil {
if err == io.EOF {
break
}
return agent.Computation{}, err
}
buffer = append(buffer, chunk[:n]...)
}
ac := agent.Computation{
AgentConfig: agent.AgentConfig{},
}
if err := json.Unmarshal(buffer, &ac); err != nil {
return agent.Computation{}, err
}
if ac.AgentConfig.LogLevel == "" {
ac.AgentConfig.LogLevel = "info"
}
if ac.AgentConfig.Port == "" {
ac.AgentConfig.Port = defSvcGRPCPort
}
return ac, nil
}
+56 -14
View File
@@ -6,12 +6,13 @@ import (
"fmt"
"log"
"os"
"path"
mglog "github.com/absmach/magistrala/logger"
"github.com/caarlos0/env/v11"
"github.com/spf13/cobra"
"github.com/spf13/pflag"
"github.com/ultravioletrs/cocos/cli"
"github.com/ultravioletrs/cocos/internal/env"
"github.com/ultravioletrs/cocos/pkg/clients/grpc"
"github.com/ultravioletrs/cocos/pkg/clients/grpc/agent"
"github.com/ultravioletrs/cocos/pkg/sdk"
@@ -20,10 +21,13 @@ import (
const (
svcName = "cli"
envPrefixAgentGRPC = "AGENT_GRPC_"
completion = "completion"
filePermision = 0o755
cocosDirectory = ".cocos"
)
type config struct {
LogLevel string `env:"AGENT_LOG_LEVEL" envDefault:"info"`
LogLevel string `env:"AGENT_LOG_LEVEL" envDefault:"info"`
}
func main() {
@@ -32,38 +36,51 @@ func main() {
log.Fatalf("failed to load %s configuration : %s", svcName, err)
}
homePath, err := os.UserHomeDir()
if err != nil {
log.Fatalf("Error fetching user home directory: %v", err)
}
directoryCachePath := path.Join(homePath, cocosDirectory)
if err := os.MkdirAll(directoryCachePath, filePermision); err != nil {
log.Fatalf("Error while creating directory %s, error: %v", directoryCachePath, err)
}
logger, err := mglog.New(os.Stdout, cfg.LogLevel)
if err != nil {
log.Fatalf("Error creating logger: %s", err)
}
agentGRPCConfig := grpc.Config{}
if err := env.Parse(&agentGRPCConfig, env.Options{Prefix: envPrefixAgentGRPC}); err != nil {
logger.Fatal(fmt.Sprintf("failed to load %s gRPC client configuration : %s", svcName, err))
if err := env.ParseWithOptions(&agentGRPCConfig, env.Options{Prefix: envPrefixAgentGRPC}); err != nil {
logger.Error(fmt.Sprintf("failed to load %s gRPC client configuration : %s", svcName, err))
return
}
agentGRPCClient, agentClient, err := agent.NewAgentClient(agentGRPCConfig)
if err != nil {
logger.Fatal(err.Error())
logger.Error(err.Error())
return
}
defer agentGRPCClient.Close()
agentSDK := sdk.NewAgentSDK(logger, agentClient)
cli.SetSDK(agentSDK)
cliSVC := cli.New(agentSDK)
rootCmd := &cobra.Command{
Use: "cocos-cli [command]",
Short: "CLI application for Computation Service API",
Short: "CLI application for CoCos Service API",
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("CLI application for Computation Service API\n\n")
fmt.Printf("CLI application for CoCos Service API\n\n")
fmt.Printf("Usage:\n %s [command]\n\n", cmd.CommandPath())
fmt.Printf("Available Commands:\n")
// Filter out "completion" command
availableCommands := make([]*cobra.Command, 0)
for _, subCmd := range cmd.Commands() {
if subCmd.Name() != "completion" {
if subCmd.Name() != completion {
availableCommands = append(availableCommands, subCmd)
}
}
@@ -80,11 +97,36 @@ func main() {
},
}
// Root Commands
rootCmd.AddCommand(cli.NewAlgorithmsCmd(agentSDK))
rootCmd.AddCommand(cli.NewDatasetsCmd(agentSDK))
rootCmd.AddCommand(cli.NewResultsCmd(agentSDK))
rootCmd.AddCommand(cli.NewAttestationCmd(agentSDK))
keysCmd := cliSVC.NewKeysCmd()
attestationCmd := cliSVC.NewAttestationCmd()
backendCmd := cliSVC.NewBackendCmd()
// Agent Commands
rootCmd.AddCommand(cliSVC.NewAlgorithmCmd())
rootCmd.AddCommand(cliSVC.NewDatasetsCmd())
rootCmd.AddCommand(cliSVC.NewResultsCmd())
rootCmd.AddCommand(attestationCmd)
rootCmd.AddCommand(cliSVC.NewFileHashCmd())
rootCmd.AddCommand(backendCmd)
rootCmd.AddCommand(keysCmd)
rootCmd.AddCommand(cliSVC.NewCABundleCmd(directoryCachePath))
// Attestation commands
attestationCmd.AddCommand(cliSVC.NewGetAttestationCmd())
attestationCmd.AddCommand(cliSVC.NewValidateAttestationValidationCmd())
// Flags
keysCmd.PersistentFlags().StringVarP(
&cli.KeyType,
"key-type",
"k",
"rsa",
"User Key type",
)
// Backend information commands
backendCmd.AddCommand(cliSVC.NewAddMeasurementCmd())
backendCmd.AddCommand(cliSVC.NewAddHostDataCmd())
if err := rootCmd.Execute(); err != nil {
logger.Error(fmt.Sprintf("Command execution failed: %s", err))
+53 -66
View File
@@ -7,47 +7,40 @@ import (
"context"
"fmt"
"log"
"log/slog"
"net/url"
"os"
"strings"
mglog "github.com/absmach/magistrala/logger"
"github.com/absmach/magistrala/pkg/jaeger"
"github.com/absmach/magistrala/pkg/prometheus"
"github.com/absmach/magistrala/pkg/uuid"
"github.com/ultravioletrs/cocos/agent"
"github.com/caarlos0/env/v11"
"github.com/ultravioletrs/cocos/internal"
"github.com/ultravioletrs/cocos/internal/env"
"github.com/ultravioletrs/cocos/internal/events"
jaegerclient "github.com/ultravioletrs/cocos/internal/jaeger"
"github.com/ultravioletrs/cocos/internal/server"
grpcserver "github.com/ultravioletrs/cocos/internal/server/grpc"
httpserver "github.com/ultravioletrs/cocos/internal/server/http"
"github.com/ultravioletrs/cocos/manager"
"github.com/ultravioletrs/cocos/manager/api"
managergrpc "github.com/ultravioletrs/cocos/manager/api/grpc"
httpapi "github.com/ultravioletrs/cocos/manager/api/http"
managerapi "github.com/ultravioletrs/cocos/manager/api/grpc"
"github.com/ultravioletrs/cocos/manager/qemu"
"github.com/ultravioletrs/cocos/manager/tracing"
"github.com/ultravioletrs/cocos/pkg/clients/grpc"
managergrpc "github.com/ultravioletrs/cocos/pkg/clients/grpc/manager"
pkgmanager "github.com/ultravioletrs/cocos/pkg/manager"
"go.opentelemetry.io/otel/trace"
"golang.org/x/sync/errgroup"
"google.golang.org/grpc"
"google.golang.org/grpc/reflection"
)
const (
svcName = "manager"
envPrefixHTTP = "MANAGER_HTTP_"
envPrefixGRPC = "MANAGER_GRPC_"
envPrefixAgentGRPC = "AGENT_GRPC_"
envPrefixQemu = "MANAGER_QEMU_"
defSvcGRPCPort = "7001"
defSvcHTTPPort = "9021"
svcName = "manager"
envPrefixGRPC = "MANAGER_GRPC_"
envPrefixQemu = "MANAGER_QEMU_"
)
type config struct {
LogLevel string `env:"MANAGER_LOG_LEVEL" envDefault:"info"`
JaegerURL string `env:"COCOS_JAEGER_URL" envDefault:"http://localhost:14268/api/traces"`
InstanceID string `env:"MANAGER_INSTANCE_ID" envDefault:""`
NotificationServerURL string `env:"COCOS_NOTIFICATION_SERVER_URL" envDefault:"http://localhost:9000"`
HostIP string `env:"MANAGER_HOST_IP" envDefault:"localhost"`
LogLevel string `env:"MANAGER_LOG_LEVEL" envDefault:"info"`
JaegerURL url.URL `env:"COCOS_JAEGER_URL" envDefault:"http://localhost:4318"`
TraceRatio float64 `env:"MG_JAEGER_TRACE_RATIO" envDefault:"1.0"`
InstanceID string `env:"MANAGER_INSTANCE_ID" envDefault:""`
}
func main() {
@@ -65,13 +58,13 @@ func main() {
}
if cfg.InstanceID == "" {
cfg.InstanceID, err = uuid.New().ID()
if err != nil {
logger.Fatal(fmt.Sprintf("Failed to generate instance ID: %s", err))
if cfg.InstanceID, err = uuid.New().ID(); err != nil {
logger.Error(fmt.Sprintf("Failed to generate instance ID: %s", err))
return
}
}
tp, err := jaegerclient.NewProvider(ctx, svcName, cfg.JaegerURL, cfg.InstanceID)
tp, err := jaeger.NewProvider(ctx, svcName, cfg.JaegerURL, cfg.InstanceID, cfg.TraceRatio)
if err != nil {
logger.Error(fmt.Sprintf("Failed to init Jaeger: %s", err))
}
@@ -83,61 +76,55 @@ func main() {
tracer := tp.Tracer(svcName)
qemuCfg := qemu.Config{}
if err := env.Parse(&qemuCfg, env.Options{Prefix: envPrefixQemu}); err != nil {
logger.Fatal(fmt.Sprintf("failed to load QEMU configuration: %s", err))
}
exe, args, err := qemu.ExecutableAndArgs(qemuCfg, agent.Computation{})
if err != nil {
logger.Fatal(fmt.Sprintf("failed to parse QEMU configuration: %s", err))
}
logger.Info(fmt.Sprintf("%s %s", exe, strings.Join(args, " ")))
svc := newService(logger, tracer, qemuCfg, events.New(svcName, cfg.NotificationServerURL), cfg)
httpServerConfig := server.Config{Port: defSvcHTTPPort}
if err := env.Parse(&httpServerConfig, env.Options{Prefix: envPrefixHTTP}); err != nil {
logger.Fatal(fmt.Sprintf("failed to load %s gRPC server configuration: %s", svcName, err))
}
hs := httpserver.New(ctx, cancel, svcName, httpServerConfig, httpapi.MakeHandler(svc, cfg.InstanceID), logger)
grpcServerConfig := server.Config{Port: defSvcGRPCPort}
if err := env.Parse(&grpcServerConfig, env.Options{Prefix: envPrefixGRPC}); err != nil {
log.Printf("failed to load %s gRPC server configuration: %s", svcName, err.Error())
if err := env.ParseWithOptions(&qemuCfg, env.Options{Prefix: envPrefixQemu}); err != nil {
logger.Error(fmt.Sprintf("failed to load QEMU configuration: %s", err))
return
}
registerManagerServiceServer := func(srv *grpc.Server) {
reflection.Register(srv)
manager.RegisterManagerServiceServer(srv, managergrpc.NewServer(svc))
args := qemuCfg.ConstructQemuArgs()
logger.Info(strings.Join(args, " "))
managerGRPCConfig := grpc.Config{}
if err := env.ParseWithOptions(&managerGRPCConfig, env.Options{Prefix: envPrefixGRPC}); err != nil {
logger.Error(fmt.Sprintf("failed to load %s gRPC client configuration : %s", svcName, err))
return
}
gs := grpcserver.New(ctx, cancel, svcName, grpcServerConfig, registerManagerServiceServer, logger)
managerGRPCClient, managerClient, err := managergrpc.NewManagerClient(managerGRPCConfig)
if err != nil {
logger.Error(err.Error())
return
}
defer managerGRPCClient.Close()
pc, err := managerClient.Process(ctx)
if err != nil {
logger.Error(err.Error())
return
}
eventsChan := make(chan *pkgmanager.ClientStreamMessage)
svc := newService(logger, tracer, qemuCfg, eventsChan)
mc := managerapi.NewClient(pc, svc, eventsChan)
g.Go(func() error {
return hs.Start()
})
g.Go(func() error {
return gs.Start()
})
g.Go(func() error {
return server.StopHandler(ctx, cancel, logger, svcName, hs, gs)
return mc.Process(ctx, cancel)
})
if err := g.Wait(); err != nil {
logger.Error(fmt.Sprintf("%s service terminated: %s", svcName, err))
}
err = internal.DeleteFilesInDir(qemuCfg.TmpFileLoc)
if err != nil {
if err = internal.DeleteFilesInDir(qemuCfg.TmpFileLoc); err != nil {
logger.Error(err.Error())
}
}
func newService(logger mglog.Logger, tracer trace.Tracer, qemuCfg qemu.Config, eventSvc events.Service, cfg config) manager.Service {
svc := manager.New(qemuCfg, logger, eventSvc, cfg.HostIP)
func newService(logger *slog.Logger, tracer trace.Tracer, qemuCfg qemu.Config, eventsChan chan *pkgmanager.ClientStreamMessage) manager.Service {
svc := manager.New(qemuCfg, logger, eventsChan, qemu.NewVM)
go svc.RetrieveAgentEventsLogs()
svc = api.LoggingMiddleware(svc, logger)
counter, latency := internal.MakeMetrics(svcName, "api")
counter, latency := prometheus.MakeMetrics(svcName, "api")
svc = api.MetricsMiddleware(svc, counter, latency)
svc = tracing.New(svc, tracer)
-21
View File
@@ -1,21 +0,0 @@
FROM golang:1.21-alpine AS builder
ARG SVC
ARG GOARCH
ARG GOARM
ARG VERSION
ARG COMMIT
ARG TIME
WORKDIR /go/src/github.com/ultravioletrs/cocos-ai
COPY . .
RUN apk update \
&& apk add make \
&& make $SVC \
&& mv build/cocos-$SVC /exe
FROM alpine:latest
RUN apk add qemu-system-x86_64
COPY --from=builder /exe /
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENTRYPOINT ["/exe"]
-7
View File
@@ -1,7 +0,0 @@
FROM alpine:latest
ARG SVC
RUN apk add --no-cache qemu-system-x86_64
COPY cocos-$SVC /exe
COPY --from=alpine:latest /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENTRYPOINT ["/exe"]
-36
View File
@@ -1,36 +0,0 @@
version: "3.7"
name: "cocos"
networks:
cocos-base-net:
driver: bridge
services:
jaeger:
image: jaegertracing/all-in-one:1.38.0
container_name: cocos-jaeger
ports:
- ${COCOS_JAEGER_PORT}:${COCOS_JAEGER_PORT}/udp
- ${COCOS_JAEGER_FRONTEND}:${COCOS_JAEGER_FRONTEND}
- ${COCOS_JAEGER_COLLECTOR}:${COCOS_JAEGER_COLLECTOR}
- ${COCOS_JAEGER_CONFIGS}:${COCOS_JAEGER_CONFIGS}
networks:
- cocos-base-net
cocos-manager:
image: ghcr.io/ultravioletrs/cocos/manager:latest
container_name: cocos-manager
env_file:
- .env
devices:
- /dev/kvm
volumes:
- ../cmd/manager/img:/img
- /usr/share/edk2/x64/OVMF_CODE.fd:/usr/share/OVMF/OVMF_CODE.fd
- /usr/share/edk2/x64/OVMF_VARS.fd:/usr/share/OVMF/OVMF_VARS.fd
depends_on:
- jaeger
networks:
- cocos-base-net
ports:
- ${MANAGER_HTTP_PORT}:${MANAGER_HTTP_PORT}
- ${MANAGER_GRPC_PORT}:${MANAGER_GRPC_PORT}
+49 -35
View File
@@ -1,51 +1,65 @@
module github.com/ultravioletrs/cocos
go 1.21.3
go 1.22.4
require (
github.com/absmach/magistrala v0.0.0-20240110171157-9f573850fc4b
github.com/caarlos0/env/v7 v7.1.0
github.com/digitalocean/go-libvirt v0.0.0-20221205150000-2939327a8519
github.com/absmach/magistrala v0.14.1-0.20240709113739-04c359462746
github.com/caarlos0/env/v11 v11.1.0
github.com/cenkalti/backoff/v4 v4.3.0
github.com/digitalocean/go-libvirt v0.0.0-20240709142323-d8406205c752
github.com/go-kit/kit v0.13.0
github.com/go-zoo/bone v1.3.0
github.com/gofrs/uuid v4.4.0+incompatible
github.com/prometheus/client_golang v1.18.0
github.com/spf13/cobra v1.8.0
github.com/google/go-sev-guest v0.11.1
github.com/mdlayher/vsock v1.2.1
github.com/spf13/cobra v1.8.1
github.com/spf13/pflag v1.0.5
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.46.1
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.46.1
go.opentelemetry.io/otel v1.21.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.21.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.21.0
go.opentelemetry.io/otel/sdk v1.21.0
go.opentelemetry.io/otel/trace v1.21.0
golang.org/x/sync v0.6.0
google.golang.org/grpc v1.60.1
google.golang.org/protobuf v1.32.0
github.com/stretchr/testify v1.9.0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.53.0
go.opentelemetry.io/otel/trace v1.28.0
golang.org/x/crypto v0.25.0
golang.org/x/sync v0.7.0
google.golang.org/grpc v1.65.0
google.golang.org/protobuf v1.34.2
)
require (
go.opentelemetry.io/otel v1.28.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.28.0 // indirect
go.opentelemetry.io/otel/sdk v1.28.0 // indirect
)
require (
cloud.google.com/go/compute v1.23.3 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/go-kit/log v0.2.1 // indirect
github.com/go-logfmt/logfmt v0.6.0 // indirect
github.com/go-logr/logr v1.4.1 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.0 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/go-configfs-tsm v0.2.2 // indirect
github.com/google/logger v1.1.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect
github.com/prometheus/client_model v0.5.0 // indirect
github.com/prometheus/common v0.45.0 // indirect
github.com/prometheus/procfs v0.12.0 // indirect
go.opentelemetry.io/otel/metric v1.21.0 // indirect
go.opentelemetry.io/proto/otlp v1.0.0 // indirect
golang.org/x/net v0.20.0 // indirect
golang.org/x/sys v0.16.0 // indirect
golang.org/x/text v0.14.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240108191215-35c7eff3a6b1 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240108191215-35c7eff3a6b1 // indirect
github.com/mdlayher/socket v0.4.1 // indirect
github.com/pborman/uuid v1.2.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_golang v1.19.1 // indirect
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/common v0.52.2 // indirect
github.com/prometheus/procfs v0.13.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
go.opentelemetry.io/otel/metric v1.28.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/net v0.26.0 // indirect
golang.org/x/sys v0.22.0 // indirect
golang.org/x/term v0.22.0
golang.org/x/text v0.16.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240701130421-f6361c86f094 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240701130421-f6361c86f094 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
+95 -122
View File
@@ -1,33 +1,20 @@
cloud.google.com/go v0.110.8 h1:tyNdfIxjzaWctIiLYOTalaLKZ17SI44SKFW26QbOhME=
cloud.google.com/go/compute v1.23.3 h1:6sVlXXBmbd7jNX0Ipq0trII3e4n1/MsADLK6a+aiVlk=
cloud.google.com/go/compute v1.23.3/go.mod h1:VCgBUoMnIVIR0CscqQiPJLAG25E3ZRZMzcFZeQ+h8CI=
cloud.google.com/go/compute/metadata v0.2.3 h1:mg4jlk7mCAj6xXp9UJ4fjI9VUI5rubuGBW5aJ7UnBMY=
cloud.google.com/go/compute/metadata v0.2.3/go.mod h1:VAV5nSsACxMJvgaAuX6Pk2AawlZn8kiOGuCv6gTkwuA=
github.com/VividCortex/gohistogram v1.0.0 h1:6+hBz+qvs0JOrrNhhmR7lFxo5sINxBCGXrdtl/UvroE=
github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/4+TcAqDqk/vUH7g=
github.com/absmach/magistrala v0.0.0-20240110171157-9f573850fc4b h1:5dwCCLImHebX52aVApF15ruZ/b0MB9Y4zECr3pmF0A4=
github.com/absmach/magistrala v0.0.0-20240110171157-9f573850fc4b/go.mod h1:7D+27TxXQwMqTmlerdulb0QtvXpMcSgeGoyvzhxr8MQ=
github.com/absmach/magistrala v0.14.1-0.20240709113739-04c359462746 h1:Tj567KeGVygjTsSCxn4++skKiz9GkPugM1KMdIFxvfw=
github.com/absmach/magistrala v0.14.1-0.20240709113739-04c359462746/go.mod h1:CIx3OsPFc4doJZmBWSA6LNWefcznKv9c3cLOxNxL4q4=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/caarlos0/env/v7 v7.1.0 h1:9lzTF5amyQeWHZzuZeKlCb5FWSUxpG1js43mhbY8ozg=
github.com/caarlos0/env/v7 v7.1.0/go.mod h1:LPPWniDUq4JaO6Q41vtlyikhMknqymCLBw0eX4dcH1E=
github.com/cenkalti/backoff/v4 v4.2.1 h1:y4OZtCnogmCPw98Zjyt5a6+QwPLGkiQsYW5oUqylYbM=
github.com/cenkalti/backoff/v4 v4.2.1/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cncf/xds/go v0.0.0-20230607035331-e9ce68804cb4 h1:/inchEIKaYC1Akx+H+gqO04wryn5h75LSazbRlnya1k=
github.com/cncf/xds/go v0.0.0-20230607035331-e9ce68804cb4/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/caarlos0/env/v11 v11.1.0 h1:a5qZqieE9ZfzdvbbdhTalRrHT5vu/4V1/ad1Ka6frhI=
github.com/caarlos0/env/v11 v11.1.0/go.mod h1:LwgkYk1kDvfGpHthrWWLof3Ny7PezzFwS4QrsJdHTMo=
github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/digitalocean/go-libvirt v0.0.0-20221205150000-2939327a8519 h1:OpkN/n40cmKenDQS+IOAeW9DLhYy4DADSeZnouCEV/E=
github.com/digitalocean/go-libvirt v0.0.0-20221205150000-2939327a8519/go.mod h1:WyJJyfmJ0gWJvjV+ZH4DOgtOYZc1KOvYyBXWCLKxsUU=
github.com/envoyproxy/protoc-gen-validate v1.0.2 h1:QkIBuU5k+x7/QXPvPPnWXWlCdaBFApVqftFV6k087DA=
github.com/envoyproxy/protoc-gen-validate v1.0.2/go.mod h1:GpiZQP3dDbg4JouG/NNS7QWXpgx6x8QiMKdmN72jogE=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/digitalocean/go-libvirt v0.0.0-20240709142323-d8406205c752 h1:NI7XEcHzWVvBfVjSVK6Qk4wmrUfoyQxCNpBjrHelZFk=
github.com/digitalocean/go-libvirt v0.0.0-20240709142323-d8406205c752/go.mod h1:/Ok8PA2qi/ve0Py38+oL+VxoYmlowigYRyLEODRYdgc=
github.com/go-kit/kit v0.13.0 h1:OoneCcHKHQ03LfBpoQCUfCluwd2Vt3ohz+kvbJneZAU=
github.com/go-kit/kit v0.13.0/go.mod h1:phqEHMMUbyrCFCTgH48JueqrM3md2HcAZ8N3XE4FKDg=
github.com/go-kit/log v0.2.1 h1:MRVx0/zhvdseW+Gza6N9rVzU/IVzaeE1SFI4raAhmBU=
@@ -35,117 +22,103 @@ github.com/go-kit/log v0.2.1/go.mod h1:NwTd00d/i8cPZ3xOwwiv2PO5MOcx78fFErGNcVmBj
github.com/go-logfmt/logfmt v0.6.0 h1:wGYYu3uicYdqXVgoYbvnkrPVXkuLM1p1ifugDMEdRi4=
github.com/go-logfmt/logfmt v0.6.0/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.1 h1:pKouT5E8xu9zeFC39JXRDukb6JFQPXM5p5I91188VAQ=
github.com/go-logr/logr v1.4.1/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-zoo/bone v1.3.0 h1:PY6sHq37FnQhj+4ZyqFIzJQHvrrGx0GEc3vTZZC/OsI=
github.com/go-zoo/bone v1.3.0/go.mod h1:HI3Lhb7G3UQcAwEhOJ2WyNcsFtQX1WYHa0Hl4OBbhW8=
github.com/gofrs/uuid v4.4.0+incompatible h1:3qXRTX8/NbyulANqlc0lchS1gqAVxRgsuW1YrTJupqA=
github.com/gofrs/uuid v4.4.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg=
github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.0 h1:Wqo399gCIufwto+VfwCSvsnfGpF/w5E9CNxSwbpD6No=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.0/go.mod h1:qmOFXW2epJhM0qSnUUYpldc7gVz2KMQwJ/QYCDIa7XU=
github.com/google/go-configfs-tsm v0.2.2 h1:YnJ9rXIOj5BYD7/0DNnzs8AOp7UcvjfTvt215EWcs98=
github.com/google/go-configfs-tsm v0.2.2/go.mod h1:EL1GTDFMb5PZQWDviGfZV9n87WeGTR/JUg13RfwkgRo=
github.com/google/go-sev-guest v0.11.1 h1:gnww4U8fHV5DCPz4gykr1s8SEX1fFNcxCBy+vvXN24k=
github.com/google/go-sev-guest v0.11.1/go.mod h1:qBOfb+JmgsUI3aUyzQoGC13Kpp9zwLeWvuyXmA9q77w=
github.com/google/logger v1.1.1 h1:+6Z2geNxc9G+4D4oDO9njjjn2d0wN5d7uOo0vOIW1NQ=
github.com/google/logger v1.1.1/go.mod h1:BkeJZ+1FhQ+/d087r4dzojEg1u2ZX+ZqG1jTUrLM+zQ=
github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0 h1:bkypFPDjIYGfCYD5mRBvpqxfYX1YCS1PXdKYWi8FsN0=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0/go.mod h1:P+Lt/0by1T8bfcF3z737NnSbmxQAppXMRziHUxPOC8k=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 h1:jWpvCLoY8Z/e3VKvlsiIGKtc+UG6U5vzxaoagmhXfyg=
github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0/go.mod h1:QUyp042oQthUoa9bqDv0ER0wrtXnBruoNd7aNjkbP+k=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/mdlayher/socket v0.4.1 h1:eM9y2/jlbs1M615oshPQOHZzj6R6wMT7bX5NPiQvn2U=
github.com/mdlayher/socket v0.4.1/go.mod h1:cAqeGjoufqdxWkD7DkpyS+wcefOtmu5OQ8KuoJGIReA=
github.com/mdlayher/vsock v1.2.1 h1:pC1mTJTvjo1r9n9fbm7S1j04rCgCzhCOS5DY0zqHlnQ=
github.com/mdlayher/vsock v1.2.1/go.mod h1:NRfCibel++DgeMD8z/hP+PPTjlNJsdPOmxcnENvE+SE=
github.com/pborman/uuid v1.2.1 h1:+ZZIw58t/ozdjRaXh/3awHfmWRbzYxJoAdNJxe/3pvw=
github.com/pborman/uuid v1.2.1/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.18.0 h1:HzFfmkOzH5Q8L8G+kSJKUx5dtG87sewO+FoDDqP5Tbk=
github.com/prometheus/client_golang v1.18.0/go.mod h1:T+GXkCk5wSJyOqMIzVgvvjFDlkOQntgjkJWKrN5txjA=
github.com/prometheus/client_model v0.5.0 h1:VQw1hfvPvk3Uv6Qf29VrPF32JB6rtbgI6cYPYQjL0Qw=
github.com/prometheus/client_model v0.5.0/go.mod h1:dTiFglRmd66nLR9Pv9f0mZi7B7fk5Pm3gvsjB5tr+kI=
github.com/prometheus/common v0.45.0 h1:2BGz0eBc2hdMDLnO/8n0jeB3oPrt2D08CekT0lneoxM=
github.com/prometheus/common v0.45.0/go.mod h1:YJmSTw9BoKxJplESWWxlbyttQR4uaEcGyv9MZjVOJsY=
github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k6Bo=
github.com/prometheus/procfs v0.12.0/go.mod h1:pcuDEFsWDnvcgNzo4EEweacyhjeA9Zk3cnaOZAZEfOo=
github.com/prometheus/client_golang v1.19.1 h1:wZWJDwK+NameRJuPGDhlnFgx8e8HN3XHQeLaYJFJBOE=
github.com/prometheus/client_golang v1.19.1/go.mod h1:mP78NwGzrVks5S2H6ab8+ZZGJLZUq1hoULYBAYBw1Ho=
github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY=
github.com/prometheus/common v0.52.2 h1:LW8Vk7BccEdONfrJBDffQGRtpSzi5CQaRZGtboOO2ck=
github.com/prometheus/common v0.52.2/go.mod h1:lrWtQx+iDfn2mbH5GUzlH9TSHyfZpHkSiG1W7y3sF2Q=
github.com/prometheus/procfs v0.13.0 h1:GqzLlQyfsPbaEHaQkO7tbDlriv/4o5Hudv6OXHGKX7o=
github.com/prometheus/procfs v0.13.0/go.mod h1:cd4PFCR54QLnGKPaKGA6l+cfuNXtht43ZKY6tow0Y1g=
github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8=
github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0=
github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho=
github.com/spf13/cobra v1.8.1 h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM=
github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.46.1 h1:SpGay3w+nEwMpfVnbqOLH5gY52/foP8RE8UzTZ1pdSE=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.46.1/go.mod h1:4UoMYEZOC0yN/sPGH76KPkkU7zgiEWYWL9vwmbnTJPE=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.46.1 h1:aFJWCqJMNjENlcleuuOkGAPH82y0yULBScfXcIEdS24=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.46.1/go.mod h1:sEGXWArGqc3tVa+ekntsN65DmVbVeW+7lTKTjZF3/Fo=
go.opentelemetry.io/otel v1.21.0 h1:hzLeKBZEL7Okw2mGzZ0cc4k/A7Fta0uoPgaJCr8fsFc=
go.opentelemetry.io/otel v1.21.0/go.mod h1:QZzNPQPm1zLX4gZK4cMi+71eaorMSGT3A4znnUvNNEo=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.21.0 h1:cl5P5/GIfFh4t6xyruOgJP5QiA1pw4fYYdv6nc6CBWw=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.21.0/go.mod h1:zgBdWWAu7oEEMC06MMKc5NLbA/1YDXV1sMpSqEeLQLg=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.21.0 h1:digkEZCJWobwBqMwC0cwCq8/wkkRy/OowZg5OArWZrM=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.21.0/go.mod h1:/OpE/y70qVkndM0TrxT4KBoN3RsFZP0QaofcfYrj76I=
go.opentelemetry.io/otel/metric v1.21.0 h1:tlYWfeo+Bocx5kLEloTjbcDwBuELRrIFxwdQ36PlJu4=
go.opentelemetry.io/otel/metric v1.21.0/go.mod h1:o1p3CA8nNHW8j5yuQLdc1eeqEaPfzug24uvsyIEJRWM=
go.opentelemetry.io/otel/sdk v1.21.0 h1:FTt8qirL1EysG6sTQRZ5TokkU8d0ugCj8htOgThZXQ8=
go.opentelemetry.io/otel/sdk v1.21.0/go.mod h1:Nna6Yv7PWTdgJHVRD9hIYywQBRx7pbox6nwBnZIxl/E=
go.opentelemetry.io/otel/trace v1.21.0 h1:WD9i5gzvoUPuXIXH24ZNBudiarZDKuekPqi/E8fpfLc=
go.opentelemetry.io/otel/trace v1.21.0/go.mod h1:LGbsEB0f9LGjN+OZaQQ26sohbOmiMR+BaslueVtS/qQ=
go.opentelemetry.io/proto/otlp v1.0.0 h1:T0TX0tmXU8a3CbNXzEKGeU5mIVOdf0oykP+u2lIVU/I=
go.opentelemetry.io/proto/otlp v1.0.0/go.mod h1:Sy6pihPLfYHkr3NkUbEhGHFhINUSI/v80hjKIs5JXpM=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.20.0 h1:aCL9BSgETF1k+blQaYUBx9hJ9LOGP3gAVemcZlf1Kpo=
golang.org/x/net v0.20.0/go.mod h1:z8BVo6PvndSri0LbOE3hAn0apkU+1YvI6E70E9jsnvY=
golang.org/x/oauth2 v0.15.0 h1:s8pnnxNVzjWyrvYdFUQq5llS1PX2zhPXmccZv99h7uQ=
golang.org/x/oauth2 v0.15.0/go.mod h1:q48ptWNTY5XWf+JNten23lcvHpLJ0ZSxF5ttTHKVCAM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.6.0 h1:5BMeUDZ7vkXGfEr1x9B4bRcTH4lpkTkpdh0T/J+qjbQ=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.16.0 h1:xWw16ngr6ZMtmxDyKyIgsE93KNKz5HKmMa3b8ALHidU=
golang.org/x/sys v0.16.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/appengine v1.6.8 h1:IhEN5q69dyKagZPYMSdIjS2HqprW324FRQZJcGqPAsM=
google.golang.org/appengine v1.6.8/go.mod h1:1jJ3jBArFh5pcgW8gCtRJnepW8FzD1V44FJffLiz/Ds=
google.golang.org/genproto v0.0.0-20240102182953-50ed04b92917 h1:nz5NESFLZbJGPFxDT/HCn+V1mZ8JGNoY4nUpmW/Y2eg=
google.golang.org/genproto v0.0.0-20240102182953-50ed04b92917/go.mod h1:pZqR+glSb11aJ+JQcczCvgf47+duRuzNSKqE8YAQnV0=
google.golang.org/genproto/googleapis/api v0.0.0-20240108191215-35c7eff3a6b1 h1:OPXtXn7fNMaXwO3JvOmF1QyTc00jsSFFz1vXXBOdCDo=
google.golang.org/genproto/googleapis/api v0.0.0-20240108191215-35c7eff3a6b1/go.mod h1:B5xPO//w8qmBDjGReYLpR6UJPnkldGkCSMoH/2vxJeg=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240108191215-35c7eff3a6b1 h1:gphdwh0npgs8elJ4T6J+DQJHPVF7RsuJHCfwztUb4J4=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240108191215-35c7eff3a6b1/go.mod h1:daQN87bsDqDoe316QbbvX60nMoJQa4r6Ds0ZuoAe5yA=
google.golang.org/grpc v1.60.1 h1:26+wFr+cNqSGFcOXcabYC0lUVJVRa2Sb2ortSK7VrEU=
google.golang.org/grpc v1.60.1/go.mod h1:OlCHIeLYqSSsLi6i49B5QGdzaMZK9+M7LXN2FKz4eGM=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.32.0 h1:pPC6BG5ex8PDFnkbrGU3EixyhKcQ2aDuBS36lqK/C7I=
google.golang.org/protobuf v1.32.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.53.0 h1:9G6E0TXzGFVfTnawRzrPl83iHOAV7L8NJiR8RSGYV1g=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.53.0/go.mod h1:azvtTADFQJA8mX80jIH/akaE7h+dbm/sVuaHqN13w74=
go.opentelemetry.io/otel v1.28.0 h1:/SqNcYk+idO0CxKEUOtKQClMK/MimZihKYMruSMViUo=
go.opentelemetry.io/otel v1.28.0/go.mod h1:q68ijF8Fc8CnMHKyzqL6akLO46ePnjkgfIMIjUIX9z4=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0 h1:3Q/xZUyC1BBkualc9ROb4G8qkH90LXEIICcs5zv1OYY=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0/go.mod h1:s75jGIWA9OfCMzF0xr+ZgfrB5FEbbV7UuYo32ahUiFI=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.28.0 h1:j9+03ymgYhPKmeXGk5Zu+cIZOlVzd9Zv7QIiyItjFBU=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.28.0/go.mod h1:Y5+XiUG4Emn1hTfciPzGPJaSI+RpDts6BnCIir0SLqk=
go.opentelemetry.io/otel/metric v1.28.0 h1:f0HGvSl1KRAU1DLgLGFjrwVyismPlnuU6JD6bOeuA5Q=
go.opentelemetry.io/otel/metric v1.28.0/go.mod h1:Fb1eVBFZmLVTMb6PPohq3TO9IIhUisDsbJoL/+uQW4s=
go.opentelemetry.io/otel/sdk v1.28.0 h1:b9d7hIry8yZsgtbmM0DKyPWMMUMlK9NEKuIG4aBqWyE=
go.opentelemetry.io/otel/sdk v1.28.0/go.mod h1:oYj7ClPUA7Iw3m+r7GeEjz0qckQRJK2B8zjcZEfu7Pg=
go.opentelemetry.io/otel/trace v1.28.0 h1:GhQ9cUuQGmNDd5BTCP2dAvv75RdMxEfTmYejp+lkx9g=
go.opentelemetry.io/otel/trace v1.28.0/go.mod h1:jPyXzNPg6da9+38HEwElrQiHlVMTnVfM3/yv2OlIHaI=
go.opentelemetry.io/proto/otlp v1.3.1 h1:TrMUixzpM0yuc/znrFTP9MMRh8trP93mkCiDVeXrui0=
go.opentelemetry.io/proto/otlp v1.3.1/go.mod h1:0X1WI4de4ZsLrrJNLAQbFeLCm3T7yBkR0XqQ7niQU+8=
go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
golang.org/x/crypto v0.25.0 h1:ypSNr+bnYL2YhwoMt2zPxHFmbAN1KZs/njMG3hxUp30=
golang.org/x/crypto v0.25.0/go.mod h1:T+wALwcMOSE0kXgUAnPAHqTLW+XHgcELELW8VaDgm/M=
golang.org/x/net v0.26.0 h1:soB7SVo0PWrY4vPW/+ay0jKDNScG2X9wFeYlXIvJsOQ=
golang.org/x/net v0.26.0/go.mod h1:5YKkiSynbBIh3p6iOc/vibscux0x38BZDkn8sCUPxHE=
golang.org/x/sync v0.7.0 h1:YsImfSBoP9QPYL0xyKJPq0gcaJdG3rInoqxTWbfQu9M=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.0.0-20210426230700-d19ff857e887/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI=
golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.22.0 h1:BbsgPEJULsl2fV/AT3v15Mjva5yXKQDyKf+TbDz7QJk=
golang.org/x/term v0.22.0/go.mod h1:F3qCibpT5AMpCRfhfT53vVJwhLtIVHhB9XDjfFvnMI4=
golang.org/x/text v0.16.0 h1:a94ExnEXNtEwYLGJSIUxnWoxoRz/ZcCsV63ROupILh4=
golang.org/x/text v0.16.0/go.mod h1:GhwF1Be+LQoKShO3cGOHzqOgRrGaYc9AvblQOmPVHnI=
google.golang.org/genproto/googleapis/api v0.0.0-20240701130421-f6361c86f094 h1:0+ozOGcrp+Y8Aq8TLNN2Aliibms5LEzsq99ZZmAGYm0=
google.golang.org/genproto/googleapis/api v0.0.0-20240701130421-f6361c86f094/go.mod h1:fJ/e3If/Q67Mj99hin0hMhiNyCRmt6BQ2aWIJshUSJw=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240701130421-f6361c86f094 h1:BwIjyKYGsK9dMCBOorzRri8MQwmi7mT9rGHsCEinZkA=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240701130421-f6361c86f094/go.mod h1:Ue6ibwXGpU+dqIcODieyLOcgj7z8+IcskoNIgZxtrFY=
google.golang.org/grpc v1.65.0 h1:bs/cUb4lp1G5iImFFd3u5ixQzweKizoZJAwBNLR42lc=
google.golang.org/grpc v1.65.0/go.mod h1:WgYC2ypjlB0EiQi6wdKixMqukr6lBc0Vo+oOgjrM5ZQ=
google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+2 -1
View File
@@ -1 +1,2 @@
source "$BR2_EXTERNAL_COCOS_PATH/package/agent/Config.in"
source "$BR2_EXTERNAL_COCOS_PATH/package/agent/Config.in"
source "$BR2_EXTERNAL_COCOS_PATH/package/wasmtime/Config.in"
+1
View File
@@ -9,6 +9,7 @@ git clone git@github.com:ultravioletrs/cocos.git
git clone git@github.com:buildroot/buildroot.git
cd buildroot
make BR2_EXTERNAL=../cocos/hal/linux cocos_defconfig
# Execute 'make menuconfig' only if you want to make additional configuration changes to Buildroot.
make menuconfig
make
```
+2
View File
@@ -38,6 +38,8 @@ CONFIG_AUTOFS4_FS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_UNWINDER_FRAME_POINTER=y
CONFIG_VSOCKETS=y
CONFIG_VIRTIO_VSOCKETS=y
###
# AMD SEV-SNP
+28 -1
View File
@@ -4,8 +4,10 @@ BR2_x86_64=y
# System
BR2_TARGET_GENERIC_HOSTNAME="cocos"
BR2_TARGET_GENERIC_ISSUE="Welcome to Cocos"
BR2_SYSTEM_DHCP="eth0"
BR2_PACKAGE_DHCP=y
BR2_PACKAGE_DHCP_CLIENT=y
BR2_INIT_SYSTEMD=y
BR2_SYSTEM_BIN_SH_BASH=y
# Filesystem
# BR2_TARGET_ROOTFS_TAR is not set
@@ -42,5 +44,30 @@ BR2_LINUX_KERNEL_NEEDS_HOST_LIBELF=y
BR2_PACKAGE_HOST_QEMU=y
BR2_PACKAGE_HOST_QEMU_SYSTEM_MODE=y
# Docker
BR2_PACKAGE_LIBSECCOMP_ARCH_SUPPORTS=y
BR2_PACKAGE_LIBSECCOMP=y
BR2_PACKAGE_CA_CERTIFICATES=y
BR2_PACKAGE_DOCKER_CLI=y
BR2_PACKAGE_DOCKER_COMPOSE=y
BR2_PACKAGE_DOCKER_ENGINE=y
BR2_PACKAGE_CONTAINERD=y
BR2_PACKAGE_RUNC=y
BR2_PACKAGE_IPTABLES=y
# Python
BR2_PACKAGE_PYTHON3=y
BR2_PACKAGE_PYTHON_PIP=y
BR2_PACKAGE_BZIP2=y
BR2_PACKAGE_XZ=y
BR2_PACKAGE_ZIP=y
BR2_PACKAGE_PYTHON3_ZLIB=y
BR2_PACKAGE_PYTHON3_XZ=y
BR2_PACKAGE_PYTHON3_BZIP2=y
BR2_INSTALL_LIBSTDCPP=y
BR2_TOOLCHAIN_BUILDROOT_CXX=y
BR2_PACKAGE_HOST_GCC_TARGET=y
BR2_TOOLCHAIN_BUILDROOT_LIBSTDCPP=y
BR2_PACKAGE_GCC=y
BR2_PACKAGE_GCC_TARGET=y
BR2_PACKAGE_LIBSTDCPP=y
+1 -1
View File
@@ -3,6 +3,6 @@ config BR2_PACKAGE_AGENT
default y
help
Confidential Computing Agent is a state machine capable of
receiving data and algorithms, running computations, and
receiving datasets and algorithm, running computations, and
fetching the attestation report from within the
Confidential VM.
+5 -4
View File
@@ -12,14 +12,15 @@ define AGENT_BUILD_CMDS
endef
define AGENT_INSTALL_TARGET_CMDS
$(INSTALL) -D -m 0755 $(@D)/build/cocos-agent $(TARGET_DIR)/bin
$(INSTALL) -D -m 0750 $(@D)/build/cocos-agent $(TARGET_DIR)/bin
mkdir -p $(TARGET_DIR)/var/log/cocos
mkdir -p $(TARGET_DIR)/cocos/
endef
define AGENT_INSTALL_INIT_SYSTEMD
$(INSTALL) -D -m 0644 $(@D)/init/systemd/cocos-agent.service $(TARGET_DIR)/usr/lib/systemd/system/cocos-agent.service
$(INSTALL) -D -m 0644 $(@D)/init/systemd/eth0.network $(TARGET_DIR)/etc/systemd/network
$(INSTALL) -D -m 0644 $(@D)/init/systemd/00-network.link $(TARGET_DIR)/etc/systemd/network
$(INSTALL) -D -m 0640 $(@D)/init/systemd/cocos-agent.service $(TARGET_DIR)/usr/lib/systemd/system/cocos-agent.service
$(INSTALL) -D -m 0750 $(@D)/init/systemd/cocos_network_setup.sh $(TARGET_DIR)/cocos/cocos_network_setup.sh
$(INSTALL) -D -m 0750 $(@D)/init/systemd/agent_start_script.sh $(TARGET_DIR)/cocos/agent_start_script.sh
endef
$(eval $(golang-package))
+5
View File
@@ -0,0 +1,5 @@
config BR2_PACKAGE_WASMTIME
bool "wasmtime"
help
Wasmtime is a standalone runtime for WebAssembly.
https://github.com/bytecodealliance/wasmtime
+11
View File
@@ -0,0 +1,11 @@
WASMTIME_SITE = https://wasmtime.dev/install.sh
define WASMTIME_BUILD_CMDS
curl $(WASMTIME_SITE) -sSf | bash
endef
define WASMTIME_INSTALL_TARGET_CMDS
$(INSTALL) -D -m 0755 ~/.wasmtime/bin/wasmtime $(TARGET_DIR)/usr/bin/wasmtime
endef
$(eval $(generic-package))
-6
View File
@@ -1,6 +0,0 @@
[Match]
OriginalName=*
[Link]
NamePolicy=kernel
MACAddressPolicy=persistent
+17
View File
@@ -0,0 +1,17 @@
#!/bin/sh
NUM_OF_PERMITED_IFACE=1
NUM_OF_IFACE=$(ip route | grep -Eo 'dev [a-z0-9]+' | awk '{ print $2 }' | sort | uniq | wc -l)
if [ $NUM_OF_IFACE -gt $NUM_OF_PERMITED_IFACE ]; then
echo "More then one network interface in the VM"
exit 1
fi
DEFAULT_IFACE=$(route | grep '^default' | grep -o '[^ ]*$')
AGENT_GRPC_HOST=$(ip -4 addr show $DEFAULT_IFACE | grep inet | awk '{print $2}' | cut -d/ -f1)
export AGENT_GRPC_HOST
exec /bin/cocos-agent
+2 -2
View File
@@ -6,11 +6,11 @@ After=network.target
StandardOutput=file:/var/log/cocos/agent.stdout
StandardError=file:/var/log/cocos/agent.stderr
Environment=AGENT_GRPC_HOST=10.0.0.1
Environment=AGENT_GRPC_PORT=7002
Environment=AGENT_LOG_LEVEL=info
ExecStart=/bin/cocos-agent
ExecStartPre=/cocos/cocos_network_setup.sh
ExecStart=/cocos/agent_start_script.sh
[Install]
WantedBy=default.target
+17
View File
@@ -0,0 +1,17 @@
#!/bin/sh
# IFACES are all network interfaces excluding lo (LOOPBACK) and sit interfaces
IFACES=$(ip link show | grep -vE 'LOOPBACK|sit*' | awk -F': ' '{print $2}')
# This for loop brings up all network interfaces in IFACES and dhclient obtains an IP address for the every interface
for IFACE in $IFACES; do
STATE=$(ip link show $IFACE | grep DOWN)
if [ -n "$STATE" ]; then
ip link set $IFACE up
fi
IP_ADDR=$(ip addr show $IFACE | grep 'inet ')
if [ -z "$IP_ADDR" ]; then
dhclient $IFACE
fi
done
-5
View File
@@ -1,5 +0,0 @@
[Match]
Name=eth0
[Network]
Address=10.0.0.1/24
+1 -2
View File
@@ -67,8 +67,7 @@ func RunCmdOutput(command string, args ...string) (string, error) {
func RunCmdStart(command string, args ...string) (*exec.Cmd, error) {
cmd := exec.Command(command, args...)
err := cmd.Start()
if err != nil {
if err := cmd.Start(); err != nil {
return nil, fmt.Errorf("error starting command '%s': %s", cmd.String(), err)
}
-6
View File
@@ -1,6 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
// Package env contains utility tools for handling
// environment-based service configuration.
package env
-38
View File
@@ -1,38 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package env
import "github.com/caarlos0/env/v7"
type Options struct {
// Environment keys and values that will be accessible for the service
Environment map[string]string
// TagName specifies another tagname to use rather than the default env
TagName string
// RequiredIfNoDef automatically sets all env as required if they do not declare 'envDefault'
RequiredIfNoDef bool
// OnSet allows to run a function when a value is set
OnSet env.OnSetFn
// Prefix define a prefix for each key
Prefix string
}
func Parse(v interface{}, opts ...Options) error {
var actOpts []env.Options
for _, opt := range opts {
actOpts = append(actOpts, env.Options{
Environment: opt.Environment,
TagName: opt.TagName,
RequiredIfNoDef: opt.RequiredIfNoDef,
OnSet: opt.OnSet,
Prefix: opt.Prefix,
})
}
return env.Parse(v, actOpts...)
}
-11
View File
@@ -1,11 +0,0 @@
# Events package
## Overview
The events package provides a simple client for sending events to a server. It is designed to be used in conjunction with a server that can handle and process events.
`service`: A string representing the name of the service. Agent or manager.
`serverUrl`: A string representing the URL of the events server.
`event`: A string representing the event type of the events.
`computationId`: A string representing the computation ID associated with the events. This can be null for idle events.
The notifications are sent to endpoint `<server_url>/notifications/events` using post method.
-65
View File
@@ -1,65 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package events
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"net/http"
"time"
)
var errFailedToCreateNotification = errors.New("failed to create notification on server")
type service struct {
service string
serverUrl string
}
type Service interface {
SendEvent(event, computationId, status string, details json.RawMessage) error
}
func New(svc, serverUrl string) Service {
return &service{
service: svc,
serverUrl: serverUrl,
}
}
func (s *service) SendEvent(event, computationId, status string, details json.RawMessage) error {
body := struct {
EventType string `json:"event_type"`
Timestamp time.Time `json:"timestamp"`
ComputationID string `json:"computation_id,omitempty"`
Details json.RawMessage `json:"details,omitempty"`
Originator string `json:"originator"`
Status string `json:"status,omitempty"`
}{
EventType: event,
Timestamp: time.Now(),
ComputationID: computationId,
Originator: s.service,
Status: status,
Details: details,
}
jsonBody, err := json.Marshal(body)
if err != nil {
return err
}
req, err := http.NewRequest(http.MethodPost, fmt.Sprintf("%s/computations/events", s.serverUrl), bytes.NewReader(jsonBody))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
res, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
if res.StatusCode != http.StatusCreated {
return errFailedToCreateNotification
}
return nil
}
-4
View File
@@ -1,4 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package jaeger
-80
View File
@@ -1,80 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package jaeger
import (
"context"
"errors"
"net/url"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/resource"
"go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
)
var (
errNoURL = errors.New("URL is empty")
errNoSvcName = errors.New("Service Name is empty")
errUnsupportedTraceURLScheme = errors.New("unsupported tracing url scheme")
)
// NewProvider initializes Jaeger TraceProvider.
func NewProvider(ctx context.Context, svcName, jaegerurl, instanceID string) (*trace.TracerProvider, error) {
if jaegerurl == "" {
return nil, errNoURL
}
if svcName == "" {
return nil, errNoSvcName
}
jurl, err := url.Parse(jaegerurl)
if err != nil {
return nil, err
}
var exporter *otlptrace.Exporter
switch jurl.Scheme {
case "http":
exporter, err = otlptracehttp.New(ctx, otlptracehttp.WithEndpoint(jurl.Host), otlptracehttp.WithURLPath(jurl.Path), otlptracehttp.WithInsecure())
if err != nil {
return nil, err
}
case "https":
exporter, err = otlptracehttp.New(ctx, otlptracehttp.WithEndpoint(jurl.Host), otlptracehttp.WithURLPath(jurl.Path))
if err != nil {
return nil, err
}
default:
return nil, errUnsupportedTraceURLScheme
}
attributes := []attribute.KeyValue{
semconv.ServiceNameKey.String(svcName),
attribute.String("host.id", instanceID),
}
hostAttr, err := resource.New(ctx, resource.WithHost(), resource.WithOSDescription(), resource.WithContainer())
if err != nil {
return nil, err
}
attributes = append(attributes, hostAttr.Attributes()...)
tp := trace.NewTracerProvider(
trace.WithSampler(trace.AlwaysSample()),
trace.WithBatcher(exporter),
trace.WithResource(resource.NewWithAttributes(
semconv.SchemaURL,
attributes...,
)),
)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.TraceContext{})
return tp, nil
}
+2 -2
View File
@@ -5,14 +5,14 @@ package libvirt
import (
"fmt"
"log"
"log/slog"
"net"
"time"
mglog "github.com/absmach/magistrala/logger"
"github.com/digitalocean/go-libvirt"
)
func Connect(logger mglog.Logger) *libvirt.Libvirt {
func Connect(logger *slog.Logger) *libvirt.Libvirt {
// This dials libvirt on the local machine, but you can substitute the first
// two parameters with "tcp", "<ip address>:<port>" to connect to libvirt on
// a remote machine.
+95
View File
@@ -0,0 +1,95 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package logger
import (
"context"
"io"
"log/slog"
"github.com/ultravioletrs/cocos/pkg/manager"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/timestamppb"
)
var _ slog.Handler = (*handler)(nil)
type handler struct {
opts slog.HandlerOptions
w io.Writer
}
func NewProtoHandler(w io.Writer, opts *slog.HandlerOptions) slog.Handler {
if opts == nil {
opts = &slog.HandlerOptions{}
}
return &handler{
opts: *opts,
w: w,
}
}
// Enabled implements slog.Handler.
func (h *handler) Enabled(_ context.Context, l slog.Level) bool {
minLevel := slog.LevelInfo
if h.opts.Level != nil {
minLevel = h.opts.Level.Level()
}
return l >= minLevel
}
// Handle implements slog.Handler.
func (h *handler) Handle(_ context.Context, r slog.Record) error {
message := r.Message
timestamp := timestamppb.New(r.Time)
level := r.Level.String()
// Calculate the number of chunks
chunkSize := 500
numChunks := (len(message) + chunkSize - 1) / chunkSize
for i := 0; i < numChunks; i++ {
start := i * chunkSize
end := start + chunkSize
if end > len(message) {
end = len(message)
}
// Create a chunk of the message
chunk := message[start:end]
// Create the agent log with the chunk
agentLog := manager.ClientStreamMessage{
Message: &manager.ClientStreamMessage_AgentLog{
AgentLog: &manager.AgentLog{
Timestamp: timestamp,
Message: chunk,
Level: level,
},
},
}
// Marshal the chunk to protobuf
b, err := proto.Marshal(&agentLog)
if err != nil {
return err
}
// Write the chunk to the writer
if _, err := h.w.Write(b); err != nil {
return err
}
}
return nil
}
// WithAttrs implements slog.Handler.
func (*handler) WithAttrs(attrs []slog.Attr) slog.Handler {
panic("unimplemented")
}
// WithGroup implements slog.Handler.
func (*handler) WithGroup(name string) slog.Handler {
panic("unimplemented")
}
-27
View File
@@ -1,27 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package internal
import (
kitprometheus "github.com/go-kit/kit/metrics/prometheus"
stdprometheus "github.com/prometheus/client_golang/prometheus"
)
// MakeMetrics returns an instance of metrics.
func MakeMetrics(namespace, subsystem string) (*kitprometheus.Counter, *kitprometheus.Summary) {
counter := kitprometheus.NewCounterFrom(stdprometheus.CounterOpts{
Namespace: namespace,
Subsystem: subsystem,
Name: "request_count",
Help: "Number of requests received.",
}, []string{"method"})
latency := kitprometheus.NewSummaryFrom(stdprometheus.SummaryOpts{
Namespace: namespace,
Subsystem: subsystem,
Objectives: map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001},
Name: "request_latency_microseconds",
Help: "Total duration of requests in microseconds.",
}, []string{"method"})
return counter, latency
}
+204 -14
View File
@@ -1,37 +1,63 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package grpc
import (
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/asn1"
"encoding/pem"
"fmt"
"log/slog"
"math/big"
"net"
"os"
"time"
mglog "github.com/absmach/magistrala/logger"
"github.com/ultravioletrs/cocos/agent"
agentgrpc "github.com/ultravioletrs/cocos/agent/api/grpc"
"github.com/ultravioletrs/cocos/agent/auth"
"github.com/ultravioletrs/cocos/internal/server"
"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
"golang.org/x/crypto/sha3"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/credentials/insecure"
)
const (
stopWaitTime = 5 * time.Second
stopWaitTime = 5 * time.Second
organization = "Ultraviolet"
country = "Serbia"
province = ""
locality = "Belgrade"
streetAddress = "Bulevar Arsenija Carnojevica 103"
postalCode = "11000"
notAfterYear = 1
notAfterMonth = 0
notAfterDay = 0
)
type Server struct {
server.BaseServer
server *grpc.Server
registerService serviceRegister
agent agent.Service
authSvc auth.Authenticator
}
type serviceRegister func(srv *grpc.Server)
var _ server.Server = (*Server)(nil)
func New(ctx context.Context, cancel context.CancelFunc, name string, config server.Config, registerService serviceRegister, logger mglog.Logger) server.Server {
func New(ctx context.Context, cancel context.CancelFunc, name string, config server.Config, registerService serviceRegister, logger *slog.Logger, agentSvc agent.Service, authSvc auth.Authenticator) server.Server {
listenFullAddress := fmt.Sprintf("%s:%s", config.Host, config.Port)
return &Server{
BaseServer: server.BaseServer{
Ctx: ctx,
@@ -42,35 +68,102 @@ func New(ctx context.Context, cancel context.CancelFunc, name string, config ser
Logger: logger,
},
registerService: registerService,
agent: agentSvc,
authSvc: authSvc,
}
}
func (s *Server) Start() error {
errCh := make(chan error)
grpcServerOptions := []grpc.ServerOption{
grpc.StatsHandler(otelgrpc.NewServerHandler()),
}
if s.authSvc != nil {
unary, stream := agentgrpc.NewAuthInterceptor(s.authSvc)
grpcServerOptions = append(grpcServerOptions, grpc.UnaryInterceptor(unary))
grpcServerOptions = append(grpcServerOptions, grpc.StreamInterceptor(stream))
}
listener, err := net.Listen("tcp", s.Address)
if err != nil {
return fmt.Errorf("failed to listen on port %s: %w", s.Address, err)
}
creds := grpc.Creds(insecure.NewCredentials())
switch {
case s.Config.AttestedTLS:
certificateBytes, privateKeyBytes, err := generateCertificatesForATLS(s.agent)
if err != nil {
return fmt.Errorf("failed to create certificate: %w", err)
}
certificate, err := tls.X509KeyPair(certificateBytes, privateKeyBytes)
if err != nil {
return fmt.Errorf("falied due to invalid key pair: %w", err)
}
tlsConfig := &tls.Config{
ClientAuth: tls.NoClientCert,
Certificates: []tls.Certificate{certificate},
}
creds = grpc.Creds(credentials.NewTLS(tlsConfig))
s.Logger.Info(fmt.Sprintf("%s service gRPC server listening at %s with Attested TLS", s.Name, s.Address))
case s.Config.CertFile != "" || s.Config.KeyFile != "":
creds, err := credentials.NewServerTLSFromFile(s.Config.CertFile, s.Config.KeyFile)
certificate, err := loadX509KeyPair(s.Config.CertFile, s.Config.KeyFile)
if err != nil {
return fmt.Errorf("failed to load auth certificates: %w", err)
}
s.Logger.Info(fmt.Sprintf("%s service gRPC server listening at %s with TLS cert %s and key %s", s.Name, s.Address, s.Config.CertFile, s.Config.KeyFile))
s.server = grpc.NewServer(
grpc.Creds(creds),
grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()),
)
tlsConfig := &tls.Config{
ClientAuth: tls.RequireAndVerifyClientCert,
Certificates: []tls.Certificate{certificate},
}
var mtlsCA string
// Loading Server CA file
rootCA, err := loadCertFile(s.Config.ServerCAFile)
if err != nil {
return fmt.Errorf("failed to load root ca file: %w", err)
}
if len(rootCA) > 0 {
if tlsConfig.RootCAs == nil {
tlsConfig.RootCAs = x509.NewCertPool()
}
if !tlsConfig.RootCAs.AppendCertsFromPEM(rootCA) {
return fmt.Errorf("failed to append root ca to tls.Config")
}
mtlsCA = fmt.Sprintf("root ca %s", s.Config.ServerCAFile)
}
// Loading Client CA File
clientCA, err := loadCertFile(s.Config.ClientCAFile)
if err != nil {
return fmt.Errorf("failed to load client ca file: %w", err)
}
if len(clientCA) > 0 {
if tlsConfig.ClientCAs == nil {
tlsConfig.ClientCAs = x509.NewCertPool()
}
if !tlsConfig.ClientCAs.AppendCertsFromPEM(clientCA) {
return fmt.Errorf("failed to append client ca to tls.Config")
}
mtlsCA = fmt.Sprintf("%s client ca %s", mtlsCA, s.Config.ClientCAFile)
}
creds = grpc.Creds(credentials.NewTLS(tlsConfig))
switch {
case mtlsCA != "":
s.Logger.Info(fmt.Sprintf("%s service gRPC server listening at %s with TLS/mTLS cert %s , key %s and %s", s.Name, s.Address, s.Config.CertFile, s.Config.KeyFile, mtlsCA))
default:
s.Logger.Info(fmt.Sprintf("%s service gRPC server listening at %s with TLS cert %s and key %s", s.Name, s.Address, s.Config.CertFile, s.Config.KeyFile))
}
default:
s.Logger.Info(fmt.Sprintf("%s service gRPC server listening at %s without TLS", s.Name, s.Address))
s.server = grpc.NewServer(
grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()),
)
}
grpcServerOptions = append(grpcServerOptions, creds)
s.server = grpc.NewServer(grpcServerOptions...)
s.registerService(s.server)
go func() {
@@ -82,7 +175,6 @@ func (s *Server) Start() error {
return s.Stop()
case err := <-errCh:
s.Cancel()
return err
}
}
@@ -102,3 +194,101 @@ func (s *Server) Stop() error {
return nil
}
func loadCertFile(certFile string) ([]byte, error) {
if certFile != "" {
return os.ReadFile(certFile)
}
return []byte{}, nil
}
func loadX509KeyPair(certfile, keyfile string) (tls.Certificate, error) {
var cert, key []byte
var err error
if _, err = os.Stat(certfile); err == nil {
cert, err = os.ReadFile(certfile)
if err != nil {
return tls.Certificate{}, err
}
} else if os.IsNotExist(err) {
cert = []byte(certfile)
} else {
return tls.Certificate{}, err
}
if _, err := os.Stat(keyfile); err == nil {
key, err = os.ReadFile(keyfile)
if err != nil {
return tls.Certificate{}, err
}
} else if os.IsNotExist(err) {
key = []byte(keyfile)
} else {
return tls.Certificate{}, err
}
return tls.X509KeyPair(cert, key)
}
func generateCertificatesForATLS(svc agent.Service) ([]byte, []byte, error) {
curve := elliptic.P256()
privateKey, err := ecdsa.GenerateKey(curve, rand.Reader)
if err != nil {
return nil, nil, fmt.Errorf("failed to generate private/public key: %w", err)
}
publicKeyBytes, err := x509.MarshalPKIXPublicKey(&privateKey.PublicKey)
if err != nil {
return nil, nil, fmt.Errorf("failed to marshal the public key: %w", err)
}
// The Attestation Report will be added as an X.509 certificate extension
attestationReport, err := svc.Attestation(context.Background(), sha3.Sum512(publicKeyBytes))
if err != nil {
return nil, nil, fmt.Errorf("failed to fetch the attestation report: %w", err)
}
certTemplate := &x509.Certificate{
SerialNumber: big.NewInt(202403311),
Subject: pkix.Name{
Organization: []string{organization},
Country: []string{country},
Province: []string{province},
Locality: []string{locality},
StreetAddress: []string{streetAddress},
PostalCode: []string{postalCode},
},
NotBefore: time.Now(),
NotAfter: time.Now().AddDate(notAfterYear, notAfterMonth, notAfterDay),
KeyUsage: x509.KeyUsageDigitalSignature,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
BasicConstraintsValid: true,
ExtraExtensions: []pkix.Extension{
{
Id: asn1.ObjectIdentifier{1, 2, 3, 4, 5, 6},
Critical: false,
Value: attestationReport,
},
},
}
certDERBytes, err := x509.CreateCertificate(rand.Reader, certTemplate, certTemplate, &privateKey.PublicKey, privateKey)
if err != nil {
return nil, nil, fmt.Errorf("failed to create certificate: %w", err)
}
certBytes := pem.EncodeToMemory(&pem.Block{
Type: "CERTIFICATE",
Bytes: certDERBytes,
})
privateKeyBytes, err := x509.MarshalPKCS8PrivateKey(privateKey)
if err != nil {
return nil, nil, fmt.Errorf("failed to marshal the private key: %w", err)
}
keyBytes := pem.EncodeToMemory(&pem.Block{
Type: "PRIVATE KEY",
Bytes: privateKeyBytes,
})
return certBytes, keyBytes, nil
}
-5
View File
@@ -1,5 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
// Package http contains the HTTP server implementation.
package http
-78
View File
@@ -1,78 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package http
import (
"context"
"fmt"
"net/http"
"time"
mglog "github.com/absmach/magistrala/logger"
"github.com/ultravioletrs/cocos/internal/server"
)
const (
stopWaitTime = 5 * time.Second
httpProtocol = "http"
httpsProtocol = "https"
)
type Server struct {
server.BaseServer
server *http.Server
}
var _ server.Server = (*Server)(nil)
func New(ctx context.Context, cancel context.CancelFunc, name string, config server.Config, handler http.Handler, logger mglog.Logger) server.Server {
listenFullAddress := fmt.Sprintf("%s:%s", config.Host, config.Port)
httpServer := &http.Server{Addr: listenFullAddress, Handler: handler}
return &Server{
BaseServer: server.BaseServer{
Ctx: ctx,
Cancel: cancel,
Name: name,
Address: listenFullAddress,
Config: config,
Logger: logger,
},
server: httpServer,
}
}
func (s *Server) Start() error {
errCh := make(chan error)
s.Protocol = httpProtocol
switch {
case s.Config.CertFile != "" || s.Config.KeyFile != "":
s.Protocol = httpsProtocol
s.Logger.Info(fmt.Sprintf("%s service %s server listening at %s with TLS cert %s and key %s", s.Name, s.Protocol, s.Address, s.Config.CertFile, s.Config.KeyFile))
go func() {
errCh <- s.server.ListenAndServeTLS(s.Config.CertFile, s.Config.KeyFile)
}()
default:
s.Logger.Info(fmt.Sprintf("%s service %s server listening at %s without TLS", s.Name, s.Protocol, s.Address))
go func() {
errCh <- s.server.ListenAndServe()
}()
}
select {
case <-s.Ctx.Done():
return s.Stop()
case err := <-errCh:
return err
}
}
func (s *Server) Stop() error {
defer s.Cancel()
ctxShutdown, cancelShutdown := context.WithTimeout(context.Background(), stopWaitTime)
defer cancelShutdown()
if err := s.server.Shutdown(ctxShutdown); err != nil {
s.Logger.Error(fmt.Sprintf("%s service %s server error occurred during shutdown at %s: %s", s.Name, s.Protocol, s.Address, err))
return fmt.Errorf("%s service %s server error occurred during shutdown at %s: %w", s.Name, s.Protocol, s.Address, err)
}
s.Logger.Info(fmt.Sprintf("%s %s service shutdown of http at %s", s.Name, s.Protocol, s.Address))
return nil
}
+10 -8
View File
@@ -5,11 +5,10 @@ package server
import (
"context"
"fmt"
"log/slog"
"os"
"os/signal"
"syscall"
mglog "github.com/absmach/magistrala/logger"
)
type Server interface {
@@ -18,10 +17,13 @@ type Server interface {
}
type Config struct {
Host string `env:"HOST" envDefault:""`
Port string `env:"PORT" envDefault:""`
CertFile string `env:"SERVER_CERT" envDefault:""`
KeyFile string `env:"SERVER_KEY" envDefault:""`
Host string `env:"HOST" envDefault:""`
Port string `env:"PORT" envDefault:""`
CertFile string `env:"SERVER_CERT" envDefault:""`
KeyFile string `env:"SERVER_KEY" envDefault:""`
ServerCAFile string `env:"SERVER_CA_CERTS" envDefault:""`
ClientCAFile string `env:"CLIENT_CA_CERTS" envDefault:""`
AttestedTLS bool `env:"ATTESTED_TLS" envDefault:"false"`
}
type BaseServer struct {
@@ -30,7 +32,7 @@ type BaseServer struct {
Name string
Address string
Config Config
Logger mglog.Logger
Logger *slog.Logger
Protocol string
}
@@ -49,7 +51,7 @@ func stopAllServer(servers ...Server) error {
return nil
}
func StopHandler(ctx context.Context, cancel context.CancelFunc, logger mglog.Logger, svcName string, servers ...Server) error {
func StopHandler(ctx context.Context, cancel context.CancelFunc, logger *slog.Logger, svcName string, servers ...Server) error {
var err error
c := make(chan os.Signal, 1)
signal.Notify(c, syscall.SIGINT, syscall.SIGABRT)
+68 -81
View File
@@ -1,6 +1,6 @@
# Manager
Manager service provides a barebones HTTP and gRPC API and Service interface implementation for the development of the manager service.
Manager service provides a barebones gRPC API and Service interface implementation for the development of the manager service.
## Configuration
@@ -9,18 +9,14 @@ The service is configured using the environment variables from the following tab
| Variable | Description | Default |
| ----------------------------- | -------------------------------------------------------- | --------------------------------- |
| MANAGER_LOG_LEVEL | Log level for manager service (debug, info, warn, error) | info |
| MANAGER_HTTP_HOST | Manager service HTTP host | |
| MANAGER_HTTP_PORT | Manager service HTTP port | 9021 |
| MANAGER_HTTP_SERVER_CERT | Path to server certificate in pem format | |
| MANAGER_HTTP_SERVER_KEY | Path to server key in pem format | |
| MANAGER_GRPC_HOST | Manager service gRPC host | |
| MANAGER_GRPC_PORT | Manager service gRPC port | 7001 |
| MANAGER_GRPC_SERVER_CERT | Path to server certificate in pem format | |
| MANAGER_GRPC_SERVER_KEY | Path to server key in pem format | |
| COCOS_JAEGER_URL | Jaeger server URL | http://localhost:14268/api/traces |
| MANAGER_GRPC_SERVER_CA_CERTS | Path to gRPC server CA certificate | |
| MANAGER_GRPC_CLIENT_CA_CERTS | Path to gRPC client CA certificate | |
| COCOS_JAEGER_URL | Jaeger server URL | http://localhost:4318 |
| MANAGER_INSTANCE_ID | Manager service instance ID | |
| COCOS_NOTIFICATION_SERVER_URL | Server to receive notification events from agent. | http:/localhost:9000 |
| MANAGER_HOST_IP | Mnagaer host IP address | localhost |
## Setup
@@ -47,6 +43,22 @@ sudo apt install qemu-kvm
Create `img` directory in `cmd/manager`. Create `tmp` directory in `cmd/manager`.
#### Add Vsock
The necessary kernel modules must be loaded on the hypervisor. To check if `vhost_vsock` is loaded run:
```shell
lsmod | grep vhost_vsock
```
If `vhost_vsock` is not loaded run the following commands:
```shell
sudo modprobe vhost_vsock
ls -l /dev/vhost-vsock
# crw-rw-rw- 1 root kvm 10, 241 Jan 16 12:05 /dev/vhost-vsock
ls -l /dev/vsock
# crw-rw-rw- 1 root root 10, 121 Jan 16 12:05 /dev/vsock
```
### Prepare Cocos HAL
Cocos HAL for Linux is framework for building custom in-enclave Linux distribution. Use the instructions in [Readme](https://github.com/ultravioletrs/cocos/blob/main/hal/linux/README.md).
@@ -76,10 +88,11 @@ qemu-system-x86_64 \
-m 2048M,slots=5,maxmem=10240M \
-no-reboot \
-drive if=pflash,format=raw,unit=0,file=$OVMF_CODE,readonly=on \
-netdev user,id=vmnic,hostfwd=tcp::2222-:22,hostfwd=tcp::9301-:9031,hostfwd=tcp::7020-:7002 \
-netdev user,id=vmnic,hostfwd=tcp::7020-:7002 \
-device virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=vmnic,romfile= \
-device vhost-vsock-pci,id=vhost-vsock-pci0,guest-cid=3 -vnc :0 \
-kernel $KERNEL \
-append "earlyprintk=serial console=ttyS0 computation={\"id\":\"c0d15c5e-e37d-4426-b3b7-b432c966fb09\",\"name\":\"Sample_Computation\",\"description\":\"A_sample_computation\",\"datasets\":[{\"provider\":\"Provider1\",\"id\":\"Dataset1\"},{\"provider\":\"Provider2\",\"id\":\"Dataset2\"}],\"algorithms\":[{\"provider\":\"AlgorithmProvider1\",\"id\":\"Algorithm1\"}],\"result_consumers\":[\"Consumer1\"], \"timeout\":\"10m\"}" \
-append "earlyprintk=serial console=ttyS0" \
-initrd $INITRD \
-nographic \
-monitor pty \
@@ -88,10 +101,9 @@ qemu-system-x86_64 \
Once the VM is booted press enter and on the login use username `root`.
#### Build and run Agent
```sh
# Start the 'agent' executable in the background using '&' at the end.
cocos-agent &
Agent is started automatically in the VM.
```sh
# List running processes and use 'grep' to filter for processes containing 'agent' in their names.
ps aux | grep cocos-agent
# This command helps verify that the 'agent' process is running.
@@ -102,12 +114,6 @@ ps aux | grep cocos-agent
We can also check if `Agent` is reachable from the host machine:
```sh
# Use netcat (nc) to test the connection to localhost on port 9301.
nc -zv localhost 9301
# Output:
# nc: connect to localhost (::1) port 9301 (tcp) failed: Connection refused
# Connection to localhost (127.0.0.1) 9301 port [tcp/*] succeeded!
# Use netcat (nc) to test the connection to localhost on port 7020.
nc -zv localhost 7020
# Output:
@@ -138,23 +144,19 @@ NB: we set environment variables that we will use in the shell process where we
## Deployment
### Standalone
To start the service outside of the container, execute the following shell script:
To start the service, execute the following shell script (note a server needs to be running see [here](../test/computations/README.md)):
```bash
# download the latest version of the service
go get github.com/ultravioletrs/cocos
# Download the latest version of the service
git clone git@github.com:ultravioletrs/cocos.git
cd $GOPATH/src/github.com/ultravioletrs/cocos
cd cocos
# compile the manager
# Compile the manager
make manager
# copy binary to bin
make install
# set the environment variables and run the service
# Set the environment variables and run the service
MANAGER_GRPC_URL=localhost:7001
MANAGER_LOG_LEVEL=debug \
MANAGER_QEMU_USE_SUDO=false \
MANAGER_QEMU_ENABLE_SEV=false \
@@ -164,6 +166,7 @@ MANAGER_QEMU_ENABLE_SEV=false \
To enable [AMD SEV](https://www.amd.com/en/developer/sev.html) support, start manager like this
```sh
MANAGER_GRPC_URL=localhost:7001
MANAGER_LOG_LEVEL=debug \
MANAGER_QEMU_USE_SUDO=true \
MANAGER_QEMU_ENABLE_SEV=true \
@@ -171,66 +174,50 @@ MANAGER_QEMU_SEV_CBITPOS=51 \
./build/cocos-manager
```
The kernel hash feature might not work with the current build of OVMF and QEMU. If so, build the host kernel, QEMU, and OVMF from the [AMD SEV GitHub](https://github.com/AMDESE/AMDSEV/tree/snp-latest) repository.
### Docker
```bash
go get github.com/ultravioletrs/cocos
To build the OVMF with the kernel hash capability, we must build the AmdSev package of OVMF. The result of the build should be a single `OVMF.fd` file (unlike the regular two OVFM files). The OVMF package is located at `OvmfPkg/AmdSev/AmdSevX64.dsc`.
cd $GOPATH/src/github.com/ultravioletrs/cocos
# compile the manager
make manager
# create manager docker image
make docker_dev_manager
# start docker composition
make run
```
### Create QEMU virtual machine (VM)
To create an instance of VM and run a computation, run
To enable [AMD SEV-SNP](https://www.amd.com/en/developer/sev.html) support, start manager like this
```sh
curl -sSi -X POST \
http://localhost:9021/run \
-H "Content-Type: application/json" \
-d '{
"computation": {
"id":"c0d15c5e-e37d-4426-b3b7-b432c966fb09",
"name":"Sample_Computation",
"description":"A_sample_computation",
"datasets":[
{
"provider":"Provider1",
"id":"Dataset1"
},
{
"provider":"Provider2",
"id":"Dataset2"
}
],
"algorithms":[
{
"provider":"AlgorithmProvider1",
"id":"Algorithm1"
}
],
"result_consumers":[
"Consumer1"
],
"timeout":"10m"
}
}'
MANAGER_GRPC_URL=localhost:7001 \
MANAGER_LOG_LEVEL=debug \
MANAGER_QEMU_ENABLE_SEV=false \
MANAGER_QEMU_ENABLE_SEV_SNP=true \
MANAGER_QEMU_SEV_CBITPOS=51 \
MANAGER_QEMU_BIN_PATH=<path to QEMU binary> \
MANAGER_QEMU_QEMU_OVMF_CODE_FILE=<path to OVMF.fd Amd Sev built package> \
./build/cocos-manager
```
You should be able to create multiple instances by reruning the command.
To include the kernel hash into the measurement of the attestation report (SEV or SEV-SNP), start manager like this
```sh
MANAGER_GRPC_URL=localhost:7001 \
MANAGER_LOG_LEVEL=debug \
MANAGER_QEMU_ENABLE_SEV=false \
MANAGER_QEMU_ENABLE_SEV_SNP=true \
MANAGER_QEMU_SEV_CBITPOS=51 \
MANAGER_QEMU_KERNEL_HASH=true \
./build/cocos-manager
```
### Verifying VM launch
NB: To verify that the manager successfully launched the VM, you need to open two terminals on the same machine. In one terminal, you need to launch `go run main.go` (with the environment variables of choice) and in the other, you can run the verification commands.
NB: To verify that the manager successfully launched the VM, you need to open three terminals on the same machine. In one terminal, you need to launch the computations server by executing (with the environment variables of choice):
```bash
go run ./test/computations/main.go <dataset path> <algo path>
```
and in the second the manager by executing (with the environment variables of choice):
```bash
go run ./cmd/manager/main.go
```
Ensure that the Manager can connect to the Manager test server by setting the MANAGER_GRPC_PORT with the port value of the Manager test server. In the last terminal, you can run the verification commands.
To verify that the manager launched the VM successfully, run the following command:
+89
View File
@@ -0,0 +1,89 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package manager
import (
"fmt"
"net"
"regexp"
"strconv"
"github.com/absmach/magistrala/pkg/errors"
"github.com/mdlayher/vsock"
"github.com/ultravioletrs/cocos/pkg/manager"
"google.golang.org/protobuf/proto"
)
const (
ManagerVsockPort = 9997
messageSize int = 1024
)
var errFailedToParseCID = errors.New("failed to parse cid from remote address")
// RetrieveAgentEventsLogs Retrieve and forward agent logs and events via vsock.
func (ms *managerService) RetrieveAgentEventsLogs() {
l, err := vsock.Listen(ManagerVsockPort, nil)
if err != nil {
ms.logger.Warn(err.Error())
return
}
for {
conn, err := l.Accept()
if err != nil {
ms.logger.Warn(err.Error())
continue
}
go ms.handleConnections(conn)
}
}
func (ms *managerService) handleConnections(conn net.Conn) {
defer conn.Close()
for {
b := make([]byte, messageSize)
n, err := conn.Read(b)
if err != nil {
ms.logger.Warn(err.Error())
return
}
cmpID, err := ms.computationIDFromAddress(conn.RemoteAddr().String())
if err != nil {
ms.logger.Warn(err.Error())
continue
}
var message manager.ClientStreamMessage
if err := proto.Unmarshal(b[:n], &message); err != nil {
ms.logger.Warn(err.Error())
continue
}
switch mes := message.Message.(type) {
case *manager.ClientStreamMessage_AgentEvent:
mes.AgentEvent.ComputationId = cmpID
ms.eventsChan <- &manager.ClientStreamMessage{Message: mes}
case *manager.ClientStreamMessage_AgentLog:
mes.AgentLog.ComputationId = cmpID
ms.eventsChan <- &manager.ClientStreamMessage{Message: mes}
default:
ms.logger.Warn("Unexpected agent log or event type")
}
ms.logger.Info(fmt.Sprintf("Agent Log/Event, Computation ID: %s, Message: %s", cmpID, message.String()))
}
}
func (ms *managerService) computationIDFromAddress(address string) (string, error) {
re := regexp.MustCompile(`vm\((\d+)\)`)
matches := re.FindStringSubmatch(address)
if len(matches) > 1 {
cid, err := strconv.Atoi(matches[1])
if err != nil {
return "", err
}
return ms.agents[cid], nil
}
return "", errFailedToParseCID
}
+57 -57
View File
@@ -4,72 +4,72 @@ package grpc
import (
"context"
"fmt"
"time"
"errors"
"github.com/go-kit/kit/endpoint"
kitgrpc "github.com/go-kit/kit/transport/grpc"
"github.com/ultravioletrs/cocos/manager"
"google.golang.org/grpc"
pkgmanager "github.com/ultravioletrs/cocos/pkg/manager"
"golang.org/x/sync/errgroup"
)
const svcName = "manager.ManagerService"
var errTerminationFromServer = errors.New("server requested client termination")
type grpcClient struct {
run endpoint.Endpoint
timeout time.Duration
type ManagerClient struct {
stream pkgmanager.ManagerService_ProcessClient
svc manager.Service
responses chan *pkgmanager.ClientStreamMessage
}
// NewClient returns new gRPC client instance.
func NewClient(conn *grpc.ClientConn, timeout time.Duration) manager.ManagerServiceClient {
return &grpcClient{
run: kitgrpc.NewClient(
conn,
svcName,
"Run",
encodeRunRequest,
decodeRunResponse,
manager.RunResponse{},
).Endpoint(),
timeout: timeout,
func NewClient(stream pkgmanager.ManagerService_ProcessClient, svc manager.Service, responses chan *pkgmanager.ClientStreamMessage) ManagerClient {
return ManagerClient{
stream: stream,
svc: svc,
responses: responses,
}
}
// encodeRunRequest is a transport/grpc.EncodeRequestFunc that
// converts a user-domain runReq to a gRPC request.
func encodeRunRequest(_ context.Context, request interface{}) (interface{}, error) {
req, ok := request.(runReq)
if !ok {
return nil, fmt.Errorf("invalid request type: %T", request)
}
return &manager.RunRequest{
Computation: req.Computation,
}, nil
}
// decodeRunResponse is a transport/grpc.DecodeResponseFunc that
// converts a gRPC RunResponse to a user-domain response.
func decodeRunResponse(_ context.Context, grpcResponse interface{}) (interface{}, error) {
res, ok := grpcResponse.(*manager.RunResponse)
if !ok {
return nil, fmt.Errorf("invalid response type: %T", grpcResponse)
}
return runRes{AgentAddress: res.AgentAddress}, nil
}
func (client grpcClient) Run(ctx context.Context, req *manager.RunRequest, _ ...grpc.CallOption) (*manager.RunResponse, error) {
ctx, cancel := context.WithTimeout(ctx, client.timeout)
defer cancel()
runReq := runReq{
Computation: req.GetComputation(),
}
res, err := client.run(ctx, runReq)
if err != nil {
return nil, err
}
runRes := res.(runRes)
return &manager.RunResponse{AgentAddress: runRes.AgentAddress}, nil
func (client ManagerClient) Process(ctx context.Context, cancel context.CancelFunc) error {
eg, ctx := errgroup.WithContext(ctx)
eg.Go(func() error {
for {
req, err := client.stream.Recv()
if err != nil {
return err
}
switch mes := req.Message.(type) {
case *pkgmanager.ServerStreamMessage_RunReq:
port, err := client.svc.Run(ctx, mes.RunReq)
if err != nil {
return err
}
runRes := &pkgmanager.ClientStreamMessage_RunRes{RunRes: &pkgmanager.RunResponse{AgentPort: port, ComputationId: mes.RunReq.Id}}
if err := client.stream.Send(&pkgmanager.ClientStreamMessage{Message: runRes}); err != nil {
return err
}
case *pkgmanager.ServerStreamMessage_TerminateReq:
cancel()
return errors.Join(errTerminationFromServer, errors.New(mes.TerminateReq.Message))
case *pkgmanager.ServerStreamMessage_StopComputation:
if err := client.svc.Stop(ctx, mes.StopComputation.ComputationId); err != nil {
return err
}
}
}
})
eg.Go(func() error {
for {
select {
case <-ctx.Done():
return nil
case mes := <-client.responses:
if err := client.stream.Send(mes); err != nil {
return err
}
}
}
})
return eg.Wait()
}
-27
View File
@@ -1,27 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package grpc
import (
"context"
"github.com/go-kit/kit/endpoint"
"github.com/ultravioletrs/cocos/manager"
)
func runEndpoint(svc manager.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(runReq)
if err := req.validate(); err != nil {
return runRes{}, err
}
agAddr, err := svc.Run(ctx, req.Computation)
if err != nil {
return runRes{}, err
}
return runRes{AgentAddress: agAddr}, nil
}
}
-15
View File
@@ -1,15 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package grpc
import (
"github.com/ultravioletrs/cocos/manager"
)
type runReq struct {
Computation *manager.Computation `json:"computation"`
}
func (req runReq) validate() error {
return nil
}
-7
View File
@@ -1,7 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package grpc
type runRes struct {
AgentAddress string `json:"agent_address"`
}
+46 -30
View File
@@ -3,46 +3,62 @@
package grpc
import (
"context"
"github.com/go-kit/kit/transport/grpc"
"github.com/ultravioletrs/cocos/manager"
"github.com/ultravioletrs/cocos/pkg/manager"
"golang.org/x/sync/errgroup"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/peer"
)
var _ manager.ManagerServiceServer = (*grpcServer)(nil)
type grpcServer struct {
run grpc.Handler
manager.UnimplementedManagerServiceServer
incoming chan *manager.ClientStreamMessage
svc Service
}
type Service interface {
Run(ipAddress string, runReqChan chan *manager.ServerStreamMessage, authInfo credentials.AuthInfo)
}
// NewServer returns new AuthServiceServer instance.
func NewServer(svc manager.Service) manager.ManagerServiceServer {
func NewServer(incoming chan *manager.ClientStreamMessage, svc Service) manager.ManagerServiceServer {
return &grpcServer{
run: grpc.NewServer(
runEndpoint(svc),
decodeRunRequest,
encodeRunResponse,
),
incoming: incoming,
svc: svc,
}
}
func decodeRunRequest(_ context.Context, grpcReq interface{}) (interface{}, error) {
req := grpcReq.(*manager.RunRequest)
return runReq{
Computation: req.GetComputation(),
}, nil
}
func encodeRunResponse(_ context.Context, response interface{}) (interface{}, error) {
res := response.(runRes)
return &manager.RunResponse{AgentAddress: res.AgentAddress}, nil
}
func (s *grpcServer) Run(ctx context.Context, req *manager.RunRequest) (*manager.RunResponse, error) {
_, res, err := s.run.ServeGRPC(ctx, req)
if err != nil {
return nil, err
func (s *grpcServer) Process(stream manager.ManagerService_ProcessServer) error {
runReqChan := make(chan *manager.ServerStreamMessage)
client, ok := peer.FromContext(stream.Context())
if ok {
go s.svc.Run(client.Addr.String(), runReqChan, client.AuthInfo)
}
rr := res.(*manager.RunResponse)
return rr, nil
eg, ctx := errgroup.WithContext(stream.Context())
eg.Go(func() error {
for {
req, err := stream.Recv()
if err != nil {
return err
}
s.incoming <- req
}
})
eg.Go(func() error {
for {
select {
case <-ctx.Done():
return nil
case req := <-runReqChan:
if err := stream.Send(req); err != nil {
return err
}
}
}
})
return eg.Wait()
}
-5
View File
@@ -1,5 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
// Package http contains implementation of kit service HTTP API.
package http
-41
View File
@@ -1,41 +0,0 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package http
import (
"context"
"github.com/go-kit/kit/endpoint"
"github.com/ultravioletrs/cocos/manager"
)
func runEndpoint(svc manager.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(runReq)
if err := req.validate(); err != nil {
return nil, err
}
mc := manager.Computation{
Id: req.Computation.ID,
Name: req.Computation.Name,
Description: req.Computation.Description,
ResultConsumers: req.Computation.ResultConsumers,
}
for _, algo := range req.Computation.Algorithms {
mc.Algorithms = append(mc.Algorithms, &manager.Algorithm{Id: algo.ID, Provider: algo.Provider})
}
for _, data := range req.Computation.Datasets {
mc.Datasets = append(mc.Datasets, &manager.Dataset{Id: data.ID, Provider: data.Provider})
}
// Call the Run method on the service
agAddr, err := svc.Run(ctx, &mc)
if err != nil {
return nil, err
}
return runRes{AgentAddress: agAddr}, nil
}
}

Some files were not shown because too many files have changed in this diff Show More