mirror of
https://github.com/ultravioletrs/cocos.git
synced 2026-06-23 04:10:25 +00:00
main
17 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jovan Djukic
|
27db9b29eb |
COCOS-591: Add support for GPU CC attestation (#592)
CI / lint (push) Has been cancelled
CI / test (agent) (push) Has been cancelled
CI / test (cli) (push) Has been cancelled
CI / test (cmd) (push) Has been cancelled
CI / test (internal) (push) Has been cancelled
CI / test (manager, true) (push) Has been cancelled
CI / test (pkg) (push) Has been cancelled
CI / upload-coverage (push) Has been cancelled
* Added GPU evidence collection * Added GPU evidence verification * Added make command for nvattest helper * Added command for installing all services * changed attestion-service.service so it knows where the helper is * Possible IGVM script bug * Possible bug * Bug * bug * Revert "bug" This reverts commit |
||
 Sammy Kerata Oina
|
da31d76c94 |
NOISSUE - Agent Pull mode for remote resources (#575)
CI / checkproto (push) Has been cancelled
CI / lint (push) Has been cancelled
Rust CI Pipeline / rust-check (push) Has been cancelled
CI / test (agent) (push) Has been cancelled
CI / test (cli) (push) Has been cancelled
CI / test (cmd) (push) Has been cancelled
CI / test (internal) (push) Has been cancelled
CI / test (manager, true) (push) Has been cancelled
CI / test (pkg) (push) Has been cancelled
CI / upload-coverage (push) Has been cancelled
* feat(kbs): implement KBS client for attestation and resource retrieval - Added KBS client implementation in pkg/kbs/client.go with methods for attestation and resource retrieval. - Introduced necessary data structures for requests and responses. - Implemented error handling for various scenarios. test(kbs): add unit tests for KBS client - Created comprehensive tests for the KBS client in pkg/kbs/client_test.go. - Included tests for attestation success and failure cases, as well as resource retrieval. feat(registry): introduce HTTP and S3 registry implementations - Added HTTPRegistry for downloading resources over HTTP/HTTPS with retry logic in pkg/registry/http.go. - Implemented S3Registry for downloading resources from AWS S3 and S3-compatible services in pkg/registry/s3.go. - Included error handling and configuration options for both registries. chore(registry): define registry interface and configuration - Created registry interface and configuration struct in pkg/registry/registry.go. - Added default configuration settings for registry clients. docs(cvms): update README for CVMS server configuration and usage - Enhanced documentation for CVMS server with detailed command-line flags and usage examples. - Clarified direct upload and remote resource modes, including KBS integration. fix(cvms): integrate KBS for remote resource handling in main.go - Updated main.go to support remote datasets and algorithms using KBS. - Added validation for command-line flags to ensure proper configuration. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix: Move ifeq conditional outside define block in attestation-service.mk Make conditionals cannot be evaluated inside define...endef blocks when used as recipe bodies. Restructured to define the ATTESTATION_SERVICE_INSTALL_INIT_SYSTEMD block conditionally based on BR2_PACKAGE_CC_ATTESTATION_AGENT configuration. * feat: Implement remote resource downloading for algorithms and datasets using AWS S3/MinIO credentials. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Add comprehensive documentation and agent support for testing remote resource download with KBS attestation. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Improve agent logging for remote resource configuration and KBS status, and add a testing guide for remote resource downloads with KBS attestation. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Add a comprehensive guide for testing remote resource download with KBS attestation and update multiple package versions to a specific commit. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Add failure transitions for resource reception states and a comprehensive guide for testing remote resource downloads with KBS attestation. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Implement remote resource download with KBS attestation in the agent and add a comprehensive testing guide. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * test: Add comprehensive guide for testing remote resource download with KBS attestation and include a debug log in the attestation client. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Delegate KBS attestation and token retrieval to a new attestation-agent service and document remote resource testing. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * client fixes Signed-off-by: Sammy Oina <sammyoina@gmail.com> * raw evidence Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix: Build all Go files in cmd directories, not just main.go This fixes the issue where fetch_raw_evidence.go wasn't being included in the attestation-service build. * fix: Wrap binary evidence in JSON for KBS compatibility Fixes 'invalid character' error by wrapping raw binary evidence in a JSON structure with base64 encoding, as expected by KBS. * chore: Update buildroot packages to |
||
|
Sammy Kerata Oina
|
6043ad150b |
COCOS-256 - Progress bar on downloads (#290)
* add progress bar for downloads Signed-off-by: Sammy Oina <sammyoina@gmail.com> * better error handling Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix test and refactor Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix failing test Signed-off-by: Sammy Oina <sammyoina@gmail.com> * add test coverage Signed-off-by: Sammy Oina <sammyoina@gmail.com> --------- Signed-off-by: Sammy Oina <sammyoina@gmail.com> |
||
 b1ackd0t
|
63994d78b8 |
NOISSUE - Add Rust gitignore (#268)
* chore(backendinfo): Add rust build artefacts to gitignore Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * style: format file following rust linter guidelines Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * chore(CI): Add rust CI pipeline Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> --------- Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> |
||
|
b1ackd0t
|
742bba5f00 |
NOISSUE - Add Dockerfile For IRIS Example (#220)
* feat(Docker): Add Dockerfile for testing Add Dockerfile for testing linear regression algorithm Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * fix(docs): Update docker linear regression example Resolves https://github.com/ultravioletrs/cocos/pull/220#discussion_r1732974631 --------- Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> |
||
|
Sammy Kerata Oina
|
f906593492 |
remove tmp directory (#204)
Signed-off-by: Sammy Oina <sammyoina@gmail.com> |
||
|
Sammy Kerata Oina
|
066dacd46a |
NOISSUE - Fix docs (#203)
* fix docs Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix typos Signed-off-by: Sammy Oina <sammyoina@gmail.com> * cli Signed-off-by: Sammy Oina <sammyoina@gmail.com> * add build instructions Signed-off-by: Sammy Oina <sammyoina@gmail.com> * remove file Signed-off-by: Sammy Oina <sammyoina@gmail.com> --------- Signed-off-by: Sammy Oina <sammyoina@gmail.com> |
||
|
b1ackd0t
|
afc306a85b |
NOISSUE - Enable WASM Support and FileSystem Support (#189)
* feat(algorithm): Add wasm as an algo type Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * feat(algorithm): Use filesystem to store results Move from unix socket for results storage to filesystem * test: test new filesystem changes Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * refactor(files): rename resultFile to resultsFilePath * feat(wasm-runtime): change from wasmtime to wasmedge Wasmedge enables easier directory mapping to get results Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * feat(algorithm): send results as zipped directory Create a new function to zip the results directory and send it back to the user * fix(wasm): runtime argument Fix the directory mapping for wasm runtime arguments Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * fix(errors): provide useful error message * chore(gitignore): add results zip to gitignore * feat(filesystem): Enable storing results on filesystem for python algos * refactor: revert to upstream cocos repo Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * fix: remove AddDataset from algorithm interface * fix: agent to handle results zipping * test: test zipping directories * refactor(agent): Handle file operations from agent * test: run test inside eos Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * refactor(test): Document and test algos are running Document steps on running the 2 python exampls and ensure they are running on eos Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * fix: remove witheDataset option * test: test without dataset argument Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> --------- Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> |
||
|
b1ackd0t
|
2f8109879c |
COCOS-168 - Allow running Computations without datasets (#175)
* feat(agent): Allow empty dataset Allow running of algorithm with empty dataset since not all algorithms require datasets. Allow state-machine transition from algo-received state to running state incase of no dataset provided Fixes https://github.com/ultravioletrs/cocos/issues/168 Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * chore(gitignore): Remove build artefacts Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * feat(algorithms): Add test algorithm for addition Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * refactor(addition): Modify addition algo to one file Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * fix(agent): move state transition to callback func Move state transition from `receivingAlgorithm` to `running` to state call back function Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * feat(agent-event): Add `algoReceivedNoData` event `algoReceivedNoData` is an event that is sent if we receive an algorithm and it should not have a dataset hence changes the state from `receivingAlgorithm` to `running` * fix(agent-state): Change state depending on manifest Change state from `receivingAlgorithm` to either `receivingData` if there is a dataset or `running` if there is no dataset provided Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> --------- Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> |
||
|
Sammy Kerata Oina
|
2ce112cc1b |
COCOS-103 - User authN and AuthZ using digital signatures (#128)
* Update Go to 1.22 and enhance security features - Upgraded the Go version in GitHub Actions workflows to 1.22.x for latest features and security patches. - Added RSA public key field `UserKey` in `Dataset` and `Algorithm` to reinforce data integrity and encryption. - Refactored `Result` method in `agentService` to use `containsID` for improved readability and potential performance benefits. - Updated `grpcserver.New` and `internal/server/grpc` invocations to pass `agent.Service` by value in line with recommended Go practices. - Introduced `grpc.StreamInterceptor` with no args in `Server.Start` which seems to be an initial step for future stream interceptor configuration. These changes prepare for stronger data security measures, maintain compatibility with the latest Go features, and improve code quality regarding service struct usage. Potential follow-up is needed to configure the stream interceptor and to ensure the new RSA key field is appropriately utilized in data handling. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor auth system and protocol buffers Enhanced the authentication system by adding context support and an improved user-role model. Implemented robust RSA public key verification for users and a restructured interceptor logic specific to stream types, streamlining the auth process. Updated protocol buffers and associated structures to accommodate user keys as byte slices, aligning with standard cryptographic practice. CLI commands for algorithms and datasets now require a private key file path argument for signing, strengthening security during interactions. This comprehensive overhaul addresses security and efficiency considerations in the RPC framework and aligns with best practices for key handling. By streamlining and securing the user authentication process, the agent service's reliability is greatly improved, directly impacting the robustness of the entire computation pipeline. - Refactored auth: added role-based user validation, context handling - Reworked interceptors: separated stream types, fortified signature checks - Updated protocol buffers: user public keys as byte slices for standard compatibility - Enhanced CLI: introduced private key argument, ensuring secure algorithm and dataset submission - Improved server and SDK contracts to align with auth changes Related issues: - Implements user roles and auth context [#103] - CLI security enhancement for private key management Signed-off-by: SammyOina <sammyoina@gmail.com> * Updated PEM decoding for key parsing in CLI and tests Added `encoding/pem` to decode PEM blocks when parsing private and public keys across CLI commands and test computation scenarios, ensuring compatibility with key files. This enhances robustness in key handling by supporting PEM encoded keys. The update also includes registration of a new Keys command in the CLI. Refactored code is now compliant with common key formats, addressing potential parsing issues. Signed-off-by: SammyOina <sammyoina@gmail.com> * Fix auth signature encoding and improve CLI usage example The authentication system now decodes base64 strings before verifying signatures to align with the expected format. Additionally, the signature generation now encodes the output in base64, ensuring consistency across the auth process. The CLI help message for the `result` command is enhanced by providing a usage example, making it more user-friendly and informative. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor containsID to handle dynamic fields Updated the `containsID` function to accept a field name parameter, enabling dynamic field lookup within the reflection logic. This change facilitates the use of the function for various struct fields, improving code reusability and flexibility. CLI command 'data' now requires an additional argument for the private key file path, outlined in the usage example update, reinforcing command clarity and user guidance. Resolves issues with hardcoded field lookups and enhances CLI usability. Signed-off-by: SammyOina <sammyoina@gmail.com> * Remove extraneous newline in key generation log output A redundant newline after the success message in the key generation command was removed to clean up log output formatting. This change ensures a more consistent and professional appearance of the CLI tool's messages. Signed-off-by: SammyOina <sammyoina@gmail.com> * Implemented auth service in gRPC startup Added authentication services to the gRPC server initialization to enforce security measures. The gRPC server's New function now includes an `authSvc` parameter, requiring instantiation of the auth service before starting the server. Failure to create the auth service results in a fatal error, halting the process to avoid running without protection. Tests have been updated to include `nil` values for the auth service parameter to maintain their functionality without authentication. Refactored `grpcserver.New` to accept the new auth service, and updated the main agent startup logic to create and inject the auth service. Added the auth middleware interceptors to the server options, which ensures that each gRPC call will undergo authentication. This change is a step towards secure communication, and affected components should now consider the authentication requirement. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor config read logic and update agent setup Improved the configuration reading in `cmd/agent/main.go` to handle larger payloads by reading data in chunks and checking for EOF, ensuring that all config data is captured even if it exceeds the initial buffer size. Enhanced the `test/manual/agent-config/main.go` to require additional command-line arguments, improving the setup process by explicitly requiring paths for data, algorithm, and public key as well as a boolean for attested TLS. Also updated the hashing method to SHA3 for the algorithm and data files, and included the hash and public keys as part of the agent, dataset, and result consumer configurations. These changes will make the agent setup more robust and provide better integrity checks for the involved files. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor run method to agentService Moved the run function into agentService for better encapsulation and maintainability. This refactoring includes capturing both stdout and stderr during algorithm execution, enabling more informative debugging through enhanced logging. Consequentially, the run method now references members through the service instance, aligning with object-oriented best practices and improving code coherence. Resolves issue with insufficient execution details when computations fail. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor computation data handling to use filepaths Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor error logging and ensure consistency Replaced usage of the standard log package with a custom logger for error reporting to standardize error logging throughout the application. Additionally, introduced graceful shutdown by returning from the main function rather than forcing exit when failing to create auth service, aligning the application's error handling strategy. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor auth initialization and key file handling Improved the readability and maintainability of the authentication service initialization by adding line breaks for logical separation. Also, standardized key filenames in the CLI key generation by introducing constants, enhancing code clarity and reducing the likelihood of file-naming errors. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor auth verification logic for improved security Removed an extraneous line in the `verifySignature` function that was not necessary for the signature verification process. This change simplifies the code and improves readability. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor payload structures to simplify API Removed the 'provider', 'id', 'consumer' fields from protocol buffers, gRPC services, and related functions across various files to streamline the data model and align with the new authentication system based on cryptographic verification rather than string identifiers. This results in more efficient data handling and a reduction in unnecessary payload data, while enhancing security by making entity validation strictly cryptographic. The changes affect agent-SDK interactions, CLI tools, and related services, ensuring only the necessary data (algorithm/data bytes, user keys, and hashes) is transmitted and processed. Consequently, the core computation algorithm and dataset handlers now rely on indexes derived from context to associate data with respective manifest entries, thus maintaining the ability to link to specific computation manifests without relying on explicit IDs in the payload. Additionally, refactored authentication methods now enforce role-based security seamlessly through metadata. This approach enhances privacy by avoiding transmission of potentially sensitive strings over the network and by ensuring that only internal indices, not globally interpretable identifiers, are used to process computations. Aligned with the broader architectural goal of simplifying and securing the platform's core services, this change paves the way for upcoming revisions to the authentication scheme that will further consolidate role-based security and improve system integrity. Signed-off-by: SammyOina <sammyoina@gmail.com> * Enhance CLI security with key paths Removed the section on running computations from the CLI README as it may no longer be necessary or the functionality has been moved elsewhere. Required private key file paths for algorithm, dataset upload, and result retrieval commands to enhance security. This change associates each action with a specific identity, ensuring secure and traceable operations. Additionally, updated the manual test commands to reflect this new requirement. Signed-off-by: SammyOina <sammyoina@gmail.com> * fix ci Signed-off-by: SammyOina <sammyoina@gmail.com> * fix fmt Signed-off-by: SammyOina <sammyoina@gmail.com> --------- Signed-off-by: SammyOina <sammyoina@gmail.com> |
||
|
Sammy Kerata Oina
|
4ea9ff6531 |
NOISSUE - Add message broker on agent and manager (#17)
* Fix bug in agent state machine The bug in the agent state machine caused an error when attempting an invalid transition. This commit fixes the bug by properly locking and unlocking the state machine before and after transitioning to the next state. Additionally, the logger now correctly logs the current and next state during a valid transition. Signed-off-by: SammyOina <sammyoina@gmail.com> * Fix race condition in state machine The commit fixes a race condition in the state machine implementation in the `Start` method. The race condition occurs when multiple goroutines try to access and modify the state concurrently. To fix this, a mutex lock and unlock are added around the critical sections of code to ensure exclusive access to the state variable. This prevents race conditions and ensures the state transitions are executed correctly. Signed-off-by: SammyOina <sammyoina@gmail.com> * Fix race condition in StateMachine.Start() The StateMachine.Start() method was experiencing a race condition when multiple events were being processed concurrently. This was caused by not properly locking and unlocking the state machine before and after updating the state. This commit fixes the issue by adding proper locking and unlocking around the state update operation. Additionally, the logging statement has been updated to include the previous and next states for better debugging. Signed-off-by: SammyOina <sammyoina@gmail.com> * add magistrala dep Signed-off-by: SammyOina <sammyoina@gmail.com> * remove mainflux Signed-off-by: SammyOina <sammyoina@gmail.com> * Fix agentService New function to include messaging.Publisher parameter The agentService New function has been updated to include a messaging.Publisher parameter. This change allows the agent service to publish messages to a messaging system. The messaging.Publisher parameter has been added to the agentService struct and the New function signature has been updated accordingly. This change ensures that the agent service can communicate with other components using the messaging system. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor service.go state functions The commit refactors the state functions in the service.go file. The functions for each state have been modified to use the svc.publishEvent method to publish events with appropriate messages. - Refactor state functions in service.go - Use svc.publishEvent to publish events with messages for each state Signed-off-by: SammyOina <sammyoina@gmail.com> * Fix computation run event publishing and add pubsub functionality The computation run event publishing in the agent service was fixed to correctly call the publishEvent function. Additionally, the pubsub functionality was added to the manager package. - Fixed computation run event publishing in agent service - Added pubsub functionality to manager package Signed-off-by: SammyOina <sammyoina@gmail.com> * Fix license header in pubsub.go file The commit fixes the license header in the pubsub.go file. The copyright and SPDX-License-Identifier have been added to comply with the Apache-2.0 license. Signed-off-by: SammyOina <sammyoina@gmail.com> * Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger. The commit message should be: "Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger" Signed-off-by: SammyOina <sammyoina@gmail.com> * Fix Makefile to properly set DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE The Makefile has been updated to fix an issue with setting the DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE variables. The USER_REPO variable is now used to generate the DOCKER_PROJECT name following the Docker Compose guidelines. Additionally, the COCOS_MESSAGE_BROKER_TYPE variable is now properly set to "nats" if it is empty. This ensures that the correct values are used when compiling and installing the service. Summary: Fix Makefile to properly set DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE Details: - Update USER_REPO variable to generate DOCKER_PROJECT name - Set COCOS_MESSAGE_BROKER_TYPE to "nats" if empty Signed-off-by: SammyOina <sammyoina@gmail.com> * Fix Makefile Docker profile assignment and build flags The Makefile was updated to fix the assignment of the Docker profile and build flags. The Docker profile is now assigned based on the value of COCOS_MESSAGE_BROKER_TYPE, and if it is not provided, the default value is set to "nats". The build flags were also updated to include the COCOS_MESSAGE_BROKER_TYPE value as a tag for the Go build process. This commit addresses the issue with the Docker profile assignment and ensures that the correct build flags are used during the build process. Signed-off-by: SammyOina <sammyoina@gmail.com> * fix makefile Signed-off-by: SammyOina <sammyoina@gmail.com> * Fix notification topic in agent service and update NATS ports in Docker environment variables The agent service's notification topic was incorrectly set to "channels.manager" instead of "agent". This commit fixes the issue by updating the notification topic. Additionally, the NATS ports in the Docker environment variables were incorrect. The COCOS_NATS_PORT and COCOS_NATS_HTTP_PORT have been updated to the correct values. These changes ensure that the agent service uses the correct notification topic and the NATS ports are properly configured. Signed-off-by: SammyOina <sammyoina@gmail.com> * add pubsub Signed-off-by: SammyOina <sammyoina@gmail.com> * update protoc Signed-off-by: SammyOina <sammyoina@gmail.com> --------- Signed-off-by: SammyOina <sammyoina@gmail.com> |
||
 Drasko Draskovic
|
e9d143a7d3 |
Merge remote-tracking branch 'manager/main'
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com> |
||
 Darko Draskovic
|
b0b22aeed3 |
Add manual test procedure for CLI
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> |
||
|
Darko Draskovic
|
de9feccc51 |
Refactor run service func to create vm
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> |
||
|
Darko Draskovic
|
750bed0d76 |
Add run svc endpoint
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> |
||
|
Darko Draskovic
|
780c620b30 |
Add Makefile targets for build and copy to VM agent
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> |
||
|
Darko Draskovic
|
39c5cc2dd5 |
Add virtual machine creation
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> |