* feat(manager): start port search at minimum port range
When getting free port to be use we start at the minimum port range i.e 6000. This makes the agent url more deterministic since in most cases the agent port will be 6000
* fix(manager): Remove unused error channel
The error channel is not used
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
---------
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* feat(algorithm): Add wasm as an algo type
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* feat(algorithm): Use filesystem to store results
Move from unix socket for results storage to filesystem
* test: test new filesystem changes
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* refactor(files): rename resultFile to resultsFilePath
* feat(wasm-runtime): change from wasmtime to wasmedge
Wasmedge enables easier directory mapping to get results
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* feat(algorithm): send results as zipped directory
Create a new function to zip the results directory and send it back to the user
* fix(wasm): runtime argument
Fix the directory mapping for wasm runtime arguments
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* fix(errors): provide useful error message
* chore(gitignore): add results zip to gitignore
* feat(filesystem): Enable storing results on filesystem for python algos
* refactor: revert to upstream cocos repo
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* fix: remove AddDataset from algorithm interface
* fix: agent to handle results zipping
* test: test zipping directories
* refactor(agent): Handle file operations from agent
* test: run test inside eos
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* refactor(test): Document and test algos are running
Document steps on running the 2 python exampls and ensure they are running on eos
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* fix: remove witheDataset option
* test: test without dataset argument
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
---------
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* test(events): failing tests
* fix: Add datapath as optional parameter
If datapath is not provided if should work as assume dataset is empty
If an invalid i.e non existent path is given it should return an error
* feat(agent): Allow empty dataset
Allow running of algorithm with empty dataset since not all algorithms require datasets.
Allow state-machine transition from algo-received state to running state incase of no dataset provided
Fixes https://github.com/ultravioletrs/cocos/issues/168
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* chore(gitignore): Remove build artefacts
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* feat(algorithms): Add test algorithm for addition
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* refactor(addition): Modify addition algo to one file
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* fix(agent): move state transition to callback func
Move state transition from `receivingAlgorithm` to `running` to state call back function
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* feat(agent-event): Add `algoReceivedNoData` event
`algoReceivedNoData` is an event that is sent if we receive an algorithm and it should not have a dataset hence changes the state from `receivingAlgorithm` to `running`
* fix(agent-state): Change state depending on manifest
Change state from `receivingAlgorithm` to either `receivingData` if there is a dataset or `running` if there is no dataset provided
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
---------
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* refactor(env): remove internal env package
No need for this package since the parent package github.com/caarlos0/env does everything we need
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* refactor(jaeger): remove internal jaeger package
No need for this package since we can use magistrala exported package
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* refactor(metrics): remove internal metrics pkg
Use exported magistrala prometheus package
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* chore(dep): Update grpc and other dependencies
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* style(linter): remove enabled by default linters
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
---------
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* * fix(interceptors.go): change role from auth.DataProviderRole to auth.ConsumerRole in AuthenticateUser call
Signed-off-by: SammyOina <sammyoina@gmail.com>
* * test(interceptors_test.go): update role variable values in TestAuthUnaryInterceptor
* - Change role value from auth.DataProviderRole to auth.ConsumerRole in the "authorized result method" test case
* - Change role value from auth.DataProviderRole to auth.ConsumerRole in the "unauthorized result method" test case
* - Change role value from auth.DataProviderRole to auth.ConsumerRole in the "other method" test case
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* add host_data option
* add CLI hostdata option and rename platform to backend
* move code for computation hash to a function
* rename getComputationHash to computationHash
* add default for backend information field switch
* add cli option to download ask and ark
* add ARK and ASK to cert chain of the attestation report
* fix spelling mistake
* add explanation for aTLS testing
* remove commented code
* Implement gRPC auth interceptors and update dependencies
Introduced gRPC authentication interceptors for unary and stream calls in the agent's API to enhance security. These interceptors verify signatures based on roles before handling requests, rejecting unauthorized ones. The authentication logic was refactored, adding relevant error handling.
Added mocks for authentication interfaces to facilitate testing and replaced direct errors with wrapped ones to provide additional context.
The Makefile now includes a version pin for the Mockery tool, ensuring consistent mock generation across environments.
Updated the gRPC server setup in `grpc.go` to utilize the new interceptors. This change aims to reinforce the API's access control by authenticating users according to their roles and is expected to prevent unauthorized access to sensitive endpoints.
Lastly, dependencies in go.mod and go.sum were updated, most notably the inclusion of testify for improved testing capabilities.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix linting errors
Signed-off-by: SammyOina <sammyoina@gmail.com>
* * chore(Makefile): remove unused variables and update 'mocks' target
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
Added support for Docker in the system configuration file. This includes packages like libseccomp, CA certificates, Docker CLI, Docker Compose, Docker Engine, containerd, and runc.
Signed-off-by: SammyOina <sammyoina@gmail.com>
Corrected the struct field 'vnc' to 'Vnc' to match Go's public field naming conventions and to make it exportable. Each QEMU virtual machine's VNC display number now increments upon instantiation to avoid display conflicts.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* add rust program for fetching platform data
* fix new line error
* add CLI options to add the measurement to platform_info.json file
* add documentation for platform info testing
* add explanation for sev-snp-measure
* delete excess space
* fix minor errors
* fix minor errors
* add file permision constant
* Update Go to 1.22 and enhance security features
- Upgraded the Go version in GitHub Actions workflows to 1.22.x for latest features and security patches.
- Added RSA public key field `UserKey` in `Dataset` and `Algorithm` to reinforce data integrity and encryption.
- Refactored `Result` method in `agentService` to use `containsID` for improved readability and potential performance benefits.
- Updated `grpcserver.New` and `internal/server/grpc` invocations to pass `agent.Service` by value in line with recommended Go practices.
- Introduced `grpc.StreamInterceptor` with no args in `Server.Start` which seems to be an initial step for future stream interceptor configuration.
These changes prepare for stronger data security measures, maintain compatibility with the latest Go features, and improve code quality regarding service struct usage. Potential follow-up is needed to configure the stream interceptor and to ensure the new RSA key field is appropriately utilized in data handling.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor auth system and protocol buffers
Enhanced the authentication system by adding context support and an improved user-role model. Implemented robust RSA public key verification for users and a restructured interceptor logic specific to stream types, streamlining the auth process. Updated protocol buffers and associated structures to accommodate user keys as byte slices, aligning with standard cryptographic practice. CLI commands for algorithms and datasets now require a private key file path argument for signing, strengthening security during interactions.
This comprehensive overhaul addresses security and efficiency considerations in the RPC framework and aligns with best practices for key handling. By streamlining and securing the user authentication process, the agent service's reliability is greatly improved, directly impacting the robustness of the entire computation pipeline.
- Refactored auth: added role-based user validation, context handling
- Reworked interceptors: separated stream types, fortified signature checks
- Updated protocol buffers: user public keys as byte slices for standard compatibility
- Enhanced CLI: introduced private key argument, ensuring secure algorithm and dataset submission
- Improved server and SDK contracts to align with auth changes
Related issues:
- Implements user roles and auth context [#103]
- CLI security enhancement for private key management
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Updated PEM decoding for key parsing in CLI and tests
Added `encoding/pem` to decode PEM blocks when parsing private and public keys across CLI commands and test computation scenarios, ensuring compatibility with key files. This enhances robustness in key handling by supporting PEM encoded keys. The update also includes registration of a new Keys command in the CLI.
Refactored code is now compliant with common key formats, addressing potential parsing issues.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix auth signature encoding and improve CLI usage example
The authentication system now decodes base64 strings before verifying signatures to align with the expected format. Additionally, the signature generation now encodes the output in base64, ensuring consistency across the auth process.
The CLI help message for the `result` command is enhanced by providing a usage example, making it more user-friendly and informative.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor containsID to handle dynamic fields
Updated the `containsID` function to accept a field name parameter, enabling dynamic field lookup within the reflection logic. This change facilitates the use of the function for various struct fields, improving code reusability and flexibility.
CLI command 'data' now requires an additional argument for the private key file path, outlined in the usage example update, reinforcing command clarity and user guidance.
Resolves issues with hardcoded field lookups and enhances CLI usability.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Remove extraneous newline in key generation log output
A redundant newline after the success message in the key generation command was removed to clean up log output formatting. This change ensures a more consistent and professional appearance of the CLI tool's messages.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Implemented auth service in gRPC startup
Added authentication services to the gRPC server initialization to enforce security measures. The gRPC server's New function now includes an `authSvc` parameter, requiring instantiation of the auth service before starting the server. Failure to create the auth service results in a fatal error, halting the process to avoid running without protection. Tests have been updated to include `nil` values for the auth service parameter to maintain their functionality without authentication.
Refactored `grpcserver.New` to accept the new auth service, and updated the main agent startup logic to create and inject the auth service. Added the auth middleware interceptors to the server options, which ensures that each gRPC call will undergo authentication.
This change is a step towards secure communication, and affected components should now consider the authentication requirement.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor config read logic and update agent setup
Improved the configuration reading in `cmd/agent/main.go` to handle larger payloads by reading data in chunks and checking for EOF, ensuring that all config data is captured even if it exceeds the initial buffer size. Enhanced the `test/manual/agent-config/main.go` to require additional command-line arguments, improving the setup process by explicitly requiring paths for data, algorithm, and public key as well as a boolean for attested TLS. Also updated the hashing method to SHA3 for the algorithm and data files, and included the hash and public keys as part of the agent, dataset, and result consumer configurations. These changes will make the agent setup more robust and provide better integrity checks for the involved files.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor run method to agentService
Moved the run function into agentService for better encapsulation and maintainability. This refactoring includes capturing both stdout and stderr during algorithm execution, enabling more informative debugging through enhanced logging. Consequentially, the run method now references members through the service instance, aligning with object-oriented best practices and improving code coherence.
Resolves issue with insufficient execution details when computations fail.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor computation data handling to use filepaths
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor error logging and ensure consistency
Replaced usage of the standard log package with a custom logger for error reporting to standardize error logging throughout the application. Additionally, introduced graceful shutdown by returning from the main function rather than forcing exit when failing to create auth service, aligning the application's error handling strategy.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor auth initialization and key file handling
Improved the readability and maintainability of the authentication service initialization by adding line breaks for logical separation. Also, standardized key filenames in the CLI key generation by introducing constants, enhancing code clarity and reducing the likelihood of file-naming errors.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor auth verification logic for improved security
Removed an extraneous line in the `verifySignature` function that was not necessary for the signature verification process. This change simplifies the code and improves readability.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor payload structures to simplify API
Removed the 'provider', 'id', 'consumer' fields from protocol buffers, gRPC services, and related functions across various files to streamline the data model and align with the new authentication system based on cryptographic verification rather than string identifiers. This results in more efficient data handling and a reduction in unnecessary payload data, while enhancing security by making entity validation strictly cryptographic.
The changes affect agent-SDK interactions, CLI tools, and related services, ensuring only the necessary data (algorithm/data bytes, user keys, and hashes) is transmitted and processed. Consequently, the core computation algorithm and dataset handlers now rely on indexes derived from context to associate data with respective manifest entries, thus maintaining the ability to link to specific computation manifests without relying on explicit IDs in the payload. Additionally, refactored authentication methods now enforce role-based security seamlessly through metadata.
This approach enhances privacy by avoiding transmission of potentially sensitive strings over the network and by ensuring that only internal indices, not globally interpretable identifiers, are used to process computations.
Aligned with the broader architectural goal of simplifying and securing the platform's core services, this change paves the way for upcoming revisions to the authentication scheme that will further consolidate role-based security and improve system integrity.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enhance CLI security with key paths
Removed the section on running computations from the CLI README as it may no longer be necessary or the functionality has been moved elsewhere. Required private key file paths for algorithm, dataset upload, and result retrieval commands to enhance security. This change associates each action with a specific identity, ensuring secure and traceable operations. Additionally, updated the manual test commands to reflect this new requirement.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix ci
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix fmt
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
Enhanced the manual algorithm testing documentation by including steps to install additional dependencies and guidance on creating static binaries. This ensures wider compatibility and ease of deployment for python programs tested with this framework.
Signed-off-by: SammyOina <sammyoina@gmail.com>
Introduced a new command for the CLI to calculate the sha3-256 hash of a given file. The update enhances the utility's cryptography toolkit by allowing users to easily generate and verify file integrity directly through the command-line interface.
Signed-off-by: SammyOina <sammyoina@gmail.com>
Changed the internal representation of algorithms and datasets within the service from byte slices to file paths, writing received data directly to temp files. This modification allows for handling potentially large data sets without the need to load them entirely into memory, improving the memory efficiency and scalability of the service. Additionally, it aligns the call signature of external algorithms with the new approach, updating documentation and examples accordingly. Updated the linear regression example for consistency with the new data handling process.
Resolves issues with memory bloat when processing large datasets.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enable streaming RPCs for Algo and Data services
Modified the gRPC service definitions for Algo and Data methods to use stream processing, enabling the handling of larger datasets and algorithms without being limited by memory restrictions. This allows client and server to send chunks of data sequentially rather than requiring the entire payload to be loaded into memory at once.
Updated server implementations to accumulate data from multiple chunks, allowing for more efficient processing and communication when dealing with large files. Client implementations have been adjusted to segment and send data in a streaming fashion.
Removed previously existing synchronous client code as it became redundant with the new streaming approach, streamlining the client's communication patterns with the gRPC backend.
This change allows for better resource management, especially in systems with constraints on memory, improving overall scalability and performance of the data and algorithm processing pipeline.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor algorithm ID check logic
Simplify the algorithm validation logic in the agent service by replacing the previous containment check with direct ID comparison. This change streamlines the error handling for undeclared algorithms and hash mismatches, while also ensuring clear and direct provider validation. The modifications enhance the readability and maintainability of the code without altering functionality.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Updated README to build single-file executable with PyInstaller
Modified the PyInstaller command in the manual testing README to bundle the linear regression script into a single executable file. This simplifies distribution and execution of the script by eliminating the need for multiple dependency files.
Ref: Optimization of deployment process
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Switch to self-contained algorithms as executables
Transitioned from using Python scripts to self-contained binary executables for running algorithms, improving modularity and reducing dependencies. This change removes the reliance on a Python environment, as evident by the removal of Python setup and packages from the build configuration. The service now creates temporary executable files for algorithm runs, handling all permissions and cleanup, enhancing security and maintaining clean execution states. A warning is logged if computation fails, aiding in debugging. Additionally, updated manual tests to reflect these changes in the agent's handling of algorithms.
Refactors:
- Removed Python runtime const since it's no longer needed.
- Updated documentation and test commands to reflect the change from .py to .bin for algorithm files.
Build config:
- Removed Python and pip packages to reduce the build size and complexity.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update agent service.go file with new constants and file permission
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refine singular usage of 'algorithm' across modules
Standardized terminology throughout the project to refer to 'algorithm' in the singular form rather than plural. Streamlined various documentations, string constants, function names, and variable names to bring cohesiveness and eliminate ambiguity when handling algorithms across README files, CLI interfaces, and internal API representations.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix state names and indices in state_string.go
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Implement manager client heartbeat mechanism
Introduced a heartbeat system in the manager service to maintain an active link with client agents. The updates involve sending periodic heartbeat signals controlled via the new environment variable `MANAGER_HEARTBEAT_INTERVAL`, defaulting to 1 second. The protobuf definitions, service interfaces, and server logic have been updated accordingly to handle the incoming signals. This change allows better tracking of active client connections and could be used for features like auto-reconnect or resource cleanup for lost connections in the future.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enhance gRPC server to handle client termination
Introduce the capability for the gRPC server in the manager module to process termination requests from clients. Server and client proto buffers have been updated to include a Termination message, allowing clients to signal the server to stop processing. Handling for communication and termination flows involving context cancellation have been added across server implementation, manager client, and main execution logic to properly respond to termination signals. This ensures clean shutdowns and resource deallocation when a client requests termination, improving the robustness of the system.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor gRPC server context handling
Removed the redundant storage of context in grpcServer struct and modified the NewServer constructor to exclude the context parameter. Updated the Process method to derive context from the stream directly, ensuring better context scoping and adherence to proper gRPC patterns. This change simplifies the server's context management and ensures contexts are more accurately associated with their corresponding streams, which can improve debugging and request cancellation behavior.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update protoc to v4.25.3 and remove heartbeat system
Upgraded protoc version to 4.25.3, ensuring compatibility with the latest protocol buffer features and potential performance improvements. Additionally, removed the entire heartbeat system, involving both its gRPC server implementation and associated client-side logic. This elimination suggests a shift in the service health-check strategy, potentially to a more modern or efficient model.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Removed heartbeat functionality for manager service
The heartbeat feature and its related configuration have been removed across the application. This includes the removal of the Heartbeat method from the Service interface, the associated interval setting in the config structure, and related environment variable documentation. The change implies an architectural shift towards alternative methods of service availability checking or potentially a move to stateless service design. The cleanup removes unnecessary code, thus simplifying the overall service logic.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Introduce Terminate message and streamline gRPC server logic
- Added a new `Terminate` protobuf message to handle termination requests more explicitly.
- Removed redundant context cancels in gRPC server stream processing to rely on context propagation.
- Updated channel usage to match the altered stream message types; this aligns message handling with the updated protobuf definitions.
- Passed client authentication info to service run function to enhance the security during service execution.
- Utilized stream's context directly in error groups for improved clarity and error handling scope.
- Refactored message descriptors in protobuf to maintain correct ordering after introducing new message types.
These changes enhance the codebase's maintainability, simplify the logic, and align with the updated messaging protocol.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update agent configuration in main.go
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor single algorithm processing
Simplified the agent service's algorithm handling logic to process a single algorithm instead of multiple. This change:
- Removed the `Algorithms` type and associated stringer implementation.
- Updated the state machine and service logic to expect a singular algorithm, aligning the agent's internal state transitions with the new model.
- Adjusted the manager service and computations test server to mirror these changes in their respective payload structures, ensuring API and test consistency.
- Altered README files to reflect the simplified interaction model and removed outdated descriptions.
- Reverted the protoc-gen-go version used for generating protobuf files to maintain compatibility with the rest of the codebase.
The single-algorithm approach streamlines the computation running process, reducing complexity and potential error conditions. It directly impacts how external services will construct and send computation requests.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update protoc-gen-go version to v1.33.0
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor variable name in computations.go and grpc.go
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* added initial code for attested TLS
* added client validation and verification
* fixed bugs for attested TLS
* updated README for manual testing
* fixed CI errors
* removed SNP pollicy from agent config
* added attested TLS config param to AgentConfig
* generated manager.pb.go for protoc v25.2
* updated proto-gen-go version on CI
* generated agent.pb.go to match newest proto gen version
* define errors for error handling
* fixed comments
* Update protoc to v4.25.3 and streamline client identification
Protocol Buffer compiler `protoc` has been upgraded to version 4.25.3. This update unifies the version across multiple generated files to ensure compatibility and take advantage of any bug fixes and performance improvements in the new release.
Additionally, the client identification process has been refined. The redundant `WhoAmI` message and the corresponding checks have been removed from the gRPC server implementation. Clients are now identified via their address from the `Process` stream's context as soon as a connection is established, simplifying the code and potentially reducing handshake time.
This change sets the foundation for a leaner communication protocol between manager and agents, and could contribute to lower latencies in client-server interactions.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix goroutine bug in grpc server
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update PROTOC_VERSION to 25.3
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor server reference in readme
Updated the deployment instructions in the README to point to the correct server location after consolidating test server documentation. Removed obsolete `manager-server` module and associated main entry point to align with the new architecture and streamline workflows.
Resolves issues with outdated links and cluttered repository structure.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enhanced state machine synchronization
Introduced a wait group to the state machine to manage go-routine synchronization effectively. The Start method now ensures that the state machine's execution is tracked, preventing potential early termination of go-routines during state transitions. The test suite is updated to wait for the go-routine to complete before proceeding, ensuring more reliable test execution.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Remove unnecessary code and fix test case in agent package
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Standardize attestation report data size to 64 bytes
Redefined the report data for attestation requests from a variable byte slice to a fixed 64-byte array across various critical components. This change enforces a standardized data length, simplifying validation logic and ensuring consistency when handling the attestation data for both the server and client end-points, logging functionalities, and the CLI tool. Updated attestation data handling in the SDK to accommodate the new fixed-length constraint.
By strictly adhering to the 64-byte requirement, the modifications promote robust input checking and prevent potential issues related to dynamic data length processing.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor attestation validation logic
Streamlined attestation validation in the CLI with a new configuration parsing approach. Introduced a retry mechanism with timeout and backoff for HTTPS getters that facilitate remote trust validation. Clarified expected data lengths in proto comments across attestation and manager services, ensuring consistency and correctness of the cryptographic data fields.
This modification enhances maintainability by replacing a verbose struct with a configurable object model, thus simplifying parameter handling. Robustness is improved through failure-retrieval strategies during remote validation, and additional in-code documentation specifies cryptographic constraints for critical data elements.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add new attestation validation flags
Enhanced attestation validation in the CLI with additional flag support for FAMILY_ID, IMAGE_ID, REPORT_ID, REPORT_ID_MA, MEASUREMENT, CHIP_ID, and TCB-related fields. This update enables fine-grained control over attestation report criteria, ensuring stricter validation conforming to protocol requirements. Adjusted the default value for MinimumGuestSvn to align with the new policy specifications.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enhance attestation policy configuration
Introduced several new configuration flags to the CLI's attestation policy, including options for minimum AMD-SP firmware builds, revocation list checking, and network restrictions. Expanded trust criteria parameters with the support for trusted author and identity key hashes. The default VMPL setting is now explicitly initialized. These changes improve security controls and offer more detailed attestation verification settings.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Simplify attestation validation process
Refactored the attestation CLI command to improve usability by removing the need for explicitly provided report data as an argument. Default values for configuration parameters have been centralized into constants for maintainability. Additionally, integrated data parsing functions to streamline the validation checks with robust error handling for cases such as invalid hex strings and file I/O issues.
This enhancement reduces user error and the complexity of entering attestation data by parsing relevant information from within the environment, all while maintaining the same security standards.
The process of marking flags as required for CLI commands has been corrected to prevent runtime errors and improve command reliability.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refined attestation validation logic
Enhanced attestation command handling by removing the hardcoded report data length and replacing it with the standard SHA-512 hash size, ensuring dynamic compatibility with hash lengths. Introduced size constants to improve code readability and enforce explicit length checks on attestation input data, raising errors when requirements are unmet. This preventative measure aims to avert potential runtime errors stemming from unexpected data sizes. The change also simplifies the minimum guest SVN description for clarity.
References to removed unnecessary whitespace maintain code cleanliness. These adjustments culminate in a more robust and maintainable attestation process, in accordance with the best practices for secure handling of cryptographic data and compliance with AMD's ABI format specifications.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enhanced attestation documentation and command syntax
Updated CLI documentation to include detailed descriptions of the attestation retrieval and validation process, along with a comprehensive list of new flags for the validation command, providing users with extensive control over the attestation verification process. Updated command syntax with flags for specifying report data, improving command clarity and consistency. These changes ensure better user guidance and offer a more robust attestation handling experience.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enforce report data size validation uniformly
Refactored report data size validation to ensure it is performed consistently across the system. Removed the hard-coded length check from attestation request validation in favor of a centralized verification based on the sha512 digest size. Also, eliminated fallback mechanism to the SHA-512 hash of report data and made the length requirement explicit, leading to immediate failure if not met. This harmonizes the error handling logic, reinforces data integrity checks, and simplifies debugging by removing ambiguous length correction behavior. Adjusted related validation functions to use a common utility, enhancing maintainability.
Fix spelling errors in log messages for attestation validation commands to improve clarity in output.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Initialize attestation config with defaults
Enhanced the attestation command initialization by setting default values for the `Config` struct to prevent nil pointer exceptions and potentially streamline configuration handling. Moreover, corrected a flag declaration by removing shorthand.P support for 'permit_provisional_software', aligning it with other flags for consistency. This change should improve stability and user experience while configuring attestation policy.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* nil check prior
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix typos
Signed-off-by: SammyOina <sammyoina@gmail.com>
* use file for attestation report
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refine attestation validation with default policy values
Adjusted default values and constants related to the attestation configuration for enhanced clarity and consistency. Changed the default minimum guest SVN value to align with new policy requirements and introduced a default guest policy constant. Additionally, standardized byte array placeholders for attestation validation fields to prevent potential issues with uninitialized bytes and ensure robust input validation checks. These changes promote more intuitive and secure attestation validation procedures.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refine attestation defaults and flag requirements
Updated the default guest policy to use hexadecimal notation for clarity and added a default minimum version for the firmware API version flag. Made 'chip_id', 'measurement', 'report_id', and 'report_id_ma' flags mandatory to ensure critical attestation data is provided by the user, enhancing validation robustness. Added a check for 'measurement' field length for consistency in input validation.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Optimize attestation file format
Standardized attestation file format across the codebase by moving the 'attestationFilePath' constant from the function scope to a higher scope within the package. Switched the file extension from .txt to .bin to reflect binary data storage, improving consistency and potential performance benefits from binary file handling.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Optimize default values for attestation validation
Removed unnecessary empty initializers and replaced hardcoded empty values with nil for optional fields in attestation validation flags. Introduced a constant defaultReportIdMa for the REPORT_ID_MA field to provide a meaningful default instead of an empty one. Streamlined the command setup by removing the requirement flags for fields that are now optional and unchecked if unset, reducing burdens on the user for providing unnecessary input.
The change simplifies the command's usage and eliminates the need for users to specify values for certain fields that can be optional, improving the command's flexibility and user experience. Additionally, setting a specific default for REPORT_ID_MA ensures clarity in cases where this is not provided by the user.
This update enhances the maintainability and usability of the attestation validation functionality.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor hardcoded attestation report size
Introduced a constant `size64` to replace the previously hardcoded array size of `64` used across various Attestation methods. This change enhances code maintainability and readability by centralizing the size definition, making it easier to update in the future if necessary.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Updated protocol buffer dependencies and cleaned CLI flags
Protobuf compiler versions were updated to v4.25.2 across various RPC service definitions for agent and manager packages, ensuring compatibility with the latest features and fixes. In the CLI attestation code, unnecessary flags (`DisallowNetwork` and `PermitProvisionalSoftware`) were removed to streamline user options, reflecting a more secure and user-focused configuration interface. Added an input validation check in `attestation.go` for CA bundle presence when a product name is set, enhancing the reliability of attestation validation. Also rectified a bug in `grpc.go` by fixing an incorrect variable assignment for reading the key file content, thereby preventing potential TLS-related errors.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Updated protoc version and adjusted proto file paths
Bumped the protocol compiler version to 25.2 to align with our dependency updates and ensure compatibility. Additionally, revised the proto file paths under the continuous integration setup, moving from 'manager/' to 'pkg/manager/' to reflect the recent directory restructuring. The changes ensure that proto file checks and comparisons are conducted in the correct file locations, preventing potential build and sync issues in future developments.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enhance hash size consistency in gRPC attestation
Aligned the hard-coded byte array size for `ReportData` with `sha512.Size` constant to ensure consistency and maintainability in gRPC attestation requests and responses. This change mitigates the risk of future errors if the SHA512 hash size standard is altered.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Optimize agent-config listener loop
Removed the goroutine wrapping the listener loop in 'agent-config' manual test to streamline and simplify the connection handling logic. This change ensures the loop runs on the main goroutine, improving the readability and maintainability of the code.
Refactoring assists in avoiding potential race conditions and makes the server's flow easier to understand for future debugging and development.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor report data size checks to use constant
Centralize the report data size definition by replacing various size checks and array declarations with a constant `ReportDataSize`. This streamlines code maintenance and ensures consistency across all instances where report data size is validated or used. The changes remove direct references to the hash function output size, decoupling the report data size from the hash function's characteristics and allowing easier updates if the data size requirement changes.
These modifications bolster the codebase's flexibility for potential adjustments in security protocols or data handling specifications.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor config JSON handling in attestation CLI
Introduced an `exampleJSONConfig` constant containing a sample configuration to improve code readability and maintainability. Previously, the example JSON configuration for the attestation CLI was an inline string, making the code cluttered and less maintainable. With this change, the example configuration is now stored as a constant, resulting in cleaner command flag setup and enhanced clarity. This constant is used in the command flag description to guide users when providing their custom configuration. Additionally, refactored the command flags related to the configuration by aligning and sorting them for better code organization.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
Updated the hashing algorithm from SHA-256 to SHA-3 for improved security and reliability in the computation service. Modified the service to accept dataset and algorithm paths as command-line arguments, enhancing flexibility and user control. Increased robustness of the service's error handling by incorporating file read validations and updated usage instructions in both the README and the program's startup error messages.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* added SEV-SNP support
* updated documentation for SEV-SNP
* changed variable names to adhere to the GO naming convention
* changed the variable name from V-sock to Vsock
Implemented new GitHub templates to streamline the contribution process. Bug report and feature request templates guide users in creating detailed and actionable issues, while a universal pull request template ensures systematically structured submissions. These additions facilitate more efficient reviews, and prioritize issues effectively. Additionally, configured the repository to disallow blank issues, encouraging use of the templates for better project management.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enforce 32-byte length check for hashes
Introduced a validation step in the manager service to ensure both algorithm and dataset hashes are of byte length 32 before provisioning VMs, preventing runtime errors due to invalid hash lengths. The test manager-server now generates a valid 32-byte hash using SHA256 for mock data to align with the new validation requirement.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* use equal comparison
Signed-off-by: SammyOina <sammyoina@gmail.com>
* use a constant
Signed-off-by: SammyOina <sammyoina@gmail.com>
* use constants
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add hash verification for datasets and algorithms
Enhanced data integrity checking by incorporating hash fields in Dataset and Algorithm structures, and modified the corresponding service logic to validate these hashes during processing. The update includes SHA-3 for hash computation, replacing the former SHA-256 usage, and ensures the provided data matches the expected hash from the manifest to prevent processing malformed or corrupted data.
- Introduce `Hash` field to both Dataset and Algorithm structs to store the expected hash value.
- Implement SHA-3 hashing within service methods that process the data, ensuring consistency with newly added `Hash` fields.
- Add error handling for hash mismatches, preventing further processing and alerting to potential data integrity issues.
- Update Protocol Buffers serialization to accommodate the new hash fields for gRPC communication.
- Modify manager service's Run method to pass the hash information when creating agent configurations.
Go module dependencies were updated to include the new SHA-3 package and upgrade Go version to 1.21.6 for improved stability and compatibility.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Remove identifiers from protobuf and related code
The protobuf definitions and related service handling code have been revised to drop specific identifier fields (`AlgorithmID` and `DatasetID`) to simplify API responses and internal function signatures. These removals streamline the overall data flow between components, reduce unnecessary data transmission, and lead to an aligned server-client expectation where identifiers are no longer a part of the response payload. Consequently, these changes simplify the logic within various functions and client commands, reinforcing encapsulation by ensuring that internal identifiers do not need to be managed or exposed unnecessarily.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix lint
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* add handler
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor gRPC and Protobuf integration for manager service
- Shifted Protobuf message definitions to a separate package `pkg/manager`.
- Updated references throughout the codebase to import and use the new package for gRPC service definitions.
- Enhanced AgentLog message with additional fields `level` and `timestamp`.
- Removed direct dependencies on old Protobuf-generated types in favor of the new package.
- Deleted obsolete Protobuf-generated files as they are now superseded by the new `pkg/manager`.
- Streamlined event publishing and gRPC handling in the manager service to use the updated Protobuf messages.
This refactoring improves modularity by centralizing Protobuf message definitions and decouples internal representation from the gRPC interface, aligning with best practices for microservice architecture. Additionally, the enriched logging structure paves the way for more detailed and fine-grained log analysis.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor vsock event/log handling and config
Streamlined event and log services in the manager by moving vsock listening functions out of `managerService` initialization and into dedicated `RetrieveAgentEventsLogs` methods. This change decouples the manager service creation from the actual start of log listening, adding clarity and flexibility in service management. Also moved logging middleware invocation outside of network handling loops to avoid unnecessary overhead.
Additionally, the agent's vsock port configuration is now dynamically passed to the `New` function in the `events` package instead of relying on a hardcoded constant, allowing for greater configurability and testability.
Finally, updated message structures for event and log sending to conform with the `ClientStreamMessage` definitions. These modifications should improve parsing and handling consistency and prepare our system for future enhancements related to inter-process communication.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix linting errors
Signed-off-by: SammyOina <sammyoina@gmail.com>
* correct path to generated files
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix comments
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove uneccessary comments
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Implemented mTLS support across services
Extended gRPC configuration to support mutual TLS (mTLS) in agent and manager components for enhanced security. This includes the loading of Certificate Authority (CA) certificates, server, and client certificates, and keys. Updated README documentation to reflect the new environment variables required for mTLS configuration. Additionally, streamlined secure gRPC client connection setup and logging messages to indicate whether a service is running with TLS, mTLS, or without TLS.
The change ensures secure communication between services by verifying both client and server identities, thus addressing potential security concerns in network-level interactions.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enhance agent cert handling and update copyright
- Implement function to create certificate files for the agent configuration dynamically, ensuring file paths are updated to reflect newly created files. This improves the agent's setup process by automating the certificate handling.
- Update copyright clause to reflect the new owning entity, Ultraviolet, affirming correct attribution and compliance with legal requirements.
- Refactor gRPC client connection code to remove redundant package alias, streamlining the codebase and improving readability.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor cert loading with fallbacks
Removed redundant certificate file creation logic in the agent module and introduced a more robust loading mechanism in the gRPC server module to support direct byte content aside from file paths. This change simplifies the initial setup process for the agent by removing the need to create certificate files preemptively, thereby streamlining deployment in environments with varying filesystem access. It supports using certificate contents directly, enhancing compatibility with in-memory configurations or environments where file storage may not be ideal.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix lint
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enhanced gRPC server concurrency with errgroup
Refactored the gRPC server's `Process` method to handle incoming messages and service execution concurrently using an `errgroup.Group`. Added a context to the `grpcServer` struct and propagated it to support graceful shutdowns and better request handling. Changed the `Service` interface's `Run` method to use a channel for transmitting `ComputationRunReq`, facilitating non-blocking operation and streamlining the request-response cycle.
Asynchronous handling of stream requests with `errgroup` allows simultaneous processing and error management, improving the server's responsiveness and fault tolerance. This update paves the way for more robust service implementations that require concurrent operations.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove trailing space
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor GRPC manager service and client
The manager service and client have been restructured for stream communication, facilitating real-time agent events, logs, and run responses. The `Run` RPC is replaced by the `Process` stream RPC, enabling bidirectional streaming between clients and the manager service. This allows continuous interchange of different message types including `WhoAmIRequest`, `AgentLog`, `AgentEvent`, and `RunResponse`.
Several message types have been adjusted and new fields introduced, like `AgentPort` in `RunResponse` and various agent-config attributes including CA files and instance IDs, to support TLS client authentication and distinguish between agent instances.
We've also incorporated `google.protobuf.Timestamp` in `AgentEvent` for precise event logging. The client code reflects these modifications with updated method calls and stream handling logic for ongoing communication. Moreover, the updates necessitate corresponding changes throughout service, grpc, and sdk layers to interoperate with the new streaming approach.
The transition to streaming paves the way for a more interactive, flexible communication system that can accommodate future expansion and real-time monitoring features.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* add computation id to run response
Signed-off-by: SammyOina <sammyoina@gmail.com>
* rename request
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor GRPC manager service and client
The manager service and client have been restructured for stream communication, facilitating real-time agent events, logs, and run responses. The `Run` RPC is replaced by the `Process` stream RPC, enabling bidirectional streaming between clients and the manager service. This allows continuous interchange of different message types including `WhoAmIRequest`, `AgentLog`, `AgentEvent`, and `RunResponse`.
Several message types have been adjusted and new fields introduced, like `AgentPort` in `RunResponse` and various agent-config attributes including CA files and instance IDs, to support TLS client authentication and distinguish between agent instances.
We've also incorporated `google.protobuf.Timestamp` in `AgentEvent` for precise event logging. The client code reflects these modifications with updated method calls and stream handling logic for ongoing communication. Moreover, the updates necessitate corresponding changes throughout service, grpc, and sdk layers to interoperate with the new streaming approach.
The transition to streaming paves the way for a more interactive, flexible communication system that can accommodate future expansion and real-time monitoring features.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix lint
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update GitHub Actions to Latest Versions
Upgraded GitHub Actions 'checkout' to version 4 and 'setup-go' to version 5 across various workflow files to leverage the latest features and improvements for better performance and reliability. This also ensures compatibility with Go version 1.21.x which is specified in the workflows.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor event handling and logging
Reworked event and log processing to use channels instead of direct HTTP calls. Removed obsolete events package and consolidated event structures, leading to cleaner and more maintainable code. Updated agent events to use channels, enhanced error handling in log forwarding, and simplified manager `New` function signature to accept an event channel directly.
- Removed `events` and `agentevents` packages to reduce complexity.
- Replaced direct event server communication with internal channel usage.
- Introduced `AgentEvent` struct in events.go for standardized event objects.
- Adapted `managerService` to dispatch events and logs through channels.
- Streamlined manager construction by removing the now-unnecessary event service and host IP parameters.
This change results in a more robust and easier to extend event and log management system within the agent-manager interaction.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix ci
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove unused code
Signed-off-by: SammyOina <sammyoina@gmail.com>
* add comments
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Expand CLI with Manager Service and Enhanced Attestation Commands
The CLI has been updated to interact not only with the Agent service but also with the Manager service, reflecting an expanded scope of operations. As part of this update, 'manager' subcommands have now been incorporated, ensuring users can seamlessly initiate computations through the manager service using the CLI.
Furthermore, attestation functionality has greatly improved. In addition to retrieving attestations from the agent, users can now validate and verify them directly from the CLI, bolstering security measures. New subcommands under 'attestation' provide clear and specific actions for attestation management.
Additionally, the command structure has been refactored to categorize commands under 'agent' or 'manager' prefixes, promoting better command organization and a more intuitive user experience.
These enhancements are part of ongoing efforts to provide robust and secure tools for service interaction and computational task management in distributed networks.
Relevant documentation adjustments and example usage have been updated in README files to align with these functional improvements.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* capitalize comments
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
The HTTP server-related code, documentation, and configurations have been removed as part of a shift towards prioritizing gRPC for service communication. This update includes deletions of HTTP host and port configs across various components, the manager HTTP API alongside its Swagger definition, and the removal of related scaffolding and utility code. This change simplifies the overall architecture and eliminates redundant HTTP support, focusing on optimizing gRPC performance and security features.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Improve attestation and update deps
Enhanced the attestation process to require additional report data and a variable privacy level, catering to varying security requirements. The `Run` method's context parameter was removed to simplify invocation following API alignment. Dependencies got updated, notably adding `github.com/google/go-sev-guest` for attestation, and some were indirectly bumped. These updates ensure the agent service leverages the latest available features and improvements for security and stability.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Extend AttestationRequest with data fields
Enhanced the AttestationRequest message structure by introducing 'report_data' and 'vpml' fields to encapsulate additional attestation context. Adjusted service and middleware layers to incorporate these new parameters, enabling richer attestation workflows. The GRPC endpoint, logging, and metrics layers have been updated to handle the additional data, catering to more secure and nuanced attestation use cases.
Refactors Attestation command in CLI to accept report data argument and include an optional 'vpml' flag for specifying virtual machine privilege level. This change empowers users to request attestation with specific context directly from the CLI.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor attestation to remove vpml dependency
Removed the `vpml` field from the AttestationRequest protobuf, and all associated handling code, streamlining the attestation process. Code adjustments were made in gRPC client, server, and service layers to ensure proper request construction and validation that focuses only on report data. This change simplifies the attestation API and its usage, reducing error risk and improving maintainability. It reflects a decision to decouple the protocol from specific vendor measurement levels, potentially due to a shift towards a more generalized attestation procedure.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enhance CLI usability and streamline SDK attestation
- Added a usage example to the CLI attestation command for better user guidance.
- Simplified the SDK attestation function by removing an unused parameter and directly populating the request structure with report data to align function parameters with actual usage requirements.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add build and release GitHub workflow for tagged commits
Introduced a new GitHub Actions workflow to automate building and releasing for tagged commits. The workflow checks out the required repositories, builds the project using Buildroot configurations, and creates a release with the resultant kernel and rootfs artifacts. This streamlines the release process, ensuring consistent and reproducible builds.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update GitHub Actions to Latest Versions
Upgraded GitHub Actions in the workflow to latest or more recent versions to leverage improvements and maintain compatibility. This includes using version 4 for checkout steps and switching to 'latest' for release creation and asset upload actions, ensuring the use of up-to-date features and potential bug fixes.
Ref: Internal upgrade protocol for CI/CD consistency.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix workflow
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Optimized checkout step in CI workflow
Ensured the checked-out repository's reference matches the current workflow's triggering reference. This alignment can result in more predictable builds, especially when the workflow is triggered by pushes to branches other than the default or by tagged commits.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* test repo
Signed-off-by: SammyOina <sammyoina@gmail.com>
* test modified module
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update and upgrade Ubuntu in GitHub actions workflow
Ensure the GitHub actions runner has the latest Ubuntu updates before checking out the repository. This change significantly reduces the chances of encountering bugs or security vulnerabilities stemming from outdated packages, creating a more reliable and secure CI/CD environment.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* restore repo
Signed-off-by: SammyOina <sammyoina@gmail.com>
* install golang
Signed-off-by: SammyOina <sammyoina@gmail.com>
* create tags only for release
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Ensure graceful shutdown and improve connection handling
Refined the network connection handling in both agent events and logs to continuously process incoming data in a loop, enabling the services to handle more than a single message per connection. Additionally, instituted a deferred close operation for the event service to guarantee resources are cleanly released upon the application's termination.
Resolves potential resource leakage and enhances log processing efficiency.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Improve robustness in event and log handling
Altered handling in agent event and log services to continue processing incoming data rather than returning early upon encountering errors. This ensures that a single erroneous data point does not prematurely halt the processing loop, improving the robustness and reliability of the services. Potential errors are now reported and logged, yet the system remains operational to handle subsequent data.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Upgrade gRPC telemetry to use StatsHandler
Switched from using gRPC's UnaryInterceptor for telemetry to the more comprehensive StatsHandler provided by otelgrpc. This enhances telemetry collection by allowing the capture of a wider range of RPC stats, leading to improved monitoring and observability of the gRPC server.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* use constants
Signed-off-by: sammy <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
Signed-off-by: sammy <sammyoina@gmail.com>
* Remove Docker-related build functionality
Removed the Docker build scripts, Dockerfile, and docker-compose definitions from the Makefile and relevant directories. Updated documentation by stripping out references to building and running Docker containers. This change likely reflects a shift towards a different deployment strategy or a move away from Docker as a dependency for builds and runtime. This simplification could lead to less complexity in the build process and reduce maintenance overhead associated with Docker configurations.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Removed Docker image build and publish workflow
The workflow for building and publishing a Docker image on pushes to the main branch has been removed. This step likely reflects a change in deployment strategy or a move to a different CI/CD pipeline. Continuous deployment might now be managed by another service or process, adhering to updated operational requirements or infrastructure changes.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor CLI commands to use CLI struct
Introduced a CLI struct to encapsulate dependencies such as agentSDK and managerSDK previously passed to command constructors. This shift towards a more object-oriented approach simplifies command creation and enhances code readability. CLI-related commands are now methods on the CLI struct, accessing shared services directly, which streamlines the setup of CLI command hierarchies and reduces the need to pass dependencies around.
Removed the OpenAPI specification file as it was likely deemed obsolete or no longer necessary to be included with the CLI binary, suggesting a possible shift in how the API is managed or documented.
Lastly, the main application now sets up two distinct command groups for 'agent' and 'manager', with respective commands organized under them, making the CLI tool's structure more intuitive for users.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix ci
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Simplify event handling and config
Streamlined event service interface by consolidating `SendEvent` and introducing `SendRaw`. Removed `notification_server_url` and `instance_id` parameters from several event publication calls to leverage centralized event construction. This change not only cleans up redundancy in event-related code but also simplifies the configuration data flow across the system, making it easier to manage and less error-prone. Uniform event generation now improves consistency and maintainability.
Refactored configuration management in the agent and manager services. Removed notifications URL from the agent configuration, relying on a simplification that assumes a single source of events. Updated Manager Port to VsockConfigPort for clarity and consistency across vsock communication.
These modifications should facilitate easier integration and extension of event and configuration systems in the future.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix lint
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor error handling in agent event forwarding
Introduced context and error channel handling to the agent event forwarding process. The logger now warns on errors during forward operations asynchronously, allowing for non-blocking error reporting. Additionally, reliance on the global logger was removed in favor of passing error information via channels, improving modularity and error flow control.
Resolves issue with silent forwarding failures by providing a means to alert system operators without halting the service. This enhancement makes the error reporting more robust and reactive while maintaining service continuity.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove unused field
Signed-off-by: SammyOina <sammyoina@gmail.com>
* update package name
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Simplify event handling and config
Streamlined event service interface by consolidating `SendEvent` and introducing `SendRaw`. Removed `notification_server_url` and `instance_id` parameters from several event publication calls to leverage centralized event construction. This change not only cleans up redundancy in event-related code but also simplifies the configuration data flow across the system, making it easier to manage and less error-prone. Uniform event generation now improves consistency and maintainability.
Refactored configuration management in the agent and manager services. Removed notifications URL from the agent configuration, relying on a simplification that assumes a single source of events. Updated Manager Port to VsockConfigPort for clarity and consistency across vsock communication.
These modifications should facilitate easier integration and extension of event and configuration systems in the future.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix lint
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor error handling in agent event forwarding
Introduced context and error channel handling to the agent event forwarding process. The logger now warns on errors during forward operations asynchronously, allowing for non-blocking error reporting. Additionally, reliance on the global logger was removed in favor of passing error information via channels, improving modularity and error flow control.
Resolves issue with silent forwarding failures by providing a means to alert system operators without halting the service. This enhancement makes the error reporting more robust and reactive while maintaining service continuity.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove unused field
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enhance agent logging via vsock connection
Redirected agent logging to use a vsock connection instead of standard output, improving the process isolation and enabling centralized log management. The change involved dialing to the specified vsock log port and initializing the logger with the vsock connection rather than stdout.
Additionally, the manager service now maintains a map of agent vsock cids to computation IDs, providing better tracking of computation resources. A routine to retrieve logs from agents was also initiated during the service setup to facilitate log collection.
Consequential to these changes is the removal of a redundant os package import in the agent's main.go, further cleaning up the dependencies.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fail gracefully
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Updated backoff strategy and VM configurations
- Added `github.com/cenkalti/backoff` to direct dependencies for robust retry logic in agent configuration sending.
- Modified the vsock logs port to align with the updated port range standards.
- Enclosed kernel console arguments in quotes to ensure proper parsing in QEMU configurations.
- Implemented exponential backoff when sending agent configurations to handle transient failures.
Refactors:
- Streamlined creation of `AgentConfig` within the computation setup to avoid unnecessary initializations when `c.AgentConfig` is not nil.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor command execution and improve argument construction
Consolidated the error handling in the command execution function for better readability. In the QEMU configuration, the argument assembly process is enhanced for clarity and correctness; the VNC parameter is now separate, and string quoting is handled properly for kernel parameters. These changes result in more maintainable code and prevent potential formatting issues during QEMU argument parsing.
Resolves issues with argument construction in QEMU config module.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refine default config handling and unpacking
Improved the agent configuration by dynamically setting default values for the log level and port if they are not specified in the incoming configuration. Also streamlined configuration unpacking in the endpoint and service layers, reducing redundancy and ensuring all required fields are correctly copied over to the Manager's configuration structure. This change ensures better fault tolerance and more maintainable code by handling edge cases where configuration values might be missing.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* rename dir
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix lint
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Ensure runRes.Empty() reflects non-empty state
Changed the always-true return value of the `runRes.Empty()` method to `false` to accurately indicate the presence of a response body. This adjustment ensures downstream handling of API responses aligns with actual content state.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Replace mglog with slog across codebase
Updated various components to replace the `mglog` logger implementation with the `slog` logger. This change affects logging initialization and calls throughout the codebase including the agent, manager, and internal server components. Transitioning to `slog` is part of a broader shift to standardize the logging mechanism to improve maintainability and consistency.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Optimize QEMU launch and add V-sock support
Refactored QEMU argument construction and launching logic by removing the dependency on 'agent.Computation'. This simplification makes the VM creation process more streamlined. Additionally, introduced V-sock capabilities in the QEMU configuration to facilitate improved guest-host communication. Updated the README to include kernel module setup instructions for the new V-sock feature.
The V-sock implementation enables VMs to use a consistent communication channel that is not affected by network configuration changes, enhancing reliability and potential interoperability with host services. It's important to ensure that the necessary kernel modules are loaded as part of the setup process, as documented.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add vsock-based communication to manager
Introduced virtual socket (vsock) communication abilities in the manager package by implementing a new socket service. This includes establishing a vsock listener and stub methods for sending computation results and cleaning up resources. The addition provides the groundwork for interprocess communication between guest and host in virtualized environments.
- Integrated the `mdlayher/vsock` library for handling virtual socket operations.
- Created a new `sockService` struct to encapsulate vsock listener handling.
- Implemented `NewVsock` constructor to initialize the listener with domain value `3`.
- Added placeholder methods for future computation sending and service closing logic.
This enhancement targets scenarios where efficient VM-to-host communication is required.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove env
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor agent config and use vsock
Introduce `AgentConfig` struct to group agent-related configurations, and update `Computation` struct to include the new `AgentConfig` field. Replace command-line computation extraction with vsock-based config retrieval for robustness and decoupling. The agent configuration is now read from a vsock connection during runtime, allowing for more dynamic and flexible deployments. Adjusted the main agent application logic to support these configuration changes, and corresponding changes have been made in the manager to facilitate vsock communication.
This approach aligns with modern practices for microservices by streamlining configuration management and reducing reliance on static command-line parameters. Moreover, it enhances the scalability of the agent service by allowing configuration to be managed externally.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor agent config and remove deprecated code
Consolidated agent configuration management into a single `AgentConfig` message and pruned deprecated Protobuf `ComputationReq`, `DatasetReq`, and `AlgorithmReq` messages. Adapted corresponding manager service logic to the new configuration structure. These modifications align with updated manager API schema, facilitate clearer configuration handling, and improve maintainability.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* send configuration
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Switch agent to listen mode for manager connections
Previously, the agent established a connection to the manager using a direct dial. This change shifts the setup to where the agent listens on a specified port and accepts incoming connections. It ensures that the agent properly handles incoming requests by initiating a listening socket and waiting for the manager to connect, enhancing the system's flexibility in connection management. This adjustment also includes graceful closure of the listening socket.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add python pip to buildroot
* Use DHCP for agent
* Use systemd for naming network interfaces
* Removed eth0 dhcp setting
* Add ETH_IFACE variable
* Added explanation for ETH_IFACE name
* Further explain ETH_IFACE value
* Add guest kernel support for VSOCKETS
* Optimize QEMU launch and add V-sock support
Refactored QEMU argument construction and launching logic by removing the dependency on 'agent.Computation'. This simplification makes the VM creation process more streamlined. Additionally, introduced V-sock capabilities in the QEMU configuration to facilitate improved guest-host communication. Updated the README to include kernel module setup instructions for the new V-sock feature.
The V-sock implementation enables VMs to use a consistent communication channel that is not affected by network configuration changes, enhancing reliability and potential interoperability with host services. It's important to ensure that the necessary kernel modules are loaded as part of the setup process, as documented.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add vsock-based communication to manager
Introduced virtual socket (vsock) communication abilities in the manager package by implementing a new socket service. This includes establishing a vsock listener and stub methods for sending computation results and cleaning up resources. The addition provides the groundwork for interprocess communication between guest and host in virtualized environments.
- Integrated the `mdlayher/vsock` library for handling virtual socket operations.
- Created a new `sockService` struct to encapsulate vsock listener handling.
- Implemented `NewVsock` constructor to initialize the listener with domain value `3`.
- Added placeholder methods for future computation sending and service closing logic.
This enhancement targets scenarios where efficient VM-to-host communication is required.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Revert "Optimize QEMU launch and add V-sock support"
This reverts commit 44557c39d9.
* Revert "Add vsock-based communication to manager"
This reverts commit d1d3c01f3b.
* Remove unnecessary network link file installation
Network configuration is being handled elsewhere, making the installation of the `00-network.link` file redundant. Eliminated the line from the agent Makefile to prevent potential conflicts and streamline the initialization process.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add python pip to buildroot
* Use DHCP for agent
* Use systemd for naming network interfaces
* Removed eth0 dhcp setting
* Add ETH_IFACE variable
* Added explanation for ETH_IFACE name
* Further explain ETH_IFACE value
* Use systemd for HAL boot
* Removed unnecessary mkdir command
* Add newline at the end of file
* Configure eth0 network interface
* Keep kernel naming scheme
* Fix agent.mk file for network interfaces
* Fix bug in agent service and state
The commit fixes a bug in the agent service and state files.
Previously, the condition to check the state in the agent service
was incorrect. It was checking the state directly instead of
using the GetState() method. This has been fixed by using the
GetState() method to check the state.
Additionally, a new GetState() method has been added to the
StateMachine struct in the state file. This method retrieves
the current state by acquiring a lock and returning the state
value.
The changes have been tested and verified to resolve the bug
and improve the accuracy of state checking in the agent service.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix bug in agent state machine
The bug in the agent state machine caused an error when attempting an invalid transition. This commit fixes the bug by properly locking and unlocking the state machine before and after transitioning to the next state. Additionally, the logger now correctly logs the current and next state during a valid transition.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix race condition in state machine
The commit fixes a race condition in the state machine implementation in the `Start` method. The race condition occurs when multiple goroutines try to access and modify the state concurrently. To fix this, a mutex lock and unlock are added around the critical sections of code to ensure exclusive access to the state variable. This prevents race conditions and ensures the state transitions are executed correctly.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix race condition in StateMachine.Start()
The StateMachine.Start() method was experiencing a race condition
when multiple events were being processed concurrently. This was
caused by not properly locking and unlocking the state machine
before and after updating the state. This commit fixes the issue
by adding proper locking and unlocking around the state update
operation. Additionally, the logging statement has been updated
to include the previous and next states for better debugging.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* add magistrala dep
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove mainflux
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger.
The commit message should be:
"Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger"
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix notification topic in agent service and update NATS ports in Docker environment variables
The agent service's notification topic was incorrectly set to "channels.manager" instead of "agent". This commit fixes the issue by updating the notification topic.
Additionally, the NATS ports in the Docker environment variables were incorrect. The COCOS_NATS_PORT and COCOS_NATS_HTTP_PORT have been updated to the correct values.
These changes ensure that the agent service uses the correct notification topic and the NATS ports are properly configured.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor Makefile and add new targets for building Docker images
The Makefile has been refactored to include new targets for building Docker images. The `make_docker` and `make_docker_dev` functions have been defined to handle the Docker build process. The `dockers` and `dockers_dev` targets have been added to build the Docker images for all services and development environments respectively.
This commit introduces changes to the Makefile to improve the build process and provide support for Docker images.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* update readme
Signed-off-by: SammyOina <sammyoina@gmail.com>
* wrap env vars
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor main.go in cmd/cli
The main.go file in cmd/cli has been refactored to improve code readability and maintainability. The defURL constant has been removed as it is no longer needed. Additionally, unnecessary whitespace has been removed.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix linting
Signed-off-by: sammy <sammyoina@gmail.com>
* rename cocos-ai to cocos
Signed-off-by: sammy <sammyoina@gmail.com>
* Updated README with NATS setup instructions and correct systemd path
Expanded the project's README to include setup instructions for the NATS server, which is now a necessary component for agent and manager communication. Additionally, the systemd service file path has been corrected from 'systemd' to 'init/systemd' ensuring the agent's proper installation and configuration as a daemon. This facilitates a smoother setup experience and reflects the dependency on NATS for push notifications.
Ref: Agent and manager communication enhancement
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Added message broker URL to agent service config
Included the environment variable for the message broker URL in the systemd service configuration to facilitate agent communication with the messaging system. This ensures the agent can connect to the designated message broker for event publishing and subscription.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* update docs to hal
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Remove TLS and timeout config from gRPC client
Refactored gRPC client and associated API code by removing unnecessary TLS configuration and timeout settings. Simplified the communication setup by trusting the environment to enforce security policies and handle operation durations, instead of hardcoding these within the application.
This brings about a cleaner, more maintainable codebase and shifts responsibility for security configurations out of the code, aligning with infrastructure-as-code practices and enabling easier scalability and environment-specific adjustments.
This change also affects the API documentation and example commands, which have been updated accordingly to reflect the simplification and to guide users with the streamlined setup process.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
Signed-off-by: sammy <sammyoina@gmail.com>
* Add agent address to run responses
The manager service's Run method now retrieves the agent address upon successful computation execution, providing more informative responses across gRPC, HTTP, and logging endpoints. This change improves service transparency by returning the agent's address to be used by client services, making the manager service's external communication more comprehensive. Updated the `RunResponse` structure in the corresponding protocol buffers definition and response handling in gRPC and HTTP APIs, ensuring that agent address information is serialized appropriately. It also necessitates a slight adjustment in the QEMU configuration to manage port forwarding rules more dynamically, simplifying the process as only agent-relevant ports are incremented with each new computation. This extra detail in responses aids in debugging and offers better integration capabilities for clients.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Allocate dynamic ports for VM guests and expand error handling
Refactored service initialization to accept host IP and incorporated dynamic port allocation for VM guests, replacing the prior static increment method. Introduced a new error type 'ErrFailedToAllocatePort' to capture instances where the system is unable to find a free port. Integrated a third-party error package for improved error wrapping and context. These changes prevent port conflicts between VM guests and enhance error diagnostics for service operations.
Resolves issue with static port allocation leading to conflicts.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add HOST_IP to service configuration
Extend service configuration to include the host machine's IP address, allowing instances to be aware of their deployment environment. This update passes the new HostIP field to the service constructor, ensuring the service can now operate with host-specific logic.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Populate AgentAddress in gRPC Responses
Enhanced the gRPC encode/decode functions to properly populate the 'AgentAddress' field in 'RunResponse' objects. This ensures that consumers of the gRPC interface receive complete response data, which previously omitted the important 'AgentAddress' information. The change impacts both server-side response encoding and client-side response decoding, aligning the implementation with the expected interface contract.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix ci
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* add status endpoint
Signed-off-by: sammy <sammyoina@gmail.com>
* feat: Update code generation tools to latest versions
Update the code generation tools, including protoc-gen-go and
protoc-gen-go-grpc, to their latest versions (v1.31.0 and v1.3.0,
respectively). This ensures compatibility with the latest features and
improvements. The updated tools also require gRPC-Go v1.32.0 or later.
The new versions bring important updates and bug fixes, enhancing the
performance and stability of the generated code. By staying up-to-date
with the latest tooling, we can take advantage of the latest
functionality and ensure a smooth development experience.
No code changes are included in this commit. These updates only impact
the code generation process.
Signed-off-by: sammy <sammyoina@gmail.com>
* Update Go version to 1.21.x
Signed-off-by: sammy <sammyoina@gmail.com>
* Refactor agent and manager services to publish event notifications
The refactoring includes changes to the agent and manager services to incorporate event notifications. By publishing events, the services can inform subscribers about the current state of the computation or any updates. Specifically, the `agentService` now includes a `cmpHash` field to store the SHA-256 hash of the computation, which is subsequently used when publishing events. The `agentService` and `managerService` now use the `publisher` interface to publish events to the topic "manager". Notably, the removed `pubsub.go` file is no longer necessary.
This commit improves the service architecture by allowing subscribers to receive relevant updates and monitor the progress of computations. It enhances the overall system by providing more transparency and enabling better coordination between the agent and manager services.
Signed-off-by: sammy <sammyoina@gmail.com>
* Improve generated Go file comparison in checkproto workflow
Refactor the file comparison logic in the checkproto workflow to use the `-p` flag instead of `-s` for improved accuracy. This change ensures that the generated Go files are thoroughly compared with the original ones, detecting any discrepancies and preventing out-of-sync files from passing the validation. By using the `-p` flag, we now check both the contents and the metadata of the files, providing more robust synchronization checks. This update enhances the reliability of the checkproto workflow and helps maintain consistency between the proto files and their corresponding generated Go files.
Signed-off-by: sammy <sammyoina@gmail.com>
* Update file comparison command to detect differences line by line
The code change updates the file comparison command used in the CI workflow to detect differences line by line instead of only reporting the first difference encountered. This change improves the accuracy of detecting inconsistencies between the original protobuf files and the generated Go files. Previously, only the first difference was reported, leading to potential missed issues. By comparing the files line by line, we can now detect and report all differences accurately. This change enhances the reliability of our CI pipeline and ensures that the generated Go files stay in sync with the protobuf files.
Signed-off-by: sammy <sammyoina@gmail.com>
* add event exporting to external server
Signed-off-by: sammy <sammyoina@gmail.com>
* feat: Add support for notification server URL
The commit adds a new environment variable, `COCOS_NOTIFICATION_SERVER_URL`, which allows specifying the server to receive notification events from the agent. This addition provides flexibility to configure the notification server URL based on the deployment environment. This change enables seamless integration with different notification server instances and enhances the extensibility of the system. It resolves the need to modify the code directly when changing the server URL.
Signed-off-by: sammy <sammyoina@gmail.com>
* Refactor gRPC client and server, remove unused handlers
The commit refactors the gRPC client and server code by removing the unused `nopDecoder` and `status` handlers from the client and server, respectively. This cleanup reduces code clutter and improves maintenance. No significant consequences are expected.
Signed-off-by: sammy <sammyoina@gmail.com>
* Ensure generated Go files stay in sync with proto files during the CI workflow
Fixes an issue in the CI workflow where proto files and their corresponding generated Go files were not being properly compared for synchronization. Previously, the `cmp -l` command was used, which only printed differing byte positions, leading to false negatives. This has been corrected by using `cmp -s` instead, which outputs nothing if the files are identical. This change ensures that any differences between the proto files and their generated Go files will be detected, helping to maintain consistency and accuracy in the codebase.
Signed-off-by: sammy <sammyoina@gmail.com>
* Enhance notification payload and endpoint
Extended the notification system to include 'status' and 'details' in the payload, improving traceability and debugging. Adapted the serialized JSON structure for clarity and added an 'originator' field to track the source service. Transitioned to a generalized event endpoint, facilitating a more streamlined event handling process.
Refactors POST request to a more appropriate endpoint and updates the notification service interface to reflect new payload requirements.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor event notification logic
Removed the legacy notifications package and consolidated event notification functionality using the new internal events service. Modified agent, manager, and main application code to use this service for consistent event reporting and error handling workflows across services. This change simplifies event management, improves error visibility, and allows for more maintainable code by centralizing event-related logic. The substitution of verbose state-specific publishEvent calls with generic status reporting aligns with the new service's capabilities.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Expand agent protobuf message types and improve error handling
The protobuf definition for agent messages has been updated to include an additional message type, facilitating future data structure expansions. Additionally, error handling for event sending in the main agent execution has been enhanced to log errors when sending 'init' events fail, ensuring issues are properly tracked. The unused `notificationTopic` constant in the manager service has been removed for cleaner code maintenance.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Optimize JSON parsing and fix header omission
Removed unnecessary unquoting of a JSON string before unmarshaling, streamlining the computation value extraction process. Also corrected a missing Content-Type header in the event sending function, ensuring proper handling of JSON requests by recipients. These changes improve performance and communication reliability.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* align vars
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Remove computation request timeout feature
The timeout feature for computation requests has been removed to simplify the computation execution flow. This involved changes across multiple files, including protobuf definitions, HTTP endpoint handling, and the internal computation logic. We eliminated the timeout field, associated logic, and error handling to ensure the system no longer supports timeouts for computations, mitigating any unintended timeout impacts on long-running processes.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: sammy <sammyoina@gmail.com>
Signed-off-by: SammyOina <sammyoina@gmail.com>
Standardized the variable name for computation parameter across various service and middleware implementations to improve code consistency. This enhances readability and maintainability.
Additionally, tracing functionality has been removed completely, signifying a potential shift in the telemetry strategy or removal of an unused feature. The run command associated with cli package has also been deleted, suggesting a change in the CLI tool's capabilities or its command handling strategy.
The service interface's Run method definition and its implementations now align with the updated parameter naming, ensuring coherence throughout the codebase.
Removed files:
- The entire `tracing` package has been deleted.
- Removed the `cli/run.go` file, indicating the removal of the 'run' command from the CLI.
Please note that these changes may affect any existing workflows or systems that rely on the removed tracing capabilities or the CLI run command, and appropriate adjustments will need to be made to accommodate these updates.
Refer to internal documentation or release notes for migration paths or additional context.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enable passing computation details to QEMU args
Refactored the QEMU argument construction to accept computation details as JSON. The changes include updating function signatures to pass an `agent.Computation` object and serialize it into the QEMU boot parameters. These modifications allow for custom computations to be specified per VM, facilitating more dynamic and configurable virtual machine creation. This update also integrates timing and data transformation necessary for the agent computation setup.
Resolves computation configurability for virtual machines.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enable extraction of computation value from cmdline
Introduced a feature to parse the computation parameter directly from the machine's command line, thus enabling dynamic configuration of agent computations without the need for a preset environment. This change enhances the agent's ability to adapt to different computation requirements on startup. A fallback warning informs of cases where the computation parameter was not found, ensuring traceability of operational state during service initialization.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Remove RunRequest and RunResponse from Agent gRPC
Refactor the Agent service by deleting the RunRequest and RunResponse messages along with the corresponding method implementations in the gRPC interface. This simplifies the gRPC protocol by removing an obsolete RPC method, streamlining communication between services. The deletion also impacts the manager service, which no longer uses the removed RPC method. Consequently, the service definitions across related files like endpoints, servers, and clients have been adjusted. This adjustment enforces a more direct approach for starting computations, reflecting a shift toward manager-initiated computation execution.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Switch qemu config to use kernel and initrd images
Refactored qemu configuration to load the system using separate kernel and initial ramdisk images instead of a QCOW2 image. Removed previous disk image configurations and replaced them with directives to specify the kernel (`bzImage`) and root filesystem (`rootfs.cpio.gz`) files. Adjusted the constructQemuArgs function accordingly to set the kernel path, kernel command line options, and initrd path. Updated CreateVM function to handle copying of the new kernel and rootfs images. These changes align the virtual machine setup process with a more modular and potentially customizable boot configuration.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix lint
Signed-off-by: SammyOina <sammyoina@gmail.com>
* capcomments
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor RunRequest to use structured Computation
The protobuf and associated service implementations for the RunRequest message were refactored to replace the raw Computation byte slice with a structured ComputationReq object. This allows clearer and more type-safe manipulation of computation requests. The grpc, http, and agent service layers were updated to build and parse ComputationReq accordingly. The ComputationReq structure includes details like IDs, names, time stamps, and metadata, forming a well-defined contract for computation tasks.
This change aligns with efforts to standardize request formats and improve clarity in inter-service communication. It impacts all systems interfacing with the RunRequest service and thus requires coordinated updates to the entire stack.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Initialize metadata maps and handle nil values
Improved the robustness of metadata handling in gRPC endpoints and SDK by initializing metadata maps and explicitly checking for nil values before converting them. This ensures that both the agent's gRPC endpoint and the SDK properly handle cases where metadata fields may be uninitialized or contain nil values, preventing potential null pointer exceptions.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor computation request handling
Refactored the endpoint to construct Computation object from gRPC request, incorporating structpb for metadata handling and timestamppb for StartTime and EndTime fields. The management service and API requests are also updated to align with these changes, improving type safety and ensuring data is correctly marshalled when making service calls.
Resolves data marshalling issues for computation requests.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* use singular
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove unuse fields
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove unused fields
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
Improved the timing logic for setting computation deadlines by replacing the channel-based pattern with a direct time addition. This change enhances the performance by preventing unnecessary channel operations.
Additionally, relocated the select block that checks for computation timeout and data reception errors. This alteration ensures that the socket connection acceptance routine starts before any timeouts or errors are checked, thereby avoiding potential race conditions and making the code more resilient to early timeout scenarios.
Signed-off-by: SammyOina <sammyoina@gmail.com>
Refactored the Dockerfile to use multi-stage builds, improving the image size and separation of concerns. Removed `qemu-system-x86_64` from the initial build stage and included it in a new, minimal final image based on `alpine:latest`. This change minimizes the production image size by avoiding unnecessary build dependencies in the final artefact. Additionally, the CA certificates are now copied into the final stage to maintain SSL functionality.
The development Dockerfile has been cleaned to match new conventions.
Resolves issues with bloated Docker images and unneeded dependencies in production.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add gRPC TLS config and update protoc versions
Enhanced manager service with TLS configuration options and timeout settings for gRPC communication with agent services. Updated corresponding protobuf definitions to include new fields for CA certificates, TLS status, and timeouts. The added TLS support ensures secure inter-service communication while flexible timeout configurations improve the robustness of network interactions.
The following adjustments and additions were made:
- Included new gRPC configuration fields (ca_certs, client_tls, timeout) for agent communication in protobuf definitions.
- Injected agent gRPC configuration into service endpoints, ensuring secure TLS setup and compliance with provided settings.
- Revised main service function signatures to accept the gRPC configuration object.
- Incremented port forwarding counters post-computation to avoid port conflicts.
- Conducted compatibility update of protobuf version comments to reflect minor version bump (v4.25.0 to v4.25.1).
This change impacts service deployment that requires proper configuration of TLS credentials and mindful determination of timeouts for efficient network use.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* update proto
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Set default timeout for agent configuration
Introduced a default timeout of 60 seconds in both gRPC and HTTP endpoints for agent configuration when none is specified. This change ensures that operations do not hang indefinitely and provides a reasonable default for client interactions. Additionally, the instantiation of a new agent client is now outside the retry loop to avoid repeated setup on transient failures.
Refactors service logic to optimize client connection handling by moving the agent client setup to occur before attempting retries, which should reduce overhead and improve clarity in error situations.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Subject: Move agent client creation into retry loop
Body:
Refactored the service manager's Run method to initialize the agent gRPC client inside the exponential backoff retry loop. This change addresses intermittent connection issues by reattempting client creation on temporary network failures, ensuring a robust setup before calling the Run method on the client.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove multiple brokers
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Standardize build process on NATS messaging
Removed conditional build flags for different message broker types, making NATS the default and only option. Simplified the Makefile by eliminating the toggleable message broker configuration and related docker-based setup. Updated the NATS URL in docker `.env` and docker-compose service dependency to reflect this change. This adjustment streamlines the build process and reduces configuration complexity.
Resolves issues with inconsistent messaging setups across various environments.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor message broker configuration
Standardized the environment variable for the message broker URL across services and updated documentation to reflect this change. Renamed the environment variable from COCOS_NATS_URL to COCOS_MESSAGE_BROKER_URL to improve clarity and maintain consistency in service configuration. This alteration facilitates future support for different message broker backends beyond NATS without further variable name changes.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Create and publish a Docker image
This commit adds a new file `.github/workflows/build.yml` which contains the workflow configuration for creating and publishing a Docker image. The workflow is triggered on push events to the `main` branch. It sets environment variables for the registry and image name. The job runs on `ubuntu-latest` and has necessary permissions for reading contents and writing packages. The steps include checking out the repository, logging in to the container registry using the GitHub token, and extracting metadata for Docker.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor build workflow and update Makefile
The build workflow has been refactored to build the Manager image instead of the API Gateway image. The Makefile has also been updated.
The changes include:
- Renaming the build step to "Build Manager and push Docker image"
- Updating the image name to include "/manager"
- Updating the build arguments to use "SVC=manager"
- Updating the tags to use the output from the metadata action
These changes ensure that the correct Docker image is built and pushed.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove labels
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix bug in agent service and state
The commit fixes a bug in the agent service and state files.
Previously, the condition to check the state in the agent service
was incorrect. It was checking the state directly instead of
using the GetState() method. This has been fixed by using the
GetState() method to check the state.
Additionally, a new GetState() method has been added to the
StateMachine struct in the state file. This method retrieves
the current state by acquiring a lock and returning the state
value.
The changes have been tested and verified to resolve the bug
and improve the accuracy of state checking in the agent service.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix bug in agent state machine
The bug in the agent state machine caused an error when attempting an invalid transition. This commit fixes the bug by properly locking and unlocking the state machine before and after transitioning to the next state. Additionally, the logger now correctly logs the current and next state during a valid transition.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix race condition in state machine
The commit fixes a race condition in the state machine implementation in the `Start` method. The race condition occurs when multiple goroutines try to access and modify the state concurrently. To fix this, a mutex lock and unlock are added around the critical sections of code to ensure exclusive access to the state variable. This prevents race conditions and ensures the state transitions are executed correctly.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix race condition in StateMachine.Start()
The StateMachine.Start() method was experiencing a race condition
when multiple events were being processed concurrently. This was
caused by not properly locking and unlocking the state machine
before and after updating the state. This commit fixes the issue
by adding proper locking and unlocking around the state update
operation. Additionally, the logging statement has been updated
to include the previous and next states for better debugging.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* add magistrala dep
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove mainflux
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger.
The commit message should be:
"Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger"
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix Makefile to properly set DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE
The Makefile has been updated to fix an issue with setting the DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE variables. The USER_REPO variable is now used to generate the DOCKER_PROJECT name following the Docker Compose guidelines. Additionally, the COCOS_MESSAGE_BROKER_TYPE variable is now properly set to "nats" if it is empty. This ensures that the correct values are used when compiling and installing the service.
Summary:
Fix Makefile to properly set DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE
Details:
- Update USER_REPO variable to generate DOCKER_PROJECT name
- Set COCOS_MESSAGE_BROKER_TYPE to "nats" if empty
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix notification topic in agent service and update NATS ports in Docker environment variables
The agent service's notification topic was incorrectly set to "channels.manager" instead of "agent". This commit fixes the issue by updating the notification topic.
Additionally, the NATS ports in the Docker environment variables were incorrect. The COCOS_NATS_PORT and COCOS_NATS_HTTP_PORT have been updated to the correct values.
These changes ensure that the agent service uses the correct notification topic and the NATS ports are properly configured.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add Dockerfiles for production and development environments
This commit adds two new Dockerfiles, one for the production environment and one for the development environment. The production Dockerfile sets up the necessary dependencies and builds the application, while the development Dockerfile simply copies the built application. Both Dockerfiles include the necessary SSL certificates for the application to work properly.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor Makefile and add new targets for building Docker images
The Makefile has been refactored to include new targets for building Docker images. The `make_docker` and `make_docker_dev` functions have been defined to handle the Docker build process. The `dockers` and `dockers_dev` targets have been added to build the Docker images for all services and development environments respectively.
This commit introduces changes to the Makefile to improve the build process and provide support for Docker images.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix makefile to generate Docker images for services
- Added DOCKERS and DOCKERS_DEV variables to generate Docker images for services
- Updated the dockers target to include DOCKERS
- Updated the dockers_dev target to include DOCKERS_DEV
- Updated the $(DOCKERS) and $(DOCKERS_DEV) targets to call the respective make_docker and make_docker_dev functions
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor Makefile and Dockerfile
The Makefile has been refactored to remove unnecessary sed commands and improve code readability. The changes include:
- Removed sed commands related to changing the broker in docker-compose.yml for nats and rabbitmq profiles.
- Removed the eer target from the Makefile.
The Dockerfile has been modified to include the WORKDIR directive.
These changes improve the maintainability and readability of the codebase.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix NATS URL in .env file and add Manager configuration in docker-compose.yml
The NATS URL in the .env file was updated to use the correct hostname for the broker. Additionally, the Manager configuration was added to the docker-compose.yml file.
This commit fixes the NATS URL in the .env file and adds the necessary Manager configuration to the docker-compose.yml file.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix incorrect host and port values in Makefile and .env
The commit updates the host and port values in the Makefile and .env files to fix incorrect values. The host and port values are updated to "cocos-manager" and "7003" respectively. This ensures that the correct host and port are used for the Manager HTTP and gRPC services.
This commit resolves the issue with the incorrect host and port values and ensures that the Manager services are configured correctly.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix Dockerfile and docker-compose.yml configurations
The Dockerfile and docker-compose.yml configurations have been fixed to address the following issues:
- Removed unnecessary COPY commands in the Dockerfile.
- Removed the "privileged" flag in the docker-compose.yml file.
These changes ensure that the Docker image is built correctly and the container is launched with the appropriate configurations.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove manual assignment
Signed-off-by: SammyOina <sammyoina@gmail.com>
* update protoc CI
Signed-off-by: SammyOina <sammyoina@gmail.com>
* add paths
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove profiles
Signed-off-by: SammyOina <sammyoina@gmail.com>
* restore default config
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix Docker project name in Makefile
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add CoCos-AI Manager API specification
This commit adds the CoCos-AI Manager API specification
in the form of a YAML file. The specification includes
information about the title, description, contact,
license, and version of the API. It also defines the
servers where the API is hosted and the paths and
operations available, such as running computation on
a virtual machine.
The API specification is based on OpenAPI 3.0.1 and
provides a clear and concise overview of the CoCos-AI
Manager service.
The commit also includes a link to the CoCos-AI repository
and the license information.
This commit is necessary to provide a clear and documented
API specification for the CoCos-AI Manager service.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor go.mod and go.sum files
The go.mod and go.sum files have been refactored to remove the go.opentelemetry.io/contrib/propagators/jaeger package, which is no longer needed. This package was causing compatibility issues with the current version of the project. The refactoring ensures that the project is using the latest compatible versions of the required dependencies.
This commit removes the go.opentelemetry.io/contrib/propagators/jaeger package from the go.mod file and updates the go.sum file accordingly.
Note: The go.mod file now uses go.opentelemetry.io/otel v1.19.0 and go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.19.0.
Please review the changes to ensure compatibility and functionality.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix typo
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* add stringer
Signed-off-by: SammyOina <sammyoina@gmail.com>
* rename module to cocos
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix fmt.Stringer implementation in computations.go
The fmt.Stringer implementation for Datasets and Algorithms in computations.go was fixed to correctly use pointers.
This commit addresses the issue where the String() method for Datasets and Algorithms in computations.go was not correctly implemented. The fix ensures that the String() method now correctly marshals the data to JSON and returns the string representation.
The changes made in this commit will improve the functionality and accuracy of the String() method for Datasets and Algorithms.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix bug in agent state machine
The bug in the agent state machine caused an error when attempting an invalid transition. This commit fixes the bug by properly locking and unlocking the state machine before and after transitioning to the next state. Additionally, the logger now correctly logs the current and next state during a valid transition.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix race condition in state machine
The commit fixes a race condition in the state machine implementation in the `Start` method. The race condition occurs when multiple goroutines try to access and modify the state concurrently. To fix this, a mutex lock and unlock are added around the critical sections of code to ensure exclusive access to the state variable. This prevents race conditions and ensures the state transitions are executed correctly.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix race condition in StateMachine.Start()
The StateMachine.Start() method was experiencing a race condition
when multiple events were being processed concurrently. This was
caused by not properly locking and unlocking the state machine
before and after updating the state. This commit fixes the issue
by adding proper locking and unlocking around the state update
operation. Additionally, the logging statement has been updated
to include the previous and next states for better debugging.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* add magistrala dep
Signed-off-by: SammyOina <sammyoina@gmail.com>
* remove mainflux
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix agentService New function to include messaging.Publisher parameter
The agentService New function has been updated to include a messaging.Publisher parameter. This change allows the agent service to publish messages to a messaging system. The messaging.Publisher parameter has been added to the agentService struct and the New function signature has been updated accordingly. This change ensures that the agent service can communicate with other components using the messaging system.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor service.go state functions
The commit refactors the state functions in the service.go file.
The functions for each state have been modified to use the svc.publishEvent
method to publish events with appropriate messages.
- Refactor state functions in service.go
- Use svc.publishEvent to publish events with messages for each state
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix computation run event publishing and add pubsub functionality
The computation run event publishing in the agent service was fixed to correctly call the publishEvent function. Additionally, the pubsub functionality was added to the manager package.
- Fixed computation run event publishing in agent service
- Added pubsub functionality to manager package
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix license header in pubsub.go file
The commit fixes the license header in the pubsub.go file.
The copyright and SPDX-License-Identifier have been added
to comply with the Apache-2.0 license.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger.
The commit message should be:
"Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger"
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix Makefile to properly set DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE
The Makefile has been updated to fix an issue with setting the DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE variables. The USER_REPO variable is now used to generate the DOCKER_PROJECT name following the Docker Compose guidelines. Additionally, the COCOS_MESSAGE_BROKER_TYPE variable is now properly set to "nats" if it is empty. This ensures that the correct values are used when compiling and installing the service.
Summary:
Fix Makefile to properly set DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE
Details:
- Update USER_REPO variable to generate DOCKER_PROJECT name
- Set COCOS_MESSAGE_BROKER_TYPE to "nats" if empty
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix Makefile Docker profile assignment and build flags
The Makefile was updated to fix the assignment of the Docker profile and build flags. The Docker profile is now assigned based on the value of COCOS_MESSAGE_BROKER_TYPE, and if it is not provided, the default value is set to "nats". The build flags were also updated to include the COCOS_MESSAGE_BROKER_TYPE value as a tag for the Go build process.
This commit addresses the issue with the Docker profile assignment and ensures that the correct build flags are used during the build process.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix makefile
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix notification topic in agent service and update NATS ports in Docker environment variables
The agent service's notification topic was incorrectly set to "channels.manager" instead of "agent". This commit fixes the issue by updating the notification topic.
Additionally, the NATS ports in the Docker environment variables were incorrect. The COCOS_NATS_PORT and COCOS_NATS_HTTP_PORT have been updated to the correct values.
These changes ensure that the agent service uses the correct notification topic and the NATS ports are properly configured.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* add pubsub
Signed-off-by: SammyOina <sammyoina@gmail.com>
* update protoc
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* add state machine
Signed-off-by: SammyOina <sammyoina@gmail.com>
* This commit addresses the following changes:
- In agent/service.go:
- Added a goroutine to start the StateMachine
- Sent the start event to the StateMachine
- In agent/state.go:
- Added a copyright notice and license identifier
These changes ensure that the StateMachine starts correctly and that the agent/state.go file includes the necessary copyright and license information.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix race condition in agent service initialization
The commit fixes a race condition in the agent service initialization.
Previously, the `New` function in `agent/service.go` would start the state machine
goroutine before initializing the `sm` field. This could lead to a race condition
where the state machine would access uninitialized fields.
To fix this, the `New` function now takes a `context.Context` argument and passes it
to the `Start` method of the state machine. Additionally, a `sync.Mutex` has been
added to the `StateMachine` struct to ensure thread safety.
This commit addresses the race condition and ensures proper initialization of the agent service.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix sending events when algorithms and datasets are received
This commit fixes a bug where events were not being sent when algorithms and datasets were received in the agent service. The bug was causing the events to not be triggered, leading to incorrect behavior. This commit adds the necessary code to send the events when algorithms and datasets are received, ensuring that the events are properly triggered.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix bug: computation results not ready
The computation results were not being marked as ready,
resulting in an error when trying to access them.
This commit fixes the issue by adding a check for
the computation results before returning them.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix bug in agent service and state
The commit fixes a bug in the agent service and state files.
Previously, the condition to check the state in the agent service
was incorrect. It was checking the state directly instead of
using the GetState() method. This has been fixed by using the
GetState() method to check the state.
Additionally, a new GetState() method has been added to the
StateMachine struct in the state file. This method retrieves
the current state by acquiring a lock and returning the state
value.
The changes have been tested and verified to resolve the bug
and improve the accuracy of state checking in the agent service.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* check for time outs
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix bug in agent state machine
The bug in the agent state machine caused an error when attempting an invalid transition. This commit fixes the bug by properly locking and unlocking the state machine before and after transitioning to the next state. Additionally, the logger now correctly logs the current and next state during a valid transition.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix race condition in state machine
The commit fixes a race condition in the state machine implementation in the `Start` method. The race condition occurs when multiple goroutines try to access and modify the state concurrently. To fix this, a mutex lock and unlock are added around the critical sections of code to ensure exclusive access to the state variable. This prevents race conditions and ensures the state transitions are executed correctly.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix race condition in StateMachine.Start()
The StateMachine.Start() method was experiencing a race condition
when multiple events were being processed concurrently. This was
caused by not properly locking and unlocking the state machine
before and after updating the state. This commit fixes the issue
by adding proper locking and unlocking around the state update
operation. Additionally, the logging statement has been updated
to include the previous and next states for better debugging.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update agent.pb.go to include a new field "Provider" in the AlgoRequest struct.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix build
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix undeclared consumer error in agent service
The commit fixes an error in the agent service where an undeclared consumer was requesting results in the computation manifest. The error is resolved by adding an error message for the undeclared consumer case.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
The Makefile was updated to fix the version and commit variables.
Previously, the VERSION variable was using "git describe" to get the latest tag,
but it was failing when there were no tags available.
The updated VERSION variable now uses "git describe --always" to fallback to the commit hash if no tags are available.
The COMMIT variable was also updated to use "git rev-parse HEAD" to get the commit hash.
Summary:
Fix Makefile version and commit variables
Body:
- Update VERSION variable to use "git describe --always" to fallback to commit hash
- Update COMMIT variable to use "git rev-parse HEAD" to get commit hash
Signed-off-by: SammyOina <sammyoina@gmail.com>
The Go version in the GitHub Actions workflow was updated from 1.20.x to 1.21.x. This change ensures that the correct version of Go is used during the workflow execution.
Signed-off-by: SammyOina <sammyoina@gmail.com>
Pull request title should be `COCOS-XXX - description` or `NOISSUE - description` where XXX is ID of the issue that this PR relate to.
Please review the [CONTRIBUTING.md](https://github.com/ultravioletrs/cocos/blob/master/CONTRIBUTING.md) file for detailed contributing guidelines.
For Work In Progress Pull Requests, please use the Draft PR feature, see https://github.blog/2019-02-14-introducing-draft-pull-requests/ for further details.
For a timely review/response, please avoid force-pushing additional commits if your PR already received reviews or comments.
- Provide tests for your changes.
- Use descriptive commit messages.
- Comment your code where appropriate.
- Squash your commits
- Update any related documentation.
-->
# What type of PR is this?
<!--This represents the type of PR you are submitting.
For example:
This is a bug fix because it fixes the following issue: #1234
This is a feature because it adds the following functionality: ...
This is a refactor because it changes the following functionality: ...
This is a documentation update because it updates the following documentation: ...
This is a dependency update because it updates the following dependencies: ...
This is an optimization because it improves the following functionality: ...
-->
## What does this do?
<!--
Please provide a brief description of what this PR is intended to do.
Include List any changes that modify/break current functionality.
-->
## Which issue(s) does this PR fix/relate to?
<!--
For pull requests that relate or close an issue, please include them below. We like to follow [Github's guidance on linking issues to pull requests](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue).
For example having the text: "Resolves #1234" would connect the current pull request to issue 1234. And when we merge the pull request, Github will automatically close the issue.
-->
- Related Issue #
- Resolves #
## Have you included tests for your changes?
<!--If you have not included tests, please explain why.
For example:
Yes, I have included tests for my changes.
No, I have not included tests because I do not know how to.
-->
## Did you document any new/modified feature?
<!--If you have not included documentation, please explain why.
For example:
Yes, I have updated the documentation for the new feature.
No, I have not updated the documentation because I do not know how to.
-->
### Notes
<!--Please provide any additional information you feel is important.-->
@@ -13,7 +13,7 @@ Reporting issues are a great way to contribute to the project. We are perpetuall
thorough bug report.
Before raising a new issue, check [our issue
list](https://github.com/mainflux/mainflux/issues) to determine if it already contains the
list](https://github.com/ultravioletrs/cocos/issues) to determine if it already contains the
problem that you are facing.
A good bug report shouldn't leave others needing to chase you for more information. Please be as detailed as possible. The following questions might serve as a template for writing a detailed
@@ -41,23 +41,23 @@ To contribute to the project, [fork](https://help.github.com/articles/fork-a-rep
clone your fork repository, and configure the remotes:
Once the images are built (`docker images` command should show you `cocos/computations` and `cocos/datasets`),
composition can be run:
```bash
docker-compose -f docker/docker-compose.yml up
```
This will bring up the Cocos docker services and interconnect them. This command can also be executed using the project's included Makefile:
```bash
make run
```
- TEE enablement, deployment and monitoring (secure VM manager)
- HAL for TEEs based on hardened Linux kernel, secure bootloader and custom-tailored embedded rootfs for minimal TCB
- In-enclave agent, netowrking controller and other system software
- Encrypted asynchronous data transfer and result delivery
- API for programmable platform manipulation
- HW and SW supported attestation with verification tools
- CLI for system interaction
## Usage
The quickest way to start using Cocos is via the CLI. The latest version can be downloaded from the [official releases page][rel].
It can also be built and used from the project's root directory:
Clone the repo and create binaries:
```bash
make cli
./build/cocos-cli version
git clone git@github.com:ultravioletrs/cocos.git
make
```
Additional details on using the CLI can be found in the [CLI documentation](https://docs.mainflux.io/cli).
This will create 3 binaries:
```bash
ls build/
# cocos-agent cocos-cli cocos-manager
```
- Manager can be deployed on the AMD SEV-SNP host
- Agent can be built into [EOS][eos]-based HAL
- CLI can be used to communicate to remote Agent.
## Documentation
Official documentation is hosted at [Cocos official docs page][docs]. Documentation is auto-generated, checkout the instructions on [official docs repository](https://github.com/cocos/docs).
Project documentation is hosted at [Cocos AI official docs page][docs].
Documentation is generated from the [docs repository](https://github.com/ultravioletrs/docs).
## License
Cocos AI is a proprietary product created by Ultraviolet company.
Cocos AI is published under permissive open-source [Apache-2.0](LICENSE) license.
Agent service provides a barebones HTTP and gRPC API and Service interface implementation for the development of the agent service.
## Configuration
The service is configured using the environment variables from the following table. Note that any unset variables will be replaced with their default values.
// AgentServiceClient is the client API for AgentService service.
//
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
// NewService creates a new instance of Service. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
// The first argument is typically a *testing.T value.
// NewService creates a new instance of Service. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
// The first argument is typically a *testing.T value.
// NewAuthenticator creates a new instance of Authenticator. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
// The first argument is typically a *testing.T value.
This repository contains the command-line interface (CLI) tool for interacting with the Agent and manager service. The CLI allows you to perform various tasks such as running computations, uploading algorithm and datasets, and retrieving results.
## Build
From the project root:
```bash
make cli
```
## Usage
#### Get attestation
Retrieves attestation information from the SEV guest and saves it to a file.
To retrieve attestation from agent, use the following command:
```bash
./build/cocos-cli attestation get '<report_data>'
```
#### Validate attestation
Validates the retrieved attestation information against a specified policy and checks its authenticity.
To validate and verify attestation from agent, use the following command:
./build/cocos-cli data /path/to/dataset.csv <private_key_file_path>
```
Users can also upload directories which will be compressed on transit. Once received by agent they will be stored as compressed files or decompressed if the user passed the decompression argument.
##### Flags
- -d, --decompress Decompress the dataset on agent
#### Retrieve result
To retrieve the computation result, use the following command:
```bash
./build/cocos-cli result <private_key_file_path>
```
#### Checksum
When defining the manifest dataset and algorithm checksums are required. This can be done as below:
// needed to support dataset service functionality.
packagepostgres
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.