* feat(manager): start port search at minimum port range
When getting free port to be use we start at the minimum port range i.e 6000. This makes the agent url more deterministic since in most cases the agent port will be 6000
* fix(manager): Remove unused error channel
The error channel is not used
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
---------
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* refactor(env): remove internal env package
No need for this package since the parent package github.com/caarlos0/env does everything we need
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* refactor(jaeger): remove internal jaeger package
No need for this package since we can use magistrala exported package
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* refactor(metrics): remove internal metrics pkg
Use exported magistrala prometheus package
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* chore(dep): Update grpc and other dependencies
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* style(linter): remove enabled by default linters
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
---------
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
* add host_data option
* add CLI hostdata option and rename platform to backend
* move code for computation hash to a function
* rename getComputationHash to computationHash
* add default for backend information field switch
Corrected the struct field 'vnc' to 'Vnc' to match Go's public field naming conventions and to make it exportable. Each QEMU virtual machine's VNC display number now increments upon instantiation to avoid display conflicts.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update Go to 1.22 and enhance security features
- Upgraded the Go version in GitHub Actions workflows to 1.22.x for latest features and security patches.
- Added RSA public key field `UserKey` in `Dataset` and `Algorithm` to reinforce data integrity and encryption.
- Refactored `Result` method in `agentService` to use `containsID` for improved readability and potential performance benefits.
- Updated `grpcserver.New` and `internal/server/grpc` invocations to pass `agent.Service` by value in line with recommended Go practices.
- Introduced `grpc.StreamInterceptor` with no args in `Server.Start` which seems to be an initial step for future stream interceptor configuration.
These changes prepare for stronger data security measures, maintain compatibility with the latest Go features, and improve code quality regarding service struct usage. Potential follow-up is needed to configure the stream interceptor and to ensure the new RSA key field is appropriately utilized in data handling.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor auth system and protocol buffers
Enhanced the authentication system by adding context support and an improved user-role model. Implemented robust RSA public key verification for users and a restructured interceptor logic specific to stream types, streamlining the auth process. Updated protocol buffers and associated structures to accommodate user keys as byte slices, aligning with standard cryptographic practice. CLI commands for algorithms and datasets now require a private key file path argument for signing, strengthening security during interactions.
This comprehensive overhaul addresses security and efficiency considerations in the RPC framework and aligns with best practices for key handling. By streamlining and securing the user authentication process, the agent service's reliability is greatly improved, directly impacting the robustness of the entire computation pipeline.
- Refactored auth: added role-based user validation, context handling
- Reworked interceptors: separated stream types, fortified signature checks
- Updated protocol buffers: user public keys as byte slices for standard compatibility
- Enhanced CLI: introduced private key argument, ensuring secure algorithm and dataset submission
- Improved server and SDK contracts to align with auth changes
Related issues:
- Implements user roles and auth context [#103]
- CLI security enhancement for private key management
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Updated PEM decoding for key parsing in CLI and tests
Added `encoding/pem` to decode PEM blocks when parsing private and public keys across CLI commands and test computation scenarios, ensuring compatibility with key files. This enhances robustness in key handling by supporting PEM encoded keys. The update also includes registration of a new Keys command in the CLI.
Refactored code is now compliant with common key formats, addressing potential parsing issues.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix auth signature encoding and improve CLI usage example
The authentication system now decodes base64 strings before verifying signatures to align with the expected format. Additionally, the signature generation now encodes the output in base64, ensuring consistency across the auth process.
The CLI help message for the `result` command is enhanced by providing a usage example, making it more user-friendly and informative.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor containsID to handle dynamic fields
Updated the `containsID` function to accept a field name parameter, enabling dynamic field lookup within the reflection logic. This change facilitates the use of the function for various struct fields, improving code reusability and flexibility.
CLI command 'data' now requires an additional argument for the private key file path, outlined in the usage example update, reinforcing command clarity and user guidance.
Resolves issues with hardcoded field lookups and enhances CLI usability.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Remove extraneous newline in key generation log output
A redundant newline after the success message in the key generation command was removed to clean up log output formatting. This change ensures a more consistent and professional appearance of the CLI tool's messages.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Implemented auth service in gRPC startup
Added authentication services to the gRPC server initialization to enforce security measures. The gRPC server's New function now includes an `authSvc` parameter, requiring instantiation of the auth service before starting the server. Failure to create the auth service results in a fatal error, halting the process to avoid running without protection. Tests have been updated to include `nil` values for the auth service parameter to maintain their functionality without authentication.
Refactored `grpcserver.New` to accept the new auth service, and updated the main agent startup logic to create and inject the auth service. Added the auth middleware interceptors to the server options, which ensures that each gRPC call will undergo authentication.
This change is a step towards secure communication, and affected components should now consider the authentication requirement.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor config read logic and update agent setup
Improved the configuration reading in `cmd/agent/main.go` to handle larger payloads by reading data in chunks and checking for EOF, ensuring that all config data is captured even if it exceeds the initial buffer size. Enhanced the `test/manual/agent-config/main.go` to require additional command-line arguments, improving the setup process by explicitly requiring paths for data, algorithm, and public key as well as a boolean for attested TLS. Also updated the hashing method to SHA3 for the algorithm and data files, and included the hash and public keys as part of the agent, dataset, and result consumer configurations. These changes will make the agent setup more robust and provide better integrity checks for the involved files.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor run method to agentService
Moved the run function into agentService for better encapsulation and maintainability. This refactoring includes capturing both stdout and stderr during algorithm execution, enabling more informative debugging through enhanced logging. Consequentially, the run method now references members through the service instance, aligning with object-oriented best practices and improving code coherence.
Resolves issue with insufficient execution details when computations fail.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor computation data handling to use filepaths
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor error logging and ensure consistency
Replaced usage of the standard log package with a custom logger for error reporting to standardize error logging throughout the application. Additionally, introduced graceful shutdown by returning from the main function rather than forcing exit when failing to create auth service, aligning the application's error handling strategy.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor auth initialization and key file handling
Improved the readability and maintainability of the authentication service initialization by adding line breaks for logical separation. Also, standardized key filenames in the CLI key generation by introducing constants, enhancing code clarity and reducing the likelihood of file-naming errors.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor auth verification logic for improved security
Removed an extraneous line in the `verifySignature` function that was not necessary for the signature verification process. This change simplifies the code and improves readability.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor payload structures to simplify API
Removed the 'provider', 'id', 'consumer' fields from protocol buffers, gRPC services, and related functions across various files to streamline the data model and align with the new authentication system based on cryptographic verification rather than string identifiers. This results in more efficient data handling and a reduction in unnecessary payload data, while enhancing security by making entity validation strictly cryptographic.
The changes affect agent-SDK interactions, CLI tools, and related services, ensuring only the necessary data (algorithm/data bytes, user keys, and hashes) is transmitted and processed. Consequently, the core computation algorithm and dataset handlers now rely on indexes derived from context to associate data with respective manifest entries, thus maintaining the ability to link to specific computation manifests without relying on explicit IDs in the payload. Additionally, refactored authentication methods now enforce role-based security seamlessly through metadata.
This approach enhances privacy by avoiding transmission of potentially sensitive strings over the network and by ensuring that only internal indices, not globally interpretable identifiers, are used to process computations.
Aligned with the broader architectural goal of simplifying and securing the platform's core services, this change paves the way for upcoming revisions to the authentication scheme that will further consolidate role-based security and improve system integrity.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enhance CLI security with key paths
Removed the section on running computations from the CLI README as it may no longer be necessary or the functionality has been moved elsewhere. Required private key file paths for algorithm, dataset upload, and result retrieval commands to enhance security. This change associates each action with a specific identity, ensuring secure and traceable operations. Additionally, updated the manual test commands to reflect this new requirement.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix ci
Signed-off-by: SammyOina <sammyoina@gmail.com>
* fix fmt
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Implement manager client heartbeat mechanism
Introduced a heartbeat system in the manager service to maintain an active link with client agents. The updates involve sending periodic heartbeat signals controlled via the new environment variable `MANAGER_HEARTBEAT_INTERVAL`, defaulting to 1 second. The protobuf definitions, service interfaces, and server logic have been updated accordingly to handle the incoming signals. This change allows better tracking of active client connections and could be used for features like auto-reconnect or resource cleanup for lost connections in the future.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Enhance gRPC server to handle client termination
Introduce the capability for the gRPC server in the manager module to process termination requests from clients. Server and client proto buffers have been updated to include a Termination message, allowing clients to signal the server to stop processing. Handling for communication and termination flows involving context cancellation have been added across server implementation, manager client, and main execution logic to properly respond to termination signals. This ensures clean shutdowns and resource deallocation when a client requests termination, improving the robustness of the system.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor gRPC server context handling
Removed the redundant storage of context in grpcServer struct and modified the NewServer constructor to exclude the context parameter. Updated the Process method to derive context from the stream directly, ensuring better context scoping and adherence to proper gRPC patterns. This change simplifies the server's context management and ensures contexts are more accurately associated with their corresponding streams, which can improve debugging and request cancellation behavior.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update protoc to v4.25.3 and remove heartbeat system
Upgraded protoc version to 4.25.3, ensuring compatibility with the latest protocol buffer features and potential performance improvements. Additionally, removed the entire heartbeat system, involving both its gRPC server implementation and associated client-side logic. This elimination suggests a shift in the service health-check strategy, potentially to a more modern or efficient model.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Removed heartbeat functionality for manager service
The heartbeat feature and its related configuration have been removed across the application. This includes the removal of the Heartbeat method from the Service interface, the associated interval setting in the config structure, and related environment variable documentation. The change implies an architectural shift towards alternative methods of service availability checking or potentially a move to stateless service design. The cleanup removes unnecessary code, thus simplifying the overall service logic.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Introduce Terminate message and streamline gRPC server logic
- Added a new `Terminate` protobuf message to handle termination requests more explicitly.
- Removed redundant context cancels in gRPC server stream processing to rely on context propagation.
- Updated channel usage to match the altered stream message types; this aligns message handling with the updated protobuf definitions.
- Passed client authentication info to service run function to enhance the security during service execution.
- Utilized stream's context directly in error groups for improved clarity and error handling scope.
- Refactored message descriptors in protobuf to maintain correct ordering after introducing new message types.
These changes enhance the codebase's maintainability, simplify the logic, and align with the updated messaging protocol.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update agent configuration in main.go
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor single algorithm processing
Simplified the agent service's algorithm handling logic to process a single algorithm instead of multiple. This change:
- Removed the `Algorithms` type and associated stringer implementation.
- Updated the state machine and service logic to expect a singular algorithm, aligning the agent's internal state transitions with the new model.
- Adjusted the manager service and computations test server to mirror these changes in their respective payload structures, ensuring API and test consistency.
- Altered README files to reflect the simplified interaction model and removed outdated descriptions.
- Reverted the protoc-gen-go version used for generating protobuf files to maintain compatibility with the rest of the codebase.
The single-algorithm approach streamlines the computation running process, reducing complexity and potential error conditions. It directly impacts how external services will construct and send computation requests.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update protoc-gen-go version to v1.33.0
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor variable name in computations.go and grpc.go
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
* added initial code for attested TLS
* added client validation and verification
* fixed bugs for attested TLS
* updated README for manual testing
* fixed CI errors
* removed SNP pollicy from agent config
* added attested TLS config param to AgentConfig
* generated manager.pb.go for protoc v25.2
* updated proto-gen-go version on CI
* generated agent.pb.go to match newest proto gen version
* define errors for error handling
* fixed comments
* Update protoc to v4.25.3 and streamline client identification
Protocol Buffer compiler `protoc` has been upgraded to version 4.25.3. This update unifies the version across multiple generated files to ensure compatibility and take advantage of any bug fixes and performance improvements in the new release.
Additionally, the client identification process has been refined. The redundant `WhoAmI` message and the corresponding checks have been removed from the gRPC server implementation. Clients are now identified via their address from the `Process` stream's context as soon as a connection is established, simplifying the code and potentially reducing handshake time.
This change sets the foundation for a leaner communication protocol between manager and agents, and could contribute to lower latencies in client-server interactions.
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Fix goroutine bug in grpc server
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Update PROTOC_VERSION to 25.3
Signed-off-by: SammyOina <sammyoina@gmail.com>
* Refactor server reference in readme
Updated the deployment instructions in the README to point to the correct server location after consolidating test server documentation. Removed obsolete `manager-server` module and associated main entry point to align with the new architecture and streamline workflows.
Resolves issues with outdated links and cluttered repository structure.
Signed-off-by: SammyOina <sammyoina@gmail.com>
---------
Signed-off-by: SammyOina <sammyoina@gmail.com>
Sammy Kerata Oina
Danko Miladinovic
Washington Kigani Kamadi
b1ackd0t
Smith Jilks
Filip Bugarski