18 Commits

Author	SHA1	Message	Date
Sammy Kerata Oina	da31d76c94	NOISSUE - Agent Pull mode for remote resources (#575) ... * feat(kbs): implement KBS client for attestation and resource retrieval - Added KBS client implementation in pkg/kbs/client.go with methods for attestation and resource retrieval. - Introduced necessary data structures for requests and responses. - Implemented error handling for various scenarios. test(kbs): add unit tests for KBS client - Created comprehensive tests for the KBS client in pkg/kbs/client_test.go. - Included tests for attestation success and failure cases, as well as resource retrieval. feat(registry): introduce HTTP and S3 registry implementations - Added HTTPRegistry for downloading resources over HTTP/HTTPS with retry logic in pkg/registry/http.go. - Implemented S3Registry for downloading resources from AWS S3 and S3-compatible services in pkg/registry/s3.go. - Included error handling and configuration options for both registries. chore(registry): define registry interface and configuration - Created registry interface and configuration struct in pkg/registry/registry.go. - Added default configuration settings for registry clients. docs(cvms): update README for CVMS server configuration and usage - Enhanced documentation for CVMS server with detailed command-line flags and usage examples. - Clarified direct upload and remote resource modes, including KBS integration. fix(cvms): integrate KBS for remote resource handling in main.go - Updated main.go to support remote datasets and algorithms using KBS. - Added validation for command-line flags to ensure proper configuration. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix: Move ifeq conditional outside define block in attestation-service.mk Make conditionals cannot be evaluated inside define...endef blocks when used as recipe bodies. Restructured to define the ATTESTATION_SERVICE_INSTALL_INIT_SYSTEMD block conditionally based on BR2_PACKAGE_CC_ATTESTATION_AGENT configuration. * feat: Implement remote resource downloading for algorithms and datasets using AWS S3/MinIO credentials. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Add comprehensive documentation and agent support for testing remote resource download with KBS attestation. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Improve agent logging for remote resource configuration and KBS status, and add a testing guide for remote resource downloads with KBS attestation. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Add a comprehensive guide for testing remote resource download with KBS attestation and update multiple package versions to a specific commit. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Add failure transitions for resource reception states and a comprehensive guide for testing remote resource downloads with KBS attestation. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Implement remote resource download with KBS attestation in the agent and add a comprehensive testing guide. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * test: Add comprehensive guide for testing remote resource download with KBS attestation and include a debug log in the attestation client. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Delegate KBS attestation and token retrieval to a new attestation-agent service and document remote resource testing. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * client fixes Signed-off-by: Sammy Oina <sammyoina@gmail.com> * raw evidence Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix: Build all Go files in cmd directories, not just main.go This fixes the issue where fetch_raw_evidence.go wasn't being included in the attestation-service build. * fix: Wrap binary evidence in JSON for KBS compatibility Fixes 'invalid character' error by wrapping raw binary evidence in a JSON structure with base64 encoding, as expected by KBS. * chore: Update buildroot packages to c28cefae Includes fixes for: 1. attestation-service build (including fetch_raw_evidence.go) 2. Agent KBS evidence format (wrapping binary in JSON) * fix: Implement KBS RCAR handshake with cookies Fixes 'cookie not found' error (401) from KBS by: 1. Adding CookieJar support to KBS client 2. Implementing GetChallenge() to perform /auth handshake and capture session cookie 3. Updating Agent to get challenge, decode nonce, and use it for evidence generation 4. Regenerating mocks * chore: Update buildroot packages to f6981ac5 Includes KBS RCAR handshake fix (cookie support + GetChallenge loop) * fix: Update KBS client JSON tags to kebab-case Fixes deserialization error (401) from KBS by: 1. Using kebab-case (e.g. extra-params) for JSON tags as per protocol. 2. Initializing ExtraParams as empty object {} instead of null/omitted. * fix: Wrap attestation evidence in primary_evidence format Updates Agent to construct 'tee-evidence' payload with: - primary_evidence: containing the actual quote/data - additional_evidence: empty JSON object This matches the Confidential Containers KBS Attestation Protocol requirements. * fix: Update KBS protocol version to 0.4.0 KBS rejected 0.1.0 with a version mismatch error. Bumping to 0.4.0 to match server expectation. * fix: Generate ephemeral key for KBS RuntimeData Updates RuntimeData to include a valid ephemeral EC P-256 public key in JWK format, as required by the KBS RCAR protocol. Also fixes the KBS client struct to support TEEPubKey as an object. * fix: Update sample attestation quote to valid JSON The default attestation.bin was binary, but the KBS Sample Verifier expects a valid JSON quote containing 'svn' and 'report_data'. Updated the embedded bin file to contain this JSON structure. * fix: Generate dynamic JSON quote for Sample TEE in FetchRawEvidence The KBS Sample Verifier expects a JSON object with 'svn' and 'report_data'. Previously, we were returning raw binary data (reportData+nonce). This commit updates FetchRawEvidence to return a marshaled JSON structure with: - svn: "1" - report_data: base64(req.ReportData) * refactor: Delegate Sample Attestation to Provider Refactored sample attestation logic: - Moved JSON Quote generation into EmptyProvider (standalone mode). - Updated FetchRawEvidence to call provider.TeeAttestation instead of manual generation. This enables using the real CC Attestation Agent for UNSPECIFIED platform if configured. * feat: Add comprehensive debug logging and enforce CC AA usage Changes: - Updated EmptyProvider to return error instead of generating mock data This forces proper use of CC Attestation Agent's sample attester - Added detailed logging to attestation-service FetchRawEvidence: * Hex dump of evidence (first 200 bytes) * String preview of evidence * Total evidence length - Added detailed logging to agent service: * Raw evidence hex and string previews * KBS evidence JSON preview (first 500 bytes) * Evidence lengths at each transformation step This logging will help diagnose why KBS Sample Verifier is rejecting evidence. * fix: Enable CC AA by default and add attestation-service log forwarding Changes: - Set USE_CC_ATTESTATION_AGENT=true by default in systemd service - Added StandardOutput/StandardError to forward logs to /var/log/cocos/ - Updated HAL makefile to handle new default value - This ensures attestation-service uses CC AA's sample attester - Logs will now be visible in CVMS output for debugging * feat: Add gRPC log forwarding to attestation-service Implemented the same log forwarding mechanism used by the agent: - Added ProtoHandler to write logs to both stdout and logQueue - Connected to log client (/run/cocos/log.sock) for gRPC forwarding - Added goroutine to forward logs to CVMS via log client - Logs will now appear in CVMS output during computation runs This enables visibility into attestation-service debug output including: - CC AA connection status - Evidence generation details (hex dumps, string previews) - Any errors from providers * fix: Parse sample evidence JSON instead of base64-encoding it The attestation-service returns sample evidence as JSON: {"svn":"1","report_data":"base64..."} The agent was incorrectly base64-encoding this JSON string again. KBS Sample Verifier expects the parsed JSON object directly. Fixed by: - Parsing the JSON evidence from attestation-service - Passing the parsed object directly in primary_evidence.evidence - This matches what KBS Sample Verifier expects * debug: Increase KBS evidence logging preview to 1000 bytes Show the complete JSON structure being sent to KBS to debug the attestation failure. * debug: Add comprehensive CC AA configuration logging Added debug logs to show: - Whether CC AA is enabled in config - CC AA address being used - Connection success/failure - Which provider is ultimately selected - Warning when falling back to EmptyProvider This will help diagnose why EmptyProvider is being used instead of CC Attestation Agent. * debug: Add startup logging for log client connection Added log message to show if log client connection succeeds at attestation-service startup. This will help diagnose why logs aren't appearing in CVMS output. * feat: Add retry logic with exponential backoff to log client Added simple retry mechanism to handle concurrent log requests: - 3 retry attempts with exponential backoff (10ms, 20ms, 40ms) - Applies to both SendLog and SendEvent methods - Centralized in log client so all services benefit - Should eliminate 'failed to send log' errors from concurrent requests This fixes the issue where attestation-service logs weren't appearing in CVMS output due to dropped messages. * fix: Flatten sample evidence fields in primary_evidence for KBS KBS Sample Verifier expects svn and report_data at the top level of primary_evidence, not nested under an 'evidence' key. Changed structure from: {"primary_evidence": {"tee": "sample", "evidence": {"svn": "1", ...}}} To: {"primary_evidence": {"tee": "sample", "svn": "1", "report_data": "...", ...}} This matches what KBS expects when deserializing the Quote structure. * fix: Use sample quote directly as primary_evidence per KBS protocol According to KBS attestation protocol spec, for sample TEE type, primary_evidence should be the sample quote JSON directly: {"svn": "1", "report_data": "..."} Removed extra 'tee' and 'platform' fields that were causing KBS to fail deserializing the Quote structure. The 'tee' field is already sent in the Request payload during RCAR handshake. Refs: - https://github.com/confidential-containers/trustee/blob/main/kbs/docs/kbs_attestation_protocol.md - https://github.com/confidential-containers/guest-components/blob/main/attestation-agent/attester/src/sample/mod.rs * fix: Make CC AA required for sample attestation when configured When USE_CC_ATTESTATION_AGENT=true, attestation-service now requires AA to be available for NoCC/sample platform. This ensures sample evidence always comes from AA with the correct KBS format. Changes: - Error out if AA connection fails for NoCC platform when AA is configured - Only use EmptyProvider if AA is explicitly NOT configured - Prevents incorrect sample evidence format from EmptyProvider This ensures attestation-service delegates to AA for sample evidence generation instead of creating it itself. * fix: Implement proper RCAR protocol with tee-pubkey and runtime-data hash Fixed KBS attestation error 'REPORT_DATA is different from that in Sample Quote' Changes: 1. Generate ephemeral EC key pair BEFORE getting evidence from AA 2. Create runtime-data with nonce + tee-pubkey (JWK format) 3. Hash runtime-data (SHA-256) and use as report_data for AA 4. This binds the tee-pubkey to the TEE evidence per RCAR protocol The report_data in the evidence now matches what KBS expects: hash(runtime-data) instead of computation ID. This completes the full RCAR protocol implementation: - Request → Challenge → Attestation (with bound tee-pubkey) → Response * fix(agent): use simple nonce for Sample attestation report_data For Sample/NoCC attestation, use the raw nonce bytes directly as report_data instead of hashing runtime-data. This avoids JSON serialization mismatches with the KBS Sample verifier. Real TEEs (TDX/SNP) still use runtime-data hash binding to cryptographically bind the ephemeral tee-pubkey to the evidence. * fix(agent): use RFC 8785 canonical JSON for runtime-data hashing The KBS Sample attestation verifier (and likely others) expects the report_data to be the SHA-256 hash of the *canonical* JSON serialization (RFC 8785) of the runtime-data. Standard Go JSON marshaling does not guarantee key ordering, leading to hash mismatches. This change uses github.com/gowebpki/jcs to canonicalize the runtime-data before hashing, ensuring compatibility with the KBS RCAR implementation. Also reverted the temporary 'simple nonce' workaround. * feat(hal): add CoCo Keyprovider and Skopeo packages - Add coco-keyprovider buildroot package with systemd service - Add skopeo buildroot package for OCI image handling - Add ocicrypt_keyprovider.conf for encrypted image decryption - Update Config.in to include new packages This enables standard CoCo ecosystem integration for encrypted OCI images instead of custom S3/HTTP registry clients. * feat(oci): add OCI image handling package with Skopeo integration - Add pkg/oci/types.go with ResourceSource and ImageManifest types - Add pkg/oci/skopeo.go with Skopeo wrapper for pull/decrypt - Add pkg/oci/extract.go for extracting algorithms and datasets from layers This package provides OCI image handling using Skopeo and CoCo Keyprovider for encrypted image decryption, replacing custom S3/HTTP registry clients. * chore: regenerate protobuf files for updated cvms.proto * refactor(agent): replace S3/HTTP/KBS with OCI package - Remove pkg/kbs and pkg/registry imports - Add pkg/oci import for OCI image handling - Replace downloadAndDecryptResource with OCI-based implementation - Use Skopeo + CoCo Keyprovider for automatic decryption - Reduce code from ~240 lines to ~70 lines This eliminates custom KBS RCAR handshake, S3/HTTP registry clients, and manual decryption logic. CoCo Keyprovider handles all decryption automatically via ocicrypt protocol. * chore: remove obsolete pkg/kbs and pkg/registry packages - Delete pkg/kbs/ (custom KBS client, ~300 lines) - Delete pkg/registry/ (S3/HTTP registry clients, ~400 lines) - Remove unused imports from agent/service.go - Run go mod tidy to clean up dependencies These packages have been replaced by pkg/oci with Skopeo and CoCo Keyprovider for standard CoCo ecosystem integration. * fix(agent): update ResourceSource struct to include type and encryption fields Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix(hal): update CoCo Keyprovider to v0.16.0 and fix build path - Update version from v0.11.0 to v0.16.0 (matches attestation agent) - Fix install path: target is at repo root, not in coco_keyprovider subdir - This fixes the build error where coco_keyprovider binary wasn't found The cargo workspace in guest-components builds to a shared target/ directory at the repository root, not within each crate's subdirectory. * feat: Update remote resources testing guide to use kbs-client and coco-keyprovider for key management and encryption, enable insecure TLS for Skopeo, and enhance CVMS with Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Update component versions, revise image encryption documentation, and sanitize OCI image paths for Skopeo compatibility. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Add `decompress` option to Dataset and `algo_type`/`algo_args` to Algorithm protobuf messages, updating client, test, and build configurations. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Update multiple package versions and enhance OCI image extraction error reporting for missing algorithm files. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * chore: Bump package versions, improve OCI image extraction debugging by returning seen files, and remove unused dataset type parsing from test code. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * refactor: Migrate OCI extraction to use structured logging with `slog` and `context`, and update package versions. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Bump multiple component versions, add encrypted status for computation inputs and algorithms, and refine OCI layer extraction warnings. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * logging Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: Add `Encrypted` field to algorithm and dataset resource sources and update all component versions. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: update component versions, integrate coco-keyprovider service, and configure ocicrypt key provider. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: add support for KBS parameters and dataset/algorithm hash calculations in CVMS Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: update resource download and extraction logic to support requirements.txt and improve hash verification Signed-off-by: Sammy Oina <sammyoina@gmail.com> * chore: Update dependencies, improve code style, and add GetRawEvidence to attestation client mocks. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Refactor code structure for improved readability and maintainability Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix: update golangci configuration to include errcheck for build path and remove unnecessary exclusions Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix: streamline kernel command line handling in QEMU args construction Signed-off-by: Sammy Oina <sammyoina@gmail.com> * feat: add attestation binary and update checksum tests and policy structure Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Add unit tests for attestation agent, attestation, log, crypto, OCI, and Skopeo clients - Implement tests for the attestation agent client including Unix socket and TCP address handling, token retrieval, and error scenarios. - Enhance attestation client tests to cover fetching raw evidence for various platforms (SNP, TDX, VTPM, SNPvTPM) and validate error handling. - Introduce log client tests to verify retry behavior for sending logs and events. - Create comprehensive tests for crypto package focusing on AES-GCM decryption, encrypted resource parsing, and key unwrapping. - Add tests for OCI package to validate algorithm and dataset extraction, including JSON serialization of OCILayout. - Implement Skopeo client tests to ensure proper functionality for image pulling, inspecting, and resource source handling. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix: handle JSON marshal errors in test cases for decrypt and extract functions Signed-off-by: Sammy Oina <sammyoina@gmail.com> * test: add comprehensive tests for algorithm and dataset extraction with various scenarios Signed-off-by: Sammy Oina <sammyoina@gmail.com> * refactor: replace hardcoded Python script content with constant variable Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix: remove redundant mock expectation for SendAgentConfig in TestCreateVMWithAaKbsParams Signed-off-by: Sammy Oina <sammyoina@gmail.com> * test: add tests for event sending failure, dataset extraction with path traversal, and Skopeo client behavior Signed-off-by: Sammy Oina <sammyoina@gmail.com> * test: add tests for download and decryption of resources with various URL formats Signed-off-by: Sammy Oina <sammyoina@gmail.com> * refactor: Introduce OCIClient interface for agent service to improve testability of OCI image operations and enhance related tests. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * refactor: Change `get_uint64_from_tcb` to accept `TcbVersion` by value and use `u64::from` for type conversions. --------- Signed-off-by: Sammy Oina <sammyoina@gmail.com>	2026-03-16 14:48:55 +01:00
Sammy Kerata Oina	5377dd4d7f	NOISSUE - Prepare cocos for v0.8.0 (#512) ... * Refactor mock interfaces to use 'any' instead of 'interface{}' for improved type safety and readability across multiple files in the manager and pkg directories. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Update Go version to 1.25.x in CI workflows and remove obsolete Go package files Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Add mock implementations for various components in the attestation and SDK packages - Created mock for MeasurementProvider in pkg/attestation/cmdconfig/mocks/mocks_test.go - Created mock for Provider in pkg/attestation/mocks/mocks_test.go - Created mock for Client in pkg/clients/grpc/mocks/mocks_test.go - Created mock for SDK in pkg/sdk/mocks/mocks_test.go These mocks are generated using mockery and are intended for unit testing purposes. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Remove autogenerated mock files and update mock usage in tests - Deleted mocks for gRPC clients in pkg/clients/grpc/mocks/mocks_test.go and pkg/sdk/mocks/mocks_test.go. - Updated test files in pkg/progressbar/progress_test.go to use the new mock structure without type parameters for gRPC client interfaces. - Refactored mock generation in pkg/sdk/mocks/sdk.go to streamline the mock creation process and ensure consistency across mock methods. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Update protobuf generated files for events and manager - Bump protoc-gen-go version from v1.36.5 to v1.36.8 in events.pb.go and manager.pb.go. - Refactor raw descriptor definitions in events.pb.go and manager.pb.go to use string concatenation for better readability and maintainability. - Ensure compatibility with the latest protobuf specifications and improve code generation consistency. Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Update test commands to use GOTOOLCHAIN for consistent Go version handling Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Fix GOTOOLCHAIN usage in test command for consistency Signed-off-by: Sammy Oina <sammyoina@gmail.com> --------- Signed-off-by: Sammy Oina <sammyoina@gmail.com>	2025-09-01 14:28:11 +02:00
Sammy Kerata Oina	ccab296b62	NOISSUE - Downgrade Golang version for HAL (#494) ... * Update Go version to 1.24.x in CI workflows and fix supermq version in go.mod Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Refactor CI workflow to separate linting and testing jobs, and streamline test execution for multiple modules Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Downgrade Go version from 1.23.10 to 1.23.8 in go.mod Signed-off-by: Sammy Oina <sammyoina@gmail.com> --------- Signed-off-by: Sammy Oina <sammyoina@gmail.com>	2025-08-07 13:51:23 +02:00
Danko Miladinovic	3102114ff3	COCOS-395 - Cloud Provider Firmware Integration (#415) ... * add CC platform identification capability * add token verification * add snp azure * add azure snp report verification * fix linter errors * fix agent tests * expand the CC provider * fix azure atls * rebase branch * add nonce check for azure token * rename package attestations * remove alias attestations --------- Co-authored-by: Ubuntu <azureuser@UVCTestCVM.bu0p0zdolasezg1jifpyqhaxuc.dx.internal.cloudapp.net>	2025-05-19 16:42:39 +02:00
Sammy Kerata Oina	12a20c74f7	NOISSUE - Reset agent state machine when computation is stopped (#417) ... * Bump mockery version to 2.53.3 in generated mock files Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Update golangci-lint action to v7 and bump version to v1.64.7 Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Update golangci-lint action to v7 and configure linters in .golangci.yaml Signed-off-by: Sammy Oina <sammyoina@gmail.com> --------- Signed-off-by: Sammy Oina <sammyoina@gmail.com>	2025-04-02 14:36:35 +02:00
Sammy Kerata Oina	913c6138cf	NOISSUE - Update Dependabot configuration and Go dependencies (#378) ... * Update Dependabot configuration and Go dependencies; modify test command to include embed tag Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor CVMSClient computation handling and improve test message queue capacity Signed-off-by: Sammy Oina <sammyoina@gmail.com> * Refactor TestManagerClient_Process to remove buffer from messageQueue and ensure proper handling of messages Signed-off-by: Sammy Oina <sammyoina@gmail.com> --------- Signed-off-by: SammyOina <sammyoina@gmail.com> Signed-off-by: Sammy Oina <sammyoina@gmail.com>	2025-02-26 14:04:09 +01:00
Sammy Kerata Oina	184617da9e	NOISSUE - Host data verification (#275) ... * host data verification Signed-off-by: Sammy Oina <sammyoina@gmail.com> * update mocks Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix test Signed-off-by: Sammy Oina <sammyoina@gmail.com> * debug host data Signed-off-by: Sammy Oina <sammyoina@gmail.com> * debug Signed-off-by: Sammy Oina <sammyoina@gmail.com> * check device Signed-off-by: Sammy Oina <sammyoina@gmail.com> * imorove test Signed-off-by: Sammy Oina <sammyoina@gmail.com> * missing header Signed-off-by: Sammy Oina <sammyoina@gmail.com> * update embed option Signed-off-by: Sammy Oina <sammyoina@gmail.com> * minor fixes Signed-off-by: Sammy Oina <sammyoina@gmail.com> * add tests Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix lint Signed-off-by: Sammy Oina <sammyoina@gmail.com> * update deps Signed-off-by: Sammy Oina <sammyoina@gmail.com> * each case is unique Signed-off-by: Sammy Oina <sammyoina@gmail.com> * all files Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix coverage Signed-off-by: Sammy Oina <sammyoina@gmail.com> * resolve comments Signed-off-by: Sammy Oina <sammyoina@gmail.com> * improve coverage Signed-off-by: Sammy Oina <sammyoina@gmail.com> * add test case Signed-off-by: Sammy Oina <sammyoina@gmail.com> * add test cases Signed-off-by: Sammy Oina <sammyoina@gmail.com> * use consts Signed-off-by: Sammy Oina <sammyoina@gmail.com> * add coverage Signed-off-by: Sammy Oina <sammyoina@gmail.com> * make sure pid is exited Signed-off-by: Sammy Oina <sammyoina@gmail.com> --------- Signed-off-by: Sammy Oina <sammyoina@gmail.com>	2024-10-14 11:12:49 +02:00
Sammy Kerata Oina	115c6c24c0	NOISSUE - Fix file format (#261) ... * add coverage Signed-off-by: Sammy Oina <sammyoina@gmail.com> * use codecov Signed-off-by: Sammy Oina <sammyoina@gmail.com> * rename extension Signed-off-by: Sammy Oina <sammyoina@gmail.com> --------- Signed-off-by: Sammy Oina <sammyoina@gmail.com>	2024-09-27 10:52:52 +02:00
Sammy Kerata Oina	c69dcd0e2d	NOISSUE - Improve reliability of state machine test (#260) ... * add sleep to prevent test failing Signed-off-by: Sammy Oina <sammyoina@gmail.com> * add coverage Signed-off-by: Sammy Oina <sammyoina@gmail.com> * use codecov Signed-off-by: Sammy Oina <sammyoina@gmail.com> * create dir Signed-off-by: Sammy Oina <sammyoina@gmail.com> --------- Signed-off-by: Sammy Oina <sammyoina@gmail.com>	2024-09-26 11:59:26 +02:00
Sammy Kerata Oina	46d24f928a	NOISSUE - Add events for disconnection of agent (#233) ... * add events Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix lint Signed-off-by: Sammy Oina <sammyoina@gmail.com> * typo Signed-off-by: Sammy Oina <sammyoina@gmail.com> * group logs Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix error Signed-off-by: Sammy Oina <sammyoina@gmail.com> * fix initialization of goroutine Signed-off-by: Sammy Oina <sammyoina@gmail.com> * add comment Signed-off-by: SammyOina <sammyoina@gmail.com> * update comment Signed-off-by: SammyOina <sammyoina@gmail.com> * fix lint Signed-off-by: SammyOina <sammyoina@gmail.com> * remove naked return Signed-off-by: Sammy Oina <sammyoina@gmail.com> --------- Signed-off-by: Sammy Oina <sammyoina@gmail.com> Signed-off-by: SammyOina <sammyoina@gmail.com>	2024-09-11 14:26:46 +02:00
b1ackd0t	dc16e8a997	NOISSUE - Use Magistrala Shared Packages (#174) ... * refactor(env): remove internal env package No need for this package since the parent package github.com/caarlos0/env does everything we need Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * refactor(jaeger): remove internal jaeger package No need for this package since we can use magistrala exported package Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * refactor(metrics): remove internal metrics pkg Use exported magistrala prometheus package Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * chore(dep): Update grpc and other dependencies Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> * style(linter): remove enabled by default linters Signed-off-by: Rodney Osodo <socials@rodneyosodo.com> --------- Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>	2024-07-10 15:48:37 +02:00
Sammy Kerata Oina	2ce112cc1b	COCOS-103 - User authN and AuthZ using digital signatures (#128) ... * Update Go to 1.22 and enhance security features - Upgraded the Go version in GitHub Actions workflows to 1.22.x for latest features and security patches. - Added RSA public key field `UserKey` in `Dataset` and `Algorithm` to reinforce data integrity and encryption. - Refactored `Result` method in `agentService` to use `containsID` for improved readability and potential performance benefits. - Updated `grpcserver.New` and `internal/server/grpc` invocations to pass `agent.Service` by value in line with recommended Go practices. - Introduced `grpc.StreamInterceptor` with no args in `Server.Start` which seems to be an initial step for future stream interceptor configuration. These changes prepare for stronger data security measures, maintain compatibility with the latest Go features, and improve code quality regarding service struct usage. Potential follow-up is needed to configure the stream interceptor and to ensure the new RSA key field is appropriately utilized in data handling. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor auth system and protocol buffers Enhanced the authentication system by adding context support and an improved user-role model. Implemented robust RSA public key verification for users and a restructured interceptor logic specific to stream types, streamlining the auth process. Updated protocol buffers and associated structures to accommodate user keys as byte slices, aligning with standard cryptographic practice. CLI commands for algorithms and datasets now require a private key file path argument for signing, strengthening security during interactions. This comprehensive overhaul addresses security and efficiency considerations in the RPC framework and aligns with best practices for key handling. By streamlining and securing the user authentication process, the agent service's reliability is greatly improved, directly impacting the robustness of the entire computation pipeline. - Refactored auth: added role-based user validation, context handling - Reworked interceptors: separated stream types, fortified signature checks - Updated protocol buffers: user public keys as byte slices for standard compatibility - Enhanced CLI: introduced private key argument, ensuring secure algorithm and dataset submission - Improved server and SDK contracts to align with auth changes Related issues: - Implements user roles and auth context [#103] - CLI security enhancement for private key management Signed-off-by: SammyOina <sammyoina@gmail.com> * Updated PEM decoding for key parsing in CLI and tests Added `encoding/pem` to decode PEM blocks when parsing private and public keys across CLI commands and test computation scenarios, ensuring compatibility with key files. This enhances robustness in key handling by supporting PEM encoded keys. The update also includes registration of a new Keys command in the CLI. Refactored code is now compliant with common key formats, addressing potential parsing issues. Signed-off-by: SammyOina <sammyoina@gmail.com> * Fix auth signature encoding and improve CLI usage example The authentication system now decodes base64 strings before verifying signatures to align with the expected format. Additionally, the signature generation now encodes the output in base64, ensuring consistency across the auth process. The CLI help message for the `result` command is enhanced by providing a usage example, making it more user-friendly and informative. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor containsID to handle dynamic fields Updated the `containsID` function to accept a field name parameter, enabling dynamic field lookup within the reflection logic. This change facilitates the use of the function for various struct fields, improving code reusability and flexibility. CLI command 'data' now requires an additional argument for the private key file path, outlined in the usage example update, reinforcing command clarity and user guidance. Resolves issues with hardcoded field lookups and enhances CLI usability. Signed-off-by: SammyOina <sammyoina@gmail.com> * Remove extraneous newline in key generation log output A redundant newline after the success message in the key generation command was removed to clean up log output formatting. This change ensures a more consistent and professional appearance of the CLI tool's messages. Signed-off-by: SammyOina <sammyoina@gmail.com> * Implemented auth service in gRPC startup Added authentication services to the gRPC server initialization to enforce security measures. The gRPC server's New function now includes an `authSvc` parameter, requiring instantiation of the auth service before starting the server. Failure to create the auth service results in a fatal error, halting the process to avoid running without protection. Tests have been updated to include `nil` values for the auth service parameter to maintain their functionality without authentication. Refactored `grpcserver.New` to accept the new auth service, and updated the main agent startup logic to create and inject the auth service. Added the auth middleware interceptors to the server options, which ensures that each gRPC call will undergo authentication. This change is a step towards secure communication, and affected components should now consider the authentication requirement. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor config read logic and update agent setup Improved the configuration reading in `cmd/agent/main.go` to handle larger payloads by reading data in chunks and checking for EOF, ensuring that all config data is captured even if it exceeds the initial buffer size. Enhanced the `test/manual/agent-config/main.go` to require additional command-line arguments, improving the setup process by explicitly requiring paths for data, algorithm, and public key as well as a boolean for attested TLS. Also updated the hashing method to SHA3 for the algorithm and data files, and included the hash and public keys as part of the agent, dataset, and result consumer configurations. These changes will make the agent setup more robust and provide better integrity checks for the involved files. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor run method to agentService Moved the run function into agentService for better encapsulation and maintainability. This refactoring includes capturing both stdout and stderr during algorithm execution, enabling more informative debugging through enhanced logging. Consequentially, the run method now references members through the service instance, aligning with object-oriented best practices and improving code coherence. Resolves issue with insufficient execution details when computations fail. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor computation data handling to use filepaths Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor error logging and ensure consistency Replaced usage of the standard log package with a custom logger for error reporting to standardize error logging throughout the application. Additionally, introduced graceful shutdown by returning from the main function rather than forcing exit when failing to create auth service, aligning the application's error handling strategy. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor auth initialization and key file handling Improved the readability and maintainability of the authentication service initialization by adding line breaks for logical separation. Also, standardized key filenames in the CLI key generation by introducing constants, enhancing code clarity and reducing the likelihood of file-naming errors. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor auth verification logic for improved security Removed an extraneous line in the `verifySignature` function that was not necessary for the signature verification process. This change simplifies the code and improves readability. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor payload structures to simplify API Removed the 'provider', 'id', 'consumer' fields from protocol buffers, gRPC services, and related functions across various files to streamline the data model and align with the new authentication system based on cryptographic verification rather than string identifiers. This results in more efficient data handling and a reduction in unnecessary payload data, while enhancing security by making entity validation strictly cryptographic. The changes affect agent-SDK interactions, CLI tools, and related services, ensuring only the necessary data (algorithm/data bytes, user keys, and hashes) is transmitted and processed. Consequently, the core computation algorithm and dataset handlers now rely on indexes derived from context to associate data with respective manifest entries, thus maintaining the ability to link to specific computation manifests without relying on explicit IDs in the payload. Additionally, refactored authentication methods now enforce role-based security seamlessly through metadata. This approach enhances privacy by avoiding transmission of potentially sensitive strings over the network and by ensuring that only internal indices, not globally interpretable identifiers, are used to process computations. Aligned with the broader architectural goal of simplifying and securing the platform's core services, this change paves the way for upcoming revisions to the authentication scheme that will further consolidate role-based security and improve system integrity. Signed-off-by: SammyOina <sammyoina@gmail.com> * Enhance CLI security with key paths Removed the section on running computations from the CLI README as it may no longer be necessary or the functionality has been moved elsewhere. Required private key file paths for algorithm, dataset upload, and result retrieval commands to enhance security. This change associates each action with a specific identity, ensuring secure and traceable operations. Additionally, updated the manual test commands to reflect this new requirement. Signed-off-by: SammyOina <sammyoina@gmail.com> * fix ci Signed-off-by: SammyOina <sammyoina@gmail.com> * fix fmt Signed-off-by: SammyOina <sammyoina@gmail.com> --------- Signed-off-by: SammyOina <sammyoina@gmail.com>	2024-05-28 14:10:13 +02:00
Sammy Kerata Oina	8975e28437	COCOS-77 - Stream agent logs and events over gRPC (#78) ... * Refactor GRPC manager service and client The manager service and client have been restructured for stream communication, facilitating real-time agent events, logs, and run responses. The `Run` RPC is replaced by the `Process` stream RPC, enabling bidirectional streaming between clients and the manager service. This allows continuous interchange of different message types including `WhoAmIRequest`, `AgentLog`, `AgentEvent`, and `RunResponse`. Several message types have been adjusted and new fields introduced, like `AgentPort` in `RunResponse` and various agent-config attributes including CA files and instance IDs, to support TLS client authentication and distinguish between agent instances. We've also incorporated `google.protobuf.Timestamp` in `AgentEvent` for precise event logging. The client code reflects these modifications with updated method calls and stream handling logic for ongoing communication. Moreover, the updates necessitate corresponding changes throughout service, grpc, and sdk layers to interoperate with the new streaming approach. The transition to streaming paves the way for a more interactive, flexible communication system that can accommodate future expansion and real-time monitoring features. Signed-off-by: SammyOina <sammyoina@gmail.com> * fix lint Signed-off-by: SammyOina <sammyoina@gmail.com> * Update GitHub Actions to Latest Versions Upgraded GitHub Actions 'checkout' to version 4 and 'setup-go' to version 5 across various workflow files to leverage the latest features and improvements for better performance and reliability. This also ensures compatibility with Go version 1.21.x which is specified in the workflows. Signed-off-by: SammyOina <sammyoina@gmail.com> * Refactor event handling and logging Reworked event and log processing to use channels instead of direct HTTP calls. Removed obsolete events package and consolidated event structures, leading to cleaner and more maintainable code. Updated agent events to use channels, enhanced error handling in log forwarding, and simplified manager `New` function signature to accept an event channel directly. - Removed `events` and `agentevents` packages to reduce complexity. - Replaced direct event server communication with internal channel usage. - Introduced `AgentEvent` struct in events.go for standardized event objects. - Adapted `managerService` to dispatch events and logs through channels. - Streamlined manager construction by removing the now-unnecessary event service and host IP parameters. This change results in a more robust and easier to extend event and log management system within the agent-manager interaction. Signed-off-by: SammyOina <sammyoina@gmail.com> * fix ci Signed-off-by: SammyOina <sammyoina@gmail.com> * remove unused code Signed-off-by: SammyOina <sammyoina@gmail.com> * add comments Signed-off-by: SammyOina <sammyoina@gmail.com> --------- Signed-off-by: SammyOina <sammyoina@gmail.com>	2024-02-07 12:08:39 +01:00
Sammy Kerata Oina	f1f8f95653	NOISSUE - Fix dependencies (#13) ... * vendor Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Return agent changes Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Add missing import Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * remove vendor Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Fix formatting Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Formatting errors Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Update agent/api/grpc/client.go Signed off: WashingtonKK washingtonkigan@gmail.com Co-authored-by: Sammy Kerata Oina <44265300+SammyOina@users.noreply.github.com> * add linters and fix Signed-off-by: SammyOina <sammyoina@gmail.com> * update ci Signed-off-by: SammyOina <sammyoina@gmail.com> * remove deprecated dependencies and use local agent and manager Signed-off-by: SammyOina <sammyoina@gmail.com> * update mainflux Signed-off-by: SammyOina <sammyoina@gmail.com> * Fix Jaeger URL in agent and manager main.go files The Jaeger URL in the agent and manager main.go files was incorrect. This commit fixes the Jaeger URL by updating it to "http://localhost::4318/v1/traces". Signed-off-by: SammyOina <sammyoina@gmail.com> --------- Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> Signed-off-by: SammyOina <sammyoina@gmail.com> Co-authored-by: WashingtonKK <washingtonkigan@gmail.com>	2023-10-27 22:12:53 +02:00
Washington Kigani Kamadi	7d47ed7a20	Fix CI (#6) ... * Fix CI Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Fix CI Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> * Use single make Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> --------- Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>	2023-10-20 17:25:14 +02:00
SammyOina	7cb04499e8	Update Go version to 1.21.x in GitHub Actions workflow ... The Go version in the GitHub Actions workflow was updated from 1.20.x to 1.21.x. This change ensures that the correct version of Go is used during the workflow execution. Signed-off-by: SammyOina <sammyoina@gmail.com>	2023-10-19 12:30:00 +03:00
rodneyosodo	2c54aa78d3	Update Build Command ... Signed-off-by: rodneyosodo <blackd0t@protonmail.com>	2023-07-12 13:34:56 +03:00
rodneyosodo	f4e71e74b2	Initial Commit: Add CI ... Signed-off-by: rodneyosodo <blackd0t@protonmail.com>	2023-07-10 15:06:45 +03:00