Compare commits

..

341 Commits

Author SHA1 Message Date
Sammy Kerata Oina e376cf35a4 feat: Add support for GCC and libstdc++ packages in cocos_defconfig (#188)
Signed-off-by: Sammy Oina <sammyoina@gmail.com>
2024-07-31 12:43:40 +02:00
Danko Miladinovic 24a76a1685 NOISSUE - Fix attested TLS and attestation CLI (#184)
* update code to reflect updated libraries

* fix attestation CLI

* update manual test for attestation CLI
2024-07-30 16:05:49 +02:00
b1ackd0t 6d170e9ca4 NOISSUE - Fix failing tests (#186)
* test(events): failing tests

* fix: Add datapath as optional parameter

If datapath is not provided if should work as assume dataset is empty
If an invalid i.e non existent path is given it should return an error
2024-07-26 12:05:26 +02:00
b1ackd0t c9ec42deb2 fix(computation): Allow computation with empty dataset (#185) 2024-07-26 11:22:13 +02:00
Washington Kigani Kamadi 7498800d2d update attestation command example (#183)
Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2024-07-25 01:25:31 +02:00
Sammy Kerata Oina 46555a72ba NOISSUE - Error logs instead of failure on stderr (#182)
* warning logs

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* switch to error

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

---------

Signed-off-by: Sammy Oina <sammyoina@gmail.com>
2024-07-24 14:28:51 +02:00
Sammy Kerata Oina 67d01e39be COCOS-155 - Add python algo support (#178)
* * feat(algorithm.go): add support for algorithm type context
* feat(python.go): implement Python algorithm runtime
* fix(cocos_defconfig): add IPTABLES package

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update proto

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* small fixes

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* add metadata

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* debug

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* debug

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* chunk logger

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* debug logger

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* test lock

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* add req file

Signed-off-by: SammyOina <sammyoina@gmail.com>

* stream result

Signed-off-by: SammyOina <sammyoina@gmail.com>

* test with venv

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* fix missing requirements file

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* result stream

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* modify test server

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* remove debugging and cleaning up

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* original repo

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* add missing header

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

* downgrade protoc

Signed-off-by: Sammy Oina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
Signed-off-by: Sammy Oina <sammyoina@gmail.com>
2024-07-23 16:38:03 +02:00
b1ackd0t 2f8109879c COCOS-168 - Allow running Computations without datasets (#175)
* feat(agent): Allow empty dataset

Allow running of algorithm with empty dataset since not all algorithms require datasets.
Allow state-machine transition from algo-received state to running state incase of no dataset provided

Fixes https://github.com/ultravioletrs/cocos/issues/168

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* chore(gitignore): Remove build artefacts

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* feat(algorithms): Add test algorithm for addition

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* refactor(addition): Modify addition algo to one file

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* fix(agent): move state transition to callback func

Move state transition from `receivingAlgorithm` to `running` to state call back function

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* feat(agent-event): Add `algoReceivedNoData` event

`algoReceivedNoData` is an event that is sent if we receive an algorithm and it should not have a dataset hence changes the state from `receivingAlgorithm` to `running`

* fix(agent-state): Change state depending on manifest

Change state from `receivingAlgorithm` to either `receivingData` if there is a dataset or `running` if there is no dataset provided

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

---------

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
2024-07-22 14:51:16 +02:00
Smith Jilks 88707c4668 NOISSUE - Fix Jaeger tracer (#177) 2024-07-22 14:49:42 +02:00
Washington Kigani Kamadi d76074ae41 COCOS-144 - Add Agent SDK Tests (#167)
* add tests and mocks

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix ci

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

update test

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix(agent/grpc): revert change

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix ci

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* refactor attestation and report tests

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

refactor tests

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

remove commented code

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

remove comment

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

remove comments

* add test cases

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

export agent errors

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

remove comm

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* fix tests

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

---------

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2024-07-11 20:53:31 +02:00
Sammy Kerata Oina 2ceb1c3562 COCOS-152 - Refactor algorithm execution and add visibility through logging and events (#176)
* * feat(agent): add support for binary algorithm execution
*
* feat(agent/algorithm): add Algorithm interface and binary implementation
*
* feat(agent/algorithm/binary): implement Run method for binary algorithm execution
*
* feat(agent/algorithm/logging): implement Stdout and Stderr writers for algorithm logging
*
* feat(agent/algorithm/logging_test): add tests for Stdout and Stderr writers
*
* feat(agent/events): add Service interface for sending events
*
* feat(agent/events/mocks): add mock implementation for Service interface
*
* refactor(agent/service): update runComputation method to use binary algorithm implementation

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * fix(logging.go): handle error when sending event in Write method of Stderr struct
* test(logging_test.go): add copyright header
* fix(backend_info.go): add missing type declaration in function signature
* fix(agent.go): rename progressbar variable to pb for clarity and consistency

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-07-11 12:24:19 +02:00
b1ackd0t dc16e8a997 NOISSUE - Use Magistrala Shared Packages (#174)
* refactor(env): remove internal env package

No need for this package since the parent package github.com/caarlos0/env does everything we need

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* refactor(jaeger): remove internal jaeger package

No need for this package since we can use magistrala exported package

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* refactor(metrics): remove internal metrics pkg

Use exported magistrala prometheus package

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* chore(dep): Update grpc and other dependencies

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

* style(linter): remove enabled by default linters

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

---------

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
2024-07-10 15:48:37 +02:00
Sammy Kerata Oina f4e3e8e09c COCOS-157 - Provide abstractions for VM management (#171)
* abstract vm creation and allow stopping computation

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor QEMU configuration loading and execution in main.go

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * feat(agent-config): add support for sending agent configuration to manager

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * chore(checkproto.yaml): update protoc-gen and protoc-grpc versions

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * chore(auth): update mockery version to v2.43.2
* chore(main.go): update import path for vm package in agent
* chore(main.go): update import path for vm package in manager
* chore(go.mod): add github.com/google/logger v1.1.1 as a required dependency
* chore(manager_test.go): update import path for vm package in manager
* chore(logging.go): move logging.go to manager/qemu/vm package
* chore(logging_test.go): move logging_test.go to manager/qemu/vm package
* chore(vm_factory.go): rename vm_factory.go to provider.go in manager/qemu/vm/mocks package
* chore(vm.go): move vm.go to manager/qemu/vm package
* chore(vm.go): update import path for vm package in manager
* chore(vm_test.go): move vm_test.go to manager/qemu/vm package
* chore(vsock.go): move vsock.go to manager

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * fix(main.go): change import path for 'github.com/ultravioletrs/cocos/manager/qemu/vm' to 'github.com/ultravioletrs/cocos/manager/vm'
* fix(main.go): change vsock.Dial argument from 'vm.VsockConfigPort' to 'qemu.VsockConfigPort'
* fix(main.go): change import path for 'github.com/ultravioletrs/cocos/manager/qemu' to 'github.com/ultravioletrs/cocos/manager/qemu'

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-07-10 15:33:54 +02:00
Sammy Kerata Oina d1c9834d86 NOISSUE - Fix user role on RPC call (#173)
* * fix(interceptors.go): change role from auth.DataProviderRole to auth.ConsumerRole in AuthenticateUser call

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * test(interceptors_test.go): update role variable values in TestAuthUnaryInterceptor
* - Change role value from auth.DataProviderRole to auth.ConsumerRole in the "authorized result method" test case
* - Change role value from auth.DataProviderRole to auth.ConsumerRole in the "unauthorized result method" test case
* - Change role value from auth.DataProviderRole to auth.ConsumerRole in the "other method" test case

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-07-10 13:42:41 +02:00
Danko Miladinovic 654be60a80 COCOS-132 - Add progress bar for algo and data uploads (#162)
* add progress bar to CLI

* fix error handling

* fix comments errors

* add header

* add wraper for AlgoClient and DataClient

* add compile time check for wrapper structs

* refactor code
2024-07-09 09:35:40 +02:00
Danko Miladinovic 006897a57c COCOS-153 - Add host-data option (#163)
* add host_data option

* add CLI hostdata option and rename platform to backend

* move code for computation hash to a function

* rename getComputationHash to computationHash

* add default for backend information field switch
2024-07-08 15:32:13 +02:00
Sammy Kerata Oina 0e236bf2e8 * feat(hal/linux): add support for wasmtime package (#166)
* fix(hal/linux/Config.in): add source for wasmtime Config.in file

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-07-04 17:04:51 +02:00
Washington Kigani Kamadi ff5bade57d COCOS-145 : Support Other Private/Public Key Pair Algorithms (#147)
* feat: generate multiple key types using cli

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

pass cert to agent

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix ci

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

standardize key types

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

marshal public key

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

use openssl format for keys

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* update auth

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

update makefile

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

update agent

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

enhance parsing logic

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* pass key type as flag

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix ci

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

return key on successful signature verification

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix cli

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

fix ed25519 signing

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

ci

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* refactor code

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* fix(cli/result.go): dry code

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

---------

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2024-07-03 22:56:18 +02:00
Danko Miladinovic 7c090fd19f NOISSUE - Add CLI option to download ASK and ARK (#149)
* add cli option to download ask and ark

* add ARK and ASK to cert chain of the attestation report

* fix spelling mistake

* add explanation for aTLS testing

* remove commented code
2024-07-03 16:07:19 +02:00
Sammy Kerata Oina 30092c6a45 COCOS-135 - Refactor gRPC auth interceptors and update dependencies (#140)
* Implement gRPC auth interceptors and update dependencies

Introduced gRPC authentication interceptors for unary and stream calls in the agent's API to enhance security. These interceptors verify signatures based on roles before handling requests, rejecting unauthorized ones. The authentication logic was refactored, adding relevant error handling.

Added mocks for authentication interfaces to facilitate testing and replaced direct errors with wrapped ones to provide additional context.

The Makefile now includes a version pin for the Mockery tool, ensuring consistent mock generation across environments.

Updated the gRPC server setup in `grpc.go` to utilize the new interceptors. This change aims to reinforce the API's access control by authenticating users according to their roles and is expected to prevent unauthorized access to sensitive endpoints.

Lastly, dependencies in go.mod and go.sum were updated, most notably the inclusion of testify for improved testing capabilities.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix linting errors

Signed-off-by: SammyOina <sammyoina@gmail.com>

* * chore(Makefile): remove unused variables and update 'mocks' target

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-07-01 21:39:35 +02:00
Sammy Kerata Oina 617b595559 Add Docker support to system configuration (#164)
Added support for Docker in the system configuration file. This includes packages like libseccomp, CA certificates, Docker CLI, Docker Compose, Docker Engine, containerd, and runc.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-07-01 21:37:58 +02:00
Sammy Kerata Oina 4c4161ccfb Fix capitalized Vnc field and increment Vnc on run (#139)
Corrected the struct field 'vnc' to 'Vnc' to match Go's public field naming conventions and to make it exportable. Each QEMU virtual machine's VNC display number now increments upon instantiation to avoid display conflicts.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-06-06 16:34:13 +02:00
Sammy Kerata Oina ff64c10ac3 NOISSUE - Add new key management command to CLI (#136)
Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-05-29 13:09:54 +02:00
Danko Miladinovic 0574abc228 NOISSUE - Add Rust script for fetching platform data (#133)
* add rust program for fetching platform data

* fix new line error

* add CLI options to add the measurement to platform_info.json file

* add documentation for platform info testing

* add explanation for sev-snp-measure

* delete excess space

* fix minor errors

* fix minor errors

* add file permision constant
2024-05-28 18:08:07 +02:00
Sammy Kerata Oina 2ce112cc1b COCOS-103 - User authN and AuthZ using digital signatures (#128)
* Update Go to 1.22 and enhance security features

- Upgraded the Go version in GitHub Actions workflows to 1.22.x for latest features and security patches.
- Added RSA public key field `UserKey` in `Dataset` and `Algorithm` to reinforce data integrity and encryption.
- Refactored `Result` method in `agentService` to use `containsID` for improved readability and potential performance benefits.
- Updated `grpcserver.New` and `internal/server/grpc` invocations to pass `agent.Service` by value in line with recommended Go practices.
- Introduced `grpc.StreamInterceptor` with no args in `Server.Start` which seems to be an initial step for future stream interceptor configuration.

These changes prepare for stronger data security measures, maintain compatibility with the latest Go features, and improve code quality regarding service struct usage. Potential follow-up is needed to configure the stream interceptor and to ensure the new RSA key field is appropriately utilized in data handling.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor auth system and protocol buffers

Enhanced the authentication system by adding context support and an improved user-role model. Implemented robust RSA public key verification for users and a restructured interceptor logic specific to stream types, streamlining the auth process. Updated protocol buffers and associated structures to accommodate user keys as byte slices, aligning with standard cryptographic practice. CLI commands for algorithms and datasets now require a private key file path argument for signing, strengthening security during interactions.

This comprehensive overhaul addresses security and efficiency considerations in the RPC framework and aligns with best practices for key handling. By streamlining and securing the user authentication process, the agent service's reliability is greatly improved, directly impacting the robustness of the entire computation pipeline.

- Refactored auth: added role-based user validation, context handling
- Reworked interceptors: separated stream types, fortified signature checks
- Updated protocol buffers: user public keys as byte slices for standard compatibility
- Enhanced CLI: introduced private key argument, ensuring secure algorithm and dataset submission
- Improved server and SDK contracts to align with auth changes

Related issues:
- Implements user roles and auth context [#103]
- CLI security enhancement for private key management

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Updated PEM decoding for key parsing in CLI and tests

Added `encoding/pem` to decode PEM blocks when parsing private and public keys across CLI commands and test computation scenarios, ensuring compatibility with key files. This enhances robustness in key handling by supporting PEM encoded keys. The update also includes registration of a new Keys command in the CLI.

Refactored code is now compliant with common key formats, addressing potential parsing issues.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix auth signature encoding and improve CLI usage example

The authentication system now decodes base64 strings before verifying signatures to align with the expected format. Additionally, the signature generation now encodes the output in base64, ensuring consistency across the auth process.

The CLI help message for the `result` command is enhanced by providing a usage example, making it more user-friendly and informative.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor containsID to handle dynamic fields

Updated the `containsID` function to accept a field name parameter, enabling dynamic field lookup within the reflection logic. This change facilitates the use of the function for various struct fields, improving code reusability and flexibility.

CLI command 'data' now requires an additional argument for the private key file path, outlined in the usage example update, reinforcing command clarity and user guidance.

Resolves issues with hardcoded field lookups and enhances CLI usability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Remove extraneous newline in key generation log output

A redundant newline after the success message in the key generation command was removed to clean up log output formatting. This change ensures a more consistent and professional appearance of the CLI tool's messages.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Implemented auth service in gRPC startup

Added authentication services to the gRPC server initialization to enforce security measures. The gRPC server's New function now includes an `authSvc` parameter, requiring instantiation of the auth service before starting the server. Failure to create the auth service results in a fatal error, halting the process to avoid running without protection. Tests have been updated to include `nil` values for the auth service parameter to maintain their functionality without authentication.

Refactored `grpcserver.New` to accept the new auth service, and updated the main agent startup logic to create and inject the auth service. Added the auth middleware interceptors to the server options, which ensures that each gRPC call will undergo authentication.

This change is a step towards secure communication, and affected components should now consider the authentication requirement.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor config read logic and update agent setup

Improved the configuration reading in `cmd/agent/main.go` to handle larger payloads by reading data in chunks and checking for EOF, ensuring that all config data is captured even if it exceeds the initial buffer size. Enhanced the `test/manual/agent-config/main.go` to require additional command-line arguments, improving the setup process by explicitly requiring paths for data, algorithm, and public key as well as a boolean for attested TLS. Also updated the hashing method to SHA3 for the algorithm and data files, and included the hash and public keys as part of the agent, dataset, and result consumer configurations. These changes will make the agent setup more robust and provide better integrity checks for the involved files.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor run method to agentService

Moved the run function into agentService for better encapsulation and maintainability. This refactoring includes capturing both stdout and stderr during algorithm execution, enabling more informative debugging through enhanced logging. Consequentially, the run method now references members through the service instance, aligning with object-oriented best practices and improving code coherence.

Resolves issue with insufficient execution details when computations fail.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor computation data handling to use filepaths

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor error logging and ensure consistency

Replaced usage of the standard log package with a custom logger for error reporting to standardize error logging throughout the application. Additionally, introduced graceful shutdown by returning from the main function rather than forcing exit when failing to create auth service, aligning the application's error handling strategy.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor auth initialization and key file handling

Improved the readability and maintainability of the authentication service initialization by adding line breaks for logical separation. Also, standardized key filenames in the CLI key generation by introducing constants, enhancing code clarity and reducing the likelihood of file-naming errors.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor auth verification logic for improved security

Removed an extraneous line in the `verifySignature` function that was not necessary for the signature verification process. This change simplifies the code and improves readability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor payload structures to simplify API

Removed the 'provider', 'id', 'consumer' fields from protocol buffers, gRPC services, and related functions across various files to streamline the data model and align with the new authentication system based on cryptographic verification rather than string identifiers. This results in more efficient data handling and a reduction in unnecessary payload data, while enhancing security by making entity validation strictly cryptographic.

The changes affect agent-SDK interactions, CLI tools, and related services, ensuring only the necessary data (algorithm/data bytes, user keys, and hashes) is transmitted and processed. Consequently, the core computation algorithm and dataset handlers now rely on indexes derived from context to associate data with respective manifest entries, thus maintaining the ability to link to specific computation manifests without relying on explicit IDs in the payload. Additionally, refactored authentication methods now enforce role-based security seamlessly through metadata.

This approach enhances privacy by avoiding transmission of potentially sensitive strings over the network and by ensuring that only internal indices, not globally interpretable identifiers, are used to process computations.

Aligned with the broader architectural goal of simplifying and securing the platform's core services, this change paves the way for upcoming revisions to the authentication scheme that will further consolidate role-based security and improve system integrity.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance CLI security with key paths

Removed the section on running computations from the CLI README as it may no longer be necessary or the functionality has been moved elsewhere. Required private key file paths for algorithm, dataset upload, and result retrieval commands to enhance security. This change associates each action with a specific identity, ensuring secure and traceable operations. Additionally, updated the manual test commands to reflect this new requirement.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix ci

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix fmt

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-05-28 14:10:13 +02:00
Sammy Kerata Oina aebe01a873 NOISSUE - Updated README with static binary instructions (#130)
Enhanced the manual algorithm testing documentation by including steps to install additional dependencies and guidance on creating static binaries. This ensures wider compatibility and ease of deployment for python programs tested with this framework.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-05-22 14:40:34 +02:00
Sammy Kerata Oina 4c681569f1 NOISSUE - Add sha3-256 file hashing command to CLI (#129)
Introduced a new command for the CLI to calculate the sha3-256 hash of a given file. The update enhances the utility's cryptography toolkit by allowing users to easily generate and verify file integrity directly through the command-line interface.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-05-20 16:56:42 +02:00
Sammy Kerata Oina c274521faf Refactor computation data handling to use filepaths (#126)
Changed the internal representation of algorithms and datasets within the service from byte slices to file paths, writing received data directly to temp files. This modification allows for handling potentially large data sets without the need to load them entirely into memory, improving the memory efficiency and scalability of the service. Additionally, it aligns the call signature of external algorithms with the new approach, updating documentation and examples accordingly. Updated the linear regression example for consistency with the new data handling process.

Resolves issues with memory bloat when processing large datasets.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-05-13 17:16:39 +02:00
Sammy Kerata Oina 226704cf0d COCOS-122 - Enable streaming RPCs for Algo and Data services (#123)
* Enable streaming RPCs for Algo and Data services

Modified the gRPC service definitions for Algo and Data methods to use stream processing, enabling the handling of larger datasets and algorithms without being limited by memory restrictions. This allows client and server to send chunks of data sequentially rather than requiring the entire payload to be loaded into memory at once.

Updated server implementations to accumulate data from multiple chunks, allowing for more efficient processing and communication when dealing with large files. Client implementations have been adjusted to segment and send data in a streaming fashion.

Removed previously existing synchronous client code as it became redundant with the new streaming approach, streamlining the client's communication patterns with the gRPC backend.

This change allows for better resource management, especially in systems with constraints on memory, improving overall scalability and performance of the data and algorithm processing pipeline.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor algorithm ID check logic

Simplify the algorithm validation logic in the agent service by replacing the previous containment check with direct ID comparison. This change streamlines the error handling for undeclared algorithms and hash mismatches, while also ensuring clear and direct provider validation. The modifications enhance the readability and maintainability of the code without altering functionality.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Updated README to build single-file executable with PyInstaller

Modified the PyInstaller command in the manual testing README to bundle the linear regression script into a single executable file. This simplifies distribution and execution of the script by eliminating the need for multiple dependency files.

Ref: Optimization of deployment process
Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-05-13 13:14:50 +02:00
Danko Miladinovic 479598a663 COCOS-116 -Separated networking setup from agent start up script (#118)
* added network setup script

* removed fixed name enp0s2 from code

* fixed small typo

* changed the capitalization of a letter
2024-05-01 17:03:28 +02:00
Sammy Kerata Oina 8d082567d7 COCOS-101 - Switch to self-contained algorithms as executables (#119)
* Switch to self-contained algorithms as executables

Transitioned from using Python scripts to self-contained binary executables for running algorithms, improving modularity and reducing dependencies. This change removes the reliance on a Python environment, as evident by the removal of Python setup and packages from the build configuration. The service now creates temporary executable files for algorithm runs, handling all permissions and cleanup, enhancing security and maintaining clean execution states. A warning is logged if computation fails, aiding in debugging. Additionally, updated manual tests to reflect these changes in the agent's handling of algorithms.

Refactors:
- Removed Python runtime const since it's no longer needed.
- Updated documentation and test commands to reflect the change from .py to .bin for algorithm files.

Build config:
- Removed Python and pip packages to reduce the build size and complexity.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update agent service.go file with new constants and file permission

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refine singular usage of 'algorithm' across modules

Standardized terminology throughout the project to refer to 'algorithm' in the singular form rather than plural. Streamlined various documentations, string constants, function names, and variable names to bring cohesiveness and eliminate ambiguity when handling algorithms across README files, CLI interfaces, and internal API representations.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix state names and indices in state_string.go

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-29 15:48:17 +02:00
Sammy Kerata Oina 4b5000d107 NOISSUE - Allow termination of manager (#95)
* Implement manager client heartbeat mechanism

Introduced a heartbeat system in the manager service to maintain an active link with client agents. The updates involve sending periodic heartbeat signals controlled via the new environment variable `MANAGER_HEARTBEAT_INTERVAL`, defaulting to 1 second. The protobuf definitions, service interfaces, and server logic have been updated accordingly to handle the incoming signals. This change allows better tracking of active client connections and could be used for features like auto-reconnect or resource cleanup for lost connections in the future.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance gRPC server to handle client termination

Introduce the capability for the gRPC server in the manager module to process termination requests from clients. Server and client proto buffers have been updated to include a Termination message, allowing clients to signal the server to stop processing. Handling for communication and termination flows involving context cancellation have been added across server implementation, manager client, and main execution logic to properly respond to termination signals. This ensures clean shutdowns and resource deallocation when a client requests termination, improving the robustness of the system.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor gRPC server context handling

Removed the redundant storage of context in grpcServer struct and modified the NewServer constructor to exclude the context parameter. Updated the Process method to derive context from the stream directly, ensuring better context scoping and adherence to proper gRPC patterns. This change simplifies the server's context management and ensures contexts are more accurately associated with their corresponding streams, which can improve debugging and request cancellation behavior.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update protoc to v4.25.3 and remove heartbeat system

Upgraded protoc version to 4.25.3, ensuring compatibility with the latest protocol buffer features and potential performance improvements. Additionally, removed the entire heartbeat system, involving both its gRPC server implementation and associated client-side logic. This elimination suggests a shift in the service health-check strategy, potentially to a more modern or efficient model.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Removed heartbeat functionality for manager service

The heartbeat feature and its related configuration have been removed across the application. This includes the removal of the Heartbeat method from the Service interface, the associated interval setting in the config structure, and related environment variable documentation. The change implies an architectural shift towards alternative methods of service availability checking or potentially a move to stateless service design. The cleanup removes unnecessary code, thus simplifying the overall service logic.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Introduce Terminate message and streamline gRPC server logic

- Added a new `Terminate` protobuf message to handle termination requests more explicitly.
- Removed redundant context cancels in gRPC server stream processing to rely on context propagation.
- Updated channel usage to match the altered stream message types; this aligns message handling with the updated protobuf definitions.
- Passed client authentication info to service run function to enhance the security during service execution.
- Utilized stream's context directly in error groups for improved clarity and error handling scope.
- Refactored message descriptors in protobuf to maintain correct ordering after introducing new message types.

These changes enhance the codebase's maintainability, simplify the logic, and align with the updated messaging protocol.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update agent configuration in main.go

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-29 14:08:42 +02:00
Sammy Kerata Oina 64f7e7f7fd NOISSUE - Refactor single algorithm processing (#117)
* Refactor single algorithm processing

Simplified the agent service's algorithm handling logic to process a single algorithm instead of multiple. This change:
- Removed the `Algorithms` type and associated stringer implementation.
- Updated the state machine and service logic to expect a singular algorithm, aligning the agent's internal state transitions with the new model.
- Adjusted the manager service and computations test server to mirror these changes in their respective payload structures, ensuring API and test consistency.
- Altered README files to reflect the simplified interaction model and removed outdated descriptions.
- Reverted the protoc-gen-go version used for generating protobuf files to maintain compatibility with the rest of the codebase.

The single-algorithm approach streamlines the computation running process, reducing complexity and potential error conditions. It directly impacts how external services will construct and send computation requests.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update protoc-gen-go version to v1.33.0

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor variable name in computations.go and grpc.go

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-17 13:34:54 +02:00
Danko Miladinovic 3a14896555 NOISSUE - Attested TLS (#99)
* added initial code for attested TLS

* added client validation and verification

* fixed bugs for attested TLS

* updated README for manual testing

* fixed CI errors

* removed SNP pollicy from agent config

* added attested TLS config param to AgentConfig

* generated manager.pb.go for protoc v25.2

* updated proto-gen-go version on CI

* generated agent.pb.go to match newest proto gen version

* define errors for error handling

* fixed comments
2024-04-16 15:19:19 +02:00
Drasko DRASKOVIC b57bafd310 Fix usage (#114)
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2024-04-16 00:42:47 +02:00
Drasko DRASKOVIC 9e6fb31d90 Fix links and add features (#113)
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2024-04-15 22:26:41 +02:00
Sammy Kerata Oina 2b760ec207 NOISSUE - Streamline client identification (#105)
* Update protoc to v4.25.3 and streamline client identification

Protocol Buffer compiler `protoc` has been upgraded to version 4.25.3. This update unifies the version across multiple generated files to ensure compatibility and take advantage of any bug fixes and performance improvements in the new release.

Additionally, the client identification process has been refined. The redundant `WhoAmI` message and the corresponding checks have been removed from the gRPC server implementation. Clients are now identified via their address from the `Process` stream's context as soon as a connection is established, simplifying the code and potentially reducing handshake time.

This change sets the foundation for a leaner communication protocol between manager and agents, and could contribute to lower latencies in client-server interactions.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix goroutine bug in grpc server

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update PROTOC_VERSION to 25.3

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor server reference in readme

Updated the deployment instructions in the README to point to the correct server location after consolidating test server documentation. Removed obsolete `manager-server` module and associated main entry point to align with the new architecture and streamline workflows.

Resolves issues with outdated links and cluttered repository structure.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-15 22:08:37 +02:00
Sammy Kerata Oina 2e612a2867 COCOS-108 - Enhanced state machine Synchronization (#112)
* Enhanced state machine synchronization

Introduced a wait group to the state machine to manage go-routine synchronization effectively. The Start method now ensures that the state machine's execution is tracked, preventing potential early termination of go-routines during state transitions. The test suite is updated to wait for the go-routine to complete before proceeding, ensuring more reliable test execution.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Remove unnecessary code and fix test case in agent package

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-15 18:55:58 +02:00
Drasko DRASKOVIC f2f57ac413 Fix license (#107)
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2024-04-14 18:05:16 +02:00
Drasko Draskovic d50a6fcfda Fix releases link
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2024-04-14 17:55:29 +02:00
Drasko DRASKOVIC d323fd3b7e Fix docs link (#106)
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2024-04-14 17:49:10 +02:00
Filip Bugarski 35c2a9e92a Update cocos documentation (#89)
* Update cocos documentation

Signed-off-by: fbugarski <filipbugarski@gmail.com>

* inserted go run commands into separate lines

* updated syntax higlighting to bash

---------

Signed-off-by: fbugarski <filipbugarski@gmail.com>
Co-authored-by: Danko Miladinovic <danko@etf.rs>
2024-04-13 18:33:10 +02:00
Sammy Kerata Oina 5c406743e0 NOISSUE - Add attesatation validation/verification options (#87)
* Standardize attestation report data size to 64 bytes

Redefined the report data for attestation requests from a variable byte slice to a fixed 64-byte array across various critical components. This change enforces a standardized data length, simplifying validation logic and ensuring consistency when handling the attestation data for both the server and client end-points, logging functionalities, and the CLI tool. Updated attestation data handling in the SDK to accommodate the new fixed-length constraint.

By strictly adhering to the 64-byte requirement, the modifications promote robust input checking and prevent potential issues related to dynamic data length processing.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor attestation validation logic

Streamlined attestation validation in the CLI with a new configuration parsing approach. Introduced a retry mechanism with timeout and backoff for HTTPS getters that facilitate remote trust validation. Clarified expected data lengths in proto comments across attestation and manager services, ensuring consistency and correctness of the cryptographic data fields.

This modification enhances maintainability by replacing a verbose struct with a configurable object model, thus simplifying parameter handling. Robustness is improved through failure-retrieval strategies during remote validation, and additional in-code documentation specifies cryptographic constraints for critical data elements.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add new attestation validation flags

Enhanced attestation validation in the CLI with additional flag support for FAMILY_ID, IMAGE_ID, REPORT_ID, REPORT_ID_MA, MEASUREMENT, CHIP_ID, and TCB-related fields. This update enables fine-grained control over attestation report criteria, ensuring stricter validation conforming to protocol requirements. Adjusted the default value for MinimumGuestSvn to align with the new policy specifications.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance attestation policy configuration

Introduced several new configuration flags to the CLI's attestation policy, including options for minimum AMD-SP firmware builds, revocation list checking, and network restrictions. Expanded trust criteria parameters with the support for trusted author and identity key hashes. The default VMPL setting is now explicitly initialized. These changes improve security controls and offer more detailed attestation verification settings.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Simplify attestation validation process

Refactored the attestation CLI command to improve usability by removing the need for explicitly provided report data as an argument. Default values for configuration parameters have been centralized into constants for maintainability. Additionally, integrated data parsing functions to streamline the validation checks with robust error handling for cases such as invalid hex strings and file I/O issues.

This enhancement reduces user error and the complexity of entering attestation data by parsing relevant information from within the environment, all while maintaining the same security standards.

The process of marking flags as required for CLI commands has been corrected to prevent runtime errors and improve command reliability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refined attestation validation logic

Enhanced attestation command handling by removing the hardcoded report data length and replacing it with the standard SHA-512 hash size, ensuring dynamic compatibility with hash lengths. Introduced size constants to improve code readability and enforce explicit length checks on attestation input data, raising errors when requirements are unmet. This preventative measure aims to avert potential runtime errors stemming from unexpected data sizes. The change also simplifies the minimum guest SVN description for clarity.

References to removed unnecessary whitespace maintain code cleanliness. These adjustments culminate in a more robust and maintainable attestation process, in accordance with the best practices for secure handling of cryptographic data and compliance with AMD's ABI format specifications.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhanced attestation documentation and command syntax

Updated CLI documentation to include detailed descriptions of the attestation retrieval and validation process, along with a comprehensive list of new flags for the validation command, providing users with extensive control over the attestation verification process. Updated command syntax with flags for specifying report data, improving command clarity and consistency. These changes ensure better user guidance and offer a more robust attestation handling experience.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enforce report data size validation uniformly

Refactored report data size validation to ensure it is performed consistently across the system. Removed the hard-coded length check from attestation request validation in favor of a centralized verification based on the sha512 digest size. Also, eliminated fallback mechanism to the SHA-512 hash of report data and made the length requirement explicit, leading to immediate failure if not met. This harmonizes the error handling logic, reinforces data integrity checks, and simplifies debugging by removing ambiguous length correction behavior. Adjusted related validation functions to use a common utility, enhancing maintainability.

Fix spelling errors in log messages for attestation validation commands to improve clarity in output.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Initialize attestation config with defaults

Enhanced the attestation command initialization by setting default values for the `Config` struct to prevent nil pointer exceptions and potentially streamline configuration handling. Moreover, corrected a flag declaration by removing shorthand.P support for 'permit_provisional_software', aligning it with other flags for consistency. This change should improve stability and user experience while configuring attestation policy.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* nil check prior

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix typos

Signed-off-by: SammyOina <sammyoina@gmail.com>

* use file for attestation report

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refine attestation validation with default policy values

Adjusted default values and constants related to the attestation configuration for enhanced clarity and consistency. Changed the default minimum guest SVN value to align with new policy requirements and introduced a default guest policy constant. Additionally, standardized byte array placeholders for attestation validation fields to prevent potential issues with uninitialized bytes and ensure robust input validation checks. These changes promote more intuitive and secure attestation validation procedures.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refine attestation defaults and flag requirements

Updated the default guest policy to use hexadecimal notation for clarity and added a default minimum version for the firmware API version flag. Made 'chip_id', 'measurement', 'report_id', and 'report_id_ma' flags mandatory to ensure critical attestation data is provided by the user, enhancing validation robustness. Added a check for 'measurement' field length for consistency in input validation.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Optimize attestation file format

Standardized attestation file format across the codebase by moving the 'attestationFilePath' constant from the function scope to a higher scope within the package. Switched the file extension from .txt to .bin to reflect binary data storage, improving consistency and potential performance benefits from binary file handling.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Optimize default values for attestation validation

Removed unnecessary empty initializers and replaced hardcoded empty values with nil for optional fields in attestation validation flags. Introduced a constant defaultReportIdMa for the REPORT_ID_MA field to provide a meaningful default instead of an empty one. Streamlined the command setup by removing the requirement flags for fields that are now optional and unchecked if unset, reducing burdens on the user for providing unnecessary input.

The change simplifies the command's usage and eliminates the need for users to specify values for certain fields that can be optional, improving the command's flexibility and user experience. Additionally, setting a specific default for REPORT_ID_MA ensures clarity in cases where this is not provided by the user.

This update enhances the maintainability and usability of the attestation validation functionality.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor hardcoded attestation report size

Introduced a constant `size64` to replace the previously hardcoded array size of `64` used across various Attestation methods. This change enhances code maintainability and readability by centralizing the size definition, making it easier to update in the future if necessary.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Updated protocol buffer dependencies and cleaned CLI flags

Protobuf compiler versions were updated to v4.25.2 across various RPC service definitions for agent and manager packages, ensuring compatibility with the latest features and fixes. In the CLI attestation code, unnecessary flags (`DisallowNetwork` and `PermitProvisionalSoftware`) were removed to streamline user options, reflecting a more secure and user-focused configuration interface. Added an input validation check in `attestation.go` for CA bundle presence when a product name is set, enhancing the reliability of attestation validation. Also rectified a bug in `grpc.go` by fixing an incorrect variable assignment for reading the key file content, thereby preventing potential TLS-related errors.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Updated protoc version and adjusted proto file paths

Bumped the protocol compiler version to 25.2 to align with our dependency updates and ensure compatibility. Additionally, revised the proto file paths under the continuous integration setup, moving from 'manager/' to 'pkg/manager/' to reflect the recent directory restructuring. The changes ensure that proto file checks and comparisons are conducted in the correct file locations, preventing potential build and sync issues in future developments.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance hash size consistency in gRPC attestation

Aligned the hard-coded byte array size for `ReportData` with `sha512.Size` constant to ensure consistency and maintainability in gRPC attestation requests and responses. This change mitigates the risk of future errors if the SHA512 hash size standard is altered.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Optimize agent-config listener loop

Removed the goroutine wrapping the listener loop in 'agent-config' manual test to streamline and simplify the connection handling logic. This change ensures the loop runs on the main goroutine, improving the readability and maintainability of the code.

Refactoring assists in avoiding potential race conditions and makes the server's flow easier to understand for future debugging and development.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor report data size checks to use constant

Centralize the report data size definition by replacing various size checks and array declarations with a constant `ReportDataSize`. This streamlines code maintenance and ensures consistency across all instances where report data size is validated or used. The changes remove direct references to the hash function output size, decoupling the report data size from the hash function's characteristics and allowing easier updates if the data size requirement changes.

These modifications bolster the codebase's flexibility for potential adjustments in security protocols or data handling specifications.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor config JSON handling in attestation CLI

Introduced an `exampleJSONConfig` constant containing a sample configuration to improve code readability and maintainability. Previously, the example JSON configuration for the attestation CLI was an inline string, making the code cluttered and less maintainable. With this change, the example configuration is now stored as a constant, resulting in cleaner command flag setup and enhanced clarity. This constant is used in the command flag description to guide users when providing their custom configuration. Additionally, refactored the command flags related to the configuration by aligning and sorting them for better code organization.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-09 10:33:26 +02:00
Smith Jilks b6f116474d Cocos-91 - Update Makefile and cocos documentation (#93)
* Update Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update cocos documentation

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update cocos documentation

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update cocos documentation

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Remove install rule from Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update agent README.md

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update agent README.md

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Update README.md files

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Remove INSTALL_DIR from Makefile

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

* Fix comments in README.md files

Signed-off-by: Jilks Smith <smithjilks@gmail.com>

---------

Signed-off-by: Jilks Smith <smithjilks@gmail.com>
2024-04-08 15:59:38 +02:00
Sammy Kerata Oina 3e3e02f132 Enhance security with SHA-3 and command-line args (#97)
Updated the hashing algorithm from SHA-256 to SHA-3 for improved security and reliability in the computation service. Modified the service to accept dataset and algorithm paths as command-line arguments, enhancing flexibility and user control. Increased robustness of the service's error handling by incorporating file read validations and updated usage instructions in both the README and the program's startup error messages.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-04-06 23:45:13 +02:00
Danko Miladinovic 4b38ac09dd NOISSUE - Added SEV-SNP support (#98)
* added SEV-SNP support

* updated documentation for SEV-SNP

* changed variable names to adhere to the GO naming convention

* changed the variable name from V-sock to Vsock
2024-03-22 10:55:09 +01:00
Danko Miladinovic 1a322f8ecb NOISSUE - Fix network interface problem for agent (#96)
* Added bus/addr for pcie net

* Remove SCSI parameters

* Remove bus parameter

* Removed blank lines
2024-03-15 16:37:41 +01:00
Sammy Kerata Oina 8c2caec395 NOISSUE - Use standard GitHub templates for issues and PRs (#92)
Implemented new GitHub templates to streamline the contribution process. Bug report and feature request templates guide users in creating detailed and actionable issues, while a universal pull request template ensures systematically structured submissions. These additions facilitate more efficient reviews, and prioritize issues effectively. Additionally, configured the repository to disallow blank issues, encouraging use of the templates for better project management.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-03-13 13:06:51 +01:00
Sammy Kerata Oina de60358c02 NOISSUE - Enforce 32-byte length check for hashes (#86)
* Enforce 32-byte length check for hashes

Introduced a validation step in the manager service to ensure both algorithm and dataset hashes are of byte length 32 before provisioning VMs, preventing runtime errors due to invalid hash lengths. The test manager-server now generates a valid 32-byte hash using SHA256 for mock data to align with the new validation requirement.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* use equal comparison

Signed-off-by: SammyOina <sammyoina@gmail.com>

* use a constant

Signed-off-by: SammyOina <sammyoina@gmail.com>

* use constants

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-24 22:22:25 +01:00
Sammy Kerata Oina 997fb3bf48 COCOS-83 - Add hash verification for datasets and algorithms (#84)
* Add hash verification for datasets and algorithms

Enhanced data integrity checking by incorporating hash fields in Dataset and Algorithm structures, and modified the corresponding service logic to validate these hashes during processing. The update includes SHA-3 for hash computation, replacing the former SHA-256 usage, and ensures the provided data matches the expected hash from the manifest to prevent processing malformed or corrupted data.

- Introduce `Hash` field to both Dataset and Algorithm structs to store the expected hash value.
- Implement SHA-3 hashing within service methods that process the data, ensuring consistency with newly added `Hash` fields.
- Add error handling for hash mismatches, preventing further processing and alerting to potential data integrity issues.
- Update Protocol Buffers serialization to accommodate the new hash fields for gRPC communication.
- Modify manager service's Run method to pass the hash information when creating agent configurations.

Go module dependencies were updated to include the new SHA-3 package and upgrade Go version to 1.21.6 for improved stability and compatibility.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Remove identifiers from protobuf and related code

The protobuf definitions and related service handling code have been revised to drop specific identifier fields (`AlgorithmID` and `DatasetID`) to simplify API responses and internal function signatures. These removals streamline the overall data flow between components, reduce unnecessary data transmission, and lead to an aligned server-client expectation where identifiers are no longer a part of the response payload. Consequently, these changes simplify the logic within various functions and client commands, reinforcing encapsulation by ensuring that internal identifiers do not need to be managed or exposed unnecessarily.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-19 15:31:35 +01:00
Sammy Kerata Oina 722b463b6a NOISSUE - Use a single listener for logs and events (#82)
* add handler

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor gRPC and Protobuf integration for manager service

- Shifted Protobuf message definitions to a separate package `pkg/manager`.
- Updated references throughout the codebase to import and use the new package for gRPC service definitions.
- Enhanced AgentLog message with additional fields `level` and `timestamp`.
- Removed direct dependencies on old Protobuf-generated types in favor of the new package.
- Deleted obsolete Protobuf-generated files as they are now superseded by the new `pkg/manager`.
- Streamlined event publishing and gRPC handling in the manager service to use the updated Protobuf messages.

This refactoring improves modularity by centralizing Protobuf message definitions and decouples internal representation from the gRPC interface, aligning with best practices for microservice architecture. Additionally, the enriched logging structure paves the way for more detailed and fine-grained log analysis.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor vsock event/log handling and config

Streamlined event and log services in the manager by moving vsock listening functions out of `managerService` initialization and into dedicated `RetrieveAgentEventsLogs` methods. This change decouples the manager service creation from the actual start of log listening, adding clarity and flexibility in service management. Also moved logging middleware invocation outside of network handling loops to avoid unnecessary overhead.

Additionally, the agent's vsock port configuration is now dynamically passed to the `New` function in the `events` package instead of relying on a hardcoded constant, allowing for greater configurability and testability.

Finally, updated message structures for event and log sending to conform with the `ClientStreamMessage` definitions. These modifications should improve parsing and handling consistency and prepare our system for future enhancements related to inter-process communication.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix linting errors

Signed-off-by: SammyOina <sammyoina@gmail.com>

* correct path to generated files

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix comments

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove uneccessary comments

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-15 17:02:05 +01:00
Sammy Kerata Oina 938dd6cb78 NOISSUE - mTLS support across services (#71)
* Implemented mTLS support across services

Extended gRPC configuration to support mutual TLS (mTLS) in agent and manager components for enhanced security. This includes the loading of Certificate Authority (CA) certificates, server, and client certificates, and keys. Updated README documentation to reflect the new environment variables required for mTLS configuration. Additionally, streamlined secure gRPC client connection setup and logging messages to indicate whether a service is running with TLS, mTLS, or without TLS.

The change ensures secure communication between services by verifying both client and server identities, thus addressing potential security concerns in network-level interactions.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance agent cert handling and update copyright

- Implement function to create certificate files for the agent configuration dynamically, ensuring file paths are updated to reflect newly created files. This improves the agent's setup process by automating the certificate handling.
- Update copyright clause to reflect the new owning entity, Ultraviolet, affirming correct attribution and compliance with legal requirements.
- Refactor gRPC client connection code to remove redundant package alias, streamlining the codebase and improving readability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor cert loading with fallbacks

Removed redundant certificate file creation logic in the agent module and introduced a more robust loading mechanism in the gRPC server module to support direct byte content aside from file paths. This change simplifies the initial setup process for the agent by removing the need to create certificate files preemptively, thereby streamlining deployment in environments with varying filesystem access. It supports using certificate contents directly, enhancing compatibility with in-memory configurations or environments where file storage may not be ideal.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-08 10:07:51 +01:00
Sammy Kerata Oina e86860b9ea NOISSUE - Enhanced gRPC server concurrency with errgroup (#81)
* Enhanced gRPC server concurrency with errgroup

Refactored the gRPC server's `Process` method to handle incoming messages and service execution concurrently using an `errgroup.Group`. Added a context to the `grpcServer` struct and propagated it to support graceful shutdowns and better request handling. Changed the `Service` interface's `Run` method to use a channel for transmitting `ComputationRunReq`, facilitating non-blocking operation and streamlining the request-response cycle.

Asynchronous handling of stream requests with `errgroup` allows simultaneous processing and error management, improving the server's responsiveness and fault tolerance. This update paves the way for more robust service implementations that require concurrent operations.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove trailing space

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-08 10:05:41 +01:00
Sammy Kerata Oina e2fb7ea88d NOISSUE - Add test server (#80)
* add test server

Signed-off-by: SammyOina <sammyoina@gmail.com>

* clean up and update docs

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update docs

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-07 16:16:27 +01:00
Sammy Kerata Oina 6d0100c096 COCOS-73 - Remove server (#76)
* Refactor GRPC manager service and client

The manager service and client have been restructured for stream communication, facilitating real-time agent events, logs, and run responses. The `Run` RPC is replaced by the `Process` stream RPC, enabling bidirectional streaming between clients and the manager service. This allows continuous interchange of different message types including `WhoAmIRequest`, `AgentLog`, `AgentEvent`, and `RunResponse`.

Several message types have been adjusted and new fields introduced, like `AgentPort` in `RunResponse` and various agent-config attributes including CA files and instance IDs, to support TLS client authentication and distinguish between agent instances.

We've also incorporated `google.protobuf.Timestamp` in `AgentEvent` for precise event logging. The client code reflects these modifications with updated method calls and stream handling logic for ongoing communication. Moreover, the updates necessitate corresponding changes throughout service, grpc, and sdk layers to interoperate with the new streaming approach.

The transition to streaming paves the way for a more interactive, flexible communication system that can accommodate future expansion and real-time monitoring features.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* add computation id to run response

Signed-off-by: SammyOina <sammyoina@gmail.com>

* rename request

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-07 15:32:35 +01:00
Sammy Kerata Oina 8975e28437 COCOS-77 - Stream agent logs and events over gRPC (#78)
* Refactor GRPC manager service and client

The manager service and client have been restructured for stream communication, facilitating real-time agent events, logs, and run responses. The `Run` RPC is replaced by the `Process` stream RPC, enabling bidirectional streaming between clients and the manager service. This allows continuous interchange of different message types including `WhoAmIRequest`, `AgentLog`, `AgentEvent`, and `RunResponse`.

Several message types have been adjusted and new fields introduced, like `AgentPort` in `RunResponse` and various agent-config attributes including CA files and instance IDs, to support TLS client authentication and distinguish between agent instances.

We've also incorporated `google.protobuf.Timestamp` in `AgentEvent` for precise event logging. The client code reflects these modifications with updated method calls and stream handling logic for ongoing communication. Moreover, the updates necessitate corresponding changes throughout service, grpc, and sdk layers to interoperate with the new streaming approach.

The transition to streaming paves the way for a more interactive, flexible communication system that can accommodate future expansion and real-time monitoring features.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update GitHub Actions to Latest Versions

Upgraded GitHub Actions 'checkout' to version 4 and 'setup-go' to version 5 across various workflow files to leverage the latest features and improvements for better performance and reliability. This also ensures compatibility with Go version 1.21.x which is specified in the workflows.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor event handling and logging

Reworked event and log processing to use channels instead of direct HTTP calls. Removed obsolete events package and consolidated event structures, leading to cleaner and more maintainable code. Updated agent events to use channels, enhanced error handling in log forwarding, and simplified manager `New` function signature to accept an event channel directly.

- Removed `events` and `agentevents` packages to reduce complexity.
- Replaced direct event server communication with internal channel usage.
- Introduced `AgentEvent` struct in events.go for standardized event objects.
- Adapted `managerService` to dispatch events and logs through channels.
- Streamlined manager construction by removing the now-unnecessary event service and host IP parameters.

This change results in a more robust and easier to extend event and log management system within the agent-manager interaction.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix ci

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove unused code

Signed-off-by: SammyOina <sammyoina@gmail.com>

* add comments

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-07 12:08:39 +01:00
Sammy Kerata Oina 3e2be03047 NOISSUE - Attest validation and verification (#74)
* Expand CLI with Manager Service and Enhanced Attestation Commands

The CLI has been updated to interact not only with the Agent service but also with the Manager service, reflecting an expanded scope of operations. As part of this update, 'manager' subcommands have now been incorporated, ensuring users can seamlessly initiate computations through the manager service using the CLI.

Furthermore, attestation functionality has greatly improved. In addition to retrieving attestations from the agent, users can now validate and verify them directly from the CLI, bolstering security measures. New subcommands under 'attestation' provide clear and specific actions for attestation management.

Additionally, the command structure has been refactored to categorize commands under 'agent' or 'manager' prefixes, promoting better command organization and a more intuitive user experience.

These enhancements are part of ongoing efforts to provide robust and secure tools for service interaction and computational task management in distributed networks.

Relevant documentation adjustments and example usage have been updated in README files to align with these functional improvements.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* capitalize comments

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-07 12:02:41 +01:00
Sammy Kerata Oina 046b549079 Remove HTTP server support and streamline config (#75)
The HTTP server-related code, documentation, and configurations have been removed as part of a shift towards prioritizing gRPC for service communication. This update includes deletions of HTTP host and port configs across various components, the manager HTTP API alongside its Swagger definition, and the removal of related scaffolding and utility code. This change simplifies the overall architecture and eliminates redundant HTTP support, focusing on optimizing gRPC performance and security features.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-07 11:03:46 +01:00
Sammy Kerata Oina 6f42135ec0 NOISSUE - Fetch attestation report (#66)
* Improve attestation and update deps

Enhanced the attestation process to require additional report data and a variable privacy level, catering to varying security requirements. The `Run` method's context parameter was removed to simplify invocation following API alignment. Dependencies got updated, notably adding `github.com/google/go-sev-guest` for attestation, and some were indirectly bumped. These updates ensure the agent service leverages the latest available features and improvements for security and stability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Extend AttestationRequest with data fields

Enhanced the AttestationRequest message structure by introducing 'report_data' and 'vpml' fields to encapsulate additional attestation context. Adjusted service and middleware layers to incorporate these new parameters, enabling richer attestation workflows. The GRPC endpoint, logging, and metrics layers have been updated to handle the additional data, catering to more secure and nuanced attestation use cases.

Refactors Attestation command in CLI to accept report data argument and include an optional 'vpml' flag for specifying virtual machine privilege level. This change empowers users to request attestation with specific context directly from the CLI.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor attestation to remove vpml dependency

Removed the `vpml` field from the AttestationRequest protobuf, and all associated handling code, streamlining the attestation process. Code adjustments were made in gRPC client, server, and service layers to ensure proper request construction and validation that focuses only on report data. This change simplifies the attestation API and its usage, reducing error risk and improving maintainability. It reflects a decision to decouple the protocol from specific vendor measurement levels, potentially due to a shift towards a more generalized attestation procedure.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance CLI usability and streamline SDK attestation

- Added a usage example to the CLI attestation command for better user guidance.
- Simplified the SDK attestation function by removing an unused parameter and directly populating the request structure with report data to align function parameters with actual usage requirements.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-02-06 17:55:42 +01:00
Sammy Kerata Oina e5c8243f5f COCOS-62 - Add EOS images to release (#72)
* Add build and release GitHub workflow for tagged commits

Introduced a new GitHub Actions workflow to automate building and releasing for tagged commits. The workflow checks out the required repositories, builds the project using Buildroot configurations, and creates a release with the resultant kernel and rootfs artifacts. This streamlines the release process, ensuring consistent and reproducible builds.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update GitHub Actions to Latest Versions

Upgraded GitHub Actions in the workflow to latest or more recent versions to leverage improvements and maintain compatibility. This includes using version 4 for checkout steps and switching to 'latest' for release creation and asset upload actions, ensuring the use of up-to-date features and potential bug fixes.

Ref: Internal upgrade protocol for CI/CD consistency.
Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix workflow

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Optimized checkout step in CI workflow

Ensured the checked-out repository's reference matches the current workflow's triggering reference. This alignment can result in more predictable builds, especially when the workflow is triggered by pushes to branches other than the default or by tagged commits.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* test repo

Signed-off-by: SammyOina <sammyoina@gmail.com>

* test modified module

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Update and upgrade Ubuntu in GitHub actions workflow

Ensure the GitHub actions runner has the latest Ubuntu updates before checking out the repository. This change significantly reduces the chances of encountering bugs or security vulnerabilities stemming from outdated packages, creating a more reliable and secure CI/CD environment.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* restore repo

Signed-off-by: SammyOina <sammyoina@gmail.com>

* install golang

Signed-off-by: SammyOina <sammyoina@gmail.com>

* create tags only for release

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-31 00:08:01 +01:00
Sammy Kerata Oina 8be6129889 NOISSUE - Fix bug on receiving logs and events, update deprecated methods (#68)
* Ensure graceful shutdown and improve connection handling

Refined the network connection handling in both agent events and logs to continuously process incoming data in a loop, enabling the services to handle more than a single message per connection. Additionally, instituted a deferred close operation for the event service to guarantee resources are cleanly released upon the application's termination.

Resolves potential resource leakage and enhances log processing efficiency.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Improve robustness in event and log handling

Altered handling in agent event and log services to continue processing incoming data rather than returning early upon encountering errors. This ensures that a single erroneous data point does not prematurely halt the processing loop, improving the robustness and reliability of the services. Potential errors are now reported and logged, yet the system remains operational to handle subsequent data.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Upgrade gRPC telemetry to use StatsHandler

Switched from using gRPC's UnaryInterceptor for telemetry to the more comprehensive StatsHandler provided by otelgrpc. This enhances telemetry collection by allowing the capture of a wider range of RPC stats, leading to improved monitoring and observability of the gRPC server.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* use constants

Signed-off-by: sammy <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
Signed-off-by: sammy <sammyoina@gmail.com>
2024-01-25 15:30:27 +01:00
Sammy Kerata Oina f2567830b3 NOISSUE - Remove docker (#67)
* Remove Docker-related build functionality

Removed the Docker build scripts, Dockerfile, and docker-compose definitions from the Makefile and relevant directories. Updated documentation by stripping out references to building and running Docker containers. This change likely reflects a shift towards a different deployment strategy or a move away from Docker as a dependency for builds and runtime. This simplification could lead to less complexity in the build process and reduce maintenance overhead associated with Docker configurations.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Removed Docker image build and publish workflow

The workflow for building and publishing a Docker image on pushes to the main branch has been removed. This step likely reflects a change in deployment strategy or a move to a different CI/CD pipeline. Continuous deployment might now be managed by another service or process, adhering to updated operational requirements or infrastructure changes.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-25 15:29:16 +01:00
Sammy Kerata Oina 55c19ca6b5 NOISSUE - Refactor SDK and CLI (#69)
* Refactor CLI commands to use CLI struct

Introduced a CLI struct to encapsulate dependencies such as agentSDK and managerSDK previously passed to command constructors. This shift towards a more object-oriented approach simplifies command creation and enhances code readability. CLI-related commands are now methods on the CLI struct, accessing shared services directly, which streamlines the setup of CLI command hierarchies and reduces the need to pass dependencies around.

Removed the OpenAPI specification file as it was likely deemed obsolete or no longer necessary to be included with the CLI binary, suggesting a possible shift in how the API is managed or documented.

Lastly, the main application now sets up two distinct command groups for 'agent' and 'manager', with respective commands organized under them, making the CLI tool's structure more intuitive for users.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix ci

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-25 15:07:33 +01:00
Sammy Kerata Oina 0c77167be1 COCOS-56 - Pass agent computation events to manager (#63)
* Simplify event handling and config

Streamlined event service interface by consolidating `SendEvent` and introducing `SendRaw`. Removed `notification_server_url` and `instance_id` parameters from several event publication calls to leverage centralized event construction. This change not only cleans up redundancy in event-related code but also simplifies the configuration data flow across the system, making it easier to manage and less error-prone. Uniform event generation now improves consistency and maintainability.

Refactored configuration management in the agent and manager services. Removed notifications URL from the agent configuration, relying on a simplification that assumes a single source of events. Updated Manager Port to VsockConfigPort for clarity and consistency across vsock communication.

These modifications should facilitate easier integration and extension of event and configuration systems in the future.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor error handling in agent event forwarding

Introduced context and error channel handling to the agent event forwarding process. The logger now warns on errors during forward operations asynchronously, allowing for non-blocking error reporting. Additionally, reliance on the global logger was removed in favor of passing error information via channels, improving modularity and error flow control.

Resolves issue with silent forwarding failures by providing a means to alert system operators without halting the service. This enhancement makes the error reporting more robust and reactive while maintaining service continuity.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove unused field

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update package name

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-22 16:36:34 +01:00
Sammy Kerata Oina 34c3bbdbd8 COCOS-55 - Pass agent logs to manager via vsock (#64)
* Simplify event handling and config

Streamlined event service interface by consolidating `SendEvent` and introducing `SendRaw`. Removed `notification_server_url` and `instance_id` parameters from several event publication calls to leverage centralized event construction. This change not only cleans up redundancy in event-related code but also simplifies the configuration data flow across the system, making it easier to manage and less error-prone. Uniform event generation now improves consistency and maintainability.

Refactored configuration management in the agent and manager services. Removed notifications URL from the agent configuration, relying on a simplification that assumes a single source of events. Updated Manager Port to VsockConfigPort for clarity and consistency across vsock communication.

These modifications should facilitate easier integration and extension of event and configuration systems in the future.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor error handling in agent event forwarding

Introduced context and error channel handling to the agent event forwarding process. The logger now warns on errors during forward operations asynchronously, allowing for non-blocking error reporting. Additionally, reliance on the global logger was removed in favor of passing error information via channels, improving modularity and error flow control.

Resolves issue with silent forwarding failures by providing a means to alert system operators without halting the service. This enhancement makes the error reporting more robust and reactive while maintaining service continuity.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove unused field

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enhance agent logging via vsock connection

Redirected agent logging to use a vsock connection instead of standard output, improving the process isolation and enabling centralized log management. The change involved dialing to the specified vsock log port and initializing the logger with the vsock connection rather than stdout.

Additionally, the manager service now maintains a map of agent vsock cids to computation IDs, providing better tracking of computation resources. A routine to retrieve logs from agents was also initiated during the service setup to facilitate log collection.

Consequential to these changes is the removal of a redundant os package import in the agent's main.go, further cleaning up the dependencies.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fail gracefully

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Updated backoff strategy and VM configurations

- Added `github.com/cenkalti/backoff` to direct dependencies for robust retry logic in agent configuration sending.
- Modified the vsock logs port to align with the updated port range standards.
- Enclosed kernel console arguments in quotes to ensure proper parsing in QEMU configurations.
- Implemented exponential backoff when sending agent configurations to handle transient failures.

Refactors:
- Streamlined creation of `AgentConfig` within the computation setup to avoid unnecessary initializations when `c.AgentConfig` is not nil.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor command execution and improve argument construction

Consolidated the error handling in the command execution function for better readability. In the QEMU configuration, the argument assembly process is enhanced for clarity and correctness; the VNC parameter is now separate, and string quoting is handled properly for kernel parameters. These changes result in more maintainable code and prevent potential formatting issues during QEMU argument parsing.

Resolves issues with argument construction in QEMU config module.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refine default config handling and unpacking

Improved the agent configuration by dynamically setting default values for the log level and port if they are not specified in the incoming configuration. Also streamlined configuration unpacking in the endpoint and service layers, reducing redundancy and ensuring all required fields are correctly copied over to the Manager's configuration structure. This change ensures better fault tolerance and more maintainable code by handling edge cases where configuration values might be missing.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* rename dir

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Ensure runRes.Empty() reflects non-empty state

Changed the always-true return value of the `runRes.Empty()` method to `false` to accurately indicate the presence of a response body. This adjustment ensures downstream handling of API responses aligns with actual content state.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Replace mglog with slog across codebase

Updated various components to replace the `mglog` logger implementation with the `slog` logger. This change affects logging initialization and calls throughout the codebase including the agent, manager, and internal server components. Transitioning to `slog` is part of a broader shift to standardize the logging mechanism to improve maintainability and consistency.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-22 14:13:47 +01:00
Sammy Kerata Oina 55afe4c038 COCOS-49 - Pass agent configuration and computation via vsock (#57)
* Optimize QEMU launch and add V-sock support

Refactored QEMU argument construction and launching logic by removing the dependency on 'agent.Computation'. This simplification makes the VM creation process more streamlined. Additionally, introduced V-sock capabilities in the QEMU configuration to facilitate improved guest-host communication. Updated the README to include kernel module setup instructions for the new V-sock feature.

The V-sock implementation enables VMs to use a consistent communication channel that is not affected by network configuration changes, enhancing reliability and potential interoperability with host services. It's important to ensure that the necessary kernel modules are loaded as part of the setup process, as documented.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add vsock-based communication to manager

Introduced virtual socket (vsock) communication abilities in the manager package by implementing a new socket service. This includes establishing a vsock listener and stub methods for sending computation results and cleaning up resources. The addition provides the groundwork for interprocess communication between guest and host in virtualized environments.

- Integrated the `mdlayher/vsock` library for handling virtual socket operations.
- Created a new `sockService` struct to encapsulate vsock listener handling.
- Implemented `NewVsock` constructor to initialize the listener with domain value `3`.
- Added placeholder methods for future computation sending and service closing logic.

This enhancement targets scenarios where efficient VM-to-host communication is required.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove env

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor agent config and use vsock

Introduce `AgentConfig` struct to group agent-related configurations, and update `Computation` struct to include the new `AgentConfig` field. Replace command-line computation extraction with vsock-based config retrieval for robustness and decoupling. The agent configuration is now read from a vsock connection during runtime, allowing for more dynamic and flexible deployments. Adjusted the main agent application logic to support these configuration changes, and corresponding changes have been made in the manager to facilitate vsock communication.

This approach aligns with modern practices for microservices by streamlining configuration management and reducing reliance on static command-line parameters. Moreover, it enhances the scalability of the agent service by allowing configuration to be managed externally.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor agent config and remove deprecated code

Consolidated agent configuration management into a single `AgentConfig` message and pruned deprecated Protobuf `ComputationReq`, `DatasetReq`, and `AlgorithmReq` messages. Adapted corresponding manager service logic to the new configuration structure. These modifications align with updated manager API schema, facilitate clearer configuration handling, and improve maintainability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* send configuration

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Switch agent to listen mode for manager connections

Previously, the agent established a connection to the manager using a direct dial. This change shifts the setup to where the agent listens on a specified port and accepts incoming connections. It ensures that the agent properly handles incoming requests by initiating a listening socket and waiting for the manager to connect, enhancing the system's flexibility in connection management. This adjustment also includes graceful closure of the listening socket.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-19 17:24:51 +01:00
Danko Miladinovic 2bb3b5650a Add support for vSockets in guest kernel (#61)
* Add python pip to buildroot

* Use DHCP for agent

* Use systemd for naming network interfaces

* Removed eth0 dhcp setting

* Add ETH_IFACE variable

* Added explanation for ETH_IFACE name

* Further explain ETH_IFACE value

* Add guest kernel support for VSOCKETS
2024-01-18 14:41:05 +01:00
Sammy Kerata Oina b4a49fb56c COCOS-58 - Remove refernece to network.link file (#59)
* Optimize QEMU launch and add V-sock support

Refactored QEMU argument construction and launching logic by removing the dependency on 'agent.Computation'. This simplification makes the VM creation process more streamlined. Additionally, introduced V-sock capabilities in the QEMU configuration to facilitate improved guest-host communication. Updated the README to include kernel module setup instructions for the new V-sock feature.

The V-sock implementation enables VMs to use a consistent communication channel that is not affected by network configuration changes, enhancing reliability and potential interoperability with host services. It's important to ensure that the necessary kernel modules are loaded as part of the setup process, as documented.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add vsock-based communication to manager

Introduced virtual socket (vsock) communication abilities in the manager package by implementing a new socket service. This includes establishing a vsock listener and stub methods for sending computation results and cleaning up resources. The addition provides the groundwork for interprocess communication between guest and host in virtualized environments.

- Integrated the `mdlayher/vsock` library for handling virtual socket operations.
- Created a new `sockService` struct to encapsulate vsock listener handling.
- Implemented `NewVsock` constructor to initialize the listener with domain value `3`.
- Added placeholder methods for future computation sending and service closing logic.

This enhancement targets scenarios where efficient VM-to-host communication is required.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Revert "Optimize QEMU launch and add V-sock support"

This reverts commit 44557c39d9.

* Revert "Add vsock-based communication to manager"

This reverts commit d1d3c01f3b.

* Remove unnecessary network link file installation

Network configuration is being handled elsewhere, making the installation of the `00-network.link` file redundant. Eliminated the line from the agent Makefile to prevent potential conflicts and streamline the initialization process.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-17 15:14:53 +01:00
Danko Miladinovic c25cfec84c Use DHCP for VM networking (#54)
* Add python pip to buildroot

* Use DHCP for agent

* Use systemd for naming network interfaces

* Removed eth0 dhcp setting

* Add ETH_IFACE variable

* Added explanation for ETH_IFACE name

* Further explain ETH_IFACE value
2024-01-15 19:27:07 +01:00
Danko Miladinovic cca75fc5cc Add python pip to buildroot (#51) 2024-01-12 17:51:26 +01:00
Filip Bugarski d24ce2b5bd NOISSUE - Fix Magistrala dependency version (#48)
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2024-01-12 14:44:41 +01:00
Sammy Kerata Oina 200f207673 fix logger (#47)
Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-12 14:34:54 +01:00
Danko Miladinovic 004f286900 Use systemd for HAL startup (#46) 2024-01-12 13:33:46 +01:00
Danko Miladinovic 0a0f757833 Use systemd for HAL boot (#40)
* Use systemd for HAL boot

* Removed unnecessary mkdir command

* Add  newline at the end of file

* Configure eth0 network interface

* Keep kernel naming scheme

* Fix agent.mk file for network interfaces
2024-01-11 22:05:42 +01:00
Sammy Kerata Oina 1167aeb53f NOISSUE - Update documentation (#20)
* Fix bug in agent service and state

The commit fixes a bug in the agent service and state files.
Previously, the condition to check the state in the agent service
was incorrect. It was checking the state directly instead of
using the GetState() method. This has been fixed by using the
GetState() method to check the state.

Additionally, a new GetState() method has been added to the
StateMachine struct in the state file. This method retrieves
the current state by acquiring a lock and returning the state
value.

The changes have been tested and verified to resolve the bug
and improve the accuracy of state checking in the agent service.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix bug in agent state machine

The bug in the agent state machine caused an error when attempting an invalid transition. This commit fixes the bug by properly locking and unlocking the state machine before and after transitioning to the next state. Additionally, the logger now correctly logs the current and next state during a valid transition.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix race condition in state machine

The commit fixes a race condition in the state machine implementation in the `Start` method. The race condition occurs when multiple goroutines try to access and modify the state concurrently. To fix this, a mutex lock and unlock are added around the critical sections of code to ensure exclusive access to the state variable. This prevents race conditions and ensures the state transitions are executed correctly.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix race condition in StateMachine.Start()

The StateMachine.Start() method was experiencing a race condition
when multiple events were being processed concurrently. This was
caused by not properly locking and unlocking the state machine
before and after updating the state. This commit fixes the issue
by adding proper locking and unlocking around the state update
operation. Additionally, the logging statement has been updated
to include the previous and next states for better debugging.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* add magistrala dep

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove mainflux

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger.

The commit message should be:

"Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger"

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix notification topic in agent service and update NATS ports in Docker environment variables

The agent service's notification topic was incorrectly set to "channels.manager" instead of "agent". This commit fixes the issue by updating the notification topic.

Additionally, the NATS ports in the Docker environment variables were incorrect. The COCOS_NATS_PORT and COCOS_NATS_HTTP_PORT have been updated to the correct values.

These changes ensure that the agent service uses the correct notification topic and the NATS ports are properly configured.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor Makefile and add new targets for building Docker images

The Makefile has been refactored to include new targets for building Docker images. The `make_docker` and `make_docker_dev` functions have been defined to handle the Docker build process. The `dockers` and `dockers_dev` targets have been added to build the Docker images for all services and development environments respectively.

This commit introduces changes to the Makefile to improve the build process and provide support for Docker images.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update readme

Signed-off-by: SammyOina <sammyoina@gmail.com>

* wrap env vars

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor main.go in cmd/cli

The main.go file in cmd/cli has been refactored to improve code readability and maintainability. The defURL constant has been removed as it is no longer needed. Additionally, unnecessary whitespace has been removed.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix linting

Signed-off-by: sammy <sammyoina@gmail.com>

* rename cocos-ai to cocos

Signed-off-by: sammy <sammyoina@gmail.com>

* Updated README with NATS setup instructions and correct systemd path

Expanded the project's README to include setup instructions for the NATS server, which is now a necessary component for agent and manager communication. Additionally, the systemd service file path has been corrected from 'systemd' to 'init/systemd' ensuring the agent's proper installation and configuration as a daemon. This facilitates a smoother setup experience and reflects the dependency on NATS for push notifications.

Ref: Agent and manager communication enhancement
Signed-off-by: SammyOina <sammyoina@gmail.com>

* Added message broker URL to agent service config

Included the environment variable for the message broker URL in the systemd service configuration to facilitate agent communication with the messaging system. This ensures the agent can connect to the designated message broker for event publishing and subscription.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update docs to hal

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Remove TLS and timeout config from gRPC client

Refactored gRPC client and associated API code by removing unnecessary TLS configuration and timeout settings. Simplified the communication setup by trusting the environment to enforce security policies and handle operation durations, instead of hardcoding these within the application.

This brings about a cleaner, more maintainable codebase and shifts responsibility for security configurations out of the code, aligning with infrastructure-as-code practices and enabling easier scalability and environment-specific adjustments.

This change also affects the API documentation and example commands, which have been updated accordingly to reflect the simplification and to guide users with the streamlined setup process.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
Signed-off-by: sammy <sammyoina@gmail.com>
2024-01-11 15:19:34 +01:00
Sammy Kerata Oina a5e6cae92c COCOS-44 - Add agent address to run responses (#45)
* Add agent address to run responses

The manager service's Run method now retrieves the agent address upon successful computation execution, providing more informative responses across gRPC, HTTP, and logging endpoints. This change improves service transparency by returning the agent's address to be used by client services, making the manager service's external communication more comprehensive. Updated the `RunResponse` structure in the corresponding protocol buffers definition and response handling in gRPC and HTTP APIs, ensuring that agent address information is serialized appropriately. It also necessitates a slight adjustment in the QEMU configuration to manage port forwarding rules more dynamically, simplifying the process as only agent-relevant ports are incremented with each new computation. This extra detail in responses aids in debugging and offers better integration capabilities for clients.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Allocate dynamic ports for VM guests and expand error handling

Refactored service initialization to accept host IP and incorporated dynamic port allocation for VM guests, replacing the prior static increment method. Introduced a new error type 'ErrFailedToAllocatePort' to capture instances where the system is unable to find a free port. Integrated a third-party error package for improved error wrapping and context. These changes prevent port conflicts between VM guests and enhance error diagnostics for service operations.

Resolves issue with static port allocation leading to conflicts.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add HOST_IP to service configuration

Extend service configuration to include the host machine's IP address, allowing instances to be aware of their deployment environment. This update passes the new HostIP field to the service constructor, ensuring the service can now operate with host-specific logic.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Populate AgentAddress in gRPC Responses

Enhanced the gRPC encode/decode functions to properly populate the 'AgentAddress' field in 'RunResponse' objects. This ensures that consumers of the gRPC interface receive complete response data, which previously omitted the important 'AgentAddress' information. The change impacts both server-side response encoding and client-side response decoding, aligning the implementation with the expected interface contract.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix ci

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-11 15:08:35 +01:00
Sammy Kerata Oina c29ef354fa COCOS-24 - Add events service via http (#27)
* add status endpoint

Signed-off-by: sammy <sammyoina@gmail.com>

* feat: Update code generation tools to latest versions

Update the code generation tools, including protoc-gen-go and
protoc-gen-go-grpc, to their latest versions (v1.31.0 and v1.3.0,
respectively). This ensures compatibility with the latest features and
improvements. The updated tools also require gRPC-Go v1.32.0 or later.

The new versions bring important updates and bug fixes, enhancing the
performance and stability of the generated code. By staying up-to-date
with the latest tooling, we can take advantage of the latest
functionality and ensure a smooth development experience.

No code changes are included in this commit. These updates only impact
the code generation process.

Signed-off-by: sammy <sammyoina@gmail.com>

* Update Go version to 1.21.x

Signed-off-by: sammy <sammyoina@gmail.com>

* Refactor agent and manager services to publish event notifications

The refactoring includes changes to the agent and manager services to incorporate event notifications. By publishing events, the services can inform subscribers about the current state of the computation or any updates. Specifically, the `agentService` now includes a `cmpHash` field to store the SHA-256 hash of the computation, which is subsequently used when publishing events. The `agentService` and `managerService` now use the `publisher` interface to publish events to the topic "manager". Notably, the removed `pubsub.go` file is no longer necessary.

This commit improves the service architecture by allowing subscribers to receive relevant updates and monitor the progress of computations. It enhances the overall system by providing more transparency and enabling better coordination between the agent and manager services.

Signed-off-by: sammy <sammyoina@gmail.com>

* Improve generated Go file comparison in checkproto workflow

Refactor the file comparison logic in the checkproto workflow to use the `-p` flag instead of `-s` for improved accuracy. This change ensures that the generated Go files are thoroughly compared with the original ones, detecting any discrepancies and preventing out-of-sync files from passing the validation. By using the `-p` flag, we now check both the contents and the metadata of the files, providing more robust synchronization checks. This update enhances the reliability of the checkproto workflow and helps maintain consistency between the proto files and their corresponding generated Go files.

Signed-off-by: sammy <sammyoina@gmail.com>

* Update file comparison command to detect differences line by line

The code change updates the file comparison command used in the CI workflow to detect differences line by line instead of only reporting the first difference encountered. This change improves the accuracy of detecting inconsistencies between the original protobuf files and the generated Go files. Previously, only the first difference was reported, leading to potential missed issues. By comparing the files line by line, we can now detect and report all differences accurately. This change enhances the reliability of our CI pipeline and ensures that the generated Go files stay in sync with the protobuf files.

Signed-off-by: sammy <sammyoina@gmail.com>

* add event exporting to external server

Signed-off-by: sammy <sammyoina@gmail.com>

* feat: Add support for notification server URL

The commit adds a new environment variable, `COCOS_NOTIFICATION_SERVER_URL`, which allows specifying the server to receive notification events from the agent. This addition provides flexibility to configure the notification server URL based on the deployment environment. This change enables seamless integration with different notification server instances and enhances the extensibility of the system. It resolves the need to modify the code directly when changing the server URL.

Signed-off-by: sammy <sammyoina@gmail.com>

* Refactor gRPC client and server, remove unused handlers

The commit refactors the gRPC client and server code by removing the unused `nopDecoder` and `status` handlers from the client and server, respectively. This cleanup reduces code clutter and improves maintenance. No significant consequences are expected.

Signed-off-by: sammy <sammyoina@gmail.com>

* Ensure generated Go files stay in sync with proto files during the CI workflow

Fixes an issue in the CI workflow where proto files and their corresponding generated Go files were not being properly compared for synchronization. Previously, the `cmp -l` command was used, which only printed differing byte positions, leading to false negatives. This has been corrected by using `cmp -s` instead, which outputs nothing if the files are identical. This change ensures that any differences between the proto files and their generated Go files will be detected, helping to maintain consistency and accuracy in the codebase.

Signed-off-by: sammy <sammyoina@gmail.com>

* Enhance notification payload and endpoint

Extended the notification system to include 'status' and 'details' in the payload, improving traceability and debugging. Adapted the serialized JSON structure for clarity and added an 'originator' field to track the source service. Transitioned to a generalized event endpoint, facilitating a more streamlined event handling process.

Refactors POST request to a more appropriate endpoint and updates the notification service interface to reflect new payload requirements.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor event notification logic

Removed the legacy notifications package and consolidated event notification functionality using the new internal events service. Modified agent, manager, and main application code to use this service for consistent event reporting and error handling workflows across services. This change simplifies event management, improves error visibility, and allows for more maintainable code by centralizing event-related logic. The substitution of verbose state-specific publishEvent calls with generic status reporting aligns with the new service's capabilities.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Expand agent protobuf message types and improve error handling

The protobuf definition for agent messages has been updated to include an additional message type, facilitating future data structure expansions. Additionally, error handling for event sending in the main agent execution has been enhanced to log errors when sending 'init' events fail, ensuring issues are properly tracked. The unused `notificationTopic` constant in the manager service has been removed for cleaner code maintenance.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Optimize JSON parsing and fix header omission

Removed unnecessary unquoting of a JSON string before unmarshaling, streamlining the computation value extraction process. Also corrected a missing Content-Type header in the event sending function, ensuring proper handling of JSON requests by recipients. These changes improve performance and communication reliability.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* align vars

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Remove computation request timeout feature

The timeout feature for computation requests has been removed to simplify the computation execution flow. This involved changes across multiple files, including protobuf definitions, HTTP endpoint handling, and the internal computation logic. We eliminated the timeout field, associated logic, and error handling to ensure the system no longer supports timeouts for computations, mitigating any unintended timeout impacts on long-running processes.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: sammy <sammyoina@gmail.com>
Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-11 14:37:14 +01:00
Sammy Kerata Oina b55b6c9734 NOISSUE - Refactor computation parameter naming and remove tracing (#42)
Standardized the variable name for computation parameter across various service and middleware implementations to improve code consistency. This enhances readability and maintainability.

Additionally, tracing functionality has been removed completely, signifying a potential shift in the telemetry strategy or removal of an unused feature. The run command associated with cli package has also been deleted, suggesting a change in the CLI tool's capabilities or its command handling strategy.

The service interface's Run method definition and its implementations now align with the updated parameter naming, ensuring coherence throughout the codebase.

Removed files:
- The entire `tracing` package has been deleted.
- Removed the `cli/run.go` file, indicating the removal of the 'run' command from the CLI.

Please note that these changes may affect any existing workflows or systems that rely on the removed tracing capabilities or the CLI run command, and appropriate adjustments will need to be made to accommodate these updates.

Refer to internal documentation or release notes for migration paths or additional context.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-10 11:52:37 +01:00
Sammy Kerata Oina 4d9f7b1a45 COCOS-25 - Remove networking between manager and agent (#43)
* Enable passing computation details to QEMU args

Refactored the QEMU argument construction to accept computation details as JSON. The changes include updating function signatures to pass an `agent.Computation` object and serialize it into the QEMU boot parameters. These modifications allow for custom computations to be specified per VM, facilitating more dynamic and configurable virtual machine creation. This update also integrates timing and data transformation necessary for the agent computation setup.

Resolves computation configurability for virtual machines.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Enable extraction of computation value from cmdline

Introduced a feature to parse the computation parameter directly from the machine's command line, thus enabling dynamic configuration of agent computations without the need for a preset environment. This change enhances the agent's ability to adapt to different computation requirements on startup. A fallback warning informs of cases where the computation parameter was not found, ensuring traceability of operational state during service initialization.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Remove RunRequest and RunResponse from Agent gRPC

Refactor the Agent service by deleting the RunRequest and RunResponse messages along with the corresponding method implementations in the gRPC interface. This simplifies the gRPC protocol by removing an obsolete RPC method, streamlining communication between services. The deletion also impacts the manager service, which no longer uses the removed RPC method. Consequently, the service definitions across related files like endpoints, servers, and clients have been adjusted. This adjustment enforces a more direct approach for starting computations, reflecting a shift toward manager-initiated computation execution.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-09 20:07:45 +01:00
Sammy Kerata Oina 46a91ccbc3 remove nats (#41)
Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-09 19:56:31 +01:00
Sammy Kerata Oina 8c5bb52c56 COCOS-7 - Switch qemu config to use kernel and initrd images (#39)
* Switch qemu config to use kernel and initrd images

Refactored qemu configuration to load the system using separate kernel and initial ramdisk images instead of a QCOW2 image. Removed previous disk image configurations and replaced them with directives to specify the kernel (`bzImage`) and root filesystem (`rootfs.cpio.gz`) files. Adjusted the constructQemuArgs function accordingly to set the kernel path, kernel command line options, and initrd path. Updated CreateVM function to handle copying of the new kernel and rootfs images. These changes align the virtual machine setup process with a more modular and potentially customizable boot configuration.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix lint

Signed-off-by: SammyOina <sammyoina@gmail.com>

* capcomments

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-08 18:05:35 +01:00
Sammy Kerata Oina ee7159a406 COCOS-35 - Refactor RunRequest to use structured Computation (#38)
* Refactor RunRequest to use structured Computation

The protobuf and associated service implementations for the RunRequest message were refactored to replace the raw Computation byte slice with a structured ComputationReq object. This allows clearer and more type-safe manipulation of computation requests. The grpc, http, and agent service layers were updated to build and parse ComputationReq accordingly. The ComputationReq structure includes details like IDs, names, time stamps, and metadata, forming a well-defined contract for computation tasks.

This change aligns with efforts to standardize request formats and improve clarity in inter-service communication. It impacts all systems interfacing with the RunRequest service and thus requires coordinated updates to the entire stack.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Initialize metadata maps and handle nil values

Improved the robustness of metadata handling in gRPC endpoints and SDK by initializing metadata maps and explicitly checking for nil values before converting them. This ensures that both the agent's gRPC endpoint and the SDK properly handle cases where metadata fields may be uninitialized or contain nil values, preventing potential null pointer exceptions.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor computation request handling

Refactored the endpoint to construct Computation object from gRPC request, incorporating structpb for metadata handling and timestamppb for StartTime and EndTime fields. The management service and API requests are also updated to align with these changes, improving type safety and ensuring data is correctly marshalled when making service calls.

Resolves data marshalling issues for computation requests.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* use singular

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove unuse fields

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove unused fields

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-08 17:51:46 +01:00
Sammy Kerata Oina ee1f5ebb3d NOISSUE - Optimize context deadline and error handling (#34)
Improved the timing logic for setting computation deadlines by replacing the channel-based pattern with a direct time addition. This change enhances the performance by preventing unnecessary channel operations.

Additionally, relocated the select block that checks for computation timeout and data reception errors. This alteration ensures that the socket connection acceptance routine starts before any timeouts or errors are checked, thereby avoiding potential race conditions and making the code more resilient to early timeout scenarios.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-04 12:53:08 +01:00
Sammy Kerata Oina 7ea8e48dbb NOISSUE - Optimized Docker build (#32)
Refactored the Dockerfile to use multi-stage builds, improving the image size and separation of concerns. Removed `qemu-system-x86_64` from the initial build stage and included it in a new, minimal final image based on `alpine:latest`. This change minimizes the production image size by avoiding unnecessary build dependencies in the final artefact. Additionally, the CA certificates are now copied into the final stage to maintain SSL functionality.

The development Dockerfile has been cleaned to match new conventions.

Resolves issues with bloated Docker images and unneeded dependencies in production.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-04 12:35:29 +01:00
Sammy Kerata Oina a3c5c765b8 NOISSUE - Connect to agent when vm is created (#33)
* Add gRPC TLS config and update protoc versions

Enhanced manager service with TLS configuration options and timeout settings for gRPC communication with agent services. Updated corresponding protobuf definitions to include new fields for CA certificates, TLS status, and timeouts. The added TLS support ensures secure inter-service communication while flexible timeout configurations improve the robustness of network interactions.

The following adjustments and additions were made:
- Included new gRPC configuration fields (ca_certs, client_tls, timeout) for agent communication in protobuf definitions.
- Injected agent gRPC configuration into service endpoints, ensuring secure TLS setup and compliance with provided settings.
- Revised main service function signatures to accept the gRPC configuration object.
- Incremented port forwarding counters post-computation to avoid port conflicts.
- Conducted compatibility update of protobuf version comments to reflect minor version bump (v4.25.0 to v4.25.1).

This change impacts service deployment that requires proper configuration of TLS credentials and mindful determination of timeouts for efficient network use.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update proto

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Set default timeout for agent configuration

Introduced a default timeout of 60 seconds in both gRPC and HTTP endpoints for agent configuration when none is specified. This change ensures that operations do not hang indefinitely and provides a reasonable default for client interactions. Additionally, the instantiation of a new agent client is now outside the retry loop to avoid repeated setup on transient failures.

Refactors service logic to optimize client connection handling by moving the agent client setup to occur before attempting retries, which should reduce overhead and improve clarity in error situations.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Subject: Move agent client creation into retry loop

Body:
Refactored the service manager's Run method to initialize the agent gRPC client inside the exponential backoff retry loop. This change addresses intermittent connection issues by reattempting client creation on temporary network failures, ensuring a robust setup before calling the Run method on the client.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2024-01-04 12:34:57 +01:00
Sammy Kerata Oina 34c5472485 NOISSUE - Remove multiple brokers (#31)
* remove multiple brokers

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Standardize build process on NATS messaging

Removed conditional build flags for different message broker types, making NATS the default and only option. Simplified the Makefile by eliminating the toggleable message broker configuration and related docker-based setup. Updated the NATS URL in docker `.env` and docker-compose service dependency to reflect this change. This adjustment streamlines the build process and reduces configuration complexity.

Resolves issues with inconsistent messaging setups across various environments.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor message broker configuration

Standardized the environment variable for the message broker URL across services and updated documentation to reflect this change. Renamed the environment variable from COCOS_NATS_URL to COCOS_MESSAGE_BROKER_URL to improve clarity and maintain consistency in service configuration. This alteration facilitates future support for different message broker backends beyond NATS without further variable name changes.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2023-12-24 19:45:04 +01:00
Sammy Kerata Oina 1fec5fe3d4 add repo readme (#30)
Signed-off-by: SammyOina <sammyoina@gmail.com>
2023-12-24 15:46:31 +01:00
Dusan Borovcanin 4c53fd26e3 NOISSUE - Fix CI Docker build
Signed-off-by: Dusan Borovcanin <borovcanindusan1@gmail.com>
2023-12-21 17:43:03 +01:00
Sammy Kerata Oina 668552256f NOISSUE - Create and publish a Docker image (#29)
* Create and publish a Docker image

This commit adds a new file `.github/workflows/build.yml` which contains the workflow configuration for creating and publishing a Docker image. The workflow is triggered on push events to the `main` branch. It sets environment variables for the registry and image name. The job runs on `ubuntu-latest` and has necessary permissions for reading contents and writing packages. The steps include checking out the repository, logging in to the container registry using the GitHub token, and extracting metadata for Docker.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor build workflow and update Makefile

The build workflow has been refactored to build the Manager image instead of the API Gateway image. The Makefile has also been updated.

The changes include:
- Renaming the build step to "Build Manager and push Docker image"
- Updating the image name to include "/manager"
- Updating the build arguments to use "SVC=manager"
- Updating the tags to use the output from the metadata action

These changes ensure that the correct Docker image is built and pushed.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove labels

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2023-12-21 17:26:34 +01:00
Danko Miladinovic 3e7fc09233 Update documentation (#28)
* Modify external.desc and README

* Added a blank line at end of external.desc file
2023-12-08 16:45:02 +01:00
Drasko DRASKOVIC 0dac01d752 NOISSUE - Add HAL (#26)
* Add HAL

Signed-off-by: Drasko DRASKOVIC <drasko.draskovic@gmail.com>

* Add Agent package in HAL

Signed-off-by: Drasko DRASKOVIC <drasko.draskovic@gmail.com>

---------

Signed-off-by: Drasko DRASKOVIC <drasko.draskovic@gmail.com>
2023-12-05 15:08:49 +01:00
Sammy Kerata Oina 18508796c1 add linters (#23)
- switchTrue
      - importShadow
      - httpNoBody
      - paramTypeCombine
      - emptyStringTest
      - builtinShadow
      - exposedSyncMutex
      - importas

Signed-off-by: SammyOina <sammyoina@gmail.com>
2023-11-24 18:59:42 +01:00
Sammy Kerata Oina 5adf0349a3 COCOS-9 - Dockerize manager (#18)
* Fix bug in agent service and state

The commit fixes a bug in the agent service and state files.
Previously, the condition to check the state in the agent service
was incorrect. It was checking the state directly instead of
using the GetState() method. This has been fixed by using the
GetState() method to check the state.

Additionally, a new GetState() method has been added to the
StateMachine struct in the state file. This method retrieves
the current state by acquiring a lock and returning the state
value.

The changes have been tested and verified to resolve the bug
and improve the accuracy of state checking in the agent service.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix bug in agent state machine

The bug in the agent state machine caused an error when attempting an invalid transition. This commit fixes the bug by properly locking and unlocking the state machine before and after transitioning to the next state. Additionally, the logger now correctly logs the current and next state during a valid transition.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix race condition in state machine

The commit fixes a race condition in the state machine implementation in the `Start` method. The race condition occurs when multiple goroutines try to access and modify the state concurrently. To fix this, a mutex lock and unlock are added around the critical sections of code to ensure exclusive access to the state variable. This prevents race conditions and ensures the state transitions are executed correctly.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix race condition in StateMachine.Start()

The StateMachine.Start() method was experiencing a race condition
when multiple events were being processed concurrently. This was
caused by not properly locking and unlocking the state machine
before and after updating the state. This commit fixes the issue
by adding proper locking and unlocking around the state update
operation. Additionally, the logging statement has been updated
to include the previous and next states for better debugging.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* add magistrala dep

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove mainflux

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger.

The commit message should be:

"Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger"

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix Makefile to properly set DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE

The Makefile has been updated to fix an issue with setting the DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE variables. The USER_REPO variable is now used to generate the DOCKER_PROJECT name following the Docker Compose guidelines. Additionally, the COCOS_MESSAGE_BROKER_TYPE variable is now properly set to "nats" if it is empty. This ensures that the correct values are used when compiling and installing the service.

Summary:
Fix Makefile to properly set DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE

Details:
- Update USER_REPO variable to generate DOCKER_PROJECT name
- Set COCOS_MESSAGE_BROKER_TYPE to "nats" if empty

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix notification topic in agent service and update NATS ports in Docker environment variables

The agent service's notification topic was incorrectly set to "channels.manager" instead of "agent". This commit fixes the issue by updating the notification topic.

Additionally, the NATS ports in the Docker environment variables were incorrect. The COCOS_NATS_PORT and COCOS_NATS_HTTP_PORT have been updated to the correct values.

These changes ensure that the agent service uses the correct notification topic and the NATS ports are properly configured.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add Dockerfiles for production and development environments

This commit adds two new Dockerfiles, one for the production environment and one for the development environment. The production Dockerfile sets up the necessary dependencies and builds the application, while the development Dockerfile simply copies the built application. Both Dockerfiles include the necessary SSL certificates for the application to work properly.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor Makefile and add new targets for building Docker images

The Makefile has been refactored to include new targets for building Docker images. The `make_docker` and `make_docker_dev` functions have been defined to handle the Docker build process. The `dockers` and `dockers_dev` targets have been added to build the Docker images for all services and development environments respectively.

This commit introduces changes to the Makefile to improve the build process and provide support for Docker images.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix makefile to generate Docker images for services

- Added DOCKERS and DOCKERS_DEV variables to generate Docker images for services
- Updated the dockers target to include DOCKERS
- Updated the dockers_dev target to include DOCKERS_DEV
- Updated the $(DOCKERS) and $(DOCKERS_DEV) targets to call the respective make_docker and make_docker_dev functions

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor Makefile and Dockerfile

The Makefile has been refactored to remove unnecessary sed commands and improve code readability. The changes include:

- Removed sed commands related to changing the broker in docker-compose.yml for nats and rabbitmq profiles.
- Removed the eer target from the Makefile.

The Dockerfile has been modified to include the WORKDIR directive.

These changes improve the maintainability and readability of the codebase.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix NATS URL in .env file and add Manager configuration in docker-compose.yml

The NATS URL in the .env file was updated to use the correct hostname for the broker. Additionally, the Manager configuration was added to the docker-compose.yml file.

This commit fixes the NATS URL in the .env file and adds the necessary Manager configuration to the docker-compose.yml file.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix incorrect host and port values in Makefile and .env

The commit updates the host and port values in the Makefile and .env files to fix incorrect values. The host and port values are updated to "cocos-manager" and "7003" respectively. This ensures that the correct host and port are used for the Manager HTTP and gRPC services.

This commit resolves the issue with the incorrect host and port values and ensures that the Manager services are configured correctly.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix Dockerfile and docker-compose.yml configurations

The Dockerfile and docker-compose.yml configurations have been fixed to address the following issues:

- Removed unnecessary COPY commands in the Dockerfile.
- Removed the "privileged" flag in the docker-compose.yml file.

These changes ensure that the Docker image is built correctly and the container is launched with the appropriate configurations.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove manual assignment

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update protoc CI

Signed-off-by: SammyOina <sammyoina@gmail.com>

* add paths

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove profiles

Signed-off-by: SammyOina <sammyoina@gmail.com>

* restore default config

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix Docker project name in Makefile

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2023-11-24 18:57:31 +01:00
Sammy Kerata Oina 0f831706f9 NOISSUE - Add CoCos-AI Manager API specification (#19)
* Add CoCos-AI Manager API specification

This commit adds the CoCos-AI Manager API specification
in the form of a YAML file. The specification includes
information about the title, description, contact,
license, and version of the API. It also defines the
servers where the API is hosted and the paths and
operations available, such as running computation on
a virtual machine.

The API specification is based on OpenAPI 3.0.1 and
provides a clear and concise overview of the CoCos-AI
Manager service.

The commit also includes a link to the CoCos-AI repository
and the license information.

This commit is necessary to provide a clear and documented
API specification for the CoCos-AI Manager service.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor go.mod and go.sum files

The go.mod and go.sum files have been refactored to remove the go.opentelemetry.io/contrib/propagators/jaeger package, which is no longer needed. This package was causing compatibility issues with the current version of the project. The refactoring ensures that the project is using the latest compatible versions of the required dependencies.

This commit removes the go.opentelemetry.io/contrib/propagators/jaeger package from the go.mod file and updates the go.sum file accordingly.

Note: The go.mod file now uses go.opentelemetry.io/otel v1.19.0 and go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.19.0.

Please review the changes to ensure compatibility and functionality.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix typo

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2023-11-24 18:54:54 +01:00
Sammy Kerata Oina 298878c96d NOISSUE - Rename module to cocos (#22)
* add stringer

Signed-off-by: SammyOina <sammyoina@gmail.com>

* rename module to cocos

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix fmt.Stringer implementation in computations.go

The fmt.Stringer implementation for Datasets and Algorithms in computations.go was fixed to correctly use pointers.

This commit addresses the issue where the String() method for Datasets and Algorithms in computations.go was not correctly implemented. The fix ensures that the String() method now correctly marshals the data to JSON and returns the string representation.

The changes made in this commit will improve the functionality and accuracy of the String() method for Datasets and Algorithms.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2023-11-22 13:15:19 +01:00
Dušan Borovčanin 7b31e21818 NOISSUE - Fix attestation gRPC call (#21)
Signed-off-by: Dusan Borovcanin <borovcanindusan1@gmail.com>
2023-11-21 12:25:24 +01:00
Sammy Kerata Oina 4ea9ff6531 NOISSUE - Add message broker on agent and manager (#17)
* Fix bug in agent state machine

The bug in the agent state machine caused an error when attempting an invalid transition. This commit fixes the bug by properly locking and unlocking the state machine before and after transitioning to the next state. Additionally, the logger now correctly logs the current and next state during a valid transition.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix race condition in state machine

The commit fixes a race condition in the state machine implementation in the `Start` method. The race condition occurs when multiple goroutines try to access and modify the state concurrently. To fix this, a mutex lock and unlock are added around the critical sections of code to ensure exclusive access to the state variable. This prevents race conditions and ensures the state transitions are executed correctly.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix race condition in StateMachine.Start()

The StateMachine.Start() method was experiencing a race condition
when multiple events were being processed concurrently. This was
caused by not properly locking and unlocking the state machine
before and after updating the state. This commit fixes the issue
by adding proper locking and unlocking around the state update
operation. Additionally, the logging statement has been updated
to include the previous and next states for better debugging.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* add magistrala dep

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove mainflux

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix agentService New function to include messaging.Publisher parameter

The agentService New function has been updated to include a messaging.Publisher parameter. This change allows the agent service to publish messages to a messaging system. The messaging.Publisher parameter has been added to the agentService struct and the New function signature has been updated accordingly. This change ensures that the agent service can communicate with other components using the messaging system.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Refactor service.go state functions

The commit refactors the state functions in the service.go file.
The functions for each state have been modified to use the svc.publishEvent
method to publish events with appropriate messages.

- Refactor state functions in service.go
- Use svc.publishEvent to publish events with messages for each state

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix computation run event publishing and add pubsub functionality

The computation run event publishing in the agent service was fixed to correctly call the publishEvent function. Additionally, the pubsub functionality was added to the manager package.

- Fixed computation run event publishing in agent service
- Added pubsub functionality to manager package

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix license header in pubsub.go file

The commit fixes the license header in the pubsub.go file.
The copyright and SPDX-License-Identifier have been added
to comply with the Apache-2.0 license.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger.

The commit message should be:

"Add Docker environment variables for Nats, RabbitMQ, Message Broker, and Jaeger"

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix Makefile to properly set DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE

The Makefile has been updated to fix an issue with setting the DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE variables. The USER_REPO variable is now used to generate the DOCKER_PROJECT name following the Docker Compose guidelines. Additionally, the COCOS_MESSAGE_BROKER_TYPE variable is now properly set to "nats" if it is empty. This ensures that the correct values are used when compiling and installing the service.

Summary:
Fix Makefile to properly set DOCKER_PROJECT and COCOS_MESSAGE_BROKER_TYPE

Details:
- Update USER_REPO variable to generate DOCKER_PROJECT name
- Set COCOS_MESSAGE_BROKER_TYPE to "nats" if empty

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix Makefile Docker profile assignment and build flags

The Makefile was updated to fix the assignment of the Docker profile and build flags. The Docker profile is now assigned based on the value of COCOS_MESSAGE_BROKER_TYPE, and if it is not provided, the default value is set to "nats". The build flags were also updated to include the COCOS_MESSAGE_BROKER_TYPE value as a tag for the Go build process.

This commit addresses the issue with the Docker profile assignment and ensures that the correct build flags are used during the build process.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix makefile

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix notification topic in agent service and update NATS ports in Docker environment variables

The agent service's notification topic was incorrectly set to "channels.manager" instead of "agent". This commit fixes the issue by updating the notification topic.

Additionally, the NATS ports in the Docker environment variables were incorrect. The COCOS_NATS_PORT and COCOS_NATS_HTTP_PORT have been updated to the correct values.

These changes ensure that the agent service uses the correct notification topic and the NATS ports are properly configured.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* add pubsub

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update protoc

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2023-11-10 16:12:20 +01:00
Sammy Kerata Oina 4958f9468f NOISSUE - Implement State-machine (#16)
* add state machine

Signed-off-by: SammyOina <sammyoina@gmail.com>

* This commit addresses the following changes:

- In agent/service.go:
  - Added a goroutine to start the StateMachine
  - Sent the start event to the StateMachine

- In agent/state.go:
  - Added a copyright notice and license identifier

These changes ensure that the StateMachine starts correctly and that the agent/state.go file includes the necessary copyright and license information.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix race condition in agent service initialization

The commit fixes a race condition in the agent service initialization.
Previously, the `New` function in `agent/service.go` would start the state machine
goroutine before initializing the `sm` field. This could lead to a race condition
where the state machine would access uninitialized fields.

To fix this, the `New` function now takes a `context.Context` argument and passes it
to the `Start` method of the state machine. Additionally, a `sync.Mutex` has been
added to the `StateMachine` struct to ensure thread safety.

This commit addresses the race condition and ensures proper initialization of the agent service.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix sending events when algorithms and datasets are received

This commit fixes a bug where events were not being sent when algorithms and datasets were received in the agent service. The bug was causing the events to not be triggered, leading to incorrect behavior. This commit adds the necessary code to send the events when algorithms and datasets are received, ensuring that the events are properly triggered.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix bug: computation results not ready

The computation results were not being marked as ready,
resulting in an error when trying to access them.
This commit fixes the issue by adding a check for
the computation results before returning them.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix bug in agent service and state

The commit fixes a bug in the agent service and state files.
Previously, the condition to check the state in the agent service
was incorrect. It was checking the state directly instead of
using the GetState() method. This has been fixed by using the
GetState() method to check the state.

Additionally, a new GetState() method has been added to the
StateMachine struct in the state file. This method retrieves
the current state by acquiring a lock and returning the state
value.

The changes have been tested and verified to resolve the bug
and improve the accuracy of state checking in the agent service.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* check for time outs

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix bug in agent state machine

The bug in the agent state machine caused an error when attempting an invalid transition. This commit fixes the bug by properly locking and unlocking the state machine before and after transitioning to the next state. Additionally, the logger now correctly logs the current and next state during a valid transition.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix race condition in state machine

The commit fixes a race condition in the state machine implementation in the `Start` method. The race condition occurs when multiple goroutines try to access and modify the state concurrently. To fix this, a mutex lock and unlock are added around the critical sections of code to ensure exclusive access to the state variable. This prevents race conditions and ensures the state transitions are executed correctly.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix race condition in StateMachine.Start()

The StateMachine.Start() method was experiencing a race condition
when multiple events were being processed concurrently. This was
caused by not properly locking and unlocking the state machine
before and after updating the state. This commit fixes the issue
by adding proper locking and unlocking around the state update
operation. Additionally, the logging statement has been updated
to include the previous and next states for better debugging.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2023-11-10 14:41:05 +01:00
Washington Kigani Kamadi 4eea9c1a4c Add attestation endpoint (#5)
* vendor

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Return agent changes

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* remove vendor

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Add context to sdk functions

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Fix comment

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Add context to functions

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Remove context variable

Signed-off-by: WashingtonKK
<washingtonkigan@gmail.com>

---------

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2023-11-10 14:05:17 +01:00
Sammy Kerata Oina 8c7d4a76ec NOISSUE - Remove HTTP server from agent and update agent service (#15)
* Update agent.pb.go to include a new field "Provider" in the AlgoRequest struct.

Signed-off-by: SammyOina <sammyoina@gmail.com>

* fix build

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix undeclared consumer error in agent service

The commit fixes an error in the agent service where an undeclared consumer was requesting results in the computation manifest. The error is resolved by adding an error message for the undeclared consumer case.

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: SammyOina <sammyoina@gmail.com>
2023-11-10 13:52:25 +01:00
Sammy Kerata Oina da590bfb63 NOISSUE - Add linters and config file (#12)
* vendor

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* remove vendor

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

---------

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
Co-authored-by: WashingtonKK <washingtonkigan@gmail.com>
2023-11-10 13:49:05 +01:00
Sammy Kerata Oina 70785b96e6 return hashes on agent service. (#14)
Co-authored-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-11-01 13:23:01 +01:00
Sammy Kerata Oina f1f8f95653 NOISSUE - Fix dependencies (#13)
* vendor

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Return agent changes

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Add missing import

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* remove vendor

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Fix formatting

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Formatting errors

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Update agent/api/grpc/client.go

Signed off: WashingtonKK

washingtonkigan@gmail.com

Co-authored-by: Sammy Kerata Oina <44265300+SammyOina@users.noreply.github.com>

* add linters and fix

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update ci

Signed-off-by: SammyOina <sammyoina@gmail.com>

* remove deprecated dependencies and
 use local agent and manager

Signed-off-by: SammyOina <sammyoina@gmail.com>

* update mainflux

Signed-off-by: SammyOina <sammyoina@gmail.com>

* Fix Jaeger URL in agent and manager main.go files

The Jaeger URL in the agent and manager main.go files was incorrect. This commit fixes the Jaeger URL by updating it to "http://localhost::4318/v1/traces".

Signed-off-by: SammyOina <sammyoina@gmail.com>

---------

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
Signed-off-by: SammyOina <sammyoina@gmail.com>
Co-authored-by: WashingtonKK <washingtonkigan@gmail.com>
2023-10-27 22:12:53 +02:00
Washington Kigani Kamadi 7d47ed7a20 Fix CI (#6)
* Fix CI

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Fix CI

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

* Use single make

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

---------

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2023-10-20 17:25:14 +02:00
Drasko Draskovic 79d1834173 Fix compilation
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2023-10-19 18:23:53 +02:00
Drasko Draskovic a63cafc4d0 Fix non-vendor compilation
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2023-10-19 17:58:23 +02:00
Drasko Draskovic 63d8035e46 Fix dir structure
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2023-10-19 17:34:38 +02:00
Drasko Draskovic 96e3c48fcb Remove vendoring
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2023-10-19 17:31:55 +02:00
Drasko DRASKOVIC 91c9274885 Merge pull request #2 from SammyOina/CA-1
CA-1 - Fix build
2023-10-19 15:31:00 +02:00
SammyOina f9a2fd0b96 Fix Makefile version and commit variables
The Makefile was updated to fix the version and commit variables.
Previously, the VERSION variable was using "git describe" to get the latest tag,
but it was failing when there were no tags available.
The updated VERSION variable now uses "git describe --always" to fallback to the commit hash if no tags are available.
The COMMIT variable was also updated to use "git rev-parse HEAD" to get the commit hash.

Summary:
Fix Makefile version and commit variables

Body:
- Update VERSION variable to use "git describe --always" to fallback to commit hash
- Update COMMIT variable to use "git rev-parse HEAD" to get commit hash

Signed-off-by: SammyOina <sammyoina@gmail.com>
2023-10-19 12:58:20 +03:00
SammyOina 7cb04499e8 Update Go version to 1.21.x in GitHub Actions workflow
The Go version in the GitHub Actions workflow was updated from 1.20.x to 1.21.x. This change ensures that the correct version of Go is used during the workflow execution.

Signed-off-by: SammyOina <sammyoina@gmail.com>
2023-10-19 12:30:00 +03:00
SammyOina 7657f8bd7a fix build
Signed-off-by: SammyOina <sammyoina@gmail.com>
2023-10-19 12:22:52 +03:00
Drasko Draskovic dd08b7d0b1 Fix vendoring
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2023-10-18 21:35:35 +02:00
Drasko Draskovic e9d143a7d3 Merge remote-tracking branch 'manager/main'
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2023-10-18 12:59:28 +02:00
Drasko Draskovic f690c286f2 Merge remote-tracking branch 'agent/main'
Signed-off-by: Drasko Draskovic <drasko.draskovic@gmail.com>
2023-10-18 12:50:08 +02:00
Drasko DRASKOVIC 194fd29d60 Initial commit 2023-10-18 12:05:53 +02:00
Darko Draskovic 740a631287 Merge pull request #45 from ultravioletrs/qemuExponentialBackoff
Qemu VM exponential backoff
2023-10-09 17:21:16 +02:00
WashingtonKK 514cc18bf8 Remove debug logs
Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2023-10-09 17:46:13 +03:00
WashingtonKK d96b3414c5 Add backoff
Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2023-10-09 17:00:40 +03:00
WashingtonKK c8d1ad4793 vendor docs
Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2023-10-09 17:00:19 +03:00
WashingtonKK 7302e82173 Fix variables
Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2023-10-09 15:56:25 +03:00
WashingtonKK 2d6487790b Reduce time delay
Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2023-10-09 14:59:20 +03:00
WashingtonKK 0ae70d9d75 Add retry loop
Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>
2023-10-09 14:43:42 +03:00
Darko Draskovic 2e0f3e9240 Merge pull request #38 from ultravioletrs/remove_domain_endpoint
Remove create libvirt domain and create QEMU VM endpoints
2023-09-21 15:52:56 +02:00
Darko Draskovic 4f05805600 Add instructions for AMD-SEV tests (#46)
* Use .venv for AMD-SEV tests

Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>

* Add instructions for VM

Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>

---------

Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-21 15:38:56 +02:00
Drasko DRASKOVIC 5171882634 Merge pull request #45 from ultravioletrs/algo_data
Add basic business logic of training a model on a dataset
2023-09-21 11:43:29 +02:00
Darko Draskovic b0b22aeed3 Add manual test procedure for CLI
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-21 10:47:31 +02:00
Darko Draskovic 16cb51563c Unexport AgentSDK by renaming to agentSDK
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-21 10:22:57 +02:00
Darko Draskovic b78b15efb6 Fix algorithms cobra command to send file and not string
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-21 10:20:33 +02:00
Darko Draskovic 10f9ec8517 Rename results.go to result.go
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-19 15:37:58 +02:00
Darko Draskovic 7cb1916e7a Move unix socket handling to separate pkg and update api yaml doc
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-19 13:58:02 +02:00
Darko Draskovic 0ab28aa3cc Replace stdout pipe with unix socket
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-19 00:35:23 +02:00
Darko Draskovic 54e03f3373 Refactor /result to perform actual computation
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-18 18:09:19 +02:00
Darko Draskovic 91dd8dd7b4 Add py script std out capture by in memory io.Writer
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-18 17:00:31 +02:00
Darko Draskovic e9f4d0c8d4 Add run func to apply python script (algo) to csv data (dataset)
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-18 15:52:51 +02:00
Darko Draskovic 4d09a9f4c3 Add basic business logic of training a model on a dataset and returning a model
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-18 15:08:27 +02:00
Darko Draskovic 115ccc9f7b Remove print to console statements
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-07 16:53:43 +02:00
Darko Draskovic 70840e2a62 Update proto files
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-07 15:03:43 +02:00
Darko Draskovic 83270dc9bc Move libvirt connect func to libvirt package
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-07 14:57:16 +02:00
Darko Draskovic 4354f0574c Add separate libvirt package and move qemu related functions to qemu package
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-07 14:57:16 +02:00
Darko Draskovic db991b529c Remove custom settings from go vendor system
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-07 14:57:16 +02:00
Darko Draskovic ec865c8d68 Add instructions for focal server preparation
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-07 14:57:16 +02:00
Darko Draskovic da7705de61 Refactor README to use blob for computation run req
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-07 14:57:16 +02:00
Darko Draskovic de9feccc51 Refactor run service func to create vm
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-07 14:57:16 +02:00
Darko Draskovic 0049ce2b28 Add temporary sh script to start vm used for testing
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-07 14:57:16 +02:00
Darko Draskovic c4cc8b48bf Remove create libvirt domain and create QEMU VM endpoints
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-07 14:57:16 +02:00
Darko Draskovic eb230b5aa6 Merge pull request #40 from ultravioletrs/checkproto
Bump proto version in github actions
2023-09-07 14:56:47 +02:00
Darko Draskovic e54d2731cd Bump proto version in github actions
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-07 14:15:45 +02:00
Drasko DRASKOVIC a3c4664497 Merge pull request #43 from ultravioletrs/focal
Refactor README to simplify VM setup procedure
2023-09-05 16:51:47 +02:00
Darko Draskovic 38558dbe53 Remove obsolete agent in VM configuration info
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-09-05 16:32:37 +02:00
Darko Draskovic d70d9fe7e7 Remove alpine and packer related files
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-18 17:14:08 +02:00
Darko Draskovic c3443c5375 Refactor README to simplify VM setup procedure
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-18 13:14:39 +02:00
Drasko DRASKOVIC 2cf6e1ac3e Merge pull request #10 from ultravioletrs/manger_SEV
Refactor manager to start AMD SEV encrypted VM
2023-08-16 15:21:48 +02:00
Darko Draskovic 5028946731 Replace literal strings by consts and properly format godoc comments
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-16 11:08:38 +02:00
Darko Draskovic b3ad784ee0 Add file.go and temporary file creation to enable multi instance VM launch
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-15 21:47:23 +02:00
Darko Draskovic 0b38e94787 Add uuid to VM creation config id fields
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-15 20:28:04 +02:00
Darko Draskovic 1536c6cbf5 Revert to env var VM parametrization
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-15 16:03:26 +02:00
Darko Draskovic 6b4a7080e7 Add CreateQemuVM endoint
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-15 15:32:42 +02:00
Darko Draskovic 4137ad3513 Amend VM launch troubleshooting instructions with port forwaring settings
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-15 11:53:55 +02:00
Darko Draskovic 5558a4f72b Add ExecutableAndArgs generator func to qemu and qemuCmd struct to service
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-14 22:32:36 +02:00
Darko Draskovic 08f26af910 Add VM launch troubleshooting instructions
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-14 20:45:26 +02:00
Darko Draskovic 88e05aea0b Refactor README to use focal-server-cloudimg-amd64.img
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-14 16:22:56 +02:00
Darko Draskovic ce6131e5fd Add programmatic qemu bin abs path config
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-13 01:47:32 +02:00
Darko Draskovic 475004cc84 Add use sudo option to qemu command ctor
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-13 01:03:51 +02:00
Darko Draskovic 51e69d8ab4 Replace sh script for ctor of qemu cmd line with go code
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-12 23:14:36 +02:00
Darko Draskovic d7aca933bd Move cmd funcs to internal/cmd.go
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-10 16:49:26 +02:00
Darko Draskovic 7e21a17317 Remove tmp file creation and capture stdout with buffer
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-10 10:57:34 +02:00
Darko Draskovic 1012d2cb0e Add sudo env var
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-09 16:05:40 +02:00
Darko Draskovic 3caaf6a631 Add a tmp file to read qemu cmd line constr output
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-09 15:57:37 +02:00
Darko Draskovic 9c545d3c7c Replace cmd.Output() with cmd.Start()
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-08 18:10:47 +02:00
Darko Draskovic fd68c9712b Merge pull request #36 from ultravioletrs/CLIReadme
Write Readme.md for Agent CLI
2023-08-08 17:13:19 +02:00
fbugarski dab603fb5f Makefile clean, and Readme installation part fix.
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-08-08 16:50:25 +02:00
Darko Draskovic ff47905247 Remove sudo necessity
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-08 14:29:14 +02:00
fbugarski 44f742fdb0 Add missing parts to Readme
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-08-07 18:39:42 +02:00
Darko Draskovic 6294117de8 Add ConstructQemuCommand and fix guestgrpc bug in sh
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-07 18:37:44 +02:00
Darko Draskovic ada7cb96b7 Parse env vars for qemu cmd line config
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-07 17:46:46 +02:00
Darko Draskovic f6d5b616d2 Add uefi bios vars, exec cmd line and log to file parameters
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-08-07 17:21:14 +02:00
fbugarski a29737b296 Add instalation to Readme
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-08-07 16:53:27 +02:00
fbugarski 5762136b0a fix remarks
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-08-07 15:57:25 +02:00
fbugarski 95a6957f4e Organize help output
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-08-04 16:33:05 +02:00
fbugarski 38cdf9405b remove extra code from makefile
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-08-04 13:39:59 +02:00
fbugarski 1a1785e1c7 Write Readme.md for Agent CLI
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-08-04 13:19:44 +02:00
Darko Draskovic b92806c3b5 Refactor dom.xml to start focal-server-cloudimg-amd64.qcow2
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-07-31 17:32:59 +02:00
Darko Draskovic b92c698800 Add host-guest http and grpc port parametrization
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-07-31 12:23:49 +02:00
Darko Draskovic eaef7ddf9a Add proper formatting
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-07-31 12:23:49 +02:00
Darko Draskovic 10e2a6d0da Refactor manager to start AMD SEV encrypted VM
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-07-31 12:23:49 +02:00
Drasko DRASKOVIC 6aff8ab0bd Merge pull request #26 from ultravioletrs/NOISSUE-update
NOISSUE - Change Default Port For GRPC Client To 7001
2023-07-28 15:49:11 +02:00
rodneyosodo 4dac375b87 Change Default Port For Client To 7001 as in main
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-27 13:51:25 +03:00
Darko Draskovic 376f204f54 Merge pull request #25 from ultravioletrs/NOISSUE-update
NOISSUE - Update Version of Agent
2023-07-27 12:40:59 +02:00
rodneyosodo 2847a28a0f Update Version of Agent
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-27 13:35:16 +03:00
Darko Draskovic c2240066f9 Merge pull request #33 from ultravioletrs/NOISSUE-cleanup
NOISSUE - Update ldflags
2023-07-27 12:29:42 +02:00
Darko Draskovic f561f6c4c9 Merge pull request #34 from ultravioletrs/NOISSUE-update
NOISSUE - Upgrade Go and Dependencies
2023-07-27 12:29:07 +02:00
Darko Draskovic a7ce9b1d87 Merge pull request #23 from ultravioletrs/NOISSUE-cleanup
DRAFT - Update ProtoC workflow
2023-07-27 12:28:12 +02:00
Darko Draskovic c9a7695b64 Merge pull request #24 from ultravioletrs/NOISSUE-update
DRAFT - Upgrade Go and Dependencies
2023-07-27 12:27:21 +02:00
rodneyosodo 15d97f64cb Upgrade Go and Dependencies
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-27 12:23:45 +03:00
rodneyosodo b82b2c6269 Upgrade Go and Dependencies
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-27 12:20:28 +03:00
rodneyosodo f73c829d27 Update ProtoC workflow
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-27 12:10:59 +03:00
rodneyosodo 20962362b6 Update ldflags
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-27 12:06:34 +03:00
Darko Draskovic ac4d20e07e Merge pull request #22 from ultravioletrs/NOISSUE-rmmfxkit
NOISSUE - Remove MFXKIT Dep
2023-07-27 11:02:18 +02:00
Darko Draskovic bd884cbcc4 Merge pull request #32 from ultravioletrs/NOISSUE-rmmfxkit
NOISSUE - Remove Docker Composition
2023-07-27 11:00:07 +02:00
rodneyosodo f5f645db0a Remove MFXKIT Dep
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-27 11:59:59 +03:00
rodneyosodo d802f65450 Remove Env File
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-27 11:29:44 +03:00
rodneyosodo f6f7152015 Remove Docker Composition
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-27 11:20:15 +03:00
Darko Draskovic 9747a0b18d Merge pull request #31 from ultravioletrs/NOISSUE-rmmfxkit
NOISSUE - Remove MFXKIT Dep
2023-07-26 17:04:55 +02:00
rodneyosodo 0bf5985f5b Rename Package Name
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-26 18:03:24 +03:00
rodneyosodo d1b40f9a06 Remove MFXKIT Dep
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-26 18:01:14 +03:00
Darko Draskovic 972e15376f Merge pull request #30 from ultravioletrs/NOISSUE-cleanup
NOISSUE - Update Protocol Buffer workflow
2023-07-26 16:59:25 +02:00
rodneyosodo d2fff75874 Update ProtoC workflow
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-26 15:34:29 +03:00
Darko Draskovic 5c3413ff74 Merge pull request #21 from ultravioletrs/NOISSUE-servicelayertracing
NOISSUE - Add Service Layer Tracing
2023-07-26 13:17:46 +02:00
rodneyosodo 897e58faec Update MANAGER_JAEGER_URL
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-26 14:06:41 +03:00
rodneyosodo f5d3b7c15b Move From OpenTracing To OpenTelemetry
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 18:56:53 +03:00
Darko Draskovic 0d3630a31b Merge pull request #29 from ultravioletrs/NOISSUE-servicelayertracing
NOISSUE - Add Service Layer Tracing
2023-07-25 17:44:25 +02:00
rodneyosodo 8ea5dda282 Propagate context
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 18:38:16 +03:00
rodneyosodo c88d77ac6f Add context to agent functions
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 17:16:51 +03:00
rodneyosodo e46a2f137b Move From OpenTracing To OpenTelemetry
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 17:03:29 +03:00
rodneyosodo 60070be4d5 Add Service Layer Tracing
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 16:24:27 +03:00
rodneyosodo fb596ab1ee Add Service Layer Tracing
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 16:07:18 +03:00
Darko Draskovic f0e7c183e0 Merge pull request #20 from ultravioletrs/NOISSUE-agentclient
NOISSUE - Connect To Agent
2023-07-25 14:58:08 +02:00
rodneyosodo f379123aa7 Remove Empty Line
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 15:51:48 +03:00
rodneyosodo 54072fc052 Connect To Agent
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 15:42:17 +03:00
Darko Draskovic 2175ae3f1a Merge pull request #19 from ultravioletrs/NOISSUE-grpcclient
NOISSUE - Add GRPC Client
2023-07-25 14:33:24 +02:00
Darko Draskovic a903ca23cd Merge pull request #27 from ultravioletrs/NOISSUE-grpcclient
NOISSUE - Add GRPC Client
2023-07-25 13:55:43 +02:00
rodneyosodo d5ac5fd3a9 Add GRPC Client
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 14:39:48 +03:00
rodneyosodo 7fec10fb56 Remove IsSecure
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 14:38:50 +03:00
rodneyosodo b6809fa5b2 Rename newClient
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 14:32:15 +03:00
rodneyosodo 9e2af2c525 Rename agentGRPCConfig
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 13:58:52 +03:00
rodneyosodo 9c74a0e7db Add GRPC Client
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 13:08:24 +03:00
Darko Draskovic 7c9a5315a7 Merge pull request #18 from ultravioletrs/NOISSUE-updateserver
NOISSUE - Move HTTP and GRPC Server Implementations To Internal
2023-07-25 10:45:43 +02:00
Darko Draskovic 5bce3a8839 Merge pull request #25 from ultravioletrs/NOISSUE-updateserver
NOISSUE - Move HTTP and GRPC Server Implementations To Internal
2023-07-25 10:36:58 +02:00
rodneyosodo 9484a44eb2 Rename stopAllServers
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-25 10:51:07 +03:00
rodneyosodo 150b7ab81c Move HTTP and GRPC Server Implementations To Internal
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-24 21:32:06 +03:00
rodneyosodo a1b4a21cb6 Reformat stopAllServer
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-24 19:42:11 +03:00
rodneyosodo b82c62d9b4 Add Inline Err Check
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-24 19:21:05 +03:00
rodneyosodo 94a22d2282 Use Channel of Size 1
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-24 19:06:49 +03:00
rodneyosodo 7ac8372720 Rename StopHandler
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-24 18:57:44 +03:00
rodneyosodo 76b2bed007 Fix Remarks
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-24 15:52:00 +03:00
rodneyosodo 885f4c9be1 Move HTTP and GRPC Server Implementations To Internal
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-24 14:57:05 +03:00
Darko Draskovic 03d637b5bf Merge pull request #17 from ultravioletrs/NOISSUE-updateMF
NOISSUE - Update Mainflux Dependency
2023-07-24 11:41:26 +02:00
Darko Draskovic 0f15d8c731 Merge pull request #16 from ultravioletrs/NOISSUE-metric
NOISSUE - Add Metric To Internal Package
2023-07-24 11:40:51 +02:00
rodneyosodo 0d9aa5d88e Update Mainflux Module
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-24 12:29:45 +03:00
rodneyosodo 300cabbaca Add Metric To Internal Package
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-24 12:25:19 +03:00
Darko Draskovic bb88a71eef Merge pull request #24 from ultravioletrs/NOISSUE-updateMF
NOISSUE - Update Mainflux Dependency
2023-07-24 11:22:10 +02:00
rodneyosodo 8505deedc6 Update Mainflux Module
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-24 12:16:30 +03:00
Darko Draskovic 6cecedd1a1 Merge pull request #23 from ultravioletrs/NOISSUE-metric
NOISSUE - Add Metric To Internal Package
2023-07-24 11:15:11 +02:00
rodneyosodo c09383ac72 Add Metric To Internal Package
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-24 12:00:28 +03:00
Darko Draskovic 8c87ef947b Merge pull request #15 from ultravioletrs/NOISSUE-env
NOISSUE - Change How To Load Env Variables
2023-07-21 18:25:17 +02:00
Darko Draskovic 3a57c8d7a4 Merge pull request #22 from ultravioletrs/NOISSUE-env
NOISSUE - Change How To Load Env Variables
2023-07-21 18:24:46 +02:00
rodneyosodo 1829afd123 Change How To Load Env Variables
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-21 19:20:35 +03:00
rodneyosodo 2c354c3977 Remove MFXKIT Env Vars
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-21 19:18:26 +03:00
rodneyosodo d59b9293cb Initialixe Epty Vars More Idomatic
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-21 19:02:54 +03:00
rodneyosodo b5e64528b1 Make config items Public
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-21 18:55:30 +03:00
rodneyosodo f18e82efa7 Change How To Load Env Variables
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-21 18:20:40 +03:00
Drasko DRASKOVIC 5b60f079b5 Merge pull request #12 from ultravioletrs/docs
Amend README with default domain creation instructions
2023-07-19 13:35:50 +02:00
Darko Draskovic 52e681c28d Amend README with default domain creation instructions
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-07-19 10:43:25 +02:00
Darko Draskovic a3619098dc Merge pull request #19 from ultravioletrs/CLISDK
Connect CLI and SDK
2023-07-17 17:45:14 +02:00
fbugarski bd61a3bd9f Remove println
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-17 16:59:30 +02:00
fbugarski f4ff0feb3c fix err check
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-17 16:39:01 +02:00
fbugarski cc078a1910 Add err check
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-17 16:32:22 +02:00
fbugarski 8f0fd5a1fa Add run, and fix response ,and loglevel
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-17 16:23:22 +02:00
fbugarski 311dc7e1a7 Fix load config
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-17 12:53:46 +02:00
fbugarski cfb0918c95 Remove unused
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-17 12:36:42 +02:00
fbugarski 74c9c306e6 Fix logger error and parse duration
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-17 12:28:15 +02:00
fbugarski 31b1f397d6 Move cfg to main
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-14 18:09:48 +02:00
fbugarski 30d6f97976 Remove unused
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-14 14:16:32 +02:00
fbugarski 5b1403edfc Agent timeout
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-14 13:49:27 +02:00
fbugarski 2ec9df770c Fix grpc connection
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-14 11:21:35 +02:00
Darko Draskovic 62455e35b2 Merge pull request #9 from ultravioletrs/NOISSUE-ci
NOISSUE - Add CI
2023-07-13 09:40:25 +02:00
rodneyosodo 074186dd55 Add Error Check
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-12 19:06:44 +03:00
fbugarski 9a33eb828c Connect CLI and SDK
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-12 14:33:47 +02:00
Darko Draskovic 5938760f8f Merge pull request #16 from ultravioletrs/NOISSUE-gotidy
NOISSUE - Add CI
2023-07-12 14:21:41 +02:00
rodneyosodo 2c54aa78d3 Update Build Command
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-12 13:34:56 +03:00
rodneyosodo d32fd050f7 Initial Commit: Add CI
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-10 15:12:00 +03:00
rodneyosodo f4e71e74b2 Initial Commit: Add CI
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-07-10 15:06:45 +03:00
Dušan Borovčanin 45a92ec842 Merge pull request #18 from ultravioletrs/agentsdk
Agent service SDK
2023-07-07 12:53:20 +02:00
fbugarski 87ea08b08a Add comments and fix url's
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-07 12:43:34 +02:00
fbugarski 0b4fbd5efe Implement SDK funcs
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-07 10:49:17 +02:00
fbugarski f3cc5848ef DRAFT - Agent service SDK
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-07-07 06:24:43 +02:00
Darko Draskovic 71b5617ab3 Add wait time for Linux VM boot
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-26 17:49:30 +02:00
Drasko DRASKOVIC a917c4ba0c Merge pull request #13 from ultravioletrs/cli_openapi
Add open api spec for Agent CLI
2023-06-26 16:05:47 +02:00
fbugarski 93b47a5897 Add initial structs for CLI commands
Signed-off-by: fbugarski <filipbugarski@gmail.com>
2023-06-26 11:53:02 +02:00
Drasko DRASKOVIC 6bd89b08fd Merge pull request #15 from ultravioletrs/c_onciseness
Change client to c for consistency and conciseness
2023-06-22 19:46:38 +02:00
Darko Draskovic 6b7a81799c Change client to c for consistency and conciseness
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-22 19:41:16 +02:00
Darko Draskovic 05ce62c158 Merge pull request #14 from ultravioletrs/grpc_algo_data_result
Add grpc server stub for algo, data and result
2023-06-22 19:37:55 +02:00
Darko Draskovic b5c38308f4 Add Result grpc endpoint and service stub
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-22 14:51:05 +02:00
Darko Draskovic 1c140ba4ee Fix grpc client encode run func to start with user domain req
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-22 11:11:35 +02:00
Darko Draskovic 5277f44e41 Add Algo and Data grpc client funcs
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-21 15:34:26 +02:00
Darko Draskovic 4036dd3c85 Add grpc server stub for algo, data and result
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-21 13:52:45 +02:00
Drasko DRASKOVIC 62234dc7b1 Merge pull request #12 from ultravioletrs/cocosagent.service
Add systemd cocos agent init script
2023-06-20 15:33:40 +02:00
Darko Draskovic ce4726081a Rename cocosagent.service to cocos-agent.service
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-20 15:31:23 +02:00
Darko Draskovic 5296340326 Use /run/cocos for agent.pid file and create /var/log/cocos via systemd script
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-20 15:29:07 +02:00
Darko Draskovic d4e3811640 Use different stderr and stdout agent log files
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-20 15:03:52 +02:00
Darko Draskovic c8d2e96437 Fix Alpine script and README to use /cocos/agent name and location
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-20 12:48:44 +02:00
Darko Draskovic 53370ba0f6 Add /var/log/agent and /cocos dirs to systemd script
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-19 17:02:59 +02:00
Darko Draskovic 5a03d2ae4a Remove /status endpoint
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-19 16:32:46 +02:00
Darko Draskovic 5496af3af2 Remove redundant setup-cocosagent.sh script
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-19 16:07:45 +02:00
Darko Draskovic 01a9f27deb Remove /computation manifest and add /result endpoint
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-19 14:34:29 +02:00
Darko Draskovic d212c77e22 Replace algorithm/dataset endpoint with algorithms/datasets/endpoint
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-19 14:16:20 +02:00
Darko Draskovic 4be83a9888 Add open api spec for Agent CLI
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-19 12:50:33 +02:00
Darko Draskovic 1e9fa759dd Add log files and setup script
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-19 12:15:21 +02:00
Darko Draskovic 066c8ecfe9 Add systemd cococ agent init script
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-15 10:48:43 +02:00
Darko Draskovic f2c58adef6 Add instructions for checking if ports on guest are open
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-14 16:19:19 +02:00
Darko Draskovic 72b56ee40a Merge pull request #8 from ultravioletrs/update-gomod
NOISSUE - Update Version For Agent to Match Latest
2023-06-14 10:26:56 +02:00
rodneyosodo b7c6eadf4f Update Version For Agent to Match Latest
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
2023-06-14 10:24:46 +03:00
Darko Draskovic e05aca0b74 Add instructions related to VM management
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-13 20:41:59 +02:00
Darko Draskovic 49feeb96ee Add instructions for checking if ports on guest are open
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-12 15:54:17 +02:00
Darko Draskovic 5030a20af6 Remove redundant agent dir
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-06-12 15:18:18 +02:00
Darko Draskovic 7b18467800 Remove redundant manager dir
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-30 14:17:34 +02:00
Darko Draskovic a6d0734212 Refactor run request to use slice of bytes to represent computation
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-30 11:27:43 +02:00
Darko Draskovic d3acfd1718 Add agent gRPC client
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-30 11:26:45 +02:00
Darko Draskovic a150433982 Add instructions related to VM management
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-26 12:54:44 +02:00
Darko Draskovic cea3e059fe Add packer json for alpine qcow2 creation
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-23 14:11:55 +02:00
Darko Draskovic 5e2ee315f6 Add instructions related to VM management
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-19 14:34:55 +02:00
Darko Draskovic 79ce0e34f9 Add instructions for interaction with cocos computations microservice
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-19 14:34:07 +02:00
Darko Draskovic 90cea0d143 Add copy files and run computation instructions
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-18 18:42:51 +02:00
Darko Draskovic 2e8603208b Add fun to replace rel with abs paths in xml files
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-18 15:31:00 +02:00
Darko Draskovic 4191560130 Fix wrong xml filepath creation
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-17 15:14:17 +02:00
Darko Draskovic e9e9b8d15c Fix VM creation error when snapshot does not exist
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-17 14:43:37 +02:00
Darko Draskovic f02a500c2d Fix agent rc script to use grpc addr visible from host
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-17 14:03:53 +02:00
Darko Draskovic c1f9fa6328 Add agent grpc svc and client and Run endpoint
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-15 16:24:58 +02:00
Darko Draskovic 750bed0d76 Add run svc endpoint
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-05-12 12:52:00 +02:00
Darko Draskovic 306fa302dc Add OpenRC agent script
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-04-18 16:14:23 +02:00
Darko Draskovic 780c620b30 Add Makefile targets for build and copy to VM agent
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-04-18 10:12:30 +02:00
Darko Draskovic 7383186143 Add libvirt funcs to revert to snapshot
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-04-11 14:43:11 +02:00
Darko Draskovic 023d28ea22 Fix vol and dom config to use qcow2 format to enable snapshots
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-04-10 18:17:59 +02:00
Darko Draskovic 16fb3710ac Add Run client endpoint
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-04-10 17:28:10 +02:00
Darko Draskovic fe7c3d9014 Add Run gRPC endpoint
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-31 17:17:02 +02:00
Darko Draskovic eb4a0943f6 Add Run endpoint
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-31 16:55:32 +02:00
Darko Draskovic b400210738 Add gRPC server start routine
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-29 15:27:22 +02:00
Darko Draskovic 7c12be6a24 Add grpc client skeleton
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-29 11:02:04 +02:00
Darko Draskovic 6f014156b3 Add grpc server skeleton
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-28 17:17:12 +02:00
Darko Draskovic 122b214299 Remove ping endpoint and svc func
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-28 15:07:15 +02:00
Darko Draskovic 8831a4f957 Add create qemu domain endpoint
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-28 14:10:01 +02:00
Darko Draskovic e25abcf5e0 Replace 'cocosvm' with 'manager'
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-27 17:01:03 +02:00
Darko Draskovic ad60e95736 Rename mfxkit to agent
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-27 16:37:52 +02:00
Darko Draskovic 6ec53d074f Add CLI skeleton
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-27 15:40:14 +02:00
Darko Draskovic 35c62cb273 first commit 2023-03-27 13:56:05 +02:00
Darko Draskovic 4d3bcf99b4 Move xml strings to xml files
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-22 14:01:17 +01:00
Darko Draskovic 39c5cc2dd5 Add virtual machine creation
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-15 15:38:31 +01:00
Darko Draskovic 278331dc81 Add libvirt
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-15 11:50:39 +01:00
Darko Draskovic b7a1fbb2fa Rename mfxkit to cocos
Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2023-03-14 15:47:26 +01:00
Darko Draskovic 22c743c4dd first commit 2023-03-14 15:12:23 +01:00
1442 changed files with 13148 additions and 457008 deletions
+28
View File
@@ -0,0 +1,28 @@
## Jaeger
COCOS_JAEGER_PORT=6831
COCOS_JAEGER_FRONTEND=16686
COCOS_JAEGER_COLLECTOR=14268
COCOS_JAEGER_CONFIGS=5778
COCOS_JAEGER_URL=http://jaeger:4318
COCOS_JAEGER_TRACE_RATIO=1.0
COCOS_JAEGER_COLLECTOR_OTLP_ENABLED=true
COCOS_JAEGER_OLTP_HTTP_PORT=4318
## Core Services
### Manager
MANAGER_GRPC_HOST=""
MANAGER_GRPC_PORT=7003
MANAGER_GRPC_SERVER_CERT=""
MANAGER_GRPC_SERVER_KEY=""
AGENT_GRPC_URL="localhost:7002"
AGENT_GRPC_TIMEOUT=""
AGENT_GRPC_CA_CERTS=""
AGENT_GRPC_CLIENT_TLS=""
MANAGER_INSTANCE_ID=""
MANAGER_LOG_LEVEL=debug
MANAGER_QEMU_USE_SUDO=false
MANAGER_QEMU_ENABLE_SEV=false
MANAGER_QEMU_SEV_CBITPOS=51
MANAGER_QEMU_OVMF_CODE_FILE=/usr/share/OVMF/OVMF_CODE.fd
MANAGER_QEMU_OVMF_VARS_FILE=/usr/share/OVMF/OVMF_VARS.fd
+49
View File
@@ -0,0 +1,49 @@
name: Bug Report
description: File a bug/issue report. Make sure to search to see if an issue already exists for the bug you encountered.
title: "Bug: <title>"
labels: ["bug", "needs-review", "help wanted"]
body:
- type: textarea
attributes:
label: What were you trying to achieve?
description: A clear and concise description of what the bug is.
validations:
required: true
- type: textarea
attributes:
label: What are the expected results?
description: A concise description of what you expected to happen.
validations:
required: true
- type: textarea
attributes:
label: What are the received results?
description: A concise description of what you received.
validations:
required: true
- type: textarea
attributes:
label: Steps To Reproduce
description: What are the steps to reproduce the issue?
placeholder: |
1. In this environment...
2. With this config...
3. Run '...'
4. See error...
validations:
required: false
- type: textarea
attributes:
label: In what environment did you encounter the issue?
description: A concise description of the environment you encountered the issue in.
validations:
required: true
- type: textarea
attributes:
label: Additional information you deem important
description: |
Links? References? Anything that will give us more context about the issue you are encountering!
Tip: You can attach images or log files by clicking this area to highlight it and then dragging files in.
validations:
required: false
+1
View File
@@ -0,0 +1 @@
blank_issues_enabled: false
@@ -0,0 +1,36 @@
name: Feature Request
description: File a feature request. Make sure to search to see if a request already exists for the feature you are requesting.
title: "Feature: <title>"
labels: ["enchancement", "needs-review"]
body:
- type: textarea
attributes:
label: Is your feature request related to a problem? Please describe.
description: A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
validations:
required: true
- type: textarea
attributes:
label: Describe the feature you are requesting, as well as the possible use case(s) for it.
description: A clear and concise description of what you want to happen.
validations:
required: true
- type: dropdown
attributes:
label: Indicate the importance of this feature to you.
description: This will help us prioritize the feature request.
options:
- Must-have
- Should-have
- Nice-to-have
validations:
required: true
- type: textarea
attributes:
label: Anything else?
description: |
Links? References? Anything that will give us more context about the feature that you are requesting.
Tip: You can attach images or log files by clicking this area to highlight it and then dragging files in.
validations:
required: false
+66
View File
@@ -0,0 +1,66 @@
<!--
Pull request title should be `COCOS-XXX - description` or `NOISSUE - description` where XXX is ID of the issue that this PR relate to.
Please review the [CONTRIBUTING.md](https://github.com/ultravioletrs/cocos/blob/master/CONTRIBUTING.md) file for detailed contributing guidelines.
For Work In Progress Pull Requests, please use the Draft PR feature, see https://github.blog/2019-02-14-introducing-draft-pull-requests/ for further details.
For a timely review/response, please avoid force-pushing additional commits if your PR already received reviews or comments.
- Provide tests for your changes.
- Use descriptive commit messages.
- Comment your code where appropriate.
- Squash your commits
- Update any related documentation.
-->
# What type of PR is this?
<!--This represents the type of PR you are submitting.
For example:
This is a bug fix because it fixes the following issue: #1234
This is a feature because it adds the following functionality: ...
This is a refactor because it changes the following functionality: ...
This is a documentation update because it updates the following documentation: ...
This is a dependency update because it updates the following dependencies: ...
This is an optimization because it improves the following functionality: ...
-->
## What does this do?
<!--
Please provide a brief description of what this PR is intended to do.
Include List any changes that modify/break current functionality.
-->
## Which issue(s) does this PR fix/relate to?
<!--
For pull requests that relate or close an issue, please include them below. We like to follow [Github's guidance on linking issues to pull requests](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue).
For example having the text: "Resolves #1234" would connect the current pull request to issue 1234. And when we merge the pull request, Github will automatically close the issue.
-->
- Related Issue #
- Resolves #
## Have you included tests for your changes?
<!--If you have not included tests, please explain why.
For example:
Yes, I have included tests for my changes.
No, I have not included tests because I do not know how to.
-->
## Did you document any new/modified feature?
<!--If you have not included documentation, please explain why.
For example:
Yes, I have updated the documentation for the new feature.
No, I have not updated the documentation because I do not know how to.
-->
### Notes
<!--Please provide any additional information you feel is important.-->
+82
View File
@@ -0,0 +1,82 @@
name: CI
on:
push:
branches:
- main
paths:
- "manager/manager.proto"
- "pkg/manager/*.pb.go"
- "agent/agent.proto"
- "agent/*.pb.go"
pull_request:
branches:
- main
paths:
- "manager/manager.proto"
- "pkg/manager/*.pb.go"
- "agent/agent.proto"
- "agent/*.pb.go"
jobs:
checkproto:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: 1.22.x
- name: Set up protoc
run: |
PROTOC_VERSION=25.3
PROTOC_GEN_VERSION=v1.34.2
PROTOC_GRPC_VERSION=v1.4.0
# Download and install protoc
PROTOC_ZIP=protoc-$PROTOC_VERSION-linux-x86_64.zip
curl -0L -o $PROTOC_ZIP https://github.com/protocolbuffers/protobuf/releases/download/v$PROTOC_VERSION/$PROTOC_ZIP
unzip -o $PROTOC_ZIP -d protoc3
sudo mv protoc3/bin/* /usr/local/bin/
sudo mv protoc3/include/* /usr/local/include/
rm -rf $PROTOC_ZIP protoc3
# Install protoc-gen-go and protoc-gen-go-grpc
go install google.golang.org/protobuf/cmd/protoc-gen-go@$PROTOC_GEN_VERSION
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@$PROTOC_GRPC_VERSION
# Add protoc to the PATH
export PATH=$PATH:/usr/local/bin/protoc
- name: Set up Cocos-AI
run: |
# Rename .pb.go files to .pb.go.tmp to prevent conflicts
for p in $(ls pkg/manager/*.pb.go); do
mv $p $p.tmp
done
for p in $(ls agent/*.pb.go); do
mv $p $p.tmp
done
# Generate Go files from protobuf definitions
make protoc
# Compare generated Go files with the original ones
for p in $(ls pkg/manager/*.pb.go); do
if ! cmp -s $p $p.tmp; then
echo "Proto file and generated Go file $p are out of sync!"
exit 1
fi
done
for p in $(ls agent/*.pb.go); do
if ! cmp -s $p $p.tmp; then
echo "Proto file and generated Go file $p are out of sync!"
exit 1
fi
done
-39
View File
@@ -1,39 +0,0 @@
name: CI
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
test:
strategy:
matrix:
go-version: [1.19.x]
os: [ubuntu-latest]
runs-on: ${{ matrix.os }}
steps:
- name: Install Go
uses: actions/setup-go@v3
with:
go-version: ${{ matrix.go-version }}
- name: Checkout code
uses: actions/checkout@v2
- name: Run go vet
run: go vet ./...
- name: golangci-lint
uses: golangci/golangci-lint-action@v3
with:
version: latest
args: --no-config --disable-all --enable gosimple --enable govet --enable unused --enable deadcode --timeout 3m
- name: Build
run: go build -v ./...
- name: Run tests
run: go test -mod=vendor -v --race -covermode=atomic -coverprofile cover.out ./...
+73
View File
@@ -0,0 +1,73 @@
name: Build and Release
on:
push:
tags:
- '*'
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Update Ubuntu
run: |
sudo apt-get update
sudo apt-get upgrade -y
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: 1.22.x
cache-dependency-path: "go.sum"
- name: Checkout cocos
uses: actions/checkout@v4
with:
repository: 'ultravioletrs/cocos'
path: cocos
- name: Checkout buildroot
uses: actions/checkout@v4
with:
repository: 'buildroot/buildroot'
path: buildroot
- name: Build
run: |
cd buildroot
make BR2_EXTERNAL=../cocos/hal/linux cocos_defconfig
make
- name: Create Release
id: create_release
uses: actions/create-release@latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
tag_name: ${{ github.ref }}
release_name: Release ${{ github.ref }}
draft: false
prerelease: false
- name: Upload Release Asset
id: upload-release-kernel
uses: actions/upload-release-asset@latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: ./buildroot/output/images/bzImage
asset_name: bzImage
asset_content_type: application/octet-stream
- name: Upload Release Asset
id: upload-release-rootfs
uses: actions/upload-release-asset@latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: ./buildroot/output/images/rootfs.cpio.gz
asset_name: rootfs.cpio.gz
asset_content_type: application/gzip
+34
View File
@@ -0,0 +1,34 @@
name: CI
on:
push:
branches:
- main
pull_request:
branches:
- main
jobs:
ci:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: 1.22.x
- name: golangci-lint
uses: golangci/golangci-lint-action@v6
with:
version: v1.59.1
- name: Build
run: |
make
- name: Run tests
run: go test -v --race -covermode=atomic -coverprofile cover.out ./...
+13
View File
@@ -1 +1,14 @@
build
build
cmd/manager/img
cmd/manager/iso
cmd/manager/tmp
.cov
*.pem
dist/
result.bin
*.spec
+80
View File
@@ -0,0 +1,80 @@
run:
timeout: 3m
issues:
max-issues-per-linter: 10
max-same-issues: 10
linters-settings:
importas:
no-unaliased: true
no-extra-aliases: false
alias:
- pkg: github.com/absmach/magistrala/logger
alias: mglog
gocritic:
enabled-checks:
- dupImport
- importShadow
- httpNoBody
- paramTypeCombine
- emptyStringTest
- builtinShadow
- exposedSyncMutex
disabled-checks:
- appendAssign
enabled-tags:
- diagnostic
disabled-tags:
- performance
- style
- experimental
- opinionated
stylecheck:
checks: ["-ST1000", "-ST1003", "-ST1020", "-ST1021", "-ST1022"]
goheader:
template: |-
Copyright (c) Ultraviolet
SPDX-License-Identifier: Apache-2.0
linters:
disable-all: true
enable:
- importas
- gocritic
- gosimple
- errcheck
- govet
- unused
- goconst
- godot
- godox
- ineffassign
- misspell
- stylecheck
- whitespace
- gci
- gofmt
- goimports
- loggercheck
- goheader
- asasalint
- asciicheck
- bidichk
- contextcheck
- decorder
- dogsled
- errchkjson
- errname
- execinquery
- exportloopref
- ginkgolinter
- gocheckcompilerdirectives
- gofumpt
- goprintffuncname
- importas
- makezero
- mirror
- nakedret
- dupword
@@ -1,11 +1,11 @@
# Contributing to Mainflux
# Contributing to CoCos
The following is a set of guidelines to contribute to Mainflux and its libraries, which are
hosted on the [Mainflux Organization](https://github.com/mainflux) on GitHub.
The following is a set of guidelines to contribute to CoCos and its libraries, which are
hosted on the [Ultraviolet Organization](https://github.com/ultravioletrs) on GitHub.
This project adheres to the [Contributor Covenant 1.2](http://contributor-covenant.org/version/1/2/0).
By participating, you are expected to uphold this code. Please report unacceptable behavior to
[abuse@mainflux.com](mailto:abuse@mainflux.com).
[abuse@ultraviolet.rs](mailto:abuse@ultraviolet.rs).
## Reporting issues
@@ -13,7 +13,7 @@ Reporting issues are a great way to contribute to the project. We are perpetuall
thorough bug report.
Before raising a new issue, check [our issue
list](https://github.com/mainflux/mainflux/issues) to determine if it already contains the
list](https://github.com/ultravioletrs/cocos/issues) to determine if it already contains the
problem that you are facing.
A good bug report shouldn't leave others needing to chase you for more information. Please be as detailed as possible. The following questions might serve as a template for writing a detailed
@@ -41,23 +41,23 @@ To contribute to the project, [fork](https://help.github.com/articles/fork-a-rep
clone your fork repository, and configure the remotes:
```
git clone https://github.com/<your-username>/mainflux.git
cd mainflux
git remote add upstream https://github.com/mainflux/mainflux.git
git clone https://github.com/<your-username>/cocos.git
cd cocos
git remote add upstream https://github.com/ultravioletrs/cocos.git
```
If your cloned repository is behind the upstream commits, then get the latest changes from upstream:
```
git checkout master
git pull --rebase upstream master
git pull --rebase upstream main
```
Create a new topic branch from `master` using the naming convention `MF-[issue-number]`
Create a new topic branch from `master` using the naming convention `COCOS-[issue-number]`
to help us keep track of your contribution scope:
```
git checkout -b MF-[issue-number]
git checkout -b COCOS-[issue-number]
```
Commit your changes in logical chunks. When you are ready to commit, make sure
@@ -74,13 +74,13 @@ and `user.email` git configs, you can sign your commit automatically with `git c
Locally merge (or rebase) the upstream development branch into your topic branch:
```
git pull --rebase upstream master
git pull --rebase upstream main
```
Push your topic branch up to your fork:
```
git push origin MF-[issue-number]
git push origin COCOS-[issue-number]
```
[Open a Pull Request](https://help.github.com/articles/using-pull-requests/) with a clear title
+4 -14
View File
@@ -1,6 +1,7 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
https://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
@@ -175,24 +176,13 @@
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Copyright 2015-2020 Mainflux
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
+15 -62
View File
@@ -1,81 +1,34 @@
COCOS_DOCKER_IMAGE_NAME_PREFIX ?= cocos
BUILD_DIR = build
SERVICES = computations datasets
DOCKERS = $(addprefix docker_,$(SERVICES))
DOCKERS_DEV = $(addprefix docker_dev_,$(SERVICES))
SERVICES = manager agent cli
PLATFORM_INFO = platform_info
CGO_ENABLED ?= 0
GOARCH ?= amd64
VERSION ?= $(shell git describe --abbrev=0 --tags)
VERSION ?= $(shell git describe --abbrev=0 --tags --always)
COMMIT ?= $(shell git rev-parse HEAD)
TIME ?= $(shell date +%F_%T)
define compile_service
CGO_ENABLED=$(CGO_ENABLED) GOOS=$(GOOS) GOARCH=$(GOARCH) GOARM=$(GOARM) \
go build -mod=vendor -ldflags "-s -w \
-X 'github.com/ultravioletrs/cocos/internal/http.BuildTime=$(TIME)' \
-X 'github.com/ultravioletrs/cocos/internal/http.Version=$(VERSION)' \
-X 'github.com/ultravioletrs/cocos/internal/http.Commit=$(COMMIT)'" \
go build -ldflags "-s -w \
-X 'github.com/absmach/magistrala.BuildTime=$(TIME)' \
-X 'github.com/absmach/magistrala.Version=$(VERSION)' \
-X 'github.com/absmach/magistrala.Commit=$(COMMIT)'" \
-o ${BUILD_DIR}/cocos-$(1) cmd/$(1)/main.go
endef
define make_docker
$(eval svc=$(subst docker_,,$(1)))
docker build \
--no-cache \
--build-arg SVC=$(svc) \
--build-arg GOARCH=$(GOARCH) \
--build-arg GOARM=$(GOARM) \
--build-arg VERSION=$(VERSION) \
--build-arg COMMIT=$(COMMIT) \
--build-arg TIME=$(TIME) \
--tag=$(COCOS_DOCKER_IMAGE_NAME_PREFIX)/$(svc) \
-f docker/Dockerfile .
endef
define make_docker_dev
$(eval svc=$(subst docker_dev_,,$(1)))
docker build \
--no-cache \
--build-arg SVC=$(svc) \
--tag=$(COCOS_DOCKER_IMAGE_NAME_PREFIX)/$(svc) \
-f docker/Dockerfile.dev ./build
endef
.PHONY: all $(SERVICES) $(PLATFORM_INFO)
all: $(SERVICES)
.PHONY: all $(SERVICES) dockers dockers_dev latest release
clean:
rm -rf ${BUILD_DIR}
cleandocker:
# Stops containers and removes containers, networks, volumes, and images created by up
docker-compose -f docker/docker-compose.yml down --rmi all -v --remove-orphans
ifdef pv
# Remove unused volumes
docker volume ls -f name=$(MF_DOCKER_IMAGE_NAME_PREFIX) -f dangling=true -q | xargs -r docker volume rm
endif
install:
cp ${BUILD_DIR}/* $(GOBIN)
test:
go test -mod=vendor -v -race -count 1 -tags test $(shell go list ./... | grep -v 'vendor\|cmd')
$(SERVICES):
$(call compile_service,$(@))
$(DOCKERS):
$(call make_docker,$(@),$(GOARCH))
$(PLATFORM_INFO):
$(MAKE) -C ./scripts/platform_info
$(DOCKERS_DEV):
$(call make_docker_dev,$(@))
protoc:
protoc -I. --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative agent/agent.proto
protoc -I. --go_out=./pkg --go_opt=paths=source_relative --go-grpc_out=./pkg --go-grpc_opt=paths=source_relative manager/manager.proto
dockers: $(DOCKERS)
dockers_dev: $(DOCKERS_DEV)
run:
docker-compose -f docker/docker-compose.yml up
mocks:
go generate ./...
+30 -37
View File
@@ -7,62 +7,55 @@ based on the [Confidential Computing][cc] and [Trusted Execution Environments (T
</p>
With Cocos AI it becomes possible to run AI/ML workloads on combined datasets from multiple organizations
while guaranteeing the privacy and security of the data and the algorithms.
while guaranteeing the privacy and security of the data and the algorithm.
Data is always encrypted, protected by hardware secure enclaves (Trusted Execution Environments),
attested via secure remote attestation protocols, and invisible to cloud processors or any other
3rd party to which computation is offloaded.
## Install
The following prerequisites are needed to run Cocos:
## Features
- [Docker](https://docs.docker.com/install/)
- [Docker compose](https://docs.docker.com/compose/install/)
Cocos AI is implementing the following features:
### Build Docker Images
Currenty, there is no Docker registry, so Cocos images must be built by hand:
```
make dockers
```
For this you might be needing to setup DNS servers in your `/etc/resolf.conf` as explained
[here](https://github.com/docker/cli/issues/2618) (i.e. add Google's `nameserver 8.8.8.8`).
### Run Composition
Once the images are built (`docker images` command should show you `cocos/computations` and `cocos/datasets`),
composition can be run:
```bash
docker-compose -f docker/docker-compose.yml up
```
This will bring up the Cocos docker services and interconnect them. This command can also be executed using the project's included Makefile:
```bash
make run
```
- TEE enablement, deployment and monitoring (secure VM manager)
- HAL for TEEs based on hardened Linux kernel, secure bootloader and custom-tailored embedded rootfs for minimal TCB
- In-enclave agent, netowrking controller and other system software
- Encrypted asynchronous data transfer and result delivery
- API for programmable platform manipulation
- HW and SW supported attestation with verification tools
- CLI for system interaction
## Usage
The quickest way to start using Cocos is via the CLI. The latest version can be downloaded from the [official releases page][rel].
It can also be built and used from the project's root directory:
Clone the repo and create binaries:
```bash
make cli
./build/cocos-cli version
git clone git@github.com:ultravioletrs/cocos.git
make
```
Additional details on using the CLI can be found in the [CLI documentation](https://docs.mainflux.io/cli).
This will create 3 binaries:
```bash
ls build/
# cocos-agent cocos-cli cocos-manager
```
- Manager can be deployed on the AMD SEV-SNP host
- Agent can be built into [EOS][eos]-based HAL
- CLI can be used to communicate to remote Agent.
## Documentation
Official documentation is hosted at [Cocos official docs page][docs]. Documentation is auto-generated, checkout the instructions on [official docs repository](https://github.com/cocos/docs).
Project documentation is hosted at [Cocos AI official docs page][docs].
Documentation is generated from the [docs repository](https://github.com/ultravioletrs/docs).
## License
Cocos AI is a proprietary product created by Ultraviolet company.
Cocos AI is published under permissive open-source [Apache-2.0](LICENSE) license.
[cc]: https://confidentialcomputing.io/white-papers-reports/
[cocos]: https://cocos.ai/
[rel]: https://github.com/ultraviolet/cocos/releases
[rel]: https://github.com/ultravioletrs/cocos/releases
[tee]: https://en.wikipedia.org/wiki/Trusted_execution_environment
[docs]: https://docs.cocos.ultraviolet.rs
[cli]: https://docs.cocos.ultraviolet.rs/cli
[eos]: https://github.com/ultravioletrs/eos
+39
View File
@@ -0,0 +1,39 @@
# Agent
Agent service provides a barebones HTTP and gRPC API and Service interface implementation for the development of the agent service.
## Configuration
The service is configured using the environment variables from the following table. Note that any unset variables will be replaced with their default values.
| Variable | Description | Default |
| ----------------------------- | ------------------------------------------------------ | ------------------------------ |
| AGENT_LOG_LEVEL | Log level for agent service (debug, info, warn, error) | info |
| AGENT_GRPC_HOST | Agent service gRPC host | "" |
| AGENT_GRPC_PORT | Agent service gRPC port | 7002 |
| AGENT_GRPC_SERVER_CERT | Path to gRPC server certificate in pem format | "" |
| AGENT_GRPC_SERVER_KEY | Path to gRPC server key in pem format | "" |
| AGENT_GRPC_SERVER_CA_CERTS | Path to gRPC server CA certificate | "" |
| AGENT_GRPC_CLIENT_CA_CERTS | Path to gRPC client CA certificate | "" |
## Deployment
To start the service outside of the container, execute the following shell script:
```bash
# Download the latest version of the service
git clone git@github.com:ultravioletrs/cocos.git
cd cocos
# Compile the service
make agent
# Run the service
./build/cocos-agent
```
## Usage
For more information about service capabilities and its usage, please check out the [README documentation](../README.md).
+586
View File
@@ -0,0 +1,586 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.34.2
// protoc v4.25.3
// source: agent/agent.proto
package agent
import (
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
)
const (
// Verify that this generated code is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
// Verify that runtime/protoimpl is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)
type AlgoRequest struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Algorithm []byte `protobuf:"bytes,1,opt,name=algorithm,proto3" json:"algorithm,omitempty"`
Requirements []byte `protobuf:"bytes,2,opt,name=requirements,proto3" json:"requirements,omitempty"`
}
func (x *AlgoRequest) Reset() {
*x = AlgoRequest{}
if protoimpl.UnsafeEnabled {
mi := &file_agent_agent_proto_msgTypes[0]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *AlgoRequest) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*AlgoRequest) ProtoMessage() {}
func (x *AlgoRequest) ProtoReflect() protoreflect.Message {
mi := &file_agent_agent_proto_msgTypes[0]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use AlgoRequest.ProtoReflect.Descriptor instead.
func (*AlgoRequest) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{0}
}
func (x *AlgoRequest) GetAlgorithm() []byte {
if x != nil {
return x.Algorithm
}
return nil
}
func (x *AlgoRequest) GetRequirements() []byte {
if x != nil {
return x.Requirements
}
return nil
}
type AlgoResponse struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
}
func (x *AlgoResponse) Reset() {
*x = AlgoResponse{}
if protoimpl.UnsafeEnabled {
mi := &file_agent_agent_proto_msgTypes[1]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *AlgoResponse) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*AlgoResponse) ProtoMessage() {}
func (x *AlgoResponse) ProtoReflect() protoreflect.Message {
mi := &file_agent_agent_proto_msgTypes[1]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use AlgoResponse.ProtoReflect.Descriptor instead.
func (*AlgoResponse) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{1}
}
type DataRequest struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Dataset []byte `protobuf:"bytes,1,opt,name=dataset,proto3" json:"dataset,omitempty"`
}
func (x *DataRequest) Reset() {
*x = DataRequest{}
if protoimpl.UnsafeEnabled {
mi := &file_agent_agent_proto_msgTypes[2]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *DataRequest) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*DataRequest) ProtoMessage() {}
func (x *DataRequest) ProtoReflect() protoreflect.Message {
mi := &file_agent_agent_proto_msgTypes[2]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use DataRequest.ProtoReflect.Descriptor instead.
func (*DataRequest) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{2}
}
func (x *DataRequest) GetDataset() []byte {
if x != nil {
return x.Dataset
}
return nil
}
type DataResponse struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
}
func (x *DataResponse) Reset() {
*x = DataResponse{}
if protoimpl.UnsafeEnabled {
mi := &file_agent_agent_proto_msgTypes[3]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *DataResponse) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*DataResponse) ProtoMessage() {}
func (x *DataResponse) ProtoReflect() protoreflect.Message {
mi := &file_agent_agent_proto_msgTypes[3]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use DataResponse.ProtoReflect.Descriptor instead.
func (*DataResponse) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{3}
}
type ResultRequest struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
}
func (x *ResultRequest) Reset() {
*x = ResultRequest{}
if protoimpl.UnsafeEnabled {
mi := &file_agent_agent_proto_msgTypes[4]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *ResultRequest) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*ResultRequest) ProtoMessage() {}
func (x *ResultRequest) ProtoReflect() protoreflect.Message {
mi := &file_agent_agent_proto_msgTypes[4]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use ResultRequest.ProtoReflect.Descriptor instead.
func (*ResultRequest) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{4}
}
type ResultResponse struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
File []byte `protobuf:"bytes,1,opt,name=file,proto3" json:"file,omitempty"`
}
func (x *ResultResponse) Reset() {
*x = ResultResponse{}
if protoimpl.UnsafeEnabled {
mi := &file_agent_agent_proto_msgTypes[5]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *ResultResponse) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*ResultResponse) ProtoMessage() {}
func (x *ResultResponse) ProtoReflect() protoreflect.Message {
mi := &file_agent_agent_proto_msgTypes[5]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use ResultResponse.ProtoReflect.Descriptor instead.
func (*ResultResponse) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{5}
}
func (x *ResultResponse) GetFile() []byte {
if x != nil {
return x.File
}
return nil
}
type AttestationRequest struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
ReportData []byte `protobuf:"bytes,1,opt,name=report_data,json=reportData,proto3" json:"report_data,omitempty"` // Should be of length 64.
}
func (x *AttestationRequest) Reset() {
*x = AttestationRequest{}
if protoimpl.UnsafeEnabled {
mi := &file_agent_agent_proto_msgTypes[6]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *AttestationRequest) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*AttestationRequest) ProtoMessage() {}
func (x *AttestationRequest) ProtoReflect() protoreflect.Message {
mi := &file_agent_agent_proto_msgTypes[6]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use AttestationRequest.ProtoReflect.Descriptor instead.
func (*AttestationRequest) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{6}
}
func (x *AttestationRequest) GetReportData() []byte {
if x != nil {
return x.ReportData
}
return nil
}
type AttestationResponse struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
File []byte `protobuf:"bytes,1,opt,name=file,proto3" json:"file,omitempty"`
}
func (x *AttestationResponse) Reset() {
*x = AttestationResponse{}
if protoimpl.UnsafeEnabled {
mi := &file_agent_agent_proto_msgTypes[7]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *AttestationResponse) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*AttestationResponse) ProtoMessage() {}
func (x *AttestationResponse) ProtoReflect() protoreflect.Message {
mi := &file_agent_agent_proto_msgTypes[7]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use AttestationResponse.ProtoReflect.Descriptor instead.
func (*AttestationResponse) Descriptor() ([]byte, []int) {
return file_agent_agent_proto_rawDescGZIP(), []int{7}
}
func (x *AttestationResponse) GetFile() []byte {
if x != nil {
return x.File
}
return nil
}
var File_agent_agent_proto protoreflect.FileDescriptor
var file_agent_agent_proto_rawDesc = []byte{
0x0a, 0x11, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2f, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x70, 0x72,
0x6f, 0x74, 0x6f, 0x12, 0x05, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x22, 0x4f, 0x0a, 0x0b, 0x41, 0x6c,
0x67, 0x6f, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x1c, 0x0a, 0x09, 0x61, 0x6c, 0x67,
0x6f, 0x72, 0x69, 0x74, 0x68, 0x6d, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x09, 0x61, 0x6c,
0x67, 0x6f, 0x72, 0x69, 0x74, 0x68, 0x6d, 0x12, 0x22, 0x0a, 0x0c, 0x72, 0x65, 0x71, 0x75, 0x69,
0x72, 0x65, 0x6d, 0x65, 0x6e, 0x74, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x0c, 0x72,
0x65, 0x71, 0x75, 0x69, 0x72, 0x65, 0x6d, 0x65, 0x6e, 0x74, 0x73, 0x22, 0x0e, 0x0a, 0x0c, 0x41,
0x6c, 0x67, 0x6f, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x27, 0x0a, 0x0b, 0x44,
0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x18, 0x0a, 0x07, 0x64, 0x61,
0x74, 0x61, 0x73, 0x65, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x07, 0x64, 0x61, 0x74,
0x61, 0x73, 0x65, 0x74, 0x22, 0x0e, 0x0a, 0x0c, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x73, 0x70,
0x6f, 0x6e, 0x73, 0x65, 0x22, 0x0f, 0x0a, 0x0d, 0x52, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x52, 0x65,
0x71, 0x75, 0x65, 0x73, 0x74, 0x22, 0x24, 0x0a, 0x0e, 0x52, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x52,
0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x66, 0x69, 0x6c, 0x65, 0x18,
0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x04, 0x66, 0x69, 0x6c, 0x65, 0x22, 0x35, 0x0a, 0x12, 0x41,
0x74, 0x74, 0x65, 0x73, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73,
0x74, 0x12, 0x1f, 0x0a, 0x0b, 0x72, 0x65, 0x70, 0x6f, 0x72, 0x74, 0x5f, 0x64, 0x61, 0x74, 0x61,
0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x0a, 0x72, 0x65, 0x70, 0x6f, 0x72, 0x74, 0x44, 0x61,
0x74, 0x61, 0x22, 0x29, 0x0a, 0x13, 0x41, 0x74, 0x74, 0x65, 0x73, 0x74, 0x61, 0x74, 0x69, 0x6f,
0x6e, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x66, 0x69, 0x6c,
0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x04, 0x66, 0x69, 0x6c, 0x65, 0x32, 0xfb, 0x01,
0x0a, 0x0c, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x33,
0x0a, 0x04, 0x41, 0x6c, 0x67, 0x6f, 0x12, 0x12, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x41,
0x6c, 0x67, 0x6f, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x13, 0x2e, 0x61, 0x67, 0x65,
0x6e, 0x74, 0x2e, 0x41, 0x6c, 0x67, 0x6f, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22,
0x00, 0x28, 0x01, 0x12, 0x33, 0x0a, 0x04, 0x44, 0x61, 0x74, 0x61, 0x12, 0x12, 0x2e, 0x61, 0x67,
0x65, 0x6e, 0x74, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a,
0x13, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x73, 0x70,
0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x28, 0x01, 0x12, 0x39, 0x0a, 0x06, 0x52, 0x65, 0x73, 0x75,
0x6c, 0x74, 0x12, 0x14, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x52, 0x65, 0x73, 0x75, 0x6c,
0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x15, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74,
0x2e, 0x52, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22,
0x00, 0x30, 0x01, 0x12, 0x46, 0x0a, 0x0b, 0x41, 0x74, 0x74, 0x65, 0x73, 0x74, 0x61, 0x74, 0x69,
0x6f, 0x6e, 0x12, 0x19, 0x2e, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x41, 0x74, 0x74, 0x65, 0x73,
0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1a, 0x2e,
0x61, 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x41, 0x74, 0x74, 0x65, 0x73, 0x74, 0x61, 0x74, 0x69, 0x6f,
0x6e, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x42, 0x09, 0x5a, 0x07, 0x2e,
0x2f, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
}
var (
file_agent_agent_proto_rawDescOnce sync.Once
file_agent_agent_proto_rawDescData = file_agent_agent_proto_rawDesc
)
func file_agent_agent_proto_rawDescGZIP() []byte {
file_agent_agent_proto_rawDescOnce.Do(func() {
file_agent_agent_proto_rawDescData = protoimpl.X.CompressGZIP(file_agent_agent_proto_rawDescData)
})
return file_agent_agent_proto_rawDescData
}
var file_agent_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 8)
var file_agent_agent_proto_goTypes = []any{
(*AlgoRequest)(nil), // 0: agent.AlgoRequest
(*AlgoResponse)(nil), // 1: agent.AlgoResponse
(*DataRequest)(nil), // 2: agent.DataRequest
(*DataResponse)(nil), // 3: agent.DataResponse
(*ResultRequest)(nil), // 4: agent.ResultRequest
(*ResultResponse)(nil), // 5: agent.ResultResponse
(*AttestationRequest)(nil), // 6: agent.AttestationRequest
(*AttestationResponse)(nil), // 7: agent.AttestationResponse
}
var file_agent_agent_proto_depIdxs = []int32{
0, // 0: agent.AgentService.Algo:input_type -> agent.AlgoRequest
2, // 1: agent.AgentService.Data:input_type -> agent.DataRequest
4, // 2: agent.AgentService.Result:input_type -> agent.ResultRequest
6, // 3: agent.AgentService.Attestation:input_type -> agent.AttestationRequest
1, // 4: agent.AgentService.Algo:output_type -> agent.AlgoResponse
3, // 5: agent.AgentService.Data:output_type -> agent.DataResponse
5, // 6: agent.AgentService.Result:output_type -> agent.ResultResponse
7, // 7: agent.AgentService.Attestation:output_type -> agent.AttestationResponse
4, // [4:8] is the sub-list for method output_type
0, // [0:4] is the sub-list for method input_type
0, // [0:0] is the sub-list for extension type_name
0, // [0:0] is the sub-list for extension extendee
0, // [0:0] is the sub-list for field type_name
}
func init() { file_agent_agent_proto_init() }
func file_agent_agent_proto_init() {
if File_agent_agent_proto != nil {
return
}
if !protoimpl.UnsafeEnabled {
file_agent_agent_proto_msgTypes[0].Exporter = func(v any, i int) any {
switch v := v.(*AlgoRequest); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_agent_agent_proto_msgTypes[1].Exporter = func(v any, i int) any {
switch v := v.(*AlgoResponse); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_agent_agent_proto_msgTypes[2].Exporter = func(v any, i int) any {
switch v := v.(*DataRequest); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_agent_agent_proto_msgTypes[3].Exporter = func(v any, i int) any {
switch v := v.(*DataResponse); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_agent_agent_proto_msgTypes[4].Exporter = func(v any, i int) any {
switch v := v.(*ResultRequest); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_agent_agent_proto_msgTypes[5].Exporter = func(v any, i int) any {
switch v := v.(*ResultResponse); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_agent_agent_proto_msgTypes[6].Exporter = func(v any, i int) any {
switch v := v.(*AttestationRequest); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_agent_agent_proto_msgTypes[7].Exporter = func(v any, i int) any {
switch v := v.(*AttestationResponse); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
}
type x struct{}
out := protoimpl.TypeBuilder{
File: protoimpl.DescBuilder{
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: file_agent_agent_proto_rawDesc,
NumEnums: 0,
NumMessages: 8,
NumExtensions: 0,
NumServices: 1,
},
GoTypes: file_agent_agent_proto_goTypes,
DependencyIndexes: file_agent_agent_proto_depIdxs,
MessageInfos: file_agent_agent_proto_msgTypes,
}.Build()
File_agent_agent_proto = out.File
file_agent_agent_proto_rawDesc = nil
file_agent_agent_proto_goTypes = nil
file_agent_agent_proto_depIdxs = nil
}
+43
View File
@@ -0,0 +1,43 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
syntax = "proto3";
package agent;
option go_package = "./agent";
service AgentService {
rpc Algo(stream AlgoRequest) returns (AlgoResponse) {}
rpc Data(stream DataRequest) returns (DataResponse) {}
rpc Result(ResultRequest) returns (stream ResultResponse) {}
rpc Attestation(AttestationRequest) returns (AttestationResponse) {}
}
message AlgoRequest {
bytes algorithm = 1;
bytes requirements = 2;
}
message AlgoResponse {}
message DataRequest {
bytes dataset = 1;
}
message DataResponse {}
message ResultRequest {
}
message ResultResponse {
bytes file = 1;
}
message AttestationRequest {
bytes report_data = 1; // Should be of length 64.
}
message AttestationResponse {
bytes file = 1;
}
+323
View File
@@ -0,0 +1,323 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
// versions:
// - protoc-gen-go-grpc v1.4.0
// - protoc v4.25.3
// source: agent/agent.proto
package agent
import (
context "context"
grpc "google.golang.org/grpc"
codes "google.golang.org/grpc/codes"
status "google.golang.org/grpc/status"
)
// This is a compile-time assertion to ensure that this generated file
// is compatible with the grpc package it is being compiled against.
// Requires gRPC-Go v1.62.0 or later.
const _ = grpc.SupportPackageIsVersion8
const (
AgentService_Algo_FullMethodName = "/agent.AgentService/Algo"
AgentService_Data_FullMethodName = "/agent.AgentService/Data"
AgentService_Result_FullMethodName = "/agent.AgentService/Result"
AgentService_Attestation_FullMethodName = "/agent.AgentService/Attestation"
)
// AgentServiceClient is the client API for AgentService service.
//
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
type AgentServiceClient interface {
Algo(ctx context.Context, opts ...grpc.CallOption) (AgentService_AlgoClient, error)
Data(ctx context.Context, opts ...grpc.CallOption) (AgentService_DataClient, error)
Result(ctx context.Context, in *ResultRequest, opts ...grpc.CallOption) (AgentService_ResultClient, error)
Attestation(ctx context.Context, in *AttestationRequest, opts ...grpc.CallOption) (*AttestationResponse, error)
}
type agentServiceClient struct {
cc grpc.ClientConnInterface
}
func NewAgentServiceClient(cc grpc.ClientConnInterface) AgentServiceClient {
return &agentServiceClient{cc}
}
func (c *agentServiceClient) Algo(ctx context.Context, opts ...grpc.CallOption) (AgentService_AlgoClient, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
stream, err := c.cc.NewStream(ctx, &AgentService_ServiceDesc.Streams[0], AgentService_Algo_FullMethodName, cOpts...)
if err != nil {
return nil, err
}
x := &agentServiceAlgoClient{ClientStream: stream}
return x, nil
}
type AgentService_AlgoClient interface {
Send(*AlgoRequest) error
CloseAndRecv() (*AlgoResponse, error)
grpc.ClientStream
}
type agentServiceAlgoClient struct {
grpc.ClientStream
}
func (x *agentServiceAlgoClient) Send(m *AlgoRequest) error {
return x.ClientStream.SendMsg(m)
}
func (x *agentServiceAlgoClient) CloseAndRecv() (*AlgoResponse, error) {
if err := x.ClientStream.CloseSend(); err != nil {
return nil, err
}
m := new(AlgoResponse)
if err := x.ClientStream.RecvMsg(m); err != nil {
return nil, err
}
return m, nil
}
func (c *agentServiceClient) Data(ctx context.Context, opts ...grpc.CallOption) (AgentService_DataClient, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
stream, err := c.cc.NewStream(ctx, &AgentService_ServiceDesc.Streams[1], AgentService_Data_FullMethodName, cOpts...)
if err != nil {
return nil, err
}
x := &agentServiceDataClient{ClientStream: stream}
return x, nil
}
type AgentService_DataClient interface {
Send(*DataRequest) error
CloseAndRecv() (*DataResponse, error)
grpc.ClientStream
}
type agentServiceDataClient struct {
grpc.ClientStream
}
func (x *agentServiceDataClient) Send(m *DataRequest) error {
return x.ClientStream.SendMsg(m)
}
func (x *agentServiceDataClient) CloseAndRecv() (*DataResponse, error) {
if err := x.ClientStream.CloseSend(); err != nil {
return nil, err
}
m := new(DataResponse)
if err := x.ClientStream.RecvMsg(m); err != nil {
return nil, err
}
return m, nil
}
func (c *agentServiceClient) Result(ctx context.Context, in *ResultRequest, opts ...grpc.CallOption) (AgentService_ResultClient, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
stream, err := c.cc.NewStream(ctx, &AgentService_ServiceDesc.Streams[2], AgentService_Result_FullMethodName, cOpts...)
if err != nil {
return nil, err
}
x := &agentServiceResultClient{ClientStream: stream}
if err := x.ClientStream.SendMsg(in); err != nil {
return nil, err
}
if err := x.ClientStream.CloseSend(); err != nil {
return nil, err
}
return x, nil
}
type AgentService_ResultClient interface {
Recv() (*ResultResponse, error)
grpc.ClientStream
}
type agentServiceResultClient struct {
grpc.ClientStream
}
func (x *agentServiceResultClient) Recv() (*ResultResponse, error) {
m := new(ResultResponse)
if err := x.ClientStream.RecvMsg(m); err != nil {
return nil, err
}
return m, nil
}
func (c *agentServiceClient) Attestation(ctx context.Context, in *AttestationRequest, opts ...grpc.CallOption) (*AttestationResponse, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
out := new(AttestationResponse)
err := c.cc.Invoke(ctx, AgentService_Attestation_FullMethodName, in, out, cOpts...)
if err != nil {
return nil, err
}
return out, nil
}
// AgentServiceServer is the server API for AgentService service.
// All implementations must embed UnimplementedAgentServiceServer
// for forward compatibility
type AgentServiceServer interface {
Algo(AgentService_AlgoServer) error
Data(AgentService_DataServer) error
Result(*ResultRequest, AgentService_ResultServer) error
Attestation(context.Context, *AttestationRequest) (*AttestationResponse, error)
mustEmbedUnimplementedAgentServiceServer()
}
// UnimplementedAgentServiceServer must be embedded to have forward compatible implementations.
type UnimplementedAgentServiceServer struct {
}
func (UnimplementedAgentServiceServer) Algo(AgentService_AlgoServer) error {
return status.Errorf(codes.Unimplemented, "method Algo not implemented")
}
func (UnimplementedAgentServiceServer) Data(AgentService_DataServer) error {
return status.Errorf(codes.Unimplemented, "method Data not implemented")
}
func (UnimplementedAgentServiceServer) Result(*ResultRequest, AgentService_ResultServer) error {
return status.Errorf(codes.Unimplemented, "method Result not implemented")
}
func (UnimplementedAgentServiceServer) Attestation(context.Context, *AttestationRequest) (*AttestationResponse, error) {
return nil, status.Errorf(codes.Unimplemented, "method Attestation not implemented")
}
func (UnimplementedAgentServiceServer) mustEmbedUnimplementedAgentServiceServer() {}
// UnsafeAgentServiceServer may be embedded to opt out of forward compatibility for this service.
// Use of this interface is not recommended, as added methods to AgentServiceServer will
// result in compilation errors.
type UnsafeAgentServiceServer interface {
mustEmbedUnimplementedAgentServiceServer()
}
func RegisterAgentServiceServer(s grpc.ServiceRegistrar, srv AgentServiceServer) {
s.RegisterService(&AgentService_ServiceDesc, srv)
}
func _AgentService_Algo_Handler(srv interface{}, stream grpc.ServerStream) error {
return srv.(AgentServiceServer).Algo(&agentServiceAlgoServer{ServerStream: stream})
}
type AgentService_AlgoServer interface {
SendAndClose(*AlgoResponse) error
Recv() (*AlgoRequest, error)
grpc.ServerStream
}
type agentServiceAlgoServer struct {
grpc.ServerStream
}
func (x *agentServiceAlgoServer) SendAndClose(m *AlgoResponse) error {
return x.ServerStream.SendMsg(m)
}
func (x *agentServiceAlgoServer) Recv() (*AlgoRequest, error) {
m := new(AlgoRequest)
if err := x.ServerStream.RecvMsg(m); err != nil {
return nil, err
}
return m, nil
}
func _AgentService_Data_Handler(srv interface{}, stream grpc.ServerStream) error {
return srv.(AgentServiceServer).Data(&agentServiceDataServer{ServerStream: stream})
}
type AgentService_DataServer interface {
SendAndClose(*DataResponse) error
Recv() (*DataRequest, error)
grpc.ServerStream
}
type agentServiceDataServer struct {
grpc.ServerStream
}
func (x *agentServiceDataServer) SendAndClose(m *DataResponse) error {
return x.ServerStream.SendMsg(m)
}
func (x *agentServiceDataServer) Recv() (*DataRequest, error) {
m := new(DataRequest)
if err := x.ServerStream.RecvMsg(m); err != nil {
return nil, err
}
return m, nil
}
func _AgentService_Result_Handler(srv interface{}, stream grpc.ServerStream) error {
m := new(ResultRequest)
if err := stream.RecvMsg(m); err != nil {
return err
}
return srv.(AgentServiceServer).Result(m, &agentServiceResultServer{ServerStream: stream})
}
type AgentService_ResultServer interface {
Send(*ResultResponse) error
grpc.ServerStream
}
type agentServiceResultServer struct {
grpc.ServerStream
}
func (x *agentServiceResultServer) Send(m *ResultResponse) error {
return x.ServerStream.SendMsg(m)
}
func _AgentService_Attestation_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(AttestationRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(AgentServiceServer).Attestation(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: AgentService_Attestation_FullMethodName,
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(AgentServiceServer).Attestation(ctx, req.(*AttestationRequest))
}
return interceptor(ctx, in, info, handler)
}
// AgentService_ServiceDesc is the grpc.ServiceDesc for AgentService service.
// It's only intended for direct use with grpc.RegisterService,
// and not to be introspected or modified (even as a copy)
var AgentService_ServiceDesc = grpc.ServiceDesc{
ServiceName: "agent.AgentService",
HandlerType: (*AgentServiceServer)(nil),
Methods: []grpc.MethodDesc{
{
MethodName: "Attestation",
Handler: _AgentService_Attestation_Handler,
},
},
Streams: []grpc.StreamDesc{
{
StreamName: "Algo",
Handler: _AgentService_Algo_Handler,
ClientStreams: true,
},
{
StreamName: "Data",
Handler: _AgentService_Data_Handler,
ClientStreams: true,
},
{
StreamName: "Result",
Handler: _AgentService_Result_Handler,
ServerStreams: true,
},
},
Metadata: "agent/agent.proto",
}
+34
View File
@@ -0,0 +1,34 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package algorithm
import (
"context"
"google.golang.org/grpc/metadata"
)
type AlgorithType string
const (
AlgoTypeBin AlgorithType = "bin"
AlgoTypePython AlgorithType = "python"
AlgoTypeKey = "algo_type"
)
func AlgorithmTypeToContext(ctx context.Context, algoType string) context.Context {
return metadata.AppendToOutgoingContext(ctx, AlgoTypeKey, algoType)
}
func AlgorithmTypeFromContext(ctx context.Context) string {
return metadata.ValueFromIncomingContext(ctx, AlgoTypeKey)[0]
}
// Algorithm is an interface that specifies the API for an algorithm.
type Algorithm interface {
// Run executes the algorithm and returns the result.
Run() ([]byte, error)
// Add dataset to algorithm.
AddDataset(dataset string)
}
+82
View File
@@ -0,0 +1,82 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package binary
import (
"fmt"
"io"
"log/slog"
"os"
"os/exec"
"github.com/ultravioletrs/cocos/agent/algorithm"
"github.com/ultravioletrs/cocos/agent/events"
"github.com/ultravioletrs/cocos/pkg/socket"
)
const socketPath = "unix_socket"
var _ algorithm.Algorithm = (*binary)(nil)
type binary struct {
algoFile string
datasets []string
logger *slog.Logger
stderr io.Writer
stdout io.Writer
}
func New(logger *slog.Logger, eventsSvc events.Service, algoFile string) algorithm.Algorithm {
return &binary{
algoFile: algoFile,
logger: logger,
stderr: &algorithm.Stderr{Logger: logger, EventSvc: eventsSvc},
stdout: &algorithm.Stdout{Logger: logger},
}
}
func (b *binary) AddDataset(dataset string) {
b.datasets = append(b.datasets, dataset)
}
func (b *binary) Run() ([]byte, error) {
defer os.Remove(b.algoFile)
defer func() {
for _, file := range b.datasets {
os.Remove(file)
}
}()
listener, err := socket.StartUnixSocketServer(socketPath)
if err != nil {
return nil, fmt.Errorf("error creating stdout pipe: %v", err)
}
defer listener.Close()
// Create channels for received data and errors
dataChannel := make(chan []byte)
errorChannel := make(chan error)
var result []byte
go socket.AcceptConnection(listener, dataChannel, errorChannel)
args := append([]string{socketPath}, b.datasets...)
cmd := exec.Command(b.algoFile, args...)
cmd.Stderr = b.stderr
cmd.Stdout = b.stdout
if err := cmd.Start(); err != nil {
return nil, fmt.Errorf("error starting algorithm: %v", err)
}
if err := cmd.Wait(); err != nil {
return nil, fmt.Errorf("algorithm execution error: %v", err)
}
select {
case result = <-dataChannel:
return result, nil
case err = <-errorChannel:
return nil, fmt.Errorf("error receiving data: %v", err)
}
}
+73
View File
@@ -0,0 +1,73 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package algorithm
import (
"bytes"
"io"
"log/slog"
"github.com/ultravioletrs/cocos/agent/events"
)
var (
_ io.Writer = &Stdout{}
_ io.Writer = &Stderr{}
)
const bufSize = 1024
type Stdout struct {
Logger *slog.Logger
}
// Write implements io.Writer.
func (s *Stdout) Write(p []byte) (n int, err error) {
inBuf := bytes.NewBuffer(p)
buf := make([]byte, bufSize)
for {
n, err := inBuf.Read(buf)
if err != nil {
if err == io.EOF {
break
}
return len(p) - inBuf.Len(), err
}
s.Logger.Debug(string(buf[:n]))
}
return len(p), nil
}
type Stderr struct {
Logger *slog.Logger
EventSvc events.Service
}
// Write implements io.Writer.
func (s *Stderr) Write(p []byte) (n int, err error) {
inBuf := bytes.NewBuffer(p)
buf := make([]byte, bufSize)
for {
n, err := inBuf.Read(buf)
if err != nil {
if err == io.EOF {
break
}
return len(p) - inBuf.Len(), err
}
s.Logger.Error(string(buf[:n]))
}
if err := s.EventSvc.SendEvent("algorithm-run", "error", nil); err != nil {
return len(p), err
}
return len(p), nil
}
+85
View File
@@ -0,0 +1,85 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package algorithm
import (
"strings"
"testing"
mglog "github.com/absmach/magistrala/logger"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"github.com/ultravioletrs/cocos/agent/events/mocks"
)
func TestStdoutWrite(t *testing.T) {
tests := []struct {
name string
input string
expected []string
}{
{
name: "Single line",
input: "Hello, World!",
expected: []string{"Hello, World!"},
},
{
name: "Multiple lines",
input: "Line 1\nLine 2\nLine 3",
expected: []string{"Line 1\nLine 2\nLine 3"},
},
{
name: "Long input",
input: strings.Repeat("a", bufSize+100),
expected: []string{strings.Repeat("a", bufSize), strings.Repeat("a", 100)},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
stdout := &Stdout{Logger: mglog.NewMock()}
n, err := stdout.Write([]byte(tt.input))
assert.NoError(t, err)
assert.Equal(t, len(tt.input), n)
})
}
}
func TestStderrWrite(t *testing.T) {
tests := []struct {
name string
input string
expected []string
}{
{
name: "Single line",
input: "Error: Something went wrong",
expected: []string{"Error: Something went wrong"},
},
{
name: "Multiple lines",
input: "Error 1\nError 2\nError 3",
expected: []string{"Error 1\nError 2\nError 3"},
},
{
name: "Long input",
input: strings.Repeat("e", bufSize+100),
expected: []string{strings.Repeat("e", bufSize), strings.Repeat("e", 100)},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
mockEventService := mocks.NewService(t)
mockEventService.On("SendEvent", "algorithm-run", "error", mock.Anything).Return(nil)
stderr := &Stderr{Logger: mglog.NewMock(), EventSvc: mockEventService}
n, err := stderr.Write([]byte(tt.input))
assert.NoError(t, err)
assert.Equal(t, len(tt.input), n)
mockEventService.AssertExpectations(t)
})
}
}
+126
View File
@@ -0,0 +1,126 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package python
import (
"context"
"fmt"
"io"
"log/slog"
"os"
"os/exec"
"path/filepath"
"github.com/ultravioletrs/cocos/agent/algorithm"
"github.com/ultravioletrs/cocos/agent/events"
"github.com/ultravioletrs/cocos/pkg/socket"
"google.golang.org/grpc/metadata"
)
const (
socketPath = "unix_socket"
PyRuntime = "python3"
pyRuntimeKey = "python_runtime"
)
func PythonRunTimeToContext(ctx context.Context, runtime string) context.Context {
return metadata.AppendToOutgoingContext(ctx, pyRuntimeKey, runtime)
}
func PythonRunTimeFromContext(ctx context.Context) string {
return metadata.ValueFromIncomingContext(ctx, pyRuntimeKey)[0]
}
var _ algorithm.Algorithm = (*python)(nil)
type python struct {
algoFile string
datasets []string
logger *slog.Logger
stderr io.Writer
stdout io.Writer
runtime string
requirementsFile string
}
func New(logger *slog.Logger, eventsSvc events.Service, runtime, requirementsFile, algoFile string) algorithm.Algorithm {
p := &python{
algoFile: algoFile,
logger: logger,
stderr: &algorithm.Stderr{Logger: logger, EventSvc: eventsSvc},
stdout: &algorithm.Stdout{Logger: logger},
requirementsFile: requirementsFile,
}
if runtime != "" {
p.runtime = runtime
} else {
p.runtime = PyRuntime
}
return p
}
func (p *python) AddDataset(dataset string) {
p.datasets = append(p.datasets, dataset)
}
func (p *python) Run() ([]byte, error) {
venvPath := "venv"
createVenvCmd := exec.Command(p.runtime, "-m", "venv", venvPath)
createVenvCmd.Stderr = p.stderr
createVenvCmd.Stdout = p.stdout
if err := createVenvCmd.Run(); err != nil {
return nil, fmt.Errorf("error creating virtual environment: %v", err)
}
pythonPath := filepath.Join(venvPath, "bin", "python")
if p.requirementsFile != "" {
rcmd := exec.Command(pythonPath, "-m", "pip", "install", "-r", p.requirementsFile)
rcmd.Stderr = p.stderr
rcmd.Stdout = p.stdout
if err := rcmd.Run(); err != nil {
return nil, fmt.Errorf("error installing requirements: %v", err)
}
}
defer os.Remove(p.algoFile)
defer func() {
for _, file := range p.datasets {
os.Remove(file)
}
}()
defer os.RemoveAll(venvPath)
listener, err := socket.StartUnixSocketServer(socketPath)
if err != nil {
return nil, fmt.Errorf("error creating stdout pipe: %v", err)
}
defer listener.Close()
dataChannel := make(chan []byte)
errorChannel := make(chan error)
var result []byte
go socket.AcceptConnection(listener, dataChannel, errorChannel)
args := append([]string{p.algoFile, socketPath}, p.datasets...)
cmd := exec.Command(pythonPath, args...)
cmd.Stderr = p.stderr
cmd.Stdout = p.stdout
if err := cmd.Start(); err != nil {
return nil, fmt.Errorf("error starting algorithm: %v", err)
}
if err := cmd.Wait(); err != nil {
return nil, fmt.Errorf("algorithm execution error: %v", err)
}
select {
case result = <-dataChannel:
return result, nil
case err = <-errorChannel:
return nil, fmt.Errorf("error receiving data: %v", err)
}
}
+6
View File
@@ -0,0 +1,6 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
// Package api contains API-related concerns: endpoint definitions, middlewares
// and all resource representations.
package api
+5
View File
@@ -0,0 +1,5 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
// Package grpc contains implementation of kit service gRPC API.
package grpc
+80
View File
@@ -0,0 +1,80 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package grpc
import (
"context"
"github.com/go-kit/kit/endpoint"
"github.com/ultravioletrs/cocos/agent"
)
func algoEndpoint(svc agent.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(algoReq)
if err := req.validate(); err != nil {
return algoRes{}, err
}
algo := agent.Algorithm{Algorithm: req.Algorithm, Requirements: req.Requirements}
err := svc.Algo(ctx, algo)
if err != nil {
return algoRes{}, err
}
return algoRes{}, nil
}
}
func dataEndpoint(svc agent.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(dataReq)
if err := req.validate(); err != nil {
return dataRes{}, err
}
dataset := agent.Dataset{Dataset: req.Dataset}
err := svc.Data(ctx, dataset)
if err != nil {
return dataRes{}, err
}
return dataRes{}, nil
}
}
func resultEndpoint(svc agent.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(resultReq)
if err := req.validate(); err != nil {
return resultRes{}, err
}
file, err := svc.Result(ctx)
if err != nil {
return resultRes{}, err
}
return resultRes{File: file}, nil
}
}
func attestationEndpoint(svc agent.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(attestationReq)
if err := req.validate(); err != nil {
return attestationRes{}, err
}
file, err := svc.Attestation(ctx, req.ReportData)
if err != nil {
return attestationRes{}, err
}
return attestationRes{File: file}, nil
}
}
+74
View File
@@ -0,0 +1,74 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package grpc
import (
"context"
"github.com/ultravioletrs/cocos/agent"
"github.com/ultravioletrs/cocos/agent/auth"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
type authInterceptor struct {
auth auth.Authenticator
}
type wrappedServerStream struct {
grpc.ServerStream
ctx context.Context
}
func (s *wrappedServerStream) Context() context.Context {
return s.ctx
}
func NewAuthInterceptor(authSvc auth.Authenticator) (grpc.UnaryServerInterceptor, grpc.StreamServerInterceptor) {
ai := &authInterceptor{auth: authSvc}
return ai.AuthUnaryInterceptor(), ai.AuthStreamInterceptor()
}
func (s *authInterceptor) AuthStreamInterceptor() grpc.StreamServerInterceptor {
return func(srv interface{}, stream grpc.ServerStream, info *grpc.StreamServerInfo, handler grpc.StreamHandler) error {
switch info.FullMethod {
case agent.AgentService_Algo_FullMethodName:
if _, err := s.auth.AuthenticateUser(stream.Context(), auth.AlgorithmProviderRole); err != nil {
return status.Errorf(codes.Unauthenticated, err.Error())
}
return handler(srv, stream)
case agent.AgentService_Data_FullMethodName:
ctx, err := s.auth.AuthenticateUser(stream.Context(), auth.DataProviderRole)
if err != nil {
return status.Errorf(codes.Unauthenticated, err.Error())
}
wrapped := &wrappedServerStream{ServerStream: stream, ctx: ctx}
return handler(srv, wrapped)
case agent.AgentService_Result_FullMethodName:
ctx, err := s.auth.AuthenticateUser(stream.Context(), auth.ConsumerRole)
if err != nil {
return status.Errorf(codes.Unauthenticated, err.Error())
}
wrapped := &wrappedServerStream{ServerStream: stream, ctx: ctx}
return handler(srv, wrapped)
default:
return handler(srv, stream)
}
}
}
func (s *authInterceptor) AuthUnaryInterceptor() grpc.UnaryServerInterceptor {
return func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
switch info.FullMethod {
case agent.AgentService_Result_FullMethodName:
ctx, err := s.auth.AuthenticateUser(ctx, auth.ConsumerRole)
if err != nil {
return nil, status.Errorf(codes.Unauthenticated, err.Error())
}
return handler(ctx, req)
default:
return handler(ctx, req)
}
}
}
+152
View File
@@ -0,0 +1,152 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package grpc
import (
"context"
"testing"
"github.com/stretchr/testify/mock"
"github.com/ultravioletrs/cocos/agent"
"github.com/ultravioletrs/cocos/agent/auth"
"github.com/ultravioletrs/cocos/agent/mocks"
"google.golang.org/grpc"
"google.golang.org/grpc/metadata"
)
func TestAuthUnaryInterceptor(t *testing.T) {
authmock := new(mocks.Authenticator)
tests := []struct {
name string
authorized bool
method string
role auth.UserRole
wantErr bool
}{
{
name: "authorized result method",
authorized: true,
method: agent.AgentService_Result_FullMethodName,
role: auth.ConsumerRole,
wantErr: false,
},
{
name: "unauthorized result method",
authorized: false,
method: agent.AgentService_Result_FullMethodName,
role: auth.ConsumerRole,
wantErr: true,
},
{
name: "other method",
authorized: false,
method: "OtherMethod",
role: auth.ConsumerRole,
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
switch tt.authorized {
case true:
mockCall := authmock.On("AuthenticateUser", context.Background(), tt.role).Return(context.Background(), nil)
mockCall.Once()
case false:
mockCall := authmock.On("AuthenticateUser", context.Background(), tt.role).Return(context.Background(), auth.ErrMissingMetadata)
mockCall.Once()
}
unaryInt, _ := NewAuthInterceptor(authmock)
_, err := unaryInt(context.Background(), nil, &grpc.UnaryServerInfo{FullMethod: tt.method}, func(ctx context.Context, req interface{}) (interface{}, error) {
return nil, nil
})
if tt.wantErr && err == nil {
t.Errorf("expected error, got none")
} else if !tt.wantErr && err != nil {
t.Errorf("unexpected error: %v", err)
}
})
}
}
func TestAuthStreamInterceptor(t *testing.T) {
authmock := new(mocks.Authenticator)
tests := []struct {
name string
authorized bool
method string
role auth.UserRole
wantErr bool
}{
{
name: "authorized algo method",
authorized: true,
method: agent.AgentService_Algo_FullMethodName,
role: auth.AlgorithmProviderRole,
wantErr: false,
},
{
name: "unauthorized algo method",
authorized: false,
method: agent.AgentService_Algo_FullMethodName,
role: auth.AlgorithmProviderRole,
wantErr: true,
},
{
name: "authorized data method",
authorized: true,
method: agent.AgentService_Data_FullMethodName,
role: auth.DataProviderRole,
wantErr: false,
},
{
name: "unauthorized data method",
authorized: false,
method: agent.AgentService_Data_FullMethodName,
role: auth.DataProviderRole,
wantErr: true,
},
{
name: "other method",
authorized: false,
method: "OtherMethod",
role: auth.DataProviderRole,
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
switch tt.authorized {
case true:
mockCall := authmock.On("AuthenticateUser", mock.Anything, tt.role).Return(context.Background(), nil)
mockCall.Once()
case false:
mockCall := authmock.On("AuthenticateUser", mock.Anything, tt.role).Return(context.Background(), auth.ErrMissingMetadata)
mockCall.Once()
}
_, streamInt := NewAuthInterceptor(authmock)
err := streamInt(nil, &mockServerStream{ctx: metadata.NewIncomingContext(context.Background(), metadata.Pairs())}, &grpc.StreamServerInfo{FullMethod: tt.method}, func(srv interface{}, stream grpc.ServerStream) error {
return nil
})
if tt.wantErr && err == nil {
t.Errorf("expected error, got none")
} else if !tt.wantErr && err != nil {
t.Errorf("unexpected error: %v", err)
}
})
}
}
type mockServerStream struct {
grpc.ServerStream
ctx context.Context
}
func (m *mockServerStream) Context() context.Context {
return m.ctx
}
+45
View File
@@ -0,0 +1,45 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package grpc
import (
"errors"
)
type algoReq struct {
Algorithm []byte `protobuf:"bytes,1,opt,name=algorithm,proto3" json:"algorithm,omitempty"`
Requirements []byte
}
func (req algoReq) validate() error {
if len(req.Algorithm) == 0 {
return errors.New("algorithm binary is required")
}
return nil
}
type dataReq struct {
Dataset []byte `protobuf:"bytes,1,opt,name=dataset,proto3" json:"dataset,omitempty"`
}
func (req dataReq) validate() error {
if len(req.Dataset) == 0 {
return errors.New("dataset CSV file is required")
}
return nil
}
type resultReq struct{}
func (req resultReq) validate() error {
// No request parameters to validate, so no validation logic needed
return nil
}
type attestationReq struct {
ReportData [64]byte
}
func (req attestationReq) validate() error {
return nil
}
+15
View File
@@ -0,0 +1,15 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package grpc
type algoRes struct{}
type dataRes struct{}
type resultRes struct {
File []byte `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
}
type attestationRes struct {
File []byte
}
+184
View File
@@ -0,0 +1,184 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package grpc
import (
"bytes"
"context"
"errors"
"io"
"github.com/go-kit/kit/transport/grpc"
"github.com/ultravioletrs/cocos/agent"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
const bufferSize = 1024 * 1024
var _ agent.AgentServiceServer = (*grpcServer)(nil)
type grpcServer struct {
algo grpc.Handler
data grpc.Handler
result grpc.Handler
attestation grpc.Handler
agent.UnimplementedAgentServiceServer
}
// NewServer returns new AgentServiceServer instance.
func NewServer(svc agent.Service) agent.AgentServiceServer {
return &grpcServer{
algo: grpc.NewServer(
algoEndpoint(svc),
decodeAlgoRequest,
encodeAlgoResponse,
),
data: grpc.NewServer(
dataEndpoint(svc),
decodeDataRequest,
encodeDataResponse,
),
result: grpc.NewServer(
resultEndpoint(svc),
decodeResultRequest,
encodeResultResponse,
),
attestation: grpc.NewServer(
attestationEndpoint(svc),
decodeAttestationRequest,
encodeAttestationResponse,
),
}
}
func decodeAlgoRequest(_ context.Context, grpcReq interface{}) (interface{}, error) {
req := grpcReq.(*agent.AlgoRequest)
return algoReq{
Algorithm: req.Algorithm,
Requirements: req.Requirements,
}, nil
}
func encodeAlgoResponse(_ context.Context, response interface{}) (interface{}, error) {
return &agent.AlgoResponse{}, nil
}
func decodeDataRequest(_ context.Context, grpcReq interface{}) (interface{}, error) {
req := grpcReq.(*agent.DataRequest)
return dataReq{
Dataset: req.Dataset,
}, nil
}
func encodeDataResponse(_ context.Context, response interface{}) (interface{}, error) {
return &agent.DataResponse{}, nil
}
func decodeResultRequest(_ context.Context, grpcReq interface{}) (interface{}, error) {
return resultReq{}, nil
}
func encodeResultResponse(_ context.Context, response interface{}) (interface{}, error) {
res := response.(resultRes)
return &agent.ResultResponse{
File: res.File,
}, nil
}
func decodeAttestationRequest(_ context.Context, grpcReq interface{}) (interface{}, error) {
req := grpcReq.(*agent.AttestationRequest)
if len(req.ReportData) != agent.ReportDataSize {
return nil, errors.New("malformed report data, expect 64 bytes")
}
return attestationReq{ReportData: [agent.ReportDataSize]byte(req.ReportData)}, nil
}
func encodeAttestationResponse(_ context.Context, response interface{}) (interface{}, error) {
res := response.(attestationRes)
return &agent.AttestationResponse{
File: res.File,
}, nil
}
// Algo implements agent.AgentServiceServer.
func (s *grpcServer) Algo(stream agent.AgentService_AlgoServer) error {
var algoFile, reqFile []byte
for {
algoChunk, err := stream.Recv()
if err == io.EOF {
break
}
if err != nil {
return status.Error(codes.Internal, err.Error())
}
algoFile = append(algoFile, algoChunk.Algorithm...)
reqFile = append(reqFile, algoChunk.Requirements...)
}
_, res, err := s.algo.ServeGRPC(stream.Context(), &agent.AlgoRequest{Algorithm: algoFile, Requirements: reqFile})
if err != nil {
return err
}
ar := res.(*agent.AlgoResponse)
return stream.SendAndClose(ar)
}
// Data implements agent.AgentServiceServer.
func (s *grpcServer) Data(stream agent.AgentService_DataServer) error {
var dataFile []byte
for {
dataChunk, err := stream.Recv()
if err == io.EOF {
break
}
if err != nil {
return status.Error(codes.Internal, err.Error())
}
dataFile = append(dataFile, dataChunk.Dataset...)
}
_, res, err := s.data.ServeGRPC(stream.Context(), &agent.DataRequest{Dataset: dataFile})
if err != nil {
return err
}
ar := res.(*agent.DataResponse)
return stream.SendAndClose(ar)
}
func (s *grpcServer) Result(req *agent.ResultRequest, stream agent.AgentService_ResultServer) error {
_, res, err := s.result.ServeGRPC(stream.Context(), req)
if err != nil {
return err
}
rr := res.(*agent.ResultResponse)
reusltBuffer := bytes.NewBuffer(rr.File)
buf := make([]byte, bufferSize)
for {
n, err := reusltBuffer.Read(buf)
if err == io.EOF {
break
}
if err != nil {
return status.Error(codes.Internal, err.Error())
}
if err := stream.Send(&agent.ResultResponse{File: buf[:n]}); err != nil {
return status.Error(codes.Internal, err.Error())
}
}
return nil
}
func (s *grpcServer) Attestation(ctx context.Context, req *agent.AttestationRequest) (*agent.AttestationResponse, error) {
_, res, err := s.attestation.ServeGRPC(ctx, req)
if err != nil {
return nil, err
}
rr := res.(*agent.AttestationResponse)
return rr, nil
}
+80
View File
@@ -0,0 +1,80 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
//go:build !test
// +build !test
package api
import (
"context"
"fmt"
"log/slog"
"time"
"github.com/ultravioletrs/cocos/agent"
)
var _ agent.Service = (*loggingMiddleware)(nil)
type loggingMiddleware struct {
logger *slog.Logger
svc agent.Service
}
// LoggingMiddleware adds logging facilities to the core service.
func LoggingMiddleware(svc agent.Service, logger *slog.Logger) agent.Service {
return &loggingMiddleware{logger, svc}
}
func (lm *loggingMiddleware) Algo(ctx context.Context, algorithm agent.Algorithm) (err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method Algo took %s to complete", time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors", message))
}(time.Now())
return lm.svc.Algo(ctx, algorithm)
}
func (lm *loggingMiddleware) Data(ctx context.Context, dataset agent.Dataset) (err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method Data took %s to complete", time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors", message))
}(time.Now())
return lm.svc.Data(ctx, dataset)
}
func (lm *loggingMiddleware) Result(ctx context.Context) (response []byte, err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method Result took %s to complete", time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors", message))
}(time.Now())
return lm.svc.Result(ctx)
}
func (lm *loggingMiddleware) Attestation(ctx context.Context, reportData [agent.ReportDataSize]byte) (response []byte, err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method Attestation took %s to complete", time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors", message))
}(time.Now())
return lm.svc.Attestation(ctx, reportData)
}
+69
View File
@@ -0,0 +1,69 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
//go:build !test
// +build !test
package api
import (
"context"
"time"
"github.com/go-kit/kit/metrics"
"github.com/ultravioletrs/cocos/agent"
)
var _ agent.Service = (*metricsMiddleware)(nil)
type metricsMiddleware struct {
counter metrics.Counter
latency metrics.Histogram
svc agent.Service
}
// MetricsMiddleware instruments core service by tracking request count and
// latency.
func MetricsMiddleware(svc agent.Service, counter metrics.Counter, latency metrics.Histogram) agent.Service {
return &metricsMiddleware{
counter: counter,
latency: latency,
svc: svc,
}
}
func (ms *metricsMiddleware) Algo(ctx context.Context, algorithm agent.Algorithm) error {
defer func(begin time.Time) {
ms.counter.With("method", "algo").Add(1)
ms.latency.With("method", "algo").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.Algo(ctx, algorithm)
}
func (ms *metricsMiddleware) Data(ctx context.Context, dataset agent.Dataset) error {
defer func(begin time.Time) {
ms.counter.With("method", "data").Add(1)
ms.latency.With("method", "data").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.Data(ctx, dataset)
}
func (ms *metricsMiddleware) Result(ctx context.Context) ([]byte, error) {
defer func(begin time.Time) {
ms.counter.With("method", "result").Add(1)
ms.latency.With("method", "result").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.Result(ctx)
}
func (ms *metricsMiddleware) Attestation(ctx context.Context, reportData [agent.ReportDataSize]byte) ([]byte, error) {
defer func(begin time.Time) {
ms.counter.With("method", "attestation").Add(1)
ms.latency.With("method", "attestation").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.Attestation(ctx, reportData)
}
+174
View File
@@ -0,0 +1,174 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package auth
import (
"context"
"crypto"
"crypto/ecdsa"
"crypto/ed25519"
"crypto/rsa"
"crypto/sha256"
"crypto/x509"
"encoding/base64"
"github.com/absmach/magistrala/pkg/errors"
"github.com/ultravioletrs/cocos/agent"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/metadata"
"google.golang.org/grpc/status"
)
type UserRole string
const (
UserMetadataKey = "user-id"
SignatureMetadataKey = "signature"
ConsumerRole UserRole = "consumer"
DataProviderRole UserRole = "data-provider"
AlgorithmProviderRole UserRole = "algorithm-provider"
)
var (
ErrMissingMetadata = errors.New("missing metadata")
ErrInvalidMetadata = errors.New("invalid metadata")
ErrSignatureVerificationFailed = errors.New("signature verification failed")
)
//go:generate mockery --name Authenticator --output=../mocks --filename auth.go --quiet --note "Copyright (c) Ultraviolet \n // SPDX-License-Identifier: Apache-2.0"
type Authenticator interface {
AuthenticateUser(ctx context.Context, role UserRole) (context.Context, error)
}
type service struct {
resultConsumers []interface{}
datasetProviders []interface{}
algorithmProvider interface{}
}
func New(manifest agent.Computation) (Authenticator, error) {
s := &service{}
for _, rc := range manifest.ResultConsumers {
pubKey, err := x509.ParsePKIXPublicKey(rc.UserKey)
if err != nil {
return nil, err
}
pKey, err := decodePublicKey(pubKey)
if err != nil {
return nil, err
}
s.resultConsumers = append(s.resultConsumers, pKey)
}
for _, dp := range manifest.Datasets {
pubKey, err := x509.ParsePKIXPublicKey(dp.UserKey)
if err != nil {
return nil, err
}
pKey, err := decodePublicKey(pubKey)
if err != nil {
return nil, err
}
s.datasetProviders = append(s.datasetProviders, pKey)
}
pubKey, err := x509.ParsePKIXPublicKey(manifest.Algorithm.UserKey)
if err != nil {
return nil, err
}
pKey, err := decodePublicKey(pubKey)
if err != nil {
return nil, err
}
s.algorithmProvider = pKey
return s, nil
}
func extractSignature(md metadata.MD) (string, error) {
signature := md.Get(SignatureMetadataKey)
if len(signature) != 1 {
return "", status.Errorf(codes.Unauthenticated, "invalid metadata")
}
return signature[0], nil
}
func verifySignature(role UserRole, signature string, publicKey any) error {
hash := sha256.Sum256([]byte(role))
sigByte, err := base64.StdEncoding.DecodeString(signature)
if err != nil {
return err
}
var ok bool
switch publicKey := publicKey.(type) {
case *rsa.PublicKey:
if err = rsa.VerifyPKCS1v15(publicKey, crypto.SHA256, hash[:], sigByte); err != nil {
return err
}
return nil
case *ecdsa.PublicKey:
ok = ecdsa.VerifyASN1(publicKey, hash[:], sigByte)
case ed25519.PublicKey:
ok = ed25519.Verify(publicKey, []byte(role), sigByte)
}
if !ok {
return ErrSignatureVerificationFailed
}
return nil
}
func (s *service) AuthenticateUser(ctx context.Context, role UserRole) (context.Context, error) {
md, ok := metadata.FromIncomingContext(ctx)
if !ok {
return nil, ErrMissingMetadata
}
signature, err := extractSignature(md)
if err != nil {
return nil, errors.Wrap(err, ErrInvalidMetadata)
}
switch role {
case ConsumerRole:
for i, rc := range s.resultConsumers {
if err := verifySignature(role, signature, rc); err == nil {
return agent.IndexToContext(ctx, i), nil
}
}
case DataProviderRole:
for i, dp := range s.datasetProviders {
if err := verifySignature(role, signature, dp); err == nil {
return agent.IndexToContext(ctx, i), nil
}
}
case AlgorithmProviderRole:
if err := verifySignature(role, signature, s.algorithmProvider); err == nil {
return ctx, nil
}
}
return ctx, ErrSignatureVerificationFailed
}
func decodePublicKey(key any) (pubKey any, err error) {
switch key := key.(type) {
case *rsa.PublicKey:
return key, nil
case *ecdsa.PublicKey:
return key, nil
case ed25519.PublicKey:
return key, nil
default:
return nil, errors.New("unsupported public key type")
}
}
+158
View File
@@ -0,0 +1,158 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package auth
import (
"context"
"crypto"
"crypto/ecdsa"
"crypto/ed25519"
"crypto/elliptic"
"crypto/rand"
"crypto/rsa"
"crypto/sha256"
"crypto/x509"
"encoding/base64"
"testing"
"github.com/absmach/magistrala/pkg/errors"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/ultravioletrs/cocos/agent"
"google.golang.org/grpc/metadata"
)
func TestAuthenticateUser(t *testing.T) {
resultConsumerKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
require.NoError(t, err)
dataProviderKey, err := rsa.GenerateKey(rand.Reader, 2048)
require.NoError(t, err)
pubEd25519Key, algorithmProviderKey, err := ed25519.GenerateKey(rand.Reader)
require.NoError(t, err)
resultConsumerPubKey, err := x509.MarshalPKIXPublicKey(&resultConsumerKey.PublicKey)
require.NoError(t, err)
dataProviderPubKey, err := x509.MarshalPKIXPublicKey(&dataProviderKey.PublicKey)
require.NoError(t, err)
algorithmProviderPubKey, err := x509.MarshalPKIXPublicKey(pubEd25519Key)
require.NoError(t, err)
manifest := agent.Computation{
ResultConsumers: []agent.ResultConsumer{{UserKey: resultConsumerPubKey}},
Datasets: []agent.Dataset{{UserKey: dataProviderPubKey}},
Algorithm: agent.Algorithm{UserKey: algorithmProviderPubKey},
}
auth, err := New(manifest)
if err != nil {
t.Fatalf("failed to create authenticator: %v", err)
}
testCases := []struct {
name string
role UserRole
key any
expectedErr error
}{
{
name: "valid result consumer",
role: ConsumerRole,
key: resultConsumerKey,
expectedErr: nil,
},
{
name: "valid data provider",
role: DataProviderRole,
key: dataProviderKey,
expectedErr: nil,
},
{
name: "valid algorithm provider",
role: AlgorithmProviderRole,
key: algorithmProviderKey,
expectedErr: nil,
},
{
name: "invalid role",
role: "invalid-role",
key: resultConsumerKey,
expectedErr: ErrSignatureVerificationFailed,
},
{
name: "invalid key",
role: ConsumerRole,
key: dataProviderKey,
expectedErr: ErrSignatureVerificationFailed,
},
{
name: "missing signature",
role: ConsumerRole,
key: resultConsumerKey,
expectedErr: ErrInvalidMetadata,
},
{
name: "missing metadata",
role: ConsumerRole,
key: resultConsumerKey,
expectedErr: ErrMissingMetadata,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
signature, err := signRole(tc.role, tc.key)
if err != nil {
t.Fatalf("failed to sign role: %v", err)
}
ctx := context.Background()
switch tc.name {
case "missing signature":
ctx = metadata.NewIncomingContext(ctx, metadata.Pairs())
case "missing metadata":
default:
ctx = metadata.NewIncomingContext(ctx, metadata.Pairs(SignatureMetadataKey, signature))
}
ctx, err = auth.AuthenticateUser(ctx, tc.role)
assert.True(t, errors.Contains(err, tc.expectedErr), "expected error %v, got %v", tc.expectedErr, err)
if err == nil {
switch id, ok := agent.IndexFromContext(ctx); {
case tc.role == ConsumerRole, tc.role == DataProviderRole:
assert.True(t, ok, "expected index in context")
assert.Equal(t, 0, id, "expected index 0 in context")
default:
assert.False(t, ok, "expected no index in context")
}
}
})
}
}
func signRole(role UserRole, key crypto.PrivateKey) (string, error) {
var signature []byte
var err error
switch k := key.(type) {
case ed25519.PrivateKey:
signature, err = k.Sign(rand.Reader, []byte(role), crypto.Hash(0))
case *rsa.PrivateKey, *ecdsa.PrivateKey:
hash := sha256.Sum256([]byte(role))
signer := key.(crypto.Signer)
signature, err = signer.Sign(rand.Reader, hash[:], crypto.SHA256)
default:
return "", errors.New("unsupported key type")
}
if err != nil {
return "", err
}
return base64.StdEncoding.EncodeToString(signature), nil
}
+70
View File
@@ -0,0 +1,70 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package agent
import (
"context"
"encoding/json"
"fmt"
)
var _ fmt.Stringer = (*Datasets)(nil)
type AgentConfig struct {
LogLevel string `json:"log_level"`
Host string `json:"host"`
Port string `json:"port"`
CertFile string `json:"cert_file"`
KeyFile string `json:"server_key"`
ServerCAFile string `json:"server_ca_file"`
ClientCAFile string `json:"client_ca_file"`
AttestedTls bool `json:"attested_tls"`
}
type Computation struct {
ID string `json:"id,omitempty"`
Name string `json:"name,omitempty"`
Description string `json:"description,omitempty"`
Datasets Datasets `json:"datasets,omitempty"`
Algorithm Algorithm `json:"algorithm,omitempty"`
ResultConsumers []ResultConsumer `json:"result_consumers,omitempty"`
AgentConfig AgentConfig `json:"agent_config,omitempty"`
}
type ResultConsumer struct {
UserKey []byte `json:"user_key,omitempty"`
}
func (d *Datasets) String() string {
dat, err := json.Marshal(d)
if err != nil {
return ""
}
return string(dat)
}
type Dataset struct {
Dataset []byte `json:"-"`
Hash [32]byte `json:"hash,omitempty"`
UserKey []byte `json:"user_key,omitempty"`
}
type Datasets []Dataset
type Algorithm struct {
Algorithm []byte `json:"-"`
Hash [32]byte `json:"hash,omitempty"`
UserKey []byte `json:"user_key,omitempty"`
Requirements []byte `json:"-"`
}
type ManifestIndexKey struct{}
func IndexToContext(ctx context.Context, index int) context.Context {
return context.WithValue(ctx, ManifestIndexKey{}, index)
}
func IndexFromContext(ctx context.Context) (int, bool) {
index, ok := ctx.Value(ManifestIndexKey{}).(int)
return index, ok
}
+4
View File
@@ -0,0 +1,4 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package agent
+69
View File
@@ -0,0 +1,69 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package events
import (
"encoding/json"
"time"
"github.com/mdlayher/vsock"
"github.com/ultravioletrs/cocos/pkg/manager"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/timestamppb"
)
type service struct {
service string
computationID string
conn *vsock.Conn
}
type AgentEvent struct {
EventType string `json:"event_type"`
Timestamp time.Time `json:"timestamp"`
ComputationID string `json:"computation_id,omitempty"`
Details json.RawMessage `json:"details,omitempty"`
Originator string `json:"originator"`
Status string `json:"status,omitempty"`
}
//go:generate mockery --name Service --output=./mocks --filename events.go --quiet --note "Copyright (c) Ultraviolet \n // SPDX-License-Identifier: Apache-2.0"
type Service interface {
SendEvent(event, status string, details json.RawMessage) error
Close() error
}
func New(svc, computationID string, sockPort uint32) (Service, error) {
conn, err := vsock.Dial(vsock.Host, sockPort, nil)
if err != nil {
return nil, err
}
return &service{
service: svc,
computationID: computationID,
conn: conn,
}, nil
}
func (s *service) SendEvent(event, status string, details json.RawMessage) error {
body := manager.ClientStreamMessage{Message: &manager.ClientStreamMessage_AgentEvent{AgentEvent: &manager.AgentEvent{
EventType: event,
Timestamp: timestamppb.Now(),
ComputationId: s.computationID,
Originator: s.service,
Status: status,
Details: details,
}}}
protoBody, err := proto.Marshal(&body)
if err != nil {
return err
}
if _, err := s.conn.Write(protoBody); err != nil {
return err
}
return nil
}
func (s *service) Close() error {
return s.conn.Close()
}
+67
View File
@@ -0,0 +1,67 @@
// Code generated by mockery v2.43.2. DO NOT EDIT.
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package mocks
import (
json "encoding/json"
mock "github.com/stretchr/testify/mock"
)
// Service is an autogenerated mock type for the Service type
type Service struct {
mock.Mock
}
// Close provides a mock function with given fields:
func (_m *Service) Close() error {
ret := _m.Called()
if len(ret) == 0 {
panic("no return value specified for Close")
}
var r0 error
if rf, ok := ret.Get(0).(func() error); ok {
r0 = rf()
} else {
r0 = ret.Error(0)
}
return r0
}
// SendEvent provides a mock function with given fields: event, status, details
func (_m *Service) SendEvent(event string, status string, details json.RawMessage) error {
ret := _m.Called(event, status, details)
if len(ret) == 0 {
panic("no return value specified for SendEvent")
}
var r0 error
if rf, ok := ret.Get(0).(func(string, string, json.RawMessage) error); ok {
r0 = rf(event, status, details)
} else {
r0 = ret.Error(0)
}
return r0
}
// NewService creates a new instance of Service. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
// The first argument is typically a *testing.T value.
func NewService(t interface {
mock.TestingT
Cleanup(func())
}) *Service {
mock := &Service{}
mock.Mock.Test(t)
t.Cleanup(func() { mock.AssertExpectations(t) })
return mock
}
+129
View File
@@ -0,0 +1,129 @@
// Code generated by mockery v2.42.3. DO NOT EDIT.
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package mocks
import (
context "context"
agent "github.com/ultravioletrs/cocos/agent"
mock "github.com/stretchr/testify/mock"
)
// Service is an autogenerated mock type for the Service type
type Service struct {
mock.Mock
}
// Algo provides a mock function with given fields: ctx, algorithm
func (_m *Service) Algo(ctx context.Context, algorithm agent.Algorithm) error {
ret := _m.Called(ctx, algorithm)
if len(ret) == 0 {
panic("no return value specified for Algo")
}
var r0 error
if rf, ok := ret.Get(0).(func(context.Context, agent.Algorithm) error); ok {
r0 = rf(ctx, algorithm)
} else {
r0 = ret.Error(0)
}
return r0
}
// Attestation provides a mock function with given fields: ctx, reportData
func (_m *Service) Attestation(ctx context.Context, reportData [64]byte) ([]byte, error) {
ret := _m.Called(ctx, reportData)
if len(ret) == 0 {
panic("no return value specified for Attestation")
}
var r0 []byte
var r1 error
if rf, ok := ret.Get(0).(func(context.Context, [64]byte) ([]byte, error)); ok {
return rf(ctx, reportData)
}
if rf, ok := ret.Get(0).(func(context.Context, [64]byte) []byte); ok {
r0 = rf(ctx, reportData)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).([]byte)
}
}
if rf, ok := ret.Get(1).(func(context.Context, [64]byte) error); ok {
r1 = rf(ctx, reportData)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// Data provides a mock function with given fields: ctx, dataset
func (_m *Service) Data(ctx context.Context, dataset agent.Dataset) error {
ret := _m.Called(ctx, dataset)
if len(ret) == 0 {
panic("no return value specified for Data")
}
var r0 error
if rf, ok := ret.Get(0).(func(context.Context, agent.Dataset) error); ok {
r0 = rf(ctx, dataset)
} else {
r0 = ret.Error(0)
}
return r0
}
// Result provides a mock function with given fields: ctx
func (_m *Service) Result(ctx context.Context) ([]byte, error) {
ret := _m.Called(ctx)
if len(ret) == 0 {
panic("no return value specified for Result")
}
var r0 []byte
var r1 error
if rf, ok := ret.Get(0).(func(context.Context) ([]byte, error)); ok {
return rf(ctx)
}
if rf, ok := ret.Get(0).(func(context.Context) []byte); ok {
r0 = rf(ctx)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).([]byte)
}
}
if rf, ok := ret.Get(1).(func(context.Context) error); ok {
r1 = rf(ctx)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// NewService creates a new instance of Service. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
// The first argument is typically a *testing.T value.
func NewService(t interface {
mock.TestingT
Cleanup(func())
}) *Service {
mock := &Service{}
mock.Mock.Test(t)
t.Cleanup(func() { mock.AssertExpectations(t) })
return mock
}
+63
View File
@@ -0,0 +1,63 @@
// Code generated by mockery v2.42.3. DO NOT EDIT.
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package mocks
import (
context "context"
auth "github.com/ultravioletrs/cocos/agent/auth"
mock "github.com/stretchr/testify/mock"
)
// Authenticator is an autogenerated mock type for the Authenticator type
type Authenticator struct {
mock.Mock
}
// AuthenticateUser provides a mock function with given fields: ctx, role
func (_m *Authenticator) AuthenticateUser(ctx context.Context, role auth.UserRole) (context.Context, error) {
ret := _m.Called(ctx, role)
if len(ret) == 0 {
panic("no return value specified for AuthenticateUser")
}
var r0 context.Context
var r1 error
if rf, ok := ret.Get(0).(func(context.Context, auth.UserRole) (context.Context, error)); ok {
return rf(ctx, role)
}
if rf, ok := ret.Get(0).(func(context.Context, auth.UserRole) context.Context); ok {
r0 = rf(ctx, role)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(context.Context)
}
}
if rf, ok := ret.Get(1).(func(context.Context, auth.UserRole) error); ok {
r1 = rf(ctx, role)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// NewAuthenticator creates a new instance of Authenticator. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
// The first argument is typically a *testing.T value.
func NewAuthenticator(t interface {
mock.TestingT
Cleanup(func())
}) *Authenticator {
mock := &Authenticator{}
mock.Mock.Test(t)
t.Cleanup(func() { mock.AssertExpectations(t) })
return mock
}
+254
View File
@@ -0,0 +1,254 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package agent
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"os"
"slices"
"github.com/google/go-sev-guest/client"
"github.com/ultravioletrs/cocos/agent/algorithm"
"github.com/ultravioletrs/cocos/agent/algorithm/binary"
"github.com/ultravioletrs/cocos/agent/algorithm/python"
"github.com/ultravioletrs/cocos/agent/events"
"golang.org/x/crypto/sha3"
)
var _ Service = (*agentService)(nil)
const (
// ReportDataSize is the size of the report data expected by the attestation service.
ReportDataSize = 64
algoFilePermission = 0o700
)
var (
// ErrMalformedEntity indicates malformed entity specification (e.g.
// invalid username or password).
ErrMalformedEntity = errors.New("malformed entity specification")
// ErrUnauthorizedAccess indicates missing or invalid credentials provided
// when accessing a protected resource.
ErrUnauthorizedAccess = errors.New("missing or invalid credentials provided")
// errUndeclaredAlgorithm indicates algorithm was not declared in computation manifest.
ErrUndeclaredDataset = errors.New("dataset not declared in computation manifest")
// errAllManifestItemsReceived indicates no new computation manifest items expected.
ErrAllManifestItemsReceived = errors.New("all expected manifest Items have been received")
// errUndeclaredConsumer indicates the consumer requesting results in not declared in computation manifest.
ErrUndeclaredConsumer = errors.New("result consumer is undeclared in computation manifest")
// errResultsNotReady indicates the computation results are not ready.
ErrResultsNotReady = errors.New("computation results are not yet ready")
// errStateNotReady agent received a request in the wrong state.
ErrStateNotReady = errors.New("agent not expecting this operation in the current state")
// errHashMismatch provided algorithm/dataset does not match hash in manifest.
ErrHashMismatch = errors.New("malformed data, hash does not match manifest")
)
// Service specifies an API that must be fullfiled by the domain service
// implementation, and all of its decorators (e.g. logging & metrics).
//
//go:generate mockery --name Service --output=mocks --filename agent.go --quiet --note "Copyright (c) Ultraviolet \n // SPDX-License-Identifier: Apache-2.0"
type Service interface {
Algo(ctx context.Context, algorithm Algorithm) error
Data(ctx context.Context, dataset Dataset) error
Result(ctx context.Context) ([]byte, error)
Attestation(ctx context.Context, reportData [ReportDataSize]byte) ([]byte, error)
}
type agentService struct {
computation Computation // Holds the current computation request details.
algorithm algorithm.Algorithm // Filepath to the algorithm received for the computation.
result []byte // Stores the result of the computation.
sm *StateMachine // Manages the state transitions of the agent service.
runError error // Stores any error encountered during the computation run.
eventSvc events.Service // Service for publishing events related to computation.
}
var _ Service = (*agentService)(nil)
// New instantiates the agent service implementation.
func New(ctx context.Context, logger *slog.Logger, eventSvc events.Service, cmp Computation) Service {
svc := &agentService{
sm: NewStateMachine(logger, cmp),
eventSvc: eventSvc,
}
go svc.sm.Start(ctx)
svc.sm.SendEvent(start)
svc.sm.StateFunctions[idle] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[receivingManifest] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[receivingAlgorithm] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[receivingData] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[resultsReady] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[complete] = svc.publishEvent("in-progress", json.RawMessage{})
svc.sm.StateFunctions[running] = svc.runComputation
svc.computation = cmp
svc.sm.SendEvent(manifestReceived)
return svc
}
func (as *agentService) Algo(ctx context.Context, algo Algorithm) error {
if as.sm.GetState() != receivingAlgorithm {
return ErrStateNotReady
}
if as.algorithm != nil {
return ErrAllManifestItemsReceived
}
hash := sha3.Sum256(algo.Algorithm)
if hash != as.computation.Algorithm.Hash {
return ErrHashMismatch
}
f, err := os.CreateTemp("", "algorithm")
if err != nil {
return fmt.Errorf("error creating algorithm file: %v", err)
}
if _, err := f.Write(algo.Algorithm); err != nil {
return fmt.Errorf("error writing algorithm to file: %v", err)
}
if err := os.Chmod(f.Name(), algoFilePermission); err != nil {
return fmt.Errorf("error changing file permissions: %v", err)
}
if err := f.Close(); err != nil {
return fmt.Errorf("error closing file: %v", err)
}
algoType := algorithm.AlgorithmTypeFromContext(ctx)
if algoType == "" {
algoType = string(algorithm.AlgoTypeBin)
}
switch algoType {
case string(algorithm.AlgoTypeBin):
as.algorithm = binary.New(as.sm.logger, as.eventSvc, f.Name())
case string(algorithm.AlgoTypePython):
fr, err := os.CreateTemp("", "requirements.txt")
if err != nil {
return fmt.Errorf("error creating requirments file: %v", err)
}
if _, err := fr.Write(algo.Requirements); err != nil {
return fmt.Errorf("error writing requirements to file: %v", err)
}
if err := fr.Close(); err != nil {
return fmt.Errorf("error closing file: %v", err)
}
runtime := python.PythonRunTimeFromContext(ctx)
as.algorithm = python.New(as.sm.logger, as.eventSvc, runtime, fr.Name(), f.Name())
}
if as.algorithm != nil {
as.sm.SendEvent(algorithmReceived)
}
return nil
}
func (as *agentService) Data(ctx context.Context, dataset Dataset) error {
if as.sm.GetState() != receivingData {
return ErrStateNotReady
}
if len(as.computation.Datasets) == 0 {
return ErrAllManifestItemsReceived
}
hash := sha3.Sum256(dataset.Dataset)
index, ok := IndexFromContext(ctx)
if !ok {
return ErrUndeclaredDataset
}
if hash != as.computation.Datasets[index].Hash {
return ErrHashMismatch
}
as.computation.Datasets = slices.Delete(as.computation.Datasets, index, index+1)
f, err := os.CreateTemp("", fmt.Sprintf("dataset-%d", index))
if err != nil {
return fmt.Errorf("error creating dataset file: %v", err)
}
if _, err := f.Write(dataset.Dataset); err != nil {
return fmt.Errorf("error writing dataset to file: %v", err)
}
if err := f.Close(); err != nil {
return fmt.Errorf("error closing file: %v", err)
}
as.algorithm.AddDataset(f.Name())
if len(as.computation.Datasets) == 0 {
as.sm.SendEvent(dataReceived)
}
return nil
}
func (as *agentService) Result(ctx context.Context) ([]byte, error) {
if as.sm.GetState() != resultsReady {
return []byte{}, ErrResultsNotReady
}
if len(as.computation.ResultConsumers) == 0 {
return []byte{}, ErrAllManifestItemsReceived
}
index, ok := IndexFromContext(ctx)
if !ok {
return []byte{}, ErrUndeclaredConsumer
}
as.computation.ResultConsumers = slices.Delete(as.computation.ResultConsumers, index, index+1)
if len(as.computation.ResultConsumers) == 0 {
as.sm.SendEvent(resultsConsumed)
}
// Return the result file or an error
return as.result, as.runError
}
func (as *agentService) Attestation(ctx context.Context, reportData [ReportDataSize]byte) ([]byte, error) {
provider, err := client.GetQuoteProvider()
if err != nil {
return []byte{}, err
}
rawQuote, err := provider.GetRawQuote(reportData)
if err != nil {
return []byte{}, err
}
return rawQuote, nil
}
func (as *agentService) runComputation() {
as.publishEvent("starting", json.RawMessage{})()
as.sm.logger.Debug("computation run started")
defer as.sm.SendEvent(runComplete)
as.publishEvent("in-progress", json.RawMessage{})()
result, err := as.algorithm.Run()
if err != nil {
as.runError = err
as.sm.logger.Warn(fmt.Sprintf("computation failed with error: %s", err.Error()))
as.publishEvent("failed", json.RawMessage{})()
return
}
as.publishEvent("complete", json.RawMessage{})()
as.result = result
}
func (as *agentService) publishEvent(status string, details json.RawMessage) func() {
return func() {
if err := as.eventSvc.SendEvent(as.sm.State.String(), status, details); err != nil {
as.sm.logger.Warn(err.Error())
}
}
}
+128
View File
@@ -0,0 +1,128 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package agent
import (
"context"
"fmt"
"log/slog"
"sync"
)
//go:generate stringer -type=state
type state uint8
const (
idle state = iota
receivingManifest
receivingAlgorithm
receivingData
running
resultsReady
complete
)
type event uint8
const (
start event = iota
manifestReceived
algorithmReceived
dataReceived
runComplete
resultsConsumed
)
// StateMachine represents the state machine.
type StateMachine struct {
mu sync.Mutex
State state
EventChan chan event
Transitions map[state]map[event]state
StateFunctions map[state]func()
logger *slog.Logger
wg *sync.WaitGroup
}
// NewStateMachine creates a new StateMachine.
func NewStateMachine(logger *slog.Logger, cmp Computation) *StateMachine {
sm := &StateMachine{
State: idle,
EventChan: make(chan event),
Transitions: make(map[state]map[event]state),
StateFunctions: make(map[state]func()),
logger: logger,
wg: &sync.WaitGroup{},
}
sm.Transitions[idle] = make(map[event]state)
sm.Transitions[idle][start] = receivingManifest
sm.Transitions[receivingManifest] = make(map[event]state)
sm.Transitions[receivingManifest][manifestReceived] = receivingAlgorithm
sm.Transitions[receivingAlgorithm] = make(map[event]state)
switch len(cmp.Datasets) {
case 0:
sm.Transitions[receivingAlgorithm][algorithmReceived] = running
default:
sm.Transitions[receivingAlgorithm][algorithmReceived] = receivingData
}
sm.Transitions[receivingData] = make(map[event]state)
sm.Transitions[receivingData][dataReceived] = running
sm.Transitions[running] = make(map[event]state)
sm.Transitions[running][runComplete] = resultsReady
sm.Transitions[resultsReady] = make(map[event]state)
sm.Transitions[resultsReady][resultsConsumed] = complete
return sm
}
// Start the state machine.
func (sm *StateMachine) Start(ctx context.Context) {
sm.wg.Add(1)
defer sm.wg.Done()
for {
select {
case event := <-sm.EventChan:
sm.mu.Lock()
nextState, valid := sm.Transitions[sm.State][event]
if valid {
sm.State = nextState
sm.logger.Debug(fmt.Sprintf("Transition: %v -> %v\n", sm.State, nextState))
} else {
sm.logger.Error(fmt.Sprintf("Invalid transition: %v -> ???\n", sm.State))
}
sm.mu.Unlock()
sm.mu.Lock()
stateFunc, exists := sm.StateFunctions[sm.State]
sm.mu.Unlock()
if exists {
go stateFunc()
}
case <-ctx.Done():
return
}
}
}
// SendEvent sends an event to the state machine.
func (sm *StateMachine) SendEvent(event event) {
sm.EventChan <- event
}
func (sm *StateMachine) GetState() state {
sm.mu.Lock()
defer sm.mu.Unlock()
return sm.State
}
func (sm *StateMachine) SetState(state state) {
sm.mu.Lock()
defer sm.mu.Unlock()
sm.State = state
}
+29
View File
@@ -0,0 +1,29 @@
// Code generated by "stringer -type=state"; DO NOT EDIT.
package agent
import "strconv"
func _() {
// An "invalid array index" compiler error signifies that the constant values have changed.
// Re-run the stringer command to generate them again.
var x [1]struct{}
_ = x[idle-0]
_ = x[receivingManifest-1]
_ = x[receivingAlgorithm-2]
_ = x[receivingData-3]
_ = x[running-4]
_ = x[resultsReady-5]
_ = x[complete-6]
}
const _state_name = "idlereceivingManifestreceivingAlgorithmreceivingDatarunningresultsReadycomplete"
var _state_index = [...]uint8{0, 4, 21, 39, 52, 59, 71, 79}
func (i state) String() string {
if i < 0 || i >= state(len(_state_index)-1) {
return "state(" + strconv.FormatInt(int64(i), 10) + ")"
}
return _state_name[_state_index[i]:_state_index[i+1]]
}
+72
View File
@@ -0,0 +1,72 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package agent
import (
"context"
"fmt"
"testing"
mglog "github.com/absmach/magistrala/logger"
)
var cmp = Computation{
Datasets: []Dataset{
{
Dataset: []byte("test"),
UserKey: []byte("test"),
},
},
}
func TestStateMachineTransitions(t *testing.T) {
cases := []struct {
fromState state
event event
expected state
cmp Computation
}{
{idle, start, receivingManifest, cmp},
{receivingManifest, manifestReceived, receivingAlgorithm, cmp},
{receivingAlgorithm, algorithmReceived, receivingData, cmp},
{receivingAlgorithm, algorithmReceived, running, Computation{}},
{receivingData, dataReceived, running, cmp},
{running, runComplete, resultsReady, cmp},
{resultsReady, resultsConsumed, complete, cmp},
}
for _, tc := range cases {
t.Run(fmt.Sprintf("Transition from %v to %v", tc.fromState, tc.expected), func(t *testing.T) {
sm := NewStateMachine(mglog.NewMock(), tc.cmp)
ctx, cancel := context.WithCancel(context.Background())
go func() {
sm.Start(ctx)
}()
sm.wg.Wait()
sm.SetState(tc.fromState)
sm.SendEvent(tc.event)
if sm.GetState() != tc.expected {
t.Errorf("Expected state %v after the event, but got %v", tc.expected, sm.GetState())
}
close(sm.EventChan)
cancel()
})
}
}
func TestStateMachineInvalidTransition(t *testing.T) {
sm := NewStateMachine(mglog.NewMock(), cmp)
ctx, cancel := context.WithCancel(context.Background())
go sm.Start(ctx)
sm.SetState(idle)
sm.SendEvent(dataReceived)
if sm.State != idle {
t.Errorf("State should not change on an invalid event, but got %v", sm.State)
}
cancel()
}
-471
View File
@@ -1,471 +0,0 @@
openapi: 3.0.1
info:
title: Computations service
description: HTTP API for managing computations
version: "1.0.0"
paths:
/computations:
post:
summary: Add new computation
description: |
Adds new computation to the list of computations owned by user identified using
the provided access token.
tags:
- computations
requestBody:
$ref: "#/components/requestBodies/ComputationCreateReq"
responses:
'201':
$ref: "#/components/responses/ComputationRes"
'400':
description: Failed due to malformed JSON.
'401':
description: Missing or invalid access token provided.
'409':
description: Entity already exist.
'415':
description: Missing or invalid content type.
'422':
description: Database can't process request.
'500':
$ref: "#/components/responses/ServiceError"
get:
summary: Retrieves computations
description: |
Retrieves a list of computations. Due to performance concerns, data
is retrieved in subsets. The API user must ensure that the entire
dataset is consumed either by making subsequent requests, or by
increasing the subset size of the initial request.
tags:
- computations
parameters:
- $ref: "#/components/parameters/Limit"
- $ref: "#/components/parameters/Offset"
- $ref: "#/components/parameters/Name"
- $ref: "#/components/parameters/Order"
- $ref: "#/components/parameters/Direction"
- $ref: "#/components/parameters/Metadata"
responses:
'200':
$ref: "#/components/responses/ComputationsPageRes"
'400':
description: Failed due to malformed query parameters.
'401':
description: Missing or invalid access token provided.
'404':
description: A non-existent entity request.
'422':
description: Database can't process request.
'500':
$ref: "#/components/responses/ServiceError"
/computations/search:
post:
summary: Search and retrieves computations
description: |
Retrieves a list of computations with name and metadata filtering.
Due to performance concerns, data is retrieved in subsets.
The API user must ensure that the entire
dataset is consumed either by making subsequent requests, or by
increasing the subset size of the initial request.
tags:
- computations
requestBody:
$ref: "#/components/requestBodies/ComputationsSearchReq"
responses:
'200':
$ref: "#/components/responses/ComputationsPageRes"
'400':
description: Failed due to malformed query parameters.
'401':
description: Missing or invalid access token provided.
'404':
description: A non-existent entity request.
'422':
description: Unprocessable Entity
'500':
$ref: "#/components/responses/ServiceError"
/computations/{Id}:
get:
summary: Retrieves computation info
tags:
- computation
parameters:
- $ref: "#/components/parameters/Id"
responses:
'200':
$ref: "#/components/responses/ComputationRes"
'401':
description: Missing or invalid access token provided.
'404':
description: Computation does not exist.
'422':
description: Database can't process request.
'500':
$ref: "#/components/responses/ServiceError"
put:
summary: Updates computation info
description: |
Update is performed by replacing the current resource data with values
provided in a request payload.
tags:
- computations
parameters:
- $ref: "#/components/parameters/Id"
requestBody:
$ref: "#/components/requestBodies/ComputationUpdateReq"
responses:
'200':
description: Computation updated.
'400':
description: Failed due to malformed JSON.
'401':
description: Missing or invalid access token provided.
'404':
description: Computation does not exist.
'415':
description: Missing or invalid content type.
'500':
$ref: "#/components/responses/ServiceError"
delete:
summary: Removes a computation
description: |
Removes a computation.
tags:
- computation
parameters:
- $ref: "#/components/parameters/Id"
responses:
'204':
description: Computation removed.
'400':
description: Failed due to malformed ID.
'401':
description: Missing or invalid access token provided.
'500':
$ref: "#/components/responses/ServiceError"
components:
schemas:
Key:
type: string
format: uuid
description: |
Computation key that is used for computation auth. If there is
not one provided service will generate one in UUID
format.
Identity:
type: object
properties:
id:
type: string
format: uuid
description: Computation unique identifier. This can be either
provided by the user or left blank. If the user provides a UUID,
it would be validated. If there is not one provided then
the service will generate one in UUID format.
ComputationReqSchema:
type: object
properties:
name:
type: string
description: Free-form name.
description:
type: string
description: Free-form description
datasets:
type: array
description: An array of datasets.
minItems: 0
uniqueItems: true
items:
type: string
algorithms:
type: array
description: An array of algorithms.
minItems: 1
uniqueItems: true
items:
type: string
startTime:
type: number
description: Start timestamp of the computations
endTime:
type: number
description: End of the computation
status:
type: string
description: Current status of the computation
owner:
type: string
description: ID of the user who started computation
datasetProviders:
type: array
description: List of IDs of users who provided datasets
minItems: 0
uniqueItems: true
items:
type: string
algorithmProviders:
type: array
description: List of IDs of users who provided algorithms
minItems: 1
uniqueItems: true
items:
type: string
ttl:
type: number
description: Period for computation to live in an inactive state
metadata:
type: object
description: Arbitrary, object-encoded data.
ComputationsReqSchema:
type: object
properties:
name:
type: string
description: Name filter. Filtering is performed as a case-insensitive partial match.
metadata:
type: object
description: Metadata filter. Filtering is performed matching the parameter with metadata on top level. Parameter is json.
total:
type: integer
description: Total number of items.
offset:
type: integer
description: Number of items to skip during retrieval.
default: 0
minimum: 0
limit:
type: integer
description: Size of the subset to retrieve.
default: 10
maximum: 100
minimum: 1
order:
type: string
description: Order type.
default: id
enum:
- name
- id
dir:
type: string
description: Order direction.
default: desc
enum:
- asc
- desc
ComputationResSchema:
type: object
properties:
id:
type: string
format: uuid
description: Unique identifier generated by the service.
name:
type: string
description: Free-form name.
key:
type: string
format: uuid
description: Auto-generated access key.
metadata:
type: object
description: Arbitrary, object-encoded data.
required:
- id
- type
- key
ComputationsPage:
type: object
properties:
computations:
type: array
minItems: 0
uniqueItems: true
items:
$ref: "#/components/schemas/ComputationResSchema"
total:
type: integer
description: Total number of items.
offset:
type: integer
description: Number of items to skip during retrieval.
limit:
type: integer
description: Maximum number of items to return in one page.
required:
- computations
parameters:
Id:
name: Id
description: Unique identifier.
in: path
schema:
type: string
format: uuid
required: true
Limit:
name: limit
description: Size of the subset to retrieve.
in: query
schema:
type: integer
default: 10
maximum: 100
minimum: 1
required: false
Offset:
name: offset
description: Number of items to skip during retrieval.
in: query
schema:
type: integer
default: 0
minimum: 0
required: false
Name:
name: name
description: Name filter. Filtering is performed as a case-insensitive partial match.
in: query
schema:
type: string
required: false
Order:
name: order
description: Order type.
in: query
schema:
type: string
default: id
enum:
- name
- id
required: false
Direction:
name: dir
description: Order direction.
in: query
schema:
type: string
default: desc
enum:
- asc
- desc
required: false
Metadata:
name: metadata
description: Metadata filter. Filtering is performed matching the parameter with metadata on top level. Parameter is json.
in: query
required: false
schema:
type: object
additionalProperties: {}
requestBodies:
ComputationCreateReq:
description: JSON-formatted document describing the new computaton.
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/ComputationReqSchema"
ComputationsCreateReq:
description: JSON-formatted document describing the new computation.
required: true
content:
application/json:
schema:
type: object
properties:
key:
$ref: "#/components/schemas/Key"
computations:
type: array
items:
$ref: "#/components/schemas/ComputationsReqSchema"
ComputationUpdateReq:
description: Arbitrary, object-encoded data.
required: true
content:
application/json:
schema:
type: object
properties:
name:
type: string
description: Free-form name.
metadata:
type: object
ComputationsSearchReq:
description: JSON-formatted document describing search parameters.
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/ComputationReqSchema"
KeyUpdateReq:
required: true
description: JSON containing computations.
content:
application/json:
schema:
type: object
properties:
key:
type: string
format: uuid
description: Computation key that is used for computation auth.
responses:
CreateComputationRes:
description: Computation registered.
headers:
Location:
content:
text/plain:
schema:
type: string
description: Created computation's relative URL.
example: /computations/{id}
ComputationRes:
description: Data retrieved.
content:
application/json:
schema:
$ref: "#/components/schemas/ComputationResSchema"
ComputationsPageRes:
description: Data retrieved.
content:
application/json:
schema:
$ref: "#/components/schemas/ComputationsPage"
IdentityRes:
description: ID returned.
content:
application/json:
schema:
$ref: "#/components/schemas/Identity"
ServiceError:
description: Unexpected server-side error occurred.
content:
application/json:
schema:
type: string
format: byte
HealthRes:
description: Service Health Check.
content:
application/json:
schema:
$ref: "./schemas/HealthInfo.yml"
securitySchemes:
bearerAuth:
type: http
scheme: bearer
bearerFormat: JWT
description: |
* Users access: "Authorization: Bearer <user_token>"
security:
- bearerAuth: []
-446
View File
@@ -1,446 +0,0 @@
openapi: 3.0.1
info:
title: Datasets service
description: HTTP API for managing datasets
version: "1.0.0"
paths:
/datasets:
post:
summary: Add new dataset
description: |
Adds new dataset to the list of datasets owned by user identified using
the provided access token.
tags:
- datasets
requestBody:
$ref: "#/components/requestBodies/DatasetCreateReq"
responses:
'201':
$ref: "#/components/responses/DatasetRes"
'400':
description: Failed due to malformed JSON.
'401':
description: Missing or invalid access token provided.
'409':
description: Entity already exist.
'415':
description: Missing or invalid content type.
'422':
description: Database can't process request.
'500':
$ref: "#/components/responses/ServiceError"
get:
summary: Retrieves datasets
description: |
Retrieves a list of datasets. Due to performance concerns, data
is retrieved in subsets. The API user must ensure that the entire
dataset is consumed either by making subsequent requests, or by
increasing the subset size of the initial request.
tags:
- datasets
parameters:
- $ref: "#/components/parameters/Limit"
- $ref: "#/components/parameters/Offset"
- $ref: "#/components/parameters/Name"
- $ref: "#/components/parameters/Order"
- $ref: "#/components/parameters/Direction"
- $ref: "#/components/parameters/Metadata"
responses:
'200':
$ref: "#/components/responses/DatasetsPageRes"
'400':
description: Failed due to malformed query parameters.
'401':
description: Missing or invalid access token provided.
'404':
description: A non-existent entity request.
'422':
description: Database can't process request.
'500':
$ref: "#/components/responses/ServiceError"
/datasets/search:
post:
summary: Search and retrieves datasets
description: |
Retrieves a list of datasets with name and metadata filtering.
Due to performance concerns, data is retrieved in subsets.
The API user must ensure that the entire
dataset is consumed either by making subsequent requests, or by
increasing the subset size of the initial request.
tags:
- datasets
requestBody:
$ref: "#/components/requestBodies/DatasetsSearchReq"
responses:
'200':
$ref: "#/components/responses/DatasetsPageRes"
'400':
description: Failed due to malformed query parameters.
'401':
description: Missing or invalid access token provided.
'404':
description: A non-existent entity request.
'422':
description: Unprocessable Entity
'500':
$ref: "#/components/responses/ServiceError"
/datasets/{Id}:
get:
summary: Retrieves dataset info
tags:
- dataset
parameters:
- $ref: "#/components/parameters/Id"
responses:
'200':
$ref: "#/components/responses/DatasetRes"
'401':
description: Missing or invalid access token provided.
'404':
description: Dataset does not exist.
'422':
description: Database can't process request.
'500':
$ref: "#/components/responses/ServiceError"
put:
summary: Updates dataset info
description: |
Update is performed by replacing the current resource data with values
provided in a request payload.
tags:
- datasets
parameters:
- $ref: "#/components/parameters/Id"
requestBody:
$ref: "#/components/requestBodies/DatasetUpdateReq"
responses:
'200':
description: dataset updated.
'400':
description: Failed due to malformed JSON.
'401':
description: Missing or invalid access token provided.
'404':
description: Dataset does not exist.
'415':
description: Missing or invalid content type.
'500':
$ref: "#/components/responses/ServiceError"
delete:
summary: Removes a dataset
description: |
Removes a dataset.
tags:
- dataset
parameters:
- $ref: "#/components/parameters/Id"
responses:
'204':
description: Dataset removed.
'400':
description: Failed due to malformed ID.
'401':
description: Missing or invalid access token provided.
'500':
$ref: "#/components/responses/ServiceError"
components:
schemas:
DatasetReqSchema:
type: object
properties:
name:
type: string
description: Free-form name.
description:
type: string
description: Free-form description
datasets:
type: array
description: An array of datasets.
minItems: 0
uniqueItems: true
items:
type: string
algorithms:
type: array
description: An array of algorithms.
minItems: 1
uniqueItems: true
items:
type: string
startTime:
type: number
description: Start timestamp of the datasets
endTime:
type: number
description: End of the dataset
status:
type: string
description: Current status of the dataset
owner:
type: string
description: ID of the user who started dataset
datasetProviders:
type: array
description: List of IDs of users who provided datasets
minItems: 0
uniqueItems: true
items:
type: string
algorithmProviders:
type: array
description: List of IDs of users who provided algorithms
minItems: 1
uniqueItems: true
items:
type: string
ttl:
type: number
description: Period for dataset to live in an inactive state
metadata:
type: object
description: Arbitrary, object-encoded data.
DatasetsReqSchema:
type: object
properties:
name:
type: string
description: Name filter. Filtering is performed as a case-insensitive partial match.
metadata:
type: object
description: Metadata filter. Filtering is performed matching the parameter with metadata on top level. Parameter is json.
total:
type: integer
description: Total number of items.
offset:
type: integer
description: Number of items to skip during retrieval.
default: 0
minimum: 0
limit:
type: integer
description: Size of the subset to retrieve.
default: 10
maximum: 100
minimum: 1
order:
type: string
description: Order type.
default: id
enum:
- name
- id
dir:
type: string
description: Order direction.
default: desc
enum:
- asc
- desc
DatasetResSchema:
type: object
properties:
id:
type: string
format: uuid
description: Unique identifier generated by the service.
name:
type: string
description: Free-form name.
key:
type: string
format: uuid
description: Auto-generated access key.
metadata:
type: object
description: Arbitrary, object-encoded data.
required:
- id
- type
- key
DatasetsPage:
type: object
properties:
datasets:
type: array
minItems: 0
uniqueItems: true
items:
$ref: "#/components/schemas/DatasetResSchema"
total:
type: integer
description: Total number of items.
offset:
type: integer
description: Number of items to skip during retrieval.
limit:
type: integer
description: Maximum number of items to return in one page.
required:
- datasets
parameters:
Id:
name: Id
description: Unique identifier.
in: path
schema:
type: string
format: uuid
required: true
Limit:
name: limit
description: Size of the subset to retrieve.
in: query
schema:
type: integer
default: 10
maximum: 100
minimum: 1
required: false
Offset:
name: offset
description: Number of items to skip during retrieval.
in: query
schema:
type: integer
default: 0
minimum: 0
required: false
Name:
name: name
description: Name filter. Filtering is performed as a case-insensitive partial match.
in: query
schema:
type: string
required: false
Order:
name: order
description: Order type.
in: query
schema:
type: string
default: id
enum:
- name
- id
required: false
Direction:
name: dir
description: Order direction.
in: query
schema:
type: string
default: desc
enum:
- asc
- desc
required: false
Metadata:
name: metadata
description: Metadata filter. Filtering is performed matching the parameter with metadata on top level. Parameter is json.
in: query
required: false
schema:
type: object
additionalProperties: {}
requestBodies:
DatasetCreateReq:
description: JSON-formatted document describing the new dataset.
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/DatasetReqSchema"
DatasetsCreateReq:
description: JSON-formatted document describing the new dataset.
required: true
content:
application/json:
schema:
type: object
properties:
datasets:
type: array
items:
$ref: "#/components/schemas/DatasetsReqSchema"
DatasetUpdateReq:
description: Arbitrary, object-encoded data.
required: true
content:
application/json:
schema:
type: object
properties:
name:
type: string
description: Free-form name.
metadata:
type: object
DatasetsSearchReq:
description: JSON-formatted document describing search parameters.
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/DatasetReqSchema"
KeyUpdateReq:
required: true
description: JSON containing datasets.
content:
application/json:
schema:
type: object
properties:
key:
type: string
format: uuid
description: Thing key that is used for thing auth.
responses:
CreateDatasetRes:
description: Dataset registered.
headers:
Location:
content:
text/plain:
schema:
type: string
description: Created dataset's relative URL.
example: /datasets/{id}
DatasetRes:
description: Data retrieved.
content:
application/json:
schema:
$ref: "#/components/schemas/DatasetResSchema"
DatasetsPageRes:
description: Data retrieved.
content:
application/json:
schema:
$ref: "#/components/schemas/DatasetsPage"
ServiceError:
description: Unexpected server-side error occurred.
content:
application/json:
schema:
type: string
format: byte
HealthRes:
description: Service Health Check.
content:
application/json:
schema:
$ref: "./schemas/HealthInfo.yml"
securitySchemes:
bearerAuth:
type: http
scheme: bearer
bearerFormat: JWT
description: |
* Users access: "Authorization: Bearer <user_token>"
security:
- bearerAuth: []
@@ -1,84 +0,0 @@
{
"info": {
"_postman_id": "5ed31077-ca87-496e-9b98-b8793ce7c5d2",
"name": "CoCoS",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
},
"item": [
{
"name": "Create Computation",
"request": {
"method": "POST",
"header": [],
"body": {
"mode": "raw",
"raw": "{\n \"name\": \"Calculate salary\",\n \"description\": \"my test computation\",\n \"ttl\": 1243\n}",
"options": {
"raw": {
"language": "json"
}
}
},
"url": {
"raw": "http://localhost:9000/computations",
"protocol": "http",
"host": [
"localhost"
],
"port": "9000",
"path": [
"computations"
]
}
},
"response": []
},
{
"name": "View Computation",
"request": {
"method": "GET",
"header": [],
"url": {
"raw": "http://localhost:9000/computations/c89b53d3-c088-4b5a-addd-51a90ea94325",
"protocol": "http",
"host": [
"localhost"
],
"port": "9000",
"path": [
"computations",
"c89b53d3-c088-4b5a-addd-51a90ea94325"
]
}
},
"response": []
}
],
"event": [
{
"listen": "prerequest",
"script": {
"type": "text/javascript",
"exec": [
""
]
}
},
{
"listen": "test",
"script": {
"type": "text/javascript",
"exec": [
""
]
}
}
],
"variable": [
{
"key": "jwt",
"value": "",
"type": "string"
}
]
}
+82
View File
@@ -0,0 +1,82 @@
# Agent CLI
This repository contains the command-line interface (CLI) tool for interacting with the Agent and manager service. The CLI allows you to perform various tasks such as running computations, uploading algorithm and datasets, and retrieving results.
## Build
From the project root:
```bash
make cli
```
## Usage
#### Get attestation
Retrieves attestation information from the SEV guest and saves it to a file.
To retrieve attestation from agent, use the following command:
```bash
./build/cocos-cli agent attestation get '<report_data>'
```
#### Validate attestation
Validates the retrieved attestation information against a specified policy and checks its authenticity.
To validate and verify attestation from agent, use the following command:
```bash
./build/cocos-cli agent attestation validate '<attestation>' --report_data '<report_data>'
```
##### Flags
- --config: Path to a JSON file containing the validation configuration. This can be used to override individual flags.
- --report_data: Hex-encoded string representing the attestation report data. (Required for the validate command)
- --host_data: Hex-encoded string representing the expected HOST_DATA field.
- --family_id: Hex-encoded string representing the expected FAMILY_ID field.
- --image_id: Hex-encoded string representing the expected IMAGE_ID field.
- --report_id: Hex-encoded string representing the expected REPORT_ID field.
- --report_id_ma: Hex-encoded string representing the expected REPORT_ID_MA field.
- --measurement: Hex-encoded string representing the expected MEASUREMENT field.
- --chip_id: Hex-encoded string representing the expected CHIP_ID field.
- --minimum_tcb: Minimum acceptable value for CURRENT_TCB, COMMITTED_TCB, and REPORTED_TCB.
- --minimum_launch_tcb: Minimum acceptable value for LAUNCH_TCB.
- --minimum_guest_svn: Minimum acceptable SnpPolicy.
- --minimum_build: Minimum 8-bit build number for AMD-SP firmware.
- --check_crl: Download and check the CRL for revoked certificates (default: false).
- --disallow_network: If true, disallows network downloads for verification (default: false).
- --timeout: Duration to continue retrying failed HTTP requests (default: 2 minutes).
- --max_retry_delay: Maximum duration to wait between HTTP request retries (default: 30 seconds).
- --require_author_key: Require that AUTHOR_KEY_EN is set to 1 (default: false).
- --require_id_block: Require that the VM was launched with an ID_BLOCK signed by a trusted key (default: false).
- --permit_provisional_software: Allow provisional firmware (default: false).
- --platform_info: Maximum acceptable PLATFORM_INFO field bit-wise (optional).
- --minimum_version: Minimum AMD-SP firmware API version (major.minor) (optional).
- --trusted_author_keys: Paths to x.509 certificates of trusted author keys (optional).
- --trusted_author_key_hashes: Hex-encoded SHA-384 hashes of trusted author keys (optional).
- --trusted_id_keys: Paths to x.509 certificates of trusted identity keys (optional).
- --trusted_id_key_hashes: Hex-encoded SHA-384 hashes of trusted identity keys (optional).
- --product: AMD product name for the chip that generated the attestation report (optional).
- --stepping: Machine stepping for the chip that generated the attestation report (optional).
- --CA_bundles_paths: Paths to CA bundles for the AMD product (optional).
- --CA_bundles: PEM format CA bundles for the AMD product (optional).
#### Upload Algorithm
To upload an algorithm, use the following command:
```bash
./build/cocos-cli agent algo /path/to/algorithm <private_key_file_path>
```
#### Upload Dataset
To upload a dataset, use the following command:
```bash
./build/cocos-cli agent data /path/to/dataset.csv <private_key_file_path>
```
#### Retrieve result
To retrieve the computation result, use the following command:
```bash
./build/cocos-cli agent result <private_key_file_path>
```
+83
View File
@@ -0,0 +1,83 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package cli
import (
"context"
"encoding/pem"
"log"
"os"
"github.com/spf13/cobra"
"github.com/ultravioletrs/cocos/agent"
"github.com/ultravioletrs/cocos/agent/algorithm"
"github.com/ultravioletrs/cocos/agent/algorithm/python"
"google.golang.org/grpc/metadata"
)
var (
pythonRuntime string
algoType string
requirementsFile string
)
func (cli *CLI) NewAlgorithmCmd() *cobra.Command {
cmd := &cobra.Command{
Use: "algo",
Short: "Upload an algorithm binary",
Example: "algo <algo_file> <private_key_file_path>",
Args: cobra.ExactArgs(2),
Run: func(cmd *cobra.Command, args []string) {
algorithmFile := args[0]
log.Println("Uploading algorithm binary:", algorithmFile)
algorithm, err := os.ReadFile(algorithmFile)
if err != nil {
log.Fatalf("Error reading algorithm file: %v", err)
}
var req []byte
if requirementsFile != "" {
req, err = os.ReadFile(requirementsFile)
if err != nil {
log.Fatalf("Error reading requirments file: %v", err)
}
}
algoReq := agent.Algorithm{
Algorithm: algorithm,
Requirements: req,
}
privKeyFile, err := os.ReadFile(args[1])
if err != nil {
log.Fatalf("Error reading private key file: %v", err)
}
pemBlock, _ := pem.Decode(privKeyFile)
privKey := decodeKey(pemBlock)
ctx := metadata.NewOutgoingContext(cmd.Context(), metadata.New(make(map[string]string)))
if err := cli.agentSDK.Algo(addAlgoMetadata(ctx), algoReq, privKey); err != nil {
log.Fatalf("Error uploading algorithm with error: %v", err)
}
log.Println("Successfully uploaded algorithm")
},
}
cmd.Flags().StringVarP(&algoType, "algorithm", "a", string(algorithm.AlgoTypeBin), "Algorithm type to run")
cmd.Flags().StringVar(&pythonRuntime, "python-runtime", python.PyRuntime, "Python runtime to use")
cmd.Flags().StringVarP(&requirementsFile, "requirements", "r", "", "Python requirements file")
return cmd
}
func addAlgoMetadata(ctx context.Context) context.Context {
ctx = algorithm.AlgorithmTypeToContext(ctx, algoType)
ctx = python.PythonRunTimeToContext(ctx, pythonRuntime)
return ctx
}
+663
View File
@@ -0,0 +1,663 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package cli
import (
"encoding/hex"
"fmt"
"log"
"os"
"strconv"
"strings"
"time"
"github.com/google/go-sev-guest/abi"
"github.com/google/go-sev-guest/proto/check"
"github.com/google/go-sev-guest/proto/sevsnp"
"github.com/google/go-sev-guest/validate"
"github.com/google/go-sev-guest/verify"
"github.com/google/go-sev-guest/verify/trust"
"github.com/spf13/cobra"
"github.com/spf13/pflag"
"github.com/ultravioletrs/cocos/agent"
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/known/wrapperspb"
)
const (
defaultMinimumTcb = 0
defaultMinimumLaunchTcb = 0
defaultMinimumGuestSvn = 0
defaultGuestPolicy = 0x0000000000030000
defaultMinimumBuild = 0
defaultCheckCrl = false
defaultTimeout = 2 * time.Minute
defaultMaxRetryDelay = 30 * time.Second
defaultRequireAuthor = false
defaultRequireIdBlock = false
defaultMinVersion = "0.0"
size16 = 16
size32 = 32
size48 = 48
size64 = 64
attestationFilePath = "attestation.bin"
sevProductNameMilan = "Milan"
sevProductNameGenoa = "Genoa"
exampleJSONConfig = `
{
"rootOfTrust":{
"product":"test_product",
"cabundlePaths":[
"test_cabundlePaths"
],
"cabundles":[
"test_Cabundles"
],
"checkCrl":true,
"disallowNetwork":true
},
"policy":{
"minimumGuestSvn":1,
"policy":"1",
"familyId":"AQIDBAUGBwgJCgsMDQ4PEA==",
"imageId":"AQIDBAUGBwgJCgsMDQ4PEA==",
"vmpl":0,
"minimumTcb":"1",
"minimumLaunchTcb":"1",
"platformInfo":"1",
"requireAuthorKey":true,
"reportData":"J+60aXs8btm8VcGgaJYURGeNCu0FIyWMFXQ7ZUlJDC0FJGJizJsOzDIXgQ75UtPC+Zqe0A3dvnnf5VEeQ61RTg==",
"measurement":"8s78ewoX7Xkfy1qsgVnkZwLDotD768Nqt6qTL5wtQOxHsLczipKM6bhDmWiHLdP4",
"hostData":"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw=",
"reportId":"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw=",
"reportIdMa":"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw=",
"chipId":"J+60aXs8btm8VcGgaJYURGeNCu0FIyWMFXQ7ZUlJDC0FJGJizJsOzDIXgQ75UtPC+Zqe0A3dvnnf5VEeQ61RTg==",
"minimumBuild":1,
"minimumVersion":"0.90",
"permitProvisionalFirmware":true,
"requireIdBlock":true,
"trustedAuthorKeys":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"trustedAuthorKeyHashes":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"trustedIdKeys":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"trustedIdKeyHashes":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"product":{
"name":"SEV_PRODUCT_MILAN",
"stepping":1,
"machineStepping":1
}
}
}
`
)
var (
cfg = check.Config{Policy: &check.Policy{}, RootOfTrust: &check.RootOfTrust{}}
cfgString string
timeout time.Duration
maxRetryDelay time.Duration
platformInfo string
stepping string
trustedAuthorKeys []string
trustedAuthorHashes []string
trustedIdKeys []string
trustedIdKeyHashes []string
attestationFile string
attestation []byte
empty16 = [size16]byte{}
empty32 = [size32]byte{}
empty64 = [size64]byte{}
defaultReportIdMa = []byte{255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255}
)
func (cli *CLI) NewAttestationCmd() *cobra.Command {
return &cobra.Command{
Use: "attestation [command]",
Short: "Get and validate attestations",
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("Get and validate attestations\n\n")
fmt.Printf("Usage:\n %s [command]\n\n", cmd.CommandPath())
fmt.Printf("Available Commands:\n")
// Filter out "completion" command
availableCommands := make([]*cobra.Command, 0)
for _, subCmd := range cmd.Commands() {
if subCmd.Name() != "completion" {
availableCommands = append(availableCommands, subCmd)
}
}
for _, subCmd := range availableCommands {
fmt.Printf(" %-15s%s\n", subCmd.Name(), subCmd.Short)
}
fmt.Printf("\nFlags:\n")
cmd.Flags().VisitAll(func(flag *pflag.Flag) {
fmt.Printf(" -%s, --%s %s\n", flag.Shorthand, flag.Name, flag.Usage)
})
fmt.Printf("\nUse \"%s [command] --help\" for more information about a command.\n", cmd.CommandPath())
},
}
}
func (cli *CLI) NewGetAttestationCmd() *cobra.Command {
return &cobra.Command{
Use: "get",
Short: "Retrieve attestation information from agent. Report data expected in hex enoded string of length 64 bytes.",
Example: "get <report_data>",
Args: cobra.ExactArgs(1),
Run: func(cmd *cobra.Command, args []string) {
log.Println("Getting attestation")
reportData, err := hex.DecodeString(args[0])
if err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
if len(reportData) != agent.ReportDataSize {
log.Fatalf("report data must be a hex encoded string of length %d bytes", agent.ReportDataSize)
}
result, err := cli.agentSDK.Attestation(cmd.Context(), [agent.ReportDataSize]byte(reportData))
if err != nil {
log.Fatalf("Error retrieving attestation: %v", err)
}
if err = os.WriteFile(attestationFilePath, result, 0o644); err != nil {
log.Fatalf("Error saving attestation result: %v", err)
}
log.Println("Attestation result retrieved and saved successfully!")
},
}
}
func (cli *CLI) NewValidateAttestationValidationCmd() *cobra.Command {
cmd := &cobra.Command{
Use: "validate",
Short: "Validate and verify attestation information. The report is provided as a file path.",
Example: "validate <attestation_report_file_path>",
Args: cobra.ExactArgs(1),
Run: func(cmd *cobra.Command, args []string) {
log.Println("Checking attestation")
attestationFile = string(args[0])
if err := parseConfig(); err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
if err := parseHashes(); err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
if err := parseFiles(); err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
// This format is the attestation report in AMD's specified ABI format, immediately
// followed by the certificate table bytes.
if len(attestation) < abi.ReportSize {
log.Fatalf("attestation contents too small (0x%x bytes). Want at least 0x%x bytes", len(attestation), abi.ReportSize)
}
if err := parseUints(); err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
cfg.Policy.Vmpl = wrapperspb.UInt32(0)
if err := validateInput(); err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
if err := verifyAndValidateAttestation(attestation); err != nil {
log.Fatalf("attestation validation and verification failed with error: %s", err)
}
log.Println("Attestation validation and verification is successful!")
},
}
cmd.Flags().StringVar(
&cfgString,
"config",
"",
"Serialized json check.Config protobuf. This will overwrite individual flags. Unmarshalled as json. Example: "+exampleJSONConfig,
)
cmd.Flags().BytesHexVar(
&cfg.Policy.HostData,
"host_data",
empty32[:],
"The expected HOST_DATA field as a hex string. Must encode 32 bytes. Unchecked if unset.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.ReportData,
"report_data",
empty64[:],
"The expected REPORT_DATA field as a hex string. Must encode 64 bytes. Must be set.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.FamilyId,
"family_id",
empty16[:],
"The expected FAMILY_ID field as a hex string. Must encode 16 bytes. Unchecked if unset.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.ImageId,
"image_id",
empty16[:],
"The expected IMAGE_ID field as a hex string. Must encode 16 bytes. Unchecked if unset.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.ReportId,
"report_id",
nil,
"The expected REPORT_ID field as a hex string. Must encode 32 bytes. Unchecked if unset.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.ReportIdMa,
"report_id_ma",
defaultReportIdMa,
"The expected REPORT_ID_MA field as a hex string. Must encode 32 bytes. Unchecked if unset.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.Measurement,
"measurement",
nil,
"The expected MEASUREMENT field as a hex string. Must encode 48 bytes. Unchecked if unset.",
)
cmd.Flags().BytesHexVar(
&cfg.Policy.ChipId,
"chip_id",
nil,
"The expected MEASUREMENT field as a hex string. Must encode 48 bytes. Unchecked if unset.",
)
cmd.Flags().Uint64Var(
&cfg.Policy.MinimumTcb,
"minimum_tcb",
defaultMinimumTcb,
"The minimum acceptable value for CURRENT_TCB, COMMITTED_TCB, and REPORTED_TCB.",
)
cmd.Flags().Uint64Var(
&cfg.Policy.MinimumLaunchTcb,
"minimum_lauch_tcb",
defaultMinimumLaunchTcb,
"The minimum acceptable value for LAUNCH_TCB.",
)
cmd.Flags().Uint64Var(
&cfg.Policy.Policy,
"guest_policy",
defaultGuestPolicy,
"The most acceptable guest SnpPolicy.",
)
cmd.Flags().Uint32Var(
&cfg.Policy.MinimumGuestSvn,
"minimum_guest_svn",
defaultMinimumGuestSvn,
"The most acceptable GUEST_SVN.",
)
cmd.Flags().Uint32Var(
&cfg.Policy.MinimumBuild,
"minimum_build",
defaultMinimumBuild,
"The 8-bit minimum build number for AMD-SP firmware",
)
cmd.Flags().BoolVar(
&cfg.RootOfTrust.CheckCrl,
"check_crl",
defaultCheckCrl,
"Download and check the CRL for revoked certificates.",
)
cmd.Flags().DurationVar(
&timeout,
"timeout",
defaultTimeout,
"Duration to continue to retry failed HTTP requests.",
)
cmd.Flags().DurationVar(
&maxRetryDelay,
"max_retry_delay",
defaultMaxRetryDelay,
"Maximum Duration to wait between HTTP request retries.",
)
cmd.Flags().BoolVar(
&cfg.Policy.RequireAuthorKey,
"require_author_key",
defaultRequireAuthor,
"Require that AUTHOR_KEY_EN is 1.",
)
cmd.Flags().BoolVar(
&cfg.Policy.RequireIdBlock,
"require_id_block",
defaultRequireIdBlock,
"Require that the VM was launch with an ID_BLOCK signed by a trusted id key or author key",
)
cmd.Flags().StringVar(
&platformInfo,
"platform_info",
"",
"The maximum acceptable PLATFORM_INFO field bit-wise. May be empty or a 64-bit unsigned integer",
)
cmd.Flags().StringVar(
&cfg.Policy.MinimumVersion,
"minimum_version",
defaultMinVersion,
"Minimum AMD-SP firmware API version (major.minor). Each number must be 8-bit non-negative.",
)
cmd.Flags().StringArrayVar(
&trustedAuthorKeys,
"trusted_author_keys",
[]string{},
"Paths to x.509 certificates of trusted author keys",
)
cmd.Flags().StringArrayVar(
&trustedAuthorHashes,
"trusted_author_key_hashes",
[]string{},
"Hex-encoded SHA-384 hash values of trusted author keys in AMD public key format",
)
cmd.Flags().StringArrayVar(
&trustedIdKeys,
"trusted_id_keys",
[]string{},
"Paths to x.509 certificates of trusted author keys",
)
cmd.Flags().StringArrayVar(
&trustedIdKeyHashes,
"trusted_id_key_hashes",
[]string{},
"Hex-encoded SHA-384 hash values of trusted identity keys in AMD public key format",
)
cmd.Flags().StringVar(
&cfg.RootOfTrust.ProductLine,
"product",
"",
"The AMD product name for the chip that generated the attestation report.",
)
cmd.Flags().StringVar(
&stepping,
"stepping",
"",
"The machine stepping for the chip that generated the attestation report. Default unchecked.",
)
cmd.Flags().StringArrayVar(
&cfg.RootOfTrust.CabundlePaths,
"CA_bundles_paths",
[]string{},
"Paths to CA bundles for the AMD product. Must be in PEM format, ASK, then ARK certificates. If unset, uses embedded root certificates.",
)
cmd.Flags().StringArrayVar(
&cfg.RootOfTrust.Cabundles,
"CA_bundles",
[]string{},
"PEM format CA bundles for the AMD product. Combined with contents of cabundle_paths.",
)
if err := cmd.MarkFlagRequired("report_data"); err != nil {
log.Fatalf("Failed to mark flag as required: %s", err)
}
if err := cmd.MarkFlagRequired("product"); err != nil {
log.Fatalf("Failed to mark flag as required: %s", err)
}
return cmd
}
func verifyAndValidateAttestation(attestation []byte) error {
sopts, err := verify.RootOfTrustToOptions(cfg.RootOfTrust)
if err != nil {
return err
}
if cfg.Policy.Product == nil {
productName := sevsnp.SevProduct_SEV_PRODUCT_UNKNOWN
switch cfg.RootOfTrust.ProductLine {
case sevProductNameMilan:
productName = sevsnp.SevProduct_SEV_PRODUCT_MILAN
case sevProductNameGenoa:
productName = sevsnp.SevProduct_SEV_PRODUCT_GENOA
default:
}
if productName == sevsnp.SevProduct_SEV_PRODUCT_UNKNOWN {
return fmt.Errorf("product name must be %s or %s", sevProductNameMilan, sevProductNameGenoa)
}
sopts.Product = &sevsnp.SevProduct{
Name: productName,
}
} else {
sopts.Product = cfg.Policy.Product
}
sopts.Getter = &trust.RetryHTTPSGetter{
Timeout: timeout,
MaxRetryDelay: maxRetryDelay,
Getter: &trust.SimpleHTTPSGetter{},
}
// Only take the attestation report and ignore everything else.
attestationPB, err := abi.ReportCertsToProto(attestation[:abi.ReportSize])
if err != nil {
return err
}
if err = verify.SnpAttestation(attestationPB, sopts); err != nil {
return err
}
opts, err := validate.PolicyToOptions(cfg.Policy)
if err != nil {
return err
}
if err = validate.SnpAttestation(attestationPB, opts); err != nil {
return err
}
return nil
}
// parseConfig decodes config passed as json for check.Config struct.
// example
/* {
"rootOfTrust":{
"product":"test_product",
"cabundlePaths":[
"test_cabundlePaths"
],
"cabundles":[
"test_Cabundles"
],
"checkCrl":true,
"disallowNetwork":true
},
"policy":{
"minimumGuestSvn":1,
"policy":"1",
"familyId":"AQIDBAUGBwgJCgsMDQ4PEA==",
"imageId":"AQIDBAUGBwgJCgsMDQ4PEA==",
"vmpl":0,
"minimumTcb":"1",
"minimumLaunchTcb":"1",
"platformInfo":"1",
"requireAuthorKey":true,
"reportData":"J+60aXs8btm8VcGgaJYURGeNCu0FIyWMFXQ7ZUlJDC0FJGJizJsOzDIXgQ75UtPC+Zqe0A3dvnnf5VEeQ61RTg==",
"measurement":"8s78ewoX7Xkfy1qsgVnkZwLDotD768Nqt6qTL5wtQOxHsLczipKM6bhDmWiHLdP4",
"hostData":"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw=",
"reportId":"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw=",
"reportIdMa":"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw=",
"chipId":"J+60aXs8btm8VcGgaJYURGeNCu0FIyWMFXQ7ZUlJDC0FJGJizJsOzDIXgQ75UtPC+Zqe0A3dvnnf5VEeQ61RTg==",
"minimumBuild":1,
"minimumVersion":"0.90",
"permitProvisionalFirmware":true,
"requireIdBlock":true,
"trustedAuthorKeys":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"trustedAuthorKeyHashes":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"trustedIdKeys":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"trustedIdKeyHashes":[
"GSvLKpfu59Y9QOF6vhq0vQsOIvb4+5O/UOHLGLBTkdw="
],
"product":{
"name":"SEV_PRODUCT_MILAN",
"stepping":1,
"machineStepping":1
}
}
}*/
func parseConfig() error {
if cfgString == "" {
return nil
}
if err := protojson.Unmarshal([]byte(cfgString), &cfg); err != nil {
return err
}
// Populate fields that should not be nil
if cfg.RootOfTrust == nil {
cfg.RootOfTrust = &check.RootOfTrust{}
}
if cfg.Policy == nil {
cfg.Policy = &check.Policy{}
}
return nil
}
func parseHashes() error {
for _, hash := range trustedAuthorHashes {
hashBytes, err := hex.DecodeString(hash)
if err != nil {
return err
}
cfg.Policy.TrustedAuthorKeyHashes = append(cfg.Policy.TrustedAuthorKeyHashes, hashBytes)
}
for _, hash := range trustedIdKeyHashes {
hashBytes, err := hex.DecodeString(hash)
if err != nil {
return err
}
cfg.Policy.TrustedIdKeyHashes = append(cfg.Policy.TrustedIdKeyHashes, hashBytes)
}
return nil
}
func parseFiles() error {
file, err := os.ReadFile(attestationFile)
if err != nil {
return err
}
attestation = file
for _, path := range trustedAuthorKeys {
file, err := os.ReadFile(path)
if err != nil {
return err
}
cfg.Policy.TrustedAuthorKeys = append(cfg.Policy.TrustedAuthorKeys, file)
}
for _, path := range trustedIdKeys {
file, err := os.ReadFile(path)
if err != nil {
return err
}
cfg.Policy.TrustedIdKeys = append(cfg.Policy.TrustedIdKeys, file)
}
return nil
}
func parseUints() error {
if stepping != "" {
if base := getBase(stepping); base == 10 {
num, err := strconv.ParseUint(stepping, getBase(stepping), 8)
if err != nil {
return err
}
cfg.Policy.Product.MachineStepping = wrapperspb.UInt32(uint32(num))
} else {
num, err := strconv.ParseUint(stepping[2:], base, 8)
if err != nil {
return err
}
cfg.Policy.Product.MachineStepping = wrapperspb.UInt32(uint32(num))
}
}
if platformInfo != "" {
if base := getBase(platformInfo); base == 10 {
num, err := strconv.ParseUint(platformInfo, getBase(platformInfo), 8)
if err != nil {
return err
}
cfg.Policy.PlatformInfo = wrapperspb.UInt64(num)
} else {
num, err := strconv.ParseUint(platformInfo[2:], base, 8)
if err != nil {
return err
}
cfg.Policy.PlatformInfo = wrapperspb.UInt64(num)
}
}
return nil
}
func getBase(val string) int {
switch {
case strings.HasPrefix(val, "0x"):
return 16
case strings.HasPrefix(val, "0o"):
return 8
case strings.HasPrefix(val, "0b"):
return 2
default:
return 10
}
}
func validateInput() error {
if len(cfg.RootOfTrust.CabundlePaths) != 0 || len(cfg.RootOfTrust.Cabundles) != 0 && cfg.RootOfTrust.Product == "" {
return fmt.Errorf("product name must be set if CA bundles are provided")
}
if err := validateFieldLength("report_data", cfg.Policy.ReportData, size64); err != nil {
return err
}
if err := validateFieldLength("host_data", cfg.Policy.HostData, size32); err != nil {
return err
}
if err := validateFieldLength("family_id", cfg.Policy.FamilyId, size16); err != nil {
return err
}
if err := validateFieldLength("image_id", cfg.Policy.ImageId, size16); err != nil {
return err
}
if err := validateFieldLength("report_id", cfg.Policy.ReportId, size32); err != nil {
return err
}
if err := validateFieldLength("report_id_ma", cfg.Policy.ReportIdMa, size32); err != nil {
return err
}
if err := validateFieldLength("measurement", cfg.Policy.Measurement, size48); err != nil {
return err
}
if err := validateFieldLength("chip_id", cfg.Policy.ChipId, size64); err != nil {
return err
}
for _, hash := range cfg.Policy.TrustedAuthorKeyHashes {
if err := validateFieldLength("trusted_author_key_hash", hash, size48); err != nil {
return err
}
}
for _, hash := range cfg.Policy.TrustedIdKeyHashes {
if err := validateFieldLength("trusted_id_key_hash", hash, size48); err != nil {
return err
}
}
return nil
}
func validateFieldLength(fieldName string, field []byte, expectedLength int) error {
if field != nil && len(field) != expectedLength {
return fmt.Errorf("%s length should be at least %d bytes long", fieldName, expectedLength)
}
return nil
}
+144
View File
@@ -0,0 +1,144 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package cli
import (
"encoding/base64"
"encoding/json"
"fmt"
"log"
"os"
"github.com/absmach/magistrala/pkg/errors"
"github.com/google/go-sev-guest/proto/check"
"github.com/spf13/cobra"
"github.com/spf13/pflag"
)
type fieldType int
const (
measurementField fieldType = iota
hostDataField
)
const (
// 0o744 file permission gives RWX permission to the user and only the R permission to others.
filePermission = 0o744
// Length of the expected host data and measurement field in bytes.
hostDataLength = 32
measurementLength = 48
)
var (
errDecode = errors.New("base64 string could not be decoded")
errDataLength = errors.New("data does not have an adequate length")
errReadingBackendInfoFile = errors.New("error while reading the backend information file")
errUnmarshalJSON = errors.New("failed to unmarshal json")
errMarshalJSON = errors.New("failed to marshal json")
errWriteFile = errors.New("failed to write to file")
errBackendField = errors.New("the specified field type does not exist in the backend information")
)
type AttestationConfiguration struct {
SNPPolicy *check.Policy `json:"snp_policy,omitempty"`
RootOfTrust *check.RootOfTrust `json:"root_of_trust,omitempty"`
}
func (cli *CLI) NewBackendCmd() *cobra.Command {
return &cobra.Command{
Use: "backend [command]",
Short: "Change backend information",
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("Change backend information\n\n")
fmt.Printf("Usage:\n %s [command]\n\n", cmd.CommandPath())
fmt.Printf("Available Commands:\n")
// Filter out "completion" command
availableCommands := make([]*cobra.Command, 0)
for _, subCmd := range cmd.Commands() {
if subCmd.Name() != "completion" {
availableCommands = append(availableCommands, subCmd)
}
}
for _, subCmd := range availableCommands {
fmt.Printf(" %-15s%s\n", subCmd.Name(), subCmd.Short)
}
fmt.Printf("\nFlags:\n")
cmd.Flags().VisitAll(func(flag *pflag.Flag) {
fmt.Printf(" -%s, --%s %s\n", flag.Shorthand, flag.Name, flag.Usage)
})
fmt.Printf("\nUse \"%s [command] --help\" for more information about a command.\n", cmd.CommandPath())
},
}
}
func (cli *CLI) NewAddMeasurementCmd() *cobra.Command {
return &cobra.Command{
Use: "measurement",
Short: "Add measurement to the backend info file. The value should be in base64. The second parameter is backend_info.json file",
Example: "measurement <measurement> <backend_info.json>",
Args: cobra.ExactArgs(2),
Run: func(cmd *cobra.Command, args []string) {
if err := changeAttestationConfiguration(args[1], args[0], measurementLength, measurementField); err != nil {
log.Fatalf("Error could not change measurement data %v", err)
}
},
}
}
func (cli *CLI) NewAddHostDataCmd() *cobra.Command {
return &cobra.Command{
Use: "hostdata",
Short: "Add host data to the backend info file. The value should be in base64. The second parameter is backend_info.json file",
Example: "hostdata <host-data> <backend_info.json>",
Args: cobra.ExactArgs(2),
Run: func(cmd *cobra.Command, args []string) {
if err := changeAttestationConfiguration(args[1], args[0], hostDataLength, hostDataField); err != nil {
log.Fatalf("Error could not change host data %v", err)
}
},
}
}
func changeAttestationConfiguration(fileName, base64Data string, expectedLength int, field fieldType) error {
data, err := base64.StdEncoding.DecodeString(base64Data)
if err != nil {
return errDecode
}
if len(data) != expectedLength {
return errDataLength
}
ac := AttestationConfiguration{}
backendInfo, err := os.ReadFile(fileName)
if err != nil {
return errors.Wrap(errReadingBackendInfoFile, err)
}
if err = json.Unmarshal(backendInfo, &ac); err != nil {
return errors.Wrap(errUnmarshalJSON, err)
}
switch field {
case measurementField:
ac.SNPPolicy.Measurement = data
case hostDataField:
ac.SNPPolicy.HostData = data
default:
return errBackendField
}
fileJson, err := json.MarshalIndent(ac, "", " ")
if err != nil {
return errors.Wrap(errMarshalJSON, err)
}
if err = os.WriteFile(fileName, fileJson, filePermission); err != nil {
return errors.Wrap(errWriteFile, err)
}
return nil
}
+69
View File
@@ -0,0 +1,69 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package cli
import (
"log"
"os"
"path"
"github.com/google/go-sev-guest/abi"
"github.com/google/go-sev-guest/kds"
"github.com/google/go-sev-guest/verify/trust"
"github.com/spf13/cobra"
"github.com/ultravioletrs/cocos/pkg/clients/grpc"
)
const (
caBundleName = "ask_ark.pem"
filePermisionKeys = 0o766
)
func (cli *CLI) NewCABundleCmd(fileSavePath string) *cobra.Command {
return &cobra.Command{
Use: "ca-bundle",
Short: "Fetch AMD SEV-SNPs CA Bundle (ASK and ARK)",
Example: "ca-bundle <path_to_platform_info_json>",
Args: cobra.ExactArgs(1),
Run: func(cmd *cobra.Command, args []string) {
attestationConfiguration := grpc.AttestationConfiguration{}
err := grpc.ReadManifest(args[0], &attestationConfiguration)
if err != nil {
log.Fatalf("Error while reading manifest: %v", err)
}
product := attestationConfiguration.RootOfTrust.Product
getter := trust.DefaultHTTPSGetter()
caURL := kds.ProductCertChainURL(abi.VcekReportSigner, product)
bundle, err := getter.Get(caURL)
if err != nil {
log.Fatalf("Error fetching ARK and ASK from AMD KDS for product: %s, error: %v", product, err)
}
err = os.MkdirAll(path.Join(fileSavePath, product), filePermisionKeys)
if err != nil {
log.Fatalf("Error while creating directory for product name %s, error: %v", product, err)
}
bundleFilePath := path.Join(fileSavePath, product, caBundleName)
if err = saveToFile(bundleFilePath, bundle); err != nil {
log.Fatalf("Error while saving ARK-ASK to file: %v", err)
}
},
}
}
func saveToFile(fileSavePath string, content []byte) error {
file, err := os.OpenFile(fileSavePath, os.O_CREATE|os.O_RDWR|os.O_TRUNC, filePermisionKeys)
if err != nil {
return err
}
if _, err := file.Write(content); err != nil {
return err
}
return nil
}
+74
View File
@@ -0,0 +1,74 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package cli
import (
"crypto/x509"
"encoding/pem"
"log"
"os"
"github.com/spf13/cobra"
"github.com/ultravioletrs/cocos/agent"
)
func (cli *CLI) NewDatasetsCmd() *cobra.Command {
return &cobra.Command{
Use: "data",
Short: "Upload a dataset CSV file",
Example: "data <dataset.csv> <private_key_file_path>",
Args: cobra.ExactArgs(2),
Run: func(cmd *cobra.Command, args []string) {
datasetFile := args[0]
log.Println("Uploading dataset CSV:", datasetFile)
dataset, err := os.ReadFile(datasetFile)
if err != nil {
log.Fatalf("Error reading dataset file: %v", err)
}
dataReq := agent.Dataset{
Dataset: dataset,
}
privKeyFile, err := os.ReadFile(args[1])
if err != nil {
log.Fatalf("Error reading private key file: %v", err)
}
pemBlock, _ := pem.Decode(privKeyFile)
privKey := decodeKey(pemBlock)
if err := cli.agentSDK.Data(cmd.Context(), dataReq, privKey); err != nil {
log.Fatalf("Error uploading dataset: %v", err)
}
log.Println("Successfully uploaded dataset")
},
}
}
func decodeKey(b *pem.Block) interface{} {
switch b.Type {
case rsaKeyType:
privKey, err := x509.ParsePKCS8PrivateKey(b.Bytes)
if err != nil {
privKey, err = x509.ParsePKCS1PrivateKey(b.Bytes)
if err != nil {
log.Fatalf("Error parsing private key: %v", err)
}
}
return privKey
case ecdsaKeyType:
privKey, err := x509.ParseECPrivateKey(b.Bytes)
if err != nil {
log.Fatalf("Error parsing private key: %v", err)
}
return privKey
default:
log.Fatalf("Error decoding key")
return nil
}
}
+35
View File
@@ -0,0 +1,35 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package cli
import (
"encoding/hex"
"log"
"os"
"github.com/spf13/cobra"
"golang.org/x/crypto/sha3"
)
func (cli *CLI) NewFileHashCmd() *cobra.Command {
return &cobra.Command{
Use: "file-hash",
Short: "Compute the sha3-256 hash of a file",
Example: "file-hash <file>",
Args: cobra.ExactArgs(1),
Run: func(cmd *cobra.Command, args []string) {
fileName := args[0]
file, err := os.ReadFile(fileName)
if err != nil {
log.Fatalf("Error reading dataset file: %v", err)
}
hashBytes := sha3.Sum256(file)
hash := hex.EncodeToString(hashBytes[:])
log.Println("Hash of file:", hash)
},
}
}
+128
View File
@@ -0,0 +1,128 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package cli
import (
"crypto/ecdsa"
"crypto/ed25519"
"crypto/elliptic"
"crypto/rand"
"crypto/rsa"
"crypto/x509"
"encoding/pem"
"log"
"os"
"reflect"
"github.com/spf13/cobra"
)
const (
keyBitSize = 4096
rsaKeyType = "PRIVATE KEY"
ecdsaKeyType = "EC PRIVATE KEY"
ed25519KeyType = "PRIVATE KEY"
publicKeyType = "PUBLIC KEY"
publicKeyFile = "public.pem"
privateKeyFile = "private.pem"
)
var KeyType string
func (cli *CLI) NewKeysCmd() *cobra.Command {
return &cobra.Command{
Use: "keys",
Short: "Generate a new public/private key pair",
Long: "Generates a new public/private key pair using an algorithm of the users choice.\n" +
"Supported algorithms are RSA, ecdsa, and ed25519.",
Example: "./build/cocos-cli keys -k rsa",
Args: cobra.ExactArgs(0),
Run: func(cmd *cobra.Command, args []string) {
switch KeyType {
case "ecdsa":
privEcdsaKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
log.Fatalf("Error generating keys: %v", err)
}
pubKeyBytes, err := x509.MarshalPKIXPublicKey(&privEcdsaKey.PublicKey)
if err != nil {
log.Fatalf("Error marshalling public key: %v", err)
}
generateAndWriteKeys(privEcdsaKey, pubKeyBytes, ecdsaKeyType)
case "ed25519":
pubEd25519Key, privEd25519Key, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
log.Fatalf("Error generating keys: %v", err)
}
pubKey, err := x509.MarshalPKIXPublicKey(pubEd25519Key)
if err != nil {
log.Fatalf("Error marshalling public key: %v", err)
}
generateAndWriteKeys(privEd25519Key, pubKey, ed25519KeyType)
// Default to RSA
default:
privKey, err := rsa.GenerateKey(rand.Reader, keyBitSize)
if err != nil {
log.Fatalf("Error generating keys: %v", err)
}
pubKeyBytes, err := x509.MarshalPKIXPublicKey(&privKey.PublicKey)
if err != nil {
log.Fatalf("Error marshalling public key: %v", err)
}
generateAndWriteKeys(privKey, pubKeyBytes, rsaKeyType)
}
},
}
}
func generateAndWriteKeys(privKey interface{}, pubKeyBytes []byte, keyType string) {
privFile, err := os.Create(privateKeyFile)
if err != nil {
log.Fatalf("Error creating private key file: %v", err)
}
defer privFile.Close()
var b []byte
switch privKey := privKey.(type) {
case *rsa.PrivateKey:
b = x509.MarshalPKCS1PrivateKey(privKey)
case *ecdsa.PrivateKey:
b, err = x509.MarshalECPrivateKey(privKey)
case ed25519.PrivateKey:
b, err = x509.MarshalPKCS8PrivateKey(privKey)
}
if err != nil {
log.Printf("Error marshalling private key: %v", err)
return
}
if err := pem.Encode(privFile, &pem.Block{
Type: keyType,
Bytes: b,
}); err != nil {
log.Printf("Error encoding private key: %v", err)
return
}
pubFile, err := os.Create(publicKeyFile)
if err != nil {
log.Printf("Error creating public key file: %v", err)
return
}
defer pubFile.Close()
if err := pem.Encode(pubFile, &pem.Block{
Type: publicKeyType,
Bytes: pubKeyBytes,
}); err != nil {
log.Printf("Error encoding public key: %v", err)
return
}
log.Printf("Successfully generated public/private key pair of type: %s", reflect.TypeOf(privKey).String())
}
+46
View File
@@ -0,0 +1,46 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package cli
import (
"encoding/pem"
"log"
"os"
"github.com/spf13/cobra"
)
const resultFilePath = "result.bin"
func (cli *CLI) NewResultsCmd() *cobra.Command {
return &cobra.Command{
Use: "result",
Short: "Retrieve computation result file",
Example: "result <private_key_file_path>",
Args: cobra.ExactArgs(1),
Run: func(cmd *cobra.Command, args []string) {
log.Println("Retrieving computation result file")
privKeyFile, err := os.ReadFile(args[0])
if err != nil {
log.Fatalf("Error reading private key file: %v", err)
}
pemBlock, _ := pem.Decode(privKeyFile)
var result []byte
privKey := decodeKey(pemBlock)
result, err = cli.agentSDK.Result(cmd.Context(), privKey)
if err != nil {
log.Fatalf("Error retrieving computation result: %v", err)
}
if err := os.WriteFile(resultFilePath, result, 0o644); err != nil {
log.Fatalf("Error saving computation result to %s: %v", resultFilePath, err)
}
log.Println("Computation result retrieved and saved successfully!")
},
}
}
+15
View File
@@ -0,0 +1,15 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package cli
import "github.com/ultravioletrs/cocos/pkg/sdk"
type CLI struct {
agentSDK sdk.SDK
}
func New(agentSDK sdk.SDK) *CLI {
return &CLI{
agentSDK: agentSDK,
}
}
+152
View File
@@ -0,0 +1,152 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"log"
"log/slog"
"github.com/absmach/magistrala/pkg/prometheus"
"github.com/mdlayher/vsock"
"github.com/ultravioletrs/cocos/agent"
"github.com/ultravioletrs/cocos/agent/api"
agentgrpc "github.com/ultravioletrs/cocos/agent/api/grpc"
"github.com/ultravioletrs/cocos/agent/auth"
"github.com/ultravioletrs/cocos/agent/events"
agentlogger "github.com/ultravioletrs/cocos/internal/logger"
"github.com/ultravioletrs/cocos/internal/server"
grpcserver "github.com/ultravioletrs/cocos/internal/server/grpc"
"github.com/ultravioletrs/cocos/manager"
"github.com/ultravioletrs/cocos/manager/qemu"
"golang.org/x/sync/errgroup"
"google.golang.org/grpc"
"google.golang.org/grpc/reflection"
)
const (
svcName = "agent"
defSvcGRPCPort = "7002"
)
func main() {
ctx, cancel := context.WithCancel(context.Background())
g, ctx := errgroup.WithContext(ctx)
cfg, err := readConfig()
if err != nil {
log.Fatalf("failed to read agent configuration from vsock %s", err.Error())
}
conn, err := vsock.Dial(vsock.Host, manager.ManagerVsockPort, nil)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
var level slog.Level
if err := level.UnmarshalText([]byte(cfg.AgentConfig.LogLevel)); err != nil {
log.Println(err)
return
}
handler := agentlogger.NewProtoHandler(conn, &slog.HandlerOptions{Level: level})
logger := slog.New(handler)
eventSvc, err := events.New(svcName, cfg.ID, manager.ManagerVsockPort)
if err != nil {
logger.Error(fmt.Sprintf("failed to create events service %s", err.Error()))
return
}
defer eventSvc.Close()
svc := newService(ctx, logger, eventSvc, cfg)
grpcServerConfig := server.Config{
Port: cfg.AgentConfig.Port,
Host: cfg.AgentConfig.Host,
CertFile: cfg.AgentConfig.CertFile,
KeyFile: cfg.AgentConfig.KeyFile,
ServerCAFile: cfg.AgentConfig.ServerCAFile,
ClientCAFile: cfg.AgentConfig.ClientCAFile,
AttestedTLS: cfg.AgentConfig.AttestedTls,
}
registerAgentServiceServer := func(srv *grpc.Server) {
reflection.Register(srv)
agent.RegisterAgentServiceServer(srv, agentgrpc.NewServer(svc))
}
authSvc, err := auth.New(cfg)
if err != nil {
logger.Error(fmt.Sprintf("failed to create auth service %s", err.Error()))
return
}
gs := grpcserver.New(ctx, cancel, svcName, grpcServerConfig, registerAgentServiceServer, logger, svc, authSvc)
g.Go(func() error {
return gs.Start()
})
g.Go(func() error {
return server.StopHandler(ctx, cancel, logger, svcName, gs)
})
if err := g.Wait(); err != nil {
logger.Error(fmt.Sprintf("%s service terminated: %s", svcName, err))
}
}
func newService(ctx context.Context, logger *slog.Logger, eventSvc events.Service, cmp agent.Computation) agent.Service {
svc := agent.New(ctx, logger, eventSvc, cmp)
svc = api.LoggingMiddleware(svc, logger)
counter, latency := prometheus.MakeMetrics(svcName, "api")
svc = api.MetricsMiddleware(svc, counter, latency)
return svc
}
func readConfig() (agent.Computation, error) {
l, err := vsock.Listen(qemu.VsockConfigPort, nil)
if err != nil {
return agent.Computation{}, err
}
defer l.Close()
conn, err := l.Accept()
if err != nil {
return agent.Computation{}, err
}
defer conn.Close()
var buffer []byte
for {
chunk := make([]byte, 1024)
n, err := conn.Read(chunk)
if err != nil {
if err == io.EOF {
break
}
return agent.Computation{}, err
}
buffer = append(buffer, chunk[:n]...)
}
ac := agent.Computation{
AgentConfig: agent.AgentConfig{},
}
if err := json.Unmarshal(buffer, &ac); err != nil {
return agent.Computation{}, err
}
if ac.AgentConfig.LogLevel == "" {
ac.AgentConfig.LogLevel = "info"
}
if ac.AgentConfig.Port == "" {
ac.AgentConfig.Port = defSvcGRPCPort
}
return ac, nil
}
+135
View File
@@ -0,0 +1,135 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package main
import (
"fmt"
"log"
"os"
"path"
mglog "github.com/absmach/magistrala/logger"
"github.com/caarlos0/env/v11"
"github.com/spf13/cobra"
"github.com/spf13/pflag"
"github.com/ultravioletrs/cocos/cli"
"github.com/ultravioletrs/cocos/pkg/clients/grpc"
"github.com/ultravioletrs/cocos/pkg/clients/grpc/agent"
"github.com/ultravioletrs/cocos/pkg/sdk"
)
const (
svcName = "cli"
envPrefixAgentGRPC = "AGENT_GRPC_"
completion = "completion"
filePermision = 0o755
cocosDirectory = ".cocos"
)
type config struct {
LogLevel string `env:"AGENT_LOG_LEVEL" envDefault:"info"`
}
func main() {
var cfg config
if err := env.Parse(&cfg); err != nil {
log.Fatalf("failed to load %s configuration : %s", svcName, err)
}
homePath, err := os.UserHomeDir()
if err != nil {
log.Fatalf("Error fetching user home directory: %v", err)
}
directoryCachePath := path.Join(homePath, cocosDirectory)
if err := os.MkdirAll(directoryCachePath, filePermision); err != nil {
log.Fatalf("Error while creating directory %s, error: %v", directoryCachePath, err)
}
logger, err := mglog.New(os.Stdout, cfg.LogLevel)
if err != nil {
log.Fatalf("Error creating logger: %s", err)
}
agentGRPCConfig := grpc.Config{}
if err := env.ParseWithOptions(&agentGRPCConfig, env.Options{Prefix: envPrefixAgentGRPC}); err != nil {
logger.Error(fmt.Sprintf("failed to load %s gRPC client configuration : %s", svcName, err))
return
}
agentGRPCClient, agentClient, err := agent.NewAgentClient(agentGRPCConfig)
if err != nil {
logger.Error(err.Error())
return
}
defer agentGRPCClient.Close()
agentSDK := sdk.NewAgentSDK(logger, agentClient)
cliSVC := cli.New(agentSDK)
rootCmd := &cobra.Command{
Use: "cocos-cli [command]",
Short: "CLI application for CoCos Service API",
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("CLI application for CoCos Service API\n\n")
fmt.Printf("Usage:\n %s [command]\n\n", cmd.CommandPath())
fmt.Printf("Available Commands:\n")
// Filter out "completion" command
availableCommands := make([]*cobra.Command, 0)
for _, subCmd := range cmd.Commands() {
if subCmd.Name() != completion {
availableCommands = append(availableCommands, subCmd)
}
}
for _, subCmd := range availableCommands {
fmt.Printf(" %-15s%s\n", subCmd.Name(), subCmd.Short)
}
fmt.Printf("\nFlags:\n")
cmd.Flags().VisitAll(func(flag *pflag.Flag) {
fmt.Printf(" -%s, --%s %s\n", flag.Shorthand, flag.Name, flag.Usage)
})
fmt.Printf("\nUse \"%s [command] --help\" for more information about a command.\n", cmd.CommandPath())
},
}
keysCmd := cliSVC.NewKeysCmd()
attestationCmd := cliSVC.NewAttestationCmd()
backendCmd := cliSVC.NewBackendCmd()
// Agent Commands
rootCmd.AddCommand(cliSVC.NewAlgorithmCmd())
rootCmd.AddCommand(cliSVC.NewDatasetsCmd())
rootCmd.AddCommand(cliSVC.NewResultsCmd())
rootCmd.AddCommand(attestationCmd)
rootCmd.AddCommand(cliSVC.NewFileHashCmd())
rootCmd.AddCommand(backendCmd)
rootCmd.AddCommand(keysCmd)
rootCmd.AddCommand(cliSVC.NewCABundleCmd(directoryCachePath))
// Attestation commands
attestationCmd.AddCommand(cliSVC.NewGetAttestationCmd())
attestationCmd.AddCommand(cliSVC.NewValidateAttestationValidationCmd())
// Flags
keysCmd.PersistentFlags().StringVarP(
&cli.KeyType,
"key-type",
"k",
"rsa",
"User Key type",
)
// Backend information commands
backendCmd.AddCommand(cliSVC.NewAddMeasurementCmd())
backendCmd.AddCommand(cliSVC.NewAddHostDataCmd())
if err := rootCmd.Execute(); err != nil {
logger.Error(fmt.Sprintf("Command execution failed: %s", err))
return
}
}
-208
View File
@@ -1,208 +0,0 @@
package main
import (
"context"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"strconv"
"time"
"github.com/mainflux/mainflux"
"github.com/mainflux/mainflux/logger"
"github.com/mainflux/mainflux/pkg/uuid"
"github.com/opentracing/opentracing-go"
jconfig "github.com/uber/jaeger-client-go/config"
"github.com/ultravioletrs/clients"
authapi "github.com/ultravioletrs/clients/api/grpc"
"github.com/ultravioletrs/cocos/computations"
"github.com/ultravioletrs/cocos/computations/api/api"
"github.com/ultravioletrs/cocos/computations/postgres"
"github.com/ultravioletrs/cocos/internal/db"
"github.com/ultravioletrs/cocos/internal/env"
"github.com/ultravioletrs/cocos/internal/errors"
"github.com/ultravioletrs/cocos/internal/http"
"golang.org/x/sync/errgroup"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
)
const (
defLogLevel = "debug"
defDBHost = "localhost"
defDBPort = "5432"
defDBUser = "cocos"
defDBPass = "cocos"
defDB = "computations"
defDBSSLMode = "disable"
defDBSSLCert = ""
defDBSSLKey = ""
defDBSSLRootCert = ""
defHTTPPort = "9000"
defServerCert = ""
defServerKey = ""
defJaegerURL = ""
defAuthTLS = "false"
defAuthCACerts = ""
defAuthURL = "localhost:8181"
defAuthTimeout = "1s"
envLogLevel = "COCOS_COMPUTATIONS_LOG_LEVEL"
envDBHost = "COCOS_COMPUTATIONS_DB_HOST"
envDBPort = "COCOS_COMPUTATIONS_DB_PORT"
envDBUser = "COCOS_COMPUTATIONS_DB_USER"
envDBPass = "COCOS_COMPUTATIONS_DB_PASS"
envDB = "COCOS_COMPUTATIONS_DB"
envDBSSLMode = "COCOS_COMPUTATIONS_DB_SSL_MODE"
envDBSSLCert = "COCOS_COMPUTATIONS_DB_SSL_CERT"
envDBSSLKey = "COCOS_COMPUTATIONS_DB_SSL_KEY"
envDBSSLRootCert = "COCOS_COMPUTATIONS_DB_SSL_ROOT_CERT"
envHTTPPort = "COCOS_COMPUTATIONS_HTTP_PORT"
envServerCert = "COCOS_COMPUTATIONS_SERVER_CERT"
envServerKey = "COCOS_COMPUTATIONS_SERVER_KEY"
envJaegerURL = "MF_JAEGER_URL"
envAuthTLS = "COCOS_COMPUTATIONS_AUTH_CLIENT_TLS"
envAuthCACerts = "COCOS_COMPUTATIONS_AUTH_CA_CERTS"
envAuthURL = "COCOS_COMPUTATIONS_AUTH_GRPC_URL"
envAuthTimeout = "COCOS_COMPUTATIONS_AUTH_GRPC_TIMEOUT"
)
const svcName = "Computations"
type config struct {
logLevel string
dbConfig db.Config
httpPort string
serverCert string
serverKey string
jaegerURL string
authTLS bool
authCACerts string
authURL string
authTimeout time.Duration
}
func loadConfig() config {
authTimeout, err := time.ParseDuration(mainflux.Env(envAuthTimeout, defAuthTimeout))
if err != nil {
log.Fatalf("Invalid %s value: %s", envAuthTimeout, err.Error())
}
tls, err := strconv.ParseBool(mainflux.Env(envAuthTLS, defAuthTLS))
if err != nil {
log.Fatalf("Invalid value passed for %s\n", envAuthTLS)
}
dbConfig := db.Config{
Host: env.Load(envDBHost, defDBHost),
Port: env.Load(envDBPort, defDBPort),
User: env.Load(envDBUser, defDBUser),
Pass: env.Load(envDBPass, defDBPass),
Name: env.Load(envDB, defDB),
SSLMode: env.Load(envDBSSLMode, defDBSSLMode),
SSLCert: env.Load(envDBSSLCert, defDBSSLCert),
SSLKey: env.Load(envDBSSLKey, defDBSSLKey),
SSLRootCert: env.Load(envDBSSLRootCert, defDBSSLRootCert),
Migrations: postgres.Migrations,
}
return config{
dbConfig: dbConfig,
authTLS: tls,
authTimeout: authTimeout,
logLevel: env.Load(envLogLevel, defLogLevel),
httpPort: env.Load(envHTTPPort, defHTTPPort),
serverCert: env.Load(envServerCert, defServerCert),
serverKey: env.Load(envServerKey, defServerKey),
authCACerts: env.Load(envAuthCACerts, defAuthCACerts),
jaegerURL: env.Load(envJaegerURL, defJaegerURL),
authURL: env.Load(envAuthURL, defAuthURL),
}
}
func main() {
cfg := loadConfig()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
g, ctx := errgroup.WithContext(ctx)
l, err := logger.New(os.Stdout, cfg.logLevel)
if err != nil {
log.Fatalf(err.Error())
}
db, err := db.Connect(cfg.dbConfig)
if err != nil {
log.Fatalf(err.Error())
}
authTracer, closer := initJaeger("auth", cfg.jaegerURL, l)
defer closer.Close()
auth, close := connectToAuth(cfg, authTracer, l)
if close != nil {
defer close()
}
repo := postgres.NewRepository(db)
idp := uuid.New()
svc := computations.NewService(repo, idp, auth)
svc = api.LoggingMiddleware(svc, l)
h := api.MakeHandler(svc, nil, l)
g.Go(func() error {
return http.StartServer(ctx, svcName, cfg.httpPort, cfg.serverCert, cfg.serverKey, h, l)
})
g.Go(errors.Handle(svcName, l, ctx, cancel))
if err := g.Wait(); err != nil {
l.Error(fmt.Sprintf("%s service terminated: %s", svcName, err))
}
}
func initJaeger(svcName, url string, logger logger.Logger) (opentracing.Tracer, io.Closer) {
if url == "" {
return opentracing.NoopTracer{}, ioutil.NopCloser(nil)
}
tracer, closer, err := jconfig.Configuration{
ServiceName: svcName,
Sampler: &jconfig.SamplerConfig{
Type: "const",
Param: 1,
},
Reporter: &jconfig.ReporterConfig{
LocalAgentHostPort: url,
LogSpans: true,
},
}.NewTracer()
if err != nil {
logger.Error(fmt.Sprintf("Failed to init Jaeger: %s", err))
os.Exit(1)
}
return tracer, closer
}
func connectToAuth(cfg config, tracer opentracing.Tracer, logger logger.Logger) (clients.AuthServiceClient, func() error) {
var opts []grpc.DialOption
if cfg.authTLS {
if cfg.authCACerts != "" {
tpc, err := credentials.NewClientTLSFromFile(cfg.authCACerts, "")
if err != nil {
logger.Error(fmt.Sprintf("Failed to create tls credentials: %s", err))
os.Exit(1)
}
opts = append(opts, grpc.WithTransportCredentials(tpc))
}
} else {
opts = append(opts, grpc.WithInsecure())
logger.Info("gRPC communication is not encrypted")
}
conn, err := grpc.Dial(cfg.authURL, opts...)
if err != nil {
logger.Error(fmt.Sprintf("Failed to connect to auth service: %s", err))
os.Exit(1)
}
return authapi.NewClient(tracer, conn, cfg.authTimeout), conn.Close
}
-141
View File
@@ -1,141 +0,0 @@
package main
import (
"context"
"fmt"
"log"
"os"
"strconv"
"time"
"github.com/mainflux/mainflux"
"github.com/mainflux/mainflux/logger"
"github.com/mainflux/mainflux/pkg/uuid"
"github.com/ultravioletrs/cocos/datasets"
"github.com/ultravioletrs/cocos/datasets/api"
"github.com/ultravioletrs/cocos/datasets/postgres"
"github.com/ultravioletrs/cocos/internal/db"
"github.com/ultravioletrs/cocos/internal/env"
"github.com/ultravioletrs/cocos/internal/errors"
"github.com/ultravioletrs/cocos/internal/http"
"golang.org/x/sync/errgroup"
)
const (
defLogLevel = "debug"
defDBHost = "localhost"
defDBPort = "5432"
defDBUser = "cocos"
defDBPass = "cocos"
defDB = "datasets"
defDBSSLMode = "disable"
defDBSSLCert = ""
defDBSSLKey = ""
defDBSSLRootCert = ""
defHTTPPort = "9001"
defServerCert = ""
defServerKey = ""
defAuthTLS = "false"
defAuthCACerts = ""
defAuthURL = "localhost:8181"
defAuthTimeout = "1s"
envLogLevel = "COCOS_DATASETS_LOG_LEVEL"
envDBHost = "COCOS_DATASETS_DB_HOST"
envDBPort = "COCOS_DATASETS_DB_PORT"
envDBUser = "COCOS_DATASETS_DB_USER"
envDBPass = "COCOS_DATASETS_DB_PASS"
envDB = "COCOS_DATASETS_DB"
envDBSSLMode = "COCOS_DATASETS_DB_SSL_MODE"
envDBSSLCert = "COCOS_DATASETS_DB_SSL_CERT"
envDBSSLKey = "COCOS_DATASETS_DB_SSL_KEY"
envDBSSLRootCert = "COCOS_DATASETS_DB_SSL_ROOT_CERT"
envHTTPPort = "COCOS_DATASETS_HTTP_PORT"
envServerCert = "COCOS_DATASETS_SERVER_CERT"
envServerKey = "COCOS_DATASETS_SERVER_KEY"
envAuthTLS = "COCOS_DATASETS_AUTH_CLIENT_TLS"
envAuthCACerts = "COCOS_DATASETS_AUTH_CA_CERTS"
envAuthURL = "COCOS_DATASETS_AUTH_GRPC_URL"
envAuthTimeout = "COCOS_DATASETS_AUTH_GRPC_TIMEOUT"
)
const svcName = "Datasets"
type config struct {
logLevel string
dbConfig db.Config
httpPort string
serverCert string
serverKey string
authTLS bool
authCACerts string
authURL string
authTimeout time.Duration
}
func loadConfig() config {
authTimeout, err := time.ParseDuration(mainflux.Env(envAuthTimeout, defAuthTimeout))
if err != nil {
log.Fatalf("Invalid %s value: %s", envAuthTimeout, err.Error())
}
tls, err := strconv.ParseBool(mainflux.Env(envAuthTLS, defAuthTLS))
if err != nil {
log.Fatalf("Invalid value passed for %s\n", envAuthTLS)
}
dbConfig := db.Config{
Host: env.Load(envDBHost, defDBHost),
Port: env.Load(envDBPort, defDBPort),
User: env.Load(envDBUser, defDBUser),
Pass: env.Load(envDBPass, defDBPass),
Name: env.Load(envDB, defDB),
SSLMode: env.Load(envDBSSLMode, defDBSSLMode),
SSLCert: env.Load(envDBSSLCert, defDBSSLCert),
SSLKey: env.Load(envDBSSLKey, defDBSSLKey),
SSLRootCert: env.Load(envDBSSLRootCert, defDBSSLRootCert),
Migrations: postgres.Migrations,
}
return config{
dbConfig: dbConfig,
authTLS: tls,
authTimeout: authTimeout,
logLevel: env.Load(envLogLevel, defLogLevel),
httpPort: env.Load(envHTTPPort, defHTTPPort),
serverCert: env.Load(envServerCert, defServerCert),
serverKey: env.Load(envServerKey, defServerKey),
authCACerts: env.Load(envAuthCACerts, defAuthCACerts),
authURL: env.Load(envAuthURL, defAuthURL),
}
}
func main() {
cfg := loadConfig()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
g, ctx := errgroup.WithContext(ctx)
l, err := logger.New(os.Stdout, cfg.logLevel)
if err != nil {
log.Fatalf(err.Error())
}
db, err := db.Connect(cfg.dbConfig)
if err != nil {
log.Fatalf(err.Error())
}
repo := postgres.NewRepository(db)
idp := uuid.New()
svc := datasets.NewService(repo, idp)
svc = api.LoggingMiddleware(svc, l)
h := api.MakeHandler(svc, nil, l)
g.Go(func() error {
return http.StartServer(ctx, svcName, cfg.httpPort, cfg.serverCert, cfg.serverKey, h, l)
})
g.Go(errors.Handle(svcName, l, ctx, cancel))
if err := g.Wait(); err != nil {
l.Error(fmt.Sprintf("%s service terminated: %s", svcName, err))
}
}
+132
View File
@@ -0,0 +1,132 @@
// Copyright (c) Ultraviolet
// SPDX-License-Identifier: Apache-2.0
package main
import (
"context"
"fmt"
"log"
"log/slog"
"net/url"
"os"
"strings"
mglog "github.com/absmach/magistrala/logger"
"github.com/absmach/magistrala/pkg/jaeger"
"github.com/absmach/magistrala/pkg/prometheus"
"github.com/absmach/magistrala/pkg/uuid"
"github.com/caarlos0/env/v11"
"github.com/ultravioletrs/cocos/internal"
"github.com/ultravioletrs/cocos/manager"
"github.com/ultravioletrs/cocos/manager/api"
managerapi "github.com/ultravioletrs/cocos/manager/api/grpc"
"github.com/ultravioletrs/cocos/manager/qemu"
"github.com/ultravioletrs/cocos/manager/tracing"
"github.com/ultravioletrs/cocos/pkg/clients/grpc"
managergrpc "github.com/ultravioletrs/cocos/pkg/clients/grpc/manager"
pkgmanager "github.com/ultravioletrs/cocos/pkg/manager"
"go.opentelemetry.io/otel/trace"
"golang.org/x/sync/errgroup"
)
const (
svcName = "manager"
envPrefixGRPC = "MANAGER_GRPC_"
envPrefixQemu = "MANAGER_QEMU_"
)
type config struct {
LogLevel string `env:"MANAGER_LOG_LEVEL" envDefault:"info"`
JaegerURL url.URL `env:"COCOS_JAEGER_URL" envDefault:"http://localhost:4318"`
TraceRatio float64 `env:"MG_JAEGER_TRACE_RATIO" envDefault:"1.0"`
InstanceID string `env:"MANAGER_INSTANCE_ID" envDefault:""`
}
func main() {
ctx, cancel := context.WithCancel(context.Background())
g, ctx := errgroup.WithContext(ctx)
var cfg config
if err := env.Parse(&cfg); err != nil {
log.Fatalf("failed to load %s configuration : %s", svcName, err)
}
logger, err := mglog.New(os.Stdout, cfg.LogLevel)
if err != nil {
log.Fatalf(err.Error())
}
if cfg.InstanceID == "" {
if cfg.InstanceID, err = uuid.New().ID(); err != nil {
logger.Error(fmt.Sprintf("Failed to generate instance ID: %s", err))
return
}
}
tp, err := jaeger.NewProvider(ctx, svcName, cfg.JaegerURL, cfg.InstanceID, cfg.TraceRatio)
if err != nil {
logger.Error(fmt.Sprintf("Failed to init Jaeger: %s", err))
}
defer func() {
if err := tp.Shutdown(ctx); err != nil {
logger.Error(fmt.Sprintf("Error shutting down tracer provider: %v", err))
}
}()
tracer := tp.Tracer(svcName)
qemuCfg := qemu.Config{}
if err := env.ParseWithOptions(&qemuCfg, env.Options{Prefix: envPrefixQemu}); err != nil {
logger.Error(fmt.Sprintf("failed to load QEMU configuration: %s", err))
return
}
args := qemuCfg.ConstructQemuArgs()
logger.Info(strings.Join(args, " "))
managerGRPCConfig := grpc.Config{}
if err := env.ParseWithOptions(&managerGRPCConfig, env.Options{Prefix: envPrefixGRPC}); err != nil {
logger.Error(fmt.Sprintf("failed to load %s gRPC client configuration : %s", svcName, err))
return
}
managerGRPCClient, managerClient, err := managergrpc.NewManagerClient(managerGRPCConfig)
if err != nil {
logger.Error(err.Error())
return
}
defer managerGRPCClient.Close()
pc, err := managerClient.Process(ctx)
if err != nil {
logger.Error(err.Error())
return
}
eventsChan := make(chan *pkgmanager.ClientStreamMessage)
svc := newService(logger, tracer, qemuCfg, eventsChan)
mc := managerapi.NewClient(pc, svc, eventsChan)
g.Go(func() error {
return mc.Process(ctx, cancel)
})
if err := g.Wait(); err != nil {
logger.Error(fmt.Sprintf("%s service terminated: %s", svcName, err))
}
if err = internal.DeleteFilesInDir(qemuCfg.TmpFileLoc); err != nil {
logger.Error(err.Error())
}
}
func newService(logger *slog.Logger, tracer trace.Tracer, qemuCfg qemu.Config, eventsChan chan *pkgmanager.ClientStreamMessage) manager.Service {
svc := manager.New(qemuCfg, logger, eventsChan, qemu.NewVM)
go svc.RetrieveAgentEventsLogs()
svc = api.LoggingMiddleware(svc, logger)
counter, latency := prometheus.MakeMetrics(svcName, "api")
svc = api.MetricsMiddleware(svc, counter, latency)
svc = tracing.New(svc, tracer)
return svc
}
+270
View File
@@ -0,0 +1,270 @@
#!/bin/bash
#
# user changeable parameters
#
HDA_FILE="cmd/manager/img/focal-server-cloudimg-amd64.qcow2"
GUEST_SIZE_IN_MB="4096"
SEV_GUEST="1"
SMP_NCPUS="4"
CONSOLE="serial"
VNC_PORT=""
USE_VIRTIO="1"
UEFI_BIOS_CODE="/usr/share/OVMF/OVMF_CODE.fd"
UEFI_BIOS_VARS_ORIG="/usr/share/OVMF/OVMF_VARS.fd"
UEFI_BIOS_VARS_COPY="cmd/manager/img/OVMF_VARS.fd"
CBITPOS=51
HOST_HTTP_PORT=9301
GUEST_HTTP_PORT=9031
HOST_GRPC_PORT=7020
GUEST_GRPC_PORT=7002
ENABLE_FILE_LOG="0"
EXEC_QEMU_CMDLINE="0"
usage() {
echo "$0 [options]"
echo "Available <commands>:"
echo " -hda hard disk ($HDA_FILE)"
echo " -nosev disable sev support"
echo " -mem guest memory"
echo " -smp number of cpus"
echo " -console display console to use (serial or gxl)"
echo " -vnc VNC port to use"
echo " -bios bios to use (default $UEFI_BIOS_CODE)"
echo " -kernel kernel to use"
echo " -initrd initrd to use"
echo " -cdrom CDROM image"
echo " -virtio use virtio devices"
echo " -cbitpos location of the C-bit"
echo " -hosthttp host http port"
echo " -guesthttp guest http port"
echo " -hostgrpc host grpc port"
echo " -guestgrpc guest grpc port"
echo " -origuefivars UEFI BIOS vars original file (default $UEFI_BIOS_VARS_ORIG)"
echo " -copyuefivars UEFI BIOS vars copy file (default $UEFI_BIOS_VARS_COPY)"
echo " -exec execute the QEMU command (default $EXEC_QEMU_CMDLINE)"
echo " -filelog enable/disable QEMU cmd line file log (default: $ENABLE_FILE_LOG)"
exit 1
}
while [[ $1 != "" ]]; do
case "$1" in
-hda)
HDA_FILE=${2}
shift
;;
-nosev)
SEV_GUEST="0"
;;
-mem)
GUEST_SIZE_IN_MB=${2}
shift
;;
-console)
CONSOLE=${2}
shift
;;
-smp)
SMP_NCPUS=$2
shift
;;
-vnc)
VNC_PORT=$2
shift
;;
-bios)
UEFI_BIOS_CODE=$2
shift
;;
-initrd)
INITRD_FILE=$2
shift
;;
-kernel)
KERNEL_FILE=$2
shift
;;
-cdrom)
CDROM_FILE=$2
shift
;;
-virtio)
USE_VIRTIO="1"
;;
-cbitpos)
CBITPOS=$2
shift
;;
-hosthttp)
HOST_HTTP_PORT=$2
shift
;;
-guesthttp)
GUEST_HTTP_PORT=$2
shift
;;
-guestgrpc)
GUEST_GRPC_PORT=$2
shift
;;
-hostgrpc)
HOST_GRPC_PORT=$2
shift
;;
-origuefivars)
UEFI_BIOS_VARS_ORIG=$2
shift
;;
-copyuefivars)
UEFI_BIOS_VARS_COPY=$2
shift
;;
-exec)
EXEC_QEMU_CMDLINE="1"
;;
-filelog)
ENABLE_FILE_LOG="1"
;;
*)
usage;;
esac
shift
done
#
# func definitions
#
add_opts() {
echo -n "$* " >> ${QEMU_CMDLINE}
}
run_cmd() {
if ! "$@"; then
echo "Command '$*' failed"
exit 1
fi
}
# copy BIOS variables to new dest for VM use without modifying the original ones
cp "$UEFI_BIOS_VARS_ORIG" "$UEFI_BIOS_VARS_COPY"
#
# Qemu cmd line construction
#
# we add all the qemu command line options into a file
QEMU_CMDLINE=/tmp/cmdline.$$
rm -rf ${QEMU_CMDLINE}
add_opts "$(which qemu-system-x86_64)"
# Basic virtual machine property
add_opts "-enable-kvm -cpu EPYC -machine q35"
# add number of VCPUs
[ -n "$SMP_NCPUS" ] && add_opts "-smp ${SMP_NCPUS},maxcpus=64"
# define guest memory
add_opts "-m ${GUEST_SIZE_IN_MB}M,slots=5,maxmem=30G"
# The OVMF binary, including the non-volatile variable store, appears as a
# "normal" qemu drive on the host side, and it is exposed to the guest as a
# persistent flash device.
add_opts "-drive if=pflash,format=raw,unit=0,file=${UEFI_BIOS_CODE},readonly=on"
add_opts "-drive if=pflash,format=raw,unit=1,file=${UEFI_BIOS_VARS_COPY}"
# add CDROM if specified
[ -n "$CDROM_FILE" ] && add_opts "-drive file=${CDROM_FILE},media=cdrom -boot d"
add_opts "-netdev user,id=vmnic,hostfwd=tcp::2222-:22,hostfwd=tcp::$HOST_HTTP_PORT-:$GUEST_HTTP_PORT,hostfwd=tcp::$HOST_GRPC_PORT-:$GUEST_GRPC_PORT"
add_opts "-device virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=vmnic,romfile="
# If harddisk file is specified then add the HDD drive
if [ -n "$HDA_FILE" ]; then
if [ "$USE_VIRTIO" = "1" ]; then
if [[ ${HDA_FILE} = *"qcow2" ]]; then
add_opts "-drive file=${HDA_FILE},if=none,id=disk0,format=qcow2"
else
add_opts "-drive file=${HDA_FILE},if=none,id=disk0,format=raw"
fi
add_opts "-device virtio-scsi-pci,id=scsi,disable-legacy=on,iommu_platform=true"
add_opts "-device scsi-hd,drive=disk0"
else
if [[ ${HDA_FILE} = *"qcow2" ]]; then
add_opts "-drive file=${HDA_FILE},format=qcow2"
else
add_opts "-drive file=${HDA_FILE},format=raw"
fi
fi
fi
# If this is SEV guest then add the encryption device objects to enable support
if [ ${SEV_GUEST} = "1" ]; then
add_opts "-object sev-guest,id=sev0,cbitpos=${CBITPOS},reduced-phys-bits=1"
add_opts "-machine memory-encryption=sev0"
fi
# if console is serial then disable graphical interface
if [ "${CONSOLE}" = "serial" ]; then
add_opts "-nographic"
else
add_opts "-vga ${CONSOLE}"
fi
# if -kernel arg is specified then use the kernel provided in command line for boot
if [ "${KERNEL_FILE}" != "" ]; then
add_opts "-kernel $KERNEL_FILE"
add_opts "-append \"console=ttyS0 earlyprintk=serial root=/dev/sda2\""
[ -n "$INITRD_FILE" ] && add_opts "-initrd ${INITRD_FILE}"
fi
# start vnc server
[ -n "$VNC_PORT" ] && add_opts "-vnc :${VNC_PORT}" && echo "Starting VNC on port ${VNC_PORT}"
# start monitor on pty
add_opts "-monitor pty"
#
# Qemu cmd line log
#
# Set the log file path if ENABLE_FILE_LOG is 1
if [ "$ENABLE_FILE_LOG" = "1" ]; then
LOG_FILE=$(pwd)/stdout.log
# Save the command line args into log file
cat "$QEMU_CMDLINE" > "$LOG_FILE"
echo >> "$LOG_FILE"
fi
# Log the command line to the console
cat "$QEMU_CMDLINE"
#
# Qemu cmd line execution
#
if [[ "${EXEC_QEMU_CMDLINE}" = "0" ]]; then
exit 0
fi
# map CTRL-C to CTRL ]
echo "Mapping CTRL-C to CTRL-]"
stty intr ^]
echo "Launching VM ..."
if [ "$ENABLE_FILE_LOG" = "1" ]; then
bash ${QEMU_CMDLINE} 2>&1 | tee -a "${LOG_FILE}"
else
bash ${QEMU_CMDLINE} 2>&1
fi
# restore the mapping
stty intr ^c
rm -rf ${QEMU_CMDLINE}
+62
View File
@@ -0,0 +1,62 @@
#!/bin/bash
# Set your default values for sudo and sev
sudo_option=false
sev_option=false
# Parse command line arguments
while [[ $# -gt 0 ]]; do
key="$1"
case $key in
--sudo)
sudo_option=true
shift
;;
--sev)
sev_option=true
shift
;;
*)
echo "Unknown option: $key"
exit 1
;;
esac
done
build_qemu_command() {
local qemu_command="/usr/bin/qemu-system-x86_64 -enable-kvm -machine q35 -cpu EPYC -smp 4,maxcpus=64 -m 2048M,slots=5,maxmem=30G -drive if=pflash,format=raw,unit=0,file=$MANAGER_QEMU_OVMF_CODE_FILE,readonly=on -drive if=pflash,format=raw,unit=1,file=img/OVMF_VARS.fd -device virtio-scsi-pci,id=scsi,disable-legacy=on,iommu_platform=true -drive file=img/focal-server-cloudimg-amd64.img,if=none,id=disk0,format=qcow2 -device scsi-hd,drive=disk0 -netdev user,id=vmnic,hostfwd=tcp::2222-:22,hostfwd=tcp::9301-:9031,hostfwd=tcp::7020-:7002 -device virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=vmnic,romfile= -nographic -monitor pty"
if [ "$sev_option" = true ]; then
qemu_command="$qemu_command -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0"
fi
echo "$qemu_command"
}
if [ ! -f "img/OVMF_VARS.fd" ]; then
cp "$MANAGER_QEMU_OVMF_VARS_FILE" "img/OVMF_VARS.fd"
echo "Copied $MANAGER_QEMU_OVMF_VARS_FILE to img/OVMF_VARS.fd"
else
echo "img/OVMF_VARS.fd already exists. No need to copy."
fi
echo "Launching VM ..."
qemu_command=$(build_qemu_command)
echo "$qemu_command"
echo "Mapping CTRL-C to CTRL-]"
stty intr ^]
if [ "$sudo_option" = true ]; then
# Split the command and arguments into an array; << operator is known as a "here string"
IFS=" " read -r -a qemu_command_array <<< "$qemu_command"
# Treat each element in the array as a separate word, preserving spaces within each element
sudo "${qemu_command_array[@]}"
else
$qemu_command
fi
# Restore the mapping
stty intr ^c
+62
View File
@@ -0,0 +1,62 @@
<domain type="kvm">
<name>QEmu-alpine-standard-x86_64</name>
<uuid>c7a5fdbd-cdaf-9455-926a-d65c16db1809</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://alpinelinux.org/alpinelinux/3.15"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">4194304</memory>
<currentMemory unit="KiB">4194304</currentMemory>
<vcpu placement="static">1</vcpu>
<os>
<type arch="x86_64" machine="q35">hvm</type>
<bootmenu enable="yes"/>
<loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE.fd</loader>
<nvram template='/usr/share/OVMF/OVMF_VARS.fd'>./img/OVMF_VARS.fd</nvram>
<!-- <boot dev='hd'/> -->
</os>
<features>
<acpi/>
<apic/>
<vmport state="off"/>
</features>
<cpu mode="host-passthrough" check="none" migratable="on"/>
<clock offset="utc">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2" discard="unmap"/>
<source file="./img/focal-server-cloudimg-amd64.qcow2"/>
<target dev="vda" bus="virtio"/>
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
<boot order="1"/>
</disk>
<graphics type="spice" autoport="yes">
<listen type="address"/>
<image compression="off"/>
</graphics>
<video>
<model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
</video>
<interface type="network">
<mac address="52:54:00:03:7b:5f"/>
<source network="default"/>
<model type="virtio"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
</devices>
</domain>
+6
View File
@@ -0,0 +1,6 @@
<pool type="dir">
<name>virtimages</name>
<target>
<path>./img</path>
</target>
</pool>
+15
View File
@@ -0,0 +1,15 @@
<volume>
<name>boot.img</name>
<allocation>0</allocation>
<capacity unit="G">1</capacity>
<target>
<format type="qcow2"/>
<path>./img/boot.img</path>
<permissions>
<owner>107</owner>
<group>107</group>
<mode>0744</mode>
<label>virt_image_t</label>
</permissions>
</target>
</volume>
-4
View File
@@ -1,4 +0,0 @@
// Package api contains API-related concerns: endpoint
// definitions, middlewares, and all resource representations.
package api
-106
View File
@@ -1,106 +0,0 @@
package api
import (
"context"
"github.com/go-kit/kit/endpoint"
"github.com/mainflux/mainflux/pkg/errors"
"github.com/ultravioletrs/cocos/computations"
)
func create(svc computations.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(createReq)
if err := req.validate(); err != nil {
return createRes{}, err
}
cid, err := svc.CreateComputation(ctx, req.token, req.computation)
if err != nil {
return createRes{}, err
}
ret := createRes{
ID: cid,
created: true,
}
return ret, nil
}
}
func view(svc computations.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(viewReq)
if err := req.validate(); err != nil {
return viewRes{}, err
}
computation, err := svc.ViewComputation(ctx, req.token, req.id)
if err != nil {
return createRes{}, err
}
ret := viewRes{
Computation: computation,
}
return ret, nil
}
}
func list(svc computations.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(listReq)
if err := req.validate(); err != nil {
return listRes{}, err
}
res, err := svc.ListComputations(ctx, req.token, req.meta)
if err != nil {
return listRes{}, err
}
ret := listRes{
Page: res,
}
return ret, nil
}
}
func update(svc computations.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(updateReq)
if err := req.validate(); err != nil {
return nil, err
}
computation := computations.Computation{
ID: req.id,
Name: req.Name,
Description: req.Description,
Metadata: req.Metadata,
}
if err := svc.UpdateComputation(ctx, req.token, computation); err != nil {
return nil, err
}
res := createRes{ID: req.id, created: false}
return res, nil
}
}
func remove(svc computations.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(viewReq)
err := req.validate()
if err == errors.ErrNotFound {
return removeRes{}, nil
}
if err != nil {
return nil, err
}
if err := svc.RemoveComputation(ctx, req.token, req.id); err != nil {
return nil, err
}
return removeRes{}, nil
}
}
-84
View File
@@ -1,84 +0,0 @@
//go:build !test
package api
import (
"context"
"fmt"
"time"
log "github.com/mainflux/mainflux/logger"
"github.com/ultravioletrs/cocos/computations"
)
var _ computations.Service = (*loggingMiddleware)(nil)
type loggingMiddleware struct {
logger log.Logger
svc computations.Service
}
// LoggingMiddleware adds logging facilities to the core service.
func LoggingMiddleware(svc computations.Service, logger log.Logger) computations.Service {
return &loggingMiddleware{logger, svc}
}
func (lm *loggingMiddleware) CreateComputation(ctx context.Context, token string, computation computations.Computation) (id string, err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method create_computation for user %s took %s to complete", token, time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s.", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors.", message))
}(time.Now())
return lm.svc.CreateComputation(ctx, token, computation)
}
func (lm *loggingMiddleware) ViewComputation(ctx context.Context, token, id string) (c computations.Computation, err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method view_computation for user %s took %s to complete", token, time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s.", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors.", message))
}(time.Now())
return lm.svc.ViewComputation(ctx, token, id)
}
func (lm *loggingMiddleware) ListComputations(ctx context.Context, token string, meta computations.PageMetadata) (page computations.Page, err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method list_computations for user %s took %s to complete", token, time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s.", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors.", message))
}(time.Now())
return lm.svc.ListComputations(ctx, token, meta)
}
func (lm *loggingMiddleware) UpdateComputation(ctx context.Context, token string, computation computations.Computation) (err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method update_computation for user %s took %s to complete", token, time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s.", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors.", message))
}(time.Now())
return lm.svc.UpdateComputation(ctx, token, computation)
}
func (lm *loggingMiddleware) RemoveComputation(ctx context.Context, token, id string) (err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method remove_computation for user %s took %s to complete", token, time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s.", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors.", message))
}(time.Now())
return lm.svc.RemoveComputation(ctx, token, id)
}
-68
View File
@@ -1,68 +0,0 @@
//go:build !test
package api
import (
"context"
"time"
"github.com/go-kit/kit/metrics"
"github.com/ultravioletrs/cocos/computations"
)
var _ computations.Service = (*metricsMiddleware)(nil)
type metricsMiddleware struct {
counter metrics.Counter
latency metrics.Histogram
svc computations.Service
}
// MetricsMiddleware instruments core service by tracking request count and latency.
func MetricsMiddleware(svc computations.Service, counter metrics.Counter, latency metrics.Histogram) computations.Service {
return &metricsMiddleware{
counter: counter,
latency: latency,
svc: svc,
}
}
func (ms *metricsMiddleware) CreateComputation(ctx context.Context, token string, computation computations.Computation) (string, error) {
defer func(begin time.Time) {
ms.counter.With("method", "create_computation").Add(1)
ms.latency.With("method", "create_computation").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.CreateComputation(ctx, token, computation)
}
func (ms *metricsMiddleware) ViewComputation(ctx context.Context, token, id string) (computations.Computation, error) {
defer func(begin time.Time) {
ms.counter.With("method", "view_computation").Add(1)
ms.latency.With("method", "view_computation").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.ViewComputation(ctx, token, id)
}
func (ms *metricsMiddleware) ListComputations(ctx context.Context, token string, meta computations.PageMetadata) (page computations.Page, err error) {
defer func(begin time.Time) {
ms.counter.With("method", "list_computations").Add(1)
ms.latency.With("method", "list_computations").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.ListComputations(ctx, token, meta)
}
func (ms *metricsMiddleware) UpdateComputation(ctx context.Context, token string, computation computations.Computation) error {
defer func(begin time.Time) {
ms.counter.With("method", "update_computations").Add(1)
ms.latency.With("method", "update_computations").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.UpdateComputation(ctx, token, computation)
}
func (ms *metricsMiddleware) RemoveComputation(ctx context.Context, token, id string) error {
defer func(begin time.Time) {
ms.counter.With("method", "remove_computation").Add(1)
ms.latency.With("method", "remove_computation").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.RemoveComputation(ctx, token, id)
}
-42
View File
@@ -1,42 +0,0 @@
package api
import "github.com/ultravioletrs/cocos/computations"
type createReq struct {
computation computations.Computation
token string
}
func (req createReq) validate() error {
return req.computation.Validate()
}
type viewReq struct {
token string
id string
}
func (req viewReq) validate() error {
return nil
}
type listReq struct {
token string
meta computations.PageMetadata
}
func (req listReq) validate() error {
return nil
}
type updateReq struct {
token string
id string
Name string `json:"name,omitempty"`
Description string `json:"description,omitempty"`
Metadata map[string]interface{} `json:"metadata,omitempty"`
}
func (req updateReq) validate() error {
return nil
}
-85
View File
@@ -1,85 +0,0 @@
package api
import (
"fmt"
"net/http"
"github.com/ultravioletrs/cocos/computations"
)
type createRes struct {
ID string
created bool
}
func (res createRes) Code() int {
if res.created {
return http.StatusCreated
}
return http.StatusOK
}
func (res createRes) Headers() map[string]string {
if res.created {
return map[string]string{
"Location": fmt.Sprintf("/computations/%s", res.ID),
}
}
return map[string]string{}
}
func (res createRes) Empty() bool {
return true
}
type viewRes struct {
computations.Computation
}
func (res viewRes) Code() int {
return http.StatusOK
}
func (res viewRes) Headers() map[string]string {
return map[string]string{}
}
func (res viewRes) Empty() bool {
return false
}
type errorRes struct {
Err string `json:"error"`
}
type listRes struct {
computations.Page
}
func (res listRes) Code() int {
return http.StatusOK
}
func (res listRes) Headers() map[string]string {
return map[string]string{}
}
func (res listRes) Empty() bool {
return false
}
type removeRes struct{}
func (res removeRes) Code() int {
return http.StatusNoContent
}
func (res removeRes) Headers() map[string]string {
return map[string]string{}
}
func (res removeRes) Empty() bool {
return true
}
-201
View File
@@ -1,201 +0,0 @@
package api
import (
"context"
"encoding/json"
"net/http"
"strings"
kithttp "github.com/go-kit/kit/transport/http"
"github.com/go-zoo/bone"
"github.com/mainflux/mainflux"
log "github.com/mainflux/mainflux/logger"
"github.com/mainflux/mainflux/pkg/errors"
opentracing "github.com/opentracing/opentracing-go"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/ultravioletrs/cocos/computations"
"github.com/ultravioletrs/cocos/internal/apiutil"
httputil "github.com/ultravioletrs/cocos/internal/http"
)
const (
contentType = "application/json"
offsetKey = "offset"
limitKey = "limit"
orderKey = "order"
dirKey = "dir"
metadataKey = "metadata"
defOffset = 0
defLimit = 10
bearerPrefix = "Bearer "
)
// MakeHandler returns a HTTP handler for API endpoints.
func MakeHandler(svc computations.Service, tracer opentracing.Tracer, logger log.Logger) http.Handler {
opts := []kithttp.ServerOption{
kithttp.ServerErrorEncoder(apiutil.LoggingErrorEncoder(logger, encodeError)),
}
mux := bone.New()
mux.Post("/computations", kithttp.NewServer(
// kitot.TraceServer(tracer, "register")(createComputation(svc)),
create(svc),
decodeCreate,
encodeResponse,
opts...,
))
mux.Get("/computations", kithttp.NewServer(
// kitot.TraceServer(tracer, "register")(createComputation(svc)),
list(svc),
decodeList,
encodeResponse,
opts...,
))
mux.Get("/computations/:id", kithttp.NewServer(
view(svc),
decodeView,
encodeResponse,
opts...,
))
mux.Put("/computations/:id", kithttp.NewServer(
update(svc),
decodeUpdate,
encodeResponse,
opts...,
))
mux.Delete("/computations/:id", kithttp.NewServer(
remove(svc),
decodeView,
encodeResponse,
opts...,
))
mux.GetFunc("/health", httputil.Health("computations", "Computation management service"))
mux.Handle("/metrics", promhttp.Handler())
return mux
}
func decodeCreate(_ context.Context, r *http.Request) (interface{}, error) {
req := createReq{
token: extractBearerToken(r),
}
if err := json.NewDecoder(r.Body).Decode(&req.computation); err != nil {
return nil, errors.Wrap(errors.ErrMalformedEntity, err)
}
return req, nil
}
func decodeList(_ context.Context, r *http.Request) (interface{}, error) {
o, err := apiutil.ReadUintQuery(r, offsetKey, defOffset)
if err != nil {
return nil, err
}
l, err := apiutil.ReadUintQuery(r, limitKey, defLimit)
if err != nil {
return nil, err
}
or, err := apiutil.ReadStringQuery(r, orderKey, "")
if err != nil {
return nil, err
}
d, err := apiutil.ReadStringQuery(r, dirKey, "")
if err != nil {
return nil, err
}
m, err := apiutil.ReadMetadataQuery(r, metadataKey, nil)
if err != nil {
return nil, err
}
req := listReq{
token: extractBearerToken(r),
meta: computations.PageMetadata{
Offset: o,
Limit: l,
Order: or,
Dir: d,
Metadata: m,
},
}
return req, nil
}
func decodeView(_ context.Context, r *http.Request) (interface{}, error) {
req := viewReq{
token: extractBearerToken(r),
id: bone.GetValue(r, "id"),
}
return req, nil
}
func decodeUpdate(_ context.Context, r *http.Request) (interface{}, error) {
if !strings.Contains(r.Header.Get("Content-Type"), contentType) {
return nil, errors.ErrUnsupportedContentType
}
req := updateReq{
token: extractBearerToken(r),
id: bone.GetValue(r, "id"),
}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
return nil, errors.Wrap(errors.ErrMalformedEntity, err)
}
return req, nil
}
func encodeResponse(_ context.Context, w http.ResponseWriter, response interface{}) error {
if ar, ok := response.(mainflux.Response); ok {
for k, v := range ar.Headers() {
w.Header().Set(k, v)
}
w.Header().Set("Content-Type", contentType)
w.WriteHeader(ar.Code())
if ar.Empty() {
return nil
}
}
return json.NewEncoder(w).Encode(response)
}
func encodeError(_ context.Context, err error, w http.ResponseWriter) {
switch {
case errors.Contains(err, errors.ErrInvalidQueryParams),
errors.Contains(err, errors.ErrMalformedEntity):
w.WriteHeader(http.StatusBadRequest)
case errors.Contains(err, errors.ErrCreateEntity),
errors.Contains(err, errors.ErrUpdateEntity),
errors.Contains(err, errors.ErrViewEntity),
errors.Contains(err, errors.ErrRemoveEntity):
w.WriteHeader(http.StatusInternalServerError)
default:
w.WriteHeader(http.StatusInternalServerError)
}
if errorVal, ok := err.(errors.Error); ok {
w.Header().Set("Content-Type", contentType)
if err := json.NewEncoder(w).Encode(errorRes{Err: errorVal.Msg()}); err != nil {
w.WriteHeader(http.StatusInternalServerError)
}
}
}
func extractBearerToken(r *http.Request) string {
token := r.Header.Get("Authorization")
if !strings.HasPrefix(token, bearerPrefix) {
return ""
}
return strings.TrimPrefix(token, bearerPrefix)
}
-51
View File
@@ -1,51 +0,0 @@
package computations
import (
"context"
"time"
)
type Computation struct {
ID string `json:"id,omitempty" db:"id"`
Name string `json:"name,omitempty" db:"name"`
Description string `json:"description,omitempty" db:"description"`
Status string `json:"status,omitempty" db:"status"`
Owner string `json:"owner,omitempty" db:"owner"`
StartTime time.Time `json:"start_time,omitempty" db:"start_time"`
EndTime time.Time `json:"end_time,omitempty" db:"end_time"`
Datasets []string `json:"datasets,omitempty" db:"datasets"`
Algorithms []string `json:"algorithms,omitempty" db:"algorithms"`
DatasetProviders []string `json:"dataset_providers,omitempty" db:"dataset_providers"`
AlgorithmProviders []string `json:"algorithm_providers,omitempty" db:"algorithm_providers"`
Ttl int `json:"ttl,omitempty" db:"ttl"`
Metadata Metadata `json:"metadata,omitempty" db:"metadata"`
}
type Metadata map[string]interface{}
type PageMetadata struct {
Total uint64 `json:"total,omitempty"`
Offset uint64 `json:"offset,omitempty"`
Limit uint64 `json:"limit,omitempty"`
Name string `json:"name,omitempty"`
Order string `json:"order,omitempty"`
Dir string `json:"dir,omitempty"`
Metadata Metadata `json:"metadata,omitempty"`
}
type Page struct {
PageMetadata
Computations []Computation `json:"computations,omitempty"`
}
func (c Computation) Validate() error {
return nil
}
type Repository interface {
Save(ctx context.Context, c Computation) (string, error)
View(ctx context.Context, id string) (Computation, error)
RetrieveAll(ctx context.Context, owner string, pm PageMetadata) (Page, error)
Update(ctx context.Context, c Computation) error
Delete(ctx context.Context, id string) error
}
-4
View File
@@ -1,4 +0,0 @@
// Package computations contains domain concept definitions
// needed to support Computations service functionality.
package computations
-355
View File
@@ -1,355 +0,0 @@
package postgres
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"strings"
"time"
"github.com/lib/pq"
"github.com/mainflux/mainflux/pkg/errors"
"github.com/ultravioletrs/cocos/computations"
db "github.com/ultravioletrs/cocos/internal/db"
)
const (
errInvalid = "invalid_text_representation"
errTruncation = "string_data_right_truncation"
errDuplicate = "unique_violation"
)
var _ computations.Repository = (*computationRepo)(nil)
type computationRepo struct {
db db.Database
}
func NewRepository(db db.Database) computations.Repository {
return computationRepo{
db: db,
}
}
func (repo computationRepo) Save(ctx context.Context, c computations.Computation) (string, error) {
q := `INSERT INTO computations (id, name, description, status, owner, start_time, end_time, datasets, algorithms, dataset_providers, algorithm_providers, ttl, metadata)
VALUES (:id, :name, :description, :status, :owner, :start_time, :end_time, :datasets, :algorithms, :dataset_providers, :algorithm_providers, :ttl, :metadata) RETURNING id`
comp, err := fromComputation(c)
if err != nil {
return "", err
}
row, err := repo.db.NamedQueryContext(ctx, q, comp)
if err != nil {
pqErr, ok := err.(*pq.Error)
if ok {
switch pqErr.Code.Name() {
case errInvalid, errTruncation:
return "", errors.Wrap(errors.ErrMalformedEntity, err)
case errDuplicate:
return "", errors.Wrap(errors.ErrConflict, err)
}
}
return "", errors.Wrap(errors.ErrCreateEntity, err)
}
defer row.Close()
row.Next()
var id string
if err := row.Scan(&id); err != nil {
return "", err
}
return id, nil
}
func (repo computationRepo) View(ctx context.Context, id string) (computations.Computation, error) {
q := `SELECT id, name, description, status, owner, start_time, end_time, datasets,
algorithms, dataset_providers, algorithm_providers, ttl, metadata
FROM computations
WHERE id = $1`
c := dbComputation{
ID: id,
}
if err := repo.db.QueryRowxContext(ctx, q, id).StructScan(&c); err != nil {
if err == sql.ErrNoRows {
return computations.Computation{}, errors.Wrap(errors.ErrNotFound, err)
}
return computations.Computation{}, errors.Wrap(errors.ErrViewEntity, err)
}
return toComputation(c)
}
func (repo computationRepo) RetrieveAll(ctx context.Context, owner string, pm computations.PageMetadata) (computations.Page, error) {
nq, name := getNameQuery(pm.Name)
oq := getOrderQuery(pm.Order)
dq := getDirQuery(pm.Dir)
m, mq, err := getMetadataQuery(pm.Metadata)
if err != nil {
return computations.Page{}, errors.Wrap(errors.ErrViewEntity, err)
}
var query []string
if mq != "" {
query = append(query, mq)
}
if nq != "" {
query = append(query, nq)
}
var whereClause string
if len(query) > 0 {
whereClause = fmt.Sprintf(" WHERE %s", strings.Join(query, " AND "))
}
q := `SELECT id, name, description, status, owner, start_time, end_time, datasets,
algorithms, dataset_providers, algorithm_providers, ttl, metadata
FROM computations %s
ORDER BY %s %s LIMIT :limit OFFSET :offset`
q = fmt.Sprintf(q, whereClause, oq, dq)
params := map[string]interface{}{
"owner": owner,
"limit": pm.Limit,
"offset": pm.Offset,
"name": name,
"metadata": m,
}
rows, err := repo.db.NamedQueryContext(ctx, q, params)
if err != nil {
return computations.Page{}, errors.Wrap(errors.ErrViewEntity, err)
}
defer rows.Close()
var items []computations.Computation
for rows.Next() {
dbc := dbComputation{Owner: owner}
if err := rows.StructScan(&dbc); err != nil {
return computations.Page{}, errors.Wrap(errors.ErrViewEntity, err)
}
c, err := toComputation(dbc)
if err != nil {
return computations.Page{}, err
}
items = append(items, c)
}
cq := fmt.Sprintf(`SELECT COUNT(*) FROM computations %s;`, whereClause)
total, err := repo.total(ctx, cq, params)
if err != nil {
return computations.Page{}, errors.Wrap(errors.ErrViewEntity, err)
}
page := computations.Page{
Computations: items,
PageMetadata: computations.PageMetadata{
Total: total,
Offset: pm.Offset,
Limit: pm.Limit,
Order: pm.Order,
Dir: pm.Dir,
},
}
return page, nil
}
func (repo computationRepo) Update(ctx context.Context, c computations.Computation) error {
q := `UPDATE computations SET name = :name, metadata = :metadata, description = :description WHERE id = :id;`
dbcpt, err := toDBComputation(c)
if err != nil {
return errors.Wrap(errors.ErrUpdateEntity, err)
}
res, errdb := repo.db.NamedExecContext(ctx, q, dbcpt)
if errdb != nil {
pqErr, ok := errdb.(*pq.Error)
if ok {
switch pqErr.Code.Name() {
case errInvalid, errTruncation:
return errors.Wrap(errors.ErrMalformedEntity, errdb)
}
}
return errors.Wrap(errors.ErrUpdateEntity, errdb)
}
cnt, errdb := res.RowsAffected()
if errdb != nil {
return errors.Wrap(errors.ErrUpdateEntity, errdb)
}
if cnt == 0 {
return errors.ErrNotFound
}
return nil
}
func (repo computationRepo) Delete(ctx context.Context, id string) error {
q := `DELETE FROM computations WHERE id = :id`
c := dbComputation{
ID: id,
}
if _, err := repo.db.NamedExecContext(ctx, q, c); err != nil {
if err == sql.ErrNoRows {
return errors.Wrap(errors.ErrNotFound, err)
}
return errors.Wrap(errors.ErrViewEntity, err)
}
return nil
}
func (repo computationRepo) total(ctx context.Context, query string, params interface{}) (uint64, error) {
rows, err := repo.db.NamedQueryContext(ctx, query, params)
if err != nil {
return 0, err
}
defer rows.Close()
total := uint64(0)
if rows.Next() {
if err := rows.Scan(&total); err != nil {
return 0, err
}
}
return total, nil
}
type dbComputation struct {
ID string `db:"id"`
Name string `db:"name"`
Description string `db:"description"`
Status string `db:"status"`
Owner string `db:"owner"`
StartTime time.Time `db:"start_time"`
EndTime time.Time `db:"end_time"`
Datasets pq.StringArray `db:"datasets"`
Algorithms pq.StringArray `db:"algorithms"`
DatasetProviders pq.StringArray `db:"dataset_providers"`
AlgorithmProviders pq.StringArray `db:"algorithm_providers"`
Ttl int `db:"ttl"`
Metadata []byte `db:"metadata"`
}
func toComputation(c dbComputation) (computations.Computation, error) {
var metadata map[string]interface{}
if c.Metadata != nil {
if err := json.Unmarshal([]byte(c.Metadata), &metadata); err != nil {
return computations.Computation{}, errors.Wrap(errors.ErrMalformedEntity, err)
}
}
ret := computations.Computation{
ID: c.ID,
Name: c.Name,
Description: c.Description,
Status: c.Status,
Owner: c.Owner,
StartTime: c.StartTime,
EndTime: c.EndTime,
Datasets: c.Datasets,
Algorithms: c.Algorithms,
DatasetProviders: c.DatasetProviders,
AlgorithmProviders: c.AlgorithmProviders,
Ttl: c.Ttl,
Metadata: metadata,
}
return ret, nil
}
func fromComputation(c computations.Computation) (dbComputation, error) {
metadata := []byte("{}")
if len(c.Metadata) > 0 {
b, err := json.Marshal(c.Metadata)
if err != nil {
return dbComputation{}, errors.Wrap(errors.ErrMalformedEntity, err)
}
metadata = b
}
ret := dbComputation{
ID: c.ID,
Name: c.Name,
Description: c.Description,
Status: c.Status,
Owner: c.Owner,
StartTime: c.StartTime,
EndTime: c.EndTime,
Datasets: c.Datasets,
Algorithms: c.Algorithms,
DatasetProviders: c.DatasetProviders,
AlgorithmProviders: c.AlgorithmProviders,
Ttl: c.Ttl,
Metadata: metadata,
}
return ret, nil
}
func getNameQuery(name string) (string, string) {
if name == "" {
return "", ""
}
name = fmt.Sprintf(`%%%s%%`, strings.ToLower(name))
nq := `LOWER(name) LIKE :name`
return nq, name
}
func getOrderQuery(order string) string {
switch order {
case "name":
return "name"
default:
return "id"
}
}
func getDirQuery(dir string) string {
switch dir {
case "asc":
return "ASC"
default:
return "DESC"
}
}
func getMetadataQuery(m computations.Metadata) ([]byte, string, error) {
mq := ""
mb := []byte("{}")
if len(m) > 0 {
mq = `metadata @> :metadata`
b, err := json.Marshal(m)
if err != nil {
return nil, "", err
}
mb = b
}
return mb, mq, nil
}
func toDBComputation(cpt computations.Computation) (dbComputation, error) {
data := []byte("{}")
if len(cpt.Metadata) > 0 {
b, err := json.Marshal(cpt.Metadata)
if err != nil {
return dbComputation{}, errors.Wrap(errors.ErrMalformedEntity, err)
}
data = b
}
return dbComputation{
ID: cpt.ID,
Owner: cpt.Owner,
Name: cpt.Name,
Description: cpt.Description,
Metadata: data,
}, nil
}
-4
View File
@@ -1,4 +0,0 @@
// Package postgres contains repository implementations using
// PostgreSQL as the underlying database.
package postgres
-32
View File
@@ -1,32 +0,0 @@
package postgres
import migrate "github.com/rubenv/sql-migrate"
var Migrations = &migrate.MemoryMigrationSource{
Migrations: []*migrate.Migration{
{
Id: "computations_1",
Up: []string{
`CREATE TABLE IF NOT EXISTS computations (
id UUID,
owner VARCHAR(255),
name VARCHAR(1025),
description text,
status VARCHAR(255),
start_time timestamp NOT NULL,
end_time timestamp,
datasets text[],
algorithms text[],
dataset_providers text[],
algorithm_providers text[],
ttl integer,
metadata JSON,
PRIMARY KEY (id, owner)
)`,
},
Down: []string{
"DROP TABLE computations",
},
},
},
}
-84
View File
@@ -1,84 +0,0 @@
package computations
import (
"context"
"time"
"github.com/mainflux/mainflux"
"github.com/mainflux/mainflux/pkg/errors"
"github.com/ultravioletrs/clients"
)
type Service interface {
CreateComputation(ctx context.Context, token string, computation Computation) (string, error)
ViewComputation(ctx context.Context, token, id string) (Computation, error)
ListComputations(ctx context.Context, token string, meta PageMetadata) (Page, error)
UpdateComputation(ctx context.Context, token string, computation Computation) error
RemoveComputation(ctx context.Context, token, id string) error
}
type service struct {
repo Repository
idp mainflux.IDProvider
auth clients.AuthServiceClient
}
func NewService(repo Repository, idp mainflux.IDProvider, auth clients.AuthServiceClient) Service {
return service{
repo: repo,
idp: idp,
auth: auth,
}
}
func (svc service) CreateComputation(ctx context.Context, token string, computation Computation) (string, error) {
id, err := svc.idp.ID()
if err != nil {
return "", err
}
computation.ID = id
computation.StartTime = time.Now()
return svc.repo.Save(ctx, computation)
}
func (svc service) ViewComputation(ctx context.Context, token string, id string) (Computation, error) {
// Use id as both subject and object as token is currently empty
// c_add policy is an example of acceptable policy from clients service.
if err := svc.authorize(ctx, id, id, "c_add"); err != nil {
return Computation{}, errors.Wrap(errors.ErrAuthentication, err)
}
return svc.repo.View(ctx, id)
}
func (svc service) ListComputations(ctx context.Context, token string, meta PageMetadata) (Page, error) {
return svc.repo.RetrieveAll(ctx, token, meta)
}
func (svc service) UpdateComputation(ctx context.Context, token string, computation Computation) error {
return svc.repo.Update(ctx, computation)
}
func (svc service) RemoveComputation(ctx context.Context, token string, id string) error {
// Use id as both subject and object as token is currently empty
// c_delete policy is an example of acceptable policy from clients service.
if err := svc.authorize(ctx, id, id, "c_delete"); err != nil {
return errors.Wrap(errors.ErrAuthentication, err)
}
return svc.repo.Delete(ctx, id)
}
func (svc service) authorize(ctx context.Context, subject, object, action string) error {
req := &clients.AuthorizeReq{
Sub: subject,
Obj: object,
Act: action,
}
res, err := svc.auth.Authorize(ctx, req)
if err != nil {
return errors.Wrap(errors.ErrAuthorization, err)
}
if !res.GetAuthorized() {
return errors.ErrAuthorization
}
return nil
}
View File
-4
View File
@@ -1,4 +0,0 @@
// Package api contains API-related concerns: endpoint
// definitions, middlewares, and all resource representations.
package api
-115
View File
@@ -1,115 +0,0 @@
package api
import (
"context"
"github.com/go-kit/kit/endpoint"
"github.com/ultravioletrs/cocos/datasets"
)
func createDatasetsEndpoint(svc datasets.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(createReq)
id, err := svc.CreateDataset(ctx, req.dataset)
if err != nil {
return createRes{}, err
}
res := createRes{
ID: id,
created: true,
}
return res, nil
}
}
func viewDatasetsEndpoint(svc datasets.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(viewRequest)
ds, err := svc.ViewDataset(ctx, req.owner, req.id)
if err != nil {
return nil, err
}
res := viewRes{
Dataset: ds,
}
return res, nil
}
}
func listDatasetsEndpoint(svc datasets.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(listResourcesReq)
page, err := svc.ListDatasets(ctx, req.owner, req.pageMetadata)
if err != nil {
return nil, err
}
res := datasetsPageRes{
pageRes: pageRes{
Total: page.Total,
Offset: page.Offset,
Limit: page.Limit,
Order: page.Order,
Dir: page.Dir,
},
Datasets: []datasets.Dataset{},
}
res.Datasets = append(res.Datasets, page.Datasets...)
return res, nil
}
}
func updateDatasetEndpoint(svc datasets.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(updateReq)
dataset := datasets.Dataset{
ID: req.id,
Name: req.Name,
Description: req.Description,
Metadata: req.Metadata,
}
if err := svc.UpdateDataset(ctx, dataset); err != nil {
return nil, err
}
res := createRes{ID: req.id, created: false}
return res, nil
}
}
func uploadDatasetEndpoint(svc datasets.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(uploadReq)
if err := req.validate(); err != nil {
return nil, err
}
if err := svc.UploadDataset(ctx, req.id, req.owner, req.Payload); err != nil {
return nil, err
}
res := createRes{ID: req.id, created: false}
return res, nil
}
}
func removeDatasetEndpoint(svc datasets.Service) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(viewRequest)
if err := svc.RemoveDataset(ctx, req.id); err != nil {
return nil, err
}
return removeRes{}, nil
}
}
-95
View File
@@ -1,95 +0,0 @@
package api
import (
"context"
"fmt"
"time"
log "github.com/mainflux/mainflux/logger"
"github.com/ultravioletrs/cocos/datasets"
)
var _ datasets.Service = (*loggingMiddleware)(nil)
type loggingMiddleware struct {
logger log.Logger
svc datasets.Service
}
// LoggingMiddleware adds logging facilities to the core service.
func LoggingMiddleware(svc datasets.Service, logger log.Logger) datasets.Service {
return &loggingMiddleware{logger, svc}
}
func (lm *loggingMiddleware) CreateDataset(ctx context.Context, dts datasets.Dataset) (id string, err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method create_dataset took %s to complete", time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s.", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors.", message))
}(time.Now())
return lm.svc.CreateDataset(ctx, dts)
}
func (lm *loggingMiddleware) ViewDataset(ctx context.Context, owner, id string) (d datasets.Dataset, err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method view_dataset took %s to complete", time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s.", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors.", message))
}(time.Now())
return lm.svc.ViewDataset(ctx, owner, id)
}
func (lm *loggingMiddleware) ListDatasets(ctx context.Context, owner string, meta datasets.PageMetadata) (page datasets.Page, err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method list_datasets took %s to complete", time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s.", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors.", message))
}(time.Now())
return lm.svc.ListDatasets(ctx, owner, meta)
}
func (lm *loggingMiddleware) UpdateDataset(ctx context.Context, dataset datasets.Dataset) (err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method update_dataset took %s to complete", time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s.", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors.", message))
}(time.Now())
return lm.svc.UpdateDataset(ctx, dataset)
}
func (lm *loggingMiddleware) UploadDataset(ctx context.Context, id, owner string, payload []byte) (err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method upload_dataset took %s to complete", time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s.", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors.", message))
}(time.Now())
return lm.svc.UploadDataset(ctx, id, owner, payload)
}
func (lm *loggingMiddleware) RemoveDataset(ctx context.Context, id string) (err error) {
defer func(begin time.Time) {
message := fmt.Sprintf("Method remove_dataset took %s to complete", time.Since(begin))
if err != nil {
lm.logger.Warn(fmt.Sprintf("%s with error: %s.", message, err))
return
}
lm.logger.Info(fmt.Sprintf("%s without errors.", message))
}(time.Now())
return lm.svc.RemoveDataset(ctx, id)
}
-74
View File
@@ -1,74 +0,0 @@
package api
import (
"context"
"time"
"github.com/go-kit/kit/metrics"
"github.com/ultravioletrs/cocos/datasets"
)
var _ datasets.Service = (*metricsMiddleware)(nil)
type metricsMiddleware struct {
counter metrics.Counter
latency metrics.Histogram
svc datasets.Service
}
// MetricsMiddleware instruments core service by tracking request count and latency.
func MetricsMiddleware(svc datasets.Service, counter metrics.Counter, latency metrics.Histogram) datasets.Service {
return &metricsMiddleware{
counter: counter,
latency: latency,
svc: svc,
}
}
func (ms *metricsMiddleware) CreateDataset(ctx context.Context, dts datasets.Dataset) (id string, err error) {
defer func(begin time.Time) {
ms.counter.With("method", "create_dataset").Add(1)
ms.latency.With("method", "create_dataset").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.CreateDataset(ctx, dts)
}
func (ms *metricsMiddleware) ViewDataset(ctx context.Context, owner, id string) (datasets.Dataset, error) {
defer func(begin time.Time) {
ms.counter.With("method", "view_dataset").Add(1)
ms.latency.With("method", "view_dataset").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.ViewDataset(ctx, owner, id)
}
func (ms *metricsMiddleware) ListDatasets(ctx context.Context, owner string, meta datasets.PageMetadata) (page datasets.Page, err error) {
defer func(begin time.Time) {
ms.counter.With("method", "list_datasets").Add(1)
ms.latency.With("method", "list_datasets").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.ListDatasets(ctx, owner, meta)
}
func (ms *metricsMiddleware) UpdateDataset(ctx context.Context, dataset datasets.Dataset) error {
defer func(begin time.Time) {
ms.counter.With("method", "update_datasets").Add(1)
ms.latency.With("method", "update_datasets").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.UpdateDataset(ctx, dataset)
}
func (ms *metricsMiddleware) UploadDataset(ctx context.Context, id, owner string, payload []byte) error {
defer func(begin time.Time) {
ms.counter.With("method", "update_datasets").Add(1)
ms.latency.With("method", "update_datasets").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.UploadDataset(ctx, id, owner, payload)
}
func (ms *metricsMiddleware) RemoveDataset(ctx context.Context, id string) error {
defer func(begin time.Time) {
ms.counter.With("method", "remove_dataset").Add(1)
ms.latency.With("method", "remove_dataset").Observe(time.Since(begin).Seconds())
}(time.Now())
return ms.svc.RemoveDataset(ctx, id)
}
-38
View File
@@ -1,38 +0,0 @@
package api
import (
"github.com/ultravioletrs/cocos/datasets"
)
type createReq struct {
dataset datasets.Dataset
token string
}
type listResourcesReq struct {
owner string
pageMetadata datasets.PageMetadata
}
type updateReq struct {
id string
Name string `json:"name,omitempty"`
Description string `json:"description,omitempty"`
Metadata map[string]interface{} `json:"metadata,omitempty"`
}
type uploadReq struct {
owner string
token string
id string
Payload []byte
}
func (req uploadReq) validate() error {
return nil
}
type viewRequest struct {
id string
owner string
}
-93
View File
@@ -1,93 +0,0 @@
package api
import (
"fmt"
"net/http"
"github.com/ultravioletrs/cocos/datasets"
)
type createRes struct {
ID string
created bool
}
func (res createRes) Code() int {
if res.created {
return http.StatusCreated
}
return http.StatusOK
}
func (res createRes) Headers() map[string]string {
if res.created {
return map[string]string{
"Location": fmt.Sprintf("/datasets/%s", res.ID),
}
}
return map[string]string{}
}
func (res createRes) Empty() bool {
return true
}
type errorRes struct {
Err string `json:"error"`
}
type removeRes struct{}
func (res removeRes) Code() int {
return http.StatusNoContent
}
func (res removeRes) Headers() map[string]string {
return map[string]string{}
}
func (res removeRes) Empty() bool {
return true
}
type viewRes struct {
datasets.Dataset
}
func (res viewRes) Code() int {
return http.StatusOK
}
func (res viewRes) Headers() map[string]string {
return map[string]string{}
}
func (res viewRes) Empty() bool {
return false
}
type datasetsPageRes struct {
pageRes
Datasets []datasets.Dataset `json:"datasets"`
}
func (res datasetsPageRes) Code() int {
return http.StatusOK
}
func (res datasetsPageRes) Headers() map[string]string {
return map[string]string{}
}
func (res datasetsPageRes) Empty() bool {
return false
}
type pageRes struct {
Total uint64 `json:"total"`
Offset uint64 `json:"offset"`
Limit uint64 `json:"limit"`
Order string `json:"order"`
Dir string `json:"direction"`
}
-238
View File
@@ -1,238 +0,0 @@
package api
import (
"context"
"encoding/json"
"io/ioutil"
"net/http"
"strings"
kithttp "github.com/go-kit/kit/transport/http"
"github.com/go-zoo/bone"
"github.com/mainflux/mainflux"
"github.com/mainflux/mainflux/logger"
"github.com/mainflux/mainflux/pkg/errors"
"github.com/opentracing/opentracing-go"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/ultravioletrs/cocos/datasets"
"github.com/ultravioletrs/cocos/internal/apiutil"
httputil "github.com/ultravioletrs/cocos/internal/http"
)
const (
contentType = "application/json"
offsetKey = "offset"
nameKey = "name"
limitKey = "limit"
orderKey = "order"
dirKey = "dir"
metadataKey = "metadata"
sharedKey = "shared"
defOffset = 0
defLimit = 10
bearerPrefix = "Bearer "
)
// MakeHandler returns a HTTP handler for API endpoints.
func MakeHandler(svc datasets.Service, tracer opentracing.Tracer, logger logger.Logger) http.Handler {
opts := []kithttp.ServerOption{
kithttp.ServerErrorEncoder(apiutil.LoggingErrorEncoder(logger, encodeError)),
}
mux := bone.New()
mux.Post("/datasets", kithttp.NewServer(
createDatasetsEndpoint(svc),
decodeCreateDatasets,
encodeResponse,
opts...,
))
mux.Get("/datasets", kithttp.NewServer(
listDatasetsEndpoint(svc),
decodeList,
encodeResponse,
opts...,
))
mux.Get("/datasets/:id", kithttp.NewServer(
viewDatasetsEndpoint(svc),
decodeView,
encodeResponse,
opts...,
))
mux.Put("/datasets/:id", kithttp.NewServer(
updateDatasetEndpoint(svc),
decodeUpdateDataset,
encodeResponse,
opts...,
))
mux.Patch("/datasets/payload/:id", kithttp.NewServer(
uploadDatasetEndpoint(svc),
decodeUploadDataset,
encodeResponse,
opts...,
))
mux.Delete("/datasets/:id", kithttp.NewServer(
removeDatasetEndpoint(svc),
decodeView,
encodeResponse,
opts...,
))
mux.GetFunc("/health", httputil.Health("datasets", "Datasets management service"))
mux.Handle("/metrics", promhttp.Handler())
return mux
}
func decodeCreateDatasets(_ context.Context, r *http.Request) (interface{}, error) {
req := createReq{
token: extractBearerToken(r),
}
if err := json.NewDecoder(r.Body).Decode(&req.dataset); err != nil {
return nil, errors.Wrap(errors.ErrMalformedEntity, err)
}
return req, nil
}
func decodeList(_ context.Context, r *http.Request) (interface{}, error) {
o, err := apiutil.ReadUintQuery(r, offsetKey, defOffset)
if err != nil {
return nil, err
}
l, err := apiutil.ReadUintQuery(r, limitKey, defLimit)
if err != nil {
return nil, err
}
n, err := apiutil.ReadStringQuery(r, nameKey, "")
if err != nil {
return nil, err
}
or, err := apiutil.ReadStringQuery(r, orderKey, "")
if err != nil {
return nil, err
}
d, err := apiutil.ReadStringQuery(r, dirKey, "")
if err != nil {
return nil, err
}
m, err := apiutil.ReadMetadataQuery(r, metadataKey, nil)
if err != nil {
return nil, err
}
shared, err := apiutil.ReadBoolQuery(r, sharedKey, false)
if err != nil {
return nil, err
}
req := listResourcesReq{
pageMetadata: datasets.PageMetadata{
Offset: o,
Limit: l,
Name: n,
Order: or,
Dir: d,
Metadata: m,
FetchSharedDatasets: shared,
},
}
return req, nil
}
func decodeView(_ context.Context, r *http.Request) (interface{}, error) {
req := viewRequest{
id: bone.GetValue(r, "id"),
}
return req, nil
}
func decodeUpdateDataset(_ context.Context, r *http.Request) (interface{}, error) {
if !strings.Contains(r.Header.Get("Content-Type"), contentType) {
return nil, errors.ErrUnsupportedContentType
}
req := updateReq{
id: bone.GetValue(r, "id"),
}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
return nil, errors.Wrap(errors.ErrMalformedEntity, err)
}
return req, nil
}
func decodeUploadDataset(_ context.Context, r *http.Request) (interface{}, error) {
if !strings.Contains(r.Header.Get("Content-Type"), contentType) {
return nil, errors.ErrUnsupportedContentType
}
payload, err := ioutil.ReadAll(r.Body)
if err != nil {
return nil, errors.ErrMalformedEntity
}
defer r.Body.Close()
req := uploadReq{
token: extractBearerToken(r),
id: bone.GetValue(r, "id"),
Payload: payload,
}
return req, nil
}
func encodeResponse(_ context.Context, w http.ResponseWriter, response interface{}) error {
if ar, ok := response.(mainflux.Response); ok {
for k, v := range ar.Headers() {
w.Header().Set(k, v)
}
w.Header().Set("Content-Type", contentType)
w.WriteHeader(ar.Code())
if ar.Empty() {
return nil
}
}
return json.NewEncoder(w).Encode(response)
}
func encodeError(_ context.Context, err error, w http.ResponseWriter) {
switch {
case errors.Contains(err, errors.ErrInvalidQueryParams),
errors.Contains(err, errors.ErrMalformedEntity):
w.WriteHeader(http.StatusBadRequest)
case errors.Contains(err, errors.ErrCreateEntity),
errors.Contains(err, errors.ErrUpdateEntity),
errors.Contains(err, errors.ErrViewEntity),
errors.Contains(err, errors.ErrRemoveEntity):
w.WriteHeader(http.StatusInternalServerError)
default:
w.WriteHeader(http.StatusInternalServerError)
}
if errorVal, ok := err.(errors.Error); ok {
w.Header().Set("Content-Type", contentType)
if err := json.NewEncoder(w).Encode(errorRes{Err: errorVal.Msg()}); err != nil {
w.WriteHeader(http.StatusInternalServerError)
}
}
}
func extractBearerToken(r *http.Request) string {
token := r.Header.Get("Authorization")
if !strings.HasPrefix(token, bearerPrefix) {
return ""
}
return strings.TrimPrefix(token, bearerPrefix)
}
-65
View File
@@ -1,65 +0,0 @@
package datasets
import (
"context"
"time"
)
type Dataset struct {
ID string `json:"id,omitempty" db:"id"`
Token string `json:"token,omitempty" db:"token"`
Name string `json:"name,omitempty" db:"name"`
Description string `json:"description,omitempty" db:"description"`
Owner string `json:"owner,omitempty" db:"owner"`
Size uint64 `json:"size,omitempty" db:"size"`
CreatedAt time.Time `json:"created_at,omitempty" db:"created_at"`
UpdatedAt time.Time `json:"updated_at,omitempty" db:"updated_at"`
Location string `json:"location,omitempty" db:"location"`
Format string `json:"format,omitempty" db:"format"`
Metadata Metadata `json:"metadata,omitempty" db:"metadata"`
Path string `json:"Path,omitempty" db:"path"`
}
type Metadata map[string]interface{}
type PageMetadata struct {
Total uint64 `json:"total,omitempty"`
Offset uint64 `json:"offset,omitempty"`
Limit uint64 `json:"limit,omitempty"`
Name string `json:"name,omitempty"`
Order string `json:"order,omitempty"`
Dir string `json:"dir,omitempty"`
Metadata Metadata `json:"metadata,omitempty"`
FetchSharedDatasets bool
}
type Page struct {
PageMetadata
Datasets []Dataset `json:"datasets,omitempty"`
}
func (d Dataset) Validate() error {
return nil
}
// DatasetRepository specifies a dataset persistence API.
type DatasetRepository interface {
// Save persists multiple datasets. Datasets are saved using a transaction. If one dataset
// fails then none will be saved. Successful operation is indicated by non-nil
// error response.
Save(ctx context.Context, dts Dataset) (string, error)
// RetrieveAll retrieves the subset of datasets owned by the specified user
RetrieveByID(ctx context.Context, owner, id string) (Dataset, error)
// RetrieveAll retrieves the subset of datasets owned by the specified user
RetrieveAll(ctx context.Context, owner string, pm PageMetadata) (Page, error)
// Update performs an update to the existing dataset. A non-nil error is
// returned to indicate operation failure.
Update(ctx context.Context, d Dataset) error
// Remove removes the dataset having the provided identifier, that is owned
// by the specified user.
Delete(ctx context.Context, id string) error
}
-4
View File
@@ -1,4 +0,0 @@
// Package dataset contains domain concept definitions
// needed to support dataset service functionality.
package datasets
-360
View File
@@ -1,360 +0,0 @@
package postgres
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"strings"
"time"
"github.com/lib/pq"
"github.com/mainflux/mainflux/pkg/errors"
"github.com/ultravioletrs/cocos/datasets"
db "github.com/ultravioletrs/cocos/internal/db"
)
const (
errDuplicate = "unique_violation"
errInvalid = "invalid_text_representation"
errTruncation = "string_data_right_truncation"
)
var _ datasets.DatasetRepository = (*datasetRepo)(nil)
type datasetRepo struct {
db db.Database
}
// NewRepository returns a new Datasets repository implementation.
func NewRepository(db db.Database) datasets.DatasetRepository {
return datasetRepo{
db: db,
}
}
func (repo datasetRepo) Save(ctx context.Context, dataset datasets.Dataset) (string, error) {
q := `INSERT INTO datasets (id, owner, name, metadata, description, format, location, created_at, updated_at)
VALUES (:id, :owner, :name, :metadata, :description, :format, :location, :created_at, :updated_at) RETURNING id;`
ds, err := fromDataset(dataset)
if err != nil {
return "", errors.Wrap(errors.ErrCreateEntity, err)
}
row, err := repo.db.NamedQueryContext(ctx, q, ds)
if err != nil {
pqErr, ok := err.(*pq.Error)
if ok {
switch pqErr.Code.Name() {
case errInvalid, errTruncation:
return "", errors.Wrap(errors.ErrMalformedEntity, err)
case errDuplicate:
return "", errors.Wrap(errors.ErrConflict, err)
}
}
return "", errors.Wrap(errors.ErrCreateEntity, err)
}
defer row.Close()
row.Next()
var id string
if err := row.Scan(&id); err != nil {
return "", err
}
return id, nil
}
func (repo datasetRepo) RetrieveByID(ctx context.Context, owner, id string) (datasets.Dataset, error) {
q := `SELECT id, name, owner, metadata, description, location, format, created_at, updated_at FROM datasets WHERE id = $1;`
dbds := dbDataset{ID: id}
if err := repo.db.QueryRowxContext(ctx, q, id).StructScan(&dbds); err != nil {
pqErr, ok := err.(*pq.Error)
if err == sql.ErrNoRows || ok && errInvalid == pqErr.Code.Name() {
return datasets.Dataset{}, errors.Wrap(errors.ErrNotFound, err)
}
return datasets.Dataset{}, errors.Wrap(errors.ErrViewEntity, err)
}
return toDataset(dbds)
}
func (repo datasetRepo) RetrieveAll(ctx context.Context, owner string, pm datasets.PageMetadata) (datasets.Page, error) {
nq, name := NameQuery(pm.Name)
oq := OrderQuery(pm.Order)
dq := DirQuery(pm.Dir)
m, mq, err := MetadataQuery(pm.Metadata)
if err != nil {
return datasets.Page{}, errors.Wrap(errors.ErrViewEntity, err)
}
var query []string
if mq != "" {
query = append(query, mq)
}
if nq != "" {
query = append(query, nq)
}
var whereClause string
if len(query) > 0 {
whereClause = fmt.Sprintf(" WHERE %s", strings.Join(query, " AND "))
}
q := `SELECT id, name, COALESCE(description, '') AS description, owner, created_at,
updated_at, location, format, metadata
FROM datasets %s
ORDER BY %s %s LIMIT :limit OFFSET :offset`
q = fmt.Sprintf(q, whereClause, oq, dq)
params := map[string]interface{}{
"owner": owner,
"limit": pm.Limit,
"offset": pm.Offset,
"name": name,
"metadata": m,
}
rows, err := repo.db.NamedQueryContext(ctx, q, params)
if err != nil {
return datasets.Page{}, errors.Wrap(errors.ErrViewEntity, err)
}
defer rows.Close()
var items []datasets.Dataset
for rows.Next() {
dbd := dbDataset{Owner: owner}
if err := rows.StructScan(&dbd); err != nil {
return datasets.Page{}, errors.Wrap(errors.ErrViewEntity, err)
}
ds, err := toDataset(dbd)
if err != nil {
return datasets.Page{}, err
}
items = append(items, ds)
}
cq := fmt.Sprintf(`SELECT COUNT(*) FROM datasets %s;`, whereClause)
total, err := repo.total(ctx, cq, params)
if err != nil {
return datasets.Page{}, errors.Wrap(errors.ErrViewEntity, err)
}
page := datasets.Page{
Datasets: items,
PageMetadata: datasets.PageMetadata{
Total: total,
Offset: pm.Offset,
Limit: pm.Limit,
Order: pm.Order,
Dir: pm.Dir,
},
}
return page, nil
}
func (repo datasetRepo) Update(ctx context.Context, d datasets.Dataset) error {
q := `UPDATE datasets SET name = :name, metadata = :metadata, description = :description WHERE id = :id;`
dbdts, err := toDBDataset(d)
if err != nil {
return errors.Wrap(errors.ErrUpdateEntity, err)
}
res, errdb := repo.db.NamedExecContext(ctx, q, dbdts)
if errdb != nil {
pqErr, ok := errdb.(*pq.Error)
if ok {
switch pqErr.Code.Name() {
case errInvalid, errTruncation:
return errors.Wrap(errors.ErrMalformedEntity, errdb)
}
}
return errors.Wrap(errors.ErrUpdateEntity, errdb)
}
cnt, errdb := res.RowsAffected()
if errdb != nil {
return errors.Wrap(errors.ErrUpdateEntity, errdb)
}
if cnt == 0 {
return errors.ErrNotFound
}
return nil
}
func (repo datasetRepo) Delete(ctx context.Context, id string) error {
q := `DELETE FROM datasets WHERE id = :id`
ds := dbDataset{
ID: id,
}
if _, err := repo.db.NamedExecContext(ctx, q, ds); err != nil {
if err == sql.ErrNoRows {
return errors.Wrap(errors.ErrNotFound, err)
}
return errors.Wrap(errors.ErrViewEntity, err)
}
return nil
}
func (repo datasetRepo) total(ctx context.Context, query string, params interface{}) (uint64, error) {
rows, err := repo.db.NamedQueryContext(ctx, query, params)
if err != nil {
return 0, err
}
defer rows.Close()
total := uint64(0)
if rows.Next() {
if err := rows.Scan(&total); err != nil {
return 0, err
}
}
return total, nil
}
type dbDataset struct {
ID string `db:"id"`
Token string `db:"token"`
Name string `db:"name"`
Description string `db:"description"`
Owner string `db:"owner"`
Size uint64 `db:"size"`
CreatedAt time.Time `db:"created_at"`
UpdatedAt time.Time `db:"updated_at"`
Location string `db:"location"`
Format string `db:"format"`
Metadata []byte `db:"metadata"`
Path string `db:"path"`
}
func toDataset(d dbDataset) (datasets.Dataset, error) {
var metadata map[string]interface{}
if d.Metadata != nil {
if err := json.Unmarshal([]byte(d.Metadata), &metadata); err != nil {
return datasets.Dataset{}, errors.Wrap(errors.ErrMalformedEntity, err)
}
}
ret := datasets.Dataset{
ID: d.ID,
Token: d.Token,
Name: d.Name,
Description: d.Description,
Owner: d.Owner,
Size: d.Size,
CreatedAt: d.CreatedAt,
UpdatedAt: d.UpdatedAt,
Location: d.Location,
Format: d.Format,
Metadata: metadata,
Path: d.Path,
}
return ret, nil
}
func fromDataset(d datasets.Dataset) (dbDataset, error) {
metadata := []byte("{}")
if len(d.Metadata) > 0 {
b, err := json.Marshal(d.Metadata)
if err != nil {
return dbDataset{}, errors.Wrap(errors.ErrMalformedEntity, err)
}
metadata = b
}
ret := dbDataset{
ID: d.ID,
Token: d.Token,
Name: d.Name,
Description: d.Description,
Owner: d.Owner,
Size: d.Size,
CreatedAt: d.CreatedAt,
UpdatedAt: d.UpdatedAt,
Location: d.Location,
Format: d.Format,
Metadata: metadata,
Path: d.Path,
}
return ret, nil
}
func NameQuery(name string) (string, string) {
if name == "" {
return "", ""
}
name = fmt.Sprintf(`%%%s%%`, strings.ToLower(name))
nq := `LOWER(name) LIKE :name`
return nq, name
}
func OrderQuery(order string) string {
switch order {
case "name":
return "name"
default:
return "id"
}
}
func DirQuery(dir string) string {
switch dir {
case "asc":
return "ASC"
default:
return "DESC"
}
}
func MetadataQuery(m datasets.Metadata) ([]byte, string, error) {
mq := ""
mb := []byte("{}")
if len(m) > 0 {
mq = `metadata @> :metadata`
b, err := json.Marshal(m)
if err != nil {
return nil, "", err
}
mb = b
}
return mb, mq, nil
}
func (repo datasetRepo) Remove(ctx context.Context, id string) error {
dbth := dbDataset{
ID: id,
}
q := `DELETE FROM datasets WHERE id = :id`
if _, err := repo.db.NamedExecContext(ctx, q, dbth); err != nil {
return errors.Wrap(errors.ErrRemoveEntity, err)
}
return nil
}
func toDBDataset(ds datasets.Dataset) (dbDataset, error) {
data := []byte("{}")
if len(ds.Metadata) > 0 {
b, err := json.Marshal(ds.Metadata)
if err != nil {
return dbDataset{}, errors.Wrap(errors.ErrMalformedEntity, err)
}
data = b
}
return dbDataset{
ID: ds.ID,
Owner: ds.Owner,
Name: ds.Name,
Description: ds.Description,
Metadata: data,
Size: ds.Size,
}, nil
}
-4
View File
@@ -1,4 +0,0 @@
// Package dataset contains domain concept definitions
// needed to support dataset service functionality.
package postgres
-33
View File
@@ -1,33 +0,0 @@
package postgres
import (
// used to register Postgres driver
_ "github.com/lib/pq"
migrate "github.com/rubenv/sql-migrate"
)
var Migrations = &migrate.MemoryMigrationSource{
Migrations: []*migrate.Migration{
{
Id: "datasets_01",
Up: []string{
`CREATE TABLE IF NOT EXISTS datasets (
id UUID,
name VARCHAR(1025),
owner VARCHAR(255),
description text,
created_at timestamp NOT NULL,
updated_at timestamp NOT NULL,
size INT,
location TEXT,
format VARCHAR(255),
metadata JSON,
PRIMARY KEY (id, owner)
)`,
},
Down: []string{
"DROP TABLE datasets",
},
},
},
}
-101
View File
@@ -1,101 +0,0 @@
package datasets
import (
"context"
"io/fs"
"os"
"path/filepath"
"time"
"github.com/mainflux/mainflux"
)
type datasetsService struct {
repo DatasetRepository
idp mainflux.IDProvider
}
func NewService(repo DatasetRepository, idp mainflux.IDProvider) Service {
return &datasetsService{
repo: repo,
idp: idp,
}
}
// Service specifies an API that must be fulfiled by the domain service
// implementation, and all of its decorators (e.g. logging & metrics).
type Service interface {
// CreateDatasets adds datasets to the user.
CreateDataset(ctx context.Context, dataset Dataset) (string, error)
// ViewDataset retrieves data about the dataset identified with the provided
// ID, that belongs to the user identified by the provided token.
ViewDataset(ctx context.Context, owner, id string) (Dataset, error)
// ListDatasets retrieves data about subset of datasets that belongs to the
// user identified by the provided token.
ListDatasets(ctx context.Context, owner string, meta PageMetadata) (Page, error)
// UpdateDataset updates the dataset identified by the provided ID, that
// belongs to the user identified by the provided token.
UpdateDataset(ctx context.Context, dataset Dataset) error
// UploadDataset uploads the actual dataset content to the dataset path.
UploadDataset(ctx context.Context, id, owner string, payload []byte) error
// RemoveDataset removes the dataset identified with the provided ID, that
// belongs to the user identified by the provided token.
RemoveDataset(ctx context.Context, id string) error
}
func (ds *datasetsService) CreateDataset(ctx context.Context, dataset Dataset) (string, error) {
id, err := ds.idp.ID()
if err != nil {
return "", err
}
dataset.ID = id
dataset.CreatedAt = time.Now()
return ds.repo.Save(ctx, dataset)
}
func (ds *datasetsService) ViewDataset(ctx context.Context, owner, id string) (Dataset, error) {
return ds.repo.RetrieveByID(ctx, owner, id)
}
func (ds *datasetsService) ListDatasets(ctx context.Context, owner string, pm PageMetadata) (Page, error) {
page, err := ds.repo.RetrieveAll(ctx, owner, pm)
if err != nil {
return Page{}, err
}
return page, nil
}
func (ds *datasetsService) UpdateDataset(ctx context.Context, dataset Dataset) error {
return ds.repo.Update(ctx, dataset)
}
func (ds *datasetsService) UploadDataset(ctx context.Context, id, owner string, payload []byte) error {
dataset, err := ds.repo.RetrieveByID(ctx, owner, id)
if err != nil {
return err
}
path := filepath.Join(dataset.Path, dataset.ID)
file, err := os.Create(id)
if err != nil {
return err
}
defer file.Close()
filename := path
err = os.WriteFile(filename, payload, fs.ModePerm)
if err != nil {
return err
}
return os.WriteFile(filename, payload, fs.ModePerm)
}
func (ds *datasetsService) RemoveDataset(ctx context.Context, id string) error {
if err := ds.repo.Delete(ctx, id); err != nil {
return err
}
return ds.repo.Delete(ctx, id)
}
-4
View File
@@ -1,4 +0,0 @@
// Package cocos acts as an umbrella package containing
// multiple microservices and defines shared domain concepts.
package cocos
-42
View File
@@ -1,42 +0,0 @@
# Docker: Environment variables in Compose
## COCOS services
### Computations service
COCOS_COMPUTATIONS_LOG_LEVEL=debug
COCOS_COMPUTATIONS_DB_PORT=5432
COCOS_COMPUTATIONS_DB_USER=cocos
COCOS_COMPUTATIONS_DB_PASS=cocos
COCOS_COMPUTATIONS_DB=computations
COCOS_COMPUTATIONS_HTTP_PORT=9000
### Datasets service
COCOS_DATASETS_LOG_LEVEL=debug
COCOS_DATASETS_DB_PORT=5432
COCOS_DATASETS_DB_USER=cocos
COCOS_DATASETS_DB_PASS=cocos
COCOS_DATASETS_DB=datasets
COCOS_DATASETS_HTTP_PORT=9001
### Clients service
UV_CLIENTS_LOG_LEVEL=debug
UV_CLIENTS_DB_HOST=clients-db
UV_CLIENTS_DB_PORT=5439
UV_CLIENTS_DB_USER=uv
UV_CLIENTS_DB_PASS=uv
UV_CLIENTS_DB=clients
UV_CLIENTS_DB_SSL_MODE=
UV_CLIENTS_DB_SSL_CERT=
UV_CLIENTS_DB_SSL_KEY=
UV_CLIENTS_DB_SSL_ROOT_CERT=
UV_CLIENTS_HTTP_PORT=9191
UV_CLIENTS_GRPC_PORT=9192
UV_CLIENTS_GRPC_TIMEOUT=1s
UV_CLIENTS_SERVER_CERT=
UV_CLIENTS_SERVER_KEY=
# Docker image tag
MF_RELEASE_TAG=latest
# Docker image tag
COCOS_RELEASE_TAG=latest
-19
View File
@@ -1,19 +0,0 @@
FROM golang:1.18-alpine AS builder
ARG SVC
ARG GOARCH
ARG GOARM
ARG VERSION
ARG COMMIT
ARG TIME
WORKDIR /go/src/github.com/ultravioletrs/cocos
COPY . .
RUN apk update \
&& apk add make\
&& make $SVC \
&& mv build/cocos-$SVC /exe
FROM scratch
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=builder /exe /
ENTRYPOINT ["/exe"]
-4
View File
@@ -1,4 +0,0 @@
FROM scratch
ARG SVC
COPY cocos-$SVC /exe
ENTRYPOINT ["/exe"]

Some files were not shown because too many files have changed in this diff Show More