mirror of
https://github.com/absmach/magistrala.git
synced 2026-08-07 07:14:46 +00:00
NOISSUE - Revert alarms approach
Continuous Delivery / lint-and-build (push) Has been cancelled
Continuous Delivery / Build and Push Docker Images (push) Has been cancelled
CI Pipeline / Lint Proto (push) Has been cancelled
CI Pipeline / lint-and-build (push) Has been cancelled
CI Pipeline / Detect Changes (push) Has been cancelled
CI Pipeline / Test ${{ matrix.module }} (push) Has been cancelled
CI Pipeline / Upload Coverage (push) Has been cancelled
Property Based Tests / api-test (push) Has been cancelled
Deploy GitHub Pages / swagger-ui (push) Has been cancelled
Continuous Delivery / lint-and-build (push) Has been cancelled
Continuous Delivery / Build and Push Docker Images (push) Has been cancelled
CI Pipeline / Lint Proto (push) Has been cancelled
CI Pipeline / lint-and-build (push) Has been cancelled
CI Pipeline / Detect Changes (push) Has been cancelled
CI Pipeline / Test ${{ matrix.module }} (push) Has been cancelled
CI Pipeline / Upload Coverage (push) Has been cancelled
Property Based Tests / api-test (push) Has been cancelled
Deploy GitHub Pages / swagger-ui (push) Has been cancelled
Signed-off-by: dusan <borovcanindusan1@gmail.com>
This commit is contained in:
@@ -11,7 +11,7 @@ database. Implemented as Go scripts.
|
||||
| Topic | Decision |
|
||||
|-------|----------|
|
||||
| Passwords | **Force reset.** Users migrate with no `password` credential; they reset via Atom's email flow on first login. (bcrypt → argon2 is not convertible without plaintext.) |
|
||||
| Scope | Core IAM + roles & policies + connections + PATs + rules/reports/alarms as Atom resources. |
|
||||
| Scope | Core IAM + roles & policies + connections + PATs + rules/report configs as Atom resources. Alarm events remain in the alarms service database and are not migrated into Atom resources. |
|
||||
| Execution | **Offline one-shot.** Stop Magistrala app services, snapshot, transform, load, start Atom. |
|
||||
| IDs | **Preserve Magistrala UUIDs** as Atom UUIDs (PKs and FKs). Magistrala IDs are 36-char UUID strings — directly usable as Atom `UUID` PKs. Keeps audit trails, message payloads, external references, and SpiceDB-derived links intact. |
|
||||
|
||||
@@ -33,7 +33,6 @@ All `magistrala/magistrala`, port 5432, on network `magistrala-base-net`:
|
||||
| `auth-db` | `auth` | `pats`, `pat_scopes` (skip `keys` — short-lived JWTs; skip legacy `policies`/`domains` mirror) |
|
||||
| `re-db` | `rules_engine` | `rules`, `rules_roles*` |
|
||||
| `reports-db` | `reports` | `report_config`, `reports_roles*` |
|
||||
| `alarms-db` | `alarms` | `alarms` |
|
||||
|
||||
> Note: `groups` migrations are embedded into clients **and** channels **and** the
|
||||
> standalone groups service. The authoritative groups data for default compose is
|
||||
@@ -105,27 +104,25 @@ principal group, matching Atom's normal `create_entity` side effect.
|
||||
| `created_by` | `owner_id` (if the user migrated) |
|
||||
| `parent_group_id` | `object_group_resources` membership |
|
||||
|
||||
### 3.4b rules / reports / alarms → resources
|
||||
### 3.4b rules / report configs → resources
|
||||
|
||||
Rules-engine rules, report configs, and alarms are domain-scoped objects with no
|
||||
Atom-native table, so they become **resources** alongside channels, distinguished
|
||||
by `kind`. Service-specific columns Atom resources lack are folded into
|
||||
`attributes` (JSONB). `tenant_id=domain_id`; rows whose domain has no surviving
|
||||
tenant are skipped (§6.4). `owner_id`=`created_by` when that user migrated
|
||||
(alarms have no `created_by` → NULL). Names are deduped per tenant (§6.3) since
|
||||
these tables carry no `(domain_id, name)` constraint; alarms (no name) use
|
||||
`measurement` with an `id` fallback.
|
||||
Rules-engine rules and report configs are domain-scoped configuration objects
|
||||
with no Atom-native table, so they become **resources** alongside channels,
|
||||
distinguished by `kind`. Service-specific columns Atom resources lack are folded
|
||||
into `attributes` (JSONB). `tenant_id=domain_id`; rows whose domain has no
|
||||
surviving tenant are skipped (§6.4). `owner_id`=`created_by` when that user
|
||||
migrated. Names are deduped per tenant (§6.3) since these tables carry no
|
||||
`(domain_id, name)` constraint.
|
||||
|
||||
| Source | Atom `resources` |
|
||||
|---|---|
|
||||
| `rules_engine.rules` (id) | `kind='rule'`; `input_channel/topic, outputs, logic_type/value, recurring*, time, start_datetime, tags, status` → `attributes` |
|
||||
| `reports.report_config` (id) | `kind='report'`; `description, config, email, metrics, report_template, due, recurring*, start_datetime, status` → `attributes` |
|
||||
| `alarms.alarms` (id) | `kind='alarm'`; `rule_id, channel_id, client_id, subtopic, measurement, value, unit, threshold, cause, severity, alarm_status, assignee/assigned/acknowledged/resolved*` → `attributes` |
|
||||
|
||||
> Atom `resources.kind` must permit `rule`, `report`, `alarm` (in addition to
|
||||
> `channel`). Rules and reports have object-specific role families and those are
|
||||
> migrated as resource-scoped roles. Alarms have no role family or
|
||||
> `parent_group_id`, so only the resource rows are migrated for alarms.
|
||||
> Atom `resources.kind` must permit `rule` and `report` (in addition to
|
||||
> `channel`). Rules and report configs have object-specific role families and
|
||||
> those are migrated as resource-scoped roles. Alarm events can be high-volume
|
||||
> operational data, so they stay in `alarms-db` and are not Atom resources.
|
||||
|
||||
### 3.5 groups → object_groups
|
||||
Magistrala groups organize clients/channels within a domain (hierarchical,
|
||||
@@ -274,7 +271,7 @@ Checks below; the email check matters mainly for dumps merged across instances
|
||||
3. …then backfill tenants.created_by/updated_by and resources.owner_id
|
||||
4. entity_emails
|
||||
5. credentials (device api_key; PAT metadata)
|
||||
6. resources (channels, rules, reports, alarms)
|
||||
6. resources (channels, rules, reports)
|
||||
7. object_groups → object_group_hierarchy → object_group_entities/resources
|
||||
8. roles → permission_blocks → permission_block_actions → role_permission_blocks
|
||||
9. role_assignments, direct_policies
|
||||
|
||||
@@ -21,14 +21,14 @@ docker build -f tools/atom-migration/Dockerfile -t magistrala/atom-migration:dev
|
||||
## Start only the source databases
|
||||
|
||||
The migrator reads Postgres directly — it does **not** need the Magistrala app
|
||||
services running. To migrate from restored volumes, start just the nine source DB
|
||||
services running. To migrate from restored volumes, start just the eight source DB
|
||||
containers (`--no-deps` keeps compose from pulling in the app services they
|
||||
depend on):
|
||||
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.yaml up -d --no-deps \
|
||||
auth-db users-db domains-db clients-db channels-db groups-db \
|
||||
re-db reports-db alarms-db
|
||||
re-db reports-db
|
||||
```
|
||||
|
||||
They mount the `magistrala_magistrala-<svc>-db-volume` volumes and attach to
|
||||
|
||||
@@ -39,7 +39,6 @@ type config struct {
|
||||
Auth dbConn
|
||||
RE dbConn // rules engine
|
||||
Reports dbConn
|
||||
Alarms dbConn
|
||||
|
||||
AtomDSN string
|
||||
UnmappedAction string
|
||||
@@ -78,7 +77,6 @@ func loadConfig(envPath, atomDSN string, fromHost bool) (config, error) {
|
||||
Auth: mk("MG_AUTH"),
|
||||
RE: mk("MG_RE"),
|
||||
Reports: mk("MG_REPORTS"),
|
||||
Alarms: mk("MG_ALARMS"),
|
||||
AtomDSN: atomDSN,
|
||||
}
|
||||
|
||||
@@ -88,7 +86,6 @@ func loadConfig(envPath, atomDSN string, fromHost bool) (config, error) {
|
||||
&cfg.Clients.Name: "clients", &cfg.Channels.Name: "channels",
|
||||
&cfg.Groups.Name: "groups", &cfg.Auth.Name: "auth",
|
||||
&cfg.RE.Name: "rules_engine", &cfg.Reports.Name: "reports",
|
||||
&cfg.Alarms.Name: "alarms",
|
||||
}
|
||||
for p, n := range defName {
|
||||
if *p == "" {
|
||||
|
||||
@@ -29,7 +29,6 @@ type migrator struct {
|
||||
authDB *sqlx.DB
|
||||
reDB *sqlx.DB
|
||||
reportsDB *sqlx.DB
|
||||
alarmsDB *sqlx.DB
|
||||
atom *sqlx.DB
|
||||
|
||||
profileID map[string]string // profile key (e.g. "user","client") -> uuid
|
||||
@@ -97,7 +96,6 @@ func newMigrator(ctx context.Context, cfg config, apply bool) (*migrator, error)
|
||||
m.authDB = open("auth", cfg.Auth.DSN())
|
||||
m.reDB = open("rules_engine", cfg.RE.DSN())
|
||||
m.reportsDB = open("reports", cfg.Reports.DSN())
|
||||
m.alarmsDB = open("alarms", cfg.Alarms.DSN())
|
||||
m.atom = open("atom", cfg.AtomDSN)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -108,7 +106,7 @@ func newMigrator(ctx context.Context, cfg config, apply bool) (*migrator, error)
|
||||
const authenticatedUsersGroupID = "00000000-0000-0000-0000-000000000005"
|
||||
|
||||
func (m *migrator) Close() {
|
||||
for _, db := range []*sqlx.DB{m.domainsDB, m.usersDB, m.clientsDB, m.channelsDB, m.groupsDB, m.authDB, m.reDB, m.reportsDB, m.alarmsDB, m.atom} {
|
||||
for _, db := range []*sqlx.DB{m.domainsDB, m.usersDB, m.clientsDB, m.channelsDB, m.groupsDB, m.authDB, m.reDB, m.reportsDB, m.atom} {
|
||||
if db != nil {
|
||||
_ = db.Close()
|
||||
}
|
||||
@@ -140,7 +138,6 @@ func (m *migrator) Run(ctx context.Context, rep *report) error {
|
||||
{"resources.channels", m.phaseChannels},
|
||||
{"resources.rules", m.phaseRules},
|
||||
{"resources.reports", m.phaseReports},
|
||||
{"resources.alarms", m.phaseAlarms},
|
||||
{"object_groups", m.phaseGroups},
|
||||
{"group_membership", m.phaseGroupMembership},
|
||||
{"roles", m.phaseRoles},
|
||||
@@ -427,9 +424,9 @@ func (m *migrator) phaseChannels(ctx context.Context, rep *report) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// insertResource writes one row into Atom resources (kind = channel/rule/report/
|
||||
// alarm). Entity-specific columns Magistrala has but Atom resources lack are
|
||||
// folded into the attributes JSONB. ON CONFLICT (id) keeps it idempotent.
|
||||
// insertResource writes one row into Atom resources. Entity-specific columns
|
||||
// Magistrala has but Atom resources lack are folded into the attributes JSONB.
|
||||
// ON CONFLICT (id) keeps it idempotent.
|
||||
func (m *migrator) insertResource(ctx context.Context, id, kind, name, tenant string, owner sql.NullString, attributes string, createdAt time.Time, updatedAt any) error {
|
||||
return m.exec(ctx,
|
||||
`INSERT INTO resources (id, kind, name, tenant_id, owner_id, attributes, created_at, updated_at)
|
||||
@@ -447,7 +444,7 @@ func (m *migrator) ownerOf(createdBy sql.NullString) sql.NullString {
|
||||
}
|
||||
|
||||
// uniqueResName makes a resource name unique within a tenant (Atom enforces
|
||||
// resources(name, tenant_id); rules/reports/alarms carry no such Magistrala
|
||||
// resources(name, tenant_id); rules/reports carry no such Magistrala
|
||||
// constraint, so same-tenant dups are possible). Suffixes -2, -3, … on collision.
|
||||
func uniqueResName(seen map[string]int, tenant, name string) string {
|
||||
key := tenant + "|" + strings.ToLower(name)
|
||||
@@ -557,58 +554,6 @@ func (m *migrator) phaseReports(ctx context.Context, rep *report) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// phaseAlarms: alarms.alarms -> resources (kind=alarm). Alarms have no name in
|
||||
// Magistrala; the measurement is used (id fallback), deduped per tenant.
|
||||
func (m *migrator) phaseAlarms(ctx context.Context, rep *report) error {
|
||||
rows, err := readAlarms(ctx, m.alarmsDB)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
seen := map[string]int{}
|
||||
for _, a := range rows {
|
||||
if !m.tenants[a.DomainID] {
|
||||
rep.skip("alarm_orphan_domain")
|
||||
continue
|
||||
}
|
||||
m.resourceDomain[a.ID] = a.DomainID
|
||||
name := uniqueResName(seen, a.DomainID, firstNonEmpty(a.Measurement, a.ID))
|
||||
extra := map[string]any{
|
||||
"rule_id": a.RuleID,
|
||||
"channel_id": a.ChannelID,
|
||||
"client_id": a.ClientID,
|
||||
"subtopic": a.Subtopic,
|
||||
"measurement": a.Measurement,
|
||||
"value": a.Value,
|
||||
"unit": a.Unit,
|
||||
"threshold": a.Threshold,
|
||||
"cause": a.Cause,
|
||||
"alarm_status": a.Status,
|
||||
"severity": a.Severity,
|
||||
}
|
||||
putStr(extra, "assignee_id", a.AssigneeID)
|
||||
putStr(extra, "updated_by", a.UpdatedBy)
|
||||
putStr(extra, "assigned_by", a.AssignedBy)
|
||||
putStr(extra, "acknowledged_by", a.AcknowledgedBy)
|
||||
putStr(extra, "resolved_by", a.ResolvedBy)
|
||||
if a.AssignedAt.Valid {
|
||||
extra["assigned_at"] = a.AssignedAt.Time
|
||||
}
|
||||
if a.AcknowledgedAt.Valid {
|
||||
extra["acknowledged_at"] = a.AcknowledgedAt.Time
|
||||
}
|
||||
if a.ResolvedAt.Valid {
|
||||
extra["resolved_at"] = a.ResolvedAt.Time
|
||||
}
|
||||
// Alarms carry no created_by; owner_id stays NULL.
|
||||
if err := m.insertResource(ctx, a.ID, "alarm", name, a.DomainID, sql.NullString{},
|
||||
attrs(a.Metadata, extra), ntToTime(a.CreatedAt), ntPtr(a.UpdatedAt)); err != nil {
|
||||
return err
|
||||
}
|
||||
rep.count("resources.alarms", 1)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *migrator) phaseGroups(ctx context.Context, rep *report) error {
|
||||
rows, err := readGroups(ctx, m.groupsDB)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
// Copyright (c) Abstract Machines
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
package main
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestGroupRoleBlockPlansUseAtomSupportedScopes(t *testing.T) {
|
||||
scope := roleScope{prefix: "groups", scopeMode: "group"}
|
||||
roleID := "role-1"
|
||||
groupID := "11111111-1111-1111-1111-111111111111"
|
||||
tenantID := "22222222-2222-2222-2222-222222222222"
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
action string
|
||||
scopeMode string
|
||||
objectKind any
|
||||
objectType any
|
||||
objectID any
|
||||
groupID any
|
||||
}{
|
||||
{
|
||||
name: "direct device action", action: "client_read",
|
||||
scopeMode: "group_direct_objects", objectKind: "entity", objectType: "entity:device", groupID: groupID,
|
||||
},
|
||||
{
|
||||
name: "descendant device action", action: "subgroup_client_set_parent_group",
|
||||
scopeMode: "group_descendant_objects", objectKind: "entity", objectType: "entity:device", groupID: groupID,
|
||||
},
|
||||
{
|
||||
name: "direct channel action", action: "channel_publish",
|
||||
scopeMode: "group_direct_objects", objectKind: "resource", objectType: "resource:channel", groupID: groupID,
|
||||
},
|
||||
{
|
||||
name: "descendant channel action", action: "subgroup_channel_subscribe",
|
||||
scopeMode: "group_descendant_objects", objectKind: "resource", objectType: "resource:channel", groupID: groupID,
|
||||
},
|
||||
{
|
||||
name: "descendant group action", action: "subgroup_set_child",
|
||||
scopeMode: "group_descendant_groups", groupID: groupID,
|
||||
},
|
||||
{
|
||||
name: "self group action", action: "manage_role",
|
||||
scopeMode: "object", objectKind: "group", objectID: groupID,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
plans := scope.blockPlans(roleID, groupID, tenantID, tc.action)
|
||||
if len(plans) != 1 {
|
||||
t.Fatalf("expected one plan, got %d", len(plans))
|
||||
}
|
||||
got := plans[0]
|
||||
if got.ScopeMode != tc.scopeMode {
|
||||
t.Fatalf("scope mode = %q, want %q", got.ScopeMode, tc.scopeMode)
|
||||
}
|
||||
if got.TenantID != tenantID {
|
||||
t.Fatalf("tenant = %q, want %q", got.TenantID, tenantID)
|
||||
}
|
||||
if got.ObjectKind != tc.objectKind || got.ObjectType != tc.objectType || got.ObjectID != tc.objectID || got.GroupID != tc.groupID {
|
||||
t.Fatalf("plan = %+v, want objectKind=%v objectType=%v objectID=%v groupID=%v", got, tc.objectKind, tc.objectType, tc.objectID, tc.groupID)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMapActionPreservesChannelPublishSubscribeVariants(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"channel_publish": actionPublish,
|
||||
"subgroup_channel_publish": actionPublish,
|
||||
"channel_subscribe": actionSubscribe,
|
||||
"subgroup_channel_subscribe": actionSubscribe,
|
||||
}
|
||||
for raw, want := range cases {
|
||||
got, ok := mapAction(raw)
|
||||
if !ok {
|
||||
t.Fatalf("mapAction(%q) returned not ok", raw)
|
||||
}
|
||||
if got != want {
|
||||
t.Fatalf("mapAction(%q) = %q, want %q", raw, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -295,17 +295,6 @@ func (m *migrator) pfOrphans(ctx context.Context, rep *report) error {
|
||||
}
|
||||
return out
|
||||
}, "reports")
|
||||
alarms, err := readAlarms(ctx, m.alarmsDB)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
count(func() []string {
|
||||
out := make([]string, len(alarms))
|
||||
for i, a := range alarms {
|
||||
out[i] = a.DomainID
|
||||
}
|
||||
return out
|
||||
}, "alarms")
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -165,34 +165,6 @@ type srcReport struct {
|
||||
ReportTemplate sql.NullString `db:"report_template"`
|
||||
}
|
||||
|
||||
// srcAlarm is an alarm (alarms.alarms) -> Atom resource kind=alarm.
|
||||
type srcAlarm struct {
|
||||
ID string `db:"id"`
|
||||
RuleID string `db:"rule_id"`
|
||||
DomainID string `db:"domain_id"`
|
||||
ChannelID string `db:"channel_id"`
|
||||
Subtopic string `db:"subtopic"`
|
||||
ClientID string `db:"client_id"`
|
||||
Measurement string `db:"measurement"`
|
||||
Value string `db:"value"`
|
||||
Unit string `db:"unit"`
|
||||
Threshold string `db:"threshold"`
|
||||
Cause string `db:"cause"`
|
||||
Status int16 `db:"status"`
|
||||
Severity int16 `db:"severity"`
|
||||
AssigneeID sql.NullString `db:"assignee_id"`
|
||||
CreatedAt sql.NullTime `db:"created_at"`
|
||||
UpdatedAt sql.NullTime `db:"updated_at"`
|
||||
UpdatedBy sql.NullString `db:"updated_by"`
|
||||
AssignedAt sql.NullTime `db:"assigned_at"`
|
||||
AssignedBy sql.NullString `db:"assigned_by"`
|
||||
AcknowledgedAt sql.NullTime `db:"acknowledged_at"`
|
||||
AcknowledgedBy sql.NullString `db:"acknowledged_by"`
|
||||
ResolvedAt sql.NullTime `db:"resolved_at"`
|
||||
ResolvedBy sql.NullString `db:"resolved_by"`
|
||||
Metadata []byte `db:"metadata"`
|
||||
}
|
||||
|
||||
// --- readers ---
|
||||
|
||||
func readDomains(ctx context.Context, db *sqlx.DB) ([]srcDomain, error) {
|
||||
@@ -256,16 +228,6 @@ func readReports(ctx context.Context, db *sqlx.DB) ([]srcReport, error) {
|
||||
return out, db.SelectContext(ctx, &out, q)
|
||||
}
|
||||
|
||||
func readAlarms(ctx context.Context, db *sqlx.DB) ([]srcAlarm, error) {
|
||||
var out []srcAlarm
|
||||
q := `SELECT id, rule_id, domain_id, channel_id, subtopic, client_id, measurement, value,
|
||||
unit, threshold, cause, status, severity, assignee_id, created_at, updated_at,
|
||||
updated_by, assigned_at, assigned_by, acknowledged_at, acknowledged_by,
|
||||
resolved_at, resolved_by, metadata
|
||||
FROM alarms`
|
||||
return out, db.SelectContext(ctx, &out, q)
|
||||
}
|
||||
|
||||
// readRoleFamily reads <prefix>_roles, _role_actions, _role_members for one service.
|
||||
func readRoleFamily(ctx context.Context, db *sqlx.DB, prefix string) ([]srcRole, []srcRoleAction, []srcRoleMember, error) {
|
||||
var roles []srcRole
|
||||
|
||||
@@ -74,7 +74,7 @@ func (m *migrator) Verify(ctx context.Context, rep *report) error {
|
||||
return chans[i].ID, domSet[chans[i].DomainID]
|
||||
}), atomResources)
|
||||
|
||||
// 4b. resources: rules, reports, alarms
|
||||
// 4b. resources: rules, reports
|
||||
rules, err := readRules(ctx, m.reDB)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -89,13 +89,6 @@ func (m *migrator) Verify(ctx context.Context, rep *report) error {
|
||||
m.reconcile(rep, "resources.reports", idsOf(len(reports), func(i int) (string, bool) {
|
||||
return reports[i].ID, domSet[reports[i].DomainID]
|
||||
}), atomResources)
|
||||
alarms, err := readAlarms(ctx, m.alarmsDB)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
m.reconcile(rep, "resources.alarms", idsOf(len(alarms), func(i int) (string, bool) {
|
||||
return alarms[i].ID, domSet[alarms[i].DomainID]
|
||||
}), atomResources)
|
||||
|
||||
// 5. object_groups
|
||||
grps, err := readGroups(ctx, m.groupsDB)
|
||||
|
||||
Reference in New Issue
Block a user