NOISSUE - Revert alarms approach
Continuous Delivery / lint-and-build (push) Has been cancelled
Continuous Delivery / Build and Push Docker Images (push) Has been cancelled
CI Pipeline / Lint Proto (push) Has been cancelled
CI Pipeline / lint-and-build (push) Has been cancelled
CI Pipeline / Detect Changes (push) Has been cancelled
CI Pipeline / Test ${{ matrix.module }} (push) Has been cancelled
CI Pipeline / Upload Coverage (push) Has been cancelled
Property Based Tests / api-test (push) Has been cancelled
Deploy GitHub Pages / swagger-ui (push) Has been cancelled

Signed-off-by: dusan <borovcanindusan1@gmail.com>
This commit is contained in:
dusan
2026-06-29 19:16:16 +02:00
parent cdc065a7b2
commit c405028fc5
14 changed files with 116 additions and 329 deletions
+14 -17
View File
@@ -11,7 +11,7 @@ database. Implemented as Go scripts.
| Topic | Decision |
|-------|----------|
| Passwords | **Force reset.** Users migrate with no `password` credential; they reset via Atom's email flow on first login. (bcrypt → argon2 is not convertible without plaintext.) |
| Scope | Core IAM + roles & policies + connections + PATs + rules/reports/alarms as Atom resources. |
| Scope | Core IAM + roles & policies + connections + PATs + rules/report configs as Atom resources. Alarm events remain in the alarms service database and are not migrated into Atom resources. |
| Execution | **Offline one-shot.** Stop Magistrala app services, snapshot, transform, load, start Atom. |
| IDs | **Preserve Magistrala UUIDs** as Atom UUIDs (PKs and FKs). Magistrala IDs are 36-char UUID strings — directly usable as Atom `UUID` PKs. Keeps audit trails, message payloads, external references, and SpiceDB-derived links intact. |
@@ -33,7 +33,6 @@ All `magistrala/magistrala`, port 5432, on network `magistrala-base-net`:
| `auth-db` | `auth` | `pats`, `pat_scopes` (skip `keys` — short-lived JWTs; skip legacy `policies`/`domains` mirror) |
| `re-db` | `rules_engine` | `rules`, `rules_roles*` |
| `reports-db` | `reports` | `report_config`, `reports_roles*` |
| `alarms-db` | `alarms` | `alarms` |
> Note: `groups` migrations are embedded into clients **and** channels **and** the
> standalone groups service. The authoritative groups data for default compose is
@@ -105,27 +104,25 @@ principal group, matching Atom's normal `create_entity` side effect.
| `created_by` | `owner_id` (if the user migrated) |
| `parent_group_id` | `object_group_resources` membership |
### 3.4b rules / reports / alarms → resources
### 3.4b rules / report configs → resources
Rules-engine rules, report configs, and alarms are domain-scoped objects with no
Atom-native table, so they become **resources** alongside channels, distinguished
by `kind`. Service-specific columns Atom resources lack are folded into
`attributes` (JSONB). `tenant_id=domain_id`; rows whose domain has no surviving
tenant are skipped (§6.4). `owner_id`=`created_by` when that user migrated
(alarms have no `created_by` → NULL). Names are deduped per tenant (§6.3) since
these tables carry no `(domain_id, name)` constraint; alarms (no name) use
`measurement` with an `id` fallback.
Rules-engine rules and report configs are domain-scoped configuration objects
with no Atom-native table, so they become **resources** alongside channels,
distinguished by `kind`. Service-specific columns Atom resources lack are folded
into `attributes` (JSONB). `tenant_id=domain_id`; rows whose domain has no
surviving tenant are skipped (§6.4). `owner_id`=`created_by` when that user
migrated. Names are deduped per tenant (§6.3) since these tables carry no
`(domain_id, name)` constraint.
| Source | Atom `resources` |
|---|---|
| `rules_engine.rules` (id) | `kind='rule'`; `input_channel/topic, outputs, logic_type/value, recurring*, time, start_datetime, tags, status``attributes` |
| `reports.report_config` (id) | `kind='report'`; `description, config, email, metrics, report_template, due, recurring*, start_datetime, status``attributes` |
| `alarms.alarms` (id) | `kind='alarm'`; `rule_id, channel_id, client_id, subtopic, measurement, value, unit, threshold, cause, severity, alarm_status, assignee/assigned/acknowledged/resolved*``attributes` |
> Atom `resources.kind` must permit `rule`, `report`, `alarm` (in addition to
> `channel`). Rules and reports have object-specific role families and those are
> migrated as resource-scoped roles. Alarms have no role family or
> `parent_group_id`, so only the resource rows are migrated for alarms.
> Atom `resources.kind` must permit `rule` and `report` (in addition to
> `channel`). Rules and report configs have object-specific role families and
> those are migrated as resource-scoped roles. Alarm events can be high-volume
> operational data, so they stay in `alarms-db` and are not Atom resources.
### 3.5 groups → object_groups
Magistrala groups organize clients/channels within a domain (hierarchical,
@@ -274,7 +271,7 @@ Checks below; the email check matters mainly for dumps merged across instances
3. …then backfill tenants.created_by/updated_by and resources.owner_id
4. entity_emails
5. credentials (device api_key; PAT metadata)
6. resources (channels, rules, reports, alarms)
6. resources (channels, rules, reports)
7. object_groups → object_group_hierarchy → object_group_entities/resources
8. roles → permission_blocks → permission_block_actions → role_permission_blocks
9. role_assignments, direct_policies
+2 -2
View File
@@ -21,14 +21,14 @@ docker build -f tools/atom-migration/Dockerfile -t magistrala/atom-migration:dev
## Start only the source databases
The migrator reads Postgres directly — it does **not** need the Magistrala app
services running. To migrate from restored volumes, start just the nine source DB
services running. To migrate from restored volumes, start just the eight source DB
containers (`--no-deps` keeps compose from pulling in the app services they
depend on):
```bash
docker compose -f docker/docker-compose.yaml up -d --no-deps \
auth-db users-db domains-db clients-db channels-db groups-db \
re-db reports-db alarms-db
re-db reports-db
```
They mount the `magistrala_magistrala-<svc>-db-volume` volumes and attach to
-3
View File
@@ -39,7 +39,6 @@ type config struct {
Auth dbConn
RE dbConn // rules engine
Reports dbConn
Alarms dbConn
AtomDSN string
UnmappedAction string
@@ -78,7 +77,6 @@ func loadConfig(envPath, atomDSN string, fromHost bool) (config, error) {
Auth: mk("MG_AUTH"),
RE: mk("MG_RE"),
Reports: mk("MG_REPORTS"),
Alarms: mk("MG_ALARMS"),
AtomDSN: atomDSN,
}
@@ -88,7 +86,6 @@ func loadConfig(envPath, atomDSN string, fromHost bool) (config, error) {
&cfg.Clients.Name: "clients", &cfg.Channels.Name: "channels",
&cfg.Groups.Name: "groups", &cfg.Auth.Name: "auth",
&cfg.RE.Name: "rules_engine", &cfg.Reports.Name: "reports",
&cfg.Alarms.Name: "alarms",
}
for p, n := range defName {
if *p == "" {
+5 -60
View File
@@ -29,7 +29,6 @@ type migrator struct {
authDB *sqlx.DB
reDB *sqlx.DB
reportsDB *sqlx.DB
alarmsDB *sqlx.DB
atom *sqlx.DB
profileID map[string]string // profile key (e.g. "user","client") -> uuid
@@ -97,7 +96,6 @@ func newMigrator(ctx context.Context, cfg config, apply bool) (*migrator, error)
m.authDB = open("auth", cfg.Auth.DSN())
m.reDB = open("rules_engine", cfg.RE.DSN())
m.reportsDB = open("reports", cfg.Reports.DSN())
m.alarmsDB = open("alarms", cfg.Alarms.DSN())
m.atom = open("atom", cfg.AtomDSN)
if err != nil {
return nil, err
@@ -108,7 +106,7 @@ func newMigrator(ctx context.Context, cfg config, apply bool) (*migrator, error)
const authenticatedUsersGroupID = "00000000-0000-0000-0000-000000000005"
func (m *migrator) Close() {
for _, db := range []*sqlx.DB{m.domainsDB, m.usersDB, m.clientsDB, m.channelsDB, m.groupsDB, m.authDB, m.reDB, m.reportsDB, m.alarmsDB, m.atom} {
for _, db := range []*sqlx.DB{m.domainsDB, m.usersDB, m.clientsDB, m.channelsDB, m.groupsDB, m.authDB, m.reDB, m.reportsDB, m.atom} {
if db != nil {
_ = db.Close()
}
@@ -140,7 +138,6 @@ func (m *migrator) Run(ctx context.Context, rep *report) error {
{"resources.channels", m.phaseChannels},
{"resources.rules", m.phaseRules},
{"resources.reports", m.phaseReports},
{"resources.alarms", m.phaseAlarms},
{"object_groups", m.phaseGroups},
{"group_membership", m.phaseGroupMembership},
{"roles", m.phaseRoles},
@@ -427,9 +424,9 @@ func (m *migrator) phaseChannels(ctx context.Context, rep *report) error {
return nil
}
// insertResource writes one row into Atom resources (kind = channel/rule/report/
// alarm). Entity-specific columns Magistrala has but Atom resources lack are
// folded into the attributes JSONB. ON CONFLICT (id) keeps it idempotent.
// insertResource writes one row into Atom resources. Entity-specific columns
// Magistrala has but Atom resources lack are folded into the attributes JSONB.
// ON CONFLICT (id) keeps it idempotent.
func (m *migrator) insertResource(ctx context.Context, id, kind, name, tenant string, owner sql.NullString, attributes string, createdAt time.Time, updatedAt any) error {
return m.exec(ctx,
`INSERT INTO resources (id, kind, name, tenant_id, owner_id, attributes, created_at, updated_at)
@@ -447,7 +444,7 @@ func (m *migrator) ownerOf(createdBy sql.NullString) sql.NullString {
}
// uniqueResName makes a resource name unique within a tenant (Atom enforces
// resources(name, tenant_id); rules/reports/alarms carry no such Magistrala
// resources(name, tenant_id); rules/reports carry no such Magistrala
// constraint, so same-tenant dups are possible). Suffixes -2, -3, … on collision.
func uniqueResName(seen map[string]int, tenant, name string) string {
key := tenant + "|" + strings.ToLower(name)
@@ -557,58 +554,6 @@ func (m *migrator) phaseReports(ctx context.Context, rep *report) error {
return nil
}
// phaseAlarms: alarms.alarms -> resources (kind=alarm). Alarms have no name in
// Magistrala; the measurement is used (id fallback), deduped per tenant.
func (m *migrator) phaseAlarms(ctx context.Context, rep *report) error {
rows, err := readAlarms(ctx, m.alarmsDB)
if err != nil {
return err
}
seen := map[string]int{}
for _, a := range rows {
if !m.tenants[a.DomainID] {
rep.skip("alarm_orphan_domain")
continue
}
m.resourceDomain[a.ID] = a.DomainID
name := uniqueResName(seen, a.DomainID, firstNonEmpty(a.Measurement, a.ID))
extra := map[string]any{
"rule_id": a.RuleID,
"channel_id": a.ChannelID,
"client_id": a.ClientID,
"subtopic": a.Subtopic,
"measurement": a.Measurement,
"value": a.Value,
"unit": a.Unit,
"threshold": a.Threshold,
"cause": a.Cause,
"alarm_status": a.Status,
"severity": a.Severity,
}
putStr(extra, "assignee_id", a.AssigneeID)
putStr(extra, "updated_by", a.UpdatedBy)
putStr(extra, "assigned_by", a.AssignedBy)
putStr(extra, "acknowledged_by", a.AcknowledgedBy)
putStr(extra, "resolved_by", a.ResolvedBy)
if a.AssignedAt.Valid {
extra["assigned_at"] = a.AssignedAt.Time
}
if a.AcknowledgedAt.Valid {
extra["acknowledged_at"] = a.AcknowledgedAt.Time
}
if a.ResolvedAt.Valid {
extra["resolved_at"] = a.ResolvedAt.Time
}
// Alarms carry no created_by; owner_id stays NULL.
if err := m.insertResource(ctx, a.ID, "alarm", name, a.DomainID, sql.NullString{},
attrs(a.Metadata, extra), ntToTime(a.CreatedAt), ntPtr(a.UpdatedAt)); err != nil {
return err
}
rep.count("resources.alarms", 1)
}
return nil
}
func (m *migrator) phaseGroups(ctx context.Context, rep *report) error {
rows, err := readGroups(ctx, m.groupsDB)
if err != nil {
+85
View File
@@ -0,0 +1,85 @@
// Copyright (c) Abstract Machines
// SPDX-License-Identifier: Apache-2.0
package main
import "testing"
func TestGroupRoleBlockPlansUseAtomSupportedScopes(t *testing.T) {
scope := roleScope{prefix: "groups", scopeMode: "group"}
roleID := "role-1"
groupID := "11111111-1111-1111-1111-111111111111"
tenantID := "22222222-2222-2222-2222-222222222222"
cases := []struct {
name string
action string
scopeMode string
objectKind any
objectType any
objectID any
groupID any
}{
{
name: "direct device action", action: "client_read",
scopeMode: "group_direct_objects", objectKind: "entity", objectType: "entity:device", groupID: groupID,
},
{
name: "descendant device action", action: "subgroup_client_set_parent_group",
scopeMode: "group_descendant_objects", objectKind: "entity", objectType: "entity:device", groupID: groupID,
},
{
name: "direct channel action", action: "channel_publish",
scopeMode: "group_direct_objects", objectKind: "resource", objectType: "resource:channel", groupID: groupID,
},
{
name: "descendant channel action", action: "subgroup_channel_subscribe",
scopeMode: "group_descendant_objects", objectKind: "resource", objectType: "resource:channel", groupID: groupID,
},
{
name: "descendant group action", action: "subgroup_set_child",
scopeMode: "group_descendant_groups", groupID: groupID,
},
{
name: "self group action", action: "manage_role",
scopeMode: "object", objectKind: "group", objectID: groupID,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
plans := scope.blockPlans(roleID, groupID, tenantID, tc.action)
if len(plans) != 1 {
t.Fatalf("expected one plan, got %d", len(plans))
}
got := plans[0]
if got.ScopeMode != tc.scopeMode {
t.Fatalf("scope mode = %q, want %q", got.ScopeMode, tc.scopeMode)
}
if got.TenantID != tenantID {
t.Fatalf("tenant = %q, want %q", got.TenantID, tenantID)
}
if got.ObjectKind != tc.objectKind || got.ObjectType != tc.objectType || got.ObjectID != tc.objectID || got.GroupID != tc.groupID {
t.Fatalf("plan = %+v, want objectKind=%v objectType=%v objectID=%v groupID=%v", got, tc.objectKind, tc.objectType, tc.objectID, tc.groupID)
}
})
}
}
func TestMapActionPreservesChannelPublishSubscribeVariants(t *testing.T) {
cases := map[string]string{
"channel_publish": actionPublish,
"subgroup_channel_publish": actionPublish,
"channel_subscribe": actionSubscribe,
"subgroup_channel_subscribe": actionSubscribe,
}
for raw, want := range cases {
got, ok := mapAction(raw)
if !ok {
t.Fatalf("mapAction(%q) returned not ok", raw)
}
if got != want {
t.Fatalf("mapAction(%q) = %q, want %q", raw, got, want)
}
}
}
-11
View File
@@ -295,17 +295,6 @@ func (m *migrator) pfOrphans(ctx context.Context, rep *report) error {
}
return out
}, "reports")
alarms, err := readAlarms(ctx, m.alarmsDB)
if err != nil {
return err
}
count(func() []string {
out := make([]string, len(alarms))
for i, a := range alarms {
out[i] = a.DomainID
}
return out
}, "alarms")
return nil
}
-38
View File
@@ -165,34 +165,6 @@ type srcReport struct {
ReportTemplate sql.NullString `db:"report_template"`
}
// srcAlarm is an alarm (alarms.alarms) -> Atom resource kind=alarm.
type srcAlarm struct {
ID string `db:"id"`
RuleID string `db:"rule_id"`
DomainID string `db:"domain_id"`
ChannelID string `db:"channel_id"`
Subtopic string `db:"subtopic"`
ClientID string `db:"client_id"`
Measurement string `db:"measurement"`
Value string `db:"value"`
Unit string `db:"unit"`
Threshold string `db:"threshold"`
Cause string `db:"cause"`
Status int16 `db:"status"`
Severity int16 `db:"severity"`
AssigneeID sql.NullString `db:"assignee_id"`
CreatedAt sql.NullTime `db:"created_at"`
UpdatedAt sql.NullTime `db:"updated_at"`
UpdatedBy sql.NullString `db:"updated_by"`
AssignedAt sql.NullTime `db:"assigned_at"`
AssignedBy sql.NullString `db:"assigned_by"`
AcknowledgedAt sql.NullTime `db:"acknowledged_at"`
AcknowledgedBy sql.NullString `db:"acknowledged_by"`
ResolvedAt sql.NullTime `db:"resolved_at"`
ResolvedBy sql.NullString `db:"resolved_by"`
Metadata []byte `db:"metadata"`
}
// --- readers ---
func readDomains(ctx context.Context, db *sqlx.DB) ([]srcDomain, error) {
@@ -256,16 +228,6 @@ func readReports(ctx context.Context, db *sqlx.DB) ([]srcReport, error) {
return out, db.SelectContext(ctx, &out, q)
}
func readAlarms(ctx context.Context, db *sqlx.DB) ([]srcAlarm, error) {
var out []srcAlarm
q := `SELECT id, rule_id, domain_id, channel_id, subtopic, client_id, measurement, value,
unit, threshold, cause, status, severity, assignee_id, created_at, updated_at,
updated_by, assigned_at, assigned_by, acknowledged_at, acknowledged_by,
resolved_at, resolved_by, metadata
FROM alarms`
return out, db.SelectContext(ctx, &out, q)
}
// readRoleFamily reads <prefix>_roles, _role_actions, _role_members for one service.
func readRoleFamily(ctx context.Context, db *sqlx.DB, prefix string) ([]srcRole, []srcRoleAction, []srcRoleMember, error) {
var roles []srcRole
+1 -8
View File
@@ -74,7 +74,7 @@ func (m *migrator) Verify(ctx context.Context, rep *report) error {
return chans[i].ID, domSet[chans[i].DomainID]
}), atomResources)
// 4b. resources: rules, reports, alarms
// 4b. resources: rules, reports
rules, err := readRules(ctx, m.reDB)
if err != nil {
return err
@@ -89,13 +89,6 @@ func (m *migrator) Verify(ctx context.Context, rep *report) error {
m.reconcile(rep, "resources.reports", idsOf(len(reports), func(i int) (string, bool) {
return reports[i].ID, domSet[reports[i].DomainID]
}), atomResources)
alarms, err := readAlarms(ctx, m.alarmsDB)
if err != nil {
return err
}
m.reconcile(rep, "resources.alarms", idsOf(len(alarms), func(i int) (string, bool) {
return alarms[i].ID, domSet[alarms[i].DomainID]
}), atomResources)
// 5. object_groups
grps, err := readGroups(ctx, m.groupsDB)