mirror of
https://github.com/rajnandan1/kener.git
synced 2026-08-07 07:14:56 +00:00
Compare commits
11 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 74b6311e81 | |||
| c4c16d65a6 | |||
| 758cf5e4d5 | |||
| 7c17db12dc | |||
| d9954ea085 | |||
| 305c0a05fe | |||
| f83447f806 | |||
| 42ca67b172 | |||
| 806df0d73c | |||
| 5f66cada43 | |||
| 85788c76f7 |
+10
-2
@@ -43,7 +43,15 @@ _Avoid_: Data point, record, check result
|
||||
A Monitoring Sample produced by a check that actually ran against the target, whatever the outcome: a clean evaluation (`REALTIME`), a timed-out check (`TIMEOUT`), or a check that errored (`ERROR`). A check failing to reach the target is itself an observation — for most monitor types that is exactly what "down" looks like.
|
||||
|
||||
**Synthetic Sample**:
|
||||
A Monitoring Sample written by the system or an admin rather than by a check: a raw heartbeat receipt (`SIGNAL`), a status pushed through the data API (`MANUAL`), a default-status fill (`DEFAULT_STATUS`), or an incident/maintenance overlay (`INCIDENT`, `MAINTENANCE`).
|
||||
A Monitoring Sample written by the system or an admin rather than by a check: a raw heartbeat receipt (`SIGNAL`), a status pushed through the data API (`MANUAL`), a default-status fill (`DEFAULT_STATUS`), a last-known-status fill (`CARRIED`), or an incident/maintenance overlay (`INCIDENT`, `MAINTENANCE`).
|
||||
|
||||
**Default Status**:
|
||||
A monitor's answer to what a minute without a Monitoring Sample means. Exactly one choice from a closed set: nothing (`NONE` — the minute shows no data), a fixed status (`UP`, `DOWN`, `DEGRADED`) written as default-status fill, or Last Known Status. `MAINTENANCE` is not a Default Status (a maintenance overlay is an event, not a fill).
|
||||
_Avoid_: Fallback status, fill status
|
||||
|
||||
**Last Known Status**:
|
||||
A Default Status choice where a minute without a sample repeats the most recent Alert-Visible Sample — status and latency alike — written as a Carried Sample (`CARRIED`). Carried Samples are themselves alert-visible, so the chain continues from the last live statement: overlays and heartbeat receipts never become sticky, backdated corrections do not change the present, and alerts trigger and resolve on carried minutes like any other. Carry never expires and never backfills: it starts at the tick after the choice is made, and Carried Samples persist as history if the choice is later changed. A monitor with no Alert-Visible Sample yet has nothing to carry — its minutes show no data. Only None-type monitors may choose it; changing the monitor's type away from None resets the Default Status to UP.
|
||||
_Avoid_: Sticky status, carry-forward mode
|
||||
|
||||
**Stale Member**:
|
||||
A Member whose monitor is no longer an Eligible Monitor (paused or deleted after being added). It remains a Member until explicitly removed, but is excluded from the group score.
|
||||
@@ -63,7 +71,7 @@ A notification channel (email, webhook, Discord, Slack) that Alert Configuration
|
||||
_Avoid_: Notifier, channel
|
||||
|
||||
**Alert-Visible Sample**:
|
||||
A Monitoring Sample that alert evaluation can see: every Observed Sample, plus data-API pushes (`MANUAL`) and default-status fill (`DEFAULT_STATUS`). Raw heartbeat receipts (`SIGNAL`) and incident/maintenance overlays are never alert-visible — while an overlay is active the alert window freezes (alerts neither trigger nor resolve). All alert conditions (status and latency alike) evaluate the same alert-visible timeline.
|
||||
A Monitoring Sample that alert evaluation can see: every Observed Sample, plus data-API pushes (`MANUAL`), default-status fill (`DEFAULT_STATUS`), and last-known-status fill (`CARRIED`). Raw heartbeat receipts (`SIGNAL`) and incident/maintenance overlays are never alert-visible — while an overlay is active the alert window freezes (alerts neither trigger nor resolve). All alert conditions (status and latency alike) evaluate the same alert-visible timeline.
|
||||
|
||||
**Failure Threshold**:
|
||||
The number of consecutive Alert-Visible Samples matching the condition required to trigger an Alert.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Alerts evaluate alert-visible samples, not just REALTIME ones
|
||||
|
||||
The consecutive-sample checks behind alert evaluation (`consecutivelyStatusFor`, `consecutivelyLatencyGreaterThan`, `consecutivelyLatencyLessThan` in `src/lib/server/db/repositories/monitoring.ts`) consider samples whose type is `REALTIME`, `ERROR`, `TIMEOUT`, `MANUAL`, or `DEFAULT_STATUS` — the "alert-visible" set — instead of `REALTIME` only. Both data-API PATCH endpoints (single timestamp and range) enqueue one alert evaluation after writing `MANUAL` rows. `SIGNAL` rows and `INCIDENT`/`MAINTENANCE` overlay rows remain invisible to alerting.
|
||||
The consecutive-sample checks behind alert evaluation (`consecutivelyStatusFor`, `consecutivelyLatencyGreaterThan`, `consecutivelyLatencyLessThan` in `src/lib/server/db/repositories/monitoring.ts`) consider samples whose type is `REALTIME`, `ERROR`, `TIMEOUT`, `MANUAL`, or `DEFAULT_STATUS` — the "alert-visible" set — instead of `REALTIME` only. Both data-API PATCH endpoints (single timestamp and range) enqueue one alert evaluation after writing `MANUAL` rows. `SIGNAL` rows and `INCIDENT`/`MAINTENANCE` overlay rows remain invisible to alerting. Amended by ADR 0006: last-known-status fill (`CARRIED`) later joined the alert-visible set under the same invariant.
|
||||
|
||||
Two issues drove this. In #633, a GameDig monitor showed DOWN on the status page but never alerted: a down game server makes `GameDig.query` throw, so every down-sample is recorded as `ERROR`, which the old `type = REALTIME` filter excluded — the "N consecutive DOWN" condition could never become true. The same failure mode silently broke gRPC, SQL, and SSL monitors (hard-down records `ERROR`) and API monitors whose outage manifests as timeouts (`TIMEOUT`). In #720, a NONE monitor driven by the data API never alerted for two stacked reasons: PATCH writes `MANUAL` rows the filter excluded, and the endpoint never enqueued evaluation at all. The status page and UPTIME alerts have no type filter, which is why users saw DOWN while alerts stayed silent.
|
||||
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
# Last Known Status is a Default Status choice that repeats the latest alert-visible sample
|
||||
|
||||
Issue #721: push-driven NONE monitors lost their status between pushes in v4 — one red minute, then gray forever — because the per-minute sample model only fills gaps with a static `default_status`. We added a fifth Default Status choice, `LAST_KNOWN`, where each tick with no observed sample writes a `CARRIED` row repeating the most recent alert-visible sample — status and latency alike. It is modeled as a dropdown value rather than a separate "sticky" checkbox so that "what does a minute without a sample mean" stays a single dimension with no conflicting combinations; the same cleanup removed the `MAINTENANCE` option, which the UI offered but the fill engine had always silently ignored (stored `MAINTENANCE`/unknown values migrate to `NONE`, preserving behavior).
|
||||
|
||||
The carry source is the most recent **Alert-Visible Sample** by timestamp, and `CARRIED` itself joins the alert-visible set. That one rule does three jobs: incident/maintenance overlays and raw heartbeat `SIGNAL` receipts can never become sticky; backdated data-API corrections cannot rewrite the present (newer carried rows outrank them); and ADR 0005's invariant — every flow that enqueues alert evaluation contributes a row the evaluator can see — keeps holding. The consequences were accepted deliberately: a single DOWN push triggers status alerts once carried minutes meet the failure threshold, alerts never auto-resolve (recovery must be pushed — unlike a fixed default, nothing "resumes"), and enabling the option on a monitor whose last push was DOWN fires the alert shortly after — the bug report, inverted, same as ADR 0005.
|
||||
|
||||
Rejected alternatives: a staleness cap ("carry for at most X, then gray") re-introduces "absence means unknown" — the opposite of what the admin just selected — and dead-integration detection is what Heartbeat monitors are for; reusing the `DEFAULT` sample type for carried rows loses the stored distinction between "admin declared absence means X" and "system repeated the last live statement"; backfilling the gap on enable would mutate historical uptime, so carry is tick-forward only and `CARRIED` rows persist as history if the setting is later changed. `LAST_KNOWN` is selectable only on NONE-type monitors — for polled types an observed sample wins the merge every tick, so offering it would be a dormant knob; changing a monitor's type away from NONE auto-resets the Default Status to UP so the invalid combination never persists.
|
||||
@@ -0,0 +1,794 @@
|
||||
# Last Known Status (fix #721) Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add a `LAST_KNOWN` Default Status choice for NONE-type (Manual) monitors: each scheduler tick without new data writes a `CARRIED` sample repeating the most recent alert-visible sample (status + latency), so push-driven monitors keep their status between pushes.
|
||||
|
||||
**Architecture:** The carry fill slots into the existing `defaultData` branch of the monitor-execute worker (`monitorExecuteQueue.ts`), sourcing from a new repository query for the latest alert-visible sample. A single normalization helper in `monitorsController.ts` enforces the closed `default_status` value set (`NONE|UP|DOWN|DEGRADED|LAST_KNOWN`) and the "LAST_KNOWN only on NONE-type, auto-reset to UP otherwise" rule across all three monitor write paths (manage UI action, v4 POST, v4 PATCH). A migration normalizes legacy values (`MAINTENANCE`/unknown/NULL → `NONE`).
|
||||
|
||||
**Tech Stack:** SvelteKit 2 (Svelte 5 runes), Knex migrations, BullMQ workers, shadcn-svelte UI.
|
||||
|
||||
**Design authority:** `docs/adr/0006-last-known-status-fill.md` and the `Default Status` / `Last Known Status` / `Alert-Visible Sample` entries in `CONTEXT.md`. If a step seems to contradict those, the docs win.
|
||||
|
||||
**Verification approach:** This repo has NO test infrastructure (no test script, no tests/ dir). Backend logic is verified with throwaway `vite-node` scripts driving repository classes against in-memory better-sqlite3 (delete the scripts before committing), plus a live end-to-end pass against the dev server. `npm run check` gates every commit.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Constants + alert-visible whitelist
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/lib/global-constants.ts:43`
|
||||
- Modify: `src/lib/server/db/repositories/monitoring.ts:13-20`
|
||||
|
||||
- [ ] **Step 1: Add the two constants**
|
||||
|
||||
In `src/lib/global-constants.ts`, the default export object currently has (line 43):
|
||||
|
||||
```typescript
|
||||
DEFAULT_STATUS: "DEFAULT",
|
||||
```
|
||||
|
||||
Add two lines directly after it:
|
||||
|
||||
```typescript
|
||||
DEFAULT_STATUS: "DEFAULT",
|
||||
CARRIED: "CARRIED",
|
||||
LAST_KNOWN: "LAST_KNOWN",
|
||||
```
|
||||
|
||||
(`CARRIED` is a **sample type** written by last-known-status fill; `LAST_KNOWN` is a **default_status value** stored on the monitor. They are different namespaces that happen to live in the same constants object — keep both names exactly as above.)
|
||||
|
||||
- [ ] **Step 2: Add CARRIED to the alert-visible whitelist**
|
||||
|
||||
In `src/lib/server/db/repositories/monitoring.ts`, replace lines 13-20:
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Sample types alert evaluation can see (see docs/adr/0005-alerts-evaluate-alert-visible-samples.md).
|
||||
* Exactly the types written by flows that enqueue alert evaluation: scheduler checks
|
||||
* (REALTIME/ERROR/TIMEOUT), default-status fill (DEFAULT_STATUS), and data-API pushes (MANUAL).
|
||||
* SIGNAL rows (raw heartbeat receipts) and INCIDENT/MAINTENANCE overlays stay invisible, so the
|
||||
* alert window freezes during manual overlays instead of triggering or resolving on them.
|
||||
*/
|
||||
const ALERT_VISIBLE_TYPES = [GC.REALTIME, GC.ERROR, GC.TIMEOUT, GC.MANUAL, GC.DEFAULT_STATUS]
|
||||
```
|
||||
|
||||
with:
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Sample types alert evaluation can see (see docs/adr/0005-alerts-evaluate-alert-visible-samples.md
|
||||
* and docs/adr/0006-last-known-status-fill.md).
|
||||
* Exactly the types written by flows that enqueue alert evaluation: scheduler checks
|
||||
* (REALTIME/ERROR/TIMEOUT), default-status fill (DEFAULT_STATUS), last-known-status fill (CARRIED),
|
||||
* and data-API pushes (MANUAL).
|
||||
* SIGNAL rows (raw heartbeat receipts) and INCIDENT/MAINTENANCE overlays stay invisible, so the
|
||||
* alert window freezes during manual overlays instead of triggering or resolving on them.
|
||||
*/
|
||||
const ALERT_VISIBLE_TYPES = [GC.REALTIME, GC.ERROR, GC.TIMEOUT, GC.MANUAL, GC.DEFAULT_STATUS, GC.CARRIED]
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Type-check**
|
||||
|
||||
Run: `npm run check`
|
||||
Expected: 0 errors (same error/warning count as before the change — run it on a clean tree first if unsure).
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add src/lib/global-constants.ts src/lib/server/db/repositories/monitoring.ts
|
||||
git commit -m "feat(constants): add CARRIED sample type and LAST_KNOWN default status"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Repository — latest alert-visible sample query
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/lib/server/db/repositories/monitoring.ts` (after `getLatestMonitoringData`, line 79)
|
||||
- Modify: `src/lib/server/db/dbimpl.ts:52-53` (declaration) and `:412` (binding)
|
||||
|
||||
- [ ] **Step 1: Add the repository method**
|
||||
|
||||
In `src/lib/server/db/repositories/monitoring.ts`, directly after the `getLatestMonitoringData` method (ends line 79), add:
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Latest sample the alert evaluator (and last-known-status fill) can see.
|
||||
* Carry source for Default Status = LAST_KNOWN (docs/adr/0006): overlays
|
||||
* (INCIDENT/MAINTENANCE) and raw heartbeat receipts (SIGNAL) are excluded,
|
||||
* so they can never become sticky.
|
||||
*/
|
||||
async getLatestAlertVisibleData(monitor_tag: string): Promise<MonitoringData | undefined> {
|
||||
return await this.knex("monitoring_data")
|
||||
.where("monitor_tag", monitor_tag)
|
||||
.whereIn("type", ALERT_VISIBLE_TYPES)
|
||||
.orderBy("timestamp", "desc")
|
||||
.limit(1)
|
||||
.first();
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Expose it on the db singleton**
|
||||
|
||||
In `src/lib/server/db/dbimpl.ts`, after line 52 (`getLatestMonitoringData!: ...`), add the declaration:
|
||||
|
||||
```typescript
|
||||
getLatestAlertVisibleData!: MonitoringRepository["getLatestAlertVisibleData"];
|
||||
```
|
||||
|
||||
and after line 412 (`this.getLatestMonitoringData = ...bind(this.monitoring);`), add the binding:
|
||||
|
||||
```typescript
|
||||
this.getLatestAlertVisibleData = this.monitoring.getLatestAlertVisibleData.bind(this.monitoring)
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Write the throwaway verification script**
|
||||
|
||||
Create `scripts/tmp-verify-carry-source.ts` (will be deleted, never committed):
|
||||
|
||||
```typescript
|
||||
import Knex from "knex"
|
||||
import { MonitoringRepository } from "../src/lib/server/db/repositories/monitoring"
|
||||
|
||||
const knex = Knex({ client: "better-sqlite3", connection: { filename: ":memory:" }, useNullAsDefault: true })
|
||||
|
||||
await knex.schema.createTable("monitoring_data", (t) => {
|
||||
t.string("monitor_tag")
|
||||
t.integer("timestamp")
|
||||
t.string("status")
|
||||
t.float("latency")
|
||||
t.string("type")
|
||||
t.text("error_message")
|
||||
t.primary(["monitor_tag", "timestamp"])
|
||||
})
|
||||
|
||||
const repo = new MonitoringRepository(knex)
|
||||
|
||||
// Timeline: MANUAL DOWN, then a CARRIED copy, then an INCIDENT overlay, then a SIGNAL receipt.
|
||||
await knex("monitoring_data").insert([
|
||||
{ monitor_tag: "t", timestamp: 100, status: "DOWN", latency: 42, type: "MANUAL" },
|
||||
{ monitor_tag: "t", timestamp: 160, status: "DOWN", latency: 42, type: "CARRIED" },
|
||||
{ monitor_tag: "t", timestamp: 220, status: "UP", latency: 0, type: "INCIDENT" },
|
||||
{ monitor_tag: "t", timestamp: 280, status: "UP", latency: 0, type: "SIGNAL" }
|
||||
])
|
||||
|
||||
const latest = await repo.getLatestAlertVisibleData("t")
|
||||
console.log("latest:", latest)
|
||||
if (!latest || latest.timestamp !== 160 || latest.type !== "CARRIED" || latest.status !== "DOWN") {
|
||||
throw new Error("FAIL: expected the CARRIED row at ts=160 (INCIDENT/SIGNAL must be skipped)")
|
||||
}
|
||||
|
||||
const none = await repo.getLatestAlertVisibleData("missing")
|
||||
if (none !== undefined) throw new Error("FAIL: expected undefined for unknown tag")
|
||||
|
||||
console.log("PASS")
|
||||
await knex.destroy()
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run it**
|
||||
|
||||
Run: `npx vite-node scripts/tmp-verify-carry-source.ts`
|
||||
Expected: prints the ts=160 CARRIED row, then `PASS`.
|
||||
|
||||
- [ ] **Step 5: Delete the script, type-check, commit**
|
||||
|
||||
```bash
|
||||
rm scripts/tmp-verify-carry-source.ts
|
||||
npm run check
|
||||
git add src/lib/server/db/repositories/monitoring.ts src/lib/server/db/dbimpl.ts
|
||||
git commit -m "feat(db): add getLatestAlertVisibleData query for last-known-status carry source"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Engine — carry fill in the execute worker
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/lib/server/queues/monitorExecuteQueue.ts:125-156`
|
||||
|
||||
- [ ] **Step 1: Extend the defaultData branch**
|
||||
|
||||
In `src/lib/server/queues/monitorExecuteQueue.ts`, replace lines 125-139:
|
||||
|
||||
```typescript
|
||||
let defaultData: MonitoringResultTS = {}
|
||||
let mergedData: MonitoringResultTS = {}
|
||||
|
||||
if (monitor.default_status !== undefined && monitor.default_status !== null) {
|
||||
if (([GC.UP, GC.DOWN, GC.DEGRADED] as string[]).indexOf(monitor.default_status) !== -1) {
|
||||
defaultData[ts] = {
|
||||
status: monitor.default_status,
|
||||
latency: 0,
|
||||
type: GC.DEFAULT_STATUS
|
||||
}
|
||||
if (monitor.default_status !== GC.UP) {
|
||||
defaultData[ts].error_message = "Default status applied"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
with:
|
||||
|
||||
```typescript
|
||||
let defaultData: MonitoringResultTS = {}
|
||||
let mergedData: MonitoringResultTS = {}
|
||||
|
||||
if (monitor.default_status !== undefined && monitor.default_status !== null) {
|
||||
if (([GC.UP, GC.DOWN, GC.DEGRADED] as string[]).indexOf(monitor.default_status) !== -1) {
|
||||
defaultData[ts] = {
|
||||
status: monitor.default_status,
|
||||
latency: 0,
|
||||
type: GC.DEFAULT_STATUS
|
||||
}
|
||||
if (monitor.default_status !== GC.UP) {
|
||||
defaultData[ts].error_message = "Default status applied"
|
||||
}
|
||||
} else if (monitor.default_status === GC.LAST_KNOWN) {
|
||||
// Last Known Status fill (docs/adr/0006): repeat the most recent alert-visible
|
||||
// sample — status and latency alike. No sample yet → nothing to carry → no fill.
|
||||
const lastKnown = await db.getLatestAlertVisibleData(monitor.tag)
|
||||
if (lastKnown && lastKnown.status) {
|
||||
defaultData[ts] = {
|
||||
status: lastKnown.status,
|
||||
latency: lastKnown.latency ?? 0,
|
||||
type: GC.CARRIED
|
||||
}
|
||||
if (lastKnown.status !== GC.UP) {
|
||||
defaultData[ts].error_message = "Last known status applied"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Fix the NO_DATA-preference block to preserve the fill's type**
|
||||
|
||||
Still in the same file, the block at (previously) lines 141-156 hardcodes `type: GC.DEFAULT_STATUS` when realtime returns NO_DATA but a fill exists. A heartbeat monitor with `LAST_KNOWN` would mislabel its carried rows. Replace:
|
||||
|
||||
```typescript
|
||||
const defaultStatus = defaultData[ts]?.status
|
||||
const realtimeStatus = realtimeData[ts]?.status
|
||||
let realtimeDataForMerge = realtimeData
|
||||
if (defaultStatus && realtimeStatus === GC.NO_DATA) {
|
||||
// Apply the preference *before* merging so incident/maintenance can still override later.
|
||||
// Also avoid carrying over realtime NO_DATA error_message.
|
||||
realtimeDataForMerge = { ...realtimeData }
|
||||
realtimeDataForMerge[ts] = {
|
||||
...realtimeDataForMerge[ts],
|
||||
status: defaultStatus,
|
||||
type: GC.DEFAULT_STATUS
|
||||
}
|
||||
delete realtimeDataForMerge[ts].error_message
|
||||
}
|
||||
```
|
||||
|
||||
with:
|
||||
|
||||
```typescript
|
||||
const defaultStatus = defaultData[ts]?.status
|
||||
const realtimeStatus = realtimeData[ts]?.status
|
||||
let realtimeDataForMerge = realtimeData
|
||||
if (defaultStatus && realtimeStatus === GC.NO_DATA) {
|
||||
// Apply the preference *before* merging so incident/maintenance can still override later.
|
||||
// Also avoid carrying over realtime NO_DATA error_message.
|
||||
// Keep the fill's own type: DEFAULT for fixed fill, CARRIED for last-known fill.
|
||||
realtimeDataForMerge = { ...realtimeData }
|
||||
realtimeDataForMerge[ts] = {
|
||||
...realtimeDataForMerge[ts],
|
||||
status: defaultStatus,
|
||||
type: defaultData[ts].type
|
||||
}
|
||||
delete realtimeDataForMerge[ts].error_message
|
||||
}
|
||||
```
|
||||
|
||||
Note: `latency` in this branch intentionally stays whatever realtime reported — unchanged from today for fixed fill; for LAST_KNOWN-on-heartbeat the carried latency was already placed in `defaultData[ts]` and `mergedData` spread order (`{ ...defaultData, ...realtimeDataForMerge, ... }`) means the realtime object wins the spread; this matches existing fixed-fill behavior, do not "improve" it here.
|
||||
|
||||
- [ ] **Step 3: Type-check**
|
||||
|
||||
Run: `npm run check`
|
||||
Expected: 0 new errors.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add src/lib/server/queues/monitorExecuteQueue.ts
|
||||
git commit -m "feat(scheduler): write CARRIED samples for LAST_KNOWN default status fixes #721"
|
||||
```
|
||||
|
||||
(Live behavior is verified end-to-end in Task 8 — the worker needs Redis + the cron loop, so there is no isolated script for this task.)
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Normalization chokepoint for all monitor writes
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/lib/server/controllers/monitorsController.ts` (near `CreateUpdateMonitor`, line 193)
|
||||
- Modify: `src/routes/(api)/api/v4/monitors/+server.ts:104-121`
|
||||
- Modify: `src/routes/(api)/api/v4/monitors/[monitor_tag]/+server.ts:72-78`
|
||||
|
||||
- [ ] **Step 1: Add the helper to monitorsController.ts**
|
||||
|
||||
Directly above `CreateUpdateMonitor` (line 193), add:
|
||||
|
||||
```typescript
|
||||
const VALID_DEFAULT_STATUSES = ["NONE", GC.UP, GC.DOWN, GC.DEGRADED, GC.LAST_KNOWN] as const
|
||||
|
||||
/**
|
||||
* Enforce the closed default_status value set and the LAST_KNOWN scope rule
|
||||
* (docs/adr/0006): LAST_KNOWN is only meaningful on NONE-type (Manual) monitors;
|
||||
* on any other type it silently resets to UP so the invalid combination never persists.
|
||||
* Throws on values outside the closed set.
|
||||
*/
|
||||
export const NormalizeDefaultStatus = (monitorType: string | null | undefined, defaultStatus: string | null | undefined): string => {
|
||||
const value = defaultStatus ?? "NONE"
|
||||
if (!(VALID_DEFAULT_STATUSES as readonly string[]).includes(value)) {
|
||||
throw new Error(`default_status must be one of: ${VALID_DEFAULT_STATUSES.join(", ")}`)
|
||||
}
|
||||
if (value === GC.LAST_KNOWN && monitorType !== "NONE") {
|
||||
return GC.UP
|
||||
}
|
||||
return value
|
||||
}
|
||||
```
|
||||
|
||||
(`monitorsController.ts` already imports `GC` at line 25 — no import change needed.)
|
||||
|
||||
- [ ] **Step 2: Apply it in the manage-UI path**
|
||||
|
||||
Replace `CreateUpdateMonitor` (lines 193-201):
|
||||
|
||||
```typescript
|
||||
export const CreateUpdateMonitor = async (monitor: MonitorInput): Promise<number | number[]> => {
|
||||
let monitorData = { ...monitor }
|
||||
if (monitorData.id) {
|
||||
return await db.updateMonitor(monitorData as MonitorRecord)
|
||||
} else {
|
||||
validateMonitorTag(monitorData.tag)
|
||||
return await db.insertMonitor(monitorData)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
with:
|
||||
|
||||
```typescript
|
||||
export const CreateUpdateMonitor = async (monitor: MonitorInput): Promise<number | number[]> => {
|
||||
let monitorData = { ...monitor }
|
||||
monitorData.default_status = NormalizeDefaultStatus(monitorData.monitor_type, monitorData.default_status)
|
||||
if (monitorData.id) {
|
||||
return await db.updateMonitor(monitorData as MonitorRecord)
|
||||
} else {
|
||||
validateMonitorTag(monitorData.tag)
|
||||
return await db.insertMonitor(monitorData)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(The manage API route at `src/routes/(manage)/manage/api/+server.ts` wraps the action switch in try/catch (line 660) and surfaces thrown `Error.message` — no route change needed.)
|
||||
|
||||
- [ ] **Step 3: Apply it in v4 POST**
|
||||
|
||||
In `src/routes/(api)/api/v4/monitors/+server.ts`, add to the imports from the monitors controller (`GetMonitorsParsed` is already imported — extend that import):
|
||||
|
||||
```typescript
|
||||
import { GetMonitorsParsed, NormalizeDefaultStatus } from "$lib/server/controllers/monitorsController"
|
||||
```
|
||||
|
||||
(match the existing import line's exact shape — if `GetMonitorsParsed` is imported from a different specifier, add `NormalizeDefaultStatus` to that same line).
|
||||
|
||||
Then replace line 111:
|
||||
|
||||
```typescript
|
||||
default_status: body.default_status ?? "UP",
|
||||
```
|
||||
|
||||
with a pre-validated variable. Above the `const monitorData = {` block (line 104), insert:
|
||||
|
||||
```typescript
|
||||
let defaultStatus: string
|
||||
try {
|
||||
defaultStatus = NormalizeDefaultStatus(body.monitor_type ?? "API", body.default_status ?? "UP")
|
||||
} catch (e) {
|
||||
const errorResponse: BadRequestResponse = {
|
||||
error: {
|
||||
code: "BAD_REQUEST",
|
||||
message: e instanceof Error ? e.message : "Invalid default_status"
|
||||
}
|
||||
}
|
||||
return json(errorResponse, { status: 400 })
|
||||
}
|
||||
```
|
||||
|
||||
and in `monitorData` use:
|
||||
|
||||
```typescript
|
||||
default_status: defaultStatus,
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Apply it in v4 PATCH**
|
||||
|
||||
In `src/routes/(api)/api/v4/monitors/[monitor_tag]/+server.ts`, the handler resolves `updateData.monitor_type` at line 78 _after_ `updateData.default_status` at line 75 — the normalization must run after BOTH are resolved. Replace line 75:
|
||||
|
||||
```typescript
|
||||
updateData.default_status = body.default_status !== undefined ? body.default_status : existingMonitor.default_status
|
||||
```
|
||||
|
||||
with (keep it in place so field ordering stays readable, but move the value through the helper after line 78):
|
||||
|
||||
```typescript
|
||||
updateData.default_status = body.default_status !== undefined ? body.default_status : existingMonitor.default_status
|
||||
```
|
||||
|
||||
…and after line 78 (`updateData.monitor_type = ...`), insert:
|
||||
|
||||
```typescript
|
||||
// Closed-set validation + LAST_KNOWN scope rule (docs/adr/0006). Runs after monitor_type
|
||||
// is resolved so a type change away from NONE auto-resets LAST_KNOWN to UP.
|
||||
try {
|
||||
updateData.default_status = NormalizeDefaultStatus(updateData.monitor_type as string, updateData.default_status as string | null)
|
||||
} catch (e) {
|
||||
const errorResponse: BadRequestResponse = {
|
||||
error: {
|
||||
code: "BAD_REQUEST",
|
||||
message: e instanceof Error ? e.message : "Invalid default_status"
|
||||
}
|
||||
}
|
||||
return json(errorResponse, { status: 400 })
|
||||
}
|
||||
```
|
||||
|
||||
Add `NormalizeDefaultStatus` to this file's monitors-controller import the same way as in Step 3.
|
||||
|
||||
- [ ] **Step 5: Verify with a throwaway script**
|
||||
|
||||
Create `scripts/tmp-verify-normalize.ts`:
|
||||
|
||||
```typescript
|
||||
import { NormalizeDefaultStatus } from "../src/lib/server/controllers/monitorsController"
|
||||
|
||||
const cases: Array<[string, string | null, string]> = [
|
||||
["NONE", "LAST_KNOWN", "LAST_KNOWN"], // allowed on Manual monitors
|
||||
["API", "LAST_KNOWN", "UP"], // auto-reset on any other type
|
||||
["NONE", null, "NONE"], // null → NONE
|
||||
["API", "DOWN", "DOWN"] // fixed values pass through
|
||||
]
|
||||
for (const [type, input, expected] of cases) {
|
||||
const got = NormalizeDefaultStatus(type, input)
|
||||
if (got !== expected) throw new Error(`FAIL: (${type}, ${input}) → ${got}, expected ${expected}`)
|
||||
}
|
||||
let threw = false
|
||||
try {
|
||||
NormalizeDefaultStatus("API", "MAINTENANCE")
|
||||
} catch {
|
||||
threw = true
|
||||
}
|
||||
if (!threw) throw new Error("FAIL: MAINTENANCE must be rejected")
|
||||
console.log("PASS")
|
||||
```
|
||||
|
||||
Run: `npx vite-node scripts/tmp-verify-normalize.ts`
|
||||
Expected: `PASS`. (Importing monitorsController transitively pulls in the db singleton; vite-node loads the repo `.env` automatically, so the configured `DATABASE_URL`/`REDIS_URL` satisfy it. If module side-effects still fail outside the dev process, inline the `NormalizeDefaultStatus` cases into a temporary copy instead — the function is pure.)
|
||||
|
||||
- [ ] **Step 6: Delete script, type-check, commit**
|
||||
|
||||
```bash
|
||||
rm scripts/tmp-verify-normalize.ts
|
||||
npm run check
|
||||
git add src/lib/server/controllers/monitorsController.ts "src/routes/(api)/api/v4/monitors/+server.ts" "src/routes/(api)/api/v4/monitors/[monitor_tag]/+server.ts"
|
||||
git commit -m "feat(api): enforce closed default_status set with LAST_KNOWN scope rule"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Migration — normalize legacy default_status values
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `migrations/20260607150000_normalize_default_status.ts`
|
||||
|
||||
- [ ] **Step 1: Write the migration**
|
||||
|
||||
```typescript
|
||||
import type { Knex } from "knex"
|
||||
|
||||
// Closed default_status set as of docs/adr/0006. MAINTENANCE was offered by the old
|
||||
// UI but never honored by the fill engine — it behaved exactly like "no fill", so it
|
||||
// (and any other unknown value, and NULL) normalizes to NONE, preserving behavior.
|
||||
const VALID = ["NONE", "UP", "DOWN", "DEGRADED", "LAST_KNOWN"]
|
||||
|
||||
export async function up(knex: Knex): Promise<void> {
|
||||
await knex("monitors").whereNull("default_status").update({ default_status: "NONE" })
|
||||
await knex("monitors").whereNotIn("default_status", VALID).update({ default_status: "NONE" })
|
||||
}
|
||||
|
||||
export async function down(): Promise<void> {
|
||||
// Irreversible by design: the values rewritten to NONE were dead (never honored
|
||||
// by the fill engine), so there is nothing meaningful to restore.
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run it against the dev database**
|
||||
|
||||
Run: `npm run migrate`
|
||||
Expected: `Batch N run: 1 migrations` with no errors.
|
||||
|
||||
- [ ] **Step 3: Spot-check via the dev API**
|
||||
|
||||
With the dev server running (`npm run dev` if not already):
|
||||
|
||||
```bash
|
||||
curl -s 'http://localhost:3000/api/v4/monitors' -H 'Authorization: Bearer <API_KEY>' | node -e "let s='';process.stdin.on('data',d=>s+=d).on('end',()=>{const r=JSON.parse(s);const bad=(r.monitors||r.data||[]).filter(m=>!['NONE','UP','DOWN','DEGRADED','LAST_KNOWN'].includes(m.default_status));console.log('invalid default_status rows:',bad.length)})"
|
||||
```
|
||||
|
||||
Expected: `invalid default_status rows: 0`.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add migrations/20260607150000_normalize_default_status.ts
|
||||
git commit -m "feat(db): migrate default_status to closed value set"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Manage UI — dropdown options + callout + auto-reset
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/routes/(manage)/manage/app/monitors/[tag]/components/GeneralSettingsCard.svelte:230-247`
|
||||
|
||||
**REQUIRED SUB-SKILL for this task: `svelte-code-writer` (per CLAUDE.md, mandatory for all .svelte edits).**
|
||||
|
||||
- [ ] **Step 1: Add imports and labels**
|
||||
|
||||
In the `<script lang="ts">` block (GC is already imported at line 20), add to the imports:
|
||||
|
||||
```typescript
|
||||
import * as Alert from "$lib/components/ui/alert/index.js"
|
||||
import TriangleAlertIcon from "@lucide/svelte/icons/triangle-alert"
|
||||
```
|
||||
|
||||
and below the props destructuring add:
|
||||
|
||||
```typescript
|
||||
const defaultStatusLabels: Record<string, string> = {
|
||||
NONE: "None (show gaps as no data)",
|
||||
UP: "UP",
|
||||
DOWN: "DOWN",
|
||||
DEGRADED: "DEGRADED",
|
||||
LAST_KNOWN: "Last known status"
|
||||
}
|
||||
|
||||
// LAST_KNOWN is only valid on Manual (NONE-type) monitors; the server enforces the
|
||||
// same rule (NormalizeDefaultStatus), this effect just keeps the UI honest live.
|
||||
$effect(() => {
|
||||
if (monitor.monitor_type !== "NONE" && monitor.default_status === GC.LAST_KNOWN) {
|
||||
monitor.default_status = GC.UP
|
||||
toast.info("Default status was reset to UP — Last known status is only available for Manual monitors.")
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
(`toast` is already imported from `svelte-sonner` at line 16.)
|
||||
|
||||
- [ ] **Step 2: Replace the Default Status select**
|
||||
|
||||
Replace lines 230-247:
|
||||
|
||||
```svelte
|
||||
<Label for="monitor-default-status">Default Status</Label>
|
||||
<Select.Root
|
||||
type="single"
|
||||
value={monitor.default_status}
|
||||
onValueChange={(v) => {
|
||||
if (v) monitor.default_status = v
|
||||
}}
|
||||
>
|
||||
<Select.Trigger id="monitor-default-status" class="w-full">
|
||||
{monitor.default_status}
|
||||
</Select.Trigger>
|
||||
<Select.Content>
|
||||
<Select.Item value="UP">UP</Select.Item>
|
||||
<Select.Item value="DOWN">DOWN</Select.Item>
|
||||
<Select.Item value="DEGRADED">DEGRADED</Select.Item>
|
||||
<Select.Item value="MAINTENANCE">MAINTENANCE</Select.Item>
|
||||
</Select.Content>
|
||||
</Select.Root>
|
||||
```
|
||||
|
||||
with:
|
||||
|
||||
```svelte
|
||||
<Label for="monitor-default-status">Default Status</Label>
|
||||
<Select.Root
|
||||
type="single"
|
||||
value={monitor.default_status ?? "NONE"}
|
||||
onValueChange={(v) => {
|
||||
if (v) monitor.default_status = v
|
||||
}}
|
||||
>
|
||||
<Select.Trigger id="monitor-default-status" class="w-full">
|
||||
{defaultStatusLabels[monitor.default_status ?? "NONE"] ?? monitor.default_status}
|
||||
</Select.Trigger>
|
||||
<Select.Content>
|
||||
<Select.Item value="NONE">None (show gaps as no data)</Select.Item>
|
||||
<Select.Item value="UP">UP</Select.Item>
|
||||
<Select.Item value="DOWN">DOWN</Select.Item>
|
||||
<Select.Item value="DEGRADED">DEGRADED</Select.Item>
|
||||
{#if monitor.monitor_type === "NONE"}
|
||||
<Select.Item value="LAST_KNOWN">Last known status</Select.Item>
|
||||
{/if}
|
||||
</Select.Content>
|
||||
</Select.Root>
|
||||
{#if monitor.default_status === GC.LAST_KNOWN}
|
||||
<Alert.Root>
|
||||
<TriangleAlertIcon />
|
||||
<Alert.Title>Last known status</Alert.Title>
|
||||
<Alert.Description>
|
||||
<p>Kener will repeat the most recent status and latency every minute until your integration sends new data.</p>
|
||||
<ul class="list-disc pl-4">
|
||||
<li>
|
||||
If your integration stops sending, the page keeps showing the last status indefinitely — Kener cannot tell "still up" from "stopped reporting". Use a Heartbeat
|
||||
monitor to catch a silent integration.
|
||||
</li>
|
||||
<li>
|
||||
Carried minutes count toward alert thresholds: a single DOWN push will trigger alerts after your failure threshold, and they stay triggered until you push a
|
||||
recovery.
|
||||
</li>
|
||||
</ul>
|
||||
</Alert.Description>
|
||||
</Alert.Root>
|
||||
{/if}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run the svelte autofixer / check**
|
||||
|
||||
Run: `npm run check`
|
||||
Expected: 0 new errors or warnings for `GeneralSettingsCard.svelte`. Also run the svelte MCP autofixer on the component if the svelte-code-writer skill instructs it.
|
||||
|
||||
- [ ] **Step 4: Visual verification**
|
||||
|
||||
With `npm run dev` running, screenshot the editor for the seeded NONE-type monitor (`earth`):
|
||||
|
||||
```bash
|
||||
npx playwright screenshot --channel chrome --color-scheme light --viewport-size "1440,900" --full-page --wait-for-timeout 3000 http://localhost:3000/manage/app/monitors/earth /tmp/lk-ui.png
|
||||
```
|
||||
|
||||
(Authed page — if it renders the login screen, follow the storage-state cookie recipe in the project memory `kener-ui-verification-recipe`, or verify manually in the browser.)
|
||||
Expected: dropdown shows the five options ("Last known status" present because earth is Manual/NONE type, "MAINTENANCE" gone); selecting "Last known status" reveals the callout.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add "src/routes/(manage)/manage/app/monitors/[tag]/components/GeneralSettingsCard.svelte"
|
||||
git commit -m "feat(manage): Last known status option with callout in Default Status dropdown"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Docs — ADR cross-link + user documentation
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `docs/adr/0005-alerts-evaluate-alert-visible-samples.md` (append one sentence)
|
||||
- Modify: `src/routes/(docs)/docs/content/v4/monitors/overview.md` (Default Status section)
|
||||
|
||||
**REQUIRED SUB-SKILL for the `src/routes/(docs)/docs/content/` edit: `documentation-writer` (per CLAUDE.md, mandatory for docs content).**
|
||||
|
||||
- [ ] **Step 1: Amend ADR 0005**
|
||||
|
||||
Append this sentence to the end of the first paragraph of `docs/adr/0005-alerts-evaluate-alert-visible-samples.md` (after "...remain invisible to alerting."):
|
||||
|
||||
```
|
||||
Amended by ADR 0006: last-known-status fill (`CARRIED`) later joined the alert-visible set under the same invariant.
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Document Last Known Status in the monitor docs**
|
||||
|
||||
`src/routes/(docs)/docs/content/v4/monitors/overview.md` does not mention `default_status` today (verified) — add a new "Default Status" section to it. Following the documentation-writer skill's conventions, document:
|
||||
|
||||
- The five values: `NONE`, `UP`, `DOWN`, `DEGRADED`, `LAST_KNOWN`.
|
||||
- `LAST_KNOWN` is only accepted for Manual (`NONE`-type) monitors; on any other type the API resets it to `UP`.
|
||||
- Behavior: every minute without new data, Kener writes a `CARRIED` sample repeating the most recent alert-visible sample (status and latency). Carry never expires and starts at the next tick after the setting is saved (no backfill).
|
||||
- The two warnings from the UI callout (stale-forever if the integration goes silent → use a Heartbeat monitor; carried minutes count toward alert thresholds and alerts only resolve on a pushed recovery).
|
||||
- A curl example mirroring #721's flow:
|
||||
|
||||
```bash
|
||||
curl -X PATCH 'https://status.example.com/api/v4/monitors/my-service/data/{current_unix_minute}' \
|
||||
-H 'Authorization: Bearer <api-key>' \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data '{"status": "DOWN", "latency": 100}'
|
||||
# With Default Status = Last known status, the monitor stays DOWN until you push UP.
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/adr/0005-alerts-evaluate-alert-visible-samples.md "src/routes/(docs)/docs/content/v4/monitors/overview.md"
|
||||
git commit -m "docs: document Last known status default and amend ADR 0005"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: End-to-end verification against the dev server
|
||||
|
||||
**Files:** none (verification only; uses the running `npm run dev` with Redis + Postgres up)
|
||||
|
||||
- [ ] **Step 1: Create a throwaway NONE monitor with LAST_KNOWN**
|
||||
|
||||
```bash
|
||||
curl -s -X POST 'http://localhost:3000/api/v4/monitors' \
|
||||
-H 'Authorization: Bearer <API_KEY>' -H 'Content-Type: application/json' \
|
||||
--data '{"tag":"lk-e2e","name":"LK E2E","monitor_type":"NONE","default_status":"LAST_KNOWN","cron":"* * * * *"}'
|
||||
```
|
||||
|
||||
Expected: 201, monitor JSON with `"default_status":"LAST_KNOWN"`.
|
||||
|
||||
- [ ] **Step 2: Confirm no fill before any push**
|
||||
|
||||
Wait ~70 seconds (one scheduler tick), then:
|
||||
|
||||
```bash
|
||||
NOW=$(date -u +%s); curl -s "http://localhost:3000/api/v4/monitors/lk-e2e/data?start_ts=$((NOW-300))&end_ts=$NOW" -H 'Authorization: Bearer <API_KEY>'
|
||||
```
|
||||
|
||||
Expected: `{"data":[]}` — nothing to carry yet (never-pushed monitors stay no-data).
|
||||
|
||||
- [ ] **Step 3: Push DOWN once, watch CARRIED rows appear**
|
||||
|
||||
```bash
|
||||
NOW=$(date -u +%s)
|
||||
curl -s -X PATCH "http://localhost:3000/api/v4/monitors/lk-e2e/data/$NOW" \
|
||||
-H 'Authorization: Bearer <API_KEY>' -H 'Content-Type: application/json' \
|
||||
--data '{"status":"DOWN","latency":2201}'
|
||||
```
|
||||
|
||||
Wait ~130 seconds (two ticks), then re-run the Step 2 range query.
|
||||
Expected: one `"type":"MANUAL"` DOWN row at the pushed minute, followed by `"type":"CARRIED"` rows with `"status":"DOWN","latency":2201` for each subsequent minute.
|
||||
|
||||
- [ ] **Step 4: Push recovery, confirm carry follows**
|
||||
|
||||
Repeat Step 3's PATCH with `{"status":"UP","latency":5}`. Wait ~70s.
|
||||
Expected: subsequent CARRIED rows are `UP` with latency 5. The status page (`http://localhost:3000`) shows the monitor UP with the red DOWN window in today's bar.
|
||||
|
||||
- [ ] **Step 5: Verify the type-change auto-reset**
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH 'http://localhost:3000/api/v4/monitors/lk-e2e' \
|
||||
-H 'Authorization: Bearer <API_KEY>' -H 'Content-Type: application/json' \
|
||||
--data '{"monitor_type":"API","type_data":{"url":"https://example.com","timeout":5000}}'
|
||||
```
|
||||
|
||||
Expected: 200 with `"default_status":"UP"` in the response (LAST_KNOWN auto-reset because the type left NONE). Then verify rejection:
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH 'http://localhost:3000/api/v4/monitors/lk-e2e' \
|
||||
-H 'Authorization: Bearer <API_KEY>' -H 'Content-Type: application/json' \
|
||||
--data '{"default_status":"MAINTENANCE"}'
|
||||
```
|
||||
|
||||
Expected: 400 with `default_status must be one of: NONE, UP, DOWN, DEGRADED, LAST_KNOWN`.
|
||||
|
||||
- [ ] **Step 6: Clean up the test monitor**
|
||||
|
||||
The v4 monitor route has no DELETE handler (only GET/PATCH), so delete via the manage API action the dashboard uses, or from the UI at `http://localhost:3000/manage/app/monitors/lk-e2e` (Danger Zone → Delete). Verify it is gone from `http://localhost:3000/`.
|
||||
|
||||
- [ ] **Step 7: Final gate**
|
||||
|
||||
Run: `npm run check && npm run prettify`
|
||||
Expected: clean check; prettify produces no diff beyond the files already touched (re-commit formatting if it does).
|
||||
|
||||
---
|
||||
|
||||
## Self-review notes
|
||||
|
||||
- **Spec coverage:** Q1 dropdown (Task 6), Q2 MAINTENANCE migration + closed set (Tasks 4, 5, 6), Q3 carry source (Task 2), Q4 CARRIED type + whitelist (Task 1), Q5 status+latency payload (Task 3), Q6/Q7 NONE-only + auto-reset (Tasks 4, 6), Q8 no expiry (no code — absence is the feature; documented in Task 7), Q9 tick-forward/no-backfill (no code — the engine only writes at `ts`; verified in Task 8 Step 2-3), Q10 alert consequences (Task 1 whitelist + existing evaluator, no further code), Q11 callout copy (Task 6).
|
||||
- **Known non-goals:** no staleness cap, no backfill, no purge on disable, no change to `CloneMonitor` (it copies `monitor_type` + `default_status` together from an already-normalized source, so the pair stays valid).
|
||||
- **Pre-existing race left untouched (deliberate):** a same-minute tick can overwrite a just-pushed MANUAL row's type via the response-queue upsert; this exists today for DEFAULT fill and is orthogonal to this change.
|
||||
@@ -0,0 +1,16 @@
|
||||
import type { Knex } from "knex";
|
||||
|
||||
// Closed default_status set as of docs/adr/0006. MAINTENANCE was offered by the old
|
||||
// UI but never honored by the fill engine — it behaved exactly like "no fill", so it
|
||||
// (and any other unknown value, and NULL) normalizes to NONE, preserving behavior.
|
||||
const VALID = ["NONE", "UP", "DOWN", "DEGRADED", "LAST_KNOWN"];
|
||||
|
||||
export async function up(knex: Knex): Promise<void> {
|
||||
await knex("monitors").whereNull("default_status").update({ default_status: "NONE" });
|
||||
await knex("monitors").whereNotIn("default_status", VALID).update({ default_status: "NONE" });
|
||||
}
|
||||
|
||||
export async function down(_knex: Knex): Promise<void> {
|
||||
// Irreversible by design: the values rewritten to NONE were dead (never honored
|
||||
// by the fill engine), so there is nothing meaningful to restore.
|
||||
}
|
||||
@@ -41,6 +41,8 @@ export default {
|
||||
MANUAL: "MANUAL",
|
||||
WEBHOOK: "WEBHOOK",
|
||||
DEFAULT_STATUS: "DEFAULT",
|
||||
CARRIED: "CARRIED",
|
||||
LAST_KNOWN: "LAST_KNOWN",
|
||||
SIGNAL: "SIGNAL",
|
||||
INVITE_VERIFY_EMAIL: "invite_verify_email",
|
||||
ERROR_NO_SETUP: "Set up not done yet. Create a user first.",
|
||||
|
||||
@@ -190,8 +190,31 @@ export const GetMonitorsParsed = async (query: MonitorFilter): Promise<Array<Mon
|
||||
return parsedMonitors;
|
||||
};
|
||||
|
||||
const VALID_DEFAULT_STATUSES = ["NONE", GC.UP, GC.DOWN, GC.DEGRADED, GC.LAST_KNOWN] as const;
|
||||
|
||||
/**
|
||||
* Enforce the closed default_status value set and the LAST_KNOWN scope rule
|
||||
* (docs/adr/0006): LAST_KNOWN is only meaningful on NONE-type (Manual) monitors;
|
||||
* on any other type it silently resets to UP so the invalid combination never persists.
|
||||
* Throws on values outside the closed set.
|
||||
*/
|
||||
export const NormalizeDefaultStatus = (
|
||||
monitorType: string | null | undefined,
|
||||
defaultStatus: string | null | undefined,
|
||||
): string => {
|
||||
const value = defaultStatus ?? "NONE";
|
||||
if (!(VALID_DEFAULT_STATUSES as readonly string[]).includes(value)) {
|
||||
throw new Error(`default_status must be one of: ${VALID_DEFAULT_STATUSES.join(", ")}`);
|
||||
}
|
||||
if (value === GC.LAST_KNOWN && monitorType !== "NONE") {
|
||||
return GC.UP;
|
||||
}
|
||||
return value;
|
||||
};
|
||||
|
||||
export const CreateUpdateMonitor = async (monitor: MonitorInput): Promise<number | number[]> => {
|
||||
let monitorData = { ...monitor };
|
||||
monitorData.default_status = NormalizeDefaultStatus(monitorData.monitor_type, monitorData.default_status);
|
||||
if (monitorData.id) {
|
||||
return await db.updateMonitor(monitorData as MonitorRecord);
|
||||
} else {
|
||||
|
||||
@@ -50,6 +50,7 @@ class DbImpl {
|
||||
getMonitoringData!: MonitoringRepository["getMonitoringData"];
|
||||
getMonitoringDataAll!: MonitoringRepository["getMonitoringDataAll"];
|
||||
getLatestMonitoringData!: MonitoringRepository["getLatestMonitoringData"];
|
||||
getLatestAlertVisibleData!: MonitoringRepository["getLatestAlertVisibleData"];
|
||||
getLatestMonitoringDataN!: MonitoringRepository["getLatestMonitoringDataN"];
|
||||
getMonitoringDataPaginated!: MonitoringRepository["getMonitoringDataPaginated"];
|
||||
getMonitoringDataCount!: MonitoringRepository["getMonitoringDataCount"];
|
||||
@@ -410,6 +411,7 @@ class DbImpl {
|
||||
this.getMonitoringData = this.monitoring.getMonitoringData.bind(this.monitoring);
|
||||
this.getMonitoringDataAll = this.monitoring.getMonitoringDataAll.bind(this.monitoring);
|
||||
this.getLatestMonitoringData = this.monitoring.getLatestMonitoringData.bind(this.monitoring);
|
||||
this.getLatestAlertVisibleData = this.monitoring.getLatestAlertVisibleData.bind(this.monitoring);
|
||||
this.getLatestMonitoringDataN = this.monitoring.getLatestMonitoringDataN.bind(this.monitoring);
|
||||
this.getMonitoringDataPaginated = this.monitoring.getMonitoringDataPaginated.bind(this.monitoring);
|
||||
this.getMonitoringDataCount = this.monitoring.getMonitoringDataCount.bind(this.monitoring);
|
||||
|
||||
@@ -11,13 +11,15 @@ import type {
|
||||
} from "../../types/db.js";
|
||||
|
||||
/**
|
||||
* Sample types alert evaluation can see (see docs/adr/0005-alerts-evaluate-alert-visible-samples.md).
|
||||
* Sample types alert evaluation can see (see docs/adr/0005-alerts-evaluate-alert-visible-samples.md
|
||||
* and docs/adr/0006-last-known-status-fill.md).
|
||||
* Exactly the types written by flows that enqueue alert evaluation: scheduler checks
|
||||
* (REALTIME/ERROR/TIMEOUT), default-status fill (DEFAULT_STATUS), and data-API pushes (MANUAL).
|
||||
* (REALTIME/ERROR/TIMEOUT), default-status fill (DEFAULT_STATUS), last-known-status fill (CARRIED),
|
||||
* and data-API pushes (MANUAL).
|
||||
* SIGNAL rows (raw heartbeat receipts) and INCIDENT/MAINTENANCE overlays stay invisible, so the
|
||||
* alert window freezes during manual overlays instead of triggering or resolving on them.
|
||||
*/
|
||||
const ALERT_VISIBLE_TYPES = [GC.REALTIME, GC.ERROR, GC.TIMEOUT, GC.MANUAL, GC.DEFAULT_STATUS];
|
||||
const ALERT_VISIBLE_TYPES = [GC.REALTIME, GC.ERROR, GC.TIMEOUT, GC.MANUAL, GC.DEFAULT_STATUS, GC.CARRIED];
|
||||
|
||||
/**
|
||||
* Repository for monitoring data operations
|
||||
@@ -70,6 +72,10 @@ export class MonitoringRepository extends BaseRepository {
|
||||
.orderBy("timestamp", "asc");
|
||||
}
|
||||
|
||||
/**
|
||||
* Newest sample of ANY type (including SIGNAL receipts and overlays).
|
||||
* For the last-known-status carry source, use getLatestAlertVisibleData.
|
||||
*/
|
||||
async getLatestMonitoringData(monitor_tag: string): Promise<MonitoringData | undefined> {
|
||||
return await this.knex("monitoring_data")
|
||||
.where("monitor_tag", monitor_tag)
|
||||
@@ -78,6 +84,23 @@ export class MonitoringRepository extends BaseRepository {
|
||||
.first();
|
||||
}
|
||||
|
||||
/**
|
||||
* Latest sample the alert evaluator (and last-known-status fill) can see.
|
||||
* Carry source for Default Status = LAST_KNOWN (docs/adr/0006): overlays
|
||||
* (INCIDENT/MAINTENANCE) and raw heartbeat receipts (SIGNAL) are excluded,
|
||||
* so they can never become sticky. CARRIED rows are included — each tick's
|
||||
* carry output becomes the next tick's carry source, which is the intended
|
||||
* sticky behavior.
|
||||
*/
|
||||
async getLatestAlertVisibleData(monitor_tag: string): Promise<MonitoringData | undefined> {
|
||||
return await this.knex("monitoring_data")
|
||||
.where("monitor_tag", monitor_tag)
|
||||
.whereIn("type", ALERT_VISIBLE_TYPES)
|
||||
.orderBy("timestamp", "desc")
|
||||
.limit(1)
|
||||
.first();
|
||||
}
|
||||
|
||||
async getLatestMonitoringDataN(monitor_tag: string, limit: number): Promise<MonitoringData[]> {
|
||||
return await this.knex("monitoring_data")
|
||||
.where("monitor_tag", monitor_tag)
|
||||
|
||||
@@ -135,6 +135,20 @@ const addWorker = () => {
|
||||
if (monitor.default_status !== GC.UP) {
|
||||
defaultData[ts].error_message = "Default status applied";
|
||||
}
|
||||
} else if (monitor.default_status === GC.LAST_KNOWN) {
|
||||
// Last Known Status fill (docs/adr/0006): repeat the most recent alert-visible
|
||||
// sample — status and latency alike. No sample yet → nothing to carry → no fill.
|
||||
const lastKnown = await db.getLatestAlertVisibleData(monitor.tag);
|
||||
if (lastKnown && lastKnown.status) {
|
||||
defaultData[ts] = {
|
||||
status: lastKnown.status,
|
||||
latency: lastKnown.latency ?? 0,
|
||||
type: GC.CARRIED,
|
||||
};
|
||||
if (lastKnown.status !== GC.UP) {
|
||||
defaultData[ts].error_message = "Last known status applied";
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -146,11 +160,12 @@ const addWorker = () => {
|
||||
if (defaultStatus && realtimeStatus === GC.NO_DATA) {
|
||||
// Apply the preference *before* merging so incident/maintenance can still override later.
|
||||
// Also avoid carrying over realtime NO_DATA error_message.
|
||||
// Keep the fill's own type: DEFAULT for fixed fill, CARRIED for last-known fill.
|
||||
realtimeDataForMerge = { ...realtimeData };
|
||||
realtimeDataForMerge[ts] = {
|
||||
...realtimeDataForMerge[ts],
|
||||
status: defaultStatus,
|
||||
type: GC.DEFAULT_STATUS,
|
||||
type: defaultData[ts].type,
|
||||
};
|
||||
delete realtimeDataForMerge[ts].error_message;
|
||||
}
|
||||
|
||||
@@ -8,7 +8,7 @@ import type {
|
||||
BadRequestResponse,
|
||||
} from "$lib/types/api";
|
||||
import type { MonitorRecord } from "$lib/server/types/db";
|
||||
import { GetMonitorsParsed } from "$lib/server/controllers/monitorsController";
|
||||
import { GetMonitorsParsed, NormalizeDefaultStatus } from "$lib/server/controllers/monitorsController";
|
||||
|
||||
function formatDateToISO(date: Date | string): string {
|
||||
if (date instanceof Date) {
|
||||
@@ -101,6 +101,19 @@ export const POST: RequestHandler = async ({ request }) => {
|
||||
return json(errorResponse, { status: 400 });
|
||||
}
|
||||
|
||||
let defaultStatus: string;
|
||||
try {
|
||||
defaultStatus = NormalizeDefaultStatus(body.monitor_type ?? "API", body.default_status ?? "UP");
|
||||
} catch (e) {
|
||||
const errorResponse: BadRequestResponse = {
|
||||
error: {
|
||||
code: "BAD_REQUEST",
|
||||
message: e instanceof Error ? e.message : "Invalid default_status",
|
||||
},
|
||||
};
|
||||
return json(errorResponse, { status: 400 });
|
||||
}
|
||||
|
||||
// Prepare monitor data for insertion
|
||||
const monitorData = {
|
||||
tag: body.tag.trim(),
|
||||
@@ -108,7 +121,7 @@ export const POST: RequestHandler = async ({ request }) => {
|
||||
description: body.description ?? null,
|
||||
image: body.image ?? null,
|
||||
cron: body.cron ?? null,
|
||||
default_status: body.default_status ?? "UP",
|
||||
default_status: defaultStatus,
|
||||
status: body.status ?? "ACTIVE",
|
||||
category_name: body.category_name ?? null,
|
||||
monitor_type: body.monitor_type ?? "API",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { json, type RequestHandler } from "@sveltejs/kit";
|
||||
import db from "$lib/server/db/db";
|
||||
import { GetMonitorsParsed } from "$lib/server/controllers/monitorsController";
|
||||
import { GetMonitorsParsed, NormalizeDefaultStatus } from "$lib/server/controllers/monitorsController";
|
||||
import type {
|
||||
GetMonitorResponse,
|
||||
MonitorResponse,
|
||||
@@ -77,6 +77,23 @@ export const PATCH: RequestHandler = async ({ locals, request }) => {
|
||||
updateData.category_name = body.category_name !== undefined ? body.category_name : existingMonitor.category_name;
|
||||
updateData.monitor_type = body.monitor_type !== undefined ? body.monitor_type : existingMonitor.monitor_type;
|
||||
|
||||
// Closed-set validation + LAST_KNOWN scope rule (docs/adr/0006). Runs after monitor_type
|
||||
// is resolved so a type change away from NONE auto-resets LAST_KNOWN to UP.
|
||||
try {
|
||||
updateData.default_status = NormalizeDefaultStatus(
|
||||
updateData.monitor_type as string,
|
||||
updateData.default_status as string | null,
|
||||
);
|
||||
} catch (e) {
|
||||
const errorResponse: BadRequestResponse = {
|
||||
error: {
|
||||
code: "BAD_REQUEST",
|
||||
message: e instanceof Error ? e.message : "Invalid default_status",
|
||||
},
|
||||
};
|
||||
return json(errorResponse, { status: 400 });
|
||||
}
|
||||
|
||||
updateData.is_hidden = body.is_hidden !== undefined ? body.is_hidden : existingMonitor.is_hidden;
|
||||
|
||||
// Handle JSON fields - merge with existing data instead of replacing
|
||||
|
||||
@@ -29,6 +29,43 @@ So realtime checks do not override an active maintenance or incident state.
|
||||
|
||||
Monitors run from cron expressions (for example `* * * * *` for every minute). Use tighter schedules for critical services and relaxed schedules for low-risk dependencies.
|
||||
|
||||
## Default Status {#default-status}
|
||||
|
||||
Default Status is the monitor's answer to the question: **what does a minute with no monitoring sample mean?**
|
||||
|
||||
| Value | Behavior |
|
||||
| ------------ | ------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `NONE` | Gap minutes show as no data (gray) |
|
||||
| `UP` | A `DEFAULT` sample is written each minute marking the service UP |
|
||||
| `DOWN` | A `DEFAULT` sample is written each minute marking the service DOWN |
|
||||
| `DEGRADED` | A `DEFAULT` sample is written each minute marking the service DEGRADED |
|
||||
| `LAST_KNOWN` | Each minute without a new sample, Kener writes a `CARRIED` row repeating the most recent alert-visible status and latency |
|
||||
|
||||
### Last known status {#last-known-status}
|
||||
|
||||
`LAST_KNOWN` is only available on **Manual (`NONE`-type) monitors**. If you select it on any other monitor type, the API resets it to `UP`. Changing a monitor's type away from Manual also resets it to `UP`.
|
||||
|
||||
How it works:
|
||||
|
||||
- Every scheduler tick with no new data, Kener writes a `CARRIED` sample copying the status and latency of the most recent alert-visible sample.
|
||||
- Carry is tick-forward only — it starts at the next scheduler tick after you save the setting, with no backfill of past gaps.
|
||||
- Carried rows persist in history even if you later change the setting.
|
||||
|
||||
Example push flow:
|
||||
|
||||
```bash
|
||||
curl -X PATCH 'https://status.example.com/api/v4/monitors/my-service/data/{current_unix_minute}' \
|
||||
-H 'Authorization: Bearer <api-key>' \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data '{"status": "DOWN", "latency": 100}'
|
||||
# With Default Status = Last known status, the monitor stays DOWN until you push UP.
|
||||
```
|
||||
|
||||
> [!WARNING]
|
||||
>
|
||||
> - If your integration stops sending, the page keeps showing the last status indefinitely — Kener cannot tell "still up" from "stopped reporting". Use a [Heartbeat monitor](/docs/v4/monitors/heartbeat) to catch a silent integration.
|
||||
> - Carried minutes count toward alert thresholds: a single DOWN push will trigger alerts after your failure threshold, and they stay triggered until you push a recovery.
|
||||
|
||||
## Uptime calculation {#uptime-calculation}
|
||||
|
||||
Default uptime formula:
|
||||
|
||||
+47
-3
@@ -11,6 +11,8 @@
|
||||
import UploadIcon from "@lucide/svelte/icons/upload";
|
||||
import XIcon from "@lucide/svelte/icons/x";
|
||||
import ImageIcon from "@lucide/svelte/icons/image";
|
||||
import * as Alert from "$lib/components/ui/alert/index.js";
|
||||
import TriangleAlertIcon from "@lucide/svelte/icons/triangle-alert";
|
||||
import type { MonitorRecord } from "$lib/server/types/db.js";
|
||||
import { toast } from "svelte-sonner";
|
||||
import { goto } from "$app/navigation";
|
||||
@@ -26,6 +28,23 @@
|
||||
|
||||
let { monitor = $bindable(), typeData, isNew }: Props = $props();
|
||||
|
||||
const defaultStatusLabels: Record<string, string> = {
|
||||
NONE: "None (show gaps as no data)",
|
||||
UP: "UP",
|
||||
DOWN: "DOWN",
|
||||
DEGRADED: "DEGRADED",
|
||||
LAST_KNOWN: "Last known status"
|
||||
};
|
||||
|
||||
// LAST_KNOWN is only valid on Manual (NONE-type) monitors; the server enforces the
|
||||
// same rule (NormalizeDefaultStatus), this effect just keeps the UI honest live.
|
||||
$effect(() => {
|
||||
if (monitor.monitor_type !== "NONE" && monitor.default_status === GC.LAST_KNOWN) {
|
||||
monitor.default_status = GC.UP;
|
||||
toast.info("Default status was reset to UP — Last known status is only available for Manual monitors.");
|
||||
}
|
||||
});
|
||||
|
||||
let savingGeneral = $state(false);
|
||||
let uploadingImage = $state(false);
|
||||
|
||||
@@ -230,21 +249,46 @@
|
||||
<Label for="monitor-default-status">Default Status</Label>
|
||||
<Select.Root
|
||||
type="single"
|
||||
value={monitor.default_status}
|
||||
value={monitor.default_status ?? "NONE"}
|
||||
onValueChange={(v) => {
|
||||
if (v) monitor.default_status = v;
|
||||
}}
|
||||
>
|
||||
<Select.Trigger id="monitor-default-status" class="w-full">
|
||||
{monitor.default_status}
|
||||
{defaultStatusLabels[monitor.default_status ?? "NONE"] ?? monitor.default_status}
|
||||
</Select.Trigger>
|
||||
<Select.Content>
|
||||
<Select.Item value="NONE">None (show gaps as no data)</Select.Item>
|
||||
<Select.Item value="UP">UP</Select.Item>
|
||||
<Select.Item value="DOWN">DOWN</Select.Item>
|
||||
<Select.Item value="DEGRADED">DEGRADED</Select.Item>
|
||||
<Select.Item value="MAINTENANCE">MAINTENANCE</Select.Item>
|
||||
{#if monitor.monitor_type === "NONE"}
|
||||
<Select.Item value="LAST_KNOWN">Last known status</Select.Item>
|
||||
{/if}
|
||||
</Select.Content>
|
||||
</Select.Root>
|
||||
{#if monitor.default_status === GC.LAST_KNOWN}
|
||||
<Alert.Root>
|
||||
<TriangleAlertIcon />
|
||||
<Alert.Title>Last known status</Alert.Title>
|
||||
<Alert.Description>
|
||||
<p>
|
||||
Kener will repeat the most recent status and latency every minute until your integration sends new data.
|
||||
</p>
|
||||
<ul class="list-disc pl-4">
|
||||
<li>
|
||||
If your integration stops sending, the page keeps showing the last status indefinitely — Kener
|
||||
cannot tell "still up" from "stopped reporting". Use a Heartbeat monitor to catch
|
||||
a silent integration.
|
||||
</li>
|
||||
<li>
|
||||
Carried minutes count toward alert thresholds: a single DOWN push will trigger alerts after your
|
||||
failure threshold, and they stay triggered until you push a recovery.
|
||||
</li>
|
||||
</ul>
|
||||
</Alert.Description>
|
||||
</Alert.Root>
|
||||
{/if}
|
||||
</div>
|
||||
<div class="flex flex-col gap-2">
|
||||
<Label for="hidden-switch">Hidden in Status Page</Label>
|
||||
|
||||
Reference in New Issue
Block a user