opensource/supermq
1
0
Fork 0
mirror of https://github.com/absmach/supermq.git synced 2026-06-23 07:10:19 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Improve queue docs

Browse Source 
Signed-off-by: dusan <borovcanindusan1@gmail.com>
This commit is contained in:
dusan
2026-02-10 10:36:06 +01:00
parent 0b00a7cff8
commit 5f2919e87c
5 changed files with 157 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files
web/content/docs/architecture/overview.md
+9 -6
View File
@@ -5,11 +5,11 @@ description: High-level system design overview covering core components and how
# Architecture Overview
**Last Updated:** 2026-02-05
**Last Updated:** 2026-02-10
## Overview
FluxMQ is a multi-protocol message broker built around a shared queue manager. MQTT transports (TCP, WebSocket, HTTP bridge, CoAP) share one MQTT broker instance, while AMQP 1.0 and AMQP 0.9.1 use dedicated brokers. Durable queues are protocol-agnostic and provide cross-protocol routing and fan-out.
FluxMQ is a multi-protocol message broker built around a shared queue manager. MQTT transports (TCP, WebSocket, HTTP bridge, CoAP) share one MQTT broker instance, while AMQP 1.0 and AMQP 0.9.1 use dedicated brokers. Queues are protocol-agnostic — all protocols route through the same queue manager, and delivery semantics depend on the [queue type](/docs/concepts/queues) (ephemeral, durable, or stream), not the protocol.
## High-Level View
@@ -36,7 +36,9 @@ FluxMQ is a multi-protocol message broker built around a shared queue manager. M
▼
┌────────────────┐
│ Queue Manager │
│ (durable logs) │
│ (ephemeral, │
│ durable, │
│ stream) │
└──────┬─────────┘
▼
┌────────────────┐
@@ -63,9 +65,10 @@ FluxMQ is a multi-protocol message broker built around a shared queue manager. M
- `server/amqp`, `server/amqp1` for AMQP listeners
- **Queue Manager**: `queue/` and `logstorage/`
- Append-only logs with consumer groups
- Queue and stream modes
- Ack/Nack/Reject support and retention policies
- Three queue types: ephemeral (in-memory, best-effort), durable (persistent work queue with PEL), stream (append-only log with cursor-based consumption)
- Shared routing layer — topic bindings, fan-out, consumer filters
- Delivery semantics depend on queue type, not protocol
- See [Queue Types](/docs/concepts/queues) for the full model
- **Storage**: `storage/` (BadgerDB and memory backends)
- Sessions, subscriptions, retained messages, offline queues
web/content/docs/concepts/broker.md
+7 -3
View File
@@ -5,15 +5,17 @@ description: What the broker does in FluxMQ and how protocol brokers fit togethe
# Broker
**Last Updated:** 2026-02-05
**Last Updated:** 2026-02-10
FluxMQ runs multiple protocol brokers that share the same durable queue manager:
FluxMQ runs multiple protocol brokers that share the same queue manager:
- **MQTT broker** for MQTT 3.1.1/5.0 over TCP and WebSocket
- **AMQP 1.0 broker**
- **AMQP 0.9.1 broker**
Each broker owns its protocol state machine, but all queue-capable traffic flows into the shared queue manager. This lets you mix protocols while keeping one durability and delivery pipeline.
Each broker owns its protocol state machine, but all queue-capable traffic flows into the shared queue manager. Protocol adapters translate protocol-specific concepts (AMQP exchanges/bindings, MQTT shared subscriptions, AMQP 1.0 link capabilities) into FluxMQ queue primitives without fabricating behavior that the underlying queue type doesn't support.
Delivery semantics depend on the [queue type](/docs/concepts/queues), not the protocol. An MQTT client and an AMQP 0.9.1 client consuming from the same durable queue get the same ack/nack/reject behavior. An AMQP 1.0 client consuming from a stream queue gets cursor-based semantics regardless of the protocol's native disposition model.
## What the Broker Handles
@@ -22,8 +24,10 @@ Each broker owns its protocol state machine, but all queue-capable traffic flows
- Retained messages and wills
- QoS enforcement and retry policies
- Queue integration for `$queue/` topics
- Protocol-to-queue translation (adapting protocol semantics to queue primitives)
## Learn More
- [Queue Types](/docs/concepts/queues)
- [Architecture overview](/docs/architecture/overview)
- [Routing internals](/docs/architecture/routing)
web/content/docs/concepts/consumer-groups.md
+20 -5
View File
@@ -5,15 +5,16 @@ description: Load-balanced queue consumption and acknowledgment behavior
# Consumer Groups
**Last Updated:** 2026-02-05
**Last Updated:** 2026-02-10
Consumer groups are how FluxMQ turns a durable queue log into a scalable worker pool.
Consumer groups are how FluxMQ turns a queue log into a scalable worker pool. They apply to both durable and stream queues (ephemeral queues do not use consumer groups — delivery is best-effort with no progress tracking).
At a glance:
- One queue can have many groups (each group has independent progress).
- One group can have many consumers (messages are load-balanced within the group).
- Groups track progress using offsets (cursor/committed) and, in classic mode, a pending list (PEL).
- The group **mode** (classic or stream) determines acknowledgment semantics — see below.
## Group Modes: Classic vs Stream
@@ -59,9 +60,23 @@ So seeing `group=demo-events@#` in logs means:
## Acknowledgment Semantics
- **Ack**: confirms processing and allows committed offset to move forward.
- **Nack**: requests redelivery (often after a visibility timeout or via work stealing).
- **Reject**: removes a message from pending tracking (DLQ handling is not fully wired yet).
Ack/nack/reject mean different things depending on the queue type. Do not assume they are interchangeable.
### Durable Queue (Classic Mode)
- **Ack**: removes the message from the PEL and advances the committed offset. The message becomes eligible for log truncation.
- **Nack**: resets the PEL entry so the message is redelivered (immediately stealable by other consumers).
- **Reject**: removes from PEL without redelivery. The message is discarded (future: routes to DLQ).
### Stream Queue (Stream Mode)
- **Ack**: advances the committed offset (checkpoint). The message remains in the log — retention is policy-based, not ack-based.
- **Nack**: no-op. Stream consumers replay by seeking the cursor to a previous offset, not by nacking individual messages.
- **Reject**: advances the cursor past the message to prevent an infinite redelivery loop. The message stays in the log.
### Ephemeral Queue
Ephemeral queues do not use consumer groups or acknowledgments. Delivery is best-effort — messages are pushed to connected consumers and not tracked.
## Learn More
web/content/docs/concepts/queues.md
+86 -21
View File
@@ -1,37 +1,102 @@
---
title: Queues
description: Durable queues, delivery guarantees, and how FluxMQ stores queue messages
title: Queue Types
description: Three queue types with distinct semantics — ephemeral, durable, and stream
---
# Queues
# Queue Types
**Last Updated:** 2026-02-05
**Last Updated:** 2026-02-10
Queues in FluxMQ are backed by an append-only log. A single publish can be routed into multiple queues based on bindings, and each queue can serve one or more consumer groups (independent progress per group).
FluxMQ supports three distinct queue types. They share the same routing layer and topic namespace (`$queue/`), but differ in persistence, delivery guarantees, and acknowledgment semantics.
## Key Ideas
**Do not try to unify them.** Each type exists because it solves a different problem. If a use case needs both task semantics and replay, use two queues — one durable, one stream.
- Queue topics use the `$queue/` prefix: `$queue/<name>/...`.
- Each queue has a durable log (single-partition, ordered offsets).
- Consumer groups control how that log is consumed (classic vs stream semantics).
- Retention policies control how long data stays available.
## Overview
## Durable vs Ephemeral Queues
| | Ephemeral | Durable | Stream |
| --------------------------- | --------------------------- | --------------------------------- | ------------------------------ |
| **Storage** | In-memory only | Persistent log | Persistent append-only log |
| **Delivery model** | Best-effort push | Push with ack/nack/reject | Cursor-based pull |
| **Message lifecycle** | Fire-and-forget | Removed on ack | Immutable, retained by policy |
| **Ack means** | N/A | "Message processed, remove it" | "Checkpoint my read position" |
| **Nack means** | N/A | "Retry this message" | No-op (replay via cursor seek) |
| **Persistence** | None | Yes | Yes |
| **Replay** | No | No | Yes (seek to any offset) |
| **Retention** | None (lost on disconnect) | Truncate after ack | Time/size/count-based |
| **Optimized for** | Low latency, transient data | Task processing, failure handling | Event streams, replay, audit |
| **Message loss acceptable** | Yes | No | No |
Queues can be configured explicitly via `queues:` in the broker config. FluxMQ can also auto-create **ephemeral queues** as a fallback when a publish does not match any existing bindings.
## Ephemeral Queue
Ephemeral queues:
Ephemeral queues are auto-created when a publish targets a `$queue/` topic that doesn't match any configured queue. They are in-memory, best-effort, and cleaned up after the last consumer disconnects (plus a short grace period).
- Are created with a default config and a short expiration window after the last consumer disconnects.
- Help prevent “publish to nowhere” from silently dropping data.
- Are best treated as a development convenience; production deployments should define queues explicitly.
- No persistence — messages are lost on broker restart
- No replay — once delivered, messages are gone
- Useful as a development convenience or for transient data where loss is acceptable
- Production deployments should define queues explicitly in config
## Classic vs Stream
Ephemeral queues are a safety net, not a primary design target. They prevent "publish to nowhere" from silently dropping data, but they don't provide durability or delivery guarantees.
- Classic queues behave like a work queue: deliver once, track pending deliveries, ack/nack/reject.
- Stream queues behave like a log: sequential consumption with cursor-based progress and optional manual commit.
## Durable Queue
Durable queues are persistent work queues optimized for task processing:
- Messages are appended to a durable log and delivered to consumer groups
- Consumers **ack** to confirm processing — the message is then eligible for truncation
- Consumers **nack** to request redelivery (with backoff and retry limits)
- Consumers **reject** to discard a message (future: route to dead-letter queue)
- Unacknowledged messages are tracked in a **Pending Entry List (PEL)** with visibility timeouts
- **Work stealing** rebalances pending work from slow consumers to idle ones
- Retention is driven by acknowledgment state — once all groups have acked past an offset, the log can be truncated
Use durable queues when every message must be processed at least once, failures must be retried, and message loss is not acceptable.
See [Durable Queues](/docs/messaging/durable-queues) for configuration, addressing, and protocol-specific usage.
## Stream Queue
Stream queues are persistent append-only logs optimized for event streaming:
- Messages are **immutable** — they are never modified or deleted by consumers
- Consumers track progress with a **cursor** (read position in the log)
- **Ack** means "checkpoint" — advance the committed offset, not delete the message
- **Nack** is a no-op — replay is done by seeking the cursor, not by redelivering individual messages
- Multiple independent consumer groups can read the same stream at different positions
- Consumers can seek to any offset or timestamp to replay history
- Retention is policy-based (time, size, message count) — independent of consumer progress
Use stream queues for event sourcing, audit logs, change data capture, analytics pipelines, and any scenario where you need replay, backfill, or multiple independent readers over the same data.
See [Durable Queues — Stream](/docs/messaging/durable-queues#stream) for configuration, cursor positioning, and commit control.
## Shared Routing Layer
All three queue types use the same routing infrastructure:
- **Topic namespace**: `$queue/<name>/...` with MQTT wildcard matching (`+`, `#`)
- **Fan-out**: A single publish can match multiple queues based on their topic bindings
- **Consumer filters**: Subscribers can apply sub-patterns within a queue for fine-grained routing
- **Protocol-agnostic**: MQTT, AMQP 1.0, and AMQP 0.9.1 all route through the same queue manager
Routing determines **which queue** a message is stored in. Delivery semantics depend on **queue type**, not protocol. An MQTT client and an AMQP client consuming from the same durable queue get the same ack/nack/reject behavior.
See [Topics](/docs/concepts/topics) for topic structure and wildcard rules.
## Design Rules
These rules guide FluxMQ's queue design:
1. **Do not assume streams replace queues.** Streams and durable queues solve different problems. A task processor needs ack-based lifecycle control. An event consumer needs cursor-based replay. Using streams for task processing (or durable queues for event replay) will fight the semantics.
2. **Do not overload ack semantics across queue types.** "Ack" means "processed, remove it" in a durable queue and "checkpoint my position" in a stream. These are fundamentally different operations and the broker treats them accordingly.
3. **Prefer correctness and explicit trade-offs over abstraction purity.** Each queue type has its own code path for delivery, acknowledgment, and retention. This is intentional — collapsing them into a single abstraction would hide important behavioral differences.
4. **If a use case needs both task semantics and replay, use two queues.** Publish the same messages to a durable queue (for processing) and a stream queue (for audit/replay). The shared routing layer makes this a one-line config change via overlapping topic bindings.
## Learn More
- [Durable queues](/docs/messaging/durable-queues)
- [Storage internals](/docs/architecture/storage)
- [Durable Queues](/docs/messaging/durable-queues) — configuration, addressing, acknowledgments, and protocol-specific usage
- [Consumer Groups](/docs/concepts/consumer-groups) — group modes, progress tracking, and ack semantics per queue type
- [Topics](/docs/concepts/topics) — topic structure, wildcards, and special namespaces
- [Storage](/docs/concepts/storage) — storage layers for broker state, queue logs, and cluster metadata
web/content/docs/messaging/durable-queues.mdx
+35 -28
View File
@@ -1,36 +1,43 @@
---
title: Durable Queues
description: Shared queue system for MQTT and AMQP with consumer groups, acknowledgments, stream queues, and append-only log storage
description: Persistent queue system for MQTT and AMQP — durable work queues and stream queues with consumer groups, acknowledgments, and append-only log storage
---
# Durable Queues
**Last Updated:** 2026-02-05
**Last Updated:** 2026-02-10
FluxMQ provides durable queues shared across MQTT, AMQP 1.0, and AMQP 0.9.1. The queue manager is append-only with consumer groups and supports both classic work-queue semantics and stream-style consumption.
This page covers FluxMQ's two persistent queue types: **durable queues** (work queue semantics) and **stream queues** (append-only log semantics). Both are shared across MQTT, AMQP 1.0, and AMQP 0.9.1.
For an overview of all three queue types (including ephemeral queues), see [Queue Types](/docs/concepts/queues).
## Overview
Durable queues are built around a simple model:
Persistent queues are built around a simple model:
- Producers publish to `$queue/<queue-name>/...`.
- The broker appends the message to the queue’s log (durable storage).
- The broker appends the message to the queue's log (durable storage).
- Consumers join a **consumer group** and receive messages from that log.
- Consumers **ack**, **nack**, or **reject** deliveries to control redelivery and truncation.
- What happens next depends on the queue type:
Queues can operate in two modes:
| | Durable (Classic) | Stream |
| -------------- | -------------------------------------------------- | ------------------------------------------ |
| **Ack** | Remove from pending list, advance truncation point | Checkpoint cursor position |
| **Nack** | Redeliver the message (with backoff) | No-op (replay via cursor seek) |
| **Reject** | Discard message (future: DLQ) | Advance cursor past message |
| **Retention** | Truncate after all groups ack | Time/size/count policy |
| **Redelivery** | Automatic (visibility timeout + work stealing) | Manual (consumer restarts from checkpoint) |
- **Classic (work queue)**: messages are claimed, tracked in a pending list (PEL), and acknowledged.
- **Stream**: consumers read sequentially with cursor-based progress (replayable log semantics).
> **Design rule:** Do not overload ack semantics. "Ack" means "processed, remove it" in a durable queue and "checkpoint my position" in a stream. If you need both task processing and replay for the same data, use two queues with overlapping topic bindings.
## Do MQTT Producers Need `$queue/`?
Yes. In FluxMQ today, MQTT publishes are treated as durable-queue traffic only when the topic starts with `$queue/`.
Yes. In FluxMQ, MQTT publishes are routed to the queue manager only when the topic starts with `$queue/`.
- Publish to `sensors/temp` goes through normal pub/sub routing (subscriptions, retained, cluster pub/sub).
- Publish to `$queue/orders/...` goes through the queue manager (durable log + consumer groups + acks).
- Publish to `$queue/orders/...` goes through the queue manager (log storage + consumer groups + acks).
If you configured a queue with topic bindings that match a non-`$queue/` topic, MQTT publishes to that topic will still *not* be enqueued, because the MQTT broker does not route non-`$queue/` publishes into the queue manager.
The `$queue/` prefix is what activates queue semantics — whether the target is an ephemeral, durable, or stream queue. If you configured a queue with topic bindings that match a non-`$queue/` topic, MQTT publishes to that topic will still *not* be enqueued, because the MQTT broker does not route non-`$queue/` publishes into the queue manager.
## Architecture
@@ -173,10 +180,10 @@ FluxMQ uses **topic patterns on queues** to decide where a publish should be sto
- Each queue has one or more topic patterns (MQTT wildcard syntax).
- On publish, the queue manager calls `FindMatchingQueues(topic)` and appends the message to every matching queue.
- If **no queue matches**, FluxMQ creates an **ephemeral queue** whose name and pattern equal the topic, then appends.
- If **no queue matches**, FluxMQ creates an **[ephemeral queue](/docs/concepts/queues#ephemeral-queue)** whose name and pattern equal the topic, then appends. Ephemeral queues are in-memory, best-effort, and cleaned up after the last consumer disconnects.
- If no queues are configured at all, FluxMQ creates a reserved `mqtt` queue that matches `$queue/#` so queue publishes always have a landing zone.
Ephemeral queues are a safety net, but they can be surprising if you publish before configuring queues or before consumers subscribe. In production, prefer explicit `queues:` configuration so queue names and bindings are stable.
In production, prefer explicit `queues:` configuration so queue names, types, and bindings are stable. Ephemeral queues are a development convenience — they prevent silent message loss but don't provide durability or delivery guarantees. See [Queue Types](/docs/concepts/queues) for when to use each type.
<Mermaid
chart="
@@ -432,14 +439,14 @@ No PEL - just two offsets to track!
Stream mode is designed for different use cases than classic work queues:
| Aspect | Classic (PEL) | Stream (No PEL) |
|--------|---------------|-----------------|
| **Semantics** | Process once | Read/replay many times |
| **Delivery guarantee** | At-least-once with ack | At-least-once with cursor |
| **Redelivery trigger** | Visibility timeout | Consumer restart/seek |
| **Work stealing** | Yes (steal from slow peers) | No (each consumer independent) |
| **Memory overhead** | PEL entry per pending message | Only cursor position |
| **Replayability** | No (acked = done) | Yes (seek to any offset) |
| Aspect | Classic (PEL) | Stream (No PEL) |
| ---------------------- | ----------------------------- | ------------------------------ |
| **Semantics** | Process once | Read/replay many times |
| **Delivery guarantee** | At-least-once with ack | At-least-once with cursor |
| **Redelivery trigger** | Visibility timeout | Consumer restart/seek |
| **Work stealing** | Yes (steal from slow peers) | No (each consumer independent) |
| **Memory overhead** | PEL entry per pending message | Only cursor position |
| **Replayability** | No (acked = done) | Yes (seek to any offset) |
**Classic mode** answers: "Has this message been successfully processed?"
@@ -543,12 +550,12 @@ Practical intuition:
Imagine a queue log with offsets `0..5`.
| Event | Cursor | PEL | Committed |
| --- | --- | --- | --- |
| Group starts | 0 | empty | 0 |
| Claims offsets 0,1 | 2 | {0,1} | 0 |
| Acks offset 0 | 2 | {1} | 1 |
| Acks offset 1 | 2 | empty | 2 |
| Event | Cursor | PEL | Committed |
| ------------------ | ------ | ----- | --------- |
| Group starts | 0 | empty | 0 |
| Claims offsets 0,1 | 2 | {0,1} | 0 |
| Acks offset 0 | 2 | {1} | 1 |
| Acks offset 1 | 2 | empty | 2 |
This is why committed is the “safe truncation floor”: once everything below an offset is fully processed, the log can be truncated without breaking group semantics.