mirror of
https://github.com/absmach/supermq.git
synced 2026-06-23 06:20:18 +00:00
MF-1584 - Upgrade InfluxDB from 1.x to 2.x (#1709)
* Upgrade InfluxDB from 1.x to 2.x Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influx DB configuration updated Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connection to InfluxDBv2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Token cannot be created Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connected to InfluxDB2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connected to InfluxDB2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB v2 Consumer Implementation Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * quickfix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb-Writer Unit Tests Update Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Consumer Update Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB Writer Tests Implemented Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB Connection Check Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolving Remarks Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved consumer-test remark Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * consumer-test slow working version Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * reader changes Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Consumer tests time issue fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Eof warning fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Makefile Fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Readers Initial Setup Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * consumer json fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Bugfix and resolves comments. Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * one test fails Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * solved last page read Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * writers future time problem fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * weird Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * weird Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Tests Passes Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolve Semaphore Issues Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * UUID comment on consumer tests resolved Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Inclusive from and Exclusive to Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Inclusive from and Exclusive to Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * 1 second limits Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * 1 second limits Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fixed json time Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fixed CI error Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved request Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved request Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved requests Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * removed blank line Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved comment Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * deleted unnecessary string builder Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * initial commit Signed-off-by: aryan <aryangodara03@gmail.com> * change influxdb docker image version. Signed-off-by: aryan <aryangodara03@gmail.com> * go mod and vendor fixing Signed-off-by: aryan <aryangodara03@gmail.com> * Upgrade InfluxDB from 1.x to 2.x Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influx DB configuration updated Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connection to InfluxDBv2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Token cannot be created Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connected to InfluxDB2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connected to InfluxDB2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB v2 Consumer Implementation Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * quickfix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb-Writer Unit Tests Update Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Consumer Update Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB Writer Tests Implemented Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB Connection Check Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolving Remarks Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved consumer-test remark Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * consumer-test slow working version Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * reader changes Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Consumer tests time issue fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Readers Initial Setup Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * consumer json fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Bugfix and resolves comments. Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * one test fails Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * solved last page read Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * writers future time problem fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * weird Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * weird Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Tests Passes Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolve Semaphore Issues Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * UUID comment on consumer tests resolved Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Inclusive from and Exclusive to Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Inclusive from and Exclusive to Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * 1 second limits Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * 1 second limits Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fixed json time Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fixed CI error Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved request Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved request Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved requests Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * removed blank line Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved comment Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * deleted unnecessary string builder Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * initial commit Signed-off-by: aryan <aryangodara03@gmail.com> * change influxdb docker image version. Signed-off-by: aryan <aryangodara03@gmail.com> * go mod and vendor fixing Signed-off-by: aryan <aryangodara03@gmail.com> * go mod fixing Signed-off-by: aryan <aryangodara03@gmail.com> * make tests pass locally, fix errors Signed-off-by: aryan <aryangodara03@gmail.com> * rem unsused variables/consts Signed-off-by: aryan <aryangodara03@gmail.com> * go mod tidy vendor Signed-off-by: aryan <aryangodara03@gmail.com> * add env var and data types Signed-off-by: aryan <aryangodara03@gmail.com> * update influxdb version to latest (2.3 to 2.12) Signed-off-by: aryan <aryangodara03@gmail.com> * change time precision for message tests Signed-off-by: aryan <aryangodara03@gmail.com> * renamed influxdb2 to influxdata Signed-off-by: aryan <aryangodara03@gmail.com> * address remarks on PR Signed-off-by: aryan <aryangodara03@gmail.com> * update influxbd according to latest mf commit Signed-off-by: aryan <aryangodara03@gmail.com> * temp commit, rem before pushing Signed-off-by: aryan <aryangodara03@gmail.com> * update main files and cassandra-reader messages_tests Signed-off-by: aryan <aryangodara03@gmail.com> * fix name of logger while importing Signed-off-by: aryan <aryangodara03@gmail.com> * remove unnecessary print lines Signed-off-by: aryan <aryangodara03@gmail.com> * correct env var name Signed-off-by: aryan <aryangodara03@gmail.com> * change to async consume Signed-off-by: aryan <aryangodara03@gmail.com> * add option to switch bw sync and async Signed-off-by: aryan <aryangodara03@gmail.com> * test for both async and sync Signed-off-by: aryan <aryangodara03@gmail.com> * update consumer and add writeAPIs to config Signed-off-by: aryan <aryangodara03@gmail.com> * revert back to sync consuming Signed-off-by: aryan <aryangodara03@gmail.com> * temp fix for default timeout value Signed-off-by: aryan <aryangodara03@gmail.com> * set default timeout in config. Signed-off-by: aryan <aryangodara03@gmail.com> * remove unwanted env vars, add required ones. Signed-off-by: aryan <aryangodara03@gmail.com> * rem unused username password from config Signed-off-by: aryan <aryangodara03@gmail.com> * update readme, env vars, and remove grafana Signed-off-by: aryan <aryangodara03@gmail.com> * update readme Signed-off-by: aryan <aryangodara03@gmail.com> * fix typo Signed-off-by: aryan <aryangodara03@gmail.com> * update readme description. Signed-off-by: aryan <aryangodara03@gmail.com> * fix more typos. Signed-off-by: aryan <aryangodara03@gmail.com> * add link to official docs to readme. Signed-off-by: aryan <aryangodara03@gmail.com> --------- Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> Signed-off-by: aryan <aryangodara03@gmail.com> Co-authored-by: fatih <fatihdurmaz@sabanciuniv.edu> Co-authored-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>
This commit is contained in:
Aryan Godara
GitHub
@@ -23,7 +23,7 @@ For more details, check out the [official documentation][docs].
|
||||
- Mutual TLS Authentication (mTLS) using X.509 Certificates
|
||||
- Fine-grained access control (policies, ABAC/RBAC)
|
||||
- Message persistence (Cassandra, InfluxDB, MongoDB and PostgresSQL)
|
||||
- Platform logging and instrumentation support (Grafana, Prometheus and OpenTracing)
|
||||
- Platform logging and instrumentation support (Prometheus and OpenTracing)
|
||||
- Event sourcing
|
||||
- Container-based deployment using [Docker][docker] and [Kubernetes][kubernetes]
|
||||
- [LoRaWAN][lora] network integration
|
||||
|
||||
@@ -6,7 +6,7 @@ import (
|
||||
"log"
|
||||
"os"
|
||||
|
||||
influxdata "github.com/influxdata/influxdb/client/v2"
|
||||
influxdb2 "github.com/influxdata/influxdb-client-go/v2"
|
||||
"github.com/mainflux/mainflux/internal"
|
||||
authClient "github.com/mainflux/mainflux/internal/clients/grpc/auth"
|
||||
thingsClient "github.com/mainflux/mainflux/internal/clients/grpc/things"
|
||||
@@ -66,13 +66,19 @@ func main() {
|
||||
if err := env.Parse(&influxDBConfig, env.Options{Prefix: envPrefixInfluxdb}); err != nil {
|
||||
logger.Fatal(fmt.Sprintf("failed to load InfluxDB client configuration from environment variable : %s", err))
|
||||
}
|
||||
client, err := influxDBClient.Connect(influxDBConfig)
|
||||
influxDBConfig.DBUrl = fmt.Sprintf("%s://%s:%s", influxDBConfig.Protocol, influxDBConfig.Host, influxDBConfig.Port)
|
||||
repocfg := influxdb.RepoConfig{
|
||||
Bucket: influxDBConfig.Bucket,
|
||||
Org: influxDBConfig.Org,
|
||||
}
|
||||
|
||||
client, err := influxDBClient.Connect(influxDBConfig, ctx)
|
||||
if err != nil {
|
||||
logger.Fatal(fmt.Sprintf("failed to connect to InfluxDB : %s", err))
|
||||
}
|
||||
defer client.Close()
|
||||
|
||||
repo := newService(client, influxDBConfig.DbName, logger)
|
||||
repo := newService(client, repocfg, logger)
|
||||
|
||||
httpServerConfig := server.Config{Port: defSvcHttpPort}
|
||||
if err := env.Parse(&httpServerConfig, env.Options{Prefix: envPrefixHttp, AltPrefix: envPrefix}); err != nil {
|
||||
@@ -93,8 +99,8 @@ func main() {
|
||||
}
|
||||
}
|
||||
|
||||
func newService(client influxdata.Client, dbName string, logger mflog.Logger) readers.MessageRepository {
|
||||
repo := influxdb.New(client, dbName)
|
||||
func newService(client influxdb2.Client, repocfg influxdb.RepoConfig, logger mflog.Logger) readers.MessageRepository {
|
||||
repo := influxdb.New(client, repocfg)
|
||||
repo = api.LoggingMiddleware(repo, logger)
|
||||
counter, latency := internal.MakeMetrics("influxdb", "message_reader")
|
||||
repo = api.MetricsMiddleware(repo, counter, latency)
|
||||
|
||||
@@ -9,7 +9,7 @@ import (
|
||||
"log"
|
||||
"os"
|
||||
|
||||
influxdata "github.com/influxdata/influxdb/client/v2"
|
||||
influxdb2 "github.com/influxdata/influxdb-client-go/v2"
|
||||
"github.com/mainflux/mainflux/consumers"
|
||||
"github.com/mainflux/mainflux/consumers/writers/api"
|
||||
"github.com/mainflux/mainflux/consumers/writers/influxdb"
|
||||
@@ -61,13 +61,19 @@ func main() {
|
||||
if err := env.Parse(&influxDBConfig, env.Options{Prefix: envPrefixInfluxdb}); err != nil {
|
||||
logger.Fatal(fmt.Sprintf("failed to load InfluxDB client configuration from environment variable : %s", err))
|
||||
}
|
||||
client, err := influxDBClient.Connect(influxDBConfig)
|
||||
influxDBConfig.DBUrl = fmt.Sprintf("%s://%s:%s", influxDBConfig.Protocol, influxDBConfig.Host, influxDBConfig.Port)
|
||||
repocfg := influxdb.RepoConfig{
|
||||
Bucket: influxDBConfig.Bucket,
|
||||
Org: influxDBConfig.Org,
|
||||
}
|
||||
|
||||
client, err := influxDBClient.Connect(influxDBConfig, ctx)
|
||||
if err != nil {
|
||||
logger.Fatal(fmt.Sprintf("failed to connect to InfluxDB : %s", err))
|
||||
}
|
||||
defer client.Close()
|
||||
|
||||
repo := newService(client, influxDBConfig.DbName, logger)
|
||||
repo := newService(client, repocfg, logger)
|
||||
|
||||
if err := consumers.Start(svcName, pubSub, repo, cfg.ConfigPath, logger); err != nil {
|
||||
logger.Fatal(fmt.Sprintf("failed to start InfluxDB writer: %s", err))
|
||||
@@ -92,8 +98,8 @@ func main() {
|
||||
}
|
||||
}
|
||||
|
||||
func newService(client influxdata.Client, dbName string, logger mflog.Logger) consumers.Consumer {
|
||||
repo := influxdb.New(client, dbName)
|
||||
func newService(client influxdb2.Client, repocfg influxdb.RepoConfig, logger mflog.Logger) consumers.Consumer {
|
||||
repo := influxdb.New(client, repocfg, true)
|
||||
repo = api.LoggingMiddleware(repo, logger)
|
||||
counter, latency := internal.MakeMetrics("influxdb", "message_writer")
|
||||
repo = api.MetricsMiddleware(repo, counter, latency)
|
||||
|
||||
@@ -12,12 +12,20 @@ default values.
|
||||
| ----------------------------- | --------------------------------------------------------------------------------- | ---------------------- |
|
||||
| MF_BROKER_URL | Message broker instance URL | nats://localhost:4222 |
|
||||
| MF_INFLUX_WRITER_LOG_LEVEL | Log level for InfluxDB writer (debug, info, warn, error) | info |
|
||||
| MF_INFLUX_WRITER_PORT | Service HTTP port | 8180 |
|
||||
| MF_INFLUX_WRITER_PORT | Service HTTP port | 8900 |
|
||||
| MF_INFLUX_WRITER_DB_HOST | InfluxDB host | localhost |
|
||||
| MF_INFLUXDB_PORT | Default port of InfluxDB database | 8086 |
|
||||
| MF_INFLUXDB_ADMIN_USER | Default user of InfluxDB database | mainflux |
|
||||
| MF_INFLUXDB_ADMIN_PASSWORD | Default password of InfluxDB user | mainflux |
|
||||
| MF_INFLUXDB_DB | InfluxDB database name | mainflux |
|
||||
| MF_INFLUXDB_HOST | InfluxDB host name | mainflux-influxdb |
|
||||
| MF_INFLUXDB_PROTOCOL | InfluxDB protocol | http |
|
||||
| MF_INFLUXDB_TIMEOUT | InfluxDB client connection readiness timeout | 1s |
|
||||
| MF_INFLUXDB_ORG | InfluxDB organization name | mainflux |
|
||||
| MF_INFLUXDB_BUCKET | InfluxDB bucket name | mainflux-bucket |
|
||||
| MF_INFLUXDB_TOKEN | InfluxDB API token | mainflux-token |
|
||||
| MF_INFLUXDB_HTTP_ENABLED | InfluxDB http enabled status | true |
|
||||
| MF_INFLUXDB_INIT_MODE | InfluxDB initialization mode | setup |
|
||||
| MF_INFLUX_WRITER_CONFIG_PATH | Config file path with message broker subjects list, payload type and content-type | /configs.toml |
|
||||
|
||||
## Deployment
|
||||
@@ -47,6 +55,13 @@ MF_INFLUXDB_HOST=[InfluxDB database host] \
|
||||
MF_INFLUXDB_PORT=[InfluxDB database port] \
|
||||
MF_INFLUXDB_ADMIN_USER=[InfluxDB admin user] \
|
||||
MF_INFLUXDB_ADMIN_PASSWORD=[InfluxDB admin password] \
|
||||
MF_INFLUXDB_PROTOCOL=[InfluxDB protocol] \
|
||||
MF_INFLUXDB_TIMEOUT=[InfluxDB timeout] \
|
||||
MF_INFLUXDB_ORG=[InfluxDB org] \
|
||||
MF_INFLUXDB_BUCKET=[InfluxDB bucket] \
|
||||
MF_INFLUXDB_TOKEN=[InfluxDB token] \
|
||||
MF_INFLUXDB_HTTP_ENABLED=[InfluxDB http enabled] \
|
||||
MF_INFLUXDB_INIT_MODE=[InfluxDB init mode] \
|
||||
MF_INFLUX_WRITER_CONFIG_PATH=[Config file path with Message broker subjects list, payload type and content-type] \
|
||||
$GOBIN/mainflux-influxdb
|
||||
```
|
||||
@@ -55,17 +70,23 @@ $GOBIN/mainflux-influxdb
|
||||
|
||||
This service can be deployed using docker containers.
|
||||
Docker compose file is available in `<project_root>/docker/addons/influxdb-writer/docker-compose.yml`. Besides database
|
||||
and writer service, it contains [Grafana platform](https://grafana.com/) which can be used for database
|
||||
and writer service, it contains InfluxData Web Admin Interface which can be used for database
|
||||
exploration and data visualization and analytics. In order to run Mainflux InfluxDB writer, execute the following command:
|
||||
|
||||
```bash
|
||||
docker-compose -f docker/addons/influxdb-writer/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
And, to use the default .env file, execute the following command:
|
||||
|
||||
```bash
|
||||
docker-compose -f docker/addons/influxdb-writer/docker-compose.yml up --env-file docker/.env -d
|
||||
```
|
||||
|
||||
_Please note that you need to start core services before the additional ones._
|
||||
|
||||
## Usage
|
||||
|
||||
Starting service will start consuming normalized messages in SenML format.
|
||||
|
||||
[doc]: https://docs.mainflux.io
|
||||
Official docs can be found [here](https://docs.mainflux.io).
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
package influxdb
|
||||
|
||||
import (
|
||||
"context"
|
||||
"math"
|
||||
"time"
|
||||
|
||||
@@ -12,7 +13,9 @@ import (
|
||||
"github.com/mainflux/mainflux/pkg/transformers/json"
|
||||
"github.com/mainflux/mainflux/pkg/transformers/senml"
|
||||
|
||||
influxdata "github.com/influxdata/influxdb/client/v2"
|
||||
influxdb2 "github.com/influxdata/influxdb-client-go/v2"
|
||||
"github.com/influxdata/influxdb-client-go/v2/api"
|
||||
"github.com/influxdata/influxdb-client-go/v2/api/write"
|
||||
)
|
||||
|
||||
const senmlPoints = "messages"
|
||||
@@ -21,65 +24,63 @@ var errSaveMessage = errors.New("failed to save message to influxdb database")
|
||||
|
||||
var _ consumers.Consumer = (*influxRepo)(nil)
|
||||
|
||||
type RepoConfig struct {
|
||||
Bucket string
|
||||
Org string
|
||||
}
|
||||
|
||||
type influxRepo struct {
|
||||
client influxdata.Client
|
||||
cfg influxdata.BatchPointsConfig
|
||||
client influxdb2.Client
|
||||
cfg RepoConfig
|
||||
writeAPIBlocking api.WriteAPIBlocking
|
||||
}
|
||||
|
||||
// New returns new InfluxDB writer.
|
||||
func New(client influxdata.Client, database string) consumers.Consumer {
|
||||
func New(client influxdb2.Client, config RepoConfig, async bool) consumers.Consumer {
|
||||
return &influxRepo{
|
||||
client: client,
|
||||
cfg: influxdata.BatchPointsConfig{
|
||||
Database: database,
|
||||
},
|
||||
client: client,
|
||||
cfg: config,
|
||||
writeAPIBlocking: client.WriteAPIBlocking(config.Org, config.Bucket),
|
||||
}
|
||||
}
|
||||
|
||||
func (repo *influxRepo) Consume(message interface{}) error {
|
||||
pts, err := influxdata.NewBatchPoints(repo.cfg)
|
||||
if err != nil {
|
||||
return errors.Wrap(errSaveMessage, err)
|
||||
}
|
||||
var err error
|
||||
var pts []*write.Point
|
||||
switch m := message.(type) {
|
||||
case json.Messages:
|
||||
pts, err = repo.jsonPoints(pts, m)
|
||||
pts, err = repo.jsonPoints(m)
|
||||
default:
|
||||
pts, err = repo.senmlPoints(pts, m)
|
||||
pts, err = repo.senmlPoints(m)
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := repo.client.Write(pts); err != nil {
|
||||
return errors.Wrap(errSaveMessage, err)
|
||||
}
|
||||
return nil
|
||||
return repo.writeAPIBlocking.WritePoint(context.Background(), pts...)
|
||||
}
|
||||
|
||||
func (repo *influxRepo) senmlPoints(pts influxdata.BatchPoints, messages interface{}) (influxdata.BatchPoints, error) {
|
||||
func (repo *influxRepo) senmlPoints(messages interface{}) ([]*write.Point, error) {
|
||||
msgs, ok := messages.([]senml.Message)
|
||||
if !ok {
|
||||
return nil, errSaveMessage
|
||||
}
|
||||
|
||||
var pts []*write.Point
|
||||
for _, msg := range msgs {
|
||||
tgs, flds := senmlTags(msg), senmlFields(msg)
|
||||
|
||||
sec, dec := math.Modf(msg.Time)
|
||||
t := time.Unix(int64(sec), int64(dec*(1e9)))
|
||||
|
||||
pt, err := influxdata.NewPoint(senmlPoints, tgs, flds, t)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(errSaveMessage, err)
|
||||
}
|
||||
pts.AddPoint(pt)
|
||||
pt := influxdb2.NewPoint(senmlPoints, tgs, flds, t)
|
||||
pts = append(pts, pt)
|
||||
}
|
||||
|
||||
return pts, nil
|
||||
}
|
||||
|
||||
func (repo *influxRepo) jsonPoints(pts influxdata.BatchPoints, msgs json.Messages) (influxdata.BatchPoints, error) {
|
||||
func (repo *influxRepo) jsonPoints(msgs json.Messages) ([]*write.Point, error) {
|
||||
var pts []*write.Point
|
||||
for i, m := range msgs.Data {
|
||||
t := time.Unix(0, m.Created+int64(i))
|
||||
|
||||
@@ -96,11 +97,8 @@ func (repo *influxRepo) jsonPoints(pts influxdata.BatchPoints, msgs json.Message
|
||||
}
|
||||
// At least one known field need to exist so that COUNT can be performed.
|
||||
fields["protocol"] = m.Protocol
|
||||
pt, err := influxdata.NewPoint(msgs.Format, jsonTags(m), fields, t)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(errSaveMessage, err)
|
||||
}
|
||||
pts.AddPoint(pt)
|
||||
pt := influxdb2.NewPoint(msgs.Format, jsonTags(m), fields, t)
|
||||
pts = append(pts, pt)
|
||||
}
|
||||
|
||||
return pts, nil
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
package influxdb_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"testing"
|
||||
@@ -11,145 +12,213 @@ import (
|
||||
|
||||
"github.com/mainflux/mainflux/pkg/errors"
|
||||
|
||||
"github.com/gofrs/uuid"
|
||||
influxdata "github.com/influxdata/influxdb/client/v2"
|
||||
influxdata "github.com/influxdata/influxdb-client-go/v2"
|
||||
writer "github.com/mainflux/mainflux/consumers/writers/influxdb"
|
||||
log "github.com/mainflux/mainflux/logger"
|
||||
"github.com/mainflux/mainflux/pkg/transformers/json"
|
||||
"github.com/mainflux/mainflux/pkg/transformers/senml"
|
||||
"github.com/mainflux/mainflux/pkg/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
const valueFields = 5
|
||||
|
||||
var (
|
||||
port string
|
||||
testLog, _ = log.New(os.Stdout, log.Info.String())
|
||||
testDB = "test"
|
||||
streamsSize = 250
|
||||
selectMsgs = "SELECT * FROM test..messages"
|
||||
dropMsgs = "DROP SERIES FROM messages"
|
||||
client influxdata.Client
|
||||
clientCfg = influxdata.HTTPConfig{
|
||||
Username: "test",
|
||||
Password: "test",
|
||||
}
|
||||
subtopic = "topic"
|
||||
)
|
||||
testLog, _ = log.New(os.Stdout, log.Info.String())
|
||||
streamsSize = 250
|
||||
rowCountSenml = fmt.Sprintf(`from(bucket: "%s")
|
||||
|> range(start: -1h, stop: 1h)
|
||||
|> filter(fn: (r) => r["_measurement"] == "messages")
|
||||
|> filter(fn: (r) => r["_field"] == "dataValue" or r["_field"] == "stringValue" or r["_field"] == "value" or r["_field"] == "boolValue" or r["_field"] == "sum" )
|
||||
|> group(columns: ["_measurement"])
|
||||
|> count()
|
||||
|> yield(name: "count")`, repoCfg.Bucket)
|
||||
|
||||
var (
|
||||
rowCountJson = fmt.Sprintf(`from(bucket: "%s")
|
||||
|> range(start: -1h, stop: 1h)
|
||||
|> filter(fn: (r) => r["_measurement"] == "some_json")
|
||||
|> filter(fn: (r) => r["_field"] == "field_1" or r["_field"] == "field_2" or r["_field"] == "field_3" or r["_field"] == "field_4" or r["_field"] == "field_5/field_1" or r["_field"] == "field_5/field_2")
|
||||
|> count()
|
||||
|> yield(name: "count")`, repoCfg.Bucket)
|
||||
subtopic = "topic"
|
||||
|
||||
client influxdata.Client
|
||||
v float64 = 5
|
||||
stringV = "value"
|
||||
boolV = true
|
||||
dataV = "base64"
|
||||
sum float64 = 42
|
||||
repoCfg = writer.RepoConfig{
|
||||
Bucket: dbBucket,
|
||||
Org: dbOrg,
|
||||
}
|
||||
errUnexpectedType = errors.New("Unexpected response type")
|
||||
|
||||
idProvider = uuid.New()
|
||||
)
|
||||
|
||||
// This is utility function to query the database.
|
||||
func queryDB(cmd string) ([][]interface{}, error) {
|
||||
q := influxdata.Query{
|
||||
Command: cmd,
|
||||
Database: testDB,
|
||||
}
|
||||
response, err := client.Query(q)
|
||||
func deleteBucket() error {
|
||||
bucketsAPI := client.BucketsAPI()
|
||||
bucket, err := bucketsAPI.FindBucketByName(context.Background(), repoCfg.Bucket)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return err
|
||||
}
|
||||
if response.Error() != nil {
|
||||
return nil, response.Error()
|
||||
|
||||
if err = bucketsAPI.DeleteBucket(context.Background(), bucket); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(response.Results[0].Series) == 0 {
|
||||
return nil, nil
|
||||
|
||||
return nil
|
||||
}
|
||||
func createBucket() error {
|
||||
orgAPI := client.OrganizationsAPI()
|
||||
org, err := orgAPI.FindOrganizationByName(context.Background(), repoCfg.Org)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// There is only one query, so only one result and
|
||||
// all data are stored in the same series.
|
||||
return response.Results[0].Series[0].Values, nil
|
||||
bucketsAPI := client.BucketsAPI()
|
||||
if _, err = bucketsAPI.CreateBucketWithName(context.Background(), org, repoCfg.Bucket); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
func resetBucket() error {
|
||||
if err := deleteBucket(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := createBucket(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func queryDB(fluxQuery string) (int, error) {
|
||||
rowCount := 0
|
||||
queryAPI := client.QueryAPI(repoCfg.Org)
|
||||
|
||||
// get QueryTableResult
|
||||
result, err := queryAPI.Query(context.Background(), fluxQuery)
|
||||
if err != nil {
|
||||
return rowCount, err
|
||||
}
|
||||
if result.Next() {
|
||||
value, ok := result.Record().Value().(int64)
|
||||
if !ok {
|
||||
return rowCount, errUnexpectedType
|
||||
}
|
||||
rowCount = int(value)
|
||||
}
|
||||
if result.Err() != nil {
|
||||
return rowCount, result.Err()
|
||||
}
|
||||
|
||||
return rowCount, nil
|
||||
}
|
||||
|
||||
func TestSaveSenml(t *testing.T) {
|
||||
repo := writer.New(client, testDB)
|
||||
for i := 0; i < 2; i++ {
|
||||
// Testing both async and sync
|
||||
repo := writer.New(client, repoCfg, i == 0)
|
||||
|
||||
cases := []struct {
|
||||
desc string
|
||||
msgsNum int
|
||||
expectedSize int
|
||||
}{
|
||||
{
|
||||
desc: "save a single message",
|
||||
msgsNum: 1,
|
||||
expectedSize: 1,
|
||||
},
|
||||
{
|
||||
desc: "save a batch of messages",
|
||||
msgsNum: streamsSize,
|
||||
expectedSize: streamsSize,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
// Clean previously saved messages.
|
||||
_, err := queryDB(dropMsgs)
|
||||
require.Nil(t, err, fmt.Sprintf("Cleaning data from InfluxDB expected to succeed: %s.\n", err))
|
||||
|
||||
now := time.Now().UnixNano()
|
||||
msg := senml.Message{
|
||||
Channel: "45",
|
||||
Publisher: "2580",
|
||||
Protocol: "http",
|
||||
Name: "test name",
|
||||
Unit: "km",
|
||||
UpdateTime: 5456565466,
|
||||
cases := []struct {
|
||||
desc string
|
||||
msgsNum int
|
||||
expectedSize int
|
||||
}{
|
||||
{
|
||||
desc: "save a single message",
|
||||
msgsNum: 1,
|
||||
expectedSize: 1,
|
||||
},
|
||||
{
|
||||
desc: "save a batch of messages",
|
||||
msgsNum: streamsSize,
|
||||
expectedSize: streamsSize,
|
||||
},
|
||||
}
|
||||
var msgs []senml.Message
|
||||
|
||||
for i := 0; i < tc.msgsNum; i++ {
|
||||
// Mix possible values as well as value sum.
|
||||
count := i % valueFields
|
||||
switch count {
|
||||
case 0:
|
||||
msg.Subtopic = subtopic
|
||||
msg.Value = &v
|
||||
case 1:
|
||||
msg.BoolValue = &boolV
|
||||
case 2:
|
||||
msg.StringValue = &stringV
|
||||
case 3:
|
||||
msg.DataValue = &dataV
|
||||
case 4:
|
||||
msg.Sum = &sum
|
||||
for _, tc := range cases {
|
||||
err := resetBucket()
|
||||
assert.Nil(t, err, fmt.Sprintf("Cleaning data from InfluxDB expected to succeed: %s.\n", err))
|
||||
now := time.Now().UnixNano()
|
||||
var msgs []senml.Message
|
||||
|
||||
chanID, err := idProvider.ID()
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s\n", err))
|
||||
pubID, err := idProvider.ID()
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s\n", err))
|
||||
for i := 0; i < tc.msgsNum; i++ {
|
||||
msg := senml.Message{
|
||||
Channel: chanID,
|
||||
Publisher: pubID,
|
||||
Protocol: "http",
|
||||
Name: "test name",
|
||||
Unit: "km",
|
||||
UpdateTime: 5456565466,
|
||||
}
|
||||
// Mix possible values as well as value sum.
|
||||
count := i % valueFields
|
||||
switch count {
|
||||
case 0:
|
||||
msg.Subtopic = subtopic
|
||||
msg.Value = &v
|
||||
case 1:
|
||||
msg.BoolValue = &boolV
|
||||
case 2:
|
||||
msg.StringValue = &stringV
|
||||
case 3:
|
||||
msg.DataValue = &dataV
|
||||
case 4:
|
||||
msg.Sum = &sum
|
||||
}
|
||||
|
||||
msg.Time = float64(now)/float64(1e9) - float64(i)
|
||||
msgs = append(msgs, msg)
|
||||
}
|
||||
|
||||
msg.Time = float64(now)/float64(1e9) + float64(i)
|
||||
msgs = append(msgs, msg)
|
||||
err = repo.Consume(msgs)
|
||||
assert.Nil(t, err, fmt.Sprintf("Save operation expected to succeed: %s.\n", err))
|
||||
|
||||
count, err := queryDB(rowCountSenml)
|
||||
assert.Nil(t, err, fmt.Sprintf("Querying InfluxDB to retrieve data expected to succeed: %s.\n", err))
|
||||
assert.Equal(t, tc.expectedSize, count, fmt.Sprintf("Expected to have %d messages saved, found %d instead.\n", tc.expectedSize, count))
|
||||
}
|
||||
|
||||
err = repo.Consume(msgs)
|
||||
assert.Nil(t, err, fmt.Sprintf("Save operation expected to succeed: %s.\n", err))
|
||||
|
||||
row, err := queryDB(selectMsgs)
|
||||
assert.Nil(t, err, fmt.Sprintf("Querying InfluxDB to retrieve data expected to succeed: %s.\n", err))
|
||||
|
||||
count := len(row)
|
||||
assert.Equal(t, tc.expectedSize, count, fmt.Sprintf("Expected to have %d messages saved, found %d instead.\n", tc.expectedSize, count))
|
||||
}
|
||||
}
|
||||
|
||||
func TestSaveJSON(t *testing.T) {
|
||||
repo := writer.New(client, testDB)
|
||||
// Testing both async and sync
|
||||
for i := 0; i < 2; i++ {
|
||||
// Testing both async and sync
|
||||
repo := writer.New(client, repoCfg, i == 0)
|
||||
|
||||
chid, err := uuid.NewV4()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
pubid, err := uuid.NewV4()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
chanID, err := idProvider.ID()
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
pubID, err := idProvider.ID()
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
|
||||
msg := json.Message{
|
||||
Channel: chid.String(),
|
||||
Publisher: pubid.String(),
|
||||
Created: time.Now().Unix(),
|
||||
Subtopic: "subtopic/format/some_json",
|
||||
Protocol: "mqtt",
|
||||
Payload: map[string]interface{}{
|
||||
msg := json.Message{
|
||||
Channel: chanID,
|
||||
Publisher: pubID,
|
||||
Created: time.Now().UnixNano(),
|
||||
Subtopic: "subtopic/format/some_json",
|
||||
Protocol: "mqtt",
|
||||
Payload: map[string]interface{}{
|
||||
"field_1": 123,
|
||||
"field_2": "value",
|
||||
"field_3": false,
|
||||
"field_4": 12.344,
|
||||
"field_5": map[string]interface{}{
|
||||
"field_1": "value",
|
||||
"field_2": 42,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
invalidKeySepMsg := msg
|
||||
invalidKeySepMsg.Payload = map[string]interface{}{
|
||||
"field_1": 123,
|
||||
"field_2": "value",
|
||||
"field_3": false,
|
||||
@@ -158,81 +227,72 @@ func TestSaveJSON(t *testing.T) {
|
||||
"field_1": "value",
|
||||
"field_2": 42,
|
||||
},
|
||||
},
|
||||
}
|
||||
"field_6/field_7": "value",
|
||||
}
|
||||
invalidKeyNameMsg := msg
|
||||
invalidKeyNameMsg.Payload = map[string]interface{}{
|
||||
"field_1": 123,
|
||||
"field_2": "value",
|
||||
"field_3": false,
|
||||
"field_4": 12.344,
|
||||
"field_5": map[string]interface{}{
|
||||
"field_1": "value",
|
||||
"field_2": 42,
|
||||
},
|
||||
"publisher": "value",
|
||||
}
|
||||
|
||||
invalidKeySepMsg := msg
|
||||
invalidKeySepMsg.Payload = map[string]interface{}{
|
||||
"field_1": 123,
|
||||
"field_2": "value",
|
||||
"field_3": false,
|
||||
"field_4": 12.344,
|
||||
"field_5": map[string]interface{}{
|
||||
"field_1": "value",
|
||||
"field_2": 42,
|
||||
},
|
||||
"field_6/field_7": "value",
|
||||
}
|
||||
invalidKeyNameMsg := msg
|
||||
invalidKeyNameMsg.Payload = map[string]interface{}{
|
||||
"field_1": 123,
|
||||
"field_2": "value",
|
||||
"field_3": false,
|
||||
"field_4": 12.344,
|
||||
"field_5": map[string]interface{}{
|
||||
"field_1": "value",
|
||||
"field_2": 42,
|
||||
},
|
||||
"publisher": "value",
|
||||
}
|
||||
now := time.Now().UnixNano()
|
||||
msgs := json.Messages{
|
||||
Format: "some_json",
|
||||
}
|
||||
invalidKeySepMsgs := json.Messages{
|
||||
Format: "some_json",
|
||||
}
|
||||
invalidKeyNameMsgs := json.Messages{
|
||||
Format: "some_json",
|
||||
}
|
||||
|
||||
now := time.Now().Unix()
|
||||
msgs := json.Messages{
|
||||
Format: "some_json",
|
||||
}
|
||||
invalidKeySepMsgs := json.Messages{
|
||||
Format: "some_json",
|
||||
}
|
||||
invalidKeyNameMsgs := json.Messages{
|
||||
Format: "some_json",
|
||||
}
|
||||
for i := 0; i < streamsSize; i++ {
|
||||
msg.Created = now
|
||||
msgs.Data = append(msgs.Data, msg)
|
||||
invalidKeySepMsgs.Data = append(invalidKeySepMsgs.Data, invalidKeySepMsg)
|
||||
invalidKeyNameMsgs.Data = append(invalidKeyNameMsgs.Data, invalidKeyNameMsg)
|
||||
}
|
||||
|
||||
for i := 0; i < streamsSize; i++ {
|
||||
msg.Created = now + int64(i)
|
||||
msgs.Data = append(msgs.Data, msg)
|
||||
invalidKeySepMsgs.Data = append(invalidKeySepMsgs.Data, invalidKeySepMsg)
|
||||
invalidKeyNameMsgs.Data = append(invalidKeyNameMsgs.Data, invalidKeyNameMsg)
|
||||
}
|
||||
cases := []struct {
|
||||
desc string
|
||||
msgs json.Messages
|
||||
err error
|
||||
}{
|
||||
{
|
||||
desc: "consume valid json messages",
|
||||
msgs: msgs,
|
||||
err: nil,
|
||||
},
|
||||
{
|
||||
desc: "consume invalid json messages containing invalid key separator",
|
||||
msgs: invalidKeySepMsgs,
|
||||
err: json.ErrInvalidKey,
|
||||
},
|
||||
{
|
||||
desc: "consume invalid json messages containing invalid key name",
|
||||
msgs: invalidKeySepMsgs,
|
||||
err: json.ErrInvalidKey,
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
err := resetBucket()
|
||||
assert.Nil(t, err, fmt.Sprintf("Cleaning data from InfluxDB expected to succeed: %s.\n", err))
|
||||
|
||||
cases := []struct {
|
||||
desc string
|
||||
msgs json.Messages
|
||||
err error
|
||||
}{
|
||||
{
|
||||
desc: "consume valid json messages",
|
||||
msgs: msgs,
|
||||
err: nil,
|
||||
},
|
||||
{
|
||||
desc: "consume invalid json messages containing invalid key separator",
|
||||
msgs: invalidKeySepMsgs,
|
||||
err: json.ErrInvalidKey,
|
||||
},
|
||||
{
|
||||
desc: "consume invalid json messages containing invalid key name",
|
||||
msgs: invalidKeySepMsgs,
|
||||
err: json.ErrInvalidKey,
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
err = repo.Consume(tc.msgs)
|
||||
assert.True(t, errors.Contains(err, tc.err), fmt.Sprintf("%s expected %s, got %s", tc.desc, tc.err, err))
|
||||
|
||||
row, err := queryDB(selectMsgs)
|
||||
assert.Nil(t, err, fmt.Sprintf("Querying InfluxDB to retrieve data expected to succeed: %s.\n", err))
|
||||
|
||||
count := len(row)
|
||||
assert.Equal(t, streamsSize, count, fmt.Sprintf("Expected to have %d messages saved, found %d instead.\n", streamsSize, count))
|
||||
switch err = repo.Consume(tc.msgs); err {
|
||||
case nil:
|
||||
count, err := queryDB(rowCountJson)
|
||||
assert.Nil(t, err, fmt.Sprintf("Querying InfluxDB to retrieve data expected to succeed: %s.\n", err))
|
||||
assert.Equal(t, streamsSize, count, fmt.Sprintf("Expected to have %d messages saved, found %d instead.\n", streamsSize, count))
|
||||
default:
|
||||
assert.True(t, errors.Contains(err, tc.err), fmt.Sprintf("%s expected %s, got %s", tc.desc, tc.err, err))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,15 +4,36 @@
|
||||
package influxdb_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
influxdb "github.com/influxdata/influxdb/client/v2"
|
||||
influxdata "github.com/influxdata/influxdb-client-go/v2"
|
||||
dockertest "github.com/ory/dockertest/v3"
|
||||
)
|
||||
|
||||
const (
|
||||
dbToken = "test-token"
|
||||
dbOrg = "test-org"
|
||||
dbAdmin = "test-admin"
|
||||
dbPass = "test-password"
|
||||
dbBucket = "test-bucket"
|
||||
dbInitMode = "setup"
|
||||
dbFluxEnabled = "true"
|
||||
dbBindAddress = ":8088"
|
||||
port = "8086/tcp"
|
||||
broker = "influxdb"
|
||||
brokerVersion = "2.2-alpine"
|
||||
poolMaxWait = 120 * time.Second
|
||||
)
|
||||
|
||||
var address string
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
pool, err := dockertest.NewPool("")
|
||||
if err != nil {
|
||||
@@ -20,31 +41,49 @@ func TestMain(m *testing.M) {
|
||||
}
|
||||
|
||||
cfg := []string{
|
||||
"INFLUXDB_USER=test",
|
||||
"INFLUXDB_USER_PASSWORD=test",
|
||||
"INFLUXDB_DB=test",
|
||||
fmt.Sprintf("DOCKER_INFLUXDB_INIT_MODE=%s", dbInitMode),
|
||||
fmt.Sprintf("DOCKER_INFLUXDB_INIT_USERNAME=%s", dbAdmin),
|
||||
fmt.Sprintf("DOCKER_INFLUXDB_INIT_PASSWORD=%s", dbPass),
|
||||
fmt.Sprintf("DOCKER_INFLUXDB_INIT_ORG=%s", dbOrg),
|
||||
fmt.Sprintf("DOCKER_INFLUXDB_INIT_BUCKET=%s", dbBucket),
|
||||
fmt.Sprintf("DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=%s", dbToken),
|
||||
fmt.Sprintf("INFLUXDB_HTTP_FLUX_ENABLED=%s", dbFluxEnabled),
|
||||
fmt.Sprintf("INFLUXDB_BIND_ADDRESS=%s", dbBindAddress),
|
||||
}
|
||||
container, err := pool.Run("influxdb", "1.8.5", cfg)
|
||||
container, err := pool.Run(broker, brokerVersion, cfg)
|
||||
if err != nil {
|
||||
testLog.Error(fmt.Sprintf("Could not start container: %s", err))
|
||||
}
|
||||
|
||||
port = container.GetPort("8086/tcp")
|
||||
clientCfg.Addr = fmt.Sprintf("http://localhost:%s", port)
|
||||
handleInterrupt(m, pool, container)
|
||||
|
||||
address = fmt.Sprintf("%s:%s", "http://localhost", container.GetPort(port))
|
||||
pool.MaxWait = poolMaxWait
|
||||
|
||||
if err := pool.Retry(func() error {
|
||||
client, err = influxdb.NewHTTPClient(clientCfg)
|
||||
_, _, err = client.Ping(5 * time.Millisecond)
|
||||
client = influxdata.NewClientWithOptions(address, dbToken, influxdata.DefaultOptions())
|
||||
_, err = client.Ready(context.Background())
|
||||
return err
|
||||
}); err != nil {
|
||||
testLog.Error(fmt.Sprintf("Could not connect to docker: %s", err))
|
||||
}
|
||||
|
||||
code := m.Run()
|
||||
|
||||
if err := pool.Purge(container); err != nil {
|
||||
testLog.Error(fmt.Sprintf("Could not purge container: %s", err))
|
||||
}
|
||||
|
||||
os.Exit(code)
|
||||
}
|
||||
|
||||
func handleInterrupt(m *testing.M, pool *dockertest.Pool, container *dockertest.Resource) {
|
||||
c := make(chan os.Signal, 2)
|
||||
signal.Notify(c, os.Interrupt, syscall.SIGTERM)
|
||||
go func() {
|
||||
<-c
|
||||
if err := pool.Purge(container); err != nil {
|
||||
log.Fatalf("Could not purge container: %s", err)
|
||||
}
|
||||
os.Exit(0)
|
||||
}()
|
||||
}
|
||||
|
||||
+9
-6
@@ -26,9 +26,6 @@ MF_BROKER_URL=${MF_NATS_URL}
|
||||
## Redis
|
||||
MF_REDIS_TCP_PORT=6379
|
||||
|
||||
## Grafana
|
||||
MF_GRAFANA_PORT=3000
|
||||
|
||||
## Jaeger
|
||||
MF_JAEGER_PORT=6831
|
||||
MF_JAEGER_FRONTEND=16686
|
||||
@@ -240,23 +237,29 @@ MF_CASSANDRA_READER_SERVER_KEY=
|
||||
|
||||
### InfluxDB
|
||||
MF_INFLUXDB_PORT=8086
|
||||
MF_INFLUXDB_DB=mainflux
|
||||
MF_INFLUXDB_HOST=mainflux-influxdb
|
||||
MF_INFLUXDB_ADMIN_USER=mainflux
|
||||
MF_INFLUXDB_ADMIN_PASSWORD=mainflux
|
||||
MF_INFLUXDB_HTTP_AUTH_ENABLED=true
|
||||
MF_INFLUXDB_PROTOCOL=http
|
||||
MF_INFLUXDB_TIMEOUT=1s
|
||||
MF_INFLUXDB_ORG=mainflux
|
||||
MF_INFLUXDB_BUCKET=mainflux-bucket
|
||||
MF_INFLUXDB_TOKEN=mainflux-token
|
||||
MF_INFLUXDB_HTTP_ENABLED=true
|
||||
MF_INFLUXDB_INIT_MODE=setup
|
||||
|
||||
### InfluxDB Writer
|
||||
MF_INFLUX_WRITER_LOG_LEVEL=debug
|
||||
MF_INFLUX_WRITER_PORT=8900
|
||||
MF_INFLUX_WRITER_BATCH_SIZE=5000
|
||||
MF_INFLUX_WRITER_BATCH_TIMEOUT=5
|
||||
MF_INFLUX_WRITER_GRAFANA_PORT=3001
|
||||
|
||||
### InfluxDB Reader
|
||||
MF_INFLUX_READER_LOG_LEVEL=debug
|
||||
MF_INFLUX_READER_PORT=8905
|
||||
MF_INFLUX_READER_SERVER_KEY=
|
||||
MF_INFLUX_READER_SERVER_CERT=
|
||||
MF_INFLUXDB_DB=mainflux
|
||||
|
||||
### MongoDB Writer
|
||||
MF_MONGO_WRITER_LOG_LEVEL=debug
|
||||
|
||||
@@ -25,6 +25,8 @@ services:
|
||||
MF_INFLUXDB_DB: ${MF_INFLUXDB_DB}
|
||||
MF_INFLUXDB_HOST: mainflux-influxdb
|
||||
MF_INFLUXDB_PORT: ${MF_INFLUXDB_PORT}
|
||||
MF_INFLUXDB_TIMEOUT: ${MF_INFLUXDB_TIMEOUT}
|
||||
MF_INFLUXDB_PROTOCOL: ${MF_INFLUXDB_PROTOCOL}
|
||||
MF_INFLUXDB_ADMIN_USER: ${MF_INFLUXDB_ADMIN_USER}
|
||||
MF_INFLUXDB_ADMIN_PASSWORD: ${MF_INFLUXDB_ADMIN_PASSWORD}
|
||||
MF_INFLUX_READER_SERVER_CERT: ${MF_INFLUX_READER_SERVER_CERT}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Copyright (c) Mainflux
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
# This docker-compose file contains optional InfluxDB, InfluxDB-writer and Grafana services
|
||||
# This docker-compose file contains optional InfluxDB and InfluxDB-writer services
|
||||
# for the Mainflux platform. Since this services are optional, this file is dependent on the
|
||||
# docker-compose.yml file from <project_root>/docker/. In order to run these services,
|
||||
# core services, as well as the network from the core composition, should be already running.
|
||||
@@ -14,18 +14,20 @@ networks:
|
||||
|
||||
volumes:
|
||||
mainflux-influxdb-volume:
|
||||
mainflux-grafana-volume:
|
||||
|
||||
services:
|
||||
influxdb:
|
||||
image: influxdb:1.8.5
|
||||
image: influxdb:2.5
|
||||
container_name: mainflux-influxdb
|
||||
restart: on-failure
|
||||
environment:
|
||||
INFLUXDB_DB: ${MF_INFLUXDB_DB}
|
||||
INFLUXDB_ADMIN_USER: ${MF_INFLUXDB_ADMIN_USER}
|
||||
INFLUXDB_ADMIN_PASSWORD: ${MF_INFLUXDB_ADMIN_PASSWORD}
|
||||
INFLUXDB_HTTP_AUTH_ENABLED: ${MF_INFLUXDB_HTTP_AUTH_ENABLED}
|
||||
DOCKER_INFLUXDB_INIT_MODE: ${MF_INFLUXDB_INIT_MODE}
|
||||
DOCKER_INFLUXDB_INIT_USERNAME: ${MF_INFLUXDB_ADMIN_USER}
|
||||
DOCKER_INFLUXDB_INIT_PASSWORD: ${MF_INFLUXDB_ADMIN_PASSWORD}
|
||||
DOCKER_INFLUXDB_INIT_ORG: ${MF_INFLUXDB_ORG}
|
||||
DOCKER_INFLUXDB_INIT_BUCKET: ${MF_INFLUXDB_BUCKET}
|
||||
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: ${MF_INFLUXDB_TOKEN}
|
||||
INFLUXDB_HTTP_FLUX_ENABLED: ${MF_INFLUXDB_HTTP_ENABLED}
|
||||
networks:
|
||||
- docker_mainflux-base-net
|
||||
ports:
|
||||
@@ -45,9 +47,10 @@ services:
|
||||
MF_INFLUX_WRITER_PORT: ${MF_INFLUX_WRITER_PORT}
|
||||
MF_INFLUX_WRITER_BATCH_SIZE: ${MF_INFLUX_WRITER_BATCH_SIZE}
|
||||
MF_INFLUX_WRITER_BATCH_TIMEOUT: ${MF_INFLUX_WRITER_BATCH_TIMEOUT}
|
||||
MF_INFLUXDB_DB: ${MF_INFLUXDB_DB}
|
||||
MF_INFLUXDB_HOST: mainflux-influxdb
|
||||
MF_INFLUXDB_HOST: ${MF_INFLUXDB_HOST}
|
||||
MF_INFLUXDB_PORT: ${MF_INFLUXDB_PORT}
|
||||
MF_INFLUXDB_TIMEOUT: ${MF_INFLUXDB_TIMEOUT}
|
||||
MF_INFLUXDB_PROTOCOL: ${MF_INFLUXDB_PROTOCOL}
|
||||
MF_INFLUXDB_ADMIN_USER: ${MF_INFLUXDB_ADMIN_USER}
|
||||
MF_INFLUXDB_ADMIN_PASSWORD: ${MF_INFLUXDB_ADMIN_PASSWORD}
|
||||
ports:
|
||||
@@ -56,16 +59,3 @@ services:
|
||||
- docker_mainflux-base-net
|
||||
volumes:
|
||||
- ./config.toml:/config.toml
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:7.3.7
|
||||
container_name: mainflux-grafana
|
||||
depends_on:
|
||||
- influxdb
|
||||
restart: on-failure
|
||||
ports:
|
||||
- ${MF_INFLUX_WRITER_GRAFANA_PORT}:${MF_GRAFANA_PORT}
|
||||
networks:
|
||||
- docker_mainflux-base-net
|
||||
volumes:
|
||||
- mainflux-grafana-volume:/var/lib/grafana
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Copyright (c) Mainflux
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
# This docker-compose file contains optional InfluxDB, InfluxDB-writer and Grafana services
|
||||
# This docker-compose file contains optional InfluxDB and InfluxDB-writer services
|
||||
# for the Mainflux platform. Since this services are optional, this file is dependent on the
|
||||
# docker-compose.yml file from <project_root>/docker/. In order to run these services,
|
||||
# core services, as well as the network from the core composition, should be already running.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Copyright (c) Mainflux
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
# This docker-compose file contains optional InfluxDB, InfluxDB-writer and Grafana services
|
||||
# This docker-compose file contains optional InfluxDB and InfluxDB-writer services
|
||||
# for the Mainflux platform. Since this services are optional, this file is dependent on the
|
||||
# docker-compose.yml file from <project_root>/docker/. In order to run these services,
|
||||
# core services, as well as the network from the core composition, should be already running.
|
||||
|
||||
@@ -21,7 +21,7 @@ require (
|
||||
github.com/gorilla/websocket v1.5.0
|
||||
github.com/hashicorp/vault/api v1.8.1
|
||||
github.com/hokaccha/go-prettyjson v0.0.0-20211117102719-0474bc63780f
|
||||
github.com/influxdata/influxdb v1.10.0
|
||||
github.com/influxdata/influxdb-client-go/v2 v2.12.2
|
||||
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa
|
||||
github.com/jackc/pgtype v1.13.0
|
||||
github.com/jackc/pgx/v5 v5.2.0
|
||||
@@ -65,6 +65,7 @@ require (
|
||||
github.com/cespare/xxhash/v2 v2.1.2 // indirect
|
||||
github.com/containerd/continuity v0.3.0 // indirect
|
||||
github.com/davecgh/go-spew v1.1.1 // indirect
|
||||
github.com/deepmap/oapi-codegen v1.8.2 // indirect
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
|
||||
github.com/docker/cli v20.10.21+incompatible // indirect
|
||||
github.com/docker/go-connections v0.4.0 // indirect
|
||||
@@ -98,6 +99,7 @@ require (
|
||||
github.com/hashicorp/yamux v0.1.1 // indirect
|
||||
github.com/imdario/mergo v0.3.13 // indirect
|
||||
github.com/inconshreveable/mousetrap v1.0.1 // indirect
|
||||
github.com/influxdata/line-protocol v0.0.0-20200327222509-2487e7298839 // indirect
|
||||
github.com/jackc/pgio v1.0.0 // indirect
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b // indirect
|
||||
|
||||
@@ -113,10 +113,14 @@ github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46t
|
||||
github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY=
|
||||
github.com/creack/pty v1.1.11 h1:07n33Z8lZxZ2qwegKbObQohDhXDQxiMMz1NOUGYlesw=
|
||||
github.com/creack/pty v1.1.11/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
|
||||
github.com/cyberdelia/templates v0.0.0-20141128023046-ca7fffd4298c/go.mod h1:GyV+0YP4qX0UQ7r2MoYZ+AvYDp12OF5yg4q8rGnyNh4=
|
||||
github.com/cyphar/filepath-securejoin v0.2.3/go.mod h1:aPGpWjXOXUn2NCNjFvBE6aRxGGx79pTxQpKOJNYHHl4=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/deepmap/oapi-codegen v1.6.0/go.mod h1:ryDa9AgbELGeB+YEXE1dR53yAjHwFvE9iAUlWl9Al3M=
|
||||
github.com/deepmap/oapi-codegen v1.8.2 h1:SegyeYGcdi0jLLrpbCMoJxnUUn8GBXHsvr4rbzjuhfU=
|
||||
github.com/deepmap/oapi-codegen v1.8.2/go.mod h1:YLgSKSDv/bZQB7N4ws6luhozi3cEdRktEqrX88CvjIw=
|
||||
github.com/denisenkom/go-mssqldb v0.9.0/go.mod h1:xbL0rPBG9cCiLr28tMa8zpbdarY27NDyej4t/EjAShU=
|
||||
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
|
||||
@@ -157,8 +161,11 @@ github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbS
|
||||
github.com/fxamacker/cbor/v2 v2.2.0/go.mod h1:TA1xS00nchWmaBnEIxPSE5oHLuJBAVvqrtAnWBwBCVo=
|
||||
github.com/fxamacker/cbor/v2 v2.4.0 h1:ri0ArlOR+5XunOP8CRUowT0pSJOwhW098ZCUyskZD88=
|
||||
github.com/fxamacker/cbor/v2 v2.4.0/go.mod h1:TA1xS00nchWmaBnEIxPSE5oHLuJBAVvqrtAnWBwBCVo=
|
||||
github.com/getkin/kin-openapi v0.53.0/go.mod h1:7Yn5whZr5kJi6t+kShccXS8ae1APpYTW6yheSwk8Yi4=
|
||||
github.com/getkin/kin-openapi v0.61.0/go.mod h1:7Yn5whZr5kJi6t+kShccXS8ae1APpYTW6yheSwk8Yi4=
|
||||
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
|
||||
github.com/go-acme/lego v2.7.2+incompatible/go.mod h1:yzMNe9CasVUhkquNvti5nAtPmG94USbYxYrZfTkIn0M=
|
||||
github.com/go-chi/chi/v5 v5.0.0/go.mod h1:BBug9lr0cqtdAhsu6R4AAdvufI0/XBzAQSsUqJpoZOs=
|
||||
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
|
||||
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
|
||||
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
|
||||
@@ -180,6 +187,8 @@ github.com/go-logfmt/logfmt v0.5.1 h1:otpy5pqBCBZ1ng9RQ0dPu4PN7ba75Y/aA+UpowDyNV
|
||||
github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs=
|
||||
github.com/go-ocf/go-coap/v2 v2.0.4-0.20200728125043-f38b86f047a7/go.mod h1:X9wVKcaOSx7wBxKcvrWgMQq1R2DNeA7NBLW2osIb8TM=
|
||||
github.com/go-ocf/kit v0.0.0-20200728130040-4aebdb6982bc/go.mod h1:TIsoMT/iB7t9P6ahkcOnsmvS83SIJsv9qXRfz/yLf6M=
|
||||
github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
|
||||
github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
|
||||
github.com/go-redis/redis/v8 v8.11.5 h1:AcZZR7igkdvfVmQTPnu9WE37LRrO/YrBH5zWyjDC0oI=
|
||||
github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq8Jd4h5lpwo=
|
||||
github.com/go-sql-driver/mysql v1.4.1/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w=
|
||||
@@ -247,6 +256,7 @@ github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEW
|
||||
github.com/golang/snappy v0.0.3/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
|
||||
github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=
|
||||
github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
|
||||
github.com/golangci/lint-1 v0.0.0-20181222135242-d2cdd8c08219/go.mod h1:/X8TswGSh1pIozq4ZwCfxS0WA5JGXguxk94ar/4c87Y=
|
||||
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
|
||||
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
|
||||
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
|
||||
@@ -289,6 +299,7 @@ github.com/gopcua/opcua v0.1.6 h1:B9SVRKQGzcWcwP2QPYN93Uku32+3wL+v5cgzBxE6V5I=
|
||||
github.com/gopcua/opcua v0.1.6/go.mod h1:INwnDoRxmNWAt7+tzqxuGqQkSF2c1C69VAL0c2q6AcY=
|
||||
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
|
||||
github.com/gorilla/mux v1.7.4/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So=
|
||||
github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So=
|
||||
github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
|
||||
github.com/gorilla/websocket v1.5.0 h1:PPwGk2jz7EePpoHN/+ClbZu8SPxiqlu12wZP/3sWmnc=
|
||||
github.com/gorilla/websocket v1.5.0/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
|
||||
@@ -369,8 +380,12 @@ github.com/imdario/mergo v0.3.13/go.mod h1:4lJ1jqUDcsbIECGy0RUJAXNIhg+6ocWgb1ALK
|
||||
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
|
||||
github.com/inconshreveable/mousetrap v1.0.1 h1:U3uMjPSQEBMNp1lFxmllqCPM6P5u/Xq7Pgzkat/bFNc=
|
||||
github.com/inconshreveable/mousetrap v1.0.1/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
|
||||
github.com/influxdata/influxdb v1.10.0 h1:8xDpt8KO3lzrzf/ss+l8r42AGUZvoITu5824berK7SE=
|
||||
github.com/influxdata/influxdb v1.10.0/go.mod h1:IVPuoA2pOOxau/NguX7ipW0Jp9Bn+dMWlo0+VOscevU=
|
||||
github.com/influxdata/influxdb-client-go/v2 v2.3.1-0.20210518120617-5d1fff431040 h1:MBLCfcSsUyFPDJp6T7EoHp/Ph3Jkrm4EuUKLD2rUWHg=
|
||||
github.com/influxdata/influxdb-client-go/v2 v2.3.1-0.20210518120617-5d1fff431040/go.mod h1:vLNHdxTJkIf2mSLvGrpj8TCcISApPoXkaxP8g9uRlW8=
|
||||
github.com/influxdata/influxdb-client-go/v2 v2.12.2 h1:uYABKdrEKlYm+++qfKdbgaHKBPmoWR5wpbmj6MBB/2g=
|
||||
github.com/influxdata/influxdb-client-go/v2 v2.12.2/go.mod h1:YteV91FiQxRdccyJ2cHvj2f/5sq4y4Njqu1fQzsQCOU=
|
||||
github.com/influxdata/line-protocol v0.0.0-20200327222509-2487e7298839 h1:W9WBk7wlPfJLvMCdtV4zPulc4uCPrlywQOmbFOhgQNU=
|
||||
github.com/influxdata/line-protocol v0.0.0-20200327222509-2487e7298839/go.mod h1:xaLFMmpvUxqXtVkUJfg9QmT88cDaCJ3ZKgdZ78oO8Qo=
|
||||
github.com/jackc/chunkreader v1.0.0 h1:4s39bBR8ByfqH+DKm8rQA3E1LHZWB9XWcrz8fqaZbe0=
|
||||
github.com/jackc/chunkreader v1.0.0/go.mod h1:RT6O25fNZIuasFJRyZ4R/Y2BbhasbmZXF9QQ7T3kePo=
|
||||
github.com/jackc/chunkreader/v2 v2.0.0/go.mod h1:odVSm741yZoC3dpHEUXIqA9tQRhFrgOHwnPIn9lDKlk=
|
||||
@@ -458,6 +473,8 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
|
||||
github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
|
||||
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
|
||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||
github.com/labstack/echo/v4 v4.2.1/go.mod h1:AA49e0DZ8kk5jTOOCKNuPR6oTnBS0dYiM4FW1e6jwpg=
|
||||
github.com/labstack/gommon v0.3.0/go.mod h1:MULnywXg0yavhxWKc+lOruYdAhDwPK9wf0OL7NoOu+k=
|
||||
github.com/lestrrat-go/iter v0.0.0-20200422075355-fc1769541911/go.mod h1:zIdgO1mRKhn8l9vrZJZz9TUMMFbQbLeTsbqPDrJ/OJc=
|
||||
github.com/lestrrat-go/jwx v1.0.2/go.mod h1:TPF17WiSFegZo+c20fdpw49QD+/7n4/IsGvEmCSWwT0=
|
||||
github.com/lestrrat-go/pdebug v0.0.0-20200204225717-4d6bd78da58d/go.mod h1:B06CSso/AWxiPejj+fheUINGeBKeeEZNt8w+EoU7+L8=
|
||||
@@ -470,6 +487,8 @@ github.com/lib/pq v1.10.7 h1:p7ZhMD+KsSRozJr34udlUrhboJwWAgCg34+/ZZNvZZw=
|
||||
github.com/magiconair/properties v1.8.5/go.mod h1:y3VJvCyxH9uVvJTWEGAELF3aiYNyPKd5NZ3oSwXrF60=
|
||||
github.com/magiconair/properties v1.8.6 h1:5ibWZ6iY0NctNGWo87LalDlEZ6R41TqbbDamhfG/Qzo=
|
||||
github.com/magiconair/properties v1.8.6/go.mod h1:y3VJvCyxH9uVvJTWEGAELF3aiYNyPKd5NZ3oSwXrF60=
|
||||
github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
|
||||
github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
|
||||
github.com/mainflux/mproxy v0.2.3 h1:tKc3i+/fNLUb2qwLVDinAJmtW7SMHAE9cgoueJmYsIU=
|
||||
github.com/mainflux/mproxy v0.2.3/go.mod h1:qJGrcNHt2e63IEIXowgHyRqgjIqsgOoU0CjAUTOUvII=
|
||||
github.com/mainflux/senml v1.5.0 h1:GAd1y1eMohfa6sVYcr2iQfVfkkh9l/q7B1TWF5L68xs=
|
||||
@@ -480,9 +499,13 @@ github.com/markbates/oncer v1.0.0 h1:E83IaVAHygyndzPimgUYJjbshhDTALZyXxvk9FOlQRY
|
||||
github.com/markbates/oncer v1.0.0/go.mod h1:Z59JA581E9GP6w96jai+TGqafHPW+cPfRxz2aSZ0mcI=
|
||||
github.com/markbates/safe v1.0.1 h1:yjZkbvRM6IzKj9tlu/zMJLS0n/V351OZWRnF3QfaUxI=
|
||||
github.com/markbates/safe v1.0.1/go.mod h1:nAqgmRi7cY2nqMc92/bSEeQA+R4OheNU2T1kNSCBdG0=
|
||||
github.com/matryer/moq v0.0.0-20190312154309-6cfb0558e1bd/go.mod h1:9ELz6aaclSIGnZBoaSLZ3NAl1VTufbOrXBPvtcy6WiQ=
|
||||
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
|
||||
github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ=
|
||||
github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
|
||||
github.com/mattn/go-colorable v0.1.6/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
|
||||
github.com/mattn/go-colorable v0.1.7/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
|
||||
github.com/mattn/go-colorable v0.1.8/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
|
||||
github.com/mattn/go-colorable v0.1.9/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
|
||||
github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4=
|
||||
github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
|
||||
@@ -490,6 +513,8 @@ github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovk
|
||||
github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
|
||||
github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
|
||||
github.com/mattn/go-isatty v0.0.7/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
|
||||
github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
|
||||
github.com/mattn/go-isatty v0.0.9/go.mod h1:YNRxwqDuOph6SZLI9vUUz6OYw3QyUt7WiY2yME+cCiQ=
|
||||
github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
|
||||
github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94=
|
||||
github.com/mattn/go-isatty v0.0.16 h1:bq3VjFmv/sOjHtdEhmkEV4x1AJtvUvOJ2PFAZ5+peKQ=
|
||||
@@ -731,6 +756,8 @@ github.com/urfave/cli v1.22.1/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtX
|
||||
github.com/urfave/cli v1.22.5/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
|
||||
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
|
||||
github.com/valyala/fasthttp v1.12.0/go.mod h1:229t1eWu9UXTPmoUkbpN/fctKPBY4IJoFXQnxHGXy6E=
|
||||
github.com/valyala/fasttemplate v1.0.1/go.mod h1:UQGH1tvbgY+Nz5t2n7tXsz52dQxojPUpymEIMZ47gx8=
|
||||
github.com/valyala/fasttemplate v1.2.1/go.mod h1:KHLXt3tVN2HBp8eijSv/kGJopbvo7S+qRAEEKiv+SiQ=
|
||||
github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a/go.mod h1:v3UYOV9WzVtRmSR+PDvWpU/qWl4Wa5LApYYX4ZtKbio=
|
||||
github.com/vishvananda/netlink v1.1.0/go.mod h1:cTgwzPIzzgDAYoQrMm0EdrjRUBkTqKYppBueQtXaqoE=
|
||||
github.com/vishvananda/netns v0.0.0-20191106174202-0a2b9b5464df/go.mod h1:JP3t17pCcGlemwknint6hfoeCVQrEMVwxRLRjXpq+BU=
|
||||
@@ -808,6 +835,7 @@ golang.org/x/crypto v0.0.0-20200510223506-06a226fb4e37/go.mod h1:LzIPMQfyMNhhGPh
|
||||
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||
golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||
golang.org/x/crypto v0.0.0-20201203163018-be400aefbc4c/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
|
||||
golang.org/x/crypto v0.0.0-20201221181555-eec23a3978ad/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
|
||||
golang.org/x/crypto v0.0.0-20210314154223-e6e6c4f2bb5b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
|
||||
golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
|
||||
golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
|
||||
@@ -981,6 +1009,7 @@ golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7w
|
||||
golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20200826173525-f9321e4c35a6/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
@@ -1042,6 +1071,7 @@ golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
|
||||
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/time v0.0.0-20201208040808-7e3f01d25324/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/time v0.1.0 h1:xYY+Bajn2a7VBmTM5GikTmnK8ZuX8YgnQCqZpbBNtmA=
|
||||
golang.org/x/time v0.1.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
|
||||
@@ -18,7 +18,6 @@ func sendMessageEndpoint(svc http.Service) endpoint.Endpoint {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
err := svc.Publish(ctx, req.token, req.msg)
|
||||
return nil, err
|
||||
return nil, svc.Publish(ctx, req.token, req.msg)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
package influxdb
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/influxdata/influxdb/client/v2"
|
||||
influxdb2 "github.com/influxdata/influxdb-client-go/v2"
|
||||
"github.com/mainflux/mainflux/internal/env"
|
||||
"github.com/mainflux/mainflux/pkg/errors"
|
||||
)
|
||||
@@ -15,38 +15,31 @@ var (
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
Protocol string `env:"PROTOCOL" envDefault:"http"`
|
||||
Host string `env:"HOST" envDefault:"localhost"`
|
||||
Port string `env:"PORT" envDefault:"8086"`
|
||||
Username string `env:"ADMIN_USER" envDefault:"mainflux"`
|
||||
Password string `env:"ADMIN_PASSWORD" envDefault:"mainflux"`
|
||||
DbName string `env:"DB" envDefault:"mainflux"`
|
||||
UserAgent string `env:"USER_AGENT" envDefault:"InfluxDBClient"`
|
||||
Timeout time.Duration `env:"TIMEOUT"` // Influxdb client configuration by default there is no timeout duration , this field will not have fallback default timeout duration Reference: https://pkg.go.dev/github.com/influxdata/influxdb@v1.10.0/client/v2#HTTPConfig
|
||||
InsecureSkipVerify bool `env:"INSECURE_SKIP_VERIFY" envDefault:"false"`
|
||||
Protocol string `env:"PROTOCOL" envDefault:"http"`
|
||||
Host string `env:"HOST" envDefault:"localhost"`
|
||||
Port string `env:"PORT" envDefault:"8086"`
|
||||
Bucket string `env:"BUCKET" envDefault:"mainflux-bucket"`
|
||||
Org string `env:"ORG" envDefault:"mainflux"`
|
||||
Token string `env:"TOKEN" envDefault:"mainflux-token"`
|
||||
DBUrl string `env:"DBURL" envDefault:""`
|
||||
Timeout time.Duration `env:"TIMEOUT" envDefault:"1s"`
|
||||
}
|
||||
|
||||
// Setup load configuration from environment variable, create InfluxDB client and connect to InfluxDB server
|
||||
func Setup(envPrefix string) (client.HTTPClient, error) {
|
||||
func Setup(envPrefix string, ctx context.Context) (influxdb2.Client, error) {
|
||||
config := Config{}
|
||||
if err := env.Parse(&config, env.Options{Prefix: envPrefix}); err != nil {
|
||||
return nil, errors.Wrap(errConfig, err)
|
||||
}
|
||||
return Connect(config)
|
||||
return Connect(config, ctx)
|
||||
}
|
||||
|
||||
// Connect create InfluxDB client and connect to InfluxDB server
|
||||
func Connect(config Config) (client.HTTPClient, error) {
|
||||
address := fmt.Sprintf("%s://%s:%s", config.Protocol, config.Host, config.Port)
|
||||
clientConfig := client.HTTPConfig{
|
||||
Addr: address,
|
||||
Username: config.Username,
|
||||
Password: config.Password,
|
||||
UserAgent: config.UserAgent,
|
||||
Timeout: config.Timeout,
|
||||
}
|
||||
client, err := client.NewHTTPClient(clientConfig)
|
||||
if err != nil {
|
||||
func Connect(config Config, ctx context.Context) (influxdb2.Client, error) {
|
||||
client := influxdb2.NewClient(config.DBUrl, config.Token)
|
||||
ctx, cancel := context.WithTimeout(ctx, config.Timeout)
|
||||
defer cancel()
|
||||
if _, err := client.Ready(ctx); err != nil {
|
||||
return nil, errors.Wrap(errConnect, err)
|
||||
}
|
||||
return client, nil
|
||||
|
||||
@@ -1,771 +0,0 @@
|
||||
{
|
||||
"__inputs": [
|
||||
{
|
||||
"name": "DS_MAINFLUX",
|
||||
"label": "mainflux",
|
||||
"description": "",
|
||||
"type": "datasource",
|
||||
"pluginId": "prometheus",
|
||||
"pluginName": "Prometheus"
|
||||
}
|
||||
],
|
||||
"__requires": [
|
||||
{
|
||||
"type": "grafana",
|
||||
"id": "grafana",
|
||||
"name": "Grafana",
|
||||
"version": "5.1.3"
|
||||
},
|
||||
{
|
||||
"type": "panel",
|
||||
"id": "graph",
|
||||
"name": "Graph",
|
||||
"version": "5.0.0"
|
||||
},
|
||||
{
|
||||
"type": "datasource",
|
||||
"id": "prometheus",
|
||||
"name": "Prometheus",
|
||||
"version": "5.0.0"
|
||||
}
|
||||
],
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": "-- Grafana --",
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"gnetId": null,
|
||||
"graphTooltip": 0,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"aliasColors": {},
|
||||
"bars": false,
|
||||
"dashLength": 10,
|
||||
"dashes": false,
|
||||
"datasource": "${DS_MAINFLUX}",
|
||||
"fill": 1,
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 10,
|
||||
"legend": {
|
||||
"avg": false,
|
||||
"current": false,
|
||||
"max": false,
|
||||
"min": false,
|
||||
"show": true,
|
||||
"total": false,
|
||||
"values": false
|
||||
},
|
||||
"lines": true,
|
||||
"linewidth": 1,
|
||||
"links": [],
|
||||
"nullPointMode": "null",
|
||||
"percentage": false,
|
||||
"pointradius": 5,
|
||||
"points": false,
|
||||
"renderer": "flot",
|
||||
"seriesOverrides": [],
|
||||
"spaceLength": 10,
|
||||
"stack": false,
|
||||
"steppedLine": false,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(ws_adapter_api_request_count{job=\"mainflux\"}[30s])",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "ws_count_{{method}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"thresholds": [],
|
||||
"timeFrom": null,
|
||||
"timeShift": null,
|
||||
"title": "WebSocket adapter request count",
|
||||
"tooltip": {
|
||||
"shared": true,
|
||||
"sort": 0,
|
||||
"value_type": "individual"
|
||||
},
|
||||
"type": "graph",
|
||||
"xaxis": {
|
||||
"buckets": null,
|
||||
"mode": "time",
|
||||
"name": null,
|
||||
"show": true,
|
||||
"values": []
|
||||
},
|
||||
"yaxes": [
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
},
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
}
|
||||
],
|
||||
"yaxis": {
|
||||
"align": false,
|
||||
"alignLevel": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"aliasColors": {},
|
||||
"bars": false,
|
||||
"dashLength": 10,
|
||||
"dashes": false,
|
||||
"datasource": "${DS_MAINFLUX}",
|
||||
"fill": 1,
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 18,
|
||||
"legend": {
|
||||
"avg": false,
|
||||
"current": false,
|
||||
"max": false,
|
||||
"min": false,
|
||||
"show": true,
|
||||
"total": false,
|
||||
"values": false
|
||||
},
|
||||
"lines": true,
|
||||
"linewidth": 1,
|
||||
"links": [],
|
||||
"nullPointMode": "null",
|
||||
"percentage": false,
|
||||
"pointradius": 5,
|
||||
"points": false,
|
||||
"renderer": "flot",
|
||||
"seriesOverrides": [],
|
||||
"spaceLength": 10,
|
||||
"stack": false,
|
||||
"steppedLine": false,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "avg(avg_over_time(ws_adapter_api_request_latency_microseconds{job=\"mainflux\"}[30s])) by (method)",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "ws_latency_{{method}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"thresholds": [],
|
||||
"timeFrom": null,
|
||||
"timeShift": null,
|
||||
"title": "WebSocket adapter request latency",
|
||||
"tooltip": {
|
||||
"shared": true,
|
||||
"sort": 0,
|
||||
"value_type": "individual"
|
||||
},
|
||||
"type": "graph",
|
||||
"xaxis": {
|
||||
"buckets": null,
|
||||
"mode": "time",
|
||||
"name": null,
|
||||
"show": true,
|
||||
"values": []
|
||||
},
|
||||
"yaxes": [
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
},
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
}
|
||||
],
|
||||
"yaxis": {
|
||||
"align": false,
|
||||
"alignLevel": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"aliasColors": {},
|
||||
"bars": false,
|
||||
"dashLength": 10,
|
||||
"dashes": false,
|
||||
"datasource": "${DS_MAINFLUX}",
|
||||
"fill": 1,
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 9
|
||||
},
|
||||
"id": 8,
|
||||
"legend": {
|
||||
"avg": false,
|
||||
"current": false,
|
||||
"max": false,
|
||||
"min": false,
|
||||
"show": true,
|
||||
"total": false,
|
||||
"values": false
|
||||
},
|
||||
"lines": true,
|
||||
"linewidth": 1,
|
||||
"links": [],
|
||||
"nullPointMode": "null",
|
||||
"percentage": false,
|
||||
"pointradius": 5,
|
||||
"points": false,
|
||||
"renderer": "flot",
|
||||
"seriesOverrides": [],
|
||||
"spaceLength": 10,
|
||||
"stack": false,
|
||||
"steppedLine": false,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(http_adapter_api_request_count{job=\"mainflux\"}[30s])",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "http_count_{{method}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"thresholds": [],
|
||||
"timeFrom": null,
|
||||
"timeShift": null,
|
||||
"title": "HTTP adapter request count",
|
||||
"tooltip": {
|
||||
"shared": true,
|
||||
"sort": 0,
|
||||
"value_type": "individual"
|
||||
},
|
||||
"type": "graph",
|
||||
"xaxis": {
|
||||
"buckets": null,
|
||||
"mode": "time",
|
||||
"name": null,
|
||||
"show": true,
|
||||
"values": []
|
||||
},
|
||||
"yaxes": [
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
},
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
}
|
||||
],
|
||||
"yaxis": {
|
||||
"align": false,
|
||||
"alignLevel": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"aliasColors": {},
|
||||
"bars": false,
|
||||
"dashLength": 10,
|
||||
"dashes": false,
|
||||
"datasource": "${DS_MAINFLUX}",
|
||||
"fill": 1,
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 9
|
||||
},
|
||||
"id": 16,
|
||||
"legend": {
|
||||
"avg": false,
|
||||
"current": false,
|
||||
"max": false,
|
||||
"min": false,
|
||||
"show": true,
|
||||
"total": false,
|
||||
"values": false
|
||||
},
|
||||
"lines": true,
|
||||
"linewidth": 1,
|
||||
"links": [],
|
||||
"nullPointMode": "null",
|
||||
"percentage": false,
|
||||
"pointradius": 5,
|
||||
"points": false,
|
||||
"renderer": "flot",
|
||||
"seriesOverrides": [],
|
||||
"spaceLength": 10,
|
||||
"stack": false,
|
||||
"steppedLine": false,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "avg(avg_over_time(http_adapter_api_request_latency_microseconds{job=\"mainflux\"}[30s])) by (method)",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "http_latency_{{method}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"thresholds": [],
|
||||
"timeFrom": null,
|
||||
"timeShift": null,
|
||||
"title": "HTTP adapter request latency",
|
||||
"tooltip": {
|
||||
"shared": true,
|
||||
"sort": 0,
|
||||
"value_type": "individual"
|
||||
},
|
||||
"type": "graph",
|
||||
"xaxis": {
|
||||
"buckets": null,
|
||||
"mode": "time",
|
||||
"name": null,
|
||||
"show": true,
|
||||
"values": []
|
||||
},
|
||||
"yaxes": [
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
},
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
}
|
||||
],
|
||||
"yaxis": {
|
||||
"align": false,
|
||||
"alignLevel": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"aliasColors": {},
|
||||
"bars": false,
|
||||
"dashLength": 10,
|
||||
"dashes": false,
|
||||
"datasource": "${DS_MAINFLUX}",
|
||||
"fill": 1,
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 18
|
||||
},
|
||||
"id": 4,
|
||||
"legend": {
|
||||
"avg": false,
|
||||
"current": false,
|
||||
"max": false,
|
||||
"min": false,
|
||||
"show": true,
|
||||
"total": false,
|
||||
"values": false
|
||||
},
|
||||
"lines": true,
|
||||
"linewidth": 1,
|
||||
"links": [],
|
||||
"nullPointMode": "null",
|
||||
"percentage": false,
|
||||
"pointradius": 5,
|
||||
"points": false,
|
||||
"renderer": "flot",
|
||||
"seriesOverrides": [],
|
||||
"spaceLength": 10,
|
||||
"stack": false,
|
||||
"steppedLine": false,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(things_api_request_count{job=\"mainflux\"}[30s])",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "things_count_{{method}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"thresholds": [],
|
||||
"timeFrom": null,
|
||||
"timeShift": null,
|
||||
"title": "Things request count",
|
||||
"tooltip": {
|
||||
"shared": true,
|
||||
"sort": 0,
|
||||
"value_type": "individual"
|
||||
},
|
||||
"type": "graph",
|
||||
"xaxis": {
|
||||
"buckets": null,
|
||||
"mode": "time",
|
||||
"name": null,
|
||||
"show": true,
|
||||
"values": []
|
||||
},
|
||||
"yaxes": [
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
},
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
}
|
||||
],
|
||||
"yaxis": {
|
||||
"align": false,
|
||||
"alignLevel": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"aliasColors": {},
|
||||
"bars": false,
|
||||
"dashLength": 10,
|
||||
"dashes": false,
|
||||
"datasource": "${DS_MAINFLUX}",
|
||||
"fill": 1,
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 18
|
||||
},
|
||||
"id": 14,
|
||||
"legend": {
|
||||
"avg": false,
|
||||
"current": false,
|
||||
"max": false,
|
||||
"min": false,
|
||||
"show": true,
|
||||
"total": false,
|
||||
"values": false
|
||||
},
|
||||
"lines": true,
|
||||
"linewidth": 1,
|
||||
"links": [],
|
||||
"nullPointMode": "null",
|
||||
"percentage": false,
|
||||
"pointradius": 5,
|
||||
"points": false,
|
||||
"renderer": "flot",
|
||||
"seriesOverrides": [],
|
||||
"spaceLength": 10,
|
||||
"stack": false,
|
||||
"steppedLine": false,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "avg(avg_over_time(things_api_request_latency_microseconds{job=\"mainflux\"}[30s])) by (method)",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "things_latency_{{method}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"thresholds": [],
|
||||
"timeFrom": null,
|
||||
"timeShift": null,
|
||||
"title": "Things request latency",
|
||||
"tooltip": {
|
||||
"shared": true,
|
||||
"sort": 0,
|
||||
"value_type": "individual"
|
||||
},
|
||||
"type": "graph",
|
||||
"xaxis": {
|
||||
"buckets": null,
|
||||
"mode": "time",
|
||||
"name": null,
|
||||
"show": true,
|
||||
"values": []
|
||||
},
|
||||
"yaxes": [
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
},
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
}
|
||||
],
|
||||
"yaxis": {
|
||||
"align": false,
|
||||
"alignLevel": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"aliasColors": {
|
||||
"users_count": "#7eb26d"
|
||||
},
|
||||
"bars": false,
|
||||
"dashLength": 10,
|
||||
"dashes": false,
|
||||
"datasource": "${DS_MAINFLUX}",
|
||||
"fill": 1,
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 27
|
||||
},
|
||||
"id": 2,
|
||||
"legend": {
|
||||
"avg": false,
|
||||
"current": false,
|
||||
"max": false,
|
||||
"min": false,
|
||||
"show": true,
|
||||
"total": false,
|
||||
"values": false
|
||||
},
|
||||
"lines": true,
|
||||
"linewidth": 1,
|
||||
"links": [],
|
||||
"nullPointMode": "null",
|
||||
"percentage": false,
|
||||
"pointradius": 5,
|
||||
"points": false,
|
||||
"renderer": "flot",
|
||||
"seriesOverrides": [
|
||||
{
|
||||
"alias": "users_count",
|
||||
"yaxis": 1
|
||||
}
|
||||
],
|
||||
"spaceLength": 10,
|
||||
"stack": false,
|
||||
"steppedLine": false,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(users_api_request_count{job=\"mainflux\"}[30s])",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "users_count_{{method}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"thresholds": [],
|
||||
"timeFrom": null,
|
||||
"timeShift": null,
|
||||
"title": "Users request count",
|
||||
"tooltip": {
|
||||
"shared": true,
|
||||
"sort": 0,
|
||||
"value_type": "individual"
|
||||
},
|
||||
"type": "graph",
|
||||
"xaxis": {
|
||||
"buckets": null,
|
||||
"mode": "time",
|
||||
"name": null,
|
||||
"show": true,
|
||||
"values": []
|
||||
},
|
||||
"yaxes": [
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
},
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
}
|
||||
],
|
||||
"yaxis": {
|
||||
"align": false,
|
||||
"alignLevel": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"aliasColors": {},
|
||||
"bars": false,
|
||||
"dashLength": 10,
|
||||
"dashes": false,
|
||||
"datasource": "${DS_MAINFLUX}",
|
||||
"fill": 1,
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 27
|
||||
},
|
||||
"id": 12,
|
||||
"legend": {
|
||||
"avg": false,
|
||||
"current": false,
|
||||
"max": false,
|
||||
"min": false,
|
||||
"show": true,
|
||||
"total": false,
|
||||
"values": false
|
||||
},
|
||||
"lines": true,
|
||||
"linewidth": 1,
|
||||
"links": [],
|
||||
"nullPointMode": "null",
|
||||
"percentage": false,
|
||||
"pointradius": 5,
|
||||
"points": false,
|
||||
"renderer": "flot",
|
||||
"seriesOverrides": [],
|
||||
"spaceLength": 10,
|
||||
"stack": false,
|
||||
"steppedLine": false,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "avg(avg_over_time(users_api_request_latency_microseconds{job=\"mainflux\"}[30s])) by (method)",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "users_latency_{{method}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"thresholds": [],
|
||||
"timeFrom": null,
|
||||
"timeShift": null,
|
||||
"title": "Users request latency",
|
||||
"tooltip": {
|
||||
"shared": true,
|
||||
"sort": 0,
|
||||
"value_type": "individual"
|
||||
},
|
||||
"type": "graph",
|
||||
"xaxis": {
|
||||
"buckets": null,
|
||||
"mode": "time",
|
||||
"name": null,
|
||||
"show": true,
|
||||
"values": []
|
||||
},
|
||||
"yaxes": [
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
},
|
||||
{
|
||||
"format": "short",
|
||||
"label": null,
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
}
|
||||
],
|
||||
"yaxis": {
|
||||
"align": false,
|
||||
"alignLevel": null
|
||||
}
|
||||
}
|
||||
],
|
||||
"refresh": "5s",
|
||||
"schemaVersion": 16,
|
||||
"style": "dark",
|
||||
"tags": [],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-5m",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {
|
||||
"refresh_intervals": [
|
||||
"5s",
|
||||
"10s",
|
||||
"30s",
|
||||
"1m",
|
||||
"5m",
|
||||
"15m",
|
||||
"30m",
|
||||
"1h",
|
||||
"2h",
|
||||
"1d"
|
||||
],
|
||||
"time_options": [
|
||||
"5m",
|
||||
"15m",
|
||||
"1h",
|
||||
"6h",
|
||||
"12h",
|
||||
"24h",
|
||||
"2d",
|
||||
"7d",
|
||||
"30d"
|
||||
]
|
||||
},
|
||||
"timezone": "",
|
||||
"title": "MainfluxDashboard",
|
||||
"uid": "NOIbtxImk",
|
||||
"version": 29
|
||||
}
|
||||
@@ -30,10 +30,6 @@ func (req listMessagesReq) validate() error {
|
||||
return apiutil.ErrLimitSize
|
||||
}
|
||||
|
||||
if req.pageMeta.Offset < 0 {
|
||||
return apiutil.ErrOffsetSize
|
||||
}
|
||||
|
||||
if req.pageMeta.Comparator != "" &&
|
||||
req.pageMeta.Comparator != readers.EqualKey &&
|
||||
req.pageMeta.Comparator != readers.LowerThanKey &&
|
||||
|
||||
@@ -50,20 +50,22 @@ func TestReadSenml(t *testing.T) {
|
||||
Hosts: []string{addr},
|
||||
Keyspace: keyspace,
|
||||
})
|
||||
require.Nil(t, err, fmt.Sprintf("failed to connect to Cassandra: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("failed to connect to Cassandra: %s", err))
|
||||
defer session.Close()
|
||||
err = casClient.InitDB(session, cwriter.Table)
|
||||
assert.Nil(t, err, fmt.Sprintf("failed to initialize to Cassandra: %s", err))
|
||||
err = casClient.InitDB(session, cwriter.Table)
|
||||
require.Nil(t, err, fmt.Sprintf("failed to initialize to Cassandra: %s", err))
|
||||
writer := cwriter.New(session)
|
||||
|
||||
chanID, err := idProvider.ID()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
pubID, err := idProvider.ID()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
pubID2, err := idProvider.ID()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
wrongID, err := idProvider.ID()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
|
||||
m := senml.Message{
|
||||
Channel: chanID,
|
||||
@@ -111,19 +113,21 @@ func TestReadSenml(t *testing.T) {
|
||||
}
|
||||
|
||||
err = writer.Consume(messages)
|
||||
require.Nil(t, err, fmt.Sprintf("failed to store message to Cassandra: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("failed to store message to Cassandra: %s", err))
|
||||
|
||||
reader := creader.New(session)
|
||||
|
||||
// Since messages are not saved in natural order,
|
||||
// cases that return subset of messages are only
|
||||
// checking data result set size, but not content.
|
||||
cases := map[string]struct {
|
||||
cases := []struct {
|
||||
desc string
|
||||
chanID string
|
||||
pageMeta readers.PageMetadata
|
||||
page readers.MessagesPage
|
||||
}{
|
||||
"read message page for existing channel": {
|
||||
{
|
||||
desc: "read message page for existing channel",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -134,7 +138,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(messages),
|
||||
},
|
||||
},
|
||||
"read message page for non-existent channel": {
|
||||
{
|
||||
desc: "read message page for non-existent channel",
|
||||
chanID: wrongID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -144,7 +149,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: []readers.Message{},
|
||||
},
|
||||
},
|
||||
"read message last page": {
|
||||
{
|
||||
desc: "read message last page",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: msgsNum - 20,
|
||||
@@ -155,7 +161,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(messages[msgsNum-20 : msgsNum]),
|
||||
},
|
||||
},
|
||||
"read message with non-existent subtopic": {
|
||||
{
|
||||
desc: "read message with non-existent subtopic",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -166,7 +173,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: []readers.Message{},
|
||||
},
|
||||
},
|
||||
"read message with subtopic": {
|
||||
{
|
||||
desc: "read message with subtopic",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -178,7 +186,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(queryMsgs),
|
||||
},
|
||||
},
|
||||
"read message with publisher": {
|
||||
{
|
||||
desc: "read message with publisher",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -190,7 +199,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(queryMsgs),
|
||||
},
|
||||
},
|
||||
"read message with wrong format": {
|
||||
{
|
||||
desc: "read message with wrong format",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Format: "messagess",
|
||||
@@ -203,7 +213,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: []readers.Message{},
|
||||
},
|
||||
},
|
||||
"read message with protocol": {
|
||||
{
|
||||
desc: "read message with protocol",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -215,7 +226,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(queryMsgs),
|
||||
},
|
||||
},
|
||||
"read message with name": {
|
||||
{
|
||||
desc: "read message with name",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -227,7 +239,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(queryMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with value": {
|
||||
{
|
||||
desc: "read message with value",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -239,7 +252,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(valueMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with value and equal comparator": {
|
||||
{
|
||||
desc: "read message with value and equal comparator",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -252,7 +266,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(valueMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with value and lower-than comparator": {
|
||||
{
|
||||
desc: "read message with value and lower-than comparator",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -265,7 +280,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(valueMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with value and lower-than-or-equal comparator": {
|
||||
{
|
||||
desc: "read message with value and lower-than-or-equal comparator",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -278,7 +294,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(valueMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with value and greater-than comparator": {
|
||||
{
|
||||
desc: "read message with value and greater-than comparator",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -291,7 +308,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(valueMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with value and greater-than-or-equal comparator": {
|
||||
{
|
||||
desc: "read message with value and greater-than-or-equal comparator",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -304,7 +322,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(valueMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with boolean value": {
|
||||
{
|
||||
desc: "read message with boolean value",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -316,7 +335,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(boolMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with string value": {
|
||||
{
|
||||
desc: "read message with string value",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -328,7 +348,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(stringMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with data value": {
|
||||
{
|
||||
desc: "read message with data value",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -340,7 +361,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(dataMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with from": {
|
||||
{
|
||||
desc: "read message with from",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -352,7 +374,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(messages[0:21]),
|
||||
},
|
||||
},
|
||||
"read message with to": {
|
||||
{
|
||||
desc: "read message with to",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -364,7 +387,8 @@ func TestReadSenml(t *testing.T) {
|
||||
Messages: fromSenml(messages[21:]),
|
||||
},
|
||||
},
|
||||
"read message with from/to": {
|
||||
{
|
||||
desc: "read message with from/to",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -379,11 +403,11 @@ func TestReadSenml(t *testing.T) {
|
||||
},
|
||||
}
|
||||
|
||||
for desc, tc := range cases {
|
||||
for _, tc := range cases {
|
||||
result, err := reader.ReadAll(tc.chanID, tc.pageMeta)
|
||||
assert.Nil(t, err, fmt.Sprintf("%s: expected no error got %s", desc, err))
|
||||
assert.ElementsMatch(t, tc.page.Messages, result.Messages, fmt.Sprintf("%s: expected %v got %v", desc, tc.page.Messages, result.Messages))
|
||||
assert.Equal(t, tc.page.Total, result.Total, fmt.Sprintf("%s: expected %v got %v", desc, tc.page.Total, result.Total))
|
||||
assert.Nil(t, err, fmt.Sprintf("%s: got unexpected error: %s", tc.desc, err))
|
||||
assert.ElementsMatch(t, tc.page.Messages, result.Messages, fmt.Sprintf("%s: expected %v got %v\n", tc.desc, tc.page.Messages, result.Messages))
|
||||
assert.Equal(t, tc.page.Total, result.Total, fmt.Sprintf("%s: expected %v got %v\n", tc.desc, tc.page.Total, result.Total))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -392,12 +416,12 @@ func TestReadJSON(t *testing.T) {
|
||||
Hosts: []string{addr},
|
||||
Keyspace: keyspace,
|
||||
})
|
||||
require.Nil(t, err, fmt.Sprintf("failed to connect to Cassandra: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("failed to connect to Cassandra: %s", err))
|
||||
defer session.Close()
|
||||
writer := cwriter.New(session)
|
||||
|
||||
id1, err := idProvider.ID()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
m := json.Message{
|
||||
Channel: id1,
|
||||
Publisher: id1,
|
||||
@@ -428,7 +452,7 @@ func TestReadJSON(t *testing.T) {
|
||||
assert.Nil(t, err, fmt.Sprintf("expected no error got %s\n", err))
|
||||
|
||||
id2, err := idProvider.ID()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
m = json.Message{
|
||||
Channel: id2,
|
||||
Publisher: id2,
|
||||
@@ -467,12 +491,14 @@ func TestReadJSON(t *testing.T) {
|
||||
|
||||
reader := creader.New(session)
|
||||
|
||||
cases := map[string]struct {
|
||||
cases := []struct {
|
||||
desc string
|
||||
chanID string
|
||||
pageMeta readers.PageMetadata
|
||||
page readers.MessagesPage
|
||||
}{
|
||||
"read message page for existing channel": {
|
||||
{
|
||||
desc: "read message page for existing channel",
|
||||
chanID: id1,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Format: messages1.Format,
|
||||
@@ -484,7 +510,8 @@ func TestReadJSON(t *testing.T) {
|
||||
Messages: fromJSON(msgs1[:10]),
|
||||
},
|
||||
},
|
||||
"read message page for non-existent channel": {
|
||||
{
|
||||
desc: "read message page for non-existent channel",
|
||||
chanID: wrongID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Format: messages1.Format,
|
||||
@@ -495,7 +522,8 @@ func TestReadJSON(t *testing.T) {
|
||||
Messages: []readers.Message{},
|
||||
},
|
||||
},
|
||||
"read message last page": {
|
||||
{
|
||||
desc: "read message last page",
|
||||
chanID: id2,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Format: messages2.Format,
|
||||
@@ -507,7 +535,8 @@ func TestReadJSON(t *testing.T) {
|
||||
Messages: fromJSON(msgs2[msgsNum-20 : msgsNum]),
|
||||
},
|
||||
},
|
||||
"read message with protocol": {
|
||||
{
|
||||
desc: "read message with protocol",
|
||||
chanID: id2,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Format: messages2.Format,
|
||||
@@ -522,7 +551,7 @@ func TestReadJSON(t *testing.T) {
|
||||
},
|
||||
}
|
||||
|
||||
for desc, tc := range cases {
|
||||
for _, tc := range cases {
|
||||
result, err := reader.ReadAll(tc.chanID, tc.pageMeta)
|
||||
for i := 0; i < len(result.Messages); i++ {
|
||||
m := result.Messages[i]
|
||||
@@ -530,9 +559,9 @@ func TestReadJSON(t *testing.T) {
|
||||
delete(m.(map[string]interface{}), "id")
|
||||
result.Messages[i] = m
|
||||
}
|
||||
assert.Nil(t, err, fmt.Sprintf("%s: expected no error got %s", desc, err))
|
||||
assert.ElementsMatch(t, tc.page.Messages, result.Messages, fmt.Sprintf("%s: expected %v got %v", desc, tc.page.Messages, result.Messages))
|
||||
assert.Equal(t, tc.page.Total, result.Total, fmt.Sprintf("%s: expected %v got %v", desc, tc.page.Total, result.Total))
|
||||
assert.Nil(t, err, fmt.Sprintf("%s: got unexpected error: %s", tc.desc, err))
|
||||
assert.ElementsMatch(t, tc.page.Messages, result.Messages, fmt.Sprintf("%s: expected %v got %v\n", tc.desc, tc.page.Messages, result.Messages))
|
||||
assert.Equal(t, tc.page.Total, result.Total, fmt.Sprintf("%s: expected %v got %v\n", tc.desc, tc.page.Total, result.Total))
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+40
-19
@@ -8,24 +8,32 @@ The service is configured using the environment variables presented in the
|
||||
following table. Note that any unset variables will be replaced with their
|
||||
default values.
|
||||
|
||||
| Variable | Description | Default |
|
||||
|------------------------------|-----------------------------------------------------|----------------|
|
||||
| MF_INFLUX_READER_LOG_LEVEL | Service log level | info |
|
||||
| MF_INFLUX_READER_PORT | Service HTTP port | 8180 |
|
||||
| MF_INFLUXDB_HOST | InfluxDB host | localhost |
|
||||
| MF_INFLUXDB_PORT | Default port of InfluxDB database | 8086 |
|
||||
| MF_INFLUXDB_ADMIN_USER | Default user of InfluxDB database | mainflux |
|
||||
| MF_INFLUXDB_ADMIN_PASSWORD | Default password of InfluxDB user | mainflux |
|
||||
| MF_INFLUXDB_DB | InfluxDB database name | mainflux |
|
||||
| MF_INFLUX_READER_CLIENT_TLS | Flag that indicates if TLS should be turned on | false |
|
||||
| MF_INFLUX_READER_CA_CERTS | Path to trusted CAs in PEM format | |
|
||||
| MF_INFLUX_READER_SERVER_CERT | Path to server certificate in pem format | |
|
||||
| MF_INFLUX_READER_SERVER_KEY | Path to server key in pem format | |
|
||||
| MF_JAEGER_URL | Jaeger server URL | localhost:6831 |
|
||||
| MF_THINGS_AUTH_GRPC_URL | Things service Auth gRPC URL | localhost:8183 |
|
||||
| MF_THINGS_AUTH_GRPC_TIMEOUT | Things service Auth gRPC request timeout in seconds | 1s |
|
||||
| MF_AUTH_GRPC_URL | Auth service gRPC URL | localhost:8181 |
|
||||
| MF_AUTH_GRPC_TIMEOUT | Auth service gRPC request timeout in seconds | 1s |
|
||||
| Variable | Description | Default |
|
||||
|------------------------------|-----------------------------------------------------|-------------------|
|
||||
| MF_INFLUX_READER_LOG_LEVEL | Service log level | info |
|
||||
| MF_INFLUX_READER_PORT | Service HTTP port | 8905 |
|
||||
| MF_INFLUXDB_HOST | InfluxDB host | localhost |
|
||||
| MF_INFLUXDB_PORT | Default port of InfluxDB database | 8086 |
|
||||
| MF_INFLUXDB_ADMIN_USER | Default user of InfluxDB database | mainflux |
|
||||
| MF_INFLUXDB_ADMIN_PASSWORD | Default password of InfluxDB user | mainflux |
|
||||
| MF_INFLUXDB_DB | InfluxDB database name | mainflux |
|
||||
| MF_INFLUXDB_HOST | InfluxDB host name | mainflux-influxdb |
|
||||
| MF_INFLUXDB_PROTOCOL | InfluxDB protocol | http |
|
||||
| MF_INFLUXDB_TIMEOUT | InfluxDB client connection readiness timeout | 1s |
|
||||
| MF_INFLUXDB_ORG | InfluxDB organization name | mainflux |
|
||||
| MF_INFLUXDB_BUCKET | InfluxDB bucket name | mainflux-bucket |
|
||||
| MF_INFLUXDB_TOKEN | InfluxDB API token | mainflux-token |
|
||||
| MF_INFLUXDB_HTTP_ENABLED | InfluxDB http enabled status | true |
|
||||
| MF_INFLUXDB_INIT_MODE | InfluxDB initialization mode | setup |
|
||||
| MF_INFLUX_READER_CLIENT_TLS | Flag that indicates if TLS should be turned on | false |
|
||||
| MF_INFLUX_READER_CA_CERTS | Path to trusted CAs in PEM format | |
|
||||
| MF_INFLUX_READER_SERVER_CERT | Path to server certificate in pem format | |
|
||||
| MF_INFLUX_READER_SERVER_KEY | Path to server key in pem format | |
|
||||
| MF_JAEGER_URL | Jaeger server URL | localhost:6831 |
|
||||
| MF_THINGS_AUTH_GRPC_URL | Things service Auth gRPC URL | localhost:8183 |
|
||||
| MF_THINGS_AUTH_GRPC_TIMEOUT | Things service Auth gRPC request timeout in seconds | 1s |
|
||||
| MF_AUTH_GRPC_URL | Auth service gRPC URL | localhost:8181 |
|
||||
| MF_AUTH_GRPC_TIMEOUT | Auth service gRPC request timeout in seconds | 1s |
|
||||
|
||||
|
||||
## Deployment
|
||||
@@ -53,6 +61,13 @@ MF_INFLUXDB_HOST=[InfluxDB database host] \
|
||||
MF_INFLUXDB_ADMIN_USER=[InfluxDB database port] \
|
||||
MF_INFLUXDB_ADMIN_USER=[InfluxDB admin user] \
|
||||
MF_INFLUXDB_ADMIN_PASSWORD=[InfluxDB admin password] \
|
||||
MF_INFLUXDB_PROTOCOL=[InfluxDB protocol] \
|
||||
MF_INFLUXDB_TIMEOUT=[InfluxDB timeout] \
|
||||
MF_INFLUXDB_ORG=[InfluxDB org] \
|
||||
MF_INFLUXDB_BUCKET=[InfluxDB bucket] \
|
||||
MF_INFLUXDB_TOKEN=[InfluxDB token] \
|
||||
MF_INFLUXDB_HTTP_ENABLED=[InfluxDB http enabled] \
|
||||
MF_INFLUXDB_INIT_MODE=[InfluxDB init mode] \
|
||||
MF_INFLUX_READER_CLIENT_TLS=[Flag that indicates if TLS should be turned on] \
|
||||
MF_INFLUX_READER_CA_CERTS=[Path to trusted CAs in PEM format] \
|
||||
MF_INFLUX_READER_SERVER_CERT=[Path to server pem certificate file] \
|
||||
@@ -76,8 +91,14 @@ docker-compose -f docker/docker-compose.yml up -d
|
||||
docker-compose -f docker/addons/influxdb-reader/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
And, to use the default .env file, execute the following command:
|
||||
|
||||
```bash
|
||||
docker-compose -f docker/addons/influxdb-reader/docker-compose.yml up --env-file docker/.env -d
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
Service exposes [HTTP API](https://api.mainflux.io/?urls.primaryName=readers-openapi.yml) for fetching messages.
|
||||
|
||||
[doc]: https://docs.mainflux.io
|
||||
Official docs can be found [here](https://docs.mainflux.io).
|
||||
|
||||
+148
-114
@@ -1,22 +1,22 @@
|
||||
package influxdb
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
"github.com/mainflux/mainflux/pkg/errors"
|
||||
"github.com/mainflux/mainflux/readers"
|
||||
|
||||
influxdata "github.com/influxdata/influxdb/client/v2"
|
||||
influxdb2 "github.com/influxdata/influxdb-client-go/v2"
|
||||
jsont "github.com/mainflux/mainflux/pkg/transformers/json"
|
||||
"github.com/mainflux/mainflux/pkg/transformers/senml"
|
||||
)
|
||||
|
||||
const (
|
||||
countCol = "count_protocol"
|
||||
// Measurement for SenML messages
|
||||
defMeasurement = "messages"
|
||||
)
|
||||
@@ -24,19 +24,22 @@ const (
|
||||
var _ readers.MessageRepository = (*influxRepository)(nil)
|
||||
|
||||
var (
|
||||
errResultSet = errors.New("invalid result set")
|
||||
errResultTime = errors.New("invalid result time")
|
||||
)
|
||||
|
||||
type RepoConfig struct {
|
||||
Bucket string
|
||||
Org string
|
||||
}
|
||||
type influxRepository struct {
|
||||
database string
|
||||
client influxdata.Client
|
||||
cfg RepoConfig
|
||||
client influxdb2.Client
|
||||
}
|
||||
|
||||
// New returns new InfluxDB reader.
|
||||
func New(client influxdata.Client, database string) readers.MessageRepository {
|
||||
func New(client influxdb2.Client, repoCfg RepoConfig) readers.MessageRepository {
|
||||
return &influxRepository{
|
||||
database,
|
||||
repoCfg,
|
||||
client,
|
||||
}
|
||||
}
|
||||
@@ -47,37 +50,45 @@ func (repo *influxRepository) ReadAll(chanID string, rpm readers.PageMetadata) (
|
||||
format = rpm.Format
|
||||
}
|
||||
|
||||
condition := fmtCondition(chanID, rpm)
|
||||
queryAPI := repo.client.QueryAPI(repo.cfg.Org)
|
||||
condition, timeRange := fmtCondition(chanID, rpm)
|
||||
query := fmt.Sprintf(`
|
||||
import "influxdata/influxdb/v1" from(bucket: "%s")
|
||||
%s
|
||||
|> v1.fieldsAsCols()
|
||||
|> group()
|
||||
|> filter(fn: (r) => r._measurement == "%s")
|
||||
%s
|
||||
|> sort(columns: ["_time"], desc: true)
|
||||
|> limit(n:%d,offset:%d)
|
||||
|> yield(name: "sort")`,
|
||||
repo.cfg.Bucket,
|
||||
timeRange,
|
||||
format,
|
||||
condition,
|
||||
rpm.Limit, rpm.Offset,
|
||||
)
|
||||
|
||||
cmd := fmt.Sprintf(`SELECT * FROM %s WHERE %s ORDER BY time DESC LIMIT %d OFFSET %d`, format, condition, rpm.Limit, rpm.Offset)
|
||||
q := influxdata.Query{
|
||||
Command: cmd,
|
||||
Database: repo.database,
|
||||
}
|
||||
|
||||
resp, err := repo.client.Query(q)
|
||||
resp, err := queryAPI.Query(context.Background(), query)
|
||||
if err != nil {
|
||||
return readers.MessagesPage{}, errors.Wrap(readers.ErrReadMessages, err)
|
||||
}
|
||||
if resp.Error() != nil {
|
||||
return readers.MessagesPage{}, errors.Wrap(readers.ErrReadMessages, resp.Error())
|
||||
}
|
||||
|
||||
if len(resp.Results) == 0 || len(resp.Results[0].Series) == 0 {
|
||||
return readers.MessagesPage{}, nil
|
||||
}
|
||||
|
||||
var messages []readers.Message
|
||||
result := resp.Results[0].Series[0]
|
||||
for _, v := range result.Values {
|
||||
msg, err := parseMessage(format, result.Columns, v)
|
||||
var valueMap map[string]interface{}
|
||||
for resp.Next() {
|
||||
valueMap = resp.Record().Values()
|
||||
msg, err := parseMessage(format, valueMap)
|
||||
if err != nil {
|
||||
return readers.MessagesPage{}, err
|
||||
}
|
||||
messages = append(messages, msg)
|
||||
}
|
||||
if resp.Err() != nil {
|
||||
return readers.MessagesPage{}, errors.Wrap(readers.ErrReadMessages, resp.Err())
|
||||
}
|
||||
|
||||
total, err := repo.count(format, condition)
|
||||
total, err := repo.count(format, condition, timeRange)
|
||||
if err != nil {
|
||||
return readers.MessagesPage{}, errors.Wrap(readers.ErrReadMessages, err)
|
||||
}
|
||||
@@ -89,62 +100,79 @@ func (repo *influxRepository) ReadAll(chanID string, rpm readers.PageMetadata) (
|
||||
}
|
||||
|
||||
return page, nil
|
||||
|
||||
}
|
||||
|
||||
func (repo *influxRepository) count(measurement, condition string) (uint64, error) {
|
||||
cmd := fmt.Sprintf(`SELECT COUNT(*) FROM %s WHERE %s`, measurement, condition)
|
||||
q := influxdata.Query{
|
||||
Command: cmd,
|
||||
Database: repo.database,
|
||||
}
|
||||
func (repo *influxRepository) count(measurement, condition string, timeRange string) (uint64, error) {
|
||||
cmd := fmt.Sprintf(`
|
||||
import "influxdata/influxdb/v1" from(bucket: "%s")
|
||||
%s
|
||||
|> v1.fieldsAsCols()
|
||||
|> filter(fn: (r) => r._measurement == "%s")
|
||||
%s
|
||||
|> group()
|
||||
|> count(column:"_measurement")
|
||||
|> yield(name: "count")
|
||||
`,
|
||||
repo.cfg.Bucket,
|
||||
timeRange,
|
||||
measurement,
|
||||
condition)
|
||||
queryAPI := repo.client.QueryAPI(repo.cfg.Org)
|
||||
resp, err := queryAPI.Query(context.Background(), cmd)
|
||||
|
||||
resp, err := repo.client.Query(q)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if resp.Error() != nil {
|
||||
return 0, resp.Error()
|
||||
}
|
||||
|
||||
if len(resp.Results) == 0 ||
|
||||
len(resp.Results[0].Series) == 0 ||
|
||||
len(resp.Results[0].Series[0].Values) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
switch resp.Next() {
|
||||
case true:
|
||||
valueMap := resp.Record().Values()
|
||||
|
||||
countIndex := 0
|
||||
for i, col := range resp.Results[0].Series[0].Columns {
|
||||
if col == countCol {
|
||||
countIndex = i
|
||||
break
|
||||
val, ok := valueMap["_measurement"].(int64)
|
||||
if !ok {
|
||||
return 0, nil
|
||||
}
|
||||
}
|
||||
return uint64(val), nil
|
||||
|
||||
result := resp.Results[0].Series[0].Values[0]
|
||||
if len(result) < countIndex+1 {
|
||||
default:
|
||||
// same as no rows.
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
count, ok := result[countIndex].(json.Number)
|
||||
if !ok {
|
||||
return 0, nil
|
||||
}
|
||||
return strconv.ParseUint(count.String(), 10, 64)
|
||||
}
|
||||
|
||||
func fmtCondition(chanID string, rpm readers.PageMetadata) string {
|
||||
condition := fmt.Sprintf(`channel='%s'`, chanID)
|
||||
func fmtCondition(chanID string, rpm readers.PageMetadata) (string, string) {
|
||||
var timeRange string
|
||||
var sb strings.Builder
|
||||
sb.WriteString(fmt.Sprintf(`|> filter(fn: (r) => r["channel"] == "%s" )`, chanID))
|
||||
|
||||
var query map[string]interface{}
|
||||
meta, err := json.Marshal(rpm)
|
||||
if err != nil {
|
||||
return condition
|
||||
return sb.String(), timeRange
|
||||
}
|
||||
|
||||
if err := json.Unmarshal(meta, &query); err != nil {
|
||||
return condition
|
||||
return sb.String(), timeRange
|
||||
}
|
||||
|
||||
//range(start:...) is a must for FluxQL syntax
|
||||
from := `start: time(v:0)`
|
||||
if value, ok := query["from"]; ok {
|
||||
fromValue := int64(value.(float64)*1e9) - 1
|
||||
from = fmt.Sprintf(`start: time(v: %d )`, fromValue)
|
||||
}
|
||||
//range(...,stop:) is an option for FluxQL syntax
|
||||
to := ""
|
||||
if value, ok := query["to"]; ok {
|
||||
toValue := int64(value.(float64) * 1e9)
|
||||
to = fmt.Sprintf(`, stop: time(v: %d )`, toValue)
|
||||
}
|
||||
// timeRange returned seperately because
|
||||
// in FluxQL time range must be at the
|
||||
// beginning of the query.
|
||||
timeRange = fmt.Sprintf(`|> range(%s %s)`, from, to)
|
||||
|
||||
for name, value := range query {
|
||||
switch name {
|
||||
case
|
||||
@@ -153,95 +181,101 @@ func fmtCondition(chanID string, rpm readers.PageMetadata) string {
|
||||
"publisher",
|
||||
"name",
|
||||
"protocol":
|
||||
condition = fmt.Sprintf(`%s AND "%s"='%s'`, condition, name, value)
|
||||
sb.WriteString(fmt.Sprintf(`|> filter(fn: (r) => r.%s == "%s" )`, name, value))
|
||||
case "v":
|
||||
comparator := readers.ParseValueComparator(query)
|
||||
condition = fmt.Sprintf(`%s AND value %s %f`, condition, comparator, value)
|
||||
// flux eq comparator is different
|
||||
if comparator == "=" {
|
||||
comparator = "=="
|
||||
}
|
||||
sb.WriteString(`|> filter(fn: (r) => exists r.value)`)
|
||||
sb.WriteString(fmt.Sprintf(`|> filter(fn: (r) => r.value %s %v)`, comparator, value))
|
||||
case "vb":
|
||||
condition = fmt.Sprintf(`%s AND boolValue = %t`, condition, value)
|
||||
sb.WriteString(`|> filter(fn: (r) => exists r.boolValue)`)
|
||||
sb.WriteString(fmt.Sprintf(`|> filter(fn: (r) => r.boolValue == %v)`, value))
|
||||
case "vs":
|
||||
condition = fmt.Sprintf(`%s AND stringValue = '%s'`, condition, value)
|
||||
sb.WriteString(`|> filter(fn: (r) => exists r.stringValue)`)
|
||||
sb.WriteString(fmt.Sprintf(`|> filter(fn: (r) => r.stringValue == "%s")`, value))
|
||||
case "vd":
|
||||
condition = fmt.Sprintf(`%s AND dataValue = '%s'`, condition, value)
|
||||
case "from":
|
||||
iVal := int64(value.(float64) * 1e9)
|
||||
condition = fmt.Sprintf(`%s AND time >= %d`, condition, iVal)
|
||||
case "to":
|
||||
iVal := int64(value.(float64) * 1e9)
|
||||
condition = fmt.Sprintf(`%s AND time < %d`, condition, iVal)
|
||||
sb.WriteString(`|> filter(fn: (r) => exists r.dataValue)`)
|
||||
sb.WriteString(fmt.Sprintf(`|> filter(fn: (r) => r.dataValue == "%s")`, value))
|
||||
}
|
||||
}
|
||||
return condition
|
||||
|
||||
return sb.String(), timeRange
|
||||
}
|
||||
|
||||
func parseMessage(measurement string, names []string, fields []interface{}) (interface{}, error) {
|
||||
func parseMessage(measurement string, valueMap map[string]interface{}) (interface{}, error) {
|
||||
switch measurement {
|
||||
case defMeasurement:
|
||||
return parseSenml(names, fields)
|
||||
return parseSenml(valueMap)
|
||||
default:
|
||||
return parseJSON(names, fields)
|
||||
return parseJSON(valueMap)
|
||||
}
|
||||
}
|
||||
|
||||
func underscore(names []string) {
|
||||
for i, name := range names {
|
||||
var buff []rune
|
||||
idx := 0
|
||||
for i, c := range name {
|
||||
if unicode.IsUpper(c) {
|
||||
buff = append(buff, []rune(name[idx:i])...)
|
||||
buff = append(buff, []rune{'_', unicode.ToLower(c)}...)
|
||||
idx = i + 1
|
||||
continue
|
||||
}
|
||||
func underscore(name string) string {
|
||||
var buff []rune
|
||||
idx := 0
|
||||
for i, c := range name {
|
||||
if unicode.IsUpper(c) {
|
||||
buff = append(buff, []rune(name[idx:i])...)
|
||||
buff = append(buff, []rune{'_', unicode.ToLower(c)}...)
|
||||
idx = i + 1
|
||||
continue
|
||||
}
|
||||
buff = append(buff, []rune(name[idx:])...)
|
||||
names[i] = string(buff)
|
||||
}
|
||||
buff = append(buff, []rune(name[idx:])...)
|
||||
return string(buff)
|
||||
}
|
||||
|
||||
func parseSenml(names []string, fields []interface{}) (interface{}, error) {
|
||||
m := make(map[string]interface{})
|
||||
if len(names) > len(fields) {
|
||||
return nil, errResultSet
|
||||
}
|
||||
underscore(names)
|
||||
for i, name := range names {
|
||||
if name == "time" {
|
||||
val, ok := fields[i].(string)
|
||||
func parseSenml(valueMap map[string]interface{}) (interface{}, error) {
|
||||
msg := make(map[string]interface{})
|
||||
|
||||
for k, v := range valueMap {
|
||||
k = underscore(k)
|
||||
if k == "_time" {
|
||||
k = "time"
|
||||
t, ok := v.(time.Time)
|
||||
if !ok {
|
||||
return nil, errResultTime
|
||||
}
|
||||
t, err := time.Parse(time.RFC3339Nano, val)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
v := float64(t.UnixNano()) / 1e9
|
||||
m[name] = v
|
||||
msg[k] = v
|
||||
continue
|
||||
}
|
||||
m[name] = fields[i]
|
||||
msg[k] = v
|
||||
}
|
||||
data, err := json.Marshal(m)
|
||||
data, err := json.Marshal(msg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
msg := senml.Message{}
|
||||
if err := json.Unmarshal(data, &msg); err != nil {
|
||||
senmlMsg := senml.Message{}
|
||||
if err := json.Unmarshal(data, &senmlMsg); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return msg, nil
|
||||
return senmlMsg, nil
|
||||
}
|
||||
|
||||
func parseJSON(names []string, fields []interface{}) (interface{}, error) {
|
||||
func parseJSON(valueMap map[string]interface{}) (interface{}, error) {
|
||||
ret := make(map[string]interface{})
|
||||
pld := make(map[string]interface{})
|
||||
for i, n := range names {
|
||||
switch n {
|
||||
case "channel", "created", "subtopic", "publisher", "protocol", "time":
|
||||
ret[n] = fields[i]
|
||||
for name, field := range valueMap {
|
||||
switch name {
|
||||
case "channel", "created", "subtopic", "publisher", "protocol":
|
||||
ret[name] = field
|
||||
case "_time":
|
||||
name = "time"
|
||||
t, ok := field.(time.Time)
|
||||
if !ok {
|
||||
return nil, errResultTime
|
||||
}
|
||||
v := float64(t.UnixNano()) / 1e9
|
||||
ret[name] = v
|
||||
continue
|
||||
case "table", "_start", "_stop", "result", "_measurement":
|
||||
default:
|
||||
v := fields[i]
|
||||
v := field
|
||||
if val, ok := v.(json.Number); ok {
|
||||
var err error
|
||||
v, err = val.Float64()
|
||||
@@ -249,7 +283,7 @@ func parseJSON(names []string, fields []interface{}) (interface{}, error) {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
pld[n] = v
|
||||
pld[name] = v
|
||||
}
|
||||
}
|
||||
ret["payload"] = jsont.ParseFlat(pld)
|
||||
|
||||
@@ -2,11 +2,10 @@ package influxdb_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
influxdata "github.com/influxdata/influxdb/client/v2"
|
||||
influxdata "github.com/influxdata/influxdb-client-go/v2"
|
||||
iwriter "github.com/mainflux/mainflux/consumers/writers/influxdb"
|
||||
"github.com/mainflux/mainflux/pkg/transformers/json"
|
||||
"github.com/mainflux/mainflux/pkg/transformers/senml"
|
||||
@@ -15,11 +14,9 @@ import (
|
||||
ireader "github.com/mainflux/mainflux/readers/influxdb"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
const (
|
||||
testDB = "test"
|
||||
subtopic = "topic"
|
||||
msgsNum = 100
|
||||
limit = 10
|
||||
@@ -28,34 +25,38 @@ const (
|
||||
httpProt = "http"
|
||||
msgName = "temperature"
|
||||
offset = 21
|
||||
|
||||
format1 = "format1"
|
||||
format2 = "format2"
|
||||
wrongID = "wrong_id"
|
||||
format1 = "format1"
|
||||
format2 = "format2"
|
||||
wrongID = "wrong_id"
|
||||
)
|
||||
|
||||
var (
|
||||
v float64 = 5
|
||||
vs = "a"
|
||||
vb = true
|
||||
vd = "dataValue"
|
||||
vs string = "a"
|
||||
vb bool = true
|
||||
vd string = "dataValue"
|
||||
sum float64 = 42
|
||||
|
||||
client influxdata.Client
|
||||
client influxdata.Client
|
||||
repoCfg = struct {
|
||||
Bucket string
|
||||
Org string
|
||||
}{
|
||||
Bucket: dbBucket,
|
||||
Org: dbOrg,
|
||||
}
|
||||
idProvider = uuid.New()
|
||||
)
|
||||
|
||||
func TestReadAll(t *testing.T) {
|
||||
writer := iwriter.New(client, testDB)
|
||||
func TestReadSenml(t *testing.T) {
|
||||
writer := iwriter.New(client, repoCfg, true)
|
||||
|
||||
chanID, err := idProvider.ID()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
pubID, err := idProvider.ID()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
pubID2, err := idProvider.ID()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
wrongID, err := idProvider.ID()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
|
||||
m := senml.Message{
|
||||
Channel: chanID,
|
||||
@@ -72,9 +73,7 @@ func TestReadAll(t *testing.T) {
|
||||
stringMsgs := []senml.Message{}
|
||||
dataMsgs := []senml.Message{}
|
||||
queryMsgs := []senml.Message{}
|
||||
rand.Seed(time.Now().UnixNano())
|
||||
to := msgsNum
|
||||
now := float64(rand.Intn(to) + offset)
|
||||
now := float64(time.Now().Unix())
|
||||
|
||||
for i := 0; i < msgsNum; i++ {
|
||||
// Mix possible values as well as value sum.
|
||||
@@ -103,21 +102,22 @@ func TestReadAll(t *testing.T) {
|
||||
msg.Name = msgName
|
||||
queryMsgs = append(queryMsgs, msg)
|
||||
}
|
||||
|
||||
messages = append(messages, msg)
|
||||
}
|
||||
|
||||
err = writer.Consume(messages)
|
||||
require.Nil(t, err, fmt.Sprintf("failed to store message to InfluxDB: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("failed to store message to InfluxDB: %s", err))
|
||||
|
||||
reader := ireader.New(client, testDB)
|
||||
reader := ireader.New(client, repoCfg)
|
||||
|
||||
cases := map[string]struct {
|
||||
cases := []struct {
|
||||
desc string
|
||||
chanID string
|
||||
pageMeta readers.PageMetadata
|
||||
page readers.MessagesPage
|
||||
}{
|
||||
"read message page for existing channel": {
|
||||
{
|
||||
desc: "read message page for existing channel",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -128,7 +128,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(messages),
|
||||
},
|
||||
},
|
||||
"read message page for non-existent channel": {
|
||||
{
|
||||
desc: "read message page for non-existent channel",
|
||||
chanID: wrongID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -138,7 +139,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: []readers.Message{},
|
||||
},
|
||||
},
|
||||
"read message last page": {
|
||||
{
|
||||
desc: "read message last page",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: msgsNum - 20,
|
||||
@@ -149,7 +151,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(messages[msgsNum-20 : msgsNum]),
|
||||
},
|
||||
},
|
||||
"read message with non-existent subtopic": {
|
||||
{
|
||||
desc: "read message with non-existent subtopic",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -160,7 +163,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: []readers.Message{},
|
||||
},
|
||||
},
|
||||
"read message with subtopic": {
|
||||
{
|
||||
desc: "read message with subtopic",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -172,7 +176,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(queryMsgs),
|
||||
},
|
||||
},
|
||||
"read message with publisher": {
|
||||
{
|
||||
desc: "read message with publisher",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -184,7 +189,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(queryMsgs),
|
||||
},
|
||||
},
|
||||
"read message with wrong format": {
|
||||
{
|
||||
desc: "read message with wrong format",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Format: "messagess",
|
||||
@@ -197,7 +203,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: []readers.Message{},
|
||||
},
|
||||
},
|
||||
"read message with protocol": {
|
||||
{
|
||||
desc: "read message with protocol",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -209,7 +216,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(queryMsgs),
|
||||
},
|
||||
},
|
||||
"read message with name": {
|
||||
{
|
||||
desc: "read message with name",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -221,7 +229,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(queryMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with value": {
|
||||
{
|
||||
desc: "read message with value",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -233,7 +242,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(valueMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with value and equal comparator": {
|
||||
{
|
||||
desc: "read message with value and equal comparator",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -246,7 +256,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(valueMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with value and lower-than comparator": {
|
||||
{
|
||||
desc: "read message with value and lower-than comparator",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -259,7 +270,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(valueMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with value and lower-than-or-equal comparator": {
|
||||
{
|
||||
desc: "read message with value and lower-than-or-equal comparator",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -272,7 +284,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(valueMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with value and greater-than comparator": {
|
||||
{
|
||||
desc: "read message with value and greater-than comparator",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -285,7 +298,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(valueMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with value and greater-than-or-equal comparator": {
|
||||
{
|
||||
desc: "read message with value and greater-than-or-equal comparator",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -298,7 +312,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(valueMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with boolean value": {
|
||||
{
|
||||
desc: "read message with boolean value",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -310,7 +325,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(boolMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with string value": {
|
||||
{
|
||||
desc: "read message with string value",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -322,7 +338,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(stringMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with data value": {
|
||||
{
|
||||
desc: "read message with data value",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -334,23 +351,25 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(dataMsgs[0:limit]),
|
||||
},
|
||||
},
|
||||
"read message with from": {
|
||||
{
|
||||
desc: "failing test case : read message with from",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
Limit: uint64(len(messages[0:offset])),
|
||||
From: messages[offset-1].Time,
|
||||
Limit: uint64(len(messages[0 : offset+1])),
|
||||
From: messages[offset].Time,
|
||||
},
|
||||
page: readers.MessagesPage{
|
||||
Total: uint64(len(messages[0:offset])),
|
||||
Messages: fromSenml(messages[0:offset]),
|
||||
Total: uint64(len(messages[0 : offset+1])),
|
||||
Messages: fromSenml(messages[0 : offset+1]),
|
||||
},
|
||||
},
|
||||
"read message with to": {
|
||||
{
|
||||
desc: "failing test case : read message with to",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
Limit: uint64(len(messages[offset:])),
|
||||
Limit: uint64(len(messages[offset-1:])),
|
||||
To: messages[offset-1].Time,
|
||||
},
|
||||
page: readers.MessagesPage{
|
||||
@@ -358,7 +377,8 @@ func TestReadAll(t *testing.T) {
|
||||
Messages: fromSenml(messages[offset:]),
|
||||
},
|
||||
},
|
||||
"read message with from/to": {
|
||||
{
|
||||
desc: "read message with from/to",
|
||||
chanID: chanID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Offset: 0,
|
||||
@@ -367,29 +387,30 @@ func TestReadAll(t *testing.T) {
|
||||
To: messages[0].Time,
|
||||
},
|
||||
page: readers.MessagesPage{
|
||||
Total: 5,
|
||||
Messages: fromSenml(messages[1:6]),
|
||||
Total: uint64(len(messages[0+1 : 5+1])),
|
||||
Messages: fromSenml(messages[0+1 : 5+1]),
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for desc, tc := range cases {
|
||||
for _, tc := range cases {
|
||||
result, err := reader.ReadAll(tc.chanID, tc.pageMeta)
|
||||
assert.Nil(t, err, fmt.Sprintf("%s: expected no error got %s", desc, err))
|
||||
assert.ElementsMatch(t, tc.page.Messages, result.Messages, fmt.Sprintf("%s: expected: %v, got: %v", desc, tc.page.Messages, result.Messages))
|
||||
assert.Equal(t, tc.page.Total, result.Total, fmt.Sprintf("%s: expected %d got %d", desc, tc.page.Total, result.Total))
|
||||
assert.Nil(t, err, fmt.Sprintf("%s: got unexpected error: %s\n", tc.desc, err))
|
||||
assert.ElementsMatch(t, tc.page.Messages, result.Messages, fmt.Sprintf("%s: expected: %v, got: %v\n", tc.desc, tc.page.Messages, result.Messages))
|
||||
assert.Equal(t, tc.page.Total, result.Total, fmt.Sprintf("%s: expected %d got %d\n", tc.desc, tc.page.Total, result.Total))
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadJSON(t *testing.T) {
|
||||
writer := iwriter.New(client, testDB)
|
||||
writer := iwriter.New(client, repoCfg, true)
|
||||
|
||||
id1, err := idProvider.ID()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
|
||||
m := json.Message{
|
||||
Channel: id1,
|
||||
Publisher: id1,
|
||||
Created: time.Now().UnixNano(),
|
||||
Created: time.Now().Unix() * 1e9,
|
||||
Subtopic: "subtopic/format/some_json",
|
||||
Protocol: "coap",
|
||||
Payload: map[string]interface{}{
|
||||
@@ -411,11 +432,12 @@ func TestReadJSON(t *testing.T) {
|
||||
assert.Nil(t, err, fmt.Sprintf("expected no error got %s\n", err))
|
||||
|
||||
id2, err := idProvider.ID()
|
||||
require.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
assert.Nil(t, err, fmt.Sprintf("got unexpected error: %s", err))
|
||||
|
||||
m = json.Message{
|
||||
Channel: id2,
|
||||
Publisher: id2,
|
||||
Created: time.Now().UnixNano() + msgsNum,
|
||||
Created: time.Now().Unix()*1e9 + msgsNum,
|
||||
Subtopic: "subtopic/other_format/some_other_json",
|
||||
Protocol: "udp",
|
||||
Payload: map[string]interface{}{
|
||||
@@ -442,14 +464,16 @@ func TestReadJSON(t *testing.T) {
|
||||
for i := 0; i < msgsNum; i += 2 {
|
||||
httpMsgs = append(httpMsgs, msgs2[i])
|
||||
}
|
||||
reader := ireader.New(client, testDB)
|
||||
reader := ireader.New(client, repoCfg)
|
||||
|
||||
cases := map[string]struct {
|
||||
cases := []struct {
|
||||
desc string
|
||||
chanID string
|
||||
pageMeta readers.PageMetadata
|
||||
page readers.MessagesPage
|
||||
}{
|
||||
"read message page for existing channel": {
|
||||
{
|
||||
desc: "read message page for existing channel",
|
||||
chanID: id1,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Format: messages1.Format,
|
||||
@@ -461,7 +485,8 @@ func TestReadJSON(t *testing.T) {
|
||||
Messages: fromJSON(msgs1[:1]),
|
||||
},
|
||||
},
|
||||
"read message page for non-existent channel": {
|
||||
{
|
||||
desc: "read message page for non-existent channel",
|
||||
chanID: wrongID,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Format: messages1.Format,
|
||||
@@ -472,7 +497,8 @@ func TestReadJSON(t *testing.T) {
|
||||
Messages: []readers.Message{},
|
||||
},
|
||||
},
|
||||
"read message last page": {
|
||||
{
|
||||
desc: "read message last page",
|
||||
chanID: id2,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Format: messages2.Format,
|
||||
@@ -484,7 +510,8 @@ func TestReadJSON(t *testing.T) {
|
||||
Messages: fromJSON(msgs2[msgsNum-20 : msgsNum]),
|
||||
},
|
||||
},
|
||||
"read message with protocol": {
|
||||
{
|
||||
desc: "read message with protocol",
|
||||
chanID: id2,
|
||||
pageMeta: readers.PageMetadata{
|
||||
Format: messages2.Format,
|
||||
@@ -499,9 +526,9 @@ func TestReadJSON(t *testing.T) {
|
||||
},
|
||||
}
|
||||
|
||||
for desc, tc := range cases {
|
||||
for _, tc := range cases {
|
||||
result, err := reader.ReadAll(tc.chanID, tc.pageMeta)
|
||||
assert.Nil(t, err, fmt.Sprintf("%s: expected no error got %s", desc, err))
|
||||
assert.Nil(t, err, fmt.Sprintf("%s: got unexpected error: %s", tc.desc, err))
|
||||
|
||||
for i := 0; i < len(result.Messages); i++ {
|
||||
m := result.Messages[i]
|
||||
@@ -510,8 +537,8 @@ func TestReadJSON(t *testing.T) {
|
||||
|
||||
result.Messages[i] = m
|
||||
}
|
||||
assert.ElementsMatch(t, tc.page.Messages, result.Messages, fmt.Sprintf("%s: expected \n%v got \n%v", desc, tc.page.Messages, result.Messages))
|
||||
assert.Equal(t, tc.page.Total, result.Total, fmt.Sprintf("%s: expected %v got %v", desc, tc.page.Total, result.Total))
|
||||
assert.ElementsMatch(t, tc.page.Messages, result.Messages, fmt.Sprintf("%s: expected \n%v got \n%v", tc.desc, tc.page.Messages, result.Messages))
|
||||
assert.Equal(t, tc.page.Total, result.Total, fmt.Sprintf("%s: expected %v got %v", tc.desc, tc.page.Total, result.Total))
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,24 +1,38 @@
|
||||
package influxdb_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
influxdata "github.com/influxdata/influxdb/client/v2"
|
||||
influxdb "github.com/influxdata/influxdb/client/v2"
|
||||
log "github.com/mainflux/mainflux/logger"
|
||||
influxdata "github.com/influxdata/influxdb-client-go/v2"
|
||||
mainflux_log "github.com/mainflux/mainflux/logger"
|
||||
dockertest "github.com/ory/dockertest/v3"
|
||||
)
|
||||
|
||||
var (
|
||||
testLog, _ = log.New(os.Stdout, log.Info.String())
|
||||
testLog, _ = mainflux_log.New(os.Stdout, mainflux_log.Info.String())
|
||||
address string
|
||||
)
|
||||
|
||||
clientCfg = influxdata.HTTPConfig{
|
||||
Username: "test",
|
||||
Password: "test",
|
||||
}
|
||||
const (
|
||||
dbToken = "test-token"
|
||||
dbOrg = "test-org"
|
||||
dbAdmin = "test-admin"
|
||||
dbPass = "test-password"
|
||||
dbBucket = "test-bucket"
|
||||
dbInitMode = "setup"
|
||||
dbFluxEnabled = "true"
|
||||
dbBindAddress = ":8088"
|
||||
port = "8086/tcp"
|
||||
broker = "influxdb"
|
||||
brokerVersion = "2.2-alpine"
|
||||
poolMaxWait = 120 * time.Second
|
||||
)
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
@@ -28,26 +42,32 @@ func TestMain(m *testing.M) {
|
||||
}
|
||||
|
||||
cfg := []string{
|
||||
"INFLUXDB_USER=test",
|
||||
"INFLUXDB_USER_PASSWORD=test",
|
||||
"INFLUXDB_DB=test",
|
||||
fmt.Sprintf("DOCKER_INFLUXDB_INIT_MODE=%s", dbInitMode),
|
||||
fmt.Sprintf("DOCKER_INFLUXDB_INIT_USERNAME=%s", dbAdmin),
|
||||
fmt.Sprintf("DOCKER_INFLUXDB_INIT_PASSWORD=%s", dbPass),
|
||||
fmt.Sprintf("DOCKER_INFLUXDB_INIT_ORG=%s", dbOrg),
|
||||
fmt.Sprintf("DOCKER_INFLUXDB_INIT_BUCKET=%s", dbBucket),
|
||||
fmt.Sprintf("DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=%s", dbToken),
|
||||
fmt.Sprintf("INFLUXDB_HTTP_FLUX_ENABLED=%s", dbFluxEnabled),
|
||||
fmt.Sprintf("INFLUXDB_BIND_ADDRESS=%s", dbBindAddress),
|
||||
}
|
||||
container, err := pool.Run("influxdb", "1.8.4", cfg)
|
||||
container, err := pool.Run(broker, brokerVersion, cfg)
|
||||
if err != nil {
|
||||
testLog.Error(fmt.Sprintf("Could not start container: %s", err))
|
||||
}
|
||||
|
||||
port := container.GetPort("8086/tcp")
|
||||
clientCfg.Addr = fmt.Sprintf("http://localhost:%s", port)
|
||||
handleInterrupt(m, pool, container)
|
||||
|
||||
address = fmt.Sprintf("%s:%s", "http://localhost", container.GetPort(port))
|
||||
pool.MaxWait = poolMaxWait
|
||||
|
||||
if err := pool.Retry(func() error {
|
||||
client, err = influxdb.NewHTTPClient(clientCfg)
|
||||
_, _, err = client.Ping(5 * time.Millisecond)
|
||||
client = influxdata.NewClient(address, dbToken)
|
||||
_, err = client.Ready(context.Background())
|
||||
return err
|
||||
}); err != nil {
|
||||
testLog.Error(fmt.Sprintf("Could not connect to docker: %s", err))
|
||||
}
|
||||
|
||||
code := m.Run()
|
||||
|
||||
if err := pool.Purge(container); err != nil {
|
||||
@@ -56,3 +76,15 @@ func TestMain(m *testing.M) {
|
||||
|
||||
os.Exit(code)
|
||||
}
|
||||
|
||||
func handleInterrupt(m *testing.M, pool *dockertest.Pool, container *dockertest.Resource) {
|
||||
c := make(chan os.Signal, 2)
|
||||
signal.Notify(c, os.Interrupt, syscall.SIGTERM)
|
||||
go func() {
|
||||
<-c
|
||||
if err := pool.Purge(container); err != nil {
|
||||
log.Fatalf("Could not purge container: %s", err)
|
||||
}
|
||||
os.Exit(0)
|
||||
}()
|
||||
}
|
||||
|
||||
+201
@@ -0,0 +1,201 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
// Copyright 2021 DeepMap, Inc.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
package runtime
|
||||
|
||||
// Binder is the interface implemented by types that can be bound to a query string or a parameter string
|
||||
// The input can be assumed to be a valid string. If you define a Bind method you are responsible for all
|
||||
// data being completely bound to the type.
|
||||
//
|
||||
// By convention, to approximate the behavior of Bind functions themselves,
|
||||
// Binder implements Bind("") as a no-op.
|
||||
type Binder interface {
|
||||
Bind(src string) error
|
||||
}
|
||||
+502
@@ -0,0 +1,502 @@
|
||||
// Copyright 2019 DeepMap, Inc.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"encoding"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"reflect"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
|
||||
"github.com/deepmap/oapi-codegen/pkg/types"
|
||||
)
|
||||
|
||||
// This function binds a parameter as described in the Path Parameters
|
||||
// section here to a Go object:
|
||||
// https://swagger.io/docs/specification/serialization/
|
||||
// It is a backward compatible function to clients generated with codegen
|
||||
// up to version v1.5.5. v1.5.6+ calls the function below.
|
||||
func BindStyledParameter(style string, explode bool, paramName string,
|
||||
value string, dest interface{}) error {
|
||||
return BindStyledParameterWithLocation(style, explode, paramName, ParamLocationUndefined, value, dest)
|
||||
}
|
||||
|
||||
// This function binds a parameter as described in the Path Parameters
|
||||
// section here to a Go object:
|
||||
// https://swagger.io/docs/specification/serialization/
|
||||
func BindStyledParameterWithLocation(style string, explode bool, paramName string,
|
||||
paramLocation ParamLocation, value string, dest interface{}) error {
|
||||
|
||||
if value == "" {
|
||||
return fmt.Errorf("parameter '%s' is empty, can't bind its value", paramName)
|
||||
}
|
||||
|
||||
// Based on the location of the parameter, we need to unescape it properly.
|
||||
var err error
|
||||
switch paramLocation {
|
||||
case ParamLocationQuery, ParamLocationUndefined:
|
||||
// We unescape undefined parameter locations here for older generated code,
|
||||
// since prior to this refactoring, they always query unescaped.
|
||||
value, err = url.QueryUnescape(value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("error unescaping query parameter '%s': %v", paramName, err)
|
||||
}
|
||||
case ParamLocationPath:
|
||||
value, err = url.PathUnescape(value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("error unescaping path parameter '%s': %v", paramName, err)
|
||||
}
|
||||
default:
|
||||
// Headers and cookies aren't escaped.
|
||||
}
|
||||
|
||||
// If the destination implements encoding.TextUnmarshaler we use it for binding
|
||||
if tu, ok := dest.(encoding.TextUnmarshaler); ok {
|
||||
if err := tu.UnmarshalText([]byte(value)); err != nil {
|
||||
return fmt.Errorf("error unmarshaling '%s' text as %T: %s", value, dest, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Everything comes in by pointer, dereference it
|
||||
v := reflect.Indirect(reflect.ValueOf(dest))
|
||||
|
||||
// This is the basic type of the destination object.
|
||||
t := v.Type()
|
||||
|
||||
if t.Kind() == reflect.Struct {
|
||||
// We've got a destination object, we'll create a JSON representation
|
||||
// of the input value, and let the json library deal with the unmarshaling
|
||||
parts, err := splitStyledParameter(style, explode, true, paramName, value)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return bindSplitPartsToDestinationStruct(paramName, parts, explode, dest)
|
||||
}
|
||||
|
||||
if t.Kind() == reflect.Slice {
|
||||
// Chop up the parameter into parts based on its style
|
||||
parts, err := splitStyledParameter(style, explode, false, paramName, value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("error splitting input '%s' into parts: %s", value, err)
|
||||
}
|
||||
|
||||
return bindSplitPartsToDestinationArray(parts, dest)
|
||||
}
|
||||
|
||||
// Try to bind the remaining types as a base type.
|
||||
return BindStringToObject(value, dest)
|
||||
}
|
||||
|
||||
// This is a complex set of operations, but each given parameter style can be
|
||||
// packed together in multiple ways, using different styles of separators, and
|
||||
// different packing strategies based on the explode flag. This function takes
|
||||
// as input any parameter format, and unpacks it to a simple list of strings
|
||||
// or key-values which we can then treat generically.
|
||||
// Why, oh why, great Swagger gods, did you have to make this so complicated?
|
||||
func splitStyledParameter(style string, explode bool, object bool, paramName string, value string) ([]string, error) {
|
||||
switch style {
|
||||
case "simple":
|
||||
// In the simple case, we always split on comma
|
||||
parts := strings.Split(value, ",")
|
||||
return parts, nil
|
||||
case "label":
|
||||
// In the label case, it's more tricky. In the no explode case, we have
|
||||
// /users/.3,4,5 for arrays
|
||||
// /users/.role,admin,firstName,Alex for objects
|
||||
// in the explode case, we have:
|
||||
// /users/.3.4.5
|
||||
// /users/.role=admin.firstName=Alex
|
||||
if explode {
|
||||
// In the exploded case, split everything on periods.
|
||||
parts := strings.Split(value, ".")
|
||||
// The first part should be an empty string because we have a
|
||||
// leading period.
|
||||
if parts[0] != "" {
|
||||
return nil, fmt.Errorf("invalid format for label parameter '%s', should start with '.'", paramName)
|
||||
}
|
||||
return parts[1:], nil
|
||||
|
||||
} else {
|
||||
// In the unexploded case, we strip off the leading period.
|
||||
if value[0] != '.' {
|
||||
return nil, fmt.Errorf("invalid format for label parameter '%s', should start with '.'", paramName)
|
||||
}
|
||||
// The rest is comma separated.
|
||||
return strings.Split(value[1:], ","), nil
|
||||
}
|
||||
|
||||
case "matrix":
|
||||
if explode {
|
||||
// In the exploded case, we break everything up on semicolon
|
||||
parts := strings.Split(value, ";")
|
||||
// The first part should always be empty string, since we started
|
||||
// with ;something
|
||||
if parts[0] != "" {
|
||||
return nil, fmt.Errorf("invalid format for matrix parameter '%s', should start with ';'", paramName)
|
||||
}
|
||||
parts = parts[1:]
|
||||
// Now, if we have an object, we just have a list of x=y statements.
|
||||
// for a non-object, like an array, we have id=x, id=y. id=z, etc,
|
||||
// so we need to strip the prefix from each of them.
|
||||
if !object {
|
||||
prefix := paramName + "="
|
||||
for i := range parts {
|
||||
parts[i] = strings.TrimPrefix(parts[i], prefix)
|
||||
}
|
||||
}
|
||||
return parts, nil
|
||||
} else {
|
||||
// In the unexploded case, parameters will start with ;paramName=
|
||||
prefix := ";" + paramName + "="
|
||||
if !strings.HasPrefix(value, prefix) {
|
||||
return nil, fmt.Errorf("expected parameter '%s' to start with %s", paramName, prefix)
|
||||
}
|
||||
str := strings.TrimPrefix(value, prefix)
|
||||
return strings.Split(str, ","), nil
|
||||
}
|
||||
case "form":
|
||||
var parts []string
|
||||
if explode {
|
||||
parts = strings.Split(value, "&")
|
||||
if !object {
|
||||
prefix := paramName + "="
|
||||
for i := range parts {
|
||||
parts[i] = strings.TrimPrefix(parts[i], prefix)
|
||||
}
|
||||
}
|
||||
return parts, nil
|
||||
} else {
|
||||
parts = strings.Split(value, ",")
|
||||
prefix := paramName + "="
|
||||
for i := range parts {
|
||||
parts[i] = strings.TrimPrefix(parts[i], prefix)
|
||||
}
|
||||
}
|
||||
return parts, nil
|
||||
}
|
||||
|
||||
return nil, fmt.Errorf("unhandled parameter style: %s", style)
|
||||
}
|
||||
|
||||
// Given a set of values as a slice, create a slice to hold them all, and
|
||||
// assign to each one by one.
|
||||
func bindSplitPartsToDestinationArray(parts []string, dest interface{}) error {
|
||||
// Everything comes in by pointer, dereference it
|
||||
v := reflect.Indirect(reflect.ValueOf(dest))
|
||||
|
||||
// This is the basic type of the destination object.
|
||||
t := v.Type()
|
||||
|
||||
// We've got a destination array, bind each object one by one.
|
||||
// This generates a slice of the correct element type and length to
|
||||
// hold all the parts.
|
||||
newArray := reflect.MakeSlice(t, len(parts), len(parts))
|
||||
for i, p := range parts {
|
||||
err := BindStringToObject(p, newArray.Index(i).Addr().Interface())
|
||||
if err != nil {
|
||||
return fmt.Errorf("error setting array element: %s", err)
|
||||
}
|
||||
}
|
||||
v.Set(newArray)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Given a set of chopped up parameter parts, bind them to a destination
|
||||
// struct. The exploded parameter controls whether we send key value pairs
|
||||
// in the exploded case, or a sequence of values which are interpreted as
|
||||
// tuples.
|
||||
// Given the struct Id { firstName string, role string }, as in the canonical
|
||||
// swagger examples, in the exploded case, we would pass
|
||||
// ["firstName=Alex", "role=admin"], where in the non-exploded case, we would
|
||||
// pass "firstName", "Alex", "role", "admin"]
|
||||
//
|
||||
// We punt the hard work of binding these values to the object to the json
|
||||
// library. We'll turn those arrays into JSON strings, and unmarshal
|
||||
// into the struct.
|
||||
func bindSplitPartsToDestinationStruct(paramName string, parts []string, explode bool, dest interface{}) error {
|
||||
// We've got a destination object, we'll create a JSON representation
|
||||
// of the input value, and let the json library deal with the unmarshaling
|
||||
var fields []string
|
||||
if explode {
|
||||
fields = make([]string, len(parts))
|
||||
for i, property := range parts {
|
||||
propertyParts := strings.Split(property, "=")
|
||||
if len(propertyParts) != 2 {
|
||||
return fmt.Errorf("parameter '%s' has invalid exploded format", paramName)
|
||||
}
|
||||
fields[i] = "\"" + propertyParts[0] + "\":\"" + propertyParts[1] + "\""
|
||||
}
|
||||
} else {
|
||||
if len(parts)%2 != 0 {
|
||||
return fmt.Errorf("parameter '%s' has invalid format, property/values need to be pairs", paramName)
|
||||
}
|
||||
fields = make([]string, len(parts)/2)
|
||||
for i := 0; i < len(parts); i += 2 {
|
||||
key := parts[i]
|
||||
value := parts[i+1]
|
||||
fields[i/2] = "\"" + key + "\":\"" + value + "\""
|
||||
}
|
||||
}
|
||||
jsonParam := "{" + strings.Join(fields, ",") + "}"
|
||||
err := json.Unmarshal([]byte(jsonParam), dest)
|
||||
if err != nil {
|
||||
return fmt.Errorf("error binding parameter %s fields: %s", paramName, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// This works much like BindStyledParameter, however it takes a query argument
|
||||
// input array from the url package, since query arguments come through a
|
||||
// different path than the styled arguments. They're also exceptionally fussy.
|
||||
// For example, consider the exploded and unexploded form parameter examples:
|
||||
// (exploded) /users?role=admin&firstName=Alex
|
||||
// (unexploded) /users?id=role,admin,firstName,Alex
|
||||
//
|
||||
// In the first case, we can pull the "id" parameter off the context,
|
||||
// and unmarshal via json as an intermediate. Easy. In the second case, we
|
||||
// don't have the id QueryParam present, but must find "role", and "firstName".
|
||||
// what if there is another parameter similar to "ID" named "role"? We can't
|
||||
// tell them apart. This code tries to fail, but the moral of the story is that
|
||||
// you shouldn't pass objects via form styled query arguments, just use
|
||||
// the Content parameter form.
|
||||
func BindQueryParameter(style string, explode bool, required bool, paramName string,
|
||||
queryParams url.Values, dest interface{}) error {
|
||||
|
||||
// dv = destination value.
|
||||
dv := reflect.Indirect(reflect.ValueOf(dest))
|
||||
|
||||
// intermediate value form which is either dv or dv dereferenced.
|
||||
v := dv
|
||||
|
||||
// inner code will bind the string's value to this interface.
|
||||
var output interface{}
|
||||
|
||||
if required {
|
||||
// If the parameter is required, then the generated code will pass us
|
||||
// a pointer to it: &int, &object, and so forth. We can directly set
|
||||
// them.
|
||||
output = dest
|
||||
} else {
|
||||
// For optional parameters, we have an extra indirect. An optional
|
||||
// parameter of type "int" will be *int on the struct. We pass that
|
||||
// in by pointer, and have **int.
|
||||
|
||||
// If the destination, is a nil pointer, we need to allocate it.
|
||||
if v.IsNil() {
|
||||
t := v.Type()
|
||||
newValue := reflect.New(t.Elem())
|
||||
// for now, hang onto the output buffer separately from destination,
|
||||
// as we don't want to write anything to destination until we can
|
||||
// unmarshal successfully, and check whether a field is required.
|
||||
output = newValue.Interface()
|
||||
} else {
|
||||
// If the destination isn't nil, just use that.
|
||||
output = v.Interface()
|
||||
}
|
||||
|
||||
// Get rid of that extra indirect as compared to the required case,
|
||||
// so the code below doesn't have to care.
|
||||
v = reflect.Indirect(reflect.ValueOf(output))
|
||||
}
|
||||
|
||||
// This is the basic type of the destination object.
|
||||
t := v.Type()
|
||||
k := t.Kind()
|
||||
|
||||
switch style {
|
||||
case "form":
|
||||
var parts []string
|
||||
if explode {
|
||||
// ok, the explode case in query arguments is very, very annoying,
|
||||
// because an exploded object, such as /users?role=admin&firstName=Alex
|
||||
// isn't actually present in the parameter array. We have to do
|
||||
// different things based on destination type.
|
||||
values, found := queryParams[paramName]
|
||||
var err error
|
||||
|
||||
switch k {
|
||||
case reflect.Slice:
|
||||
// In the slice case, we simply use the arguments provided by
|
||||
// http library.
|
||||
if !found {
|
||||
if required {
|
||||
return fmt.Errorf("query parameter '%s' is required", paramName)
|
||||
} else {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
err = bindSplitPartsToDestinationArray(values, output)
|
||||
case reflect.Struct:
|
||||
// This case is really annoying, and error prone, but the
|
||||
// form style object binding doesn't tell us which arguments
|
||||
// in the query string correspond to the object's fields. We'll
|
||||
// try to bind field by field.
|
||||
err = bindParamsToExplodedObject(paramName, queryParams, output)
|
||||
default:
|
||||
// Primitive object case. We expect to have 1 value to
|
||||
// unmarshal.
|
||||
if len(values) == 0 {
|
||||
if required {
|
||||
return fmt.Errorf("query parameter '%s' is required", paramName)
|
||||
} else {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if len(values) != 1 {
|
||||
return fmt.Errorf("multiple values for single value parameter '%s'", paramName)
|
||||
}
|
||||
err = BindStringToObject(values[0], output)
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// If the parameter is required, and we've successfully unmarshaled
|
||||
// it, this assigns the new object to the pointer pointer.
|
||||
if !required {
|
||||
dv.Set(reflect.ValueOf(output))
|
||||
}
|
||||
return nil
|
||||
} else {
|
||||
values, found := queryParams[paramName]
|
||||
if !found {
|
||||
if required {
|
||||
return fmt.Errorf("query parameter '%s' is required", paramName)
|
||||
} else {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if len(values) != 1 {
|
||||
return fmt.Errorf("parameter '%s' is not exploded, but is specified multiple times", paramName)
|
||||
}
|
||||
parts = strings.Split(values[0], ",")
|
||||
}
|
||||
var err error
|
||||
switch k {
|
||||
case reflect.Slice:
|
||||
err = bindSplitPartsToDestinationArray(parts, output)
|
||||
case reflect.Struct:
|
||||
err = bindSplitPartsToDestinationStruct(paramName, parts, explode, output)
|
||||
default:
|
||||
if len(parts) == 0 {
|
||||
if required {
|
||||
return fmt.Errorf("query parameter '%s' is required", paramName)
|
||||
} else {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if len(parts) != 1 {
|
||||
return fmt.Errorf("multiple values for single value parameter '%s'", paramName)
|
||||
}
|
||||
err = BindStringToObject(parts[0], output)
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !required {
|
||||
dv.Set(reflect.ValueOf(output))
|
||||
}
|
||||
return nil
|
||||
case "deepObject":
|
||||
if !explode {
|
||||
return errors.New("deepObjects must be exploded")
|
||||
}
|
||||
return UnmarshalDeepObject(dest, paramName, queryParams)
|
||||
case "spaceDelimited", "pipeDelimited":
|
||||
return fmt.Errorf("query arguments of style '%s' aren't yet supported", style)
|
||||
default:
|
||||
return fmt.Errorf("style '%s' on parameter '%s' is invalid", style, paramName)
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
// This function reflects the destination structure, and pulls the value for
|
||||
// each settable field from the given parameters map. This is to deal with the
|
||||
// exploded form styled object which may occupy any number of parameter names.
|
||||
// We don't try to be smart here, if the field exists as a query argument,
|
||||
// set its value.
|
||||
func bindParamsToExplodedObject(paramName string, values url.Values, dest interface{}) error {
|
||||
// Dereference pointers to their destination values
|
||||
binder, v, t := indirect(dest)
|
||||
if binder != nil {
|
||||
return BindStringToObject(values.Get(paramName), dest)
|
||||
}
|
||||
if t.Kind() != reflect.Struct {
|
||||
return fmt.Errorf("unmarshaling query arg '%s' into wrong type", paramName)
|
||||
}
|
||||
|
||||
for i := 0; i < t.NumField(); i++ {
|
||||
fieldT := t.Field(i)
|
||||
|
||||
// Skip unsettable fields, such as internal ones.
|
||||
if !v.Field(i).CanSet() {
|
||||
continue
|
||||
}
|
||||
|
||||
// Find the json annotation on the field, and use the json specified
|
||||
// name if available, otherwise, just the field name.
|
||||
tag := fieldT.Tag.Get("json")
|
||||
fieldName := fieldT.Name
|
||||
if tag != "" {
|
||||
tagParts := strings.Split(tag, ",")
|
||||
name := tagParts[0]
|
||||
if name != "" {
|
||||
fieldName = name
|
||||
}
|
||||
}
|
||||
|
||||
// At this point, we look up field name in the parameter list.
|
||||
fieldVal, found := values[fieldName]
|
||||
if found {
|
||||
if len(fieldVal) != 1 {
|
||||
return fmt.Errorf("field '%s' specified multiple times for param '%s'", fieldName, paramName)
|
||||
}
|
||||
err := BindStringToObject(fieldVal[0], v.Field(i).Addr().Interface())
|
||||
if err != nil {
|
||||
return fmt.Errorf("could not bind query arg '%s' to request object: %s'", paramName, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// indirect
|
||||
func indirect(dest interface{}) (interface{}, reflect.Value, reflect.Type) {
|
||||
v := reflect.ValueOf(dest)
|
||||
if v.Type().NumMethod() > 0 && v.CanInterface() {
|
||||
if u, ok := v.Interface().(Binder); ok {
|
||||
return u, reflect.Value{}, nil
|
||||
}
|
||||
}
|
||||
v = reflect.Indirect(v)
|
||||
t := v.Type()
|
||||
// special handling for custom types which might look like an object. We
|
||||
// don't want to use object binding on them, but rather treat them as
|
||||
// primitive types. time.Time{} is a unique case since we can't add a Binder
|
||||
// to it without changing the underlying generated code.
|
||||
if t.ConvertibleTo(reflect.TypeOf(time.Time{})) {
|
||||
return dest, reflect.Value{}, nil
|
||||
}
|
||||
if t.ConvertibleTo(reflect.TypeOf(types.Date{})) {
|
||||
return dest, reflect.Value{}, nil
|
||||
}
|
||||
return nil, v, t
|
||||
}
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
// Copyright 2019 DeepMap, Inc.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/deepmap/oapi-codegen/pkg/types"
|
||||
)
|
||||
|
||||
// This function takes a string, and attempts to assign it to the destination
|
||||
// interface via whatever type conversion is necessary. We have to do this
|
||||
// via reflection instead of a much simpler type switch so that we can handle
|
||||
// type aliases. This function was the easy way out, the better way, since we
|
||||
// know the destination type each place that we use this, is to generate code
|
||||
// to read each specific type.
|
||||
func BindStringToObject(src string, dst interface{}) error {
|
||||
var err error
|
||||
|
||||
v := reflect.ValueOf(dst)
|
||||
t := reflect.TypeOf(dst)
|
||||
|
||||
// We need to dereference pointers
|
||||
if t.Kind() == reflect.Ptr {
|
||||
v = reflect.Indirect(v)
|
||||
t = v.Type()
|
||||
}
|
||||
|
||||
// The resulting type must be settable. reflect will catch issues like
|
||||
// passing the destination by value.
|
||||
if !v.CanSet() {
|
||||
return errors.New("destination is not settable")
|
||||
}
|
||||
|
||||
switch t.Kind() {
|
||||
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
|
||||
var val int64
|
||||
val, err = strconv.ParseInt(src, 10, 64)
|
||||
if err == nil {
|
||||
v.SetInt(val)
|
||||
}
|
||||
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
|
||||
var val uint64
|
||||
val, err = strconv.ParseUint(src, 10, 64)
|
||||
if err == nil {
|
||||
v.SetUint(val)
|
||||
}
|
||||
case reflect.String:
|
||||
v.SetString(src)
|
||||
err = nil
|
||||
case reflect.Float64, reflect.Float32:
|
||||
var val float64
|
||||
val, err = strconv.ParseFloat(src, 64)
|
||||
if err == nil {
|
||||
v.SetFloat(val)
|
||||
}
|
||||
case reflect.Bool:
|
||||
var val bool
|
||||
val, err = strconv.ParseBool(src)
|
||||
if err == nil {
|
||||
v.SetBool(val)
|
||||
}
|
||||
case reflect.Struct:
|
||||
// if this is not of type Time or of type Date look to see if this is of type Binder.
|
||||
if dstType, ok := dst.(Binder); ok {
|
||||
return dstType.Bind(src)
|
||||
}
|
||||
|
||||
if t.ConvertibleTo(reflect.TypeOf(time.Time{})) {
|
||||
// Don't fail on empty string.
|
||||
if src == "" {
|
||||
return nil
|
||||
}
|
||||
// Time is a special case of a struct that we handle
|
||||
parsedTime, err := time.Parse(time.RFC3339Nano, src)
|
||||
if err != nil {
|
||||
parsedTime, err = time.Parse(types.DateFormat, src)
|
||||
if err != nil {
|
||||
return fmt.Errorf("error parsing '%s' as RFC3339 or 2006-01-02 time: %s", src, err)
|
||||
}
|
||||
}
|
||||
// So, assigning this gets a little fun. We have a value to the
|
||||
// dereference destination. We can't do a conversion to
|
||||
// time.Time because the result isn't assignable, so we need to
|
||||
// convert pointers.
|
||||
if t != reflect.TypeOf(time.Time{}) {
|
||||
vPtr := v.Addr()
|
||||
vtPtr := vPtr.Convert(reflect.TypeOf(&time.Time{}))
|
||||
v = reflect.Indirect(vtPtr)
|
||||
}
|
||||
v.Set(reflect.ValueOf(parsedTime))
|
||||
return nil
|
||||
}
|
||||
|
||||
if t.ConvertibleTo(reflect.TypeOf(types.Date{})) {
|
||||
// Don't fail on empty string.
|
||||
if src == "" {
|
||||
return nil
|
||||
}
|
||||
parsedTime, err := time.Parse(types.DateFormat, src)
|
||||
if err != nil {
|
||||
return fmt.Errorf("error parsing '%s' as date: %s", src, err)
|
||||
}
|
||||
parsedDate := types.Date{Time: parsedTime}
|
||||
|
||||
// We have to do the same dance here to assign, just like with times
|
||||
// above.
|
||||
if t != reflect.TypeOf(types.Date{}) {
|
||||
vPtr := v.Addr()
|
||||
vtPtr := vPtr.Convert(reflect.TypeOf(&types.Date{}))
|
||||
v = reflect.Indirect(vtPtr)
|
||||
}
|
||||
v.Set(reflect.ValueOf(parsedDate))
|
||||
return nil
|
||||
}
|
||||
|
||||
// We fall through to the error case below if we haven't handled the
|
||||
// destination type above.
|
||||
fallthrough
|
||||
default:
|
||||
// We've got a bunch of types unimplemented, don't fail silently.
|
||||
err = fmt.Errorf("can not bind to destination of type: %s", t.Kind())
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("error binding string parameter: %s", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+357
@@ -0,0 +1,357 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"github.com/deepmap/oapi-codegen/pkg/types"
|
||||
"net/url"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
)
|
||||
|
||||
func marshalDeepObject(in interface{}, path []string) ([]string, error) {
|
||||
var result []string
|
||||
|
||||
switch t := in.(type) {
|
||||
case []interface{}:
|
||||
// For the array, we will use numerical subscripts of the form [x],
|
||||
// in the same order as the array.
|
||||
for i, iface := range t {
|
||||
newPath := append(path, strconv.Itoa(i))
|
||||
fields, err := marshalDeepObject(iface, newPath)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "error traversing array")
|
||||
}
|
||||
result = append(result, fields...)
|
||||
}
|
||||
case map[string]interface{}:
|
||||
// For a map, each key (field name) becomes a member of the path, and
|
||||
// we recurse. First, sort the keys.
|
||||
keys := make([]string, len(t))
|
||||
i := 0
|
||||
for k := range t {
|
||||
keys[i] = k
|
||||
i++
|
||||
}
|
||||
sort.Strings(keys)
|
||||
|
||||
// Now, for each key, we recursively marshal it.
|
||||
for _, k := range keys {
|
||||
newPath := append(path, k)
|
||||
fields, err := marshalDeepObject(t[k], newPath)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "error traversing map")
|
||||
}
|
||||
result = append(result, fields...)
|
||||
}
|
||||
default:
|
||||
// Now, for a concrete value, we will turn the path elements
|
||||
// into a deepObject style set of subscripts. [a, b, c] turns into
|
||||
// [a][b][c]
|
||||
prefix := "[" + strings.Join(path, "][") + "]"
|
||||
result = []string{
|
||||
prefix + fmt.Sprintf("=%v", t),
|
||||
}
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func MarshalDeepObject(i interface{}, paramName string) (string, error) {
|
||||
// We're going to marshal to JSON and unmarshal into an interface{},
|
||||
// which will use the json pkg to deal with all the field annotations. We
|
||||
// can then walk the generic object structure to produce a deepObject. This
|
||||
// isn't efficient and it would be more efficient to reflect on our own,
|
||||
// but it's complicated, error-prone code.
|
||||
buf, err := json.Marshal(i)
|
||||
if err != nil {
|
||||
return "", errors.Wrap(err, "failed to marshal input to JSON")
|
||||
}
|
||||
var i2 interface{}
|
||||
err = json.Unmarshal(buf, &i2)
|
||||
if err != nil {
|
||||
return "", errors.Wrap(err, "failed to unmarshal JSON")
|
||||
}
|
||||
fields, err := marshalDeepObject(i2, nil)
|
||||
if err != nil {
|
||||
return "", errors.Wrap(err, "error traversing JSON structure")
|
||||
}
|
||||
|
||||
// Prefix the param name to each subscripted field.
|
||||
for i := range fields {
|
||||
fields[i] = paramName + fields[i]
|
||||
}
|
||||
return strings.Join(fields, "&"), nil
|
||||
}
|
||||
|
||||
type fieldOrValue struct {
|
||||
fields map[string]fieldOrValue
|
||||
value string
|
||||
}
|
||||
|
||||
func (f *fieldOrValue) appendPathValue(path []string, value string) {
|
||||
fieldName := path[0]
|
||||
if len(path) == 1 {
|
||||
f.fields[fieldName] = fieldOrValue{value: value}
|
||||
return
|
||||
}
|
||||
|
||||
pv, found := f.fields[fieldName]
|
||||
if !found {
|
||||
pv = fieldOrValue{
|
||||
fields: make(map[string]fieldOrValue),
|
||||
}
|
||||
f.fields[fieldName] = pv
|
||||
}
|
||||
pv.appendPathValue(path[1:], value)
|
||||
}
|
||||
|
||||
func makeFieldOrValue(paths [][]string, values []string) fieldOrValue {
|
||||
|
||||
f := fieldOrValue{
|
||||
fields: make(map[string]fieldOrValue),
|
||||
}
|
||||
for i := range paths {
|
||||
path := paths[i]
|
||||
value := values[i]
|
||||
f.appendPathValue(path, value)
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
func UnmarshalDeepObject(dst interface{}, paramName string, params url.Values) error {
|
||||
// Params are all the query args, so we need those that look like
|
||||
// "paramName["...
|
||||
var fieldNames []string
|
||||
var fieldValues []string
|
||||
searchStr := paramName + "["
|
||||
for pName, pValues := range params {
|
||||
if strings.HasPrefix(pName, searchStr) {
|
||||
// trim the parameter name from the full name.
|
||||
pName = pName[len(paramName):]
|
||||
fieldNames = append(fieldNames, pName)
|
||||
if len(pValues) != 1 {
|
||||
return fmt.Errorf("%s has multiple values", pName)
|
||||
}
|
||||
fieldValues = append(fieldValues, pValues[0])
|
||||
}
|
||||
}
|
||||
|
||||
// Now, for each field, reconstruct its subscript path and value
|
||||
paths := make([][]string, len(fieldNames))
|
||||
for i, path := range fieldNames {
|
||||
path = strings.TrimLeft(path, "[")
|
||||
path = strings.TrimRight(path, "]")
|
||||
paths[i] = strings.Split(path, "][")
|
||||
}
|
||||
|
||||
fieldPaths := makeFieldOrValue(paths, fieldValues)
|
||||
err := assignPathValues(dst, fieldPaths)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "error assigning value to destination")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// This returns a field name, either using the variable name, or the json
|
||||
// annotation if that exists.
|
||||
func getFieldName(f reflect.StructField) string {
|
||||
n := f.Name
|
||||
tag, found := f.Tag.Lookup("json")
|
||||
if found {
|
||||
// If we have a json field, and the first part of it before the
|
||||
// first comma is non-empty, that's our field name.
|
||||
parts := strings.Split(tag, ",")
|
||||
if parts[0] != "" {
|
||||
n = parts[0]
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// Create a map of field names that we'll see in the deepObject to reflect
|
||||
// field indices on the given type.
|
||||
func fieldIndicesByJsonTag(i interface{}) (map[string]int, error) {
|
||||
t := reflect.TypeOf(i)
|
||||
if t.Kind() != reflect.Struct {
|
||||
return nil, errors.New("expected a struct as input")
|
||||
}
|
||||
|
||||
n := t.NumField()
|
||||
fieldMap := make(map[string]int)
|
||||
for i := 0; i < n; i++ {
|
||||
field := t.Field(i)
|
||||
fieldName := getFieldName(field)
|
||||
fieldMap[fieldName] = i
|
||||
}
|
||||
return fieldMap, nil
|
||||
}
|
||||
|
||||
func assignPathValues(dst interface{}, pathValues fieldOrValue) error {
|
||||
//t := reflect.TypeOf(dst)
|
||||
v := reflect.ValueOf(dst)
|
||||
|
||||
iv := reflect.Indirect(v)
|
||||
it := iv.Type()
|
||||
|
||||
switch it.Kind() {
|
||||
case reflect.Slice:
|
||||
sliceLength := len(pathValues.fields)
|
||||
dstSlice := reflect.MakeSlice(it, sliceLength, sliceLength)
|
||||
err := assignSlice(dstSlice, pathValues)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "error assigning slice")
|
||||
}
|
||||
iv.Set(dstSlice)
|
||||
return nil
|
||||
case reflect.Struct:
|
||||
// Some special types we care about are structs. Handle them
|
||||
// here. They may be redefined, so we need to do some hoop
|
||||
// jumping. If the types are aliased, we need to type convert
|
||||
// the pointer, then set the value of the dereference pointer.
|
||||
|
||||
// We check to see if the object implements the Binder interface first.
|
||||
if dst, isBinder := v.Interface().(Binder); isBinder {
|
||||
return dst.Bind(pathValues.value)
|
||||
}
|
||||
// Then check the legacy types
|
||||
if it.ConvertibleTo(reflect.TypeOf(types.Date{})) {
|
||||
var date types.Date
|
||||
var err error
|
||||
date.Time, err = time.Parse(types.DateFormat, pathValues.value)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "invalid date format")
|
||||
}
|
||||
dst := iv
|
||||
if it != reflect.TypeOf(types.Date{}) {
|
||||
// Types are aliased, convert the pointers.
|
||||
ivPtr := iv.Addr()
|
||||
aPtr := ivPtr.Convert(reflect.TypeOf(&types.Date{}))
|
||||
dst = reflect.Indirect(aPtr)
|
||||
}
|
||||
dst.Set(reflect.ValueOf(date))
|
||||
}
|
||||
if it.ConvertibleTo(reflect.TypeOf(time.Time{})) {
|
||||
var tm time.Time
|
||||
var err error
|
||||
tm, err = time.Parse(time.RFC3339Nano, pathValues.value)
|
||||
if err != nil {
|
||||
// Fall back to parsing it as a date.
|
||||
tm, err = time.Parse(types.DateFormat, pathValues.value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("error parsing tim as RFC3339 or 2006-01-02 time: %s", err)
|
||||
}
|
||||
return errors.Wrap(err, "invalid date format")
|
||||
}
|
||||
dst := iv
|
||||
if it != reflect.TypeOf(time.Time{}) {
|
||||
// Types are aliased, convert the pointers.
|
||||
ivPtr := iv.Addr()
|
||||
aPtr := ivPtr.Convert(reflect.TypeOf(&time.Time{}))
|
||||
dst = reflect.Indirect(aPtr)
|
||||
}
|
||||
dst.Set(reflect.ValueOf(tm))
|
||||
}
|
||||
fieldMap, err := fieldIndicesByJsonTag(iv.Interface())
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed enumerating fields")
|
||||
}
|
||||
for _, fieldName := range sortedFieldOrValueKeys(pathValues.fields) {
|
||||
fieldValue := pathValues.fields[fieldName]
|
||||
fieldIndex, found := fieldMap[fieldName]
|
||||
if !found {
|
||||
return fmt.Errorf("field [%s] is not present in destination object", fieldName)
|
||||
}
|
||||
field := iv.Field(fieldIndex)
|
||||
err = assignPathValues(field.Addr().Interface(), fieldValue)
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "error assigning field [%s]", fieldName)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
case reflect.Ptr:
|
||||
// If we have a pointer after redirecting, it means we're dealing with
|
||||
// an optional field, such as *string, which was passed in as &foo. We
|
||||
// will allocate it if necessary, and call ourselves with a different
|
||||
// interface.
|
||||
dstVal := reflect.New(it.Elem())
|
||||
dstPtr := dstVal.Interface()
|
||||
err := assignPathValues(dstPtr, pathValues)
|
||||
iv.Set(dstVal)
|
||||
return err
|
||||
case reflect.Bool:
|
||||
val, err := strconv.ParseBool(pathValues.value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("expected a valid bool, got %s", pathValues.value)
|
||||
}
|
||||
iv.SetBool(val)
|
||||
return nil
|
||||
case reflect.Float32:
|
||||
val, err := strconv.ParseFloat(pathValues.value, 32)
|
||||
if err != nil {
|
||||
return fmt.Errorf("expected a valid float, got %s", pathValues.value)
|
||||
}
|
||||
iv.SetFloat(val)
|
||||
return nil
|
||||
case reflect.Float64:
|
||||
val, err := strconv.ParseFloat(pathValues.value, 64)
|
||||
if err != nil {
|
||||
return fmt.Errorf("expected a valid float, got %s", pathValues.value)
|
||||
}
|
||||
iv.SetFloat(val)
|
||||
return nil
|
||||
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
|
||||
val, err := strconv.ParseInt(pathValues.value, 10, 64)
|
||||
if err != nil {
|
||||
return fmt.Errorf("expected a valid int, got %s", pathValues.value)
|
||||
}
|
||||
iv.SetInt(val)
|
||||
return nil
|
||||
case reflect.String:
|
||||
iv.SetString(pathValues.value)
|
||||
return nil
|
||||
default:
|
||||
return errors.New("unhandled type: " + it.String())
|
||||
}
|
||||
}
|
||||
|
||||
func assignSlice(dst reflect.Value, pathValues fieldOrValue) error {
|
||||
// Gather up the values
|
||||
nValues := len(pathValues.fields)
|
||||
values := make([]string, nValues)
|
||||
// We expect to have consecutive array indices in the map
|
||||
for i := 0; i < nValues; i++ {
|
||||
indexStr := strconv.Itoa(i)
|
||||
fv, found := pathValues.fields[indexStr]
|
||||
if !found {
|
||||
return errors.New("array deepObjects must have consecutive indices")
|
||||
}
|
||||
values[i] = fv.value
|
||||
}
|
||||
|
||||
// This could be cleaner, but we can call into assignPathValues to
|
||||
// avoid recreating this logic.
|
||||
for i := 0; i < nValues; i++ {
|
||||
dstElem := dst.Index(i).Addr()
|
||||
err := assignPathValues(dstElem.Interface(), fieldOrValue{value: values[i]})
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "error binding array")
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func sortedFieldOrValueKeys(m map[string]fieldOrValue) []string {
|
||||
keys := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
return keys
|
||||
}
|
||||
+390
@@ -0,0 +1,390 @@
|
||||
// Copyright 2019 DeepMap, Inc.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/url"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
|
||||
"github.com/deepmap/oapi-codegen/pkg/types"
|
||||
)
|
||||
|
||||
// Parameter escaping works differently based on where a header is found
|
||||
type ParamLocation int
|
||||
|
||||
const (
|
||||
ParamLocationUndefined ParamLocation = iota
|
||||
ParamLocationQuery
|
||||
ParamLocationPath
|
||||
ParamLocationHeader
|
||||
ParamLocationCookie
|
||||
)
|
||||
|
||||
// This function is used by older generated code, and must remain compatible
|
||||
// with that code. It is not to be used in new templates. Please see the
|
||||
// function below, which can specialize its output based on the location of
|
||||
// the parameter.
|
||||
func StyleParam(style string, explode bool, paramName string, value interface{}) (string, error) {
|
||||
return StyleParamWithLocation(style, explode, paramName, ParamLocationUndefined, value)
|
||||
}
|
||||
|
||||
// Given an input value, such as a primitive type, array or object, turn it
|
||||
// into a parameter based on style/explode definition, performing whatever
|
||||
// escaping is necessary based on parameter location
|
||||
func StyleParamWithLocation(style string, explode bool, paramName string, paramLocation ParamLocation, value interface{}) (string, error) {
|
||||
t := reflect.TypeOf(value)
|
||||
v := reflect.ValueOf(value)
|
||||
|
||||
// Things may be passed in by pointer, we need to dereference, so return
|
||||
// error on nil.
|
||||
if t.Kind() == reflect.Ptr {
|
||||
if v.IsNil() {
|
||||
return "", fmt.Errorf("value is a nil pointer")
|
||||
}
|
||||
v = reflect.Indirect(v)
|
||||
t = v.Type()
|
||||
}
|
||||
|
||||
switch t.Kind() {
|
||||
case reflect.Slice:
|
||||
n := v.Len()
|
||||
sliceVal := make([]interface{}, n)
|
||||
for i := 0; i < n; i++ {
|
||||
sliceVal[i] = v.Index(i).Interface()
|
||||
}
|
||||
return styleSlice(style, explode, paramName, paramLocation, sliceVal)
|
||||
case reflect.Struct:
|
||||
return styleStruct(style, explode, paramName, paramLocation, value)
|
||||
case reflect.Map:
|
||||
return styleMap(style, explode, paramName, paramLocation, value)
|
||||
default:
|
||||
return stylePrimitive(style, explode, paramName, paramLocation, value)
|
||||
}
|
||||
}
|
||||
|
||||
func styleSlice(style string, explode bool, paramName string, paramLocation ParamLocation, values []interface{}) (string, error) {
|
||||
if style == "deepObject" {
|
||||
if !explode {
|
||||
return "", errors.New("deepObjects must be exploded")
|
||||
}
|
||||
return MarshalDeepObject(values, paramName)
|
||||
}
|
||||
|
||||
var prefix string
|
||||
var separator string
|
||||
|
||||
switch style {
|
||||
case "simple":
|
||||
separator = ","
|
||||
case "label":
|
||||
prefix = "."
|
||||
if explode {
|
||||
separator = "."
|
||||
} else {
|
||||
separator = ","
|
||||
}
|
||||
case "matrix":
|
||||
prefix = fmt.Sprintf(";%s=", paramName)
|
||||
if explode {
|
||||
separator = prefix
|
||||
} else {
|
||||
separator = ","
|
||||
}
|
||||
case "form":
|
||||
prefix = fmt.Sprintf("%s=", paramName)
|
||||
if explode {
|
||||
separator = "&" + prefix
|
||||
} else {
|
||||
separator = ","
|
||||
}
|
||||
case "spaceDelimited":
|
||||
prefix = fmt.Sprintf("%s=", paramName)
|
||||
if explode {
|
||||
separator = "&" + prefix
|
||||
} else {
|
||||
separator = " "
|
||||
}
|
||||
case "pipeDelimited":
|
||||
prefix = fmt.Sprintf("%s=", paramName)
|
||||
if explode {
|
||||
separator = "&" + prefix
|
||||
} else {
|
||||
separator = "|"
|
||||
}
|
||||
default:
|
||||
return "", fmt.Errorf("unsupported style '%s'", style)
|
||||
}
|
||||
|
||||
// We're going to assume here that the array is one of simple types.
|
||||
var err error
|
||||
var part string
|
||||
parts := make([]string, len(values))
|
||||
for i, v := range values {
|
||||
part, err = primitiveToString(v)
|
||||
part = escapeParameterString(part, paramLocation)
|
||||
parts[i] = part
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("error formatting '%s': %s", paramName, err)
|
||||
}
|
||||
}
|
||||
return prefix + strings.Join(parts, separator), nil
|
||||
}
|
||||
|
||||
func sortedKeys(strMap map[string]string) []string {
|
||||
keys := make([]string, len(strMap))
|
||||
i := 0
|
||||
for k := range strMap {
|
||||
keys[i] = k
|
||||
i++
|
||||
}
|
||||
sort.Strings(keys)
|
||||
return keys
|
||||
}
|
||||
|
||||
// This is a special case. The struct may be a date or time, in
|
||||
// which case, marshal it in correct format.
|
||||
func marshalDateTimeValue(value interface{}) (string, bool) {
|
||||
v := reflect.Indirect(reflect.ValueOf(value))
|
||||
t := v.Type()
|
||||
|
||||
if t.ConvertibleTo(reflect.TypeOf(time.Time{})) {
|
||||
tt := v.Convert(reflect.TypeOf(time.Time{}))
|
||||
timeVal := tt.Interface().(time.Time)
|
||||
return timeVal.Format(time.RFC3339Nano), true
|
||||
}
|
||||
|
||||
if t.ConvertibleTo(reflect.TypeOf(types.Date{})) {
|
||||
d := v.Convert(reflect.TypeOf(types.Date{}))
|
||||
dateVal := d.Interface().(types.Date)
|
||||
return dateVal.Format(types.DateFormat), true
|
||||
}
|
||||
|
||||
return "", false
|
||||
}
|
||||
|
||||
func styleStruct(style string, explode bool, paramName string, paramLocation ParamLocation, value interface{}) (string, error) {
|
||||
|
||||
if timeVal, ok := marshalDateTimeValue(value); ok {
|
||||
styledVal, err := stylePrimitive(style, explode, paramName, paramLocation, timeVal)
|
||||
if err != nil {
|
||||
return "", errors.Wrap(err, "failed to style time")
|
||||
}
|
||||
return styledVal, nil
|
||||
}
|
||||
|
||||
if style == "deepObject" {
|
||||
if !explode {
|
||||
return "", errors.New("deepObjects must be exploded")
|
||||
}
|
||||
return MarshalDeepObject(value, paramName)
|
||||
}
|
||||
|
||||
// Otherwise, we need to build a dictionary of the struct's fields. Each
|
||||
// field may only be a primitive value.
|
||||
v := reflect.ValueOf(value)
|
||||
t := reflect.TypeOf(value)
|
||||
fieldDict := make(map[string]string)
|
||||
|
||||
for i := 0; i < t.NumField(); i++ {
|
||||
fieldT := t.Field(i)
|
||||
// Find the json annotation on the field, and use the json specified
|
||||
// name if available, otherwise, just the field name.
|
||||
tag := fieldT.Tag.Get("json")
|
||||
fieldName := fieldT.Name
|
||||
if tag != "" {
|
||||
tagParts := strings.Split(tag, ",")
|
||||
name := tagParts[0]
|
||||
if name != "" {
|
||||
fieldName = name
|
||||
}
|
||||
}
|
||||
f := v.Field(i)
|
||||
|
||||
// Unset optional fields will be nil pointers, skip over those.
|
||||
if f.Type().Kind() == reflect.Ptr && f.IsNil() {
|
||||
continue
|
||||
}
|
||||
str, err := primitiveToString(f.Interface())
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("error formatting '%s': %s", paramName, err)
|
||||
}
|
||||
fieldDict[fieldName] = str
|
||||
}
|
||||
|
||||
return processFieldDict(style, explode, paramName, paramLocation, fieldDict)
|
||||
}
|
||||
|
||||
func styleMap(style string, explode bool, paramName string, paramLocation ParamLocation, value interface{}) (string, error) {
|
||||
if style == "deepObject" {
|
||||
if !explode {
|
||||
return "", errors.New("deepObjects must be exploded")
|
||||
}
|
||||
return MarshalDeepObject(value, paramName)
|
||||
}
|
||||
|
||||
dict, ok := value.(map[string]interface{})
|
||||
if !ok {
|
||||
return "", errors.New("map not of type map[string]interface{}")
|
||||
}
|
||||
|
||||
fieldDict := make(map[string]string)
|
||||
for fieldName, value := range dict {
|
||||
str, err := primitiveToString(value)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("error formatting '%s': %s", paramName, err)
|
||||
}
|
||||
fieldDict[fieldName] = str
|
||||
}
|
||||
return processFieldDict(style, explode, paramName, paramLocation, fieldDict)
|
||||
}
|
||||
|
||||
func processFieldDict(style string, explode bool, paramName string, paramLocation ParamLocation, fieldDict map[string]string) (string, error) {
|
||||
var parts []string
|
||||
|
||||
// This works for everything except deepObject. We'll handle that one
|
||||
// separately.
|
||||
if style != "deepObject" {
|
||||
if explode {
|
||||
for _, k := range sortedKeys(fieldDict) {
|
||||
v := escapeParameterString(fieldDict[k], paramLocation)
|
||||
parts = append(parts, k+"="+v)
|
||||
}
|
||||
} else {
|
||||
for _, k := range sortedKeys(fieldDict) {
|
||||
v := escapeParameterString(fieldDict[k], paramLocation)
|
||||
parts = append(parts, k)
|
||||
parts = append(parts, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var prefix string
|
||||
var separator string
|
||||
|
||||
switch style {
|
||||
case "simple":
|
||||
separator = ","
|
||||
case "label":
|
||||
prefix = "."
|
||||
if explode {
|
||||
separator = prefix
|
||||
} else {
|
||||
separator = ","
|
||||
}
|
||||
case "matrix":
|
||||
if explode {
|
||||
separator = ";"
|
||||
prefix = ";"
|
||||
} else {
|
||||
separator = ","
|
||||
prefix = fmt.Sprintf(";%s=", paramName)
|
||||
}
|
||||
case "form":
|
||||
if explode {
|
||||
separator = "&"
|
||||
} else {
|
||||
prefix = fmt.Sprintf("%s=", paramName)
|
||||
separator = ","
|
||||
}
|
||||
case "deepObject":
|
||||
{
|
||||
if !explode {
|
||||
return "", fmt.Errorf("deepObject parameters must be exploded")
|
||||
}
|
||||
for _, k := range sortedKeys(fieldDict) {
|
||||
v := fieldDict[k]
|
||||
part := fmt.Sprintf("%s[%s]=%s", paramName, k, v)
|
||||
parts = append(parts, part)
|
||||
}
|
||||
separator = "&"
|
||||
}
|
||||
default:
|
||||
return "", fmt.Errorf("unsupported style '%s'", style)
|
||||
}
|
||||
|
||||
return prefix + strings.Join(parts, separator), nil
|
||||
}
|
||||
|
||||
func stylePrimitive(style string, explode bool, paramName string, paramLocation ParamLocation, value interface{}) (string, error) {
|
||||
strVal, err := primitiveToString(value)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
var prefix string
|
||||
switch style {
|
||||
case "simple":
|
||||
case "label":
|
||||
prefix = "."
|
||||
case "matrix":
|
||||
prefix = fmt.Sprintf(";%s=", paramName)
|
||||
case "form":
|
||||
prefix = fmt.Sprintf("%s=", paramName)
|
||||
default:
|
||||
return "", fmt.Errorf("unsupported style '%s'", style)
|
||||
}
|
||||
return prefix + escapeParameterString(strVal, paramLocation), nil
|
||||
}
|
||||
|
||||
// Converts a primitive value to a string. We need to do this based on the
|
||||
// Kind of an interface, not the Type to work with aliased types.
|
||||
func primitiveToString(value interface{}) (string, error) {
|
||||
var output string
|
||||
|
||||
// Values may come in by pointer for optionals, so make sure to dereferene.
|
||||
v := reflect.Indirect(reflect.ValueOf(value))
|
||||
t := v.Type()
|
||||
kind := t.Kind()
|
||||
|
||||
switch kind {
|
||||
case reflect.Int8, reflect.Int32, reflect.Int64, reflect.Int:
|
||||
output = strconv.FormatInt(v.Int(), 10)
|
||||
case reflect.Float64:
|
||||
output = strconv.FormatFloat(v.Float(), 'f', -1, 64)
|
||||
case reflect.Float32:
|
||||
output = strconv.FormatFloat(v.Float(), 'f', -1, 32)
|
||||
case reflect.Bool:
|
||||
if v.Bool() {
|
||||
output = "true"
|
||||
} else {
|
||||
output = "false"
|
||||
}
|
||||
case reflect.String:
|
||||
output = v.String()
|
||||
default:
|
||||
return "", fmt.Errorf("unsupported type %s", reflect.TypeOf(value).String())
|
||||
}
|
||||
return output, nil
|
||||
}
|
||||
|
||||
// This function escapes a parameter value bas on the location of that parameter.
|
||||
// Query params and path params need different kinds of escaping, while header
|
||||
// and cookie params seem not to need escaping.
|
||||
func escapeParameterString(value string, paramLocation ParamLocation) string {
|
||||
switch paramLocation {
|
||||
case ParamLocationQuery:
|
||||
return url.QueryEscape(value)
|
||||
case ParamLocationPath:
|
||||
return url.PathEscape(value)
|
||||
default:
|
||||
return value
|
||||
}
|
||||
}
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
package types
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"time"
|
||||
)
|
||||
|
||||
const DateFormat = "2006-01-02"
|
||||
|
||||
type Date struct {
|
||||
time.Time
|
||||
}
|
||||
|
||||
func (d Date) MarshalJSON() ([]byte, error) {
|
||||
return json.Marshal(d.Time.Format(DateFormat))
|
||||
}
|
||||
|
||||
func (d *Date) UnmarshalJSON(data []byte) error {
|
||||
var dateStr string
|
||||
err := json.Unmarshal(data, &dateStr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
parsed, err := time.Parse(DateFormat, dateStr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
d.Time = parsed
|
||||
return nil
|
||||
}
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
package types
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
)
|
||||
|
||||
type Email string
|
||||
|
||||
func (e Email) MarshalJSON() ([]byte, error) {
|
||||
if !emailRegex.MatchString(string(e)) {
|
||||
return nil, errors.New("email: failed to pass regex validation")
|
||||
}
|
||||
return json.Marshal(string(e))
|
||||
}
|
||||
|
||||
func (e *Email) UnmarshalJSON(data []byte) error {
|
||||
var s string
|
||||
if err := json.Unmarshal(data, &s); err != nil {
|
||||
return err
|
||||
}
|
||||
if !emailRegex.MatchString(s) {
|
||||
return errors.New("email: failed to pass regex validation")
|
||||
}
|
||||
*e = Email(s)
|
||||
return nil
|
||||
}
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
package types
|
||||
|
||||
import "regexp"
|
||||
|
||||
const (
|
||||
emailRegexString = "^(?:(?:(?:(?:[a-zA-Z]|\\d|[!#\\$%&'\\*\\+\\-\\/=\\?\\^_`{\\|}~]|[\\x{00A0}-\\x{D7FF}\\x{F900}-\\x{FDCF}\\x{FDF0}-\\x{FFEF}])+(?:\\.([a-zA-Z]|\\d|[!#\\$%&'\\*\\+\\-\\/=\\?\\^_`{\\|}~]|[\\x{00A0}-\\x{D7FF}\\x{F900}-\\x{FDCF}\\x{FDF0}-\\x{FFEF}])+)*)|(?:(?:\\x22)(?:(?:(?:(?:\\x20|\\x09)*(?:\\x0d\\x0a))?(?:\\x20|\\x09)+)?(?:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x7f]|\\x21|[\\x23-\\x5b]|[\\x5d-\\x7e]|[\\x{00A0}-\\x{D7FF}\\x{F900}-\\x{FDCF}\\x{FDF0}-\\x{FFEF}])|(?:(?:[\\x01-\\x09\\x0b\\x0c\\x0d-\\x7f]|[\\x{00A0}-\\x{D7FF}\\x{F900}-\\x{FDCF}\\x{FDF0}-\\x{FFEF}]))))*(?:(?:(?:\\x20|\\x09)*(?:\\x0d\\x0a))?(\\x20|\\x09)+)?(?:\\x22))))@(?:(?:(?:[a-zA-Z]|\\d|[\\x{00A0}-\\x{D7FF}\\x{F900}-\\x{FDCF}\\x{FDF0}-\\x{FFEF}])|(?:(?:[a-zA-Z]|\\d|[\\x{00A0}-\\x{D7FF}\\x{F900}-\\x{FDCF}\\x{FDF0}-\\x{FFEF}])(?:[a-zA-Z]|\\d|-|\\.|~|[\\x{00A0}-\\x{D7FF}\\x{F900}-\\x{FDCF}\\x{FDF0}-\\x{FFEF}])*(?:[a-zA-Z]|\\d|[\\x{00A0}-\\x{D7FF}\\x{F900}-\\x{FDCF}\\x{FDF0}-\\x{FFEF}])))\\.)+(?:(?:[a-zA-Z]|[\\x{00A0}-\\x{D7FF}\\x{F900}-\\x{FDCF}\\x{FDF0}-\\x{FFEF}])|(?:(?:[a-zA-Z]|[\\x{00A0}-\\x{D7FF}\\x{F900}-\\x{FDCF}\\x{FDF0}-\\x{FFEF}])(?:[a-zA-Z]|\\d|-|\\.|~|[\\x{00A0}-\\x{D7FF}\\x{F900}-\\x{FDCF}\\x{FDF0}-\\x{FFEF}])*(?:[a-zA-Z]|[\\x{00A0}-\\x{D7FF}\\x{F900}-\\x{FDCF}\\x{FDF0}-\\x{FFEF}])))\\.?$"
|
||||
)
|
||||
|
||||
var (
|
||||
emailRegex = regexp.MustCompile(emailRegexString)
|
||||
)
|
||||
@@ -0,0 +1,20 @@
|
||||
# Binaries for programs and plugins
|
||||
*.exe
|
||||
*.bat
|
||||
*.exe~
|
||||
*.dll
|
||||
*.so
|
||||
*.dylib
|
||||
|
||||
# Test binary, built with `go test -c`
|
||||
*.test
|
||||
|
||||
# Output of the go coverage tool, specifically when used with LiteIDE
|
||||
*.out
|
||||
|
||||
# Dependency directories (remove the comment below to include it)
|
||||
# vendor/
|
||||
|
||||
# IntelliJ IDEA
|
||||
.IDEA
|
||||
*.IML
|
||||
+276
@@ -0,0 +1,276 @@
|
||||
## 2.12.2 [2023-01-26]
|
||||
### Bug fixes
|
||||
- [#368](https://github.com/influxdata/influxdb-client-go/pull/368) Allowing proxy from environment variable
|
||||
|
||||
## 2.12.1 [2022-12-01]
|
||||
### Bug fixes
|
||||
- [#363](https://github.com/influxdata/influxdb-client-go/pull/363) Generated server stubs return also error message from InfluxDB 1.x forward compatible API.
|
||||
- [#364](https://github.com/influxdata/influxdb-client-go/pull/364) Fixed panic when retrying over a long period without a server connection.
|
||||
|
||||
### Documentation
|
||||
- [#366](https://github.com/influxdata/influxdb-client-go/pull/366) Readme improvements:
|
||||
- Added GOPATH installation description
|
||||
- Added error handling to Basic Example.
|
||||
|
||||
## 2.12.0 [2022-10-27]
|
||||
### Features
|
||||
- [#358](https://github.com/influxdata/influxdb-client-go/pull/358):
|
||||
- Added possibility to set an application name, which will be part of the User-Agent HTTP header:
|
||||
- Set using `Options.SetApplicationName`
|
||||
- Warning message is written to log if an application name is not set
|
||||
- This may change to be logged as an error in a future release
|
||||
- Added example how to fully override `User-Agent` header using `Doer` interface
|
||||
|
||||
### Bug fixes
|
||||
- [#359](https://github.com/influxdata/influxdb-client-go/pull/359) `WriteAPIBlocking.Flush()` correctly returns nil error.
|
||||
|
||||
## 2.11.0 [2022-09-29]
|
||||
### Features
|
||||
- [#353](https://github.com/influxdata/influxdb-client-go/pull/353) Simplify generated code.
|
||||
- [#353](https://github.com/influxdata/influxdb-client-go/pull/353) Regenerate code from swagger.
|
||||
- [#355](https://github.com/influxdata/influxdb-client-go/pull/355) Upgrade of lib gopkg.in/yaml from v2 to v3
|
||||
|
||||
### Bug fixes
|
||||
- [#354](https://github.com/influxdata/influxdb-client-go/pull/354) More efficient synchronization in WriteAPIBlocking.
|
||||
|
||||
### Breaking change
|
||||
- [#353](https://github.com/influxdata/influxdb-client-go/pull/353):
|
||||
- Interface `Client` has been extended with `APIClient()` function.
|
||||
- The generated client API changed:
|
||||
- Function names are simplified (was `PostDBRPWithResponse`, now `PostDBRP`)
|
||||
- All functions now accept a context and a single wrapper structure with request body and HTTP parameters
|
||||
- The functions return deserialized response body or an error (it was a response wrapper with a status code that had to be then validated)
|
||||
|
||||
## 2.10.0 [2022-08-25]
|
||||
### Features
|
||||
- [#348](https://github.com/influxdata/influxdb-client-go/pull/348) Added `write.Options.Consitency` parameter to support InfluxDB Enterprise.
|
||||
- [#350](https://github.com/influxdata/influxdb-client-go/pull/350) Added support for implicit batching to `WriteAPIBlocking`. It's off by default, enabled by `EnableBatching()`.
|
||||
|
||||
### Bug fixes
|
||||
- [#349](https://github.com/influxdata/influxdb-client-go/pull/349) Skip retrying on specific write errors (mostly partial write error).
|
||||
|
||||
### Breaking change
|
||||
- [#350](https://github.com/influxdata/influxdb-client-go/pull/350) Interface `WriteAPIBlocking` is extend with `EnableBatching()` and `Flush()`.
|
||||
|
||||
## 2.9.2 [2022-07-29]
|
||||
### Bug fixes
|
||||
- [#341](https://github.com/influxdata/influxdb-client-go/pull/341) Changing logging level of messages about discarding batch to Error.
|
||||
- [#344](https://github.com/influxdata/influxdb-client-go/pull/344) `WriteAPI.Flush()` writes also batches from the retry queue.
|
||||
|
||||
### Test
|
||||
- [#345](https://github.com/influxdata/influxdb-client-go/pul/345) Added makefile for simplifying testing from command line.
|
||||
|
||||
## 2.9.1 [2022-06-24]
|
||||
### Bug fixes
|
||||
- [#332](https://github.com/influxdata/influxdb-client-go/pull/332) Retry strategy drops expired batches as soon as they expire.
|
||||
- [#335](https://github.com/influxdata/influxdb-client-go/pull/335) Retry strategy keeps max retry delay for new batches.
|
||||
|
||||
## 2.9.0 [2022-05-20]
|
||||
### Features
|
||||
- [#323](https://github.com/influxdata/influxdb-client-go/pull/323) Added `TasksAPI.CreateTaskByFlux` to allow full control of task script.
|
||||
- [#328](https://github.com/influxdata/influxdb-client-go/pull/328) Added `Client.SetupWithToken` allowing to specify a custom token.
|
||||
|
||||
### Bug fixes
|
||||
- [#324](https://github.com/influxdata/influxdb-client-go/pull/324) Non-empty error channel will not block writes
|
||||
|
||||
## 2.8.2 [2022-04-19]
|
||||
### Bug fixes
|
||||
- [#319](https://github.com/influxdata/influxdb-client-go/pull/319) Synchronize `WriteAPIImpl.Close` to prevent panic when closing client by multiple go-routines.
|
||||
|
||||
## 2.8.1 [2022-03-21]
|
||||
### Bug fixes
|
||||
- [#311](https://github.com/influxdata/influxdb-client-go/pull/311) Correctly unwrapping http.Error from Server API calls
|
||||
- [#315](https://github.com/influxdata/influxdb-client-go/pull/315) Masking authorization token in log
|
||||
|
||||
## 2.8.0 [2022-02-18]
|
||||
### Features
|
||||
- [#304](https://github.com/influxdata/influxdb-client-go/pull/304) Added public constructor for `QueryTableResult`
|
||||
- [#307](https://github.com/influxdata/influxdb-client-go/pull/307) Synced generated server API with the latest [oss.yml](https://github.com/influxdata/openapi/blob/master/contracts/oss.yml).
|
||||
- [#308](https://github.com/influxdata/influxdb-client-go/pull/308) Added Flux query parameters. Supported by InfluxDB Cloud only now.
|
||||
- [#308](https://github.com/influxdata/influxdb-client-go/pull/308) Go 1.17 is required
|
||||
|
||||
## 2.7.0[2022-01-20]
|
||||
### Features
|
||||
- [#297](https://github.com/influxdata/influxdb-client-go/pull/297),[#298](https://github.com/influxdata/influxdb-client-go/pull/298) Optimized `WriteRecord` of [WriteAPIBlocking](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api#WriteAPIBlocking). Custom batch can be written by single argument.
|
||||
|
||||
### Bug fixes
|
||||
- [#294](https://github.com/influxdata/influxdb-client-go/pull/294) `WritePoint` and `WriteRecord` of [WriteAPIBlocking](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api#WriteAPIBlocking) returns always full error information.
|
||||
- [300](https://github.com/influxdata/influxdb-client-go/pull/300) Closing the response body after write batch.
|
||||
- [302](https://github.com/influxdata/influxdb-client-go/pull/302) FluxRecord.Table() returns value of the table column.
|
||||
|
||||
## 2.6.0[2021-11-26]
|
||||
### Features
|
||||
- [#285](https://github.com/influxdata/influxdb-client-go/pull/285) Added *Client.Ping()* function as the only validation method available in both OSS and Cloud.
|
||||
- [#286](https://github.com/influxdata/influxdb-client-go/pull/286) Synced generated server API with the latest [oss.yml](https://github.com/influxdata/openapi/blob/master/contracts/oss.yml).
|
||||
- [#287](https://github.com/influxdata/influxdb-client-go/pull/287) Added *FluxRecord.Result()* function as a convenient way to retrieve the Flux result name of data.
|
||||
|
||||
### Bug fixes
|
||||
- [#285](https://github.com/influxdata/influxdb-client-go/pull/285) Functions *Client.Health()* and *Client.Ready()* correctly report an error when called against InfluxDB Cloud.
|
||||
|
||||
### Breaking change
|
||||
- [#285](https://github.com/influxdata/influxdb-client-go/pull/285) Function *Client.Ready()* now returns `*domain.Ready` with full uptime info.
|
||||
|
||||
## 2.5.1[2021-09-17]
|
||||
### Bug fixes
|
||||
- [#276](https://github.com/influxdata/influxdb-client-go/pull/276) Synchronized logging methods of _log.Logger_.
|
||||
|
||||
## 2.5.0 [2021-08-20]
|
||||
### Features
|
||||
- [#264](https://github.com/influxdata/influxdb-client-go/pull/264) Synced generated server API with the latest [oss.yml](https://github.com/influxdata/openapi/blob/master/contracts/oss.yml).
|
||||
- [#271](https://github.com/influxdata/influxdb-client-go/pull/271) Use exponential _random_ retry strategy
|
||||
- [#273](https://github.com/influxdata/influxdb-client-go/pull/273) Added `WriteFailedCallback` for `WriteAPI` allowing to be _synchronously_ notified about failed writes and decide on further batch processing.
|
||||
|
||||
### Bug fixes
|
||||
- [#269](https://github.com/influxdata/influxdb-client-go/pull/269) Synchronized setters of _log.Logger_ to allow concurrent usage
|
||||
- [#270](https://github.com/influxdata/influxdb-client-go/pull/270) Fixed duplicate `Content-Type` header in requests to managemet API
|
||||
|
||||
### Documentation
|
||||
- [#261](https://github.com/influxdata/influxdb-client-go/pull/261) Update Line Protocol document link to v2.0
|
||||
- [#274](https://github.com/influxdata/influxdb-client-go/pull/274) Documenting proxy configuration and HTTP redirects handling
|
||||
|
||||
## 2.4.0 [2021-06-04]
|
||||
### Features
|
||||
- [#256](https://github.com/influxdata/influxdb-client-go/pull/256) Allowing 'Doer' interface for HTTP requests
|
||||
|
||||
### Bug fixes
|
||||
- [#259](https://github.com/influxdata/influxdb-client-go/pull/259) Fixed leaking connection in case of not reading whole query result on TLS connection
|
||||
|
||||
|
||||
## 2.3.0 [2021-04-30]
|
||||
### Breaking change
|
||||
- [#253](https://github.com/influxdata/influxdb-client-go/pull/253) Interface 'Logger' extended with 'LogLevel() uint' getter.
|
||||
|
||||
### Features
|
||||
- [#241](https://github.com/influxdata/influxdb-client-go/pull/241),[#248](https://github.com/influxdata/influxdb-client-go/pull/248) Synced with InfluxDB 2.0.5 swagger:
|
||||
- Setup (onboarding) now sends correctly retentionDuration if specified
|
||||
- `RetentionRule` used in `Bucket` now contains `ShardGroupDurationSeconds` to specify the shard group duration.
|
||||
|
||||
### Documentation
|
||||
1. [#242](https://github.com/influxdata/influxdb-client-go/pull/242) Documentation improvements:
|
||||
- [Custom server API example](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2#example-Client-CustomServerAPICall) now shows how to create DBRP mapping
|
||||
- Improved documentation about concurrency
|
||||
1. [#251](https://github.com/influxdata/influxdb-client-go/pull/251) Fixed Readme.md formatting
|
||||
|
||||
### Bug fixes
|
||||
1. [#252](https://github.com/influxdata/influxdb-client-go/pull/252) Fixed panic when getting not present standard Flux columns
|
||||
1. [#253](https://github.com/influxdata/influxdb-client-go/pull/253) Conditional debug logging of buffers
|
||||
1. [#254](https://github.com/influxdata/influxdb-client-go/pull/254) Fixed golint pull
|
||||
|
||||
## 2.2.3 [2021-04-01]
|
||||
### Bug fixes
|
||||
1. [#236](https://github.com/influxdata/influxdb-client-go/pull/236) Setting MaxRetries to zero value disables retry strategy.
|
||||
1. [#239](https://github.com/influxdata/influxdb-client-go/pull/239) Blocking write client doesn't use retry handling.
|
||||
|
||||
## 2.2.2 [2021-01-29]
|
||||
### Bug fixes
|
||||
1. [#229](https://github.com/influxdata/influxdb-client-go/pull/229) Connection errors are also subject for retrying.
|
||||
|
||||
## 2.2.1 [2020-12-24]
|
||||
### Bug fixes
|
||||
1. [#220](https://github.com/influxdata/influxdb-client-go/pull/220) Fixed runtime error occurring when calling v2 API on v1 server.
|
||||
|
||||
### Documentation
|
||||
1. [#218](https://github.com/influxdata/influxdb-client-go/pull/218), [#221](https://github.com/influxdata/influxdb-client-go/pull/221), [#222](https://github.com/influxdata/influxdb-client-go/pull/222), Changed links leading to sources to point to API docs in Readme, fixed broken links to InfluxDB docs.
|
||||
|
||||
## 2.2.0 [2020-10-30]
|
||||
### Features
|
||||
1. [#206](https://github.com/influxdata/influxdb-client-go/pull/206) Adding TasksAPI for managing tasks and associated logs and runs.
|
||||
|
||||
### Bug fixes
|
||||
1. [#209](https://github.com/influxdata/influxdb-client-go/pull/209) Synchronizing access to the write service in WriteAPIBlocking.
|
||||
|
||||
## 2.1.0 [2020-10-02]
|
||||
### Features
|
||||
1. [#193](https://github.com/influxdata/influxdb-client-go/pull/193) Added authentication using username and password. See `UsersAPI.SignIn()` and `UsersAPI.SignOut()`
|
||||
1. [#204](https://github.com/influxdata/influxdb-client-go/pull/204) Synced with InfluxDB 2 RC0 swagger. Added pagination to Organizations API and `After` paging param to Buckets API.
|
||||
|
||||
### Bug fixes
|
||||
1. [#191](https://github.com/influxdata/influxdb-client-go/pull/191) Fixed QueryTableResult.Next() failed to parse boolean datatype.
|
||||
1. [#192](https://github.com/influxdata/influxdb-client-go/pull/192) Client.Close() closes idle connections of internally created HTTP client
|
||||
|
||||
### Documentation
|
||||
1. [#189](https://github.com/influxdata/influxdb-client-go/pull/189) Added clarification that server URL has to be the InfluxDB server base URL to API docs and all examples.
|
||||
1. [#196](https://github.com/influxdata/influxdb-client-go/pull/196) Changed default server port 9999 to 8086 in docs and examples
|
||||
1. [#200](https://github.com/influxdata/influxdb-client-go/pull/200) Fix example code in the Readme
|
||||
|
||||
## 2.0.1 [2020-08-14]
|
||||
### Bug fixes
|
||||
1. [#187](https://github.com/influxdata/influxdb-client-go/pull/187) Properly updated library for new major version.
|
||||
|
||||
## 2.0.0 [2020-08-14]
|
||||
### Breaking changes
|
||||
1. [#173](https://github.com/influxdata/influxdb-client-go/pull/173) Removed deprecated API.
|
||||
1. [#174](https://github.com/influxdata/influxdb-client-go/pull/174) Removed orgs labels API cause [it has been removed from the server API](https://github.com/influxdata/influxdb/pull/19104)
|
||||
1. [#175](https://github.com/influxdata/influxdb-client-go/pull/175) Removed WriteAPI.Close()
|
||||
|
||||
### Features
|
||||
1. [#165](https://github.com/influxdata/influxdb-client-go/pull/165) Allow overriding the http.Client for the http service.
|
||||
1. [#179](https://github.com/influxdata/influxdb-client-go/pull/179) Unifying retry strategy among InfluxDB 2 clients: added exponential backoff.
|
||||
1. [#180](https://github.com/influxdata/influxdb-client-go/pull/180) Provided public logger API to enable overriding logging. It is also possible to disable logging.
|
||||
1. [#181](https://github.com/influxdata/influxdb-client-go/pull/181) Exposed HTTP service to allow custom server API calls. Added example.
|
||||
|
||||
### Bug fixes
|
||||
1. [#175](https://github.com/influxdata/influxdb-client-go/pull/175) Fixed WriteAPIs management. Keeping single instance for each org and bucket pair.
|
||||
|
||||
### Documentation
|
||||
1. [#185](https://github.com/influxdata/influxdb-client-go/pull/185) DeleteAPI and sample WriteAPIBlocking wrapper for implicit batching
|
||||
|
||||
## 1.4.0 [2020-07-17]
|
||||
### Breaking changes
|
||||
1. [#156](https://github.com/influxdata/influxdb-client-go/pull/156) Fixing Go naming and code style violations:
|
||||
- Introducing new *API interfaces with proper name of types, methods and arguments.
|
||||
- This also affects the `Client` interface and the `Options` type.
|
||||
- Affected types and methods have been deprecated and they will be removed in the next release.
|
||||
|
||||
### Bug fixes
|
||||
1. [#152](https://github.com/influxdata/influxdb-client-go/pull/152) Allow connecting to server on a URL path
|
||||
1. [#154](https://github.com/influxdata/influxdb-client-go/pull/154) Use idiomatic go style for write channels (internal)
|
||||
1. [#155](https://github.com/influxdata/influxdb-client-go/pull/155) Fix panic in FindOrganizationByName in case of no permissions
|
||||
|
||||
|
||||
## 1.3.0 [2020-06-19]
|
||||
### Features
|
||||
1. [#131](https://github.com/influxdata/influxdb-client-go/pull/131) Labels API
|
||||
1. [#136](https://github.com/influxdata/influxdb-client-go/pull/136) Possibility to specify default tags
|
||||
1. [#138](https://github.com/influxdata/influxdb-client-go/pull/138) Fix errors from InfluxDB 1.8 being empty
|
||||
|
||||
### Bug fixes
|
||||
1. [#132](https://github.com/influxdata/influxdb-client-go/pull/132) Handle unsupported write type as string instead of generating panic
|
||||
1. [#134](https://github.com/influxdata/influxdb-client-go/pull/134) FluxQueryResult: support reordering of annotations
|
||||
|
||||
## 1.2.0 [2020-05-15]
|
||||
### Breaking Changes
|
||||
- [#107](https://github.com/influxdata/influxdb-client-go/pull/107) Renamed `InfluxDBClient` interface to `Client`, so the full name `influxdb2.Client` suits better to Go naming conventions
|
||||
- [#125](https://github.com/influxdata/influxdb-client-go/pull/125) `WriteApi`,`WriteApiBlocking`,`QueryApi` interfaces and related objects like `Point`, `FluxTableMetadata`, `FluxTableColumn`, `FluxRecord`, moved to the `api` ( and `api/write`, `api/query`) packages
|
||||
to provide consistent interface
|
||||
|
||||
### Features
|
||||
1. [#120](https://github.com/influxdata/influxdb-client-go/pull/120) Health check API
|
||||
1. [#122](https://github.com/influxdata/influxdb-client-go/pull/122) Delete API
|
||||
1. [#124](https://github.com/influxdata/influxdb-client-go/pull/124) Buckets API
|
||||
|
||||
### Bug fixes
|
||||
1. [#108](https://github.com/influxdata/influxdb-client-go/pull/108) Fix default retry interval doc
|
||||
1. [#110](https://github.com/influxdata/influxdb-client-go/pull/110) Allowing empty (nil) values in query result
|
||||
|
||||
### Documentation
|
||||
- [#112](https://github.com/influxdata/influxdb-client-go/pull/112) Clarify how to use client with InfluxDB 1.8+
|
||||
- [#115](https://github.com/influxdata/influxdb-client-go/pull/115) Doc and examples for reading write api errors
|
||||
|
||||
## 1.1.0 [2020-04-24]
|
||||
### Features
|
||||
1. [#100](https://github.com/influxdata/influxdb-client-go/pull/100) HTTP request timeout made configurable
|
||||
1. [#99](https://github.com/influxdata/influxdb-client-go/pull/99) Organizations API and Users API
|
||||
1. [#96](https://github.com/influxdata/influxdb-client-go/pull/96) Authorization API
|
||||
|
||||
### Docs
|
||||
1. [#101](https://github.com/influxdata/influxdb-client-go/pull/101) Added examples to API docs
|
||||
|
||||
## 1.0.0 [2020-04-01]
|
||||
### Core
|
||||
|
||||
- initial release of new client version
|
||||
|
||||
### APIs
|
||||
|
||||
- initial release of new client version
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2020-2021 Influxdata, Inc.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
artifacts_path := /tmp/artifacts
|
||||
|
||||
help:
|
||||
@echo 'Targets:'
|
||||
@echo ' all - runs lint, server, coverage'
|
||||
@echo ' lint - runs code style checks'
|
||||
@echo ' shorttest - runs unit and integration tests'
|
||||
@echo ' test - runs all tests, including e2e tests - requires running influxdb 2 server'
|
||||
@echo ' coverage - runs all tests, including e2e tests, with coverage report - requires running influxdb 2 server'
|
||||
@echo ' server - prepares InfluxDB in docker environment'
|
||||
|
||||
lint:
|
||||
go vet ./...
|
||||
go install honnef.co/go/tools/cmd/staticcheck@latest && staticcheck --checks='all' --tags e2e ./...
|
||||
go install golang.org/x/lint/golint@latest && golint ./...
|
||||
|
||||
shorttest:
|
||||
go test -race -v -count=1 ./...
|
||||
|
||||
test:
|
||||
go test -race -v -count=1 --tags e2e ./...
|
||||
|
||||
coverage:
|
||||
go install gotest.tools/gotestsum@latest && gotestsum --junitfile /tmp/test-results/unit-tests.xml -- -race -coverprofile=coverage.txt -covermode=atomic -coverpkg '.,./api/...,./internal/.../,./log/...' -tags e2e ./...
|
||||
if test ! -e $(artifacts_path); then mkdir $(artifacts_path); fi
|
||||
go tool cover -html=coverage.txt -o $(artifacts_path)/coverage.html
|
||||
|
||||
server:
|
||||
./scripts/influxdb-restart.sh
|
||||
|
||||
all: lint server coverage
|
||||
+711
@@ -0,0 +1,711 @@
|
||||
# InfluxDB Client Go
|
||||
|
||||
[](https://circleci.com/gh/influxdata/influxdb-client-go)
|
||||
[](https://codecov.io/gh/influxdata/influxdb-client-go)
|
||||
[](https://github.com/influxdata/influxdb-client-go/blob/master/LICENSE)
|
||||
[](https://www.influxdata.com/slack)
|
||||
|
||||
This repository contains the reference Go client for InfluxDB 2.
|
||||
|
||||
#### Note: Use this client library with InfluxDB 2.x and InfluxDB 1.8+ ([see details](#influxdb-18-api-compatibility)). For connecting to InfluxDB 1.7 or earlier instances, use the [influxdb1-go](https://github.com/influxdata/influxdb1-client) client library.
|
||||
|
||||
- [Features](#features)
|
||||
- [Documentation](#documentation)
|
||||
- [Examples](#examples)
|
||||
- [How To Use](#how-to-use)
|
||||
- [Installation](#installation)
|
||||
- [Basic Example](#basic-example)
|
||||
- [Writes in Detail](#writes)
|
||||
- [Queries in Detail](#queries)
|
||||
- [Parametrized Queries](#parametrized-queries)
|
||||
- [Concurrency](#concurrency)
|
||||
- [Proxy and redirects](#proxy-and-redirects)
|
||||
- [Checking Server State](#checking-server-state)
|
||||
- [InfluxDB 1.8 API compatibility](#influxdb-18-api-compatibility)
|
||||
- [Contributing](#contributing)
|
||||
- [License](#license)
|
||||
|
||||
## Features
|
||||
|
||||
- InfluxDB 2 client
|
||||
- Querying data
|
||||
- using the Flux language
|
||||
- into raw data, flux table representation
|
||||
- [How to queries](#queries)
|
||||
- Writing data using
|
||||
- [Line Protocol](https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/)
|
||||
- [Data Point](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api/write#Point)
|
||||
- Both [asynchronous](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api#WriteAPI) or [synchronous](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api#WriteAPIBlocking) ways
|
||||
- [How to writes](#writes)
|
||||
- InfluxDB 2 API
|
||||
- setup, ready, health
|
||||
- authotizations, users, organizations
|
||||
- buckets, delete
|
||||
- ...
|
||||
|
||||
## Documentation
|
||||
|
||||
This section contains links to the client library documentation.
|
||||
|
||||
- [Product documentation](https://docs.influxdata.com/influxdb/v2.0/tools/client-libraries/), [Getting Started](#how-to-use)
|
||||
- [Examples](#examples)
|
||||
- [API Reference](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2)
|
||||
- [Changelog](CHANGELOG.md)
|
||||
|
||||
### Examples
|
||||
|
||||
Examples for basic writing and querying data are shown below in this document
|
||||
|
||||
There are also other examples in the API docs:
|
||||
- [Client usage](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2?tab=doc#pkg-examples)
|
||||
- [Management APIs](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api?tab=doc#pkg-examples)
|
||||
|
||||
## How To Use
|
||||
|
||||
### Installation
|
||||
**Go 1.17** or later is required.
|
||||
|
||||
#### Go mod project
|
||||
1. Add the latest version of the client package to your project dependencies (go.mod).
|
||||
```sh
|
||||
go get github.com/influxdata/influxdb-client-go/v2
|
||||
```
|
||||
1. Add import `github.com/influxdata/influxdb-client-go/v2` to your source code.
|
||||
#### GOPATH project
|
||||
```sh
|
||||
go get github.com/influxdata/influxdb-client-go
|
||||
```
|
||||
Note: To have _go get_ in the GOPATH mode, the environment variable `GO111MODULE` must have the `off` value.
|
||||
|
||||
### Basic Example
|
||||
The following example demonstrates how to write data to InfluxDB 2 and read them back using the Flux language:
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Create a new client using an InfluxDB server base URL and an authentication token
|
||||
client := influxdb2.NewClient("http://localhost:8086", "my-token")
|
||||
// Use blocking write client for writes to desired bucket
|
||||
writeAPI := client.WriteAPIBlocking("my-org", "my-bucket")
|
||||
// Create point using full params constructor
|
||||
p := influxdb2.NewPoint("stat",
|
||||
map[string]string{"unit": "temperature"},
|
||||
map[string]interface{}{"avg": 24.5, "max": 45.0},
|
||||
time.Now())
|
||||
// write point immediately
|
||||
writeAPI.WritePoint(context.Background(), p)
|
||||
// Create point using fluent style
|
||||
p = influxdb2.NewPointWithMeasurement("stat").
|
||||
AddTag("unit", "temperature").
|
||||
AddField("avg", 23.2).
|
||||
AddField("max", 45.0).
|
||||
SetTime(time.Now())
|
||||
err := writeAPI.WritePoint(context.Background(), p)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
// Or write directly line protocol
|
||||
line := fmt.Sprintf("stat,unit=temperature avg=%f,max=%f", 23.5, 45.0)
|
||||
err = writeAPI.WriteRecord(context.Background(), line)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Get query client
|
||||
queryAPI := client.QueryAPI("my-org")
|
||||
// Get parser flux query result
|
||||
result, err := queryAPI.Query(context.Background(), `from(bucket:"my-bucket")|> range(start: -1h) |> filter(fn: (r) => r._measurement == "stat")`)
|
||||
if err == nil {
|
||||
// Use Next() to iterate over query result lines
|
||||
for result.Next() {
|
||||
// Observe when there is new grouping key producing new table
|
||||
if result.TableChanged() {
|
||||
fmt.Printf("table: %s\n", result.TableMetadata().String())
|
||||
}
|
||||
// read result
|
||||
fmt.Printf("row: %s\n", result.Record().String())
|
||||
}
|
||||
if result.Err() != nil {
|
||||
fmt.Printf("Query error: %s\n", result.Err().Error())
|
||||
}
|
||||
} else {
|
||||
panic(err)
|
||||
}
|
||||
// Ensures background processes finishes
|
||||
client.Close()
|
||||
}
|
||||
```
|
||||
### Options
|
||||
The InfluxDBClient uses set of options to configure behavior. These are available in the [Options](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2#Options) object
|
||||
Creating a client instance using
|
||||
```go
|
||||
client := influxdb2.NewClient("http://localhost:8086", "my-token")
|
||||
```
|
||||
will use the default options.
|
||||
|
||||
To set different configuration values, e.g. to set gzip compression and trust all server certificates, get default options
|
||||
and change what is needed:
|
||||
```go
|
||||
client := influxdb2.NewClientWithOptions("http://localhost:8086", "my-token",
|
||||
influxdb2.DefaultOptions().
|
||||
SetUseGZip(true).
|
||||
SetTLSConfig(&tls.Config{
|
||||
InsecureSkipVerify: true,
|
||||
}))
|
||||
```
|
||||
### Writes
|
||||
|
||||
Client offers two ways of writing, non-blocking and blocking.
|
||||
|
||||
### Non-blocking write client
|
||||
Non-blocking write client uses implicit batching. Data are asynchronously
|
||||
written to the underlying buffer and they are automatically sent to a server when the size of the write buffer reaches the batch size, default 5000, or the flush interval, default 1s, times out.
|
||||
Writes are automatically retried on server back pressure.
|
||||
|
||||
This write client also offers synchronous blocking method to ensure that write buffer is flushed and all pending writes are finished,
|
||||
see [Flush()](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api#WriteAPI.Flush) method.
|
||||
Always use [Close()](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2#Client.Close) method of the client to stop all background processes.
|
||||
|
||||
Asynchronous write client is recommended for frequent periodic writes.
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"time"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Create a new client using an InfluxDB server base URL and an authentication token
|
||||
// and set batch size to 20
|
||||
client := influxdb2.NewClientWithOptions("http://localhost:8086", "my-token",
|
||||
influxdb2.DefaultOptions().SetBatchSize(20))
|
||||
// Get non-blocking write client
|
||||
writeAPI := client.WriteAPI("my-org","my-bucket")
|
||||
// write some points
|
||||
for i := 0; i <100; i++ {
|
||||
// create point
|
||||
p := influxdb2.NewPoint(
|
||||
"system",
|
||||
map[string]string{
|
||||
"id": fmt.Sprintf("rack_%v", i%10),
|
||||
"vendor": "AWS",
|
||||
"hostname": fmt.Sprintf("host_%v", i%100),
|
||||
},
|
||||
map[string]interface{}{
|
||||
"temperature": rand.Float64() * 80.0,
|
||||
"disk_free": rand.Float64() * 1000.0,
|
||||
"disk_total": (i/10 + 1) * 1000000,
|
||||
"mem_total": (i/100 + 1) * 10000000,
|
||||
"mem_free": rand.Uint64(),
|
||||
},
|
||||
time.Now())
|
||||
// write asynchronously
|
||||
writeAPI.WritePoint(p)
|
||||
}
|
||||
// Force all unwritten data to be sent
|
||||
writeAPI.Flush()
|
||||
// Ensures background processes finishes
|
||||
client.Close()
|
||||
}
|
||||
```
|
||||
### Handling of failed async writes
|
||||
WriteAPI by default continues with retrying of failed writes.
|
||||
Retried are automatically writes that fail on a connection failure or when server returns response HTTP status code >= 429.
|
||||
|
||||
Retrying algorithm uses random exponential strategy to set retry time.
|
||||
The delay for the next retry attempt is a random value in the interval _retryInterval * exponentialBase^(attempts)_ and _retryInterval * exponentialBase^(attempts+1)_.
|
||||
If writes of batch repeatedly fails, WriteAPI continues with retrying until _maxRetries_ is reached or the overall retry time of batch exceeds _maxRetryTime_.
|
||||
|
||||
The defaults parameters (part of the WriteOptions) are:
|
||||
- _retryInterval_=5,000ms
|
||||
- _exponentialBase_=2
|
||||
- _maxRetryDelay_=125,000ms
|
||||
- _maxRetries_=5
|
||||
- _maxRetryTime_=180,000ms
|
||||
|
||||
Retry delays are by default randomly distributed within the ranges:
|
||||
1. 5,000-10,000
|
||||
1. 10,000-20,000
|
||||
1. 20,000-40,000
|
||||
1. 40,000-80,000
|
||||
1. 80,000-125,000
|
||||
|
||||
Setting _retryInterval_ to 0 disables retry strategy and any failed write will discard the batch.
|
||||
|
||||
[WriteFailedCallback](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api#WriteFailedCallback) allows advanced controlling of retrying.
|
||||
It is synchronously notified in case async write fails.
|
||||
It controls further batch handling by its return value. If it returns `true`, WriteAPI continues with retrying of writes of this batch. Returned `false` means the batch should be discarded.
|
||||
|
||||
### Reading async errors
|
||||
WriteAPI automatically logs write errors. Use [Errors()](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api#WriteAPI.Errors) method, which returns the channel for reading errors occuring during async writes, for writing write error to a custom target:
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"time"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Create a new client using an InfluxDB server base URL and an authentication token
|
||||
client := influxdb2.NewClient("http://localhost:8086", "my-token")
|
||||
// Get non-blocking write client
|
||||
writeAPI := client.WriteAPI("my-org", "my-bucket")
|
||||
// Get errors channel
|
||||
errorsCh := writeAPI.Errors()
|
||||
// Create go proc for reading and logging errors
|
||||
go func() {
|
||||
for err := range errorsCh {
|
||||
fmt.Printf("write error: %s\n", err.Error())
|
||||
}
|
||||
}()
|
||||
// write some points
|
||||
for i := 0; i < 100; i++ {
|
||||
// create point
|
||||
p := influxdb2.NewPointWithMeasurement("stat").
|
||||
AddTag("id", fmt.Sprintf("rack_%v", i%10)).
|
||||
AddTag("vendor", "AWS").
|
||||
AddTag("hostname", fmt.Sprintf("host_%v", i%100)).
|
||||
AddField("temperature", rand.Float64()*80.0).
|
||||
AddField("disk_free", rand.Float64()*1000.0).
|
||||
AddField("disk_total", (i/10+1)*1000000).
|
||||
AddField("mem_total", (i/100+1)*10000000).
|
||||
AddField("mem_free", rand.Uint64()).
|
||||
SetTime(time.Now())
|
||||
// write asynchronously
|
||||
writeAPI.WritePoint(p)
|
||||
}
|
||||
// Force all unwritten data to be sent
|
||||
writeAPI.Flush()
|
||||
// Ensures background processes finishes
|
||||
client.Close()
|
||||
}
|
||||
```
|
||||
|
||||
### Blocking write client
|
||||
Blocking write client writes given point(s) synchronously. It doesn't do implicit batching. Batch is created from given set of points.
|
||||
Implicit batching can be enabled with `WriteAPIBlocking.EnableBatching()`.
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"time"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Create a new client using an InfluxDB server base URL and an authentication token
|
||||
client := influxdb2.NewClient("http://localhost:8086", "my-token")
|
||||
// Get blocking write client
|
||||
writeAPI := client.WriteAPIBlocking("my-org","my-bucket")
|
||||
// write some points
|
||||
for i := 0; i <100; i++ {
|
||||
// create data point
|
||||
p := influxdb2.NewPoint(
|
||||
"system",
|
||||
map[string]string{
|
||||
"id": fmt.Sprintf("rack_%v", i%10),
|
||||
"vendor": "AWS",
|
||||
"hostname": fmt.Sprintf("host_%v", i%100),
|
||||
},
|
||||
map[string]interface{}{
|
||||
"temperature": rand.Float64() * 80.0,
|
||||
"disk_free": rand.Float64() * 1000.0,
|
||||
"disk_total": (i/10 + 1) * 1000000,
|
||||
"mem_total": (i/100 + 1) * 10000000,
|
||||
"mem_free": rand.Uint64(),
|
||||
},
|
||||
time.Now())
|
||||
// write synchronously
|
||||
err := writeAPI.WritePoint(context.Background(), p)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}
|
||||
// Ensures background processes finishes
|
||||
client.Close()
|
||||
}
|
||||
```
|
||||
|
||||
### Queries
|
||||
Query client offers retrieving of query results to a parsed representation in a [QueryTableResult](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api#QueryTableResult) or to a raw string.
|
||||
|
||||
### QueryTableResult
|
||||
QueryTableResult offers comfortable way how to deal with flux query CSV response. It parses CSV stream into FluxTableMetaData, FluxColumn and FluxRecord objects
|
||||
for easy reading the result.
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Create a new client using an InfluxDB server base URL and an authentication token
|
||||
client := influxdb2.NewClient("http://localhost:8086", "my-token")
|
||||
// Get query client
|
||||
queryAPI := client.QueryAPI("my-org")
|
||||
// get QueryTableResult
|
||||
result, err := queryAPI.Query(context.Background(), `from(bucket:"my-bucket")|> range(start: -1h) |> filter(fn: (r) => r._measurement == "stat")`)
|
||||
if err == nil {
|
||||
// Iterate over query response
|
||||
for result.Next() {
|
||||
// Notice when group key has changed
|
||||
if result.TableChanged() {
|
||||
fmt.Printf("table: %s\n", result.TableMetadata().String())
|
||||
}
|
||||
// Access data
|
||||
fmt.Printf("value: %v\n", result.Record().Value())
|
||||
}
|
||||
// check for an error
|
||||
if result.Err() != nil {
|
||||
fmt.Printf("query parsing error: %s\n", result.Err().Error())
|
||||
}
|
||||
} else {
|
||||
panic(err)
|
||||
}
|
||||
// Ensures background processes finishes
|
||||
client.Close()
|
||||
}
|
||||
```
|
||||
|
||||
### Raw
|
||||
[QueryRaw()](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api#QueryAPI.QueryRaw) returns raw, unparsed, query result string and process it on your own. Returned csv format
|
||||
can be controlled by the third parameter, query dialect.
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Create a new client using an InfluxDB server base URL and an authentication token
|
||||
client := influxdb2.NewClient("http://localhost:8086", "my-token")
|
||||
// Get query client
|
||||
queryAPI := client.QueryAPI("my-org")
|
||||
// Query and get complete result as a string
|
||||
// Use default dialect
|
||||
result, err := queryAPI.QueryRaw(context.Background(), `from(bucket:"my-bucket")|> range(start: -1h) |> filter(fn: (r) => r._measurement == "stat")`, influxdb2.DefaultDialect())
|
||||
if err == nil {
|
||||
fmt.Println("QueryResult:")
|
||||
fmt.Println(result)
|
||||
} else {
|
||||
panic(err)
|
||||
}
|
||||
// Ensures background processes finishes
|
||||
client.Close()
|
||||
}
|
||||
```
|
||||
### Parametrized Queries
|
||||
InfluxDB Cloud supports [Parameterized Queries](https://docs.influxdata.com/influxdb/cloud/query-data/parameterized-queries/)
|
||||
that let you dynamically change values in a query using the InfluxDB API. Parameterized queries make Flux queries more
|
||||
reusable and can also be used to help prevent injection attacks.
|
||||
|
||||
InfluxDB Cloud inserts the params object into the Flux query as a Flux record named `params`. Use dot or bracket
|
||||
notation to access parameters in the `params` record in your Flux query. Parameterized Flux queries support only `int`
|
||||
, `float`, and `string` data types. To convert the supported data types into
|
||||
other [Flux basic data types, use Flux type conversion functions](https://docs.influxdata.com/influxdb/cloud/query-data/parameterized-queries/#supported-parameter-data-types).
|
||||
|
||||
Query parameters can be passed as a struct or map. Param values can be only simple types or `time.Time`.
|
||||
The name of the parameter represented by a struct field can be specified by JSON annotation.
|
||||
|
||||
Parameterized query example:
|
||||
> :warning: Parameterized Queries are supported only in InfluxDB Cloud. There is no support in InfluxDB OSS currently.
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Create a new client using an InfluxDB server base URL and an authentication token
|
||||
client := influxdb2.NewClient("http://localhost:8086", "my-token")
|
||||
// Get query client
|
||||
queryAPI := client.QueryAPI("my-org")
|
||||
// Define parameters
|
||||
parameters := struct {
|
||||
Start string `json:"start"`
|
||||
Field string `json:"field"`
|
||||
Value float64 `json:"value"`
|
||||
}{
|
||||
"-1h",
|
||||
"temperature",
|
||||
25,
|
||||
}
|
||||
// Query with parameters
|
||||
query := `from(bucket:"my-bucket")
|
||||
|> range(start: duration(params.start))
|
||||
|> filter(fn: (r) => r._measurement == "stat")
|
||||
|> filter(fn: (r) => r._field == params.field)
|
||||
|> filter(fn: (r) => r._value > params.value)`
|
||||
|
||||
// Get result
|
||||
result, err := queryAPI.QueryWithParams(context.Background(), query, parameters)
|
||||
if err == nil {
|
||||
// Iterate over query response
|
||||
for result.Next() {
|
||||
// Notice when group key has changed
|
||||
if result.TableChanged() {
|
||||
fmt.Printf("table: %s\n", result.TableMetadata().String())
|
||||
}
|
||||
// Access data
|
||||
fmt.Printf("value: %v\n", result.Record().Value())
|
||||
}
|
||||
// check for an error
|
||||
if result.Err() != nil {
|
||||
fmt.Printf("query parsing error: %s\n", result.Err().Error())
|
||||
}
|
||||
} else {
|
||||
panic(err)
|
||||
}
|
||||
// Ensures background processes finishes
|
||||
client.Close()
|
||||
}
|
||||
```
|
||||
|
||||
### Concurrency
|
||||
InfluxDB Go Client can be used in a concurrent environment. All its functions are thread-safe.
|
||||
|
||||
The best practise is to use a single `Client` instance per server URL. This ensures optimized resources usage,
|
||||
most importantly reusing HTTP connections.
|
||||
|
||||
For efficient reuse of HTTP resources among multiple clients, create an HTTP client and use `Options.SetHTTPClient()` for setting it to all clients:
|
||||
```go
|
||||
// Create HTTP client
|
||||
httpClient := &http.Client{
|
||||
Timeout: time.Second * time.Duration(60),
|
||||
Transport: &http.Transport{
|
||||
DialContext: (&net.Dialer{
|
||||
Timeout: 5 * time.Second,
|
||||
}).DialContext,
|
||||
TLSHandshakeTimeout: 5 * time.Second,
|
||||
TLSClientConfig: &tls.Config{
|
||||
InsecureSkipVerify: true,
|
||||
},
|
||||
MaxIdleConns: 100,
|
||||
MaxIdleConnsPerHost: 100,
|
||||
IdleConnTimeout: 90 * time.Second,
|
||||
},
|
||||
}
|
||||
// Client for server 1
|
||||
client1 := influxdb2.NewClientWithOptions("https://server:8086", "my-token", influxdb2.DefaultOptions().SetHTTPClient(httpClient))
|
||||
// Client for server 2
|
||||
client2 := influxdb2.NewClientWithOptions("https://server:9999", "my-token2", influxdb2.DefaultOptions().SetHTTPClient(httpClient))
|
||||
```
|
||||
|
||||
Client ensures that there is a single instance of each server API sub-client for the specific area. E.g. a single `WriteAPI` instance for each org/bucket pair,
|
||||
a single `QueryAPI` for each org.
|
||||
|
||||
Such a single API sub-client instance can be used concurrently:
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"math/rand"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
influxdb2 "github.com/influxdata/influxdb-client-go"
|
||||
"github.com/influxdata/influxdb-client-go/v2/api/write"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Create client
|
||||
client := influxdb2.NewClient("http://localhost:8086", "my-token")
|
||||
// Ensure closing the client
|
||||
defer client.Close()
|
||||
|
||||
// Get write client
|
||||
writeApi := client.WriteAPI("my-org", "my-bucket")
|
||||
|
||||
// Create channel for points feeding
|
||||
pointsCh := make(chan *write.Point, 200)
|
||||
|
||||
threads := 5
|
||||
|
||||
var wg sync.WaitGroup
|
||||
go func(points int) {
|
||||
for i := 0; i < points; i++ {
|
||||
p := influxdb2.NewPoint("meas",
|
||||
map[string]string{"tag": "tagvalue"},
|
||||
map[string]interface{}{"val1": rand.Int63n(1000), "val2": rand.Float64()*100.0 - 50.0},
|
||||
time.Now())
|
||||
pointsCh <- p
|
||||
}
|
||||
close(pointsCh)
|
||||
}(1000000)
|
||||
|
||||
// Launch write routines
|
||||
for t := 0; t < threads; t++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
for p := range pointsCh {
|
||||
writeApi.WritePoint(p)
|
||||
}
|
||||
wg.Done()
|
||||
}()
|
||||
}
|
||||
// Wait for writes complete
|
||||
wg.Wait()
|
||||
}
|
||||
```
|
||||
|
||||
### Proxy and redirects
|
||||
You can configure InfluxDB Go client behind a proxy in two ways:
|
||||
1. Using environment variable
|
||||
Set environment variable `HTTP_PROXY` (or `HTTPS_PROXY` based on the scheme of your server url).
|
||||
e.g. (linux) `export HTTP_PROXY=http://my-proxy:8080` or in Go code `os.Setenv("HTTP_PROXY","http://my-proxy:8080")`
|
||||
|
||||
1. Configure `http.Client` to use proxy<br>
|
||||
Create a custom `http.Client` with a proxy configuration:
|
||||
```go
|
||||
proxyUrl, err := url.Parse("http://my-proxy:8080")
|
||||
httpClient := &http.Client{
|
||||
Transport: &http.Transport{
|
||||
Proxy: http.ProxyURL(proxyUrl)
|
||||
}
|
||||
}
|
||||
client := influxdb2.NewClientWithOptions("http://localhost:8086", token, influxdb2.DefaultOptions().SetHTTPClient(httpClient))
|
||||
```
|
||||
|
||||
Client automatically follows HTTP redirects. The default redirect policy is to follow up to 10 consecutive requests.
|
||||
Due to a security reason _Authorization_ header is not forwarded when redirect leads to a different domain.
|
||||
To overcome this limitation you have to set a custom redirect handler:
|
||||
```go
|
||||
token := "my-token"
|
||||
|
||||
httpClient := &http.Client{
|
||||
CheckRedirect: func(req *http.Request, via []*http.Request) error {
|
||||
req.Header.Add("Authorization","Token " + token)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
client := influxdb2.NewClientWithOptions("http://localhost:8086", token, influxdb2.DefaultOptions().SetHTTPClient(httpClient))
|
||||
```
|
||||
|
||||
### Checking Server State
|
||||
There are three functions for checking whether a server is up and ready for communication:
|
||||
|
||||
| Function| Description | Availability |
|
||||
|:----------|:----------|:----------|
|
||||
| [Health()](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2#Client.Health) | Detailed info about the server status, along with version string | OSS |
|
||||
| [Ready()](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2#Client.Ready) | Server uptime info | OSS |
|
||||
| [Ping()](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2#Client.Ping) | Whether a server is up | OSS, Cloud |
|
||||
|
||||
Only the [Ping()](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2#Client.Ping) function works in InfluxDB Cloud server.
|
||||
|
||||
## InfluxDB 1.8 API compatibility
|
||||
|
||||
[InfluxDB 1.8.0 introduced forward compatibility APIs](https://docs.influxdata.com/influxdb/latest/tools/api/#influxdb-2-0-api-compatibility-endpoints) for InfluxDB 2.0. This allow you to easily move from InfluxDB 1.x to InfluxDB 2.0 Cloud or open source.
|
||||
|
||||
Client API usage differences summary:
|
||||
1. Use the form `username:password` for an **authentication token**. Example: `my-user:my-password`. Use an empty string (`""`) if the server doesn't require authentication.
|
||||
1. The organization parameter is not used. Use an empty string (`""`) where necessary.
|
||||
1. Use the form `database/retention-policy` where a **bucket** is required. Skip retention policy if the default retention policy should be used. Examples: `telegraf/autogen`, `telegraf`.
|
||||
|
||||
The following forward compatible APIs are available:
|
||||
|
||||
| API | Endpoint | Description |
|
||||
|:----------|:----------|:----------|
|
||||
| [WriteAPI](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api#WriteAPI) (also [WriteAPIBlocking](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api#WriteAPIBlocking))| [/api/v2/write](https://docs.influxdata.com/influxdb/v2.0/write-data/developer-tools/api/) | Write data to InfluxDB 1.8.0+ using the InfluxDB 2.0 API |
|
||||
| [QueryAPI](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2/api#QueryAPI) | [/api/v2/query](https://docs.influxdata.com/influxdb/v2.0/query-data/execute-queries/influx-api/) | Query data in InfluxDB 1.8.0+ using the InfluxDB 2.0 API and [Flux](https://docs.influxdata.com/flux/latest/) endpoint should be enabled by the [`flux-enabled` option](https://docs.influxdata.com/influxdb/v1.8/administration/config/#flux-enabled-false)
|
||||
| [Health()](https://pkg.go.dev/github.com/influxdata/influxdb-client-go/v2#Client.Health) | [/health](https://docs.influxdata.com/influxdb/v2.0/api/#tag/Health) | Check the health of your InfluxDB instance |
|
||||
|
||||
|
||||
### Example
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2"
|
||||
)
|
||||
|
||||
func main() {
|
||||
userName := "my-user"
|
||||
password := "my-password"
|
||||
// Create a new client using an InfluxDB server base URL and an authentication token
|
||||
// For authentication token supply a string in the form: "username:password" as a token. Set empty value for an unauthenticated server
|
||||
client := influxdb2.NewClient("http://localhost:8086", fmt.Sprintf("%s:%s",userName, password))
|
||||
// Get the blocking write client
|
||||
// Supply a string in the form database/retention-policy as a bucket. Skip retention policy for the default one, use just a database name (without the slash character)
|
||||
// Org name is not used
|
||||
writeAPI := client.WriteAPIBlocking("", "test/autogen")
|
||||
// create point using full params constructor
|
||||
p := influxdb2.NewPoint("stat",
|
||||
map[string]string{"unit": "temperature"},
|
||||
map[string]interface{}{"avg": 24.5, "max": 45},
|
||||
time.Now())
|
||||
// Write data
|
||||
err := writeAPI.WritePoint(context.Background(), p)
|
||||
if err != nil {
|
||||
fmt.Printf("Write error: %s\n", err.Error())
|
||||
}
|
||||
|
||||
// Get query client. Org name is not used
|
||||
queryAPI := client.QueryAPI("")
|
||||
// Supply string in a form database/retention-policy as a bucket. Skip retention policy for the default one, use just a database name (without the slash character)
|
||||
result, err := queryAPI.Query(context.Background(), `from(bucket:"test")|> range(start: -1h) |> filter(fn: (r) => r._measurement == "stat")`)
|
||||
if err == nil {
|
||||
for result.Next() {
|
||||
if result.TableChanged() {
|
||||
fmt.Printf("table: %s\n", result.TableMetadata().String())
|
||||
}
|
||||
fmt.Printf("row: %s\n", result.Record().String())
|
||||
}
|
||||
if result.Err() != nil {
|
||||
fmt.Printf("Query error: %s\n", result.Err().Error())
|
||||
}
|
||||
} else {
|
||||
fmt.Printf("Query error: %s\n", err.Error())
|
||||
}
|
||||
// Close client
|
||||
client.Close()
|
||||
}
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
If you would like to contribute code you can do through GitHub by forking the repository and sending a pull request into the `master` branch.
|
||||
|
||||
## License
|
||||
|
||||
The InfluxDB 2 Go Client is released under the [MIT License](https://opensource.org/licenses/MIT).
|
||||
+127
@@ -0,0 +1,127 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2/domain"
|
||||
)
|
||||
|
||||
// AuthorizationsAPI provides methods for organizing Authorization in a InfluxDB server
|
||||
type AuthorizationsAPI interface {
|
||||
// GetAuthorizations returns all authorizations
|
||||
GetAuthorizations(ctx context.Context) (*[]domain.Authorization, error)
|
||||
// FindAuthorizationsByUserName returns all authorizations for given userName
|
||||
FindAuthorizationsByUserName(ctx context.Context, userName string) (*[]domain.Authorization, error)
|
||||
// FindAuthorizationsByUserID returns all authorizations for given userID
|
||||
FindAuthorizationsByUserID(ctx context.Context, userID string) (*[]domain.Authorization, error)
|
||||
// FindAuthorizationsByOrgName returns all authorizations for given organization name
|
||||
FindAuthorizationsByOrgName(ctx context.Context, orgName string) (*[]domain.Authorization, error)
|
||||
// FindAuthorizationsByOrgID returns all authorizations for given organization id
|
||||
FindAuthorizationsByOrgID(ctx context.Context, orgID string) (*[]domain.Authorization, error)
|
||||
// CreateAuthorization creates new authorization
|
||||
CreateAuthorization(ctx context.Context, authorization *domain.Authorization) (*domain.Authorization, error)
|
||||
// CreateAuthorizationWithOrgID creates new authorization with given permissions scoped to given orgID
|
||||
CreateAuthorizationWithOrgID(ctx context.Context, orgID string, permissions []domain.Permission) (*domain.Authorization, error)
|
||||
// UpdateAuthorizationStatus updates status of authorization
|
||||
UpdateAuthorizationStatus(ctx context.Context, authorization *domain.Authorization, status domain.AuthorizationUpdateRequestStatus) (*domain.Authorization, error)
|
||||
// UpdateAuthorizationStatusWithID updates status of authorization with authID
|
||||
UpdateAuthorizationStatusWithID(ctx context.Context, authID string, status domain.AuthorizationUpdateRequestStatus) (*domain.Authorization, error)
|
||||
// DeleteAuthorization deletes authorization
|
||||
DeleteAuthorization(ctx context.Context, authorization *domain.Authorization) error
|
||||
// DeleteAuthorization deletes authorization with authID
|
||||
DeleteAuthorizationWithID(ctx context.Context, authID string) error
|
||||
}
|
||||
|
||||
// authorizationsAPI implements AuthorizationsAPI
|
||||
type authorizationsAPI struct {
|
||||
apiClient *domain.Client
|
||||
}
|
||||
|
||||
// NewAuthorizationsAPI creates new instance of AuthorizationsAPI
|
||||
func NewAuthorizationsAPI(apiClient *domain.Client) AuthorizationsAPI {
|
||||
return &authorizationsAPI{
|
||||
apiClient: apiClient,
|
||||
}
|
||||
}
|
||||
|
||||
func (a *authorizationsAPI) GetAuthorizations(ctx context.Context) (*[]domain.Authorization, error) {
|
||||
authQuery := &domain.GetAuthorizationsParams{}
|
||||
return a.listAuthorizations(ctx, authQuery)
|
||||
}
|
||||
|
||||
func (a *authorizationsAPI) FindAuthorizationsByUserName(ctx context.Context, userName string) (*[]domain.Authorization, error) {
|
||||
authQuery := &domain.GetAuthorizationsParams{User: &userName}
|
||||
return a.listAuthorizations(ctx, authQuery)
|
||||
}
|
||||
|
||||
func (a *authorizationsAPI) FindAuthorizationsByUserID(ctx context.Context, userID string) (*[]domain.Authorization, error) {
|
||||
authQuery := &domain.GetAuthorizationsParams{UserID: &userID}
|
||||
return a.listAuthorizations(ctx, authQuery)
|
||||
}
|
||||
|
||||
func (a *authorizationsAPI) FindAuthorizationsByOrgName(ctx context.Context, orgName string) (*[]domain.Authorization, error) {
|
||||
authQuery := &domain.GetAuthorizationsParams{Org: &orgName}
|
||||
return a.listAuthorizations(ctx, authQuery)
|
||||
}
|
||||
|
||||
func (a *authorizationsAPI) FindAuthorizationsByOrgID(ctx context.Context, orgID string) (*[]domain.Authorization, error) {
|
||||
authQuery := &domain.GetAuthorizationsParams{OrgID: &orgID}
|
||||
return a.listAuthorizations(ctx, authQuery)
|
||||
}
|
||||
|
||||
func (a *authorizationsAPI) listAuthorizations(ctx context.Context, query *domain.GetAuthorizationsParams) (*[]domain.Authorization, error) {
|
||||
response, err := a.apiClient.GetAuthorizations(ctx, query)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return response.Authorizations, nil
|
||||
}
|
||||
|
||||
func (a *authorizationsAPI) CreateAuthorization(ctx context.Context, authorization *domain.Authorization) (*domain.Authorization, error) {
|
||||
params := &domain.PostAuthorizationsAllParams{
|
||||
Body: domain.PostAuthorizationsJSONRequestBody{
|
||||
AuthorizationUpdateRequest: authorization.AuthorizationUpdateRequest,
|
||||
OrgID: authorization.OrgID,
|
||||
Permissions: authorization.Permissions,
|
||||
UserID: authorization.UserID,
|
||||
},
|
||||
}
|
||||
return a.apiClient.PostAuthorizations(ctx, params)
|
||||
}
|
||||
|
||||
func (a *authorizationsAPI) CreateAuthorizationWithOrgID(ctx context.Context, orgID string, permissions []domain.Permission) (*domain.Authorization, error) {
|
||||
status := domain.AuthorizationUpdateRequestStatusActive
|
||||
auth := &domain.Authorization{
|
||||
AuthorizationUpdateRequest: domain.AuthorizationUpdateRequest{Status: &status},
|
||||
OrgID: &orgID,
|
||||
Permissions: &permissions,
|
||||
}
|
||||
return a.CreateAuthorization(ctx, auth)
|
||||
}
|
||||
|
||||
func (a *authorizationsAPI) UpdateAuthorizationStatusWithID(ctx context.Context, authID string, status domain.AuthorizationUpdateRequestStatus) (*domain.Authorization, error) {
|
||||
params := &domain.PatchAuthorizationsIDAllParams{
|
||||
Body: domain.PatchAuthorizationsIDJSONRequestBody{Status: &status},
|
||||
AuthID: authID,
|
||||
}
|
||||
return a.apiClient.PatchAuthorizationsID(ctx, params)
|
||||
}
|
||||
|
||||
func (a *authorizationsAPI) UpdateAuthorizationStatus(ctx context.Context, authorization *domain.Authorization, status domain.AuthorizationUpdateRequestStatus) (*domain.Authorization, error) {
|
||||
return a.UpdateAuthorizationStatusWithID(ctx, *authorization.Id, status)
|
||||
}
|
||||
|
||||
func (a *authorizationsAPI) DeleteAuthorization(ctx context.Context, authorization *domain.Authorization) error {
|
||||
return a.DeleteAuthorizationWithID(ctx, *authorization.Id)
|
||||
}
|
||||
|
||||
func (a *authorizationsAPI) DeleteAuthorizationWithID(ctx context.Context, authID string) error {
|
||||
params := &domain.DeleteAuthorizationsIDAllParams{
|
||||
AuthID: authID,
|
||||
}
|
||||
return a.apiClient.DeleteAuthorizationsID(ctx, params)
|
||||
}
|
||||
+275
@@ -0,0 +1,275 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"github.com/influxdata/influxdb-client-go/v2/domain"
|
||||
)
|
||||
|
||||
// BucketsAPI provides methods for managing Buckets in a InfluxDB server.
|
||||
type BucketsAPI interface {
|
||||
// GetBuckets returns all buckets.
|
||||
// GetBuckets supports PagingOptions: Offset, Limit, After. Empty pagingOptions means the default paging (first 20 results).
|
||||
GetBuckets(ctx context.Context, pagingOptions ...PagingOption) (*[]domain.Bucket, error)
|
||||
// FindBucketByName returns a bucket found using bucketName.
|
||||
FindBucketByName(ctx context.Context, bucketName string) (*domain.Bucket, error)
|
||||
// FindBucketByID returns a bucket found using bucketID.
|
||||
FindBucketByID(ctx context.Context, bucketID string) (*domain.Bucket, error)
|
||||
// FindBucketsByOrgID returns buckets belonging to the organization with ID orgID.
|
||||
// FindBucketsByOrgID supports PagingOptions: Offset, Limit, After. Empty pagingOptions means the default paging (first 20 results).
|
||||
FindBucketsByOrgID(ctx context.Context, orgID string, pagingOptions ...PagingOption) (*[]domain.Bucket, error)
|
||||
// FindBucketsByOrgName returns buckets belonging to the organization with name orgName, with the specified paging. Empty pagingOptions means the default paging (first 20 results).
|
||||
FindBucketsByOrgName(ctx context.Context, orgName string, pagingOptions ...PagingOption) (*[]domain.Bucket, error)
|
||||
// CreateBucket creates a new bucket.
|
||||
CreateBucket(ctx context.Context, bucket *domain.Bucket) (*domain.Bucket, error)
|
||||
// CreateBucketWithName creates a new bucket with bucketName in organization org, with retention specified in rules. Empty rules means infinite retention.
|
||||
CreateBucketWithName(ctx context.Context, org *domain.Organization, bucketName string, rules ...domain.RetentionRule) (*domain.Bucket, error)
|
||||
// CreateBucketWithNameWithID creates a new bucket with bucketName in organization with orgID, with retention specified in rules. Empty rules means infinite retention.
|
||||
CreateBucketWithNameWithID(ctx context.Context, orgID, bucketName string, rules ...domain.RetentionRule) (*domain.Bucket, error)
|
||||
// UpdateBucket updates a bucket.
|
||||
UpdateBucket(ctx context.Context, bucket *domain.Bucket) (*domain.Bucket, error)
|
||||
// DeleteBucket deletes a bucket.
|
||||
DeleteBucket(ctx context.Context, bucket *domain.Bucket) error
|
||||
// DeleteBucketWithID deletes a bucket with bucketID.
|
||||
DeleteBucketWithID(ctx context.Context, bucketID string) error
|
||||
// GetMembers returns members of a bucket.
|
||||
GetMembers(ctx context.Context, bucket *domain.Bucket) (*[]domain.ResourceMember, error)
|
||||
// GetMembersWithID returns members of a bucket with bucketID.
|
||||
GetMembersWithID(ctx context.Context, bucketID string) (*[]domain.ResourceMember, error)
|
||||
// AddMember adds a member to a bucket.
|
||||
AddMember(ctx context.Context, bucket *domain.Bucket, user *domain.User) (*domain.ResourceMember, error)
|
||||
// AddMemberWithID adds a member with id memberID to a bucket with bucketID.
|
||||
AddMemberWithID(ctx context.Context, bucketID, memberID string) (*domain.ResourceMember, error)
|
||||
// RemoveMember removes a member from a bucket.
|
||||
RemoveMember(ctx context.Context, bucket *domain.Bucket, user *domain.User) error
|
||||
// RemoveMemberWithID removes a member with id memberID from a bucket with bucketID.
|
||||
RemoveMemberWithID(ctx context.Context, bucketID, memberID string) error
|
||||
// GetOwners returns owners of a bucket.
|
||||
GetOwners(ctx context.Context, bucket *domain.Bucket) (*[]domain.ResourceOwner, error)
|
||||
// GetOwnersWithID returns owners of a bucket with bucketID.
|
||||
GetOwnersWithID(ctx context.Context, bucketID string) (*[]domain.ResourceOwner, error)
|
||||
// AddOwner adds an owner to a bucket.
|
||||
AddOwner(ctx context.Context, bucket *domain.Bucket, user *domain.User) (*domain.ResourceOwner, error)
|
||||
// AddOwnerWithID adds an owner with id memberID to a bucket with bucketID.
|
||||
AddOwnerWithID(ctx context.Context, bucketID, memberID string) (*domain.ResourceOwner, error)
|
||||
// RemoveOwner removes an owner from a bucket.
|
||||
RemoveOwner(ctx context.Context, bucket *domain.Bucket, user *domain.User) error
|
||||
// RemoveOwnerWithID removes a member with id memberID from a bucket with bucketID.
|
||||
RemoveOwnerWithID(ctx context.Context, bucketID, memberID string) error
|
||||
}
|
||||
|
||||
// bucketsAPI implements BucketsAPI
|
||||
type bucketsAPI struct {
|
||||
apiClient *domain.Client
|
||||
}
|
||||
|
||||
// NewBucketsAPI creates new instance of BucketsAPI
|
||||
func NewBucketsAPI(apiClient *domain.Client) BucketsAPI {
|
||||
return &bucketsAPI{
|
||||
apiClient: apiClient,
|
||||
}
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) GetBuckets(ctx context.Context, pagingOptions ...PagingOption) (*[]domain.Bucket, error) {
|
||||
return b.getBuckets(ctx, nil, pagingOptions...)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) getBuckets(ctx context.Context, params *domain.GetBucketsParams, pagingOptions ...PagingOption) (*[]domain.Bucket, error) {
|
||||
if params == nil {
|
||||
params = &domain.GetBucketsParams{}
|
||||
}
|
||||
options := defaultPaging()
|
||||
for _, opt := range pagingOptions {
|
||||
opt(options)
|
||||
}
|
||||
if options.limit > 0 {
|
||||
params.Limit = &options.limit
|
||||
}
|
||||
params.Offset = &options.offset
|
||||
|
||||
response, err := b.apiClient.GetBuckets(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return response.Buckets, nil
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) FindBucketByName(ctx context.Context, bucketName string) (*domain.Bucket, error) {
|
||||
params := &domain.GetBucketsParams{Name: &bucketName}
|
||||
response, err := b.apiClient.GetBuckets(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if response.Buckets != nil && len(*response.Buckets) > 0 {
|
||||
return &(*response.Buckets)[0], nil
|
||||
}
|
||||
return nil, fmt.Errorf("bucket '%s' not found", bucketName)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) FindBucketByID(ctx context.Context, bucketID string) (*domain.Bucket, error) {
|
||||
params := &domain.GetBucketsIDAllParams{
|
||||
BucketID: bucketID,
|
||||
}
|
||||
return b.apiClient.GetBucketsID(ctx, params)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) FindBucketsByOrgID(ctx context.Context, orgID string, pagingOptions ...PagingOption) (*[]domain.Bucket, error) {
|
||||
params := &domain.GetBucketsParams{OrgID: &orgID}
|
||||
return b.getBuckets(ctx, params, pagingOptions...)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) FindBucketsByOrgName(ctx context.Context, orgName string, pagingOptions ...PagingOption) (*[]domain.Bucket, error) {
|
||||
params := &domain.GetBucketsParams{Org: &orgName}
|
||||
return b.getBuckets(ctx, params, pagingOptions...)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) createBucket(ctx context.Context, bucketReq *domain.PostBucketRequest) (*domain.Bucket, error) {
|
||||
params := &domain.PostBucketsAllParams{
|
||||
Body: domain.PostBucketsJSONRequestBody(*bucketReq),
|
||||
}
|
||||
return b.apiClient.PostBuckets(ctx, params)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) CreateBucket(ctx context.Context, bucket *domain.Bucket) (*domain.Bucket, error) {
|
||||
bucketReq := &domain.PostBucketRequest{
|
||||
Description: bucket.Description,
|
||||
Name: bucket.Name,
|
||||
OrgID: *bucket.OrgID,
|
||||
RetentionRules: &bucket.RetentionRules,
|
||||
Rp: bucket.Rp,
|
||||
}
|
||||
return b.createBucket(ctx, bucketReq)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) CreateBucketWithNameWithID(ctx context.Context, orgID, bucketName string, rules ...domain.RetentionRule) (*domain.Bucket, error) {
|
||||
rs := domain.RetentionRules(rules)
|
||||
bucket := &domain.PostBucketRequest{Name: bucketName, OrgID: orgID, RetentionRules: &rs}
|
||||
return b.createBucket(ctx, bucket)
|
||||
}
|
||||
func (b *bucketsAPI) CreateBucketWithName(ctx context.Context, org *domain.Organization, bucketName string, rules ...domain.RetentionRule) (*domain.Bucket, error) {
|
||||
return b.CreateBucketWithNameWithID(ctx, *org.Id, bucketName, rules...)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) DeleteBucket(ctx context.Context, bucket *domain.Bucket) error {
|
||||
return b.DeleteBucketWithID(ctx, *bucket.Id)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) DeleteBucketWithID(ctx context.Context, bucketID string) error {
|
||||
params := &domain.DeleteBucketsIDAllParams{
|
||||
BucketID: bucketID,
|
||||
}
|
||||
return b.apiClient.DeleteBucketsID(ctx, params)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) UpdateBucket(ctx context.Context, bucket *domain.Bucket) (*domain.Bucket, error) {
|
||||
params := &domain.PatchBucketsIDAllParams{
|
||||
Body: domain.PatchBucketsIDJSONRequestBody{
|
||||
Description: bucket.Description,
|
||||
Name: &bucket.Name,
|
||||
RetentionRules: retentionRulesToPatchRetentionRules(&bucket.RetentionRules),
|
||||
},
|
||||
BucketID: *bucket.Id,
|
||||
}
|
||||
return b.apiClient.PatchBucketsID(ctx, params)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) GetMembers(ctx context.Context, bucket *domain.Bucket) (*[]domain.ResourceMember, error) {
|
||||
return b.GetMembersWithID(ctx, *bucket.Id)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) GetMembersWithID(ctx context.Context, bucketID string) (*[]domain.ResourceMember, error) {
|
||||
params := &domain.GetBucketsIDMembersAllParams{
|
||||
BucketID: bucketID,
|
||||
}
|
||||
response, err := b.apiClient.GetBucketsIDMembers(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return response.Users, nil
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) AddMember(ctx context.Context, bucket *domain.Bucket, user *domain.User) (*domain.ResourceMember, error) {
|
||||
return b.AddMemberWithID(ctx, *bucket.Id, *user.Id)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) AddMemberWithID(ctx context.Context, bucketID, memberID string) (*domain.ResourceMember, error) {
|
||||
params := &domain.PostBucketsIDMembersAllParams{
|
||||
BucketID: bucketID,
|
||||
Body: domain.PostBucketsIDMembersJSONRequestBody{Id: memberID},
|
||||
}
|
||||
return b.apiClient.PostBucketsIDMembers(ctx, params)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) RemoveMember(ctx context.Context, bucket *domain.Bucket, user *domain.User) error {
|
||||
return b.RemoveMemberWithID(ctx, *bucket.Id, *user.Id)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) RemoveMemberWithID(ctx context.Context, bucketID, memberID string) error {
|
||||
params := &domain.DeleteBucketsIDMembersIDAllParams{
|
||||
BucketID: bucketID,
|
||||
UserID: memberID,
|
||||
}
|
||||
return b.apiClient.DeleteBucketsIDMembersID(ctx, params)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) GetOwners(ctx context.Context, bucket *domain.Bucket) (*[]domain.ResourceOwner, error) {
|
||||
return b.GetOwnersWithID(ctx, *bucket.Id)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) GetOwnersWithID(ctx context.Context, bucketID string) (*[]domain.ResourceOwner, error) {
|
||||
params := &domain.GetBucketsIDOwnersAllParams{
|
||||
BucketID: bucketID,
|
||||
}
|
||||
response, err := b.apiClient.GetBucketsIDOwners(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return response.Users, nil
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) AddOwner(ctx context.Context, bucket *domain.Bucket, user *domain.User) (*domain.ResourceOwner, error) {
|
||||
return b.AddOwnerWithID(ctx, *bucket.Id, *user.Id)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) AddOwnerWithID(ctx context.Context, bucketID, memberID string) (*domain.ResourceOwner, error) {
|
||||
params := &domain.PostBucketsIDOwnersAllParams{
|
||||
BucketID: bucketID,
|
||||
Body: domain.PostBucketsIDOwnersJSONRequestBody{Id: memberID},
|
||||
}
|
||||
return b.apiClient.PostBucketsIDOwners(ctx, params)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) RemoveOwner(ctx context.Context, bucket *domain.Bucket, user *domain.User) error {
|
||||
return b.RemoveOwnerWithID(ctx, *bucket.Id, *user.Id)
|
||||
}
|
||||
|
||||
func (b *bucketsAPI) RemoveOwnerWithID(ctx context.Context, bucketID, memberID string) error {
|
||||
params := &domain.DeleteBucketsIDOwnersIDAllParams{
|
||||
BucketID: bucketID,
|
||||
UserID: memberID,
|
||||
}
|
||||
return b.apiClient.DeleteBucketsIDOwnersID(ctx, params)
|
||||
}
|
||||
|
||||
func retentionRulesToPatchRetentionRules(rrs *domain.RetentionRules) *domain.PatchRetentionRules {
|
||||
if rrs == nil {
|
||||
return nil
|
||||
}
|
||||
prrs := make([]domain.PatchRetentionRule, len(*rrs))
|
||||
for i, rr := range *rrs {
|
||||
prrs[i] = domain.PatchRetentionRule{
|
||||
EverySeconds: rr.EverySeconds,
|
||||
ShardGroupDurationSeconds: rr.ShardGroupDurationSeconds,
|
||||
}
|
||||
if rr.Type != nil {
|
||||
rrt := domain.PatchRetentionRuleType(*rr.Type)
|
||||
prrs[i].Type = &rrt
|
||||
}
|
||||
}
|
||||
dprrs := domain.PatchRetentionRules(prrs)
|
||||
return &dprrs
|
||||
}
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"github.com/influxdata/influxdb-client-go/v2/domain"
|
||||
"time"
|
||||
)
|
||||
|
||||
// DeleteAPI provides methods for deleting time series data from buckets.
|
||||
// Deleted series are selected by the time range specified by start and stop arguments and optional predicate string which contains condition for selecting data for deletion, such as:
|
||||
// tag1="value1" and (tag2="value2" and tag3!="value3")
|
||||
// Empty predicate string means all data from the given time range will be deleted. See https://v2.docs.influxdata.com/v2.0/reference/syntax/delete-predicate/
|
||||
// for more info about predicate syntax.
|
||||
type DeleteAPI interface {
|
||||
// Delete deletes series selected by the time range specified by start and stop arguments and optional predicate string from the bucket bucket belonging to the organization org.
|
||||
Delete(ctx context.Context, org *domain.Organization, bucket *domain.Bucket, start, stop time.Time, predicate string) error
|
||||
// DeleteWithID deletes series selected by the time range specified by start and stop arguments and optional predicate string from the bucket with ID bucketID belonging to the organization with ID orgID.
|
||||
DeleteWithID(ctx context.Context, orgID, bucketID string, start, stop time.Time, predicate string) error
|
||||
// DeleteWithName deletes series selected by the time range specified by start and stop arguments and optional predicate string from the bucket with name bucketName belonging to the organization with name orgName.
|
||||
DeleteWithName(ctx context.Context, orgName, bucketName string, start, stop time.Time, predicate string) error
|
||||
}
|
||||
|
||||
// deleteAPI implements DeleteAPI
|
||||
type deleteAPI struct {
|
||||
apiClient *domain.Client
|
||||
}
|
||||
|
||||
// NewDeleteAPI creates new instance of DeleteAPI
|
||||
func NewDeleteAPI(apiClient *domain.Client) DeleteAPI {
|
||||
return &deleteAPI{
|
||||
apiClient: apiClient,
|
||||
}
|
||||
}
|
||||
|
||||
func (d *deleteAPI) delete(ctx context.Context, params *domain.PostDeleteParams, conditions *domain.DeletePredicateRequest) error {
|
||||
allParams := &domain.PostDeleteAllParams{
|
||||
PostDeleteParams: *params,
|
||||
Body: domain.PostDeleteJSONRequestBody(*conditions),
|
||||
}
|
||||
return d.apiClient.PostDelete(ctx, allParams)
|
||||
}
|
||||
|
||||
func (d *deleteAPI) Delete(ctx context.Context, org *domain.Organization, bucket *domain.Bucket, start, stop time.Time, predicate string) error {
|
||||
params := &domain.PostDeleteParams{
|
||||
OrgID: org.Id,
|
||||
BucketID: bucket.Id,
|
||||
}
|
||||
conditions := &domain.DeletePredicateRequest{
|
||||
Predicate: &predicate,
|
||||
Start: start,
|
||||
Stop: stop,
|
||||
}
|
||||
return d.delete(ctx, params, conditions)
|
||||
}
|
||||
|
||||
func (d *deleteAPI) DeleteWithID(ctx context.Context, orgID, bucketID string, start, stop time.Time, predicate string) error {
|
||||
params := &domain.PostDeleteParams{
|
||||
OrgID: &orgID,
|
||||
BucketID: &bucketID,
|
||||
}
|
||||
conditions := &domain.DeletePredicateRequest{
|
||||
Predicate: &predicate,
|
||||
Start: start,
|
||||
Stop: stop,
|
||||
}
|
||||
return d.delete(ctx, params, conditions)
|
||||
}
|
||||
|
||||
func (d *deleteAPI) DeleteWithName(ctx context.Context, orgName, bucketName string, start, stop time.Time, predicate string) error {
|
||||
params := &domain.PostDeleteParams{
|
||||
Org: &orgName,
|
||||
Bucket: &bucketName,
|
||||
}
|
||||
conditions := &domain.DeletePredicateRequest{
|
||||
Predicate: &predicate,
|
||||
Start: start,
|
||||
Stop: stop,
|
||||
}
|
||||
return d.delete(ctx, params, conditions)
|
||||
}
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package api provides clients for InfluxDB server APIs.
|
||||
package api
|
||||
+49
@@ -0,0 +1,49 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package http
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
// Error represent error response from InfluxDBServer or http error
|
||||
type Error struct {
|
||||
StatusCode int
|
||||
Code string
|
||||
Message string
|
||||
Err error
|
||||
RetryAfter uint
|
||||
}
|
||||
|
||||
// Error fulfils error interface
|
||||
func (e *Error) Error() string {
|
||||
switch {
|
||||
case e.Err != nil:
|
||||
return e.Err.Error()
|
||||
case e.Code != "" && e.Message != "":
|
||||
return fmt.Sprintf("%s: %s", e.Code, e.Message)
|
||||
default:
|
||||
return "Unexpected status code " + strconv.Itoa(e.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func (e *Error) Unwrap() error {
|
||||
if e.Err != nil {
|
||||
return e.Err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// NewError returns newly created Error initialised with nested error and default values
|
||||
func NewError(err error) *Error {
|
||||
return &Error{
|
||||
StatusCode: 0,
|
||||
Code: "",
|
||||
Message: "",
|
||||
Err: err,
|
||||
RetryAfter: 0,
|
||||
}
|
||||
}
|
||||
+139
@@ -0,0 +1,139 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package http
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"net"
|
||||
"net/http"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Options holds http configuration properties for communicating with InfluxDB server
|
||||
type Options struct {
|
||||
// HTTP client. Default is http.DefaultClient.
|
||||
httpClient *http.Client
|
||||
// doer is an http Doer - if set it overrides httpClient
|
||||
doer Doer
|
||||
// Flag whether http client was created internally
|
||||
ownClient bool
|
||||
// TLS configuration for secure connection. Default nil
|
||||
tlsConfig *tls.Config
|
||||
// HTTP request timeout in sec. Default 20
|
||||
httpRequestTimeout uint
|
||||
// Application name in the User-Agent HTTP header string
|
||||
appName string
|
||||
}
|
||||
|
||||
// HTTPClient returns the http.Client that is configured to be used
|
||||
// for HTTP requests. It will return the one that has been set using
|
||||
// SetHTTPClient or it will construct a default client using the
|
||||
// other configured options.
|
||||
// HTTPClient panics if SetHTTPDoer was called.
|
||||
func (o *Options) HTTPClient() *http.Client {
|
||||
if o.doer != nil {
|
||||
panic("HTTPClient called after SetHTTPDoer")
|
||||
}
|
||||
if o.httpClient == nil {
|
||||
o.httpClient = &http.Client{
|
||||
Timeout: time.Second * time.Duration(o.HTTPRequestTimeout()),
|
||||
Transport: &http.Transport{
|
||||
Proxy: http.ProxyFromEnvironment,
|
||||
DialContext: (&net.Dialer{
|
||||
Timeout: 5 * time.Second,
|
||||
}).DialContext,
|
||||
TLSHandshakeTimeout: 5 * time.Second,
|
||||
TLSClientConfig: o.TLSConfig(),
|
||||
MaxIdleConns: 100,
|
||||
MaxIdleConnsPerHost: 100,
|
||||
IdleConnTimeout: 90 * time.Second,
|
||||
},
|
||||
}
|
||||
o.ownClient = true
|
||||
}
|
||||
return o.httpClient
|
||||
}
|
||||
|
||||
// SetHTTPClient will configure the http.Client that is used
|
||||
// for HTTP requests. If set to nil, an HTTPClient will be
|
||||
// generated.
|
||||
//
|
||||
// Setting the HTTPClient will cause the other HTTP options
|
||||
// to be ignored.
|
||||
// In case of UsersAPI.SignIn() is used, HTTPClient.Jar will be used for storing session cookie.
|
||||
func (o *Options) SetHTTPClient(c *http.Client) *Options {
|
||||
o.httpClient = c
|
||||
o.ownClient = false
|
||||
return o
|
||||
}
|
||||
|
||||
// OwnHTTPClient returns true of HTTP client was created internally. False if it was set externally.
|
||||
func (o *Options) OwnHTTPClient() bool {
|
||||
return o.ownClient
|
||||
}
|
||||
|
||||
// Doer allows proving custom Do for HTTP operations
|
||||
type Doer interface {
|
||||
Do(*http.Request) (*http.Response, error)
|
||||
}
|
||||
|
||||
// SetHTTPDoer will configure the http.Client that is used
|
||||
// for HTTP requests. If set to nil, this has no effect.
|
||||
//
|
||||
// Setting the HTTPDoer will cause the other HTTP options
|
||||
// to be ignored.
|
||||
func (o *Options) SetHTTPDoer(d Doer) *Options {
|
||||
if d != nil {
|
||||
o.doer = d
|
||||
o.ownClient = false
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// HTTPDoer returns actual Doer if set, or http.Client
|
||||
func (o *Options) HTTPDoer() Doer {
|
||||
if o.doer != nil {
|
||||
return o.doer
|
||||
}
|
||||
return o.HTTPClient()
|
||||
}
|
||||
|
||||
// TLSConfig returns tls.Config
|
||||
func (o *Options) TLSConfig() *tls.Config {
|
||||
return o.tlsConfig
|
||||
}
|
||||
|
||||
// SetTLSConfig sets TLS configuration for secure connection
|
||||
func (o *Options) SetTLSConfig(tlsConfig *tls.Config) *Options {
|
||||
o.tlsConfig = tlsConfig
|
||||
return o
|
||||
}
|
||||
|
||||
// HTTPRequestTimeout returns HTTP request timeout
|
||||
func (o *Options) HTTPRequestTimeout() uint {
|
||||
return o.httpRequestTimeout
|
||||
}
|
||||
|
||||
// SetHTTPRequestTimeout sets HTTP request timeout in sec
|
||||
func (o *Options) SetHTTPRequestTimeout(httpRequestTimeout uint) *Options {
|
||||
o.httpRequestTimeout = httpRequestTimeout
|
||||
return o
|
||||
}
|
||||
|
||||
// ApplicationName returns application name used in the User-Agent HTTP header
|
||||
func (o *Options) ApplicationName() string {
|
||||
return o.appName
|
||||
}
|
||||
|
||||
// SetApplicationName sets an application name to the User-Agent HTTP header
|
||||
func (o *Options) SetApplicationName(appName string) *Options {
|
||||
o.appName = appName
|
||||
return o
|
||||
}
|
||||
|
||||
// DefaultOptions returns Options object with default values
|
||||
func DefaultOptions() *Options {
|
||||
return &Options{httpRequestTimeout: 20}
|
||||
}
|
||||
+192
@@ -0,0 +1,192 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package http provides HTTP servicing related code.
|
||||
//
|
||||
// Important type is Service which handles HTTP operations. It is internally used by library and it is not necessary to use it directly for common operations.
|
||||
// It can be useful when creating custom InfluxDB2 server API calls using generated code from the domain package, that are not yet exposed by API of this library.
|
||||
//
|
||||
// Service can be obtained from client using HTTPService() method.
|
||||
// It can be also created directly. To instantiate a Service use NewService(). Remember, the authorization param is in form "Token your-auth-token". e.g. "Token DXnd7annkGteV5Wqx9G3YjO9Ezkw87nHk8OabcyHCxF5451kdBV0Ag2cG7OmZZgCUTHroagUPdxbuoyen6TSPw==".
|
||||
// srv := http.NewService("http://localhost:8086", "Token my-token", http.DefaultOptions())
|
||||
package http
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"mime"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
|
||||
http2 "github.com/influxdata/influxdb-client-go/v2/internal/http"
|
||||
"github.com/influxdata/influxdb-client-go/v2/internal/log"
|
||||
)
|
||||
|
||||
// RequestCallback defines function called after a request is created before any call
|
||||
type RequestCallback func(req *http.Request)
|
||||
|
||||
// ResponseCallback defines function called after a successful response was received
|
||||
type ResponseCallback func(resp *http.Response) error
|
||||
|
||||
// Service handles HTTP operations with taking care of mandatory request headers and known errors
|
||||
type Service interface {
|
||||
// DoPostRequest sends HTTP POST request to the given url with body
|
||||
DoPostRequest(ctx context.Context, url string, body io.Reader, requestCallback RequestCallback, responseCallback ResponseCallback) *Error
|
||||
// DoHTTPRequest sends given HTTP request and handles response
|
||||
DoHTTPRequest(req *http.Request, requestCallback RequestCallback, responseCallback ResponseCallback) *Error
|
||||
// DoHTTPRequestWithResponse sends given HTTP request and returns response
|
||||
DoHTTPRequestWithResponse(req *http.Request, requestCallback RequestCallback) (*http.Response, error)
|
||||
// SetAuthorization sets the authorization header value
|
||||
SetAuthorization(authorization string)
|
||||
// Authorization returns current authorization header value
|
||||
Authorization() string
|
||||
// ServerAPIURL returns URL to InfluxDB2 server API space
|
||||
ServerAPIURL() string
|
||||
// ServerURL returns URL to InfluxDB2 server
|
||||
ServerURL() string
|
||||
}
|
||||
|
||||
// service implements Service interface
|
||||
type service struct {
|
||||
serverAPIURL string
|
||||
serverURL string
|
||||
authorization string
|
||||
client Doer
|
||||
userAgent string
|
||||
}
|
||||
|
||||
// NewService creates instance of http Service with given parameters
|
||||
func NewService(serverURL, authorization string, httpOptions *Options) Service {
|
||||
apiURL, err := url.Parse(serverURL)
|
||||
serverAPIURL := serverURL
|
||||
if err == nil {
|
||||
apiURL, err = apiURL.Parse("api/v2/")
|
||||
if err == nil {
|
||||
serverAPIURL = apiURL.String()
|
||||
}
|
||||
}
|
||||
return &service{
|
||||
serverAPIURL: serverAPIURL,
|
||||
serverURL: serverURL,
|
||||
authorization: authorization,
|
||||
client: httpOptions.HTTPDoer(),
|
||||
userAgent: http2.FormatUserAgent(httpOptions.ApplicationName()),
|
||||
}
|
||||
}
|
||||
|
||||
func (s *service) ServerAPIURL() string {
|
||||
return s.serverAPIURL
|
||||
}
|
||||
|
||||
func (s *service) ServerURL() string {
|
||||
return s.serverURL
|
||||
}
|
||||
|
||||
func (s *service) SetAuthorization(authorization string) {
|
||||
s.authorization = authorization
|
||||
}
|
||||
|
||||
func (s *service) Authorization() string {
|
||||
return s.authorization
|
||||
}
|
||||
|
||||
func (s *service) DoPostRequest(ctx context.Context, url string, body io.Reader, requestCallback RequestCallback, responseCallback ResponseCallback) *Error {
|
||||
return s.doHTTPRequestWithURL(ctx, http.MethodPost, url, body, requestCallback, responseCallback)
|
||||
}
|
||||
|
||||
func (s *service) doHTTPRequestWithURL(ctx context.Context, method, url string, body io.Reader, requestCallback RequestCallback, responseCallback ResponseCallback) *Error {
|
||||
req, err := http.NewRequestWithContext(ctx, method, url, body)
|
||||
if err != nil {
|
||||
return NewError(err)
|
||||
}
|
||||
return s.DoHTTPRequest(req, requestCallback, responseCallback)
|
||||
}
|
||||
|
||||
func (s *service) DoHTTPRequest(req *http.Request, requestCallback RequestCallback, responseCallback ResponseCallback) *Error {
|
||||
resp, err := s.DoHTTPRequestWithResponse(req, requestCallback)
|
||||
if err != nil {
|
||||
return NewError(err)
|
||||
}
|
||||
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return s.parseHTTPError(resp)
|
||||
}
|
||||
if responseCallback != nil {
|
||||
err := responseCallback(resp)
|
||||
if err != nil {
|
||||
return NewError(err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *service) DoHTTPRequestWithResponse(req *http.Request, requestCallback RequestCallback) (*http.Response, error) {
|
||||
log.Infof("HTTP %s req to %s", req.Method, req.URL.String())
|
||||
if len(s.authorization) > 0 {
|
||||
req.Header.Set("Authorization", s.authorization)
|
||||
}
|
||||
if req.Header.Get("User-Agent") == "" {
|
||||
req.Header.Set("User-Agent", s.userAgent)
|
||||
}
|
||||
if requestCallback != nil {
|
||||
requestCallback(req)
|
||||
}
|
||||
return s.client.Do(req)
|
||||
}
|
||||
|
||||
func (s *service) parseHTTPError(r *http.Response) *Error {
|
||||
// successful status code range
|
||||
if r.StatusCode >= 200 && r.StatusCode < 300 {
|
||||
return nil
|
||||
}
|
||||
defer func() {
|
||||
// discard body so connection can be reused
|
||||
_, _ = io.Copy(ioutil.Discard, r.Body)
|
||||
_ = r.Body.Close()
|
||||
}()
|
||||
|
||||
perror := NewError(nil)
|
||||
perror.StatusCode = r.StatusCode
|
||||
|
||||
if v := r.Header.Get("Retry-After"); v != "" {
|
||||
r, err := strconv.ParseUint(v, 10, 32)
|
||||
if err == nil {
|
||||
perror.RetryAfter = uint(r)
|
||||
}
|
||||
}
|
||||
|
||||
// json encoded error
|
||||
ctype, _, _ := mime.ParseMediaType(r.Header.Get("Content-Type"))
|
||||
if ctype == "application/json" {
|
||||
perror.Err = json.NewDecoder(r.Body).Decode(perror)
|
||||
} else {
|
||||
body, err := ioutil.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
perror.Err = err
|
||||
return perror
|
||||
}
|
||||
|
||||
perror.Code = r.Status
|
||||
perror.Message = string(body)
|
||||
}
|
||||
|
||||
if perror.Code == "" && perror.Message == "" {
|
||||
switch r.StatusCode {
|
||||
case http.StatusTooManyRequests:
|
||||
perror.Code = "too many requests"
|
||||
perror.Message = "exceeded rate limit"
|
||||
case http.StatusServiceUnavailable:
|
||||
perror.Code = "unavailable"
|
||||
perror.Message = "service temporarily unavailable"
|
||||
default:
|
||||
perror.Code = r.Status
|
||||
perror.Message = r.Header.Get("X-Influxdb-Error")
|
||||
}
|
||||
}
|
||||
|
||||
return perror
|
||||
}
|
||||
+155
@@ -0,0 +1,155 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2/domain"
|
||||
)
|
||||
|
||||
// LabelsAPI provides methods for managing labels in a InfluxDB server.
|
||||
type LabelsAPI interface {
|
||||
// GetLabels returns all labels.
|
||||
GetLabels(ctx context.Context) (*[]domain.Label, error)
|
||||
// FindLabelsByOrg returns labels belonging to organization org.
|
||||
FindLabelsByOrg(ctx context.Context, org *domain.Organization) (*[]domain.Label, error)
|
||||
// FindLabelsByOrgID returns labels belonging to organization with id orgID.
|
||||
FindLabelsByOrgID(ctx context.Context, orgID string) (*[]domain.Label, error)
|
||||
// FindLabelByID returns a label with labelID.
|
||||
FindLabelByID(ctx context.Context, labelID string) (*domain.Label, error)
|
||||
// FindLabelByName returns a label with name labelName under an organization orgID.
|
||||
FindLabelByName(ctx context.Context, orgID, labelName string) (*domain.Label, error)
|
||||
// CreateLabel creates a new label.
|
||||
CreateLabel(ctx context.Context, label *domain.LabelCreateRequest) (*domain.Label, error)
|
||||
// CreateLabelWithName creates a new label with label labelName and properties, under the organization org.
|
||||
// Properties example: {"color": "ffb3b3", "description": "this is a description"}.
|
||||
CreateLabelWithName(ctx context.Context, org *domain.Organization, labelName string, properties map[string]string) (*domain.Label, error)
|
||||
// CreateLabelWithNameWithID creates a new label with label labelName and properties, under the organization with id orgID.
|
||||
// Properties example: {"color": "ffb3b3", "description": "this is a description"}.
|
||||
CreateLabelWithNameWithID(ctx context.Context, orgID, labelName string, properties map[string]string) (*domain.Label, error)
|
||||
// UpdateLabel updates the label.
|
||||
// Properties can be removed by sending an update with an empty value.
|
||||
UpdateLabel(ctx context.Context, label *domain.Label) (*domain.Label, error)
|
||||
// DeleteLabelWithID deletes a label with labelID.
|
||||
DeleteLabelWithID(ctx context.Context, labelID string) error
|
||||
// DeleteLabel deletes a label.
|
||||
DeleteLabel(ctx context.Context, label *domain.Label) error
|
||||
}
|
||||
|
||||
// labelsAPI implements LabelsAPI
|
||||
type labelsAPI struct {
|
||||
apiClient *domain.Client
|
||||
}
|
||||
|
||||
// NewLabelsAPI creates new instance of LabelsAPI
|
||||
func NewLabelsAPI(apiClient *domain.Client) LabelsAPI {
|
||||
return &labelsAPI{
|
||||
apiClient: apiClient,
|
||||
}
|
||||
}
|
||||
|
||||
func (u *labelsAPI) GetLabels(ctx context.Context) (*[]domain.Label, error) {
|
||||
params := &domain.GetLabelsParams{}
|
||||
return u.getLabels(ctx, params)
|
||||
}
|
||||
|
||||
func (u *labelsAPI) getLabels(ctx context.Context, params *domain.GetLabelsParams) (*[]domain.Label, error) {
|
||||
response, err := u.apiClient.GetLabels(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return (*[]domain.Label)(response.Labels), nil
|
||||
}
|
||||
|
||||
func (u *labelsAPI) FindLabelsByOrg(ctx context.Context, org *domain.Organization) (*[]domain.Label, error) {
|
||||
return u.FindLabelsByOrgID(ctx, *org.Id)
|
||||
}
|
||||
|
||||
func (u *labelsAPI) FindLabelsByOrgID(ctx context.Context, orgID string) (*[]domain.Label, error) {
|
||||
params := &domain.GetLabelsParams{OrgID: &orgID}
|
||||
return u.getLabels(ctx, params)
|
||||
}
|
||||
|
||||
func (u *labelsAPI) FindLabelByID(ctx context.Context, labelID string) (*domain.Label, error) {
|
||||
params := &domain.GetLabelsIDAllParams{
|
||||
LabelID: labelID,
|
||||
}
|
||||
response, err := u.apiClient.GetLabelsID(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return response.Label, nil
|
||||
}
|
||||
|
||||
func (u *labelsAPI) FindLabelByName(ctx context.Context, orgID, labelName string) (*domain.Label, error) {
|
||||
labels, err := u.FindLabelsByOrgID(ctx, orgID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var label *domain.Label
|
||||
for _, u := range *labels {
|
||||
if *u.Name == labelName {
|
||||
label = &u
|
||||
break
|
||||
}
|
||||
}
|
||||
if label == nil {
|
||||
return nil, fmt.Errorf("label '%s' not found", labelName)
|
||||
}
|
||||
return label, nil
|
||||
}
|
||||
|
||||
func (u *labelsAPI) CreateLabelWithName(ctx context.Context, org *domain.Organization, labelName string, properties map[string]string) (*domain.Label, error) {
|
||||
return u.CreateLabelWithNameWithID(ctx, *org.Id, labelName, properties)
|
||||
}
|
||||
|
||||
func (u *labelsAPI) CreateLabelWithNameWithID(ctx context.Context, orgID, labelName string, properties map[string]string) (*domain.Label, error) {
|
||||
props := &domain.LabelCreateRequest_Properties{AdditionalProperties: properties}
|
||||
label := &domain.LabelCreateRequest{Name: labelName, OrgID: orgID, Properties: props}
|
||||
return u.CreateLabel(ctx, label)
|
||||
}
|
||||
|
||||
func (u *labelsAPI) CreateLabel(ctx context.Context, label *domain.LabelCreateRequest) (*domain.Label, error) {
|
||||
params := &domain.PostLabelsAllParams{
|
||||
Body: domain.PostLabelsJSONRequestBody(*label),
|
||||
}
|
||||
response, err := u.apiClient.PostLabels(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return response.Label, nil
|
||||
}
|
||||
|
||||
func (u *labelsAPI) UpdateLabel(ctx context.Context, label *domain.Label) (*domain.Label, error) {
|
||||
var props *domain.LabelUpdate_Properties
|
||||
if label.Properties != nil {
|
||||
props = &domain.LabelUpdate_Properties{AdditionalProperties: label.Properties.AdditionalProperties}
|
||||
}
|
||||
params := &domain.PatchLabelsIDAllParams{
|
||||
Body: domain.PatchLabelsIDJSONRequestBody(domain.LabelUpdate{
|
||||
Name: label.Name,
|
||||
Properties: props,
|
||||
}),
|
||||
LabelID: *label.Id,
|
||||
}
|
||||
response, err := u.apiClient.PatchLabelsID(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return response.Label, nil
|
||||
}
|
||||
|
||||
func (u *labelsAPI) DeleteLabel(ctx context.Context, label *domain.Label) error {
|
||||
return u.DeleteLabelWithID(ctx, *label.Id)
|
||||
}
|
||||
|
||||
func (u *labelsAPI) DeleteLabelWithID(ctx context.Context, labelID string) error {
|
||||
params := &domain.DeleteLabelsIDAllParams{
|
||||
LabelID: labelID,
|
||||
}
|
||||
return u.apiClient.DeleteLabelsID(ctx, params)
|
||||
}
|
||||
+233
@@ -0,0 +1,233 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2/domain"
|
||||
)
|
||||
|
||||
// OrganizationsAPI provides methods for managing Organizations in a InfluxDB server.
|
||||
type OrganizationsAPI interface {
|
||||
// GetOrganizations returns all organizations.
|
||||
// GetOrganizations supports PagingOptions: Offset, Limit, Descending
|
||||
GetOrganizations(ctx context.Context, pagingOptions ...PagingOption) (*[]domain.Organization, error)
|
||||
// FindOrganizationByName returns an organization found using orgName.
|
||||
FindOrganizationByName(ctx context.Context, orgName string) (*domain.Organization, error)
|
||||
// FindOrganizationByID returns an organization found using orgID.
|
||||
FindOrganizationByID(ctx context.Context, orgID string) (*domain.Organization, error)
|
||||
// FindOrganizationsByUserID returns organizations an user with userID belongs to.
|
||||
// FindOrganizationsByUserID supports PagingOptions: Offset, Limit, Descending
|
||||
FindOrganizationsByUserID(ctx context.Context, userID string, pagingOptions ...PagingOption) (*[]domain.Organization, error)
|
||||
// CreateOrganization creates new organization.
|
||||
CreateOrganization(ctx context.Context, org *domain.Organization) (*domain.Organization, error)
|
||||
// CreateOrganizationWithName creates new organization with orgName and with status active.
|
||||
CreateOrganizationWithName(ctx context.Context, orgName string) (*domain.Organization, error)
|
||||
// UpdateOrganization updates organization.
|
||||
UpdateOrganization(ctx context.Context, org *domain.Organization) (*domain.Organization, error)
|
||||
// DeleteOrganization deletes an organization.
|
||||
DeleteOrganization(ctx context.Context, org *domain.Organization) error
|
||||
// DeleteOrganizationWithID deletes an organization with orgID.
|
||||
DeleteOrganizationWithID(ctx context.Context, orgID string) error
|
||||
// GetMembers returns members of an organization.
|
||||
GetMembers(ctx context.Context, org *domain.Organization) (*[]domain.ResourceMember, error)
|
||||
// GetMembersWithID returns members of an organization with orgID.
|
||||
GetMembersWithID(ctx context.Context, orgID string) (*[]domain.ResourceMember, error)
|
||||
// AddMember adds a member to an organization.
|
||||
AddMember(ctx context.Context, org *domain.Organization, user *domain.User) (*domain.ResourceMember, error)
|
||||
// AddMemberWithID adds a member with id memberID to an organization with orgID.
|
||||
AddMemberWithID(ctx context.Context, orgID, memberID string) (*domain.ResourceMember, error)
|
||||
// RemoveMember removes a member from an organization.
|
||||
RemoveMember(ctx context.Context, org *domain.Organization, user *domain.User) error
|
||||
// RemoveMemberWithID removes a member with id memberID from an organization with orgID.
|
||||
RemoveMemberWithID(ctx context.Context, orgID, memberID string) error
|
||||
// GetOwners returns owners of an organization.
|
||||
GetOwners(ctx context.Context, org *domain.Organization) (*[]domain.ResourceOwner, error)
|
||||
// GetOwnersWithID returns owners of an organization with orgID.
|
||||
GetOwnersWithID(ctx context.Context, orgID string) (*[]domain.ResourceOwner, error)
|
||||
// AddOwner adds an owner to an organization.
|
||||
AddOwner(ctx context.Context, org *domain.Organization, user *domain.User) (*domain.ResourceOwner, error)
|
||||
// AddOwnerWithID adds an owner with id memberID to an organization with orgID.
|
||||
AddOwnerWithID(ctx context.Context, orgID, memberID string) (*domain.ResourceOwner, error)
|
||||
// RemoveOwner removes an owner from an organization.
|
||||
RemoveOwner(ctx context.Context, org *domain.Organization, user *domain.User) error
|
||||
// RemoveOwnerWithID removes an owner with id memberID from an organization with orgID.
|
||||
RemoveOwnerWithID(ctx context.Context, orgID, memberID string) error
|
||||
}
|
||||
|
||||
// organizationsAPI implements OrganizationsAPI
|
||||
type organizationsAPI struct {
|
||||
apiClient *domain.Client
|
||||
}
|
||||
|
||||
// NewOrganizationsAPI creates new instance of OrganizationsAPI
|
||||
func NewOrganizationsAPI(apiClient *domain.Client) OrganizationsAPI {
|
||||
return &organizationsAPI{
|
||||
apiClient: apiClient,
|
||||
}
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) getOrganizations(ctx context.Context, params *domain.GetOrgsParams, pagingOptions ...PagingOption) (*[]domain.Organization, error) {
|
||||
options := defaultPaging()
|
||||
for _, opt := range pagingOptions {
|
||||
opt(options)
|
||||
}
|
||||
if options.limit > 0 {
|
||||
params.Limit = &options.limit
|
||||
}
|
||||
params.Offset = &options.offset
|
||||
params.Descending = &options.descending
|
||||
response, err := o.apiClient.GetOrgs(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return response.Orgs, nil
|
||||
}
|
||||
func (o *organizationsAPI) GetOrganizations(ctx context.Context, pagingOptions ...PagingOption) (*[]domain.Organization, error) {
|
||||
params := &domain.GetOrgsParams{}
|
||||
return o.getOrganizations(ctx, params, pagingOptions...)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) FindOrganizationByName(ctx context.Context, orgName string) (*domain.Organization, error) {
|
||||
params := &domain.GetOrgsParams{Org: &orgName}
|
||||
organizations, err := o.getOrganizations(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if organizations != nil && len(*organizations) > 0 {
|
||||
return &(*organizations)[0], nil
|
||||
}
|
||||
return nil, fmt.Errorf("organization '%s' not found", orgName)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) FindOrganizationByID(ctx context.Context, orgID string) (*domain.Organization, error) {
|
||||
params := &domain.GetOrgsIDAllParams{
|
||||
OrgID: orgID,
|
||||
}
|
||||
return o.apiClient.GetOrgsID(ctx, params)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) FindOrganizationsByUserID(ctx context.Context, userID string, pagingOptions ...PagingOption) (*[]domain.Organization, error) {
|
||||
params := &domain.GetOrgsParams{UserID: &userID}
|
||||
return o.getOrganizations(ctx, params, pagingOptions...)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) CreateOrganization(ctx context.Context, org *domain.Organization) (*domain.Organization, error) {
|
||||
params := &domain.PostOrgsAllParams{
|
||||
Body: domain.PostOrgsJSONRequestBody{
|
||||
Name: org.Name,
|
||||
Description: org.Description,
|
||||
},
|
||||
}
|
||||
return o.apiClient.PostOrgs(ctx, params)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) CreateOrganizationWithName(ctx context.Context, orgName string) (*domain.Organization, error) {
|
||||
status := domain.OrganizationStatusActive
|
||||
org := &domain.Organization{Name: orgName, Status: &status}
|
||||
return o.CreateOrganization(ctx, org)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) DeleteOrganization(ctx context.Context, org *domain.Organization) error {
|
||||
return o.DeleteOrganizationWithID(ctx, *org.Id)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) DeleteOrganizationWithID(ctx context.Context, orgID string) error {
|
||||
params := &domain.DeleteOrgsIDAllParams{
|
||||
OrgID: orgID,
|
||||
}
|
||||
return o.apiClient.DeleteOrgsID(ctx, params)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) UpdateOrganization(ctx context.Context, org *domain.Organization) (*domain.Organization, error) {
|
||||
params := &domain.PatchOrgsIDAllParams{
|
||||
Body: domain.PatchOrgsIDJSONRequestBody{
|
||||
Name: &org.Name,
|
||||
Description: org.Description,
|
||||
},
|
||||
OrgID: *org.Id,
|
||||
}
|
||||
return o.apiClient.PatchOrgsID(ctx, params)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) GetMembers(ctx context.Context, org *domain.Organization) (*[]domain.ResourceMember, error) {
|
||||
return o.GetMembersWithID(ctx, *org.Id)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) GetMembersWithID(ctx context.Context, orgID string) (*[]domain.ResourceMember, error) {
|
||||
params := &domain.GetOrgsIDMembersAllParams{
|
||||
OrgID: orgID,
|
||||
}
|
||||
response, err := o.apiClient.GetOrgsIDMembers(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return response.Users, nil
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) AddMember(ctx context.Context, org *domain.Organization, user *domain.User) (*domain.ResourceMember, error) {
|
||||
return o.AddMemberWithID(ctx, *org.Id, *user.Id)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) AddMemberWithID(ctx context.Context, orgID, memberID string) (*domain.ResourceMember, error) {
|
||||
params := &domain.PostOrgsIDMembersAllParams{
|
||||
Body: domain.PostOrgsIDMembersJSONRequestBody{Id: memberID},
|
||||
OrgID: orgID,
|
||||
}
|
||||
return o.apiClient.PostOrgsIDMembers(ctx, params)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) RemoveMember(ctx context.Context, org *domain.Organization, user *domain.User) error {
|
||||
return o.RemoveMemberWithID(ctx, *org.Id, *user.Id)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) RemoveMemberWithID(ctx context.Context, orgID, memberID string) error {
|
||||
params := &domain.DeleteOrgsIDMembersIDAllParams{
|
||||
OrgID: orgID,
|
||||
UserID: memberID,
|
||||
}
|
||||
return o.apiClient.DeleteOrgsIDMembersID(ctx, params)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) GetOwners(ctx context.Context, org *domain.Organization) (*[]domain.ResourceOwner, error) {
|
||||
return o.GetOwnersWithID(ctx, *org.Id)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) GetOwnersWithID(ctx context.Context, orgID string) (*[]domain.ResourceOwner, error) {
|
||||
params := &domain.GetOrgsIDOwnersAllParams{
|
||||
OrgID: orgID,
|
||||
}
|
||||
response, err := o.apiClient.GetOrgsIDOwners(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return response.Users, nil
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) AddOwner(ctx context.Context, org *domain.Organization, user *domain.User) (*domain.ResourceOwner, error) {
|
||||
return o.AddOwnerWithID(ctx, *org.Id, *user.Id)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) AddOwnerWithID(ctx context.Context, orgID, memberID string) (*domain.ResourceOwner, error) {
|
||||
params := &domain.PostOrgsIDOwnersAllParams{
|
||||
Body: domain.PostOrgsIDOwnersJSONRequestBody{Id: memberID},
|
||||
OrgID: orgID,
|
||||
}
|
||||
return o.apiClient.PostOrgsIDOwners(ctx, params)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) RemoveOwner(ctx context.Context, org *domain.Organization, user *domain.User) error {
|
||||
return o.RemoveOwnerWithID(ctx, *org.Id, *user.Id)
|
||||
}
|
||||
|
||||
func (o *organizationsAPI) RemoveOwnerWithID(ctx context.Context, orgID, memberID string) error {
|
||||
params := &domain.DeleteOrgsIDOwnersIDAllParams{
|
||||
OrgID: orgID,
|
||||
UserID: memberID,
|
||||
}
|
||||
return o.apiClient.DeleteOrgsIDOwnersID(ctx, params)
|
||||
}
|
||||
+69
@@ -0,0 +1,69 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package api
|
||||
|
||||
import "github.com/influxdata/influxdb-client-go/v2/domain"
|
||||
|
||||
// PagingOption is the function type for applying paging option
|
||||
type PagingOption func(p *Paging)
|
||||
|
||||
// Paging holds pagination parameters for various Get* functions of InfluxDB 2 API
|
||||
// Not the all options are usable for some Get* functions
|
||||
type Paging struct {
|
||||
// Starting offset for returning items
|
||||
// Default 0.
|
||||
offset domain.Offset
|
||||
// Maximum number of items returned.
|
||||
// Default 0 - not applied
|
||||
limit domain.Limit
|
||||
// What field should be used for sorting
|
||||
sortBy string
|
||||
// Changes sorting direction
|
||||
descending domain.Descending
|
||||
// The last resource ID from which to seek from (but not including).
|
||||
// This is to be used instead of `offset`.
|
||||
after domain.After
|
||||
}
|
||||
|
||||
// defaultPagingOptions returns default paging options: offset 0, limit 0 (not applied), default sorting, ascending
|
||||
func defaultPaging() *Paging {
|
||||
return &Paging{limit: 0, offset: 0, sortBy: "", descending: false, after: ""}
|
||||
}
|
||||
|
||||
// PagingWithLimit sets limit option - maximum number of items returned.
|
||||
func PagingWithLimit(limit int) PagingOption {
|
||||
return func(p *Paging) {
|
||||
p.limit = domain.Limit(limit)
|
||||
}
|
||||
}
|
||||
|
||||
// PagingWithOffset set starting offset for returning items. Default 0.
|
||||
func PagingWithOffset(offset int) PagingOption {
|
||||
return func(p *Paging) {
|
||||
p.offset = domain.Offset(offset)
|
||||
}
|
||||
}
|
||||
|
||||
// PagingWithSortBy sets field name which should be used for sorting
|
||||
func PagingWithSortBy(sortBy string) PagingOption {
|
||||
return func(p *Paging) {
|
||||
p.sortBy = sortBy
|
||||
}
|
||||
}
|
||||
|
||||
// PagingWithDescending changes sorting direction
|
||||
func PagingWithDescending(descending bool) PagingOption {
|
||||
return func(p *Paging) {
|
||||
p.descending = domain.Descending(descending)
|
||||
}
|
||||
}
|
||||
|
||||
// PagingWithAfter set after option - the last resource ID from which to seek from (but not including).
|
||||
// This is to be used instead of `offset`.
|
||||
func PagingWithAfter(after string) PagingOption {
|
||||
return func(p *Paging) {
|
||||
p.after = domain.After(after)
|
||||
}
|
||||
}
|
||||
+532
@@ -0,0 +1,532 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package api
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/csv"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"path"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
http2 "github.com/influxdata/influxdb-client-go/v2/api/http"
|
||||
"github.com/influxdata/influxdb-client-go/v2/api/query"
|
||||
"github.com/influxdata/influxdb-client-go/v2/domain"
|
||||
"github.com/influxdata/influxdb-client-go/v2/internal/log"
|
||||
ilog "github.com/influxdata/influxdb-client-go/v2/log"
|
||||
)
|
||||
|
||||
const (
|
||||
stringDatatype = "string"
|
||||
doubleDatatype = "double"
|
||||
boolDatatype = "boolean"
|
||||
longDatatype = "long"
|
||||
uLongDatatype = "unsignedLong"
|
||||
durationDatatype = "duration"
|
||||
base64BinaryDataType = "base64Binary"
|
||||
timeDatatypeRFC = "dateTime:RFC3339"
|
||||
timeDatatypeRFCNano = "dateTime:RFC3339Nano"
|
||||
)
|
||||
|
||||
// QueryAPI provides methods for performing synchronously flux query against InfluxDB server.
|
||||
//
|
||||
// Flux query can contain reference to parameters, which must be passed via queryParams.
|
||||
// it can be a struct or map. Param values can be only simple types or time.Time.
|
||||
// The name of a struct field or a map key (must be a string) will be a param name.
|
||||
// The name of the parameter represented by a struct field can be specified by JSON annotation:
|
||||
//
|
||||
// type Condition struct {
|
||||
// Start time.Time `json:"start"`
|
||||
// Field string `json:"field"`
|
||||
// Value float64 `json:"value"`
|
||||
// }
|
||||
//
|
||||
// Parameters are then accessed via the Flux params object:
|
||||
//
|
||||
// query:= `from(bucket: "environment")
|
||||
// |> range(start: time(v: params.start))
|
||||
// |> filter(fn: (r) => r._measurement == "air")
|
||||
// |> filter(fn: (r) => r._field == params.field)
|
||||
// |> filter(fn: (r) => r._value > params.value)`
|
||||
//
|
||||
type QueryAPI interface {
|
||||
// QueryRaw executes flux query on the InfluxDB server and returns complete query result as a string with table annotations according to dialect
|
||||
QueryRaw(ctx context.Context, query string, dialect *domain.Dialect) (string, error)
|
||||
// QueryRawWithParams executes flux parametrized query on the InfluxDB server and returns complete query result as a string with table annotations according to dialect
|
||||
QueryRawWithParams(ctx context.Context, query string, dialect *domain.Dialect, params interface{}) (string, error)
|
||||
// Query executes flux query on the InfluxDB server and returns QueryTableResult which parses streamed response into structures representing flux table parts
|
||||
Query(ctx context.Context, query string) (*QueryTableResult, error)
|
||||
// QueryWithParams executes flux parametrized query on the InfluxDB server and returns QueryTableResult which parses streamed response into structures representing flux table parts
|
||||
QueryWithParams(ctx context.Context, query string, params interface{}) (*QueryTableResult, error)
|
||||
}
|
||||
|
||||
// NewQueryAPI returns new query client for querying buckets belonging to org
|
||||
func NewQueryAPI(org string, service http2.Service) QueryAPI {
|
||||
return &queryAPI{
|
||||
org: org,
|
||||
httpService: service,
|
||||
}
|
||||
}
|
||||
|
||||
// QueryTableResult parses streamed flux query response into structures representing flux table parts
|
||||
// Walking though the result is done by repeatedly calling Next() until returns false.
|
||||
// Actual flux table info (columns with names, data types, etc) is returned by TableMetadata() method.
|
||||
// Data are acquired by Record() method.
|
||||
// Preliminary end can be caused by an error, so when Next() return false, check Err() for an error
|
||||
type QueryTableResult struct {
|
||||
io.Closer
|
||||
csvReader *csv.Reader
|
||||
tablePosition int
|
||||
tableChanged bool
|
||||
table *query.FluxTableMetadata
|
||||
record *query.FluxRecord
|
||||
err error
|
||||
}
|
||||
|
||||
// NewQueryTableResult returns new QueryTableResult
|
||||
func NewQueryTableResult(rawResponse io.ReadCloser) *QueryTableResult {
|
||||
csvReader := csv.NewReader(rawResponse)
|
||||
csvReader.FieldsPerRecord = -1
|
||||
return &QueryTableResult{Closer: rawResponse, csvReader: csvReader}
|
||||
}
|
||||
|
||||
// queryAPI implements QueryAPI interface
|
||||
type queryAPI struct {
|
||||
org string
|
||||
httpService http2.Service
|
||||
url string
|
||||
lock sync.Mutex
|
||||
}
|
||||
|
||||
// queryBody holds the body for an HTTP query request.
|
||||
type queryBody struct {
|
||||
Dialect *domain.Dialect `json:"dialect,omitempty"`
|
||||
Query string `json:"query"`
|
||||
Type domain.QueryType `json:"type"`
|
||||
Params interface{} `json:"params,omitempty"`
|
||||
}
|
||||
|
||||
func (q *queryAPI) QueryRaw(ctx context.Context, query string, dialect *domain.Dialect) (string, error) {
|
||||
return q.QueryRawWithParams(ctx, query, dialect, nil)
|
||||
}
|
||||
|
||||
func (q *queryAPI) QueryRawWithParams(ctx context.Context, query string, dialect *domain.Dialect, params interface{}) (string, error) {
|
||||
if err := checkParamsType(params); err != nil {
|
||||
return "", err
|
||||
}
|
||||
queryURL, err := q.queryURL()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
qr := queryBody{
|
||||
Query: query,
|
||||
Type: domain.QueryTypeFlux,
|
||||
Dialect: dialect,
|
||||
Params: params,
|
||||
}
|
||||
qrJSON, err := json.Marshal(qr)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if log.Level() >= ilog.DebugLevel {
|
||||
log.Debugf("Query: %s", qrJSON)
|
||||
}
|
||||
var body string
|
||||
perror := q.httpService.DoPostRequest(ctx, queryURL, bytes.NewReader(qrJSON), func(req *http.Request) {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Accept-Encoding", "gzip")
|
||||
},
|
||||
func(resp *http.Response) error {
|
||||
if resp.Header.Get("Content-Encoding") == "gzip" {
|
||||
resp.Body, err = gzip.NewReader(resp.Body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
respBody, err := ioutil.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
body = string(respBody)
|
||||
return nil
|
||||
})
|
||||
if perror != nil {
|
||||
return "", perror
|
||||
}
|
||||
return body, nil
|
||||
}
|
||||
|
||||
// DefaultDialect return flux query Dialect with full annotations (datatype, group, default), header and comma char as a delimiter
|
||||
func DefaultDialect() *domain.Dialect {
|
||||
annotations := []domain.DialectAnnotations{domain.DialectAnnotationsDatatype, domain.DialectAnnotationsGroup, domain.DialectAnnotationsDefault}
|
||||
delimiter := ","
|
||||
header := true
|
||||
return &domain.Dialect{
|
||||
Annotations: &annotations,
|
||||
Delimiter: &delimiter,
|
||||
Header: &header,
|
||||
}
|
||||
}
|
||||
|
||||
func (q *queryAPI) Query(ctx context.Context, query string) (*QueryTableResult, error) {
|
||||
return q.QueryWithParams(ctx, query, nil)
|
||||
}
|
||||
|
||||
func (q *queryAPI) QueryWithParams(ctx context.Context, query string, params interface{}) (*QueryTableResult, error) {
|
||||
var queryResult *QueryTableResult
|
||||
if err := checkParamsType(params); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
queryURL, err := q.queryURL()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
qr := queryBody{
|
||||
Query: query,
|
||||
Type: domain.QueryTypeFlux,
|
||||
Dialect: DefaultDialect(),
|
||||
Params: params,
|
||||
}
|
||||
qrJSON, err := json.Marshal(qr)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if log.Level() >= ilog.DebugLevel {
|
||||
log.Debugf("Query: %s", qrJSON)
|
||||
}
|
||||
perror := q.httpService.DoPostRequest(ctx, queryURL, bytes.NewReader(qrJSON), func(req *http.Request) {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Accept-Encoding", "gzip")
|
||||
},
|
||||
func(resp *http.Response) error {
|
||||
if resp.Header.Get("Content-Encoding") == "gzip" {
|
||||
resp.Body, err = gzip.NewReader(resp.Body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
csvReader := csv.NewReader(resp.Body)
|
||||
csvReader.FieldsPerRecord = -1
|
||||
queryResult = &QueryTableResult{Closer: resp.Body, csvReader: csvReader}
|
||||
return nil
|
||||
})
|
||||
if perror != nil {
|
||||
return queryResult, perror
|
||||
}
|
||||
return queryResult, nil
|
||||
}
|
||||
|
||||
func (q *queryAPI) queryURL() (string, error) {
|
||||
if q.url == "" {
|
||||
u, err := url.Parse(q.httpService.ServerAPIURL())
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
u.Path = path.Join(u.Path, "query")
|
||||
|
||||
params := u.Query()
|
||||
params.Set("org", q.org)
|
||||
u.RawQuery = params.Encode()
|
||||
q.lock.Lock()
|
||||
q.url = u.String()
|
||||
q.lock.Unlock()
|
||||
}
|
||||
return q.url, nil
|
||||
}
|
||||
|
||||
// checkParamsType validates the value is struct with simple type fields
|
||||
// or a map with key as string and value as a simple type
|
||||
func checkParamsType(p interface{}) error {
|
||||
if p == nil {
|
||||
return nil
|
||||
}
|
||||
t := reflect.TypeOf(p)
|
||||
v := reflect.ValueOf(p)
|
||||
if t.Kind() == reflect.Ptr {
|
||||
t = t.Elem()
|
||||
v = v.Elem()
|
||||
}
|
||||
if t.Kind() != reflect.Struct && t.Kind() != reflect.Map {
|
||||
return fmt.Errorf("cannot use %v as query params", t)
|
||||
}
|
||||
switch t.Kind() {
|
||||
case reflect.Struct:
|
||||
fields := reflect.VisibleFields(t)
|
||||
for _, f := range fields {
|
||||
fv := v.FieldByIndex(f.Index)
|
||||
t := getFieldType(fv)
|
||||
if !validParamType(t) {
|
||||
return fmt.Errorf("cannot use field '%s' of type '%v' as a query param", f.Name, t)
|
||||
}
|
||||
|
||||
}
|
||||
case reflect.Map:
|
||||
key := t.Key()
|
||||
if key.Kind() != reflect.String {
|
||||
return fmt.Errorf("cannot use map key of type '%v' for query param name", key)
|
||||
}
|
||||
for _, k := range v.MapKeys() {
|
||||
f := v.MapIndex(k)
|
||||
t := getFieldType(f)
|
||||
if !validParamType(t) {
|
||||
return fmt.Errorf("cannot use map value type '%v' as a query param", t)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// getFieldType extracts type of value
|
||||
func getFieldType(v reflect.Value) reflect.Type {
|
||||
t := v.Type()
|
||||
if t.Kind() == reflect.Ptr {
|
||||
t = t.Elem()
|
||||
v = v.Elem()
|
||||
}
|
||||
if t.Kind() == reflect.Interface && !v.IsNil() {
|
||||
t = reflect.ValueOf(v.Interface()).Type()
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// timeType is the exact type for the Time
|
||||
var timeType = reflect.TypeOf(time.Time{})
|
||||
|
||||
// validParamType validates that t is primitive type or string or interface
|
||||
func validParamType(t reflect.Type) bool {
|
||||
return (t.Kind() > reflect.Invalid && t.Kind() < reflect.Complex64) ||
|
||||
t.Kind() == reflect.String ||
|
||||
t == timeType
|
||||
}
|
||||
|
||||
// TablePosition returns actual flux table position in the result, or -1 if no table was found yet
|
||||
// Each new table is introduced by an annotation in csv
|
||||
func (q *QueryTableResult) TablePosition() int {
|
||||
if q.table != nil {
|
||||
return q.table.Position()
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
// TableMetadata returns actual flux table metadata
|
||||
func (q *QueryTableResult) TableMetadata() *query.FluxTableMetadata {
|
||||
return q.table
|
||||
}
|
||||
|
||||
// TableChanged returns true if last call of Next() found also new result table
|
||||
// Table information is available via TableMetadata method
|
||||
func (q *QueryTableResult) TableChanged() bool {
|
||||
return q.tableChanged
|
||||
}
|
||||
|
||||
// Record returns last parsed flux table data row
|
||||
// Use Record methods to access value and row properties
|
||||
func (q *QueryTableResult) Record() *query.FluxRecord {
|
||||
return q.record
|
||||
}
|
||||
|
||||
type parsingState int
|
||||
|
||||
const (
|
||||
parsingStateNormal parsingState = iota
|
||||
parsingStateAnnotation
|
||||
parsingStateNameRow
|
||||
parsingStateError
|
||||
)
|
||||
|
||||
// Next advances to next row in query result.
|
||||
// During the first time it is called, Next creates also table metadata
|
||||
// Actual parsed row is available through Record() function
|
||||
// Returns false in case of end or an error, otherwise true
|
||||
func (q *QueryTableResult) Next() bool {
|
||||
var row []string
|
||||
// set closing query in case of preliminary return
|
||||
closer := func() {
|
||||
if err := q.Close(); err != nil {
|
||||
message := err.Error()
|
||||
if q.err != nil {
|
||||
message = fmt.Sprintf("%s,%s", message, q.err.Error())
|
||||
}
|
||||
q.err = errors.New(message)
|
||||
}
|
||||
}
|
||||
defer func() {
|
||||
closer()
|
||||
}()
|
||||
parsingState := parsingStateNormal
|
||||
q.tableChanged = false
|
||||
dataTypeAnnotationFound := false
|
||||
readRow:
|
||||
row, q.err = q.csvReader.Read()
|
||||
if q.err == io.EOF {
|
||||
q.err = nil
|
||||
return false
|
||||
}
|
||||
if q.err != nil {
|
||||
return false
|
||||
}
|
||||
|
||||
if len(row) <= 1 {
|
||||
goto readRow
|
||||
}
|
||||
if len(row[0]) > 0 && row[0][0] == '#' {
|
||||
if parsingState == parsingStateNormal {
|
||||
q.table = query.NewFluxTableMetadata(q.tablePosition)
|
||||
q.tablePosition++
|
||||
q.tableChanged = true
|
||||
for i := range row[1:] {
|
||||
q.table.AddColumn(query.NewFluxColumn(i))
|
||||
}
|
||||
parsingState = parsingStateAnnotation
|
||||
}
|
||||
}
|
||||
if q.table == nil {
|
||||
q.err = errors.New("parsing error, annotations not found")
|
||||
return false
|
||||
}
|
||||
if len(row)-1 != len(q.table.Columns()) {
|
||||
q.err = fmt.Errorf("parsing error, row has different number of columns than the table: %d vs %d", len(row)-1, len(q.table.Columns()))
|
||||
return false
|
||||
}
|
||||
switch row[0] {
|
||||
case "":
|
||||
switch parsingState {
|
||||
case parsingStateAnnotation:
|
||||
if !dataTypeAnnotationFound {
|
||||
q.err = errors.New("parsing error, datatype annotation not found")
|
||||
return false
|
||||
}
|
||||
parsingState = parsingStateNameRow
|
||||
fallthrough
|
||||
case parsingStateNameRow:
|
||||
if row[1] == "error" {
|
||||
parsingState = parsingStateError
|
||||
} else {
|
||||
for i, n := range row[1:] {
|
||||
if q.table.Column(i) != nil {
|
||||
q.table.Column(i).SetName(n)
|
||||
}
|
||||
}
|
||||
parsingState = parsingStateNormal
|
||||
}
|
||||
goto readRow
|
||||
case parsingStateError:
|
||||
var message string
|
||||
if len(row) > 1 && len(row[1]) > 0 {
|
||||
message = row[1]
|
||||
} else {
|
||||
message = "unknown query error"
|
||||
}
|
||||
reference := ""
|
||||
if len(row) > 2 && len(row[2]) > 0 {
|
||||
reference = fmt.Sprintf(",%s", row[2])
|
||||
}
|
||||
q.err = fmt.Errorf("%s%s", message, reference)
|
||||
return false
|
||||
}
|
||||
values := make(map[string]interface{})
|
||||
for i, v := range row[1:] {
|
||||
if q.table.Column(i) != nil {
|
||||
values[q.table.Column(i).Name()], q.err = toValue(stringTernary(v, q.table.Column(i).DefaultValue()), q.table.Column(i).DataType(), q.table.Column(i).Name())
|
||||
if q.err != nil {
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
q.record = query.NewFluxRecord(q.table.Position(), values)
|
||||
case "#datatype":
|
||||
dataTypeAnnotationFound = true
|
||||
for i, d := range row[1:] {
|
||||
if q.table.Column(i) != nil {
|
||||
q.table.Column(i).SetDataType(d)
|
||||
}
|
||||
}
|
||||
goto readRow
|
||||
case "#group":
|
||||
for i, g := range row[1:] {
|
||||
if q.table.Column(i) != nil {
|
||||
q.table.Column(i).SetGroup(g == "true")
|
||||
}
|
||||
}
|
||||
goto readRow
|
||||
case "#default":
|
||||
for i, c := range row[1:] {
|
||||
if q.table.Column(i) != nil {
|
||||
q.table.Column(i).SetDefaultValue(c)
|
||||
}
|
||||
}
|
||||
goto readRow
|
||||
}
|
||||
// don't close query
|
||||
closer = func() {}
|
||||
return true
|
||||
}
|
||||
|
||||
// Err returns an error raised during flux query response parsing
|
||||
func (q *QueryTableResult) Err() error {
|
||||
return q.err
|
||||
}
|
||||
|
||||
// Close reads remaining data and closes underlying Closer
|
||||
func (q *QueryTableResult) Close() error {
|
||||
var err error
|
||||
for err == nil {
|
||||
_, err = q.csvReader.Read()
|
||||
}
|
||||
return q.Closer.Close()
|
||||
}
|
||||
|
||||
// stringTernary returns a if not empty, otherwise b
|
||||
func stringTernary(a, b string) string {
|
||||
if a == "" {
|
||||
return b
|
||||
}
|
||||
return a
|
||||
}
|
||||
|
||||
// toValues converts s into type by t
|
||||
func toValue(s, t, name string) (interface{}, error) {
|
||||
if s == "" {
|
||||
return nil, nil
|
||||
}
|
||||
switch t {
|
||||
case stringDatatype:
|
||||
return s, nil
|
||||
case timeDatatypeRFC:
|
||||
return time.Parse(time.RFC3339, s)
|
||||
case timeDatatypeRFCNano:
|
||||
return time.Parse(time.RFC3339Nano, s)
|
||||
case durationDatatype:
|
||||
return time.ParseDuration(s)
|
||||
case doubleDatatype:
|
||||
return strconv.ParseFloat(s, 64)
|
||||
case boolDatatype:
|
||||
if strings.ToLower(s) == "false" {
|
||||
return false, nil
|
||||
}
|
||||
return true, nil
|
||||
case longDatatype:
|
||||
return strconv.ParseInt(s, 10, 64)
|
||||
case uLongDatatype:
|
||||
return strconv.ParseUint(s, 10, 64)
|
||||
case base64BinaryDataType:
|
||||
return base64.StdEncoding.DecodeString(s)
|
||||
default:
|
||||
return nil, fmt.Errorf("%s has unknown data type %s", name, t)
|
||||
}
|
||||
}
|
||||
+260
@@ -0,0 +1,260 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package query defined types for representing flux query result
|
||||
package query
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// FluxTableMetadata holds flux query result table information represented by collection of columns.
|
||||
// Each new table is introduced by annotations
|
||||
type FluxTableMetadata struct {
|
||||
position int
|
||||
columns []*FluxColumn
|
||||
}
|
||||
|
||||
// FluxColumn holds flux query table column properties
|
||||
type FluxColumn struct {
|
||||
index int
|
||||
name string
|
||||
dataType string
|
||||
group bool
|
||||
defaultValue string
|
||||
}
|
||||
|
||||
// FluxRecord represents row in the flux query result table
|
||||
type FluxRecord struct {
|
||||
table int
|
||||
values map[string]interface{}
|
||||
}
|
||||
|
||||
// NewFluxTableMetadata creates FluxTableMetadata for the table on position
|
||||
func NewFluxTableMetadata(position int) *FluxTableMetadata {
|
||||
return NewFluxTableMetadataFull(position, make([]*FluxColumn, 0, 10))
|
||||
}
|
||||
|
||||
// NewFluxTableMetadataFull creates FluxTableMetadata
|
||||
func NewFluxTableMetadataFull(position int, columns []*FluxColumn) *FluxTableMetadata {
|
||||
return &FluxTableMetadata{position: position, columns: columns}
|
||||
}
|
||||
|
||||
// Position returns position of the table in the flux query result
|
||||
func (f *FluxTableMetadata) Position() int {
|
||||
return f.position
|
||||
}
|
||||
|
||||
// Columns returns slice of flux query result table
|
||||
func (f *FluxTableMetadata) Columns() []*FluxColumn {
|
||||
return f.columns
|
||||
}
|
||||
|
||||
// AddColumn adds column definition to table metadata
|
||||
func (f *FluxTableMetadata) AddColumn(column *FluxColumn) *FluxTableMetadata {
|
||||
f.columns = append(f.columns, column)
|
||||
return f
|
||||
}
|
||||
|
||||
// Column returns flux table column by index.
|
||||
// Returns nil if index is out of the bounds.
|
||||
func (f *FluxTableMetadata) Column(index int) *FluxColumn {
|
||||
if len(f.columns) == 0 || index < 0 || index >= len(f.columns) {
|
||||
return nil
|
||||
}
|
||||
return f.columns[index]
|
||||
}
|
||||
|
||||
// String returns FluxTableMetadata string dump
|
||||
func (f *FluxTableMetadata) String() string {
|
||||
var buffer strings.Builder
|
||||
for i, c := range f.columns {
|
||||
if i > 0 {
|
||||
buffer.WriteString(",")
|
||||
}
|
||||
buffer.WriteString("col")
|
||||
buffer.WriteString(c.String())
|
||||
}
|
||||
return buffer.String()
|
||||
}
|
||||
|
||||
// NewFluxColumn creates FluxColumn for position
|
||||
func NewFluxColumn(index int) *FluxColumn {
|
||||
return &FluxColumn{index: index}
|
||||
}
|
||||
|
||||
// NewFluxColumnFull creates FluxColumn
|
||||
func NewFluxColumnFull(dataType string, defaultValue string, name string, group bool, index int) *FluxColumn {
|
||||
return &FluxColumn{index: index, name: name, dataType: dataType, group: group, defaultValue: defaultValue}
|
||||
}
|
||||
|
||||
// SetDefaultValue sets default value for the column
|
||||
func (f *FluxColumn) SetDefaultValue(defaultValue string) {
|
||||
f.defaultValue = defaultValue
|
||||
}
|
||||
|
||||
// SetGroup set group flag for the column
|
||||
func (f *FluxColumn) SetGroup(group bool) {
|
||||
f.group = group
|
||||
}
|
||||
|
||||
// SetDataType sets data type for the column
|
||||
func (f *FluxColumn) SetDataType(dataType string) {
|
||||
f.dataType = dataType
|
||||
}
|
||||
|
||||
// SetName sets name of the column
|
||||
func (f *FluxColumn) SetName(name string) {
|
||||
f.name = name
|
||||
}
|
||||
|
||||
// DefaultValue returns default value of the column
|
||||
func (f *FluxColumn) DefaultValue() string {
|
||||
return f.defaultValue
|
||||
}
|
||||
|
||||
// IsGroup return true if the column is grouping column
|
||||
func (f *FluxColumn) IsGroup() bool {
|
||||
return f.group
|
||||
}
|
||||
|
||||
// DataType returns data type of the column
|
||||
func (f *FluxColumn) DataType() string {
|
||||
return f.dataType
|
||||
}
|
||||
|
||||
// Name returns name of the column
|
||||
func (f *FluxColumn) Name() string {
|
||||
return f.name
|
||||
}
|
||||
|
||||
// Index returns index of the column
|
||||
func (f *FluxColumn) Index() int {
|
||||
return f.index
|
||||
}
|
||||
|
||||
// String returns FluxColumn string dump
|
||||
func (f *FluxColumn) String() string {
|
||||
return fmt.Sprintf("{%d: name: %s, datatype: %s, defaultValue: %s, group: %v}", f.index, f.name, f.dataType, f.defaultValue, f.group)
|
||||
}
|
||||
|
||||
// NewFluxRecord returns new record for the table with values
|
||||
func NewFluxRecord(table int, values map[string]interface{}) *FluxRecord {
|
||||
return &FluxRecord{table: table, values: values}
|
||||
}
|
||||
|
||||
// Table returns value of the table column
|
||||
// It returns zero if the table column is not found
|
||||
func (r *FluxRecord) Table() int {
|
||||
return int(intValue(r.values, "table"))
|
||||
}
|
||||
|
||||
// Start returns the inclusive lower time bound of all records in the current table.
|
||||
// Returns empty time.Time if there is no column "_start".
|
||||
func (r *FluxRecord) Start() time.Time {
|
||||
return timeValue(r.values, "_start")
|
||||
}
|
||||
|
||||
// Stop returns the exclusive upper time bound of all records in the current table.
|
||||
// Returns empty time.Time if there is no column "_stop".
|
||||
func (r *FluxRecord) Stop() time.Time {
|
||||
return timeValue(r.values, "_stop")
|
||||
}
|
||||
|
||||
// Time returns the time of the record.
|
||||
// Returns empty time.Time if there is no column "_time".
|
||||
func (r *FluxRecord) Time() time.Time {
|
||||
return timeValue(r.values, "_time")
|
||||
}
|
||||
|
||||
// Value returns the default _value column value or nil if not present
|
||||
func (r *FluxRecord) Value() interface{} {
|
||||
return r.ValueByKey("_value")
|
||||
}
|
||||
|
||||
// Field returns the field name.
|
||||
// Returns empty string if there is no column "_field".
|
||||
func (r *FluxRecord) Field() string {
|
||||
return stringValue(r.values, "_field")
|
||||
}
|
||||
|
||||
// Result returns the value of the _result column, which represents result name.
|
||||
// Returns empty string if there is no column "result".
|
||||
func (r *FluxRecord) Result() string {
|
||||
return stringValue(r.values, "result")
|
||||
}
|
||||
|
||||
// Measurement returns the measurement name of the record
|
||||
// Returns empty string if there is no column "_measurement".
|
||||
func (r *FluxRecord) Measurement() string {
|
||||
return stringValue(r.values, "_measurement")
|
||||
}
|
||||
|
||||
// Values returns map of the values where key is the column name
|
||||
func (r *FluxRecord) Values() map[string]interface{} {
|
||||
return r.values
|
||||
}
|
||||
|
||||
// ValueByKey returns value for given column key for the record or nil of result has no value the column key
|
||||
func (r *FluxRecord) ValueByKey(key string) interface{} {
|
||||
return r.values[key]
|
||||
}
|
||||
|
||||
// String returns FluxRecord string dump
|
||||
func (r *FluxRecord) String() string {
|
||||
if len(r.values) == 0 {
|
||||
return ""
|
||||
}
|
||||
|
||||
i := 0
|
||||
keys := make([]string, len(r.values))
|
||||
for k := range r.values {
|
||||
keys[i] = k
|
||||
i++
|
||||
}
|
||||
sort.Strings(keys)
|
||||
var buffer strings.Builder
|
||||
buffer.WriteString(fmt.Sprintf("%s:%v", keys[0], r.values[keys[0]]))
|
||||
for _, k := range keys[1:] {
|
||||
buffer.WriteString(",")
|
||||
buffer.WriteString(fmt.Sprintf("%s:%v", k, r.values[k]))
|
||||
}
|
||||
return buffer.String()
|
||||
}
|
||||
|
||||
// timeValue returns time.Time value from values map according to the key
|
||||
// Empty time.Time value is returned if key is not found
|
||||
func timeValue(values map[string]interface{}, key string) time.Time {
|
||||
if val, ok := values[key]; ok {
|
||||
if t, ok := val.(time.Time); ok {
|
||||
return t
|
||||
}
|
||||
}
|
||||
return time.Time{}
|
||||
}
|
||||
|
||||
// stringValue returns string value from values map according to the key
|
||||
// Empty string is returned if key is not found
|
||||
func stringValue(values map[string]interface{}, key string) string {
|
||||
if val, ok := values[key]; ok {
|
||||
if s, ok := val.(string); ok {
|
||||
return s
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// intValue returns int64 value from values map according to the key
|
||||
// Zero value is returned if key is not found
|
||||
func intValue(values map[string]interface{}, key string) int64 {
|
||||
if val, ok := values[key]; ok {
|
||||
if i, ok := val.(int64); ok {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
+511
@@ -0,0 +1,511 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2/domain"
|
||||
)
|
||||
|
||||
// TaskFilter defines filtering options for FindTasks functions.
|
||||
type TaskFilter struct {
|
||||
// Returns task with a specific name
|
||||
Name string
|
||||
// Filter tasks to a specific organization name.
|
||||
OrgName string
|
||||
// Filter tasks to a specific organization ID.
|
||||
OrgID string
|
||||
// Filter tasks to a specific user ID.
|
||||
User string
|
||||
// Filter tasks by a status--"inactive" or "active".
|
||||
Status domain.TaskStatusType
|
||||
// Return tasks after a specified ID.
|
||||
After string
|
||||
// The number of tasks to return.
|
||||
// Default 100, minimum: 1, maximum 500
|
||||
Limit int
|
||||
}
|
||||
|
||||
// RunFilter defines filtering options for FindRun* functions.
|
||||
type RunFilter struct {
|
||||
// Return runs after a specified ID.
|
||||
After string
|
||||
// The number of runs to return.
|
||||
// Default 100, minimum 1, maximum 500.
|
||||
Limit int
|
||||
// Filter runs to those scheduled before this time.
|
||||
BeforeTime time.Time
|
||||
// Filter runs to those scheduled after this time.
|
||||
AfterTime time.Time
|
||||
}
|
||||
|
||||
// TasksAPI provides methods for managing tasks and task runs in an InfluxDB server.
|
||||
type TasksAPI interface {
|
||||
// FindTasks retrieves tasks according to the filter. More fields can be applied. Filter can be nil.
|
||||
FindTasks(ctx context.Context, filter *TaskFilter) ([]domain.Task, error)
|
||||
// GetTask retrieves a refreshed instance of task.
|
||||
GetTask(ctx context.Context, task *domain.Task) (*domain.Task, error)
|
||||
// GetTaskByID retrieves a task found using taskID.
|
||||
GetTaskByID(ctx context.Context, taskID string) (*domain.Task, error)
|
||||
// CreateTask creates a new task according the task object.
|
||||
// It copies OrgId, Name, Description, Flux, Status and Every or Cron properties. Every and Cron are mutually exclusive.
|
||||
// Every has higher priority.
|
||||
CreateTask(ctx context.Context, task *domain.Task) (*domain.Task, error)
|
||||
// CreateTaskWithEvery creates a new task with the name, flux script and every repetition setting, in the org orgID.
|
||||
// Every means duration values.
|
||||
CreateTaskWithEvery(ctx context.Context, name, flux, every, orgID string) (*domain.Task, error)
|
||||
// CreateTaskWithCron creates a new task with the name, flux script and cron repetition setting, in the org orgID
|
||||
// Cron holds cron-like setting, e.g. once an hour at beginning of the hour "0 * * * *".
|
||||
CreateTaskWithCron(ctx context.Context, name, flux, cron, orgID string) (*domain.Task, error)
|
||||
// CreateTaskByFlux creates a new task with complete definition in flux script, in the org orgID
|
||||
CreateTaskByFlux(ctx context.Context, flux, orgID string) (*domain.Task, error)
|
||||
// UpdateTask updates a task.
|
||||
// It copies Description, Flux, Status, Offset and Every or Cron properties. Every and Cron are mutually exclusive.
|
||||
// Every has higher priority.
|
||||
UpdateTask(ctx context.Context, task *domain.Task) (*domain.Task, error)
|
||||
// DeleteTask deletes a task.
|
||||
DeleteTask(ctx context.Context, task *domain.Task) error
|
||||
// DeleteTaskWithID deletes a task with taskID.
|
||||
DeleteTaskWithID(ctx context.Context, taskID string) error
|
||||
// FindMembers retrieves members of a task.
|
||||
FindMembers(ctx context.Context, task *domain.Task) ([]domain.ResourceMember, error)
|
||||
// FindMembersWithID retrieves members of a task with taskID.
|
||||
FindMembersWithID(ctx context.Context, taskID string) ([]domain.ResourceMember, error)
|
||||
// AddMember adds a member to a task.
|
||||
AddMember(ctx context.Context, task *domain.Task, user *domain.User) (*domain.ResourceMember, error)
|
||||
// AddMemberWithID adds a member with id memberID to a task with taskID.
|
||||
AddMemberWithID(ctx context.Context, taskID, memberID string) (*domain.ResourceMember, error)
|
||||
// RemoveMember removes a member from a task.
|
||||
RemoveMember(ctx context.Context, task *domain.Task, user *domain.User) error
|
||||
// RemoveMemberWithID removes a member with id memberID from a task with taskID.
|
||||
RemoveMemberWithID(ctx context.Context, taskID, memberID string) error
|
||||
// FindOwners retrieves owners of a task.
|
||||
FindOwners(ctx context.Context, task *domain.Task) ([]domain.ResourceOwner, error)
|
||||
// FindOwnersWithID retrieves owners of a task with taskID.
|
||||
FindOwnersWithID(ctx context.Context, taskID string) ([]domain.ResourceOwner, error)
|
||||
// AddOwner adds an owner to a task.
|
||||
AddOwner(ctx context.Context, task *domain.Task, user *domain.User) (*domain.ResourceOwner, error)
|
||||
// AddOwnerWithID adds an owner with id memberID to a task with taskID.
|
||||
AddOwnerWithID(ctx context.Context, taskID, memberID string) (*domain.ResourceOwner, error)
|
||||
// RemoveOwner removes an owner from a task.
|
||||
RemoveOwner(ctx context.Context, task *domain.Task, user *domain.User) error
|
||||
// RemoveOwnerWithID removes a member with id memberID from a task with taskID.
|
||||
RemoveOwnerWithID(ctx context.Context, taskID, memberID string) error
|
||||
// FindRuns retrieves a task runs according the filter. More fields can be applied. Filter can be nil.
|
||||
FindRuns(ctx context.Context, task *domain.Task, filter *RunFilter) ([]domain.Run, error)
|
||||
// FindRunsWithID retrieves runs of a task with taskID according the filter. More fields can be applied. Filter can be nil.
|
||||
FindRunsWithID(ctx context.Context, taskID string, filter *RunFilter) ([]domain.Run, error)
|
||||
// GetRun retrieves a refreshed instance if a task run.
|
||||
GetRun(ctx context.Context, run *domain.Run) (*domain.Run, error)
|
||||
// GetRunByID retrieves a specific task run by taskID and runID
|
||||
GetRunByID(ctx context.Context, taskID, runID string) (*domain.Run, error)
|
||||
// FindRunLogs return all log events for a task run.
|
||||
FindRunLogs(ctx context.Context, run *domain.Run) ([]domain.LogEvent, error)
|
||||
// FindRunLogsWithID return all log events for a run with runID of a task with taskID.
|
||||
FindRunLogsWithID(ctx context.Context, taskID, runID string) ([]domain.LogEvent, error)
|
||||
// RunManually manually start a run of the task now, overriding the current schedule.
|
||||
RunManually(ctx context.Context, task *domain.Task) (*domain.Run, error)
|
||||
// RunManuallyWithID manually start a run of a task with taskID now, overriding the current schedule.
|
||||
RunManuallyWithID(ctx context.Context, taskID string) (*domain.Run, error)
|
||||
// RetryRun retry a task run.
|
||||
RetryRun(ctx context.Context, run *domain.Run) (*domain.Run, error)
|
||||
// RetryRunWithID retry a run with runID of a task with taskID.
|
||||
RetryRunWithID(ctx context.Context, taskID, runID string) (*domain.Run, error)
|
||||
// CancelRun cancels a running task.
|
||||
CancelRun(ctx context.Context, run *domain.Run) error
|
||||
// CancelRunWithID cancels a running task.
|
||||
CancelRunWithID(ctx context.Context, taskID, runID string) error
|
||||
// FindLogs retrieves all logs for a task.
|
||||
FindLogs(ctx context.Context, task *domain.Task) ([]domain.LogEvent, error)
|
||||
// FindLogsWithID retrieves all logs for a task with taskID.
|
||||
FindLogsWithID(ctx context.Context, taskID string) ([]domain.LogEvent, error)
|
||||
// FindLabels retrieves labels of a task.
|
||||
FindLabels(ctx context.Context, task *domain.Task) ([]domain.Label, error)
|
||||
// FindLabelsWithID retrieves labels of a task with taskID.
|
||||
FindLabelsWithID(ctx context.Context, taskID string) ([]domain.Label, error)
|
||||
// AddLabel adds a label to a task.
|
||||
AddLabel(ctx context.Context, task *domain.Task, label *domain.Label) (*domain.Label, error)
|
||||
// AddLabelWithID adds a label with id labelID to a task with taskID.
|
||||
AddLabelWithID(ctx context.Context, taskID, labelID string) (*domain.Label, error)
|
||||
// RemoveLabel removes a label from a task.
|
||||
RemoveLabel(ctx context.Context, task *domain.Task, label *domain.Label) error
|
||||
// RemoveLabelWithID removes a label with id labelID from a task with taskID.
|
||||
RemoveLabelWithID(ctx context.Context, taskID, labelID string) error
|
||||
}
|
||||
|
||||
// tasksAPI implements TasksAPI
|
||||
type tasksAPI struct {
|
||||
apiClient *domain.Client
|
||||
}
|
||||
|
||||
// NewTasksAPI creates new instance of TasksAPI
|
||||
func NewTasksAPI(apiClient *domain.Client) TasksAPI {
|
||||
return &tasksAPI{
|
||||
apiClient: apiClient,
|
||||
}
|
||||
}
|
||||
|
||||
func (t *tasksAPI) FindTasks(ctx context.Context, filter *TaskFilter) ([]domain.Task, error) {
|
||||
params := &domain.GetTasksParams{}
|
||||
if filter != nil {
|
||||
if filter.Name != "" {
|
||||
params.Name = &filter.Name
|
||||
}
|
||||
if filter.User != "" {
|
||||
params.User = &filter.User
|
||||
}
|
||||
if filter.OrgID != "" {
|
||||
params.OrgID = &filter.OrgID
|
||||
}
|
||||
if filter.OrgName != "" {
|
||||
params.Org = &filter.OrgName
|
||||
}
|
||||
if filter.Status != "" {
|
||||
status := domain.GetTasksParamsStatus(filter.Status)
|
||||
params.Status = &status
|
||||
}
|
||||
if filter.Limit > 0 {
|
||||
params.Limit = &filter.Limit
|
||||
}
|
||||
if filter.After != "" {
|
||||
params.After = &filter.After
|
||||
}
|
||||
}
|
||||
|
||||
response, err := t.apiClient.GetTasks(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return *response.Tasks, nil
|
||||
}
|
||||
|
||||
func (t *tasksAPI) GetTask(ctx context.Context, task *domain.Task) (*domain.Task, error) {
|
||||
return t.GetTaskByID(ctx, task.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) GetTaskByID(ctx context.Context, taskID string) (*domain.Task, error) {
|
||||
params := &domain.GetTasksIDAllParams{
|
||||
TaskID: taskID,
|
||||
}
|
||||
return t.apiClient.GetTasksID(ctx, params)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) createTask(ctx context.Context, taskReq *domain.TaskCreateRequest) (*domain.Task, error) {
|
||||
params := &domain.PostTasksAllParams{
|
||||
Body: domain.PostTasksJSONRequestBody(*taskReq),
|
||||
}
|
||||
return t.apiClient.PostTasks(ctx, params)
|
||||
}
|
||||
|
||||
func createTaskReqDetailed(name, flux string, every, cron *string, orgID string) *domain.TaskCreateRequest {
|
||||
repetition := ""
|
||||
if every != nil {
|
||||
repetition = fmt.Sprintf("every: %s", *every)
|
||||
} else if cron != nil {
|
||||
repetition = fmt.Sprintf(`cron: "%s"`, *cron)
|
||||
}
|
||||
fullFlux := fmt.Sprintf(`option task = { name: "%s", %s } %s`, name, repetition, flux)
|
||||
return createTaskReq(fullFlux, orgID)
|
||||
}
|
||||
func createTaskReq(flux string, orgID string) *domain.TaskCreateRequest {
|
||||
|
||||
status := domain.TaskStatusTypeActive
|
||||
taskReq := &domain.TaskCreateRequest{
|
||||
Flux: flux,
|
||||
Status: &status,
|
||||
OrgID: &orgID,
|
||||
}
|
||||
return taskReq
|
||||
}
|
||||
|
||||
func (t *tasksAPI) CreateTask(ctx context.Context, task *domain.Task) (*domain.Task, error) {
|
||||
taskReq := createTaskReqDetailed(task.Name, task.Flux, task.Every, task.Cron, task.OrgID)
|
||||
taskReq.Description = task.Description
|
||||
taskReq.Status = task.Status
|
||||
return t.createTask(ctx, taskReq)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) CreateTaskWithEvery(ctx context.Context, name, flux, every, orgID string) (*domain.Task, error) {
|
||||
taskReq := createTaskReqDetailed(name, flux, &every, nil, orgID)
|
||||
return t.createTask(ctx, taskReq)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) CreateTaskWithCron(ctx context.Context, name, flux, cron, orgID string) (*domain.Task, error) {
|
||||
taskReq := createTaskReqDetailed(name, flux, nil, &cron, orgID)
|
||||
return t.createTask(ctx, taskReq)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) CreateTaskByFlux(ctx context.Context, flux, orgID string) (*domain.Task, error) {
|
||||
taskReq := createTaskReq(flux, orgID)
|
||||
return t.createTask(ctx, taskReq)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) DeleteTask(ctx context.Context, task *domain.Task) error {
|
||||
return t.DeleteTaskWithID(ctx, task.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) DeleteTaskWithID(ctx context.Context, taskID string) error {
|
||||
params := &domain.DeleteTasksIDAllParams{
|
||||
TaskID: taskID,
|
||||
}
|
||||
return t.apiClient.DeleteTasksID(ctx, params)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) UpdateTask(ctx context.Context, task *domain.Task) (*domain.Task, error) {
|
||||
params := &domain.PatchTasksIDAllParams{
|
||||
Body: domain.PatchTasksIDJSONRequestBody(domain.TaskUpdateRequest{
|
||||
Description: task.Description,
|
||||
Flux: &task.Flux,
|
||||
Name: &task.Name,
|
||||
Offset: task.Offset,
|
||||
Status: task.Status,
|
||||
}),
|
||||
TaskID: task.Id,
|
||||
}
|
||||
if task.Every != nil {
|
||||
params.Body.Every = task.Every
|
||||
} else {
|
||||
params.Body.Cron = task.Cron
|
||||
}
|
||||
return t.apiClient.PatchTasksID(ctx, params)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) FindMembers(ctx context.Context, task *domain.Task) ([]domain.ResourceMember, error) {
|
||||
return t.FindMembersWithID(ctx, task.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) FindMembersWithID(ctx context.Context, taskID string) ([]domain.ResourceMember, error) {
|
||||
params := &domain.GetTasksIDMembersAllParams{
|
||||
TaskID: taskID,
|
||||
}
|
||||
response, err := t.apiClient.GetTasksIDMembers(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return *response.Users, nil
|
||||
}
|
||||
|
||||
func (t *tasksAPI) AddMember(ctx context.Context, task *domain.Task, user *domain.User) (*domain.ResourceMember, error) {
|
||||
return t.AddMemberWithID(ctx, task.Id, *user.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) AddMemberWithID(ctx context.Context, taskID, memberID string) (*domain.ResourceMember, error) {
|
||||
params := &domain.PostTasksIDMembersAllParams{
|
||||
TaskID: taskID,
|
||||
Body: domain.PostTasksIDMembersJSONRequestBody{Id: memberID},
|
||||
}
|
||||
|
||||
return t.apiClient.PostTasksIDMembers(ctx, params)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) RemoveMember(ctx context.Context, task *domain.Task, user *domain.User) error {
|
||||
return t.RemoveMemberWithID(ctx, task.Id, *user.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) RemoveMemberWithID(ctx context.Context, taskID, memberID string) error {
|
||||
params := &domain.DeleteTasksIDMembersIDAllParams{
|
||||
TaskID: taskID,
|
||||
UserID: memberID,
|
||||
}
|
||||
return t.apiClient.DeleteTasksIDMembersID(ctx, params)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) FindOwners(ctx context.Context, task *domain.Task) ([]domain.ResourceOwner, error) {
|
||||
return t.FindOwnersWithID(ctx, task.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) FindOwnersWithID(ctx context.Context, taskID string) ([]domain.ResourceOwner, error) {
|
||||
params := &domain.GetTasksIDOwnersAllParams{
|
||||
TaskID: taskID,
|
||||
}
|
||||
response, err := t.apiClient.GetTasksIDOwners(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return *response.Users, nil
|
||||
}
|
||||
|
||||
func (t *tasksAPI) AddOwner(ctx context.Context, task *domain.Task, user *domain.User) (*domain.ResourceOwner, error) {
|
||||
return t.AddOwnerWithID(ctx, task.Id, *user.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) AddOwnerWithID(ctx context.Context, taskID, memberID string) (*domain.ResourceOwner, error) {
|
||||
params := &domain.PostTasksIDOwnersAllParams{
|
||||
Body: domain.PostTasksIDOwnersJSONRequestBody{Id: memberID},
|
||||
TaskID: taskID,
|
||||
}
|
||||
return t.apiClient.PostTasksIDOwners(ctx, params)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) RemoveOwner(ctx context.Context, task *domain.Task, user *domain.User) error {
|
||||
return t.RemoveOwnerWithID(ctx, task.Id, *user.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) RemoveOwnerWithID(ctx context.Context, taskID, memberID string) error {
|
||||
params := &domain.DeleteTasksIDOwnersIDAllParams{
|
||||
TaskID: taskID,
|
||||
UserID: memberID,
|
||||
}
|
||||
return t.apiClient.DeleteTasksIDOwnersID(ctx, params)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) FindRuns(ctx context.Context, task *domain.Task, filter *RunFilter) ([]domain.Run, error) {
|
||||
return t.FindRunsWithID(ctx, task.Id, filter)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) FindRunsWithID(ctx context.Context, taskID string, filter *RunFilter) ([]domain.Run, error) {
|
||||
params := &domain.GetTasksIDRunsAllParams{TaskID: taskID}
|
||||
if filter != nil {
|
||||
if !filter.AfterTime.IsZero() {
|
||||
params.AfterTime = &filter.AfterTime
|
||||
}
|
||||
if !filter.BeforeTime.IsZero() {
|
||||
params.BeforeTime = &filter.BeforeTime
|
||||
}
|
||||
if filter.Limit > 0 {
|
||||
params.Limit = &filter.Limit
|
||||
}
|
||||
if filter.After != "" {
|
||||
params.After = &filter.After
|
||||
}
|
||||
}
|
||||
response, err := t.apiClient.GetTasksIDRuns(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return *response.Runs, nil
|
||||
}
|
||||
|
||||
func (t *tasksAPI) GetRun(ctx context.Context, run *domain.Run) (*domain.Run, error) {
|
||||
return t.GetRunByID(ctx, *run.TaskID, *run.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) GetRunByID(ctx context.Context, taskID, runID string) (*domain.Run, error) {
|
||||
params := &domain.GetTasksIDRunsIDAllParams{
|
||||
TaskID: taskID,
|
||||
RunID: runID,
|
||||
}
|
||||
return t.apiClient.GetTasksIDRunsID(ctx, params)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) FindRunLogs(ctx context.Context, run *domain.Run) ([]domain.LogEvent, error) {
|
||||
return t.FindRunLogsWithID(ctx, *run.TaskID, *run.Id)
|
||||
}
|
||||
func (t *tasksAPI) FindRunLogsWithID(ctx context.Context, taskID, runID string) ([]domain.LogEvent, error) {
|
||||
params := &domain.GetTasksIDRunsIDLogsAllParams{
|
||||
TaskID: taskID,
|
||||
RunID: runID,
|
||||
}
|
||||
response, err := t.apiClient.GetTasksIDRunsIDLogs(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if response.Events == nil {
|
||||
return nil, fmt.Errorf("logs for task '%s' run '%s 'not found", taskID, runID)
|
||||
}
|
||||
return *response.Events, nil
|
||||
}
|
||||
|
||||
func (t *tasksAPI) RunManually(ctx context.Context, task *domain.Task) (*domain.Run, error) {
|
||||
return t.RunManuallyWithID(ctx, task.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) RunManuallyWithID(ctx context.Context, taskID string) (*domain.Run, error) {
|
||||
params := &domain.PostTasksIDRunsAllParams{
|
||||
TaskID: taskID,
|
||||
}
|
||||
return t.apiClient.PostTasksIDRuns(ctx, params)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) RetryRun(ctx context.Context, run *domain.Run) (*domain.Run, error) {
|
||||
return t.RetryRunWithID(ctx, *run.TaskID, *run.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) RetryRunWithID(ctx context.Context, taskID, runID string) (*domain.Run, error) {
|
||||
params := &domain.PostTasksIDRunsIDRetryAllParams{
|
||||
TaskID: taskID,
|
||||
RunID: runID,
|
||||
}
|
||||
return t.apiClient.PostTasksIDRunsIDRetry(ctx, params)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) CancelRun(ctx context.Context, run *domain.Run) error {
|
||||
return t.CancelRunWithID(ctx, *run.TaskID, *run.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) CancelRunWithID(ctx context.Context, taskID, runID string) error {
|
||||
params := &domain.DeleteTasksIDRunsIDAllParams{
|
||||
TaskID: taskID,
|
||||
RunID: runID,
|
||||
}
|
||||
return t.apiClient.DeleteTasksIDRunsID(ctx, params)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) FindLogs(ctx context.Context, task *domain.Task) ([]domain.LogEvent, error) {
|
||||
return t.FindLogsWithID(ctx, task.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) FindLogsWithID(ctx context.Context, taskID string) ([]domain.LogEvent, error) {
|
||||
params := &domain.GetTasksIDLogsAllParams{
|
||||
TaskID: taskID,
|
||||
}
|
||||
response, err := t.apiClient.GetTasksIDLogs(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if response.Events == nil {
|
||||
return nil, fmt.Errorf("logs for task '%s' not found", taskID)
|
||||
}
|
||||
return *response.Events, nil
|
||||
}
|
||||
|
||||
func (t *tasksAPI) FindLabels(ctx context.Context, task *domain.Task) ([]domain.Label, error) {
|
||||
return t.FindLabelsWithID(ctx, task.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) FindLabelsWithID(ctx context.Context, taskID string) ([]domain.Label, error) {
|
||||
params := &domain.GetTasksIDLabelsAllParams{
|
||||
TaskID: taskID,
|
||||
}
|
||||
response, err := t.apiClient.GetTasksIDLabels(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if response.Labels == nil {
|
||||
return nil, fmt.Errorf("lables for task '%s' not found", taskID)
|
||||
}
|
||||
return *response.Labels, nil
|
||||
}
|
||||
|
||||
func (t *tasksAPI) AddLabel(ctx context.Context, task *domain.Task, label *domain.Label) (*domain.Label, error) {
|
||||
return t.AddLabelWithID(ctx, task.Id, *label.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) AddLabelWithID(ctx context.Context, taskID, labelID string) (*domain.Label, error) {
|
||||
params := &domain.PostTasksIDLabelsAllParams{
|
||||
Body: domain.PostTasksIDLabelsJSONRequestBody{LabelID: &labelID},
|
||||
TaskID: taskID,
|
||||
}
|
||||
response, err := t.apiClient.PostTasksIDLabels(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return response.Label, nil
|
||||
}
|
||||
|
||||
func (t *tasksAPI) RemoveLabel(ctx context.Context, task *domain.Task, label *domain.Label) error {
|
||||
return t.RemoveLabelWithID(ctx, task.Id, *label.Id)
|
||||
}
|
||||
|
||||
func (t *tasksAPI) RemoveLabelWithID(ctx context.Context, taskID, labelID string) error {
|
||||
params := &domain.DeleteTasksIDLabelsIDAllParams{
|
||||
TaskID: taskID,
|
||||
LabelID: labelID,
|
||||
}
|
||||
return t.apiClient.DeleteTasksIDLabelsID(ctx, params)
|
||||
}
|
||||
+241
@@ -0,0 +1,241 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
nethttp "net/http"
|
||||
"net/http/cookiejar"
|
||||
"sync"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2/api/http"
|
||||
"github.com/influxdata/influxdb-client-go/v2/domain"
|
||||
"golang.org/x/net/publicsuffix"
|
||||
)
|
||||
|
||||
// UsersAPI provides methods for managing users in a InfluxDB server
|
||||
type UsersAPI interface {
|
||||
// GetUsers returns all users
|
||||
GetUsers(ctx context.Context) (*[]domain.User, error)
|
||||
// FindUserByID returns user with userID
|
||||
FindUserByID(ctx context.Context, userID string) (*domain.User, error)
|
||||
// FindUserByName returns user with name userName
|
||||
FindUserByName(ctx context.Context, userName string) (*domain.User, error)
|
||||
// CreateUser creates new user
|
||||
CreateUser(ctx context.Context, user *domain.User) (*domain.User, error)
|
||||
// CreateUserWithName creates new user with userName
|
||||
CreateUserWithName(ctx context.Context, userName string) (*domain.User, error)
|
||||
// UpdateUser updates user
|
||||
UpdateUser(ctx context.Context, user *domain.User) (*domain.User, error)
|
||||
// UpdateUserPassword sets password for a user
|
||||
UpdateUserPassword(ctx context.Context, user *domain.User, password string) error
|
||||
// UpdateUserPasswordWithID sets password for a user with userID
|
||||
UpdateUserPasswordWithID(ctx context.Context, userID string, password string) error
|
||||
// DeleteUserWithID deletes an user with userID
|
||||
DeleteUserWithID(ctx context.Context, userID string) error
|
||||
// DeleteUser deletes an user
|
||||
DeleteUser(ctx context.Context, user *domain.User) error
|
||||
// Me returns actual user
|
||||
Me(ctx context.Context) (*domain.User, error)
|
||||
// MeUpdatePassword set password of actual user
|
||||
MeUpdatePassword(ctx context.Context, oldPassword, newPassword string) error
|
||||
// SignIn exchanges username and password credentials to establish an authenticated session with the InfluxDB server. The Client's authentication token is then ignored, it can be empty.
|
||||
SignIn(ctx context.Context, username, password string) error
|
||||
// SignOut signs out previously signed-in user
|
||||
SignOut(ctx context.Context) error
|
||||
}
|
||||
|
||||
// usersAPI implements UsersAPI
|
||||
type usersAPI struct {
|
||||
apiClient *domain.Client
|
||||
httpService http.Service
|
||||
httpClient *nethttp.Client
|
||||
deleteCookieJar bool
|
||||
lock sync.Mutex
|
||||
}
|
||||
|
||||
// NewUsersAPI creates new instance of UsersAPI
|
||||
func NewUsersAPI(apiClient *domain.Client, httpService http.Service, httpClient *nethttp.Client) UsersAPI {
|
||||
return &usersAPI{
|
||||
apiClient: apiClient,
|
||||
httpService: httpService,
|
||||
httpClient: httpClient,
|
||||
}
|
||||
}
|
||||
|
||||
func (u *usersAPI) GetUsers(ctx context.Context) (*[]domain.User, error) {
|
||||
params := &domain.GetUsersParams{}
|
||||
response, err := u.apiClient.GetUsers(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return userResponsesToUsers(response.Users), nil
|
||||
}
|
||||
|
||||
func (u *usersAPI) FindUserByID(ctx context.Context, userID string) (*domain.User, error) {
|
||||
params := &domain.GetUsersIDAllParams{
|
||||
UserID: userID,
|
||||
}
|
||||
response, err := u.apiClient.GetUsersID(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return userResponseToUser(response), nil
|
||||
}
|
||||
|
||||
func (u *usersAPI) FindUserByName(ctx context.Context, userName string) (*domain.User, error) {
|
||||
users, err := u.GetUsers(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var user *domain.User
|
||||
for _, u := range *users {
|
||||
if u.Name == userName {
|
||||
user = &u
|
||||
break
|
||||
}
|
||||
}
|
||||
if user == nil {
|
||||
return nil, fmt.Errorf("user '%s' not found", userName)
|
||||
}
|
||||
return user, nil
|
||||
}
|
||||
|
||||
func (u *usersAPI) CreateUserWithName(ctx context.Context, userName string) (*domain.User, error) {
|
||||
user := &domain.User{Name: userName}
|
||||
return u.CreateUser(ctx, user)
|
||||
}
|
||||
|
||||
func (u *usersAPI) CreateUser(ctx context.Context, user *domain.User) (*domain.User, error) {
|
||||
params := &domain.PostUsersAllParams{
|
||||
Body: domain.PostUsersJSONRequestBody(*user),
|
||||
}
|
||||
response, err := u.apiClient.PostUsers(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return userResponseToUser(response), nil
|
||||
}
|
||||
|
||||
func (u *usersAPI) UpdateUser(ctx context.Context, user *domain.User) (*domain.User, error) {
|
||||
params := &domain.PatchUsersIDAllParams{
|
||||
Body: domain.PatchUsersIDJSONRequestBody(*user),
|
||||
UserID: *user.Id,
|
||||
}
|
||||
response, err := u.apiClient.PatchUsersID(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return userResponseToUser(response), nil
|
||||
}
|
||||
|
||||
func (u *usersAPI) UpdateUserPassword(ctx context.Context, user *domain.User, password string) error {
|
||||
return u.UpdateUserPasswordWithID(ctx, *user.Id, password)
|
||||
}
|
||||
|
||||
func (u *usersAPI) UpdateUserPasswordWithID(ctx context.Context, userID string, password string) error {
|
||||
params := &domain.PostUsersIDPasswordAllParams{
|
||||
UserID: userID,
|
||||
Body: domain.PostUsersIDPasswordJSONRequestBody(domain.PasswordResetBody{Password: password}),
|
||||
}
|
||||
return u.apiClient.PostUsersIDPassword(ctx, params)
|
||||
}
|
||||
|
||||
func (u *usersAPI) DeleteUser(ctx context.Context, user *domain.User) error {
|
||||
return u.DeleteUserWithID(ctx, *user.Id)
|
||||
}
|
||||
|
||||
func (u *usersAPI) DeleteUserWithID(ctx context.Context, userID string) error {
|
||||
params := &domain.DeleteUsersIDAllParams{
|
||||
UserID: userID,
|
||||
}
|
||||
return u.apiClient.DeleteUsersID(ctx, params)
|
||||
}
|
||||
|
||||
func (u *usersAPI) Me(ctx context.Context) (*domain.User, error) {
|
||||
params := &domain.GetMeParams{}
|
||||
response, err := u.apiClient.GetMe(ctx, params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return userResponseToUser(response), nil
|
||||
}
|
||||
|
||||
func (u *usersAPI) MeUpdatePassword(ctx context.Context, oldPassword, newPassword string) error {
|
||||
u.lock.Lock()
|
||||
defer u.lock.Unlock()
|
||||
me, err := u.Me(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
creds := base64.StdEncoding.EncodeToString([]byte(me.Name + ":" + oldPassword))
|
||||
auth := u.httpService.Authorization()
|
||||
defer u.httpService.SetAuthorization(auth)
|
||||
u.httpService.SetAuthorization("Basic " + creds)
|
||||
params := &domain.PutMePasswordAllParams{
|
||||
Body: domain.PutMePasswordJSONRequestBody(domain.PasswordResetBody{Password: newPassword}),
|
||||
}
|
||||
return u.apiClient.PutMePassword(ctx, params)
|
||||
}
|
||||
|
||||
func (u *usersAPI) SignIn(ctx context.Context, username, password string) error {
|
||||
u.lock.Lock()
|
||||
defer u.lock.Unlock()
|
||||
if u.httpClient.Jar == nil {
|
||||
jar, err := cookiejar.New(&cookiejar.Options{PublicSuffixList: publicsuffix.List})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
u.httpClient.Jar = jar
|
||||
u.deleteCookieJar = true
|
||||
}
|
||||
creds := base64.StdEncoding.EncodeToString([]byte(username + ":" + password))
|
||||
u.httpService.SetAuthorization("Basic " + creds)
|
||||
defer u.httpService.SetAuthorization("")
|
||||
return u.apiClient.PostSignin(ctx, &domain.PostSigninParams{})
|
||||
}
|
||||
|
||||
func (u *usersAPI) SignOut(ctx context.Context) error {
|
||||
u.lock.Lock()
|
||||
defer u.lock.Unlock()
|
||||
err := u.apiClient.PostSignout(ctx, &domain.PostSignoutParams{})
|
||||
if u.deleteCookieJar {
|
||||
u.httpClient.Jar = nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func userResponseToUser(ur *domain.UserResponse) *domain.User {
|
||||
if ur == nil {
|
||||
return nil
|
||||
}
|
||||
user := &domain.User{
|
||||
Id: ur.Id,
|
||||
Name: ur.Name,
|
||||
Status: userResponseStatusToUserStatus(ur.Status),
|
||||
}
|
||||
return user
|
||||
}
|
||||
|
||||
func userResponseStatusToUserStatus(urs *domain.UserResponseStatus) *domain.UserStatus {
|
||||
if urs == nil {
|
||||
return nil
|
||||
}
|
||||
us := domain.UserStatus(*urs)
|
||||
return &us
|
||||
}
|
||||
|
||||
func userResponsesToUsers(urs *[]domain.UserResponse) *[]domain.User {
|
||||
if urs == nil {
|
||||
return nil
|
||||
}
|
||||
us := make([]domain.User, len(*urs))
|
||||
for i, ur := range *urs {
|
||||
us[i] = *userResponseToUser(&ur)
|
||||
}
|
||||
return &us
|
||||
}
|
||||
+268
@@ -0,0 +1,268 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
http2 "github.com/influxdata/influxdb-client-go/v2/api/http"
|
||||
"github.com/influxdata/influxdb-client-go/v2/api/write"
|
||||
"github.com/influxdata/influxdb-client-go/v2/internal/log"
|
||||
iwrite "github.com/influxdata/influxdb-client-go/v2/internal/write"
|
||||
)
|
||||
|
||||
// WriteFailedCallback is synchronously notified in case non-blocking write fails.
|
||||
// batch contains complete payload, error holds detailed error information,
|
||||
// retryAttempts means number of retries, 0 if it failed during first write.
|
||||
// It must return true if WriteAPI should continue with retrying, false will discard the batch.
|
||||
type WriteFailedCallback func(batch string, error http2.Error, retryAttempts uint) bool
|
||||
|
||||
// WriteAPI is Write client interface with non-blocking methods for writing time series data asynchronously in batches into an InfluxDB server.
|
||||
// WriteAPI can be used concurrently.
|
||||
// When using multiple goroutines for writing, use a single WriteAPI instance in all goroutines.
|
||||
type WriteAPI interface {
|
||||
// WriteRecord writes asynchronously line protocol record into bucket.
|
||||
// WriteRecord adds record into the buffer which is sent on the background when it reaches the batch size.
|
||||
// Blocking alternative is available in the WriteAPIBlocking interface
|
||||
WriteRecord(line string)
|
||||
// WritePoint writes asynchronously Point into bucket.
|
||||
// WritePoint adds Point into the buffer which is sent on the background when it reaches the batch size.
|
||||
// Blocking alternative is available in the WriteAPIBlocking interface
|
||||
WritePoint(point *write.Point)
|
||||
// Flush forces all pending writes from the buffer to be sent
|
||||
Flush()
|
||||
// Errors returns a channel for reading errors which occurs during async writes.
|
||||
// Must be called before performing any writes for errors to be collected.
|
||||
// The chan is unbuffered and must be drained or the writer will block.
|
||||
Errors() <-chan error
|
||||
// SetWriteFailedCallback sets callback allowing custom handling of failed writes.
|
||||
// If callback returns true, failed batch will be retried, otherwise discarded.
|
||||
SetWriteFailedCallback(cb WriteFailedCallback)
|
||||
}
|
||||
|
||||
// WriteAPIImpl provides main implementation for WriteAPI
|
||||
type WriteAPIImpl struct {
|
||||
service *iwrite.Service
|
||||
writeBuffer []string
|
||||
|
||||
errCh chan error
|
||||
writeCh chan *iwrite.Batch
|
||||
bufferCh chan string
|
||||
writeStop chan struct{}
|
||||
bufferStop chan struct{}
|
||||
bufferFlush chan struct{}
|
||||
doneCh chan struct{}
|
||||
bufferInfoCh chan writeBuffInfoReq
|
||||
writeInfoCh chan writeBuffInfoReq
|
||||
writeOptions *write.Options
|
||||
closingMu *sync.Mutex
|
||||
// more appropriate Bool type from sync/atomic cannot be used because it is available since go 1.19
|
||||
isErrChReader int32
|
||||
}
|
||||
|
||||
type writeBuffInfoReq struct {
|
||||
writeBuffLen int
|
||||
}
|
||||
|
||||
// NewWriteAPI returns new non-blocking write client for writing data to bucket belonging to org
|
||||
func NewWriteAPI(org string, bucket string, service http2.Service, writeOptions *write.Options) *WriteAPIImpl {
|
||||
w := &WriteAPIImpl{
|
||||
service: iwrite.NewService(org, bucket, service, writeOptions),
|
||||
errCh: make(chan error, 1),
|
||||
writeBuffer: make([]string, 0, writeOptions.BatchSize()+1),
|
||||
writeCh: make(chan *iwrite.Batch),
|
||||
bufferCh: make(chan string),
|
||||
bufferStop: make(chan struct{}),
|
||||
writeStop: make(chan struct{}),
|
||||
bufferFlush: make(chan struct{}),
|
||||
doneCh: make(chan struct{}),
|
||||
bufferInfoCh: make(chan writeBuffInfoReq),
|
||||
writeInfoCh: make(chan writeBuffInfoReq),
|
||||
writeOptions: writeOptions,
|
||||
closingMu: &sync.Mutex{},
|
||||
}
|
||||
|
||||
go w.bufferProc()
|
||||
go w.writeProc()
|
||||
|
||||
return w
|
||||
}
|
||||
|
||||
// SetWriteFailedCallback sets callback allowing custom handling of failed writes.
|
||||
// If callback returns true, failed batch will be retried, otherwise discarded.
|
||||
func (w *WriteAPIImpl) SetWriteFailedCallback(cb WriteFailedCallback) {
|
||||
w.service.SetBatchErrorCallback(func(batch *iwrite.Batch, error2 http2.Error) bool {
|
||||
return cb(batch.Batch, error2, batch.RetryAttempts)
|
||||
})
|
||||
}
|
||||
|
||||
// Errors returns a channel for reading errors which occurs during async writes.
|
||||
// Must be called before performing any writes for errors to be collected.
|
||||
// New error is skipped when channel is not read.
|
||||
func (w *WriteAPIImpl) Errors() <-chan error {
|
||||
w.setErrChanRead()
|
||||
return w.errCh
|
||||
}
|
||||
|
||||
// Flush forces all pending writes from the buffer to be sent.
|
||||
// Flush also tries sending batches from retry queue without additional retrying.
|
||||
func (w *WriteAPIImpl) Flush() {
|
||||
w.bufferFlush <- struct{}{}
|
||||
w.waitForFlushing()
|
||||
w.service.Flush()
|
||||
}
|
||||
|
||||
func (w *WriteAPIImpl) waitForFlushing() {
|
||||
for {
|
||||
w.bufferInfoCh <- writeBuffInfoReq{}
|
||||
writeBuffInfo := <-w.bufferInfoCh
|
||||
if writeBuffInfo.writeBuffLen == 0 {
|
||||
break
|
||||
}
|
||||
log.Info("Waiting buffer is flushed")
|
||||
<-time.After(time.Millisecond)
|
||||
}
|
||||
for {
|
||||
w.writeInfoCh <- writeBuffInfoReq{}
|
||||
writeBuffInfo := <-w.writeInfoCh
|
||||
if writeBuffInfo.writeBuffLen == 0 {
|
||||
break
|
||||
}
|
||||
log.Info("Waiting buffer is flushed")
|
||||
<-time.After(time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func (w *WriteAPIImpl) bufferProc() {
|
||||
log.Info("Buffer proc started")
|
||||
ticker := time.NewTicker(time.Duration(w.writeOptions.FlushInterval()) * time.Millisecond)
|
||||
x:
|
||||
for {
|
||||
select {
|
||||
case line := <-w.bufferCh:
|
||||
w.writeBuffer = append(w.writeBuffer, line)
|
||||
if len(w.writeBuffer) == int(w.writeOptions.BatchSize()) {
|
||||
w.flushBuffer()
|
||||
}
|
||||
case <-ticker.C:
|
||||
w.flushBuffer()
|
||||
case <-w.bufferFlush:
|
||||
w.flushBuffer()
|
||||
case <-w.bufferStop:
|
||||
ticker.Stop()
|
||||
w.flushBuffer()
|
||||
break x
|
||||
case buffInfo := <-w.bufferInfoCh:
|
||||
buffInfo.writeBuffLen = len(w.bufferInfoCh)
|
||||
w.bufferInfoCh <- buffInfo
|
||||
}
|
||||
}
|
||||
log.Info("Buffer proc finished")
|
||||
w.doneCh <- struct{}{}
|
||||
}
|
||||
|
||||
func (w *WriteAPIImpl) flushBuffer() {
|
||||
if len(w.writeBuffer) > 0 {
|
||||
log.Info("sending batch")
|
||||
batch := iwrite.NewBatch(buffer(w.writeBuffer), w.writeOptions.MaxRetryTime())
|
||||
w.writeCh <- batch
|
||||
w.writeBuffer = w.writeBuffer[:0]
|
||||
}
|
||||
}
|
||||
func (w *WriteAPIImpl) isErrChanRead() bool {
|
||||
return atomic.LoadInt32(&w.isErrChReader) > 0
|
||||
}
|
||||
|
||||
func (w *WriteAPIImpl) setErrChanRead() {
|
||||
atomic.StoreInt32(&w.isErrChReader, 1)
|
||||
}
|
||||
|
||||
func (w *WriteAPIImpl) writeProc() {
|
||||
log.Info("Write proc started")
|
||||
x:
|
||||
for {
|
||||
select {
|
||||
case batch := <-w.writeCh:
|
||||
err := w.service.HandleWrite(context.Background(), batch)
|
||||
if err != nil && w.isErrChanRead() {
|
||||
select {
|
||||
case w.errCh <- err:
|
||||
default:
|
||||
log.Warn("Cannot write error to error channel, it is not read")
|
||||
}
|
||||
}
|
||||
case <-w.writeStop:
|
||||
log.Info("Write proc: received stop")
|
||||
break x
|
||||
case buffInfo := <-w.writeInfoCh:
|
||||
buffInfo.writeBuffLen = len(w.writeCh)
|
||||
w.writeInfoCh <- buffInfo
|
||||
}
|
||||
}
|
||||
log.Info("Write proc finished")
|
||||
w.doneCh <- struct{}{}
|
||||
}
|
||||
|
||||
// Close finishes outstanding write operations,
|
||||
// stop background routines and closes all channels
|
||||
func (w *WriteAPIImpl) Close() {
|
||||
w.closingMu.Lock()
|
||||
defer w.closingMu.Unlock()
|
||||
if w.writeCh != nil {
|
||||
// Flush outstanding metrics
|
||||
w.Flush()
|
||||
|
||||
// stop and wait for buffer proc
|
||||
close(w.bufferStop)
|
||||
<-w.doneCh
|
||||
|
||||
close(w.bufferFlush)
|
||||
close(w.bufferCh)
|
||||
|
||||
// stop and wait for write proc
|
||||
close(w.writeStop)
|
||||
<-w.doneCh
|
||||
|
||||
close(w.writeCh)
|
||||
close(w.writeInfoCh)
|
||||
close(w.bufferInfoCh)
|
||||
w.writeCh = nil
|
||||
|
||||
close(w.errCh)
|
||||
w.errCh = nil
|
||||
}
|
||||
}
|
||||
|
||||
// WriteRecord writes asynchronously line protocol record into bucket.
|
||||
// WriteRecord adds record into the buffer which is sent on the background when it reaches the batch size.
|
||||
// Blocking alternative is available in the WriteAPIBlocking interface
|
||||
func (w *WriteAPIImpl) WriteRecord(line string) {
|
||||
b := []byte(line)
|
||||
b = append(b, 0xa)
|
||||
w.bufferCh <- string(b)
|
||||
}
|
||||
|
||||
// WritePoint writes asynchronously Point into bucket.
|
||||
// WritePoint adds Point into the buffer which is sent on the background when it reaches the batch size.
|
||||
// Blocking alternative is available in the WriteAPIBlocking interface
|
||||
func (w *WriteAPIImpl) WritePoint(point *write.Point) {
|
||||
line, err := w.service.EncodePoints(point)
|
||||
if err != nil {
|
||||
log.Errorf("point encoding error: %s\n", err.Error())
|
||||
if w.errCh != nil {
|
||||
w.errCh <- err
|
||||
}
|
||||
} else {
|
||||
w.bufferCh <- line
|
||||
}
|
||||
}
|
||||
|
||||
func buffer(lines []string) string {
|
||||
return strings.Join(lines, "")
|
||||
}
|
||||
+106
@@ -0,0 +1,106 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package write
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Point extension methods for test
|
||||
|
||||
// PointToLineProtocolBuffer creates InfluxDB line protocol string from the Point, converting associated timestamp according to precision
|
||||
// and write result to the string builder
|
||||
func PointToLineProtocolBuffer(p *Point, sb *strings.Builder, precision time.Duration) {
|
||||
escapeKey(sb, p.Name(), false)
|
||||
sb.WriteRune(',')
|
||||
for i, t := range p.TagList() {
|
||||
if i > 0 {
|
||||
sb.WriteString(",")
|
||||
}
|
||||
escapeKey(sb, t.Key, true)
|
||||
sb.WriteString("=")
|
||||
escapeKey(sb, t.Value, true)
|
||||
}
|
||||
sb.WriteString(" ")
|
||||
for i, f := range p.FieldList() {
|
||||
if i > 0 {
|
||||
sb.WriteString(",")
|
||||
}
|
||||
escapeKey(sb, f.Key, true)
|
||||
sb.WriteString("=")
|
||||
switch f.Value.(type) {
|
||||
case string:
|
||||
sb.WriteString(`"`)
|
||||
escapeValue(sb, f.Value.(string))
|
||||
sb.WriteString(`"`)
|
||||
default:
|
||||
sb.WriteString(fmt.Sprintf("%v", f.Value))
|
||||
}
|
||||
switch f.Value.(type) {
|
||||
case int64:
|
||||
sb.WriteString("i")
|
||||
case uint64:
|
||||
sb.WriteString("u")
|
||||
}
|
||||
}
|
||||
if !p.Time().IsZero() {
|
||||
sb.WriteString(" ")
|
||||
switch precision {
|
||||
case time.Microsecond:
|
||||
sb.WriteString(strconv.FormatInt(p.Time().UnixNano()/1000, 10))
|
||||
case time.Millisecond:
|
||||
sb.WriteString(strconv.FormatInt(p.Time().UnixNano()/1000000, 10))
|
||||
case time.Second:
|
||||
sb.WriteString(strconv.FormatInt(p.Time().Unix(), 10))
|
||||
default:
|
||||
sb.WriteString(strconv.FormatInt(p.Time().UnixNano(), 10))
|
||||
}
|
||||
}
|
||||
sb.WriteString("\n")
|
||||
}
|
||||
|
||||
// PointToLineProtocol creates InfluxDB line protocol string from the Point, converting associated timestamp according to precision
|
||||
func PointToLineProtocol(p *Point, precision time.Duration) string {
|
||||
var sb strings.Builder
|
||||
sb.Grow(1024)
|
||||
PointToLineProtocolBuffer(p, &sb, precision)
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
func escapeKey(sb *strings.Builder, key string, escapeEqual bool) {
|
||||
for _, r := range key {
|
||||
switch r {
|
||||
case '\n':
|
||||
sb.WriteString(`\\n`)
|
||||
continue
|
||||
case '\r':
|
||||
sb.WriteString(`\\r`)
|
||||
continue
|
||||
case '\t':
|
||||
sb.WriteString(`\\t`)
|
||||
continue
|
||||
case ' ', ',':
|
||||
sb.WriteString(`\`)
|
||||
case '=':
|
||||
if escapeEqual {
|
||||
sb.WriteString(`\`)
|
||||
}
|
||||
}
|
||||
sb.WriteRune(r)
|
||||
}
|
||||
}
|
||||
|
||||
func escapeValue(sb *strings.Builder, value string) {
|
||||
for _, r := range value {
|
||||
switch r {
|
||||
case '\\', '"':
|
||||
sb.WriteString(`\`)
|
||||
}
|
||||
sb.WriteRune(r)
|
||||
}
|
||||
}
|
||||
+199
@@ -0,0 +1,199 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package write
|
||||
|
||||
import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// Options holds write configuration properties
|
||||
type Options struct {
|
||||
// Maximum number of points sent to server in single request. Default 5000
|
||||
batchSize uint
|
||||
// Interval, in ms, in which is buffer flushed if it has not been already written (by reaching batch size) . Default 1000ms
|
||||
flushInterval uint
|
||||
// Precision to use in writes for timestamp. In unit of duration: time.Nanosecond, time.Microsecond, time.Millisecond, time.Second
|
||||
// Default time.Nanosecond
|
||||
precision time.Duration
|
||||
// Whether to use GZip compression in requests. Default false
|
||||
useGZip bool
|
||||
// Tags added to each point during writing. If a point already has a tag with the same key, it is left unchanged.
|
||||
defaultTags map[string]string
|
||||
// Default retry interval in ms, if not sent by server. Default 5,000.
|
||||
retryInterval uint
|
||||
// Maximum count of retry attempts of failed writes, default 5.
|
||||
maxRetries uint
|
||||
// Maximum number of points to keep for retry. Should be multiple of BatchSize. Default 50,000.
|
||||
retryBufferLimit uint
|
||||
// The maximum delay between each retry attempt in milliseconds, default 125,000.
|
||||
maxRetryInterval uint
|
||||
// The maximum total retry timeout in millisecond, default 180,000.
|
||||
maxRetryTime uint
|
||||
// The base for the exponential retry delay
|
||||
exponentialBase uint
|
||||
// InfluxDB Enterprise write consistency as explained in https://docs.influxdata.com/enterprise_influxdb/v1.9/concepts/clustering/#write-consistency
|
||||
consistency Consistency
|
||||
}
|
||||
|
||||
const (
|
||||
// ConsistencyOne requires at least one data node acknowledged a write.
|
||||
ConsistencyOne Consistency = "one"
|
||||
|
||||
// ConsistencyAll requires all data nodes to acknowledge a write.
|
||||
ConsistencyAll Consistency = "all"
|
||||
|
||||
// ConsistencyQuorum requires a quorum of data nodes to acknowledge a write.
|
||||
ConsistencyQuorum Consistency = "quorum"
|
||||
|
||||
// ConsistencyAny allows for hinted hand off, potentially no write happened yet.
|
||||
ConsistencyAny Consistency = "any"
|
||||
)
|
||||
|
||||
// Consistency defines enum for allows consistency values for InfluxDB Enterprise, as explained https://docs.influxdata.com/enterprise_influxdb/v1.9/concepts/clustering/#write-consistency
|
||||
type Consistency string
|
||||
|
||||
// BatchSize returns size of batch
|
||||
func (o *Options) BatchSize() uint {
|
||||
return o.batchSize
|
||||
}
|
||||
|
||||
// SetBatchSize sets number of points sent in single request
|
||||
func (o *Options) SetBatchSize(batchSize uint) *Options {
|
||||
o.batchSize = batchSize
|
||||
return o
|
||||
}
|
||||
|
||||
// FlushInterval returns flush interval in ms
|
||||
func (o *Options) FlushInterval() uint {
|
||||
return o.flushInterval
|
||||
}
|
||||
|
||||
// SetFlushInterval sets flush interval in ms in which is buffer flushed if it has not been already written
|
||||
func (o *Options) SetFlushInterval(flushIntervalMs uint) *Options {
|
||||
o.flushInterval = flushIntervalMs
|
||||
return o
|
||||
}
|
||||
|
||||
// RetryInterval returns the default retry interval in ms, if not sent by server. Default 5,000.
|
||||
func (o *Options) RetryInterval() uint {
|
||||
return o.retryInterval
|
||||
}
|
||||
|
||||
// SetRetryInterval sets the time to wait before retry unsuccessful write in ms, if not sent by server
|
||||
func (o *Options) SetRetryInterval(retryIntervalMs uint) *Options {
|
||||
o.retryInterval = retryIntervalMs
|
||||
return o
|
||||
}
|
||||
|
||||
// MaxRetries returns maximum count of retry attempts of failed writes, default 5.
|
||||
func (o *Options) MaxRetries() uint {
|
||||
return o.maxRetries
|
||||
}
|
||||
|
||||
// SetMaxRetries sets maximum count of retry attempts of failed writes.
|
||||
// Setting zero value disables retry strategy.
|
||||
func (o *Options) SetMaxRetries(maxRetries uint) *Options {
|
||||
o.maxRetries = maxRetries
|
||||
return o
|
||||
}
|
||||
|
||||
// RetryBufferLimit returns retry buffer limit.
|
||||
func (o *Options) RetryBufferLimit() uint {
|
||||
return o.retryBufferLimit
|
||||
}
|
||||
|
||||
// SetRetryBufferLimit sets maximum number of points to keep for retry. Should be multiple of BatchSize.
|
||||
func (o *Options) SetRetryBufferLimit(retryBufferLimit uint) *Options {
|
||||
o.retryBufferLimit = retryBufferLimit
|
||||
return o
|
||||
}
|
||||
|
||||
// MaxRetryInterval returns the maximum delay between each retry attempt in milliseconds, default 125,000.
|
||||
func (o *Options) MaxRetryInterval() uint {
|
||||
return o.maxRetryInterval
|
||||
}
|
||||
|
||||
// SetMaxRetryInterval sets the maximum delay between each retry attempt in millisecond
|
||||
func (o *Options) SetMaxRetryInterval(maxRetryIntervalMs uint) *Options {
|
||||
o.maxRetryInterval = maxRetryIntervalMs
|
||||
return o
|
||||
}
|
||||
|
||||
// MaxRetryTime returns the maximum total retry timeout in millisecond, default 180,000.
|
||||
func (o *Options) MaxRetryTime() uint {
|
||||
return o.maxRetryTime
|
||||
}
|
||||
|
||||
// SetMaxRetryTime sets the maximum total retry timeout in millisecond.
|
||||
func (o *Options) SetMaxRetryTime(maxRetryTimeMs uint) *Options {
|
||||
o.maxRetryTime = maxRetryTimeMs
|
||||
return o
|
||||
}
|
||||
|
||||
// ExponentialBase returns the base for the exponential retry delay. Default 2.
|
||||
func (o *Options) ExponentialBase() uint {
|
||||
return o.exponentialBase
|
||||
}
|
||||
|
||||
// SetExponentialBase sets the base for the exponential retry delay.
|
||||
func (o *Options) SetExponentialBase(retryExponentialBase uint) *Options {
|
||||
o.exponentialBase = retryExponentialBase
|
||||
return o
|
||||
}
|
||||
|
||||
// Precision returns time precision for writes
|
||||
func (o *Options) Precision() time.Duration {
|
||||
return o.precision
|
||||
}
|
||||
|
||||
// SetPrecision sets time precision to use in writes for timestamp. In unit of duration: time.Nanosecond, time.Microsecond, time.Millisecond, time.Second
|
||||
func (o *Options) SetPrecision(precision time.Duration) *Options {
|
||||
o.precision = precision
|
||||
return o
|
||||
}
|
||||
|
||||
// UseGZip returns true if write request are gzip`ed
|
||||
func (o *Options) UseGZip() bool {
|
||||
return o.useGZip
|
||||
}
|
||||
|
||||
// SetUseGZip specifies whether to use GZip compression in write requests.
|
||||
func (o *Options) SetUseGZip(useGZip bool) *Options {
|
||||
o.useGZip = useGZip
|
||||
return o
|
||||
}
|
||||
|
||||
// AddDefaultTag adds a default tag. DefaultTags are added to each written point.
|
||||
// If a tag with the same key already exist it is overwritten.
|
||||
// If a point already defines such a tag, it is left unchanged.
|
||||
func (o *Options) AddDefaultTag(key, value string) *Options {
|
||||
o.DefaultTags()[key] = value
|
||||
return o
|
||||
}
|
||||
|
||||
// DefaultTags returns set of default tags
|
||||
func (o *Options) DefaultTags() map[string]string {
|
||||
if o.defaultTags == nil {
|
||||
o.defaultTags = make(map[string]string)
|
||||
}
|
||||
return o.defaultTags
|
||||
}
|
||||
|
||||
// Consistency returns consistency for param value
|
||||
func (o *Options) Consistency() Consistency {
|
||||
return o.consistency
|
||||
}
|
||||
|
||||
// SetConsistency allows setting InfluxDB Enterprise write consistency, as explained in https://docs.influxdata.com/enterprise_influxdb/v1.9/concepts/clustering/#write-consistency */
|
||||
func (o *Options) SetConsistency(consistency Consistency) *Options {
|
||||
o.consistency = consistency
|
||||
return o
|
||||
}
|
||||
|
||||
// DefaultOptions returns Options object with default values
|
||||
func DefaultOptions() *Options {
|
||||
return &Options{batchSize: 5_000, flushInterval: 1_000, precision: time.Nanosecond, useGZip: false, retryBufferLimit: 50_000, defaultTags: make(map[string]string),
|
||||
maxRetries: 5, retryInterval: 5_000, maxRetryInterval: 125_000, maxRetryTime: 180_000, exponentialBase: 2}
|
||||
}
|
||||
+162
@@ -0,0 +1,162 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package write provides the Point struct
|
||||
package write
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
lp "github.com/influxdata/line-protocol"
|
||||
)
|
||||
|
||||
// Point is represents InfluxDB time series point, holding tags and fields
|
||||
type Point struct {
|
||||
measurement string
|
||||
tags []*lp.Tag
|
||||
fields []*lp.Field
|
||||
timestamp time.Time
|
||||
}
|
||||
|
||||
// TagList returns a slice containing tags of a Point.
|
||||
func (m *Point) TagList() []*lp.Tag {
|
||||
return m.tags
|
||||
}
|
||||
|
||||
// FieldList returns a slice containing the fields of a Point.
|
||||
func (m *Point) FieldList() []*lp.Field {
|
||||
return m.fields
|
||||
}
|
||||
|
||||
// SetTime set timestamp for a Point.
|
||||
func (m *Point) SetTime(timestamp time.Time) *Point {
|
||||
m.timestamp = timestamp
|
||||
return m
|
||||
}
|
||||
|
||||
// Time is the timestamp of a Point.
|
||||
func (m *Point) Time() time.Time {
|
||||
return m.timestamp
|
||||
}
|
||||
|
||||
// SortTags orders the tags of a point alphanumerically by key.
|
||||
// This is just here as a helper, to make it easy to keep tags sorted if you are creating a Point manually.
|
||||
func (m *Point) SortTags() *Point {
|
||||
sort.Slice(m.tags, func(i, j int) bool { return m.tags[i].Key < m.tags[j].Key })
|
||||
return m
|
||||
}
|
||||
|
||||
// SortFields orders the fields of a point alphanumerically by key.
|
||||
func (m *Point) SortFields() *Point {
|
||||
sort.Slice(m.fields, func(i, j int) bool { return m.fields[i].Key < m.fields[j].Key })
|
||||
return m
|
||||
}
|
||||
|
||||
// AddTag adds a tag to a point.
|
||||
func (m *Point) AddTag(k, v string) *Point {
|
||||
for i, tag := range m.tags {
|
||||
if k == tag.Key {
|
||||
m.tags[i].Value = v
|
||||
return m
|
||||
}
|
||||
}
|
||||
m.tags = append(m.tags, &lp.Tag{Key: k, Value: v})
|
||||
return m
|
||||
}
|
||||
|
||||
// AddField adds a field to a point.
|
||||
func (m *Point) AddField(k string, v interface{}) *Point {
|
||||
for i, field := range m.fields {
|
||||
if k == field.Key {
|
||||
m.fields[i].Value = v
|
||||
return m
|
||||
}
|
||||
}
|
||||
m.fields = append(m.fields, &lp.Field{Key: k, Value: convertField(v)})
|
||||
return m
|
||||
}
|
||||
|
||||
// Name returns the name of measurement of a point.
|
||||
func (m *Point) Name() string {
|
||||
return m.measurement
|
||||
}
|
||||
|
||||
// NewPointWithMeasurement creates a empty Point
|
||||
// Use AddTag and AddField to fill point with data
|
||||
func NewPointWithMeasurement(measurement string) *Point {
|
||||
return &Point{measurement: measurement}
|
||||
}
|
||||
|
||||
// NewPoint creates a Point from measurement name, tags, fields and a timestamp.
|
||||
func NewPoint(
|
||||
measurement string,
|
||||
tags map[string]string,
|
||||
fields map[string]interface{},
|
||||
ts time.Time,
|
||||
) *Point {
|
||||
m := &Point{
|
||||
measurement: measurement,
|
||||
tags: nil,
|
||||
fields: nil,
|
||||
timestamp: ts,
|
||||
}
|
||||
|
||||
if len(tags) > 0 {
|
||||
m.tags = make([]*lp.Tag, 0, len(tags))
|
||||
for k, v := range tags {
|
||||
m.tags = append(m.tags,
|
||||
&lp.Tag{Key: k, Value: v})
|
||||
}
|
||||
}
|
||||
|
||||
m.fields = make([]*lp.Field, 0, len(fields))
|
||||
for k, v := range fields {
|
||||
v := convertField(v)
|
||||
if v == nil {
|
||||
continue
|
||||
}
|
||||
m.fields = append(m.fields, &lp.Field{Key: k, Value: v})
|
||||
}
|
||||
m.SortFields()
|
||||
m.SortTags()
|
||||
return m
|
||||
}
|
||||
|
||||
// convertField converts any primitive type to types supported by line protocol
|
||||
func convertField(v interface{}) interface{} {
|
||||
switch v := v.(type) {
|
||||
case bool, int64, string, float64:
|
||||
return v
|
||||
case int:
|
||||
return int64(v)
|
||||
case uint:
|
||||
return uint64(v)
|
||||
case uint64:
|
||||
return v
|
||||
case []byte:
|
||||
return string(v)
|
||||
case int32:
|
||||
return int64(v)
|
||||
case int16:
|
||||
return int64(v)
|
||||
case int8:
|
||||
return int64(v)
|
||||
case uint32:
|
||||
return uint64(v)
|
||||
case uint16:
|
||||
return uint64(v)
|
||||
case uint8:
|
||||
return uint64(v)
|
||||
case float32:
|
||||
return float64(v)
|
||||
case time.Time:
|
||||
return v.Format(time.RFC3339Nano)
|
||||
case time.Duration:
|
||||
return v.String()
|
||||
default:
|
||||
return fmt.Sprintf("%v", v)
|
||||
}
|
||||
}
|
||||
+135
@@ -0,0 +1,135 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
|
||||
http2 "github.com/influxdata/influxdb-client-go/v2/api/http"
|
||||
"github.com/influxdata/influxdb-client-go/v2/api/write"
|
||||
iwrite "github.com/influxdata/influxdb-client-go/v2/internal/write"
|
||||
)
|
||||
|
||||
// WriteAPIBlocking offers blocking methods for writing time series data synchronously into an InfluxDB server.
|
||||
// It doesn't implicitly create batches of points by default. Batches are created from array of points/records.
|
||||
//
|
||||
// Implicit batching is enabled with EnableBatching(). In this mode, each call to WritePoint or WriteRecord adds a line
|
||||
// to internal buffer. If length ot the buffer is equal to the batch-size (set in write.Options), the buffer is sent to the server
|
||||
// and the result of the operation is returned.
|
||||
// When a point is written to the buffer, nil error is always returned.
|
||||
// Flush() can be used to trigger sending of batch when it doesn't have the batch-size.
|
||||
//
|
||||
// Synchronous writing is intended to use for writing less frequent data, such as a weather sensing, or if there is a need to have explicit control of failed batches.
|
||||
|
||||
//
|
||||
// WriteAPIBlocking can be used concurrently.
|
||||
// When using multiple goroutines for writing, use a single WriteAPIBlocking instance in all goroutines.
|
||||
type WriteAPIBlocking interface {
|
||||
// WriteRecord writes line protocol record(s) into bucket.
|
||||
// WriteRecord writes lines without implicit batching by default, batch is created from given number of records.
|
||||
// Automatic batching can be enabled by EnableBatching()
|
||||
// Individual arguments can also be batches (multiple records separated by newline).
|
||||
// Non-blocking alternative is available in the WriteAPI interface
|
||||
WriteRecord(ctx context.Context, line ...string) error
|
||||
// WritePoint data point into bucket.
|
||||
// WriteRecord writes points without implicit batching by default, batch is created from given number of points.
|
||||
// Automatic batching can be enabled by EnableBatching().
|
||||
// Non-blocking alternative is available in the WriteAPI interface
|
||||
WritePoint(ctx context.Context, point ...*write.Point) error
|
||||
// EnableBatching turns on implicit batching
|
||||
// Batch size is controlled via write.Options
|
||||
EnableBatching()
|
||||
// Flush forces write of buffer if batching is enabled, even buffer doesn't have the batch-size.
|
||||
Flush(ctx context.Context) error
|
||||
}
|
||||
|
||||
// writeAPIBlocking implements WriteAPIBlocking interface
|
||||
type writeAPIBlocking struct {
|
||||
service *iwrite.Service
|
||||
writeOptions *write.Options
|
||||
// more appropriate Bool type from sync/atomic cannot be used because it is available since go 1.19
|
||||
batching int32
|
||||
batch []string
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// NewWriteAPIBlocking creates new instance of blocking write client for writing data to bucket belonging to org
|
||||
func NewWriteAPIBlocking(org string, bucket string, service http2.Service, writeOptions *write.Options) WriteAPIBlocking {
|
||||
return &writeAPIBlocking{service: iwrite.NewService(org, bucket, service, writeOptions), writeOptions: writeOptions}
|
||||
}
|
||||
|
||||
// NewWriteAPIBlockingWithBatching creates new instance of blocking write client for writing data to bucket belonging to org with batching enabled
|
||||
func NewWriteAPIBlockingWithBatching(org string, bucket string, service http2.Service, writeOptions *write.Options) WriteAPIBlocking {
|
||||
api := &writeAPIBlocking{service: iwrite.NewService(org, bucket, service, writeOptions), writeOptions: writeOptions}
|
||||
api.EnableBatching()
|
||||
return api
|
||||
}
|
||||
|
||||
func (w *writeAPIBlocking) EnableBatching() {
|
||||
if atomic.LoadInt32(&w.batching) == 0 {
|
||||
w.mu.Lock()
|
||||
w.batching = 1
|
||||
w.batch = make([]string, 0, w.writeOptions.BatchSize())
|
||||
w.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
func (w *writeAPIBlocking) write(ctx context.Context, line string) error {
|
||||
if atomic.LoadInt32(&w.batching) > 0 {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
w.batch = append(w.batch, line)
|
||||
if len(w.batch) == int(w.writeOptions.BatchSize()) {
|
||||
return w.flush(ctx)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
err := w.service.WriteBatch(ctx, iwrite.NewBatch(line, w.writeOptions.MaxRetryTime()))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w *writeAPIBlocking) WriteRecord(ctx context.Context, line ...string) error {
|
||||
if len(line) == 0 {
|
||||
return nil
|
||||
}
|
||||
return w.write(ctx, strings.Join(line, "\n"))
|
||||
}
|
||||
|
||||
func (w *writeAPIBlocking) WritePoint(ctx context.Context, point ...*write.Point) error {
|
||||
line, err := w.service.EncodePoints(point...)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return w.write(ctx, line)
|
||||
}
|
||||
|
||||
// flush is unsychronized helper for creating and sending batch
|
||||
// Must be called from synchronized block
|
||||
func (w *writeAPIBlocking) flush(ctx context.Context) error {
|
||||
if len(w.batch) > 0 {
|
||||
body := strings.Join(w.batch, "\n")
|
||||
w.batch = w.batch[:0]
|
||||
b := iwrite.NewBatch(body, w.writeOptions.MaxRetryTime())
|
||||
if err:= w.service.WriteBatch(ctx, b); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w *writeAPIBlocking) Flush(ctx context.Context) error {
|
||||
if atomic.LoadInt32(&w.batching) > 0 {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
return w.flush(ctx)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+324
@@ -0,0 +1,324 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package influxdb2 provides API for using InfluxDB client in Go.
|
||||
// It's intended to use with InfluxDB 2 server. WriteAPI, QueryAPI and Health work also with InfluxDB 1.8
|
||||
package influxdb2
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
httpnet "net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2/api"
|
||||
"github.com/influxdata/influxdb-client-go/v2/api/http"
|
||||
"github.com/influxdata/influxdb-client-go/v2/domain"
|
||||
ilog "github.com/influxdata/influxdb-client-go/v2/internal/log"
|
||||
"github.com/influxdata/influxdb-client-go/v2/log"
|
||||
)
|
||||
|
||||
// Client provides API to communicate with InfluxDBServer.
|
||||
// There two APIs for writing, WriteAPI and WriteAPIBlocking.
|
||||
// WriteAPI provides asynchronous, non-blocking, methods for writing time series data.
|
||||
// WriteAPIBlocking provides blocking methods for writing time series data.
|
||||
type Client interface {
|
||||
// Setup sends request to initialise new InfluxDB server with user, org and bucket, and data retention period
|
||||
// and returns details about newly created entities along with the authorization object.
|
||||
// Retention period of zero will result to infinite retention.
|
||||
Setup(ctx context.Context, username, password, org, bucket string, retentionPeriodHours int) (*domain.OnboardingResponse, error)
|
||||
// SetupWithToken sends request to initialise new InfluxDB server with user, org and bucket, data retention period and token
|
||||
// and returns details about newly created entities along with the authorization object.
|
||||
// Retention period of zero will result to infinite retention.
|
||||
SetupWithToken(ctx context.Context, username, password, org, bucket string, retentionPeriodHours int, token string) (*domain.OnboardingResponse, error)
|
||||
// Ready returns InfluxDB uptime info of server. It doesn't validate authentication params.
|
||||
Ready(ctx context.Context) (*domain.Ready, error)
|
||||
// Health returns an InfluxDB server health check result. Read the HealthCheck.Status field to get server status.
|
||||
// Health doesn't validate authentication params.
|
||||
Health(ctx context.Context) (*domain.HealthCheck, error)
|
||||
// Ping validates whether InfluxDB server is running. It doesn't validate authentication params.
|
||||
Ping(ctx context.Context) (bool, error)
|
||||
// Close ensures all ongoing asynchronous write clients finish.
|
||||
// Also closes all idle connections, in case of HTTP client was created internally.
|
||||
Close()
|
||||
// Options returns the options associated with client
|
||||
Options() *Options
|
||||
// ServerURL returns the url of the server url client talks to
|
||||
ServerURL() string
|
||||
// HTTPService returns underlying HTTP service object used by client
|
||||
HTTPService() http.Service
|
||||
// WriteAPI returns the asynchronous, non-blocking, Write client.
|
||||
// Ensures using a single WriteAPI instance for each org/bucket pair.
|
||||
WriteAPI(org, bucket string) api.WriteAPI
|
||||
// WriteAPIBlocking returns the synchronous, blocking, Write client.
|
||||
// Ensures using a single WriteAPIBlocking instance for each org/bucket pair.
|
||||
WriteAPIBlocking(org, bucket string) api.WriteAPIBlocking
|
||||
// QueryAPI returns Query client.
|
||||
// Ensures using a single QueryAPI instance each org.
|
||||
QueryAPI(org string) api.QueryAPI
|
||||
// AuthorizationsAPI returns Authorizations API client.
|
||||
AuthorizationsAPI() api.AuthorizationsAPI
|
||||
// OrganizationsAPI returns Organizations API client
|
||||
OrganizationsAPI() api.OrganizationsAPI
|
||||
// UsersAPI returns Users API client.
|
||||
UsersAPI() api.UsersAPI
|
||||
// DeleteAPI returns Delete API client
|
||||
DeleteAPI() api.DeleteAPI
|
||||
// BucketsAPI returns Buckets API client
|
||||
BucketsAPI() api.BucketsAPI
|
||||
// LabelsAPI returns Labels API client
|
||||
LabelsAPI() api.LabelsAPI
|
||||
// TasksAPI returns Tasks API client
|
||||
TasksAPI() api.TasksAPI
|
||||
|
||||
APIClient() *domain.Client
|
||||
}
|
||||
|
||||
// clientImpl implements Client interface
|
||||
type clientImpl struct {
|
||||
serverURL string
|
||||
options *Options
|
||||
writeAPIs map[string]api.WriteAPI
|
||||
syncWriteAPIs map[string]api.WriteAPIBlocking
|
||||
lock sync.Mutex
|
||||
httpService http.Service
|
||||
apiClient *domain.Client
|
||||
authAPI api.AuthorizationsAPI
|
||||
orgAPI api.OrganizationsAPI
|
||||
usersAPI api.UsersAPI
|
||||
deleteAPI api.DeleteAPI
|
||||
bucketsAPI api.BucketsAPI
|
||||
labelsAPI api.LabelsAPI
|
||||
tasksAPI api.TasksAPI
|
||||
}
|
||||
|
||||
type clientDoer struct {
|
||||
service http.Service
|
||||
}
|
||||
|
||||
// NewClient creates Client for connecting to given serverURL with provided authentication token, with the default options.
|
||||
// serverURL is the InfluxDB server base URL, e.g. http://localhost:8086,
|
||||
// authToken is an authentication token. It can be empty in case of connecting to newly installed InfluxDB server, which has not been set up yet.
|
||||
// In such case, calling Setup() will set the authentication token.
|
||||
func NewClient(serverURL string, authToken string) Client {
|
||||
return NewClientWithOptions(serverURL, authToken, DefaultOptions())
|
||||
}
|
||||
|
||||
// NewClientWithOptions creates Client for connecting to given serverURL with provided authentication token
|
||||
// and configured with custom Options.
|
||||
// serverURL is the InfluxDB server base URL, e.g. http://localhost:8086,
|
||||
// authToken is an authentication token. It can be empty in case of connecting to newly installed InfluxDB server, which has not been set up yet.
|
||||
// In such case, calling Setup() will set authentication token
|
||||
func NewClientWithOptions(serverURL string, authToken string, options *Options) Client {
|
||||
normServerURL := serverURL
|
||||
if !strings.HasSuffix(normServerURL, "/") {
|
||||
// For subsequent path parts concatenation, url has to end with '/'
|
||||
normServerURL = serverURL + "/"
|
||||
}
|
||||
authorization := ""
|
||||
if len(authToken) > 0 {
|
||||
authorization = "Token " + authToken
|
||||
}
|
||||
service := http.NewService(normServerURL, authorization, options.httpOptions)
|
||||
doer := &clientDoer{service}
|
||||
|
||||
apiClient, _ := domain.NewClient(service.ServerURL(), doer)
|
||||
|
||||
client := &clientImpl{
|
||||
serverURL: serverURL,
|
||||
options: options,
|
||||
writeAPIs: make(map[string]api.WriteAPI, 5),
|
||||
syncWriteAPIs: make(map[string]api.WriteAPIBlocking, 5),
|
||||
httpService: service,
|
||||
apiClient: apiClient,
|
||||
}
|
||||
if log.Log != nil {
|
||||
log.Log.SetLogLevel(options.LogLevel())
|
||||
}
|
||||
if ilog.Level() >= log.InfoLevel {
|
||||
tokenStr := ""
|
||||
if len(authToken) > 0 {
|
||||
tokenStr = ", token '******'"
|
||||
}
|
||||
ilog.Infof("Using URL '%s'%s", serverURL, tokenStr)
|
||||
}
|
||||
if options.ApplicationName() == "" {
|
||||
ilog.Warn("Application name is not set")
|
||||
}
|
||||
return client
|
||||
}
|
||||
|
||||
func (c *clientImpl) APIClient() *domain.Client {
|
||||
return c.apiClient
|
||||
}
|
||||
|
||||
func (c *clientImpl) Options() *Options {
|
||||
return c.options
|
||||
}
|
||||
|
||||
func (c *clientImpl) ServerURL() string {
|
||||
return c.serverURL
|
||||
}
|
||||
|
||||
func (c *clientImpl) HTTPService() http.Service {
|
||||
return c.httpService
|
||||
}
|
||||
|
||||
func (c *clientDoer) Do(req *httpnet.Request) (*httpnet.Response, error) {
|
||||
return c.service.DoHTTPRequestWithResponse(req, nil)
|
||||
}
|
||||
|
||||
func (c *clientImpl) Ready(ctx context.Context) (*domain.Ready, error) {
|
||||
params := &domain.GetReadyParams{}
|
||||
return c.apiClient.GetReady(ctx, params)
|
||||
}
|
||||
|
||||
func (c *clientImpl) Setup(ctx context.Context, username, password, org, bucket string, retentionPeriodHours int) (*domain.OnboardingResponse, error) {
|
||||
return c.SetupWithToken(ctx, username, password, org, bucket, retentionPeriodHours, "")
|
||||
}
|
||||
|
||||
func (c *clientImpl) SetupWithToken(ctx context.Context, username, password, org, bucket string, retentionPeriodHours int, token string) (*domain.OnboardingResponse, error) {
|
||||
if username == "" || password == "" {
|
||||
return nil, errors.New("a username and a password is required for a setup")
|
||||
}
|
||||
c.lock.Lock()
|
||||
defer c.lock.Unlock()
|
||||
params := &domain.PostSetupAllParams{}
|
||||
retentionPeriodSeconds := int64(retentionPeriodHours * 3600)
|
||||
retentionPeriodHrs := int(time.Duration(retentionPeriodSeconds) * time.Second)
|
||||
params.Body = domain.PostSetupJSONRequestBody{
|
||||
Bucket: bucket,
|
||||
Org: org,
|
||||
Password: &password,
|
||||
RetentionPeriodSeconds: &retentionPeriodSeconds,
|
||||
RetentionPeriodHrs: &retentionPeriodHrs,
|
||||
Username: username,
|
||||
}
|
||||
if token != "" {
|
||||
params.Body.Token = &token
|
||||
}
|
||||
return c.apiClient.PostSetup(ctx, params)
|
||||
}
|
||||
|
||||
func (c *clientImpl) Health(ctx context.Context) (*domain.HealthCheck, error) {
|
||||
params := &domain.GetHealthParams{}
|
||||
return c.apiClient.GetHealth(ctx, params)
|
||||
}
|
||||
|
||||
func (c *clientImpl) Ping(ctx context.Context) (bool, error) {
|
||||
err := c.apiClient.GetPing(ctx)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func createKey(org, bucket string) string {
|
||||
return org + "\t" + bucket
|
||||
}
|
||||
|
||||
func (c *clientImpl) WriteAPI(org, bucket string) api.WriteAPI {
|
||||
c.lock.Lock()
|
||||
defer c.lock.Unlock()
|
||||
key := createKey(org, bucket)
|
||||
if _, ok := c.writeAPIs[key]; !ok {
|
||||
w := api.NewWriteAPI(org, bucket, c.httpService, c.options.writeOptions)
|
||||
c.writeAPIs[key] = w
|
||||
}
|
||||
return c.writeAPIs[key]
|
||||
}
|
||||
|
||||
func (c *clientImpl) WriteAPIBlocking(org, bucket string) api.WriteAPIBlocking {
|
||||
c.lock.Lock()
|
||||
defer c.lock.Unlock()
|
||||
key := createKey(org, bucket)
|
||||
if _, ok := c.syncWriteAPIs[key]; !ok {
|
||||
w := api.NewWriteAPIBlocking(org, bucket, c.httpService, c.options.writeOptions)
|
||||
c.syncWriteAPIs[key] = w
|
||||
}
|
||||
return c.syncWriteAPIs[key]
|
||||
}
|
||||
|
||||
func (c *clientImpl) Close() {
|
||||
for key, w := range c.writeAPIs {
|
||||
wa := w.(*api.WriteAPIImpl)
|
||||
wa.Close()
|
||||
delete(c.writeAPIs, key)
|
||||
}
|
||||
for key := range c.syncWriteAPIs {
|
||||
delete(c.syncWriteAPIs, key)
|
||||
}
|
||||
if c.options.HTTPOptions().OwnHTTPClient() {
|
||||
c.options.HTTPOptions().HTTPClient().CloseIdleConnections()
|
||||
}
|
||||
}
|
||||
|
||||
func (c *clientImpl) QueryAPI(org string) api.QueryAPI {
|
||||
return api.NewQueryAPI(org, c.httpService)
|
||||
}
|
||||
|
||||
func (c *clientImpl) AuthorizationsAPI() api.AuthorizationsAPI {
|
||||
c.lock.Lock()
|
||||
defer c.lock.Unlock()
|
||||
if c.authAPI == nil {
|
||||
c.authAPI = api.NewAuthorizationsAPI(c.apiClient)
|
||||
}
|
||||
return c.authAPI
|
||||
}
|
||||
|
||||
func (c *clientImpl) OrganizationsAPI() api.OrganizationsAPI {
|
||||
c.lock.Lock()
|
||||
defer c.lock.Unlock()
|
||||
if c.orgAPI == nil {
|
||||
c.orgAPI = api.NewOrganizationsAPI(c.apiClient)
|
||||
}
|
||||
return c.orgAPI
|
||||
}
|
||||
|
||||
func (c *clientImpl) UsersAPI() api.UsersAPI {
|
||||
c.lock.Lock()
|
||||
defer c.lock.Unlock()
|
||||
if c.usersAPI == nil {
|
||||
c.usersAPI = api.NewUsersAPI(c.apiClient, c.httpService, c.options.HTTPClient())
|
||||
}
|
||||
return c.usersAPI
|
||||
}
|
||||
|
||||
func (c *clientImpl) DeleteAPI() api.DeleteAPI {
|
||||
c.lock.Lock()
|
||||
defer c.lock.Unlock()
|
||||
if c.deleteAPI == nil {
|
||||
c.deleteAPI = api.NewDeleteAPI(c.apiClient)
|
||||
}
|
||||
return c.deleteAPI
|
||||
}
|
||||
|
||||
func (c *clientImpl) BucketsAPI() api.BucketsAPI {
|
||||
c.lock.Lock()
|
||||
defer c.lock.Unlock()
|
||||
if c.bucketsAPI == nil {
|
||||
c.bucketsAPI = api.NewBucketsAPI(c.apiClient)
|
||||
}
|
||||
return c.bucketsAPI
|
||||
}
|
||||
|
||||
func (c *clientImpl) LabelsAPI() api.LabelsAPI {
|
||||
c.lock.Lock()
|
||||
defer c.lock.Unlock()
|
||||
if c.labelsAPI == nil {
|
||||
c.labelsAPI = api.NewLabelsAPI(c.apiClient)
|
||||
}
|
||||
return c.labelsAPI
|
||||
}
|
||||
|
||||
func (c *clientImpl) TasksAPI() api.TasksAPI {
|
||||
c.lock.Lock()
|
||||
defer c.lock.Unlock()
|
||||
if c.tasksAPI == nil {
|
||||
c.tasksAPI = api.NewTasksAPI(c.apiClient)
|
||||
}
|
||||
return c.tasksAPI
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package influxdb2
|
||||
|
||||
import (
|
||||
"github.com/influxdata/influxdb-client-go/v2/api"
|
||||
"github.com/influxdata/influxdb-client-go/v2/api/write"
|
||||
"github.com/influxdata/influxdb-client-go/v2/domain"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Proxy methods for backward compatibility
|
||||
|
||||
// NewPointWithMeasurement creates a empty Point
|
||||
// Use AddTag and AddField to fill point with data
|
||||
func NewPointWithMeasurement(measurement string) *write.Point {
|
||||
return write.NewPointWithMeasurement(measurement)
|
||||
}
|
||||
|
||||
// NewPoint creates a Point from measurement name, tags, fields and a timestamp.
|
||||
func NewPoint(
|
||||
measurement string,
|
||||
tags map[string]string,
|
||||
fields map[string]interface{},
|
||||
ts time.Time,
|
||||
) *write.Point {
|
||||
return write.NewPoint(measurement, tags, fields, ts)
|
||||
}
|
||||
|
||||
// DefaultDialect return flux query Dialect with full annotations (datatype, group, default), header and comma char as a delimiter
|
||||
func DefaultDialect() *domain.Dialect {
|
||||
return api.DefaultDialect()
|
||||
}
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
# Generated types and API client
|
||||
|
||||
`oss.yml`must be periodically synced with latest changes and types and client must be re-generated
|
||||
to maintain full compatibility with the latest InfluxDB release
|
||||
|
||||
|
||||
## Install oapi generator
|
||||
`git clone git@github.com:bonitoo-io/oapi-codegen.git`
|
||||
`cd oapi-codegen`
|
||||
`git checkout feat/template_helpers`
|
||||
`go install ./cmd/oapi-codegen/oapi-codegen.go`
|
||||
|
||||
## Download latest swagger
|
||||
`wget https://raw.githubusercontent.com/influxdata/openapi/master/contracts/oss.yml`
|
||||
`cd domain`
|
||||
|
||||
## Generate
|
||||
### Generate types
|
||||
`oapi-codegen -generate types -o types.gen.go -package domain -templates .\templates oss.yml`
|
||||
|
||||
### Generate client
|
||||
`oapi-codegen -generate client -o client.gen.go -package domain -templates .\templates oss.yml`
|
||||
|
||||
+14852
File diff suppressed because it is too large
Load Diff
+17582
File diff suppressed because it is too large
Load Diff
+8860
File diff suppressed because it is too large
Load Diff
+52
@@ -0,0 +1,52 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc.. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package gzip provides GZip related functionality
|
||||
package gzip
|
||||
|
||||
import (
|
||||
"compress/gzip"
|
||||
"io"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// ReadWaitCloser is ReadCloser that waits for finishing underlying reader
|
||||
type ReadWaitCloser struct {
|
||||
pipeReader *io.PipeReader
|
||||
wg sync.WaitGroup
|
||||
}
|
||||
|
||||
// Close closes underlying reader and waits for finishing operations
|
||||
func (r *ReadWaitCloser) Close() error {
|
||||
err := r.pipeReader.Close()
|
||||
r.wg.Wait() // wait for the gzip goroutine finish
|
||||
return err
|
||||
}
|
||||
|
||||
// CompressWithGzip takes an io.Reader as input and pipes
|
||||
// it through a gzip.Writer returning an io.Reader containing
|
||||
// the gzipped data.
|
||||
// An error is returned if passing data to the gzip.Writer fails
|
||||
// this is shamelessly stolen from https://github.com/influxdata/telegraf
|
||||
func CompressWithGzip(data io.Reader) (io.ReadCloser, error) {
|
||||
pipeReader, pipeWriter := io.Pipe()
|
||||
gzipWriter := gzip.NewWriter(pipeWriter)
|
||||
|
||||
rc := &ReadWaitCloser{
|
||||
pipeReader: pipeReader,
|
||||
}
|
||||
|
||||
rc.wg.Add(1)
|
||||
var err error
|
||||
go func() {
|
||||
_, err = io.Copy(gzipWriter, data)
|
||||
gzipWriter.Close()
|
||||
// subsequent reads from the read half of the pipe will
|
||||
// return no bytes and the error err, or EOF if err is nil.
|
||||
pipeWriter.CloseWithError(err)
|
||||
rc.wg.Done()
|
||||
}()
|
||||
|
||||
return pipeReader, err
|
||||
}
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package http hold internal HTTP related stuff
|
||||
package http
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// UserAgentBase keeps once created base User-Agent string
|
||||
var UserAgentBase string
|
||||
|
||||
// FormatUserAgent creates User-Agent header value for application name
|
||||
func FormatUserAgent(appName string) string {
|
||||
if appName != "" {
|
||||
return fmt.Sprintf("%s %s", UserAgentBase, appName)
|
||||
}
|
||||
return UserAgentBase
|
||||
}
|
||||
+74
@@ -0,0 +1,74 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package log provides internal logging infrastructure
|
||||
package log
|
||||
|
||||
import (
|
||||
ilog "github.com/influxdata/influxdb-client-go/v2/log"
|
||||
)
|
||||
|
||||
// Debugf writes formatted debug message to the Logger instance
|
||||
func Debugf(format string, v ...interface{}) {
|
||||
if ilog.Log != nil {
|
||||
ilog.Log.Debugf(format, v...)
|
||||
}
|
||||
}
|
||||
|
||||
// Debug writes debug message message to the Logger instance
|
||||
func Debug(msg string) {
|
||||
if ilog.Log != nil {
|
||||
ilog.Log.Debug(msg)
|
||||
}
|
||||
}
|
||||
|
||||
// Infof writes formatted info message to the Logger instance
|
||||
func Infof(format string, v ...interface{}) {
|
||||
if ilog.Log != nil {
|
||||
ilog.Log.Infof(format, v...)
|
||||
}
|
||||
}
|
||||
|
||||
// Info writes info message message to the Logger instance
|
||||
func Info(msg string) {
|
||||
if ilog.Log != nil {
|
||||
ilog.Log.Info(msg)
|
||||
}
|
||||
}
|
||||
|
||||
// Warnf writes formatted warning message to the Logger instance
|
||||
func Warnf(format string, v ...interface{}) {
|
||||
if ilog.Log != nil {
|
||||
ilog.Log.Warnf(format, v...)
|
||||
}
|
||||
}
|
||||
|
||||
// Warn writes warning message message to the Logger instance
|
||||
func Warn(msg string) {
|
||||
if ilog.Log != nil {
|
||||
ilog.Log.Warn(msg)
|
||||
}
|
||||
}
|
||||
|
||||
// Errorf writes formatted error message to the Logger instance
|
||||
func Errorf(format string, v ...interface{}) {
|
||||
if ilog.Log != nil {
|
||||
ilog.Log.Errorf(format, v...)
|
||||
}
|
||||
}
|
||||
|
||||
// Error writes error message message to the Logger instance
|
||||
func Error(msg string) {
|
||||
if ilog.Log != nil {
|
||||
ilog.Log.Error(msg)
|
||||
}
|
||||
}
|
||||
|
||||
// Level retrieves current logging level form the Logger instance
|
||||
func Level() uint {
|
||||
if ilog.Log != nil {
|
||||
return ilog.Log.LogLevel()
|
||||
}
|
||||
return ilog.ErrorLevel
|
||||
}
|
||||
+50
@@ -0,0 +1,50 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package write
|
||||
|
||||
import (
|
||||
"container/list"
|
||||
)
|
||||
|
||||
type queue struct {
|
||||
list *list.List
|
||||
limit int
|
||||
}
|
||||
|
||||
func newQueue(limit int) *queue {
|
||||
return &queue{list: list.New(), limit: limit}
|
||||
}
|
||||
func (q *queue) push(batch *Batch) bool {
|
||||
overWrite := false
|
||||
if q.list.Len() == q.limit {
|
||||
q.pop()
|
||||
overWrite = true
|
||||
}
|
||||
q.list.PushBack(batch)
|
||||
return overWrite
|
||||
}
|
||||
|
||||
func (q *queue) pop() *Batch {
|
||||
el := q.list.Front()
|
||||
if el != nil {
|
||||
q.list.Remove(el)
|
||||
batch := el.Value.(*Batch)
|
||||
batch.Evicted = true
|
||||
return batch
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (q *queue) first() *Batch {
|
||||
el := q.list.Front()
|
||||
if el != nil {
|
||||
return el.Value.(*Batch)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (q *queue) isEmpty() bool {
|
||||
return q.list.Len() == 0
|
||||
}
|
||||
+417
@@ -0,0 +1,417 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package write provides service and its stuff
|
||||
package write
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"math/rand"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
http2 "github.com/influxdata/influxdb-client-go/v2/api/http"
|
||||
"github.com/influxdata/influxdb-client-go/v2/api/write"
|
||||
"github.com/influxdata/influxdb-client-go/v2/internal/gzip"
|
||||
"github.com/influxdata/influxdb-client-go/v2/internal/log"
|
||||
ilog "github.com/influxdata/influxdb-client-go/v2/log"
|
||||
lp "github.com/influxdata/line-protocol"
|
||||
)
|
||||
|
||||
// Batch holds information for sending points batch
|
||||
type Batch struct {
|
||||
// lines to send
|
||||
Batch string
|
||||
// retry attempts so far
|
||||
RetryAttempts uint
|
||||
// true if it was removed from queue
|
||||
Evicted bool
|
||||
// time when this batch expires
|
||||
Expires time.Time
|
||||
}
|
||||
|
||||
// NewBatch creates new batch
|
||||
func NewBatch(data string, expireDelayMs uint) *Batch {
|
||||
return &Batch{
|
||||
Batch: data,
|
||||
Expires: time.Now().Add(time.Duration(expireDelayMs) * time.Millisecond),
|
||||
}
|
||||
}
|
||||
|
||||
// BatchErrorCallback is synchronously notified in case non-blocking write fails.
|
||||
// It returns true if WriteAPI should continue with retrying, false will discard the batch.
|
||||
type BatchErrorCallback func(batch *Batch, error2 http2.Error) bool
|
||||
|
||||
// Service is responsible for reliable writing of batches
|
||||
type Service struct {
|
||||
org string
|
||||
bucket string
|
||||
httpService http2.Service
|
||||
url string
|
||||
lastWriteAttempt time.Time
|
||||
retryQueue *queue
|
||||
lock sync.Mutex
|
||||
writeOptions *write.Options
|
||||
retryExponentialBase uint
|
||||
errorCb BatchErrorCallback
|
||||
retryDelay uint
|
||||
retryAttempts uint
|
||||
}
|
||||
|
||||
// NewService creates new write service
|
||||
func NewService(org string, bucket string, httpService http2.Service, options *write.Options) *Service {
|
||||
|
||||
retryBufferLimit := options.RetryBufferLimit() / options.BatchSize()
|
||||
if retryBufferLimit == 0 {
|
||||
retryBufferLimit = 1
|
||||
}
|
||||
u, _ := url.Parse(httpService.ServerAPIURL())
|
||||
u, _ = u.Parse("write")
|
||||
params := u.Query()
|
||||
params.Set("org", org)
|
||||
params.Set("bucket", bucket)
|
||||
params.Set("precision", precisionToString(options.Precision()))
|
||||
if options.Consistency() != "" {
|
||||
params.Set("consistency", string(options.Consistency()))
|
||||
}
|
||||
u.RawQuery = params.Encode()
|
||||
writeURL := u.String()
|
||||
return &Service{
|
||||
org: org,
|
||||
bucket: bucket,
|
||||
httpService: httpService,
|
||||
url: writeURL,
|
||||
writeOptions: options,
|
||||
retryQueue: newQueue(int(retryBufferLimit)),
|
||||
retryExponentialBase: 2,
|
||||
retryDelay: options.RetryInterval(),
|
||||
retryAttempts: 0,
|
||||
}
|
||||
}
|
||||
|
||||
// SetBatchErrorCallback sets callback allowing custom handling of failed writes.
|
||||
// If callback returns true, failed batch will be retried, otherwise discarded.
|
||||
func (w *Service) SetBatchErrorCallback(cb BatchErrorCallback) {
|
||||
w.errorCb = cb
|
||||
}
|
||||
|
||||
// HandleWrite handles writes of batches and handles retrying.
|
||||
// Retrying is triggered by new writes, there is no scheduler.
|
||||
// It first checks retry queue, because it has the highest priority.
|
||||
// If there are some batches in retry queue, those are written and incoming batch is added to end of retry queue.
|
||||
// Immediate write is allowed only in case there was success or not retryable error.
|
||||
// Otherwise, delay is checked based on recent batch.
|
||||
// If write of batch fails with retryable error (connection errors and HTTP code >= 429),
|
||||
// Batch retry time is calculated based on #of attempts.
|
||||
// If writes continues failing and # of attempts reaches maximum or total retry time reaches maxRetryTime,
|
||||
// batch is discarded.
|
||||
func (w *Service) HandleWrite(ctx context.Context, batch *Batch) error {
|
||||
log.Debug("Write proc: received write request")
|
||||
batchToWrite := batch
|
||||
retrying := false
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
log.Debug("Write proc: ctx cancelled req")
|
||||
return ctx.Err()
|
||||
default:
|
||||
}
|
||||
if !w.retryQueue.isEmpty() {
|
||||
log.Debug("Write proc: taking batch from retry queue")
|
||||
if !retrying {
|
||||
b := w.retryQueue.first()
|
||||
|
||||
// Discard batches at beginning of retryQueue that have already expired
|
||||
if time.Now().After(b.Expires) {
|
||||
log.Error("Write proc: oldest batch in retry queue expired, discarding")
|
||||
if !b.Evicted {
|
||||
w.retryQueue.pop()
|
||||
}
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
// Can we write? In case of retryable error we must wait a bit
|
||||
if w.lastWriteAttempt.IsZero() || time.Now().After(w.lastWriteAttempt.Add(time.Millisecond*time.Duration(w.retryDelay))) {
|
||||
retrying = true
|
||||
} else {
|
||||
log.Warn("Write proc: cannot write yet, storing batch to queue")
|
||||
if w.retryQueue.push(batch) {
|
||||
log.Error("Write proc: Retry buffer full, discarding oldest batch")
|
||||
}
|
||||
batchToWrite = nil
|
||||
}
|
||||
}
|
||||
if retrying {
|
||||
batchToWrite = w.retryQueue.first()
|
||||
if batch != nil { //store actual batch to retry queue
|
||||
if w.retryQueue.push(batch) {
|
||||
log.Error("Write proc: Retry buffer full, discarding oldest batch")
|
||||
}
|
||||
batch = nil
|
||||
}
|
||||
}
|
||||
}
|
||||
// write batch
|
||||
if batchToWrite != nil {
|
||||
perror := w.WriteBatch(ctx, batchToWrite)
|
||||
if perror != nil {
|
||||
if isIgnorableError(perror) {
|
||||
log.Warnf("Write error: %s", perror.Error())
|
||||
} else {
|
||||
if w.writeOptions.MaxRetries() != 0 && (perror.StatusCode == 0 || perror.StatusCode >= http.StatusTooManyRequests) {
|
||||
log.Errorf("Write error: %s, batch kept for retrying\n", perror.Error())
|
||||
if perror.RetryAfter > 0 {
|
||||
w.retryDelay = perror.RetryAfter * 1000
|
||||
} else {
|
||||
w.retryDelay = w.computeRetryDelay(w.retryAttempts)
|
||||
}
|
||||
if w.errorCb != nil && !w.errorCb(batchToWrite, *perror) {
|
||||
log.Error("Callback rejected batch, discarding")
|
||||
if !batchToWrite.Evicted {
|
||||
w.retryQueue.pop()
|
||||
}
|
||||
return perror
|
||||
}
|
||||
// store new batch (not taken from queue)
|
||||
if !batchToWrite.Evicted && batchToWrite != w.retryQueue.first() {
|
||||
if w.retryQueue.push(batch) {
|
||||
log.Error("Retry buffer full, discarding oldest batch")
|
||||
}
|
||||
} else if batchToWrite.RetryAttempts == w.writeOptions.MaxRetries() {
|
||||
log.Error("Reached maximum number of retries, discarding batch")
|
||||
if !batchToWrite.Evicted {
|
||||
w.retryQueue.pop()
|
||||
}
|
||||
}
|
||||
batchToWrite.RetryAttempts++
|
||||
w.retryAttempts++
|
||||
log.Debugf("Write proc: next wait for write is %dms\n", w.retryDelay)
|
||||
} else {
|
||||
log.Errorf("Write error: %s\n", perror.Error())
|
||||
}
|
||||
return fmt.Errorf("write failed (attempts %d): %w", batchToWrite.RetryAttempts, perror)
|
||||
}
|
||||
}
|
||||
|
||||
w.retryDelay = w.writeOptions.RetryInterval()
|
||||
w.retryAttempts = 0
|
||||
if retrying && !batchToWrite.Evicted {
|
||||
w.retryQueue.pop()
|
||||
}
|
||||
batchToWrite = nil
|
||||
} else {
|
||||
break
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Non-retryable errors
|
||||
const (
|
||||
errStringHintedHandoffNotEmpty = "hinted handoff queue not empty"
|
||||
errStringPartialWrite = "partial write"
|
||||
errStringPointsBeyondRP = "points beyond retention policy"
|
||||
errStringUnableToParse = "unable to parse"
|
||||
)
|
||||
|
||||
func isIgnorableError(error *http2.Error) bool {
|
||||
// This "error" is an informational message about the state of the
|
||||
// InfluxDB cluster.
|
||||
if strings.Contains(error.Message, errStringHintedHandoffNotEmpty) {
|
||||
return true
|
||||
}
|
||||
// Points beyond retention policy is returned when points are immediately
|
||||
// discarded for being older than the retention policy. Usually this not
|
||||
// a cause for concern, and we don't want to retry.
|
||||
if strings.Contains(error.Message, errStringPointsBeyondRP) {
|
||||
return true
|
||||
}
|
||||
// Other partial write errors, such as "field type conflict", are not
|
||||
// correctable at this point and so the point is dropped instead of
|
||||
// retrying.
|
||||
if strings.Contains(error.Message, errStringPartialWrite) {
|
||||
return true
|
||||
}
|
||||
// This error indicates an error in line protocol
|
||||
// serialization, retries would not be successful.
|
||||
if strings.Contains(error.Message, errStringUnableToParse) {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// computeRetryDelay calculates retry delay.
|
||||
// Retry delay is calculated as random value within the interval
|
||||
// [retry_interval * exponential_base^(attempts) and retry_interval * exponential_base^(attempts+1)]
|
||||
func (w *Service) computeRetryDelay(attempts uint) uint {
|
||||
minDelay := int(w.writeOptions.RetryInterval() * pow(w.writeOptions.ExponentialBase(), attempts))
|
||||
maxDelay := int(w.writeOptions.RetryInterval() * pow(w.writeOptions.ExponentialBase(), attempts+1))
|
||||
diff := maxDelay - minDelay
|
||||
if diff <= 0 { //check overflows
|
||||
return w.writeOptions.MaxRetryInterval()
|
||||
}
|
||||
retryDelay := uint(rand.Intn(diff) + minDelay)
|
||||
if retryDelay > w.writeOptions.MaxRetryInterval() {
|
||||
retryDelay = w.writeOptions.MaxRetryInterval()
|
||||
}
|
||||
return retryDelay
|
||||
}
|
||||
|
||||
// pow computes x**y
|
||||
func pow(x, y uint) uint {
|
||||
p := uint(1)
|
||||
if y == 0 {
|
||||
return 1
|
||||
}
|
||||
for i := uint(1); i <= y; i++ {
|
||||
p = p * x
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// WriteBatch performs actual writing via HTTP service
|
||||
func (w *Service) WriteBatch(ctx context.Context, batch *Batch) *http2.Error {
|
||||
var body io.Reader
|
||||
var err error
|
||||
body = strings.NewReader(batch.Batch)
|
||||
|
||||
if log.Level() >= ilog.DebugLevel {
|
||||
log.Debugf("Writing batch: %s", batch.Batch)
|
||||
}
|
||||
if w.writeOptions.UseGZip() {
|
||||
body, err = gzip.CompressWithGzip(body)
|
||||
if err != nil {
|
||||
return http2.NewError(err)
|
||||
}
|
||||
}
|
||||
w.lock.Lock()
|
||||
w.lastWriteAttempt = time.Now()
|
||||
w.lock.Unlock()
|
||||
perror := w.httpService.DoPostRequest(ctx, w.url, body, func(req *http.Request) {
|
||||
if w.writeOptions.UseGZip() {
|
||||
req.Header.Set("Content-Encoding", "gzip")
|
||||
}
|
||||
}, func(r *http.Response) error {
|
||||
return r.Body.Close()
|
||||
})
|
||||
return perror
|
||||
}
|
||||
|
||||
// Flush sends batches from retry queue immediately, without retrying
|
||||
func (w *Service) Flush() {
|
||||
for !w.retryQueue.isEmpty() {
|
||||
b := w.retryQueue.pop()
|
||||
if time.Now().After(b.Expires) {
|
||||
log.Error("Oldest batch in retry queue expired, discarding")
|
||||
continue
|
||||
}
|
||||
if err := w.WriteBatch(context.Background(), b); err != nil {
|
||||
log.Errorf("Error flushing batch from retry queue: %w", err.Unwrap())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// pointWithDefaultTags encapsulates Point with default tags
|
||||
type pointWithDefaultTags struct {
|
||||
point *write.Point
|
||||
defaultTags map[string]string
|
||||
}
|
||||
|
||||
// Name returns the name of measurement of a point.
|
||||
func (p *pointWithDefaultTags) Name() string {
|
||||
return p.point.Name()
|
||||
}
|
||||
|
||||
// Time is the timestamp of a Point.
|
||||
func (p *pointWithDefaultTags) Time() time.Time {
|
||||
return p.point.Time()
|
||||
}
|
||||
|
||||
// FieldList returns a slice containing the fields of a Point.
|
||||
func (p *pointWithDefaultTags) FieldList() []*lp.Field {
|
||||
return p.point.FieldList()
|
||||
}
|
||||
|
||||
// TagList returns tags from point along with default tags
|
||||
// If point of tag can override default tag
|
||||
func (p *pointWithDefaultTags) TagList() []*lp.Tag {
|
||||
tags := make([]*lp.Tag, 0, len(p.point.TagList())+len(p.defaultTags))
|
||||
tags = append(tags, p.point.TagList()...)
|
||||
for k, v := range p.defaultTags {
|
||||
if !existTag(p.point.TagList(), k) {
|
||||
tags = append(tags, &lp.Tag{
|
||||
Key: k,
|
||||
Value: v,
|
||||
})
|
||||
}
|
||||
}
|
||||
sort.Slice(tags, func(i, j int) bool { return tags[i].Key < tags[j].Key })
|
||||
return tags
|
||||
}
|
||||
|
||||
func existTag(tags []*lp.Tag, key string) bool {
|
||||
for _, tag := range tags {
|
||||
if key == tag.Key {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// EncodePoints creates line protocol string from points
|
||||
func (w *Service) EncodePoints(points ...*write.Point) (string, error) {
|
||||
var buffer bytes.Buffer
|
||||
e := lp.NewEncoder(&buffer)
|
||||
e.SetFieldTypeSupport(lp.UintSupport)
|
||||
e.FailOnFieldErr(true)
|
||||
e.SetPrecision(w.writeOptions.Precision())
|
||||
for _, point := range points {
|
||||
_, err := e.Encode(w.pointToEncode(point))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
return buffer.String(), nil
|
||||
}
|
||||
|
||||
// pointToEncode determines whether default tags should be applied
|
||||
// and returns point with default tags instead of point
|
||||
func (w *Service) pointToEncode(point *write.Point) lp.Metric {
|
||||
var m lp.Metric
|
||||
if len(w.writeOptions.DefaultTags()) > 0 {
|
||||
m = &pointWithDefaultTags{
|
||||
point: point,
|
||||
defaultTags: w.writeOptions.DefaultTags(),
|
||||
}
|
||||
} else {
|
||||
m = point
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// WriteURL returns current write URL
|
||||
func (w *Service) WriteURL() string {
|
||||
return w.url
|
||||
}
|
||||
|
||||
func precisionToString(precision time.Duration) string {
|
||||
prec := "ns"
|
||||
switch precision {
|
||||
case time.Microsecond:
|
||||
prec = "us"
|
||||
case time.Millisecond:
|
||||
prec = "ms"
|
||||
case time.Second:
|
||||
prec = "s"
|
||||
}
|
||||
return prec
|
||||
}
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package log defines Logging API.
|
||||
// The global Log variable contains the actual logger. Set it to own implementation to override logging. Set it to nil to disable logging
|
||||
package log
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// Log is the library wide logger. Setting to nil disables logging.
|
||||
var Log Logger = &logger{logLevel: ErrorLevel, prefix: "influxdb2client"}
|
||||
|
||||
// Log levels
|
||||
const (
|
||||
ErrorLevel uint = iota
|
||||
WarningLevel
|
||||
InfoLevel
|
||||
DebugLevel
|
||||
)
|
||||
|
||||
// Logger defines interface for logging
|
||||
type Logger interface {
|
||||
// Writes formatted debug message if debug logLevel is enabled.
|
||||
Debugf(format string, v ...interface{})
|
||||
// Writes debug message if debug is enabled.
|
||||
Debug(msg string)
|
||||
// Writes formatted info message if info logLevel is enabled.
|
||||
Infof(format string, v ...interface{})
|
||||
// Writes info message if info logLevel is enabled
|
||||
Info(msg string)
|
||||
// Writes formatted warning message if warning logLevel is enabled.
|
||||
Warnf(format string, v ...interface{})
|
||||
// Writes warning message if warning logLevel is enabled.
|
||||
Warn(msg string)
|
||||
// Writes formatted error message
|
||||
Errorf(format string, v ...interface{})
|
||||
// Writes error message
|
||||
Error(msg string)
|
||||
// SetLogLevel sets allowed logging level.
|
||||
SetLogLevel(logLevel uint)
|
||||
// LogLevel retrieves current logging level
|
||||
LogLevel() uint
|
||||
// SetPrefix sets logging prefix.
|
||||
SetPrefix(prefix string)
|
||||
}
|
||||
|
||||
// logger provides default implementation for Logger. It logs using Go log API
|
||||
// mutex is needed in cases when multiple clients run concurrently
|
||||
type logger struct {
|
||||
prefix string
|
||||
logLevel uint
|
||||
lock sync.Mutex
|
||||
}
|
||||
|
||||
func (l *logger) SetLogLevel(logLevel uint) {
|
||||
l.lock.Lock()
|
||||
defer l.lock.Unlock()
|
||||
l.logLevel = logLevel
|
||||
}
|
||||
|
||||
func (l *logger) LogLevel() uint {
|
||||
l.lock.Lock()
|
||||
defer l.lock.Unlock()
|
||||
return l.logLevel
|
||||
}
|
||||
|
||||
func (l *logger) SetPrefix(prefix string) {
|
||||
l.lock.Lock()
|
||||
defer l.lock.Unlock()
|
||||
l.prefix = prefix
|
||||
}
|
||||
|
||||
func (l *logger) Debugf(format string, v ...interface{}) {
|
||||
l.lock.Lock()
|
||||
defer l.lock.Unlock()
|
||||
if l.logLevel >= DebugLevel {
|
||||
log.Print(l.prefix, " D! ", fmt.Sprintf(format, v...))
|
||||
}
|
||||
}
|
||||
func (l *logger) Debug(msg string) {
|
||||
l.lock.Lock()
|
||||
defer l.lock.Unlock()
|
||||
if l.logLevel >= DebugLevel {
|
||||
log.Print(l.prefix, " D! ", msg)
|
||||
}
|
||||
}
|
||||
|
||||
func (l *logger) Infof(format string, v ...interface{}) {
|
||||
l.lock.Lock()
|
||||
defer l.lock.Unlock()
|
||||
if l.logLevel >= InfoLevel {
|
||||
log.Print(l.prefix, " I! ", fmt.Sprintf(format, v...))
|
||||
}
|
||||
}
|
||||
func (l *logger) Info(msg string) {
|
||||
l.lock.Lock()
|
||||
defer l.lock.Unlock()
|
||||
if l.logLevel >= DebugLevel {
|
||||
log.Print(l.prefix, " I! ", msg)
|
||||
}
|
||||
}
|
||||
|
||||
func (l *logger) Warnf(format string, v ...interface{}) {
|
||||
l.lock.Lock()
|
||||
defer l.lock.Unlock()
|
||||
if l.logLevel >= WarningLevel {
|
||||
log.Print(l.prefix, " W! ", fmt.Sprintf(format, v...))
|
||||
}
|
||||
}
|
||||
func (l *logger) Warn(msg string) {
|
||||
l.lock.Lock()
|
||||
defer l.lock.Unlock()
|
||||
if l.logLevel >= WarningLevel {
|
||||
log.Print(l.prefix, " W! ", msg)
|
||||
}
|
||||
}
|
||||
|
||||
func (l *logger) Errorf(format string, v ...interface{}) {
|
||||
l.lock.Lock()
|
||||
defer l.lock.Unlock()
|
||||
log.Print(l.prefix, " E! ", fmt.Sprintf(format, v...))
|
||||
}
|
||||
|
||||
func (l *logger) Error(msg string) {
|
||||
l.lock.Lock()
|
||||
defer l.lock.Unlock()
|
||||
log.Print(l.prefix, " E! ", msg)
|
||||
}
|
||||
+236
@@ -0,0 +1,236 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package influxdb2
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
nethttp "net/http"
|
||||
"time"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2/api/http"
|
||||
"github.com/influxdata/influxdb-client-go/v2/api/write"
|
||||
)
|
||||
|
||||
// Options holds configuration properties for communicating with InfluxDB server
|
||||
type Options struct {
|
||||
// LogLevel to filter log messages. Each level mean to log all categories bellow. 0 error, 1 - warning, 2 - info, 3 - debug
|
||||
logLevel uint
|
||||
// Writing options
|
||||
writeOptions *write.Options
|
||||
// Http options
|
||||
httpOptions *http.Options
|
||||
}
|
||||
|
||||
// BatchSize returns size of batch
|
||||
func (o *Options) BatchSize() uint {
|
||||
return o.WriteOptions().BatchSize()
|
||||
}
|
||||
|
||||
// SetBatchSize sets number of points sent in single request
|
||||
func (o *Options) SetBatchSize(batchSize uint) *Options {
|
||||
o.WriteOptions().SetBatchSize(batchSize)
|
||||
return o
|
||||
}
|
||||
|
||||
// FlushInterval returns flush interval in ms
|
||||
func (o *Options) FlushInterval() uint {
|
||||
return o.WriteOptions().FlushInterval()
|
||||
}
|
||||
|
||||
// SetFlushInterval sets flush interval in ms in which is buffer flushed if it has not been already written
|
||||
func (o *Options) SetFlushInterval(flushIntervalMs uint) *Options {
|
||||
o.WriteOptions().SetFlushInterval(flushIntervalMs)
|
||||
return o
|
||||
}
|
||||
|
||||
// RetryInterval returns the retry interval in ms
|
||||
func (o *Options) RetryInterval() uint {
|
||||
return o.WriteOptions().RetryInterval()
|
||||
}
|
||||
|
||||
// SetRetryInterval sets retry interval in ms, which is set if not sent by server
|
||||
func (o *Options) SetRetryInterval(retryIntervalMs uint) *Options {
|
||||
o.WriteOptions().SetRetryInterval(retryIntervalMs)
|
||||
return o
|
||||
}
|
||||
|
||||
// MaxRetries returns maximum count of retry attempts of failed writes, default 5.
|
||||
func (o *Options) MaxRetries() uint {
|
||||
return o.WriteOptions().MaxRetries()
|
||||
}
|
||||
|
||||
// SetMaxRetries sets maximum count of retry attempts of failed writes.
|
||||
// Setting zero value disables retry strategy.
|
||||
func (o *Options) SetMaxRetries(maxRetries uint) *Options {
|
||||
o.WriteOptions().SetMaxRetries(maxRetries)
|
||||
return o
|
||||
}
|
||||
|
||||
// RetryBufferLimit returns retry buffer limit
|
||||
func (o *Options) RetryBufferLimit() uint {
|
||||
return o.WriteOptions().RetryBufferLimit()
|
||||
}
|
||||
|
||||
// SetRetryBufferLimit sets maximum number of points to keep for retry. Should be multiple of BatchSize.
|
||||
func (o *Options) SetRetryBufferLimit(retryBufferLimit uint) *Options {
|
||||
o.WriteOptions().SetRetryBufferLimit(retryBufferLimit)
|
||||
return o
|
||||
}
|
||||
|
||||
// MaxRetryInterval returns the maximum delay between each retry attempt in milliseconds, default 125,000.
|
||||
func (o *Options) MaxRetryInterval() uint {
|
||||
return o.WriteOptions().MaxRetryInterval()
|
||||
}
|
||||
|
||||
// SetMaxRetryInterval sets the maximum delay between each retry attempt in millisecond.
|
||||
func (o *Options) SetMaxRetryInterval(maxRetryIntervalMs uint) *Options {
|
||||
o.WriteOptions().SetMaxRetryInterval(maxRetryIntervalMs)
|
||||
return o
|
||||
}
|
||||
|
||||
// MaxRetryTime returns the maximum total retry timeout in millisecond, default 180,000.
|
||||
func (o *Options) MaxRetryTime() uint {
|
||||
return o.WriteOptions().MaxRetryTime()
|
||||
}
|
||||
|
||||
// SetMaxRetryTime sets the maximum total retry timeout in millisecond.
|
||||
func (o *Options) SetMaxRetryTime(maxRetryTimeMs uint) *Options {
|
||||
o.WriteOptions().SetMaxRetryTime(maxRetryTimeMs)
|
||||
return o
|
||||
}
|
||||
|
||||
// ExponentialBase returns the base for the exponential retry delay. Default 2.
|
||||
func (o *Options) ExponentialBase() uint {
|
||||
return o.WriteOptions().ExponentialBase()
|
||||
}
|
||||
|
||||
// SetExponentialBase sets the base for the exponential retry delay.
|
||||
func (o *Options) SetExponentialBase(exponentialBase uint) *Options {
|
||||
o.WriteOptions().SetExponentialBase(exponentialBase)
|
||||
return o
|
||||
}
|
||||
|
||||
// LogLevel returns log level
|
||||
func (o *Options) LogLevel() uint {
|
||||
return o.logLevel
|
||||
}
|
||||
|
||||
// SetLogLevel set level to filter log messages. Each level mean to log all categories bellow. Default is ErrorLevel.
|
||||
// There are four level constant int the log package in this library:
|
||||
// - ErrorLevel
|
||||
// - WarningLevel
|
||||
// - InfoLevel
|
||||
// - DebugLevel
|
||||
// The DebugLevel will print also content of writen batches, queries.
|
||||
// The InfoLevel prints HTTP requests info, among others.
|
||||
// Set log.Log to nil in order to completely disable logging.
|
||||
func (o *Options) SetLogLevel(logLevel uint) *Options {
|
||||
o.logLevel = logLevel
|
||||
return o
|
||||
}
|
||||
|
||||
// Precision returns time precision for writes
|
||||
func (o *Options) Precision() time.Duration {
|
||||
return o.WriteOptions().Precision()
|
||||
}
|
||||
|
||||
// SetPrecision sets time precision to use in writes for timestamp. In unit of duration: time.Nanosecond, time.Microsecond, time.Millisecond, time.Second
|
||||
func (o *Options) SetPrecision(precision time.Duration) *Options {
|
||||
o.WriteOptions().SetPrecision(precision)
|
||||
return o
|
||||
}
|
||||
|
||||
// UseGZip returns true if write request are gzip`ed
|
||||
func (o *Options) UseGZip() bool {
|
||||
return o.WriteOptions().UseGZip()
|
||||
}
|
||||
|
||||
// SetUseGZip specifies whether to use GZip compression in write requests.
|
||||
func (o *Options) SetUseGZip(useGZip bool) *Options {
|
||||
o.WriteOptions().SetUseGZip(useGZip)
|
||||
return o
|
||||
}
|
||||
|
||||
// HTTPClient returns the http.Client that is configured to be used
|
||||
// for HTTP requests. It will return the one that has been set using
|
||||
// SetHTTPClient or it will construct a default client using the
|
||||
// other configured options.
|
||||
func (o *Options) HTTPClient() *nethttp.Client {
|
||||
return o.httpOptions.HTTPClient()
|
||||
}
|
||||
|
||||
// SetHTTPClient will configure the http.Client that is used
|
||||
// for HTTP requests. If set to nil, an HTTPClient will be
|
||||
// generated.
|
||||
//
|
||||
// Setting the HTTPClient will cause the other HTTP options
|
||||
// to be ignored.
|
||||
// In case of UsersAPI.SignIn() is used, HTTPClient.Jar will be used for storing session cookie.
|
||||
func (o *Options) SetHTTPClient(c *nethttp.Client) *Options {
|
||||
o.httpOptions.SetHTTPClient(c)
|
||||
return o
|
||||
}
|
||||
|
||||
// TLSConfig returns TLS config
|
||||
func (o *Options) TLSConfig() *tls.Config {
|
||||
return o.HTTPOptions().TLSConfig()
|
||||
}
|
||||
|
||||
// SetTLSConfig sets TLS configuration for secure connection
|
||||
func (o *Options) SetTLSConfig(tlsConfig *tls.Config) *Options {
|
||||
o.HTTPOptions().SetTLSConfig(tlsConfig)
|
||||
return o
|
||||
}
|
||||
|
||||
// HTTPRequestTimeout returns HTTP request timeout
|
||||
func (o *Options) HTTPRequestTimeout() uint {
|
||||
return o.HTTPOptions().HTTPRequestTimeout()
|
||||
}
|
||||
|
||||
// SetHTTPRequestTimeout sets HTTP request timeout in sec
|
||||
func (o *Options) SetHTTPRequestTimeout(httpRequestTimeout uint) *Options {
|
||||
o.HTTPOptions().SetHTTPRequestTimeout(httpRequestTimeout)
|
||||
return o
|
||||
}
|
||||
|
||||
// WriteOptions returns write related options
|
||||
func (o *Options) WriteOptions() *write.Options {
|
||||
if o.writeOptions == nil {
|
||||
o.writeOptions = write.DefaultOptions()
|
||||
}
|
||||
return o.writeOptions
|
||||
}
|
||||
|
||||
// HTTPOptions returns HTTP related options
|
||||
func (o *Options) HTTPOptions() *http.Options {
|
||||
if o.httpOptions == nil {
|
||||
o.httpOptions = http.DefaultOptions()
|
||||
}
|
||||
return o.httpOptions
|
||||
}
|
||||
|
||||
// AddDefaultTag adds a default tag. DefaultTags are added to each written point.
|
||||
// If a tag with the same key already exist it is overwritten.
|
||||
// If a point already defines such a tag, it is left unchanged
|
||||
func (o *Options) AddDefaultTag(key, value string) *Options {
|
||||
o.WriteOptions().AddDefaultTag(key, value)
|
||||
return o
|
||||
}
|
||||
|
||||
// ApplicationName returns application name used in the User-Agent HTTP header
|
||||
func (o *Options) ApplicationName() string {
|
||||
return o.HTTPOptions().ApplicationName()
|
||||
}
|
||||
|
||||
// SetApplicationName sets an application name to the User-Agent HTTP header
|
||||
func (o *Options) SetApplicationName(appName string) *Options {
|
||||
o.HTTPOptions().SetApplicationName(appName)
|
||||
return o
|
||||
}
|
||||
|
||||
// DefaultOptions returns Options object with default values
|
||||
func DefaultOptions() *Options {
|
||||
return &Options{logLevel: 0, writeOptions: write.DefaultOptions(), httpOptions: http.DefaultOptions()}
|
||||
}
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
||||
// Use of this source code is governed by MIT
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package influxdb2
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"runtime"
|
||||
|
||||
"github.com/influxdata/influxdb-client-go/v2/internal/http"
|
||||
)
|
||||
|
||||
const (
|
||||
// Version defines current version
|
||||
Version = "2.12.2"
|
||||
)
|
||||
|
||||
func init() {
|
||||
http.UserAgentBase = fmt.Sprintf("influxdb-client-go/%s (%s; %s)", Version, runtime.GOOS, runtime.GOARCH)
|
||||
}
|
||||
-763
@@ -1,763 +0,0 @@
|
||||
// Package client (v2) is the current official Go client for InfluxDB.
|
||||
package client // import "github.com/influxdata/influxdb/client/v2"
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"mime"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/influxdata/influxdb/models"
|
||||
)
|
||||
|
||||
// HTTPConfig is the config data needed to create an HTTP Client.
|
||||
type HTTPConfig struct {
|
||||
// Addr should be of the form "http://host:port"
|
||||
// or "http://[ipv6-host%zone]:port".
|
||||
Addr string
|
||||
|
||||
// Username is the influxdb username, optional.
|
||||
Username string
|
||||
|
||||
// Password is the influxdb password, optional.
|
||||
Password string
|
||||
|
||||
// UserAgent is the http User Agent, defaults to "InfluxDBClient".
|
||||
UserAgent string
|
||||
|
||||
// Timeout for influxdb writes, defaults to no timeout.
|
||||
Timeout time.Duration
|
||||
|
||||
// InsecureSkipVerify gets passed to the http client, if true, it will
|
||||
// skip https certificate verification. Defaults to false.
|
||||
InsecureSkipVerify bool
|
||||
|
||||
// TLSConfig allows the user to set their own TLS config for the HTTP
|
||||
// Client. If set, this option overrides InsecureSkipVerify.
|
||||
TLSConfig *tls.Config
|
||||
|
||||
// Proxy configures the Proxy function on the HTTP client.
|
||||
Proxy func(req *http.Request) (*url.URL, error)
|
||||
|
||||
// DialContext specifies the dial function for creating unencrypted TCP connections.
|
||||
// If DialContext is nil then the transport dials using package net.
|
||||
DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
|
||||
}
|
||||
|
||||
// BatchPointsConfig is the config data needed to create an instance of the BatchPoints struct.
|
||||
type BatchPointsConfig struct {
|
||||
// Precision is the write precision of the points, defaults to "ns".
|
||||
Precision string
|
||||
|
||||
// Database is the database to write points to.
|
||||
Database string
|
||||
|
||||
// RetentionPolicy is the retention policy of the points.
|
||||
RetentionPolicy string
|
||||
|
||||
// Write consistency is the number of servers required to confirm write.
|
||||
WriteConsistency string
|
||||
}
|
||||
|
||||
// Client is a client interface for writing & querying the database.
|
||||
type Client interface {
|
||||
// Ping checks that status of cluster, and will always return 0 time and no
|
||||
// error for UDP clients.
|
||||
Ping(timeout time.Duration) (time.Duration, string, error)
|
||||
|
||||
// Write takes a BatchPoints object and writes all Points to InfluxDB.
|
||||
Write(bp BatchPoints) error
|
||||
|
||||
// WriteCtx takes a BatchPoints object and writes all Points to InfluxDB.
|
||||
WriteCtx(ctx context.Context, bp BatchPoints) error
|
||||
|
||||
// Query makes an InfluxDB Query on the database. This will fail if using
|
||||
// the UDP client.
|
||||
Query(q Query) (*Response, error)
|
||||
|
||||
// QueryCtx makes an InfluxDB Query on the database. This will fail if using
|
||||
// the UDP client.
|
||||
QueryCtx(ctx context.Context, q Query) (*Response, error)
|
||||
|
||||
// QueryAsChunk makes an InfluxDB Query on the database. This will fail if using
|
||||
// the UDP client.
|
||||
QueryAsChunk(q Query) (*ChunkedResponse, error)
|
||||
|
||||
// Close releases any resources a Client may be using.
|
||||
Close() error
|
||||
}
|
||||
|
||||
// For added performance users may want to send pre-serialized points.
|
||||
type HTTPClient interface {
|
||||
Client
|
||||
WriteRawCtx(ctx context.Context, bp BatchPoints, reqBody io.Reader) error
|
||||
}
|
||||
|
||||
// NewHTTPClient returns a new Client from the provided config.
|
||||
// Client is safe for concurrent use by multiple goroutines.
|
||||
func NewHTTPClient(conf HTTPConfig) (HTTPClient, error) {
|
||||
if conf.UserAgent == "" {
|
||||
conf.UserAgent = "InfluxDBClient"
|
||||
}
|
||||
|
||||
u, err := url.Parse(conf.Addr)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
} else if u.Scheme != "http" && u.Scheme != "https" {
|
||||
m := fmt.Sprintf("Unsupported protocol scheme: %s, your address"+
|
||||
" must start with http:// or https://", u.Scheme)
|
||||
return nil, errors.New(m)
|
||||
}
|
||||
|
||||
tr := &http.Transport{
|
||||
TLSClientConfig: &tls.Config{
|
||||
InsecureSkipVerify: conf.InsecureSkipVerify,
|
||||
},
|
||||
Proxy: conf.Proxy,
|
||||
DialContext: conf.DialContext,
|
||||
}
|
||||
if conf.TLSConfig != nil {
|
||||
tr.TLSClientConfig = conf.TLSConfig
|
||||
// Make sure to preserve the InsecureSkipVerify setting from the config.
|
||||
tr.TLSClientConfig.InsecureSkipVerify = conf.InsecureSkipVerify
|
||||
}
|
||||
return &client{
|
||||
url: *u,
|
||||
username: conf.Username,
|
||||
password: conf.Password,
|
||||
useragent: conf.UserAgent,
|
||||
httpClient: &http.Client{
|
||||
Timeout: conf.Timeout,
|
||||
Transport: tr,
|
||||
},
|
||||
transport: tr,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Ping will check to see if the server is up with an optional timeout on waiting for leader.
|
||||
// Ping returns how long the request took, the version of the server it connected to, and an error if one occurred.
|
||||
func (c *client) Ping(timeout time.Duration) (time.Duration, string, error) {
|
||||
now := time.Now()
|
||||
|
||||
u := c.url
|
||||
u.Path = path.Join(u.Path, "ping")
|
||||
|
||||
req, err := http.NewRequest("GET", u.String(), nil)
|
||||
if err != nil {
|
||||
return 0, "", err
|
||||
}
|
||||
|
||||
req.Header.Set("User-Agent", c.useragent)
|
||||
|
||||
if c.username != "" {
|
||||
req.SetBasicAuth(c.username, c.password)
|
||||
}
|
||||
|
||||
if timeout > 0 {
|
||||
params := req.URL.Query()
|
||||
params.Set("wait_for_leader", fmt.Sprintf("%.0fs", timeout.Seconds()))
|
||||
req.URL.RawQuery = params.Encode()
|
||||
}
|
||||
|
||||
resp, err := c.httpClient.Do(req)
|
||||
if err != nil {
|
||||
return 0, "", err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return 0, "", err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusNoContent {
|
||||
var err = errors.New(string(body))
|
||||
return 0, "", err
|
||||
}
|
||||
|
||||
version := resp.Header.Get("X-Influxdb-Version")
|
||||
return time.Since(now), version, nil
|
||||
}
|
||||
|
||||
// Close releases the client's resources.
|
||||
func (c *client) Close() error {
|
||||
c.transport.CloseIdleConnections()
|
||||
return nil
|
||||
}
|
||||
|
||||
// client is safe for concurrent use as the fields are all read-only
|
||||
// once the client is instantiated.
|
||||
type client struct {
|
||||
// N.B - if url.UserInfo is accessed in future modifications to the
|
||||
// methods on client, you will need to synchronize access to url.
|
||||
url url.URL
|
||||
username string
|
||||
password string
|
||||
useragent string
|
||||
httpClient *http.Client
|
||||
transport *http.Transport
|
||||
}
|
||||
|
||||
// BatchPoints is an interface into a batched grouping of points to write into
|
||||
// InfluxDB together. BatchPoints is NOT thread-safe, you must create a separate
|
||||
// batch for each goroutine.
|
||||
type BatchPoints interface {
|
||||
// AddPoint adds the given point to the Batch of points.
|
||||
AddPoint(p *Point)
|
||||
// AddPoints adds the given points to the Batch of points.
|
||||
AddPoints(ps []*Point)
|
||||
// Points lists the points in the Batch.
|
||||
Points() []*Point
|
||||
|
||||
// Precision returns the currently set precision of this Batch.
|
||||
Precision() string
|
||||
// SetPrecision sets the precision of this batch.
|
||||
SetPrecision(s string) error
|
||||
|
||||
// Database returns the currently set database of this Batch.
|
||||
Database() string
|
||||
// SetDatabase sets the database of this Batch.
|
||||
SetDatabase(s string)
|
||||
|
||||
// WriteConsistency returns the currently set write consistency of this Batch.
|
||||
WriteConsistency() string
|
||||
// SetWriteConsistency sets the write consistency of this Batch.
|
||||
SetWriteConsistency(s string)
|
||||
|
||||
// RetentionPolicy returns the currently set retention policy of this Batch.
|
||||
RetentionPolicy() string
|
||||
// SetRetentionPolicy sets the retention policy of this Batch.
|
||||
SetRetentionPolicy(s string)
|
||||
}
|
||||
|
||||
// NewBatchPoints returns a BatchPoints interface based on the given config.
|
||||
func NewBatchPoints(conf BatchPointsConfig) (BatchPoints, error) {
|
||||
if conf.Precision == "" {
|
||||
conf.Precision = "ns"
|
||||
}
|
||||
if _, err := time.ParseDuration("1" + conf.Precision); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
bp := &batchpoints{
|
||||
database: conf.Database,
|
||||
precision: conf.Precision,
|
||||
retentionPolicy: conf.RetentionPolicy,
|
||||
writeConsistency: conf.WriteConsistency,
|
||||
}
|
||||
return bp, nil
|
||||
}
|
||||
|
||||
type batchpoints struct {
|
||||
points []*Point
|
||||
database string
|
||||
precision string
|
||||
retentionPolicy string
|
||||
writeConsistency string
|
||||
}
|
||||
|
||||
func (bp *batchpoints) AddPoint(p *Point) {
|
||||
bp.points = append(bp.points, p)
|
||||
}
|
||||
|
||||
func (bp *batchpoints) AddPoints(ps []*Point) {
|
||||
bp.points = append(bp.points, ps...)
|
||||
}
|
||||
|
||||
func (bp *batchpoints) Points() []*Point {
|
||||
return bp.points
|
||||
}
|
||||
|
||||
func (bp *batchpoints) Precision() string {
|
||||
return bp.precision
|
||||
}
|
||||
|
||||
func (bp *batchpoints) Database() string {
|
||||
return bp.database
|
||||
}
|
||||
|
||||
func (bp *batchpoints) WriteConsistency() string {
|
||||
return bp.writeConsistency
|
||||
}
|
||||
|
||||
func (bp *batchpoints) RetentionPolicy() string {
|
||||
return bp.retentionPolicy
|
||||
}
|
||||
|
||||
func (bp *batchpoints) SetPrecision(p string) error {
|
||||
if _, err := time.ParseDuration("1" + p); err != nil {
|
||||
return err
|
||||
}
|
||||
bp.precision = p
|
||||
return nil
|
||||
}
|
||||
|
||||
func (bp *batchpoints) SetDatabase(db string) {
|
||||
bp.database = db
|
||||
}
|
||||
|
||||
func (bp *batchpoints) SetWriteConsistency(wc string) {
|
||||
bp.writeConsistency = wc
|
||||
}
|
||||
|
||||
func (bp *batchpoints) SetRetentionPolicy(rp string) {
|
||||
bp.retentionPolicy = rp
|
||||
}
|
||||
|
||||
// Point represents a single data point.
|
||||
type Point struct {
|
||||
pt models.Point
|
||||
}
|
||||
|
||||
// NewPoint returns a point with the given timestamp. If a timestamp is not
|
||||
// given, then data is sent to the database without a timestamp, in which case
|
||||
// the server will assign local time upon reception. NOTE: it is recommended to
|
||||
// send data with a timestamp.
|
||||
func NewPoint(
|
||||
name string,
|
||||
tags map[string]string,
|
||||
fields map[string]interface{},
|
||||
t ...time.Time,
|
||||
) (*Point, error) {
|
||||
var T time.Time
|
||||
if len(t) > 0 {
|
||||
T = t[0]
|
||||
}
|
||||
|
||||
pt, err := models.NewPoint(name, models.NewTags(tags), fields, T)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Point{
|
||||
pt: pt,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// String returns a line-protocol string of the Point.
|
||||
func (p *Point) String() string {
|
||||
return p.pt.String()
|
||||
}
|
||||
|
||||
// PrecisionString returns a line-protocol string of the Point,
|
||||
// with the timestamp formatted for the given precision.
|
||||
func (p *Point) PrecisionString(precision string) string {
|
||||
return p.pt.PrecisionString(precision)
|
||||
}
|
||||
|
||||
// Name returns the measurement name of the point.
|
||||
func (p *Point) Name() string {
|
||||
return string(p.pt.Name())
|
||||
}
|
||||
|
||||
// Tags returns the tags associated with the point.
|
||||
func (p *Point) Tags() map[string]string {
|
||||
return p.pt.Tags().Map()
|
||||
}
|
||||
|
||||
// Time return the timestamp for the point.
|
||||
func (p *Point) Time() time.Time {
|
||||
return p.pt.Time()
|
||||
}
|
||||
|
||||
// UnixNano returns timestamp of the point in nanoseconds since Unix epoch.
|
||||
func (p *Point) UnixNano() int64 {
|
||||
return p.pt.UnixNano()
|
||||
}
|
||||
|
||||
// Fields returns the fields for the point.
|
||||
func (p *Point) Fields() (map[string]interface{}, error) {
|
||||
return p.pt.Fields()
|
||||
}
|
||||
|
||||
// NewPointFrom returns a point from the provided models.Point.
|
||||
func NewPointFrom(pt models.Point) *Point {
|
||||
return &Point{pt: pt}
|
||||
}
|
||||
|
||||
func (c *client) Write(bp BatchPoints) error {
|
||||
return c.WriteCtx(context.Background(), bp)
|
||||
}
|
||||
|
||||
func (c *client) WriteCtx(ctx context.Context, bp BatchPoints) error {
|
||||
var b bytes.Buffer
|
||||
|
||||
for _, p := range bp.Points() {
|
||||
if p == nil {
|
||||
continue
|
||||
}
|
||||
if _, err := b.WriteString(p.pt.PrecisionString(bp.Precision())); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := b.WriteByte('\n'); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return c.WriteRawCtx(ctx, bp, &b)
|
||||
}
|
||||
|
||||
// WriteRawCtx uses reqBody instead of parsing bp.Points. Metadata still comes from bp.
|
||||
func (c *client) WriteRawCtx(ctx context.Context, bp BatchPoints, reqBody io.Reader) error {
|
||||
u := c.url
|
||||
u.Path = path.Join(u.Path, "write")
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, "POST", u.String(), reqBody)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.Header.Set("Content-Type", "")
|
||||
req.Header.Set("User-Agent", c.useragent)
|
||||
if c.username != "" {
|
||||
req.SetBasicAuth(c.username, c.password)
|
||||
}
|
||||
|
||||
params := req.URL.Query()
|
||||
params.Set("db", bp.Database())
|
||||
params.Set("rp", bp.RetentionPolicy())
|
||||
params.Set("precision", bp.Precision())
|
||||
params.Set("consistency", bp.WriteConsistency())
|
||||
req.URL.RawQuery = params.Encode()
|
||||
|
||||
resp, err := c.httpClient.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusNoContent && resp.StatusCode != http.StatusOK {
|
||||
var err = errors.New(string(body))
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Query defines a query to send to the server.
|
||||
type Query struct {
|
||||
Command string
|
||||
Database string
|
||||
RetentionPolicy string
|
||||
Precision string
|
||||
Chunked bool
|
||||
ChunkSize int
|
||||
Parameters map[string]interface{}
|
||||
}
|
||||
|
||||
// Params is a type alias to the query parameters.
|
||||
type Params map[string]interface{}
|
||||
|
||||
// NewQuery returns a query object.
|
||||
// The database and precision arguments can be empty strings if they are not needed for the query.
|
||||
func NewQuery(command, database, precision string) Query {
|
||||
return Query{
|
||||
Command: command,
|
||||
Database: database,
|
||||
Precision: precision,
|
||||
Parameters: make(map[string]interface{}),
|
||||
}
|
||||
}
|
||||
|
||||
// NewQueryWithRP returns a query object.
|
||||
// The database, retention policy, and precision arguments can be empty strings if they are not needed
|
||||
// for the query. Setting the retention policy only works on InfluxDB versions 1.6 or greater.
|
||||
func NewQueryWithRP(command, database, retentionPolicy, precision string) Query {
|
||||
return Query{
|
||||
Command: command,
|
||||
Database: database,
|
||||
RetentionPolicy: retentionPolicy,
|
||||
Precision: precision,
|
||||
Parameters: make(map[string]interface{}),
|
||||
}
|
||||
}
|
||||
|
||||
// NewQueryWithParameters returns a query object.
|
||||
// The database and precision arguments can be empty strings if they are not needed for the query.
|
||||
// parameters is a map of the parameter names used in the command to their values.
|
||||
func NewQueryWithParameters(command, database, precision string, parameters map[string]interface{}) Query {
|
||||
return Query{
|
||||
Command: command,
|
||||
Database: database,
|
||||
Precision: precision,
|
||||
Parameters: parameters,
|
||||
}
|
||||
}
|
||||
|
||||
// Response represents a list of statement results.
|
||||
type Response struct {
|
||||
Results []Result
|
||||
Err string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// Error returns the first error from any statement.
|
||||
// It returns nil if no errors occurred on any statements.
|
||||
func (r *Response) Error() error {
|
||||
if r.Err != "" {
|
||||
return errors.New(r.Err)
|
||||
}
|
||||
for _, result := range r.Results {
|
||||
if result.Err != "" {
|
||||
return errors.New(result.Err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Message represents a user message.
|
||||
type Message struct {
|
||||
Level string
|
||||
Text string
|
||||
}
|
||||
|
||||
// Result represents a resultset returned from a single statement.
|
||||
type Result struct {
|
||||
Series []models.Row
|
||||
Messages []*Message
|
||||
Err string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// Query sends a command to the server and returns the Response.
|
||||
func (c *client) Query(q Query) (*Response, error) {
|
||||
return c.QueryCtx(context.Background(), q)
|
||||
}
|
||||
|
||||
// QueryCtx sends a command to the server and returns the Response.
|
||||
func (c *client) QueryCtx(ctx context.Context, q Query) (*Response, error) {
|
||||
req, err := c.createDefaultRequest(ctx, q)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
params := req.URL.Query()
|
||||
if q.Chunked {
|
||||
params.Set("chunked", "true")
|
||||
if q.ChunkSize > 0 {
|
||||
params.Set("chunk_size", strconv.Itoa(q.ChunkSize))
|
||||
}
|
||||
req.URL.RawQuery = params.Encode()
|
||||
}
|
||||
resp, err := c.httpClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if err := checkResponse(resp); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var response Response
|
||||
if q.Chunked {
|
||||
cr := NewChunkedResponse(resp.Body)
|
||||
for {
|
||||
r, err := cr.NextResponse()
|
||||
if err != nil {
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
// If we got an error while decoding the response, send that back.
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if r == nil {
|
||||
break
|
||||
}
|
||||
|
||||
response.Results = append(response.Results, r.Results...)
|
||||
if r.Err != "" {
|
||||
response.Err = r.Err
|
||||
break
|
||||
}
|
||||
}
|
||||
} else {
|
||||
dec := json.NewDecoder(resp.Body)
|
||||
dec.UseNumber()
|
||||
decErr := dec.Decode(&response)
|
||||
|
||||
// ignore this error if we got an invalid status code
|
||||
if decErr != nil && decErr.Error() == "EOF" && resp.StatusCode != http.StatusOK {
|
||||
decErr = nil
|
||||
}
|
||||
// If we got a valid decode error, send that back
|
||||
if decErr != nil {
|
||||
return nil, fmt.Errorf("unable to decode json: received status code %d err: %s", resp.StatusCode, decErr)
|
||||
}
|
||||
}
|
||||
|
||||
// If we don't have an error in our json response, and didn't get statusOK
|
||||
// then send back an error
|
||||
if resp.StatusCode != http.StatusOK && response.Error() == nil {
|
||||
return &response, fmt.Errorf("received status code %d from server", resp.StatusCode)
|
||||
}
|
||||
return &response, nil
|
||||
}
|
||||
|
||||
// QueryAsChunk sends a command to the server and returns the Response.
|
||||
func (c *client) QueryAsChunk(q Query) (*ChunkedResponse, error) {
|
||||
req, err := c.createDefaultRequest(context.Background(), q)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
params := req.URL.Query()
|
||||
params.Set("chunked", "true")
|
||||
if q.ChunkSize > 0 {
|
||||
params.Set("chunk_size", strconv.Itoa(q.ChunkSize))
|
||||
}
|
||||
req.URL.RawQuery = params.Encode()
|
||||
resp, err := c.httpClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := checkResponse(resp); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return NewChunkedResponse(resp.Body), nil
|
||||
}
|
||||
|
||||
func checkResponse(resp *http.Response) error {
|
||||
// If we lack a X-Influxdb-Version header, then we didn't get a response from influxdb
|
||||
// but instead some other service. If the error code is also a 500+ code, then some
|
||||
// downstream loadbalancer/proxy/etc had an issue and we should report that.
|
||||
if resp.Header.Get("X-Influxdb-Version") == "" && resp.StatusCode >= http.StatusInternalServerError {
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil || len(body) == 0 {
|
||||
return fmt.Errorf("received status code %d from downstream server", resp.StatusCode)
|
||||
}
|
||||
|
||||
return fmt.Errorf("received status code %d from downstream server, with response body: %q", resp.StatusCode, body)
|
||||
}
|
||||
|
||||
// If we get an unexpected content type, then it is also not from influx direct and therefore
|
||||
// we want to know what we received and what status code was returned for debugging purposes.
|
||||
if cType, _, _ := mime.ParseMediaType(resp.Header.Get("Content-Type")); cType != "application/json" {
|
||||
// Read up to 1kb of the body to help identify downstream errors and limit the impact of things
|
||||
// like downstream serving a large file
|
||||
body, err := io.ReadAll(io.LimitReader(resp.Body, 1024))
|
||||
if err != nil || len(body) == 0 {
|
||||
return fmt.Errorf("expected json response, got empty body, with status: %v", resp.StatusCode)
|
||||
}
|
||||
|
||||
return fmt.Errorf("expected json response, got %q, with status: %v and response body: %q", cType, resp.StatusCode, body)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *client) createDefaultRequest(ctx context.Context, q Query) (*http.Request, error) {
|
||||
u := c.url
|
||||
u.Path = path.Join(u.Path, "query")
|
||||
|
||||
jsonParameters, err := json.Marshal(q.Parameters)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
req, err := http.NewRequest("POST", u.String(), nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if ctx != nil {
|
||||
req = req.WithContext(ctx)
|
||||
}
|
||||
|
||||
req.Header.Set("Content-Type", "")
|
||||
req.Header.Set("User-Agent", c.useragent)
|
||||
|
||||
if c.username != "" {
|
||||
req.SetBasicAuth(c.username, c.password)
|
||||
}
|
||||
|
||||
params := req.URL.Query()
|
||||
params.Set("q", q.Command)
|
||||
params.Set("db", q.Database)
|
||||
if q.RetentionPolicy != "" {
|
||||
params.Set("rp", q.RetentionPolicy)
|
||||
}
|
||||
params.Set("params", string(jsonParameters))
|
||||
|
||||
if q.Precision != "" {
|
||||
params.Set("epoch", q.Precision)
|
||||
}
|
||||
req.URL.RawQuery = params.Encode()
|
||||
|
||||
return req, nil
|
||||
|
||||
}
|
||||
|
||||
// duplexReader reads responses and writes it to another writer while
|
||||
// satisfying the reader interface.
|
||||
type duplexReader struct {
|
||||
r io.ReadCloser
|
||||
w io.Writer
|
||||
}
|
||||
|
||||
func (r *duplexReader) Read(p []byte) (n int, err error) {
|
||||
n, err = r.r.Read(p)
|
||||
if err == nil {
|
||||
r.w.Write(p[:n])
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
|
||||
// Close closes the response.
|
||||
func (r *duplexReader) Close() error {
|
||||
return r.r.Close()
|
||||
}
|
||||
|
||||
// ChunkedResponse represents a response from the server that
|
||||
// uses chunking to stream the output.
|
||||
type ChunkedResponse struct {
|
||||
dec *json.Decoder
|
||||
duplex *duplexReader
|
||||
buf bytes.Buffer
|
||||
}
|
||||
|
||||
// NewChunkedResponse reads a stream and produces responses from the stream.
|
||||
func NewChunkedResponse(r io.Reader) *ChunkedResponse {
|
||||
rc, ok := r.(io.ReadCloser)
|
||||
if !ok {
|
||||
rc = io.NopCloser(r)
|
||||
}
|
||||
resp := &ChunkedResponse{}
|
||||
resp.duplex = &duplexReader{r: rc, w: &resp.buf}
|
||||
resp.dec = json.NewDecoder(resp.duplex)
|
||||
resp.dec.UseNumber()
|
||||
return resp
|
||||
}
|
||||
|
||||
// NextResponse reads the next line of the stream and returns a response.
|
||||
func (r *ChunkedResponse) NextResponse() (*Response, error) {
|
||||
var response Response
|
||||
if err := r.dec.Decode(&response); err != nil {
|
||||
if err == io.EOF {
|
||||
return nil, err
|
||||
}
|
||||
// A decoding error happened. This probably means the server crashed
|
||||
// and sent a last-ditch error message to us. Ensure we have read the
|
||||
// entirety of the connection to get any remaining error text.
|
||||
io.Copy(io.Discard, r.duplex)
|
||||
return nil, errors.New(strings.TrimSpace(r.buf.String()))
|
||||
}
|
||||
|
||||
r.buf.Reset()
|
||||
return &response, nil
|
||||
}
|
||||
|
||||
// Close closes the response.
|
||||
func (r *ChunkedResponse) Close() error {
|
||||
return r.duplex.Close()
|
||||
}
|
||||
-73
@@ -1,73 +0,0 @@
|
||||
package client
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"time"
|
||||
)
|
||||
|
||||
type (
|
||||
// Identifier is an identifier value.
|
||||
Identifier string
|
||||
|
||||
// StringValue is a string literal.
|
||||
StringValue string
|
||||
|
||||
// RegexValue is a regexp literal.
|
||||
RegexValue string
|
||||
|
||||
// NumberValue is a number literal.
|
||||
NumberValue float64
|
||||
|
||||
// IntegerValue is an integer literal.
|
||||
IntegerValue int64
|
||||
|
||||
// BooleanValue is a boolean literal.
|
||||
BooleanValue bool
|
||||
|
||||
// TimeValue is a time literal.
|
||||
TimeValue time.Time
|
||||
|
||||
// DurationValue is a duration literal.
|
||||
DurationValue time.Duration
|
||||
)
|
||||
|
||||
func (v Identifier) MarshalJSON() ([]byte, error) {
|
||||
m := map[string]string{"identifier": string(v)}
|
||||
return json.Marshal(m)
|
||||
}
|
||||
|
||||
func (v StringValue) MarshalJSON() ([]byte, error) {
|
||||
m := map[string]string{"string": string(v)}
|
||||
return json.Marshal(m)
|
||||
}
|
||||
|
||||
func (v RegexValue) MarshalJSON() ([]byte, error) {
|
||||
m := map[string]string{"regex": string(v)}
|
||||
return json.Marshal(m)
|
||||
}
|
||||
|
||||
func (v NumberValue) MarshalJSON() ([]byte, error) {
|
||||
m := map[string]float64{"number": float64(v)}
|
||||
return json.Marshal(m)
|
||||
}
|
||||
|
||||
func (v IntegerValue) MarshalJSON() ([]byte, error) {
|
||||
m := map[string]int64{"integer": int64(v)}
|
||||
return json.Marshal(m)
|
||||
}
|
||||
|
||||
func (v BooleanValue) MarshalJSON() ([]byte, error) {
|
||||
m := map[string]bool{"boolean": bool(v)}
|
||||
return json.Marshal(m)
|
||||
}
|
||||
|
||||
func (v TimeValue) MarshalJSON() ([]byte, error) {
|
||||
t := time.Time(v)
|
||||
m := map[string]string{"string": t.Format(time.RFC3339Nano)}
|
||||
return json.Marshal(m)
|
||||
}
|
||||
|
||||
func (v DurationValue) MarshalJSON() ([]byte, error) {
|
||||
m := map[string]int64{"duration": int64(v)}
|
||||
return json.Marshal(m)
|
||||
}
|
||||
-125
@@ -1,125 +0,0 @@
|
||||
package client
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
// UDPPayloadSize is a reasonable default payload size for UDP packets that
|
||||
// could be travelling over the internet.
|
||||
UDPPayloadSize = 512
|
||||
)
|
||||
|
||||
// UDPConfig is the config data needed to create a UDP Client.
|
||||
type UDPConfig struct {
|
||||
// Addr should be of the form "host:port"
|
||||
// or "[ipv6-host%zone]:port".
|
||||
Addr string
|
||||
|
||||
// PayloadSize is the maximum size of a UDP client message, optional
|
||||
// Tune this based on your network. Defaults to UDPPayloadSize.
|
||||
PayloadSize int
|
||||
}
|
||||
|
||||
// NewUDPClient returns a client interface for writing to an InfluxDB UDP
|
||||
// service from the given config.
|
||||
func NewUDPClient(conf UDPConfig) (Client, error) {
|
||||
var udpAddr *net.UDPAddr
|
||||
udpAddr, err := net.ResolveUDPAddr("udp", conf.Addr)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
conn, err := net.DialUDP("udp", nil, udpAddr)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
payloadSize := conf.PayloadSize
|
||||
if payloadSize == 0 {
|
||||
payloadSize = UDPPayloadSize
|
||||
}
|
||||
|
||||
return &udpclient{
|
||||
conn: conn,
|
||||
payloadSize: payloadSize,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Close releases the udpclient's resources.
|
||||
func (uc *udpclient) Close() error {
|
||||
return uc.conn.Close()
|
||||
}
|
||||
|
||||
type udpclient struct {
|
||||
conn io.WriteCloser
|
||||
payloadSize int
|
||||
}
|
||||
|
||||
func (uc *udpclient) Write(bp BatchPoints) error {
|
||||
return uc.WriteCtx(context.Background(), bp)
|
||||
}
|
||||
|
||||
func (uc *udpclient) WriteCtx(ctx context.Context, bp BatchPoints) error {
|
||||
var b = make([]byte, 0, uc.payloadSize) // initial buffer size, it will grow as needed
|
||||
var d, _ = time.ParseDuration("1" + bp.Precision())
|
||||
|
||||
var delayedError error
|
||||
|
||||
var checkBuffer = func(n int) {
|
||||
if len(b) > 0 && len(b)+n > uc.payloadSize {
|
||||
if _, err := uc.conn.Write(b); err != nil {
|
||||
delayedError = err
|
||||
}
|
||||
b = b[:0]
|
||||
}
|
||||
}
|
||||
|
||||
for _, p := range bp.Points() {
|
||||
p.pt.Round(d)
|
||||
pointSize := p.pt.StringSize() + 1 // include newline in size
|
||||
//point := p.pt.RoundedString(d) + "\n"
|
||||
|
||||
checkBuffer(pointSize)
|
||||
|
||||
if p.Time().IsZero() || pointSize <= uc.payloadSize {
|
||||
b = p.pt.AppendString(b)
|
||||
b = append(b, '\n')
|
||||
continue
|
||||
}
|
||||
|
||||
points := p.pt.Split(uc.payloadSize - 1) // account for newline character
|
||||
for _, sp := range points {
|
||||
checkBuffer(sp.StringSize() + 1)
|
||||
b = sp.AppendString(b)
|
||||
b = append(b, '\n')
|
||||
}
|
||||
}
|
||||
|
||||
if len(b) > 0 {
|
||||
if _, err := uc.conn.Write(b); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return delayedError
|
||||
}
|
||||
|
||||
func (uc *udpclient) Query(q Query) (*Response, error) {
|
||||
return nil, fmt.Errorf("querying via UDP is not supported")
|
||||
}
|
||||
|
||||
func (uc *udpclient) QueryCtx(ctx context.Context, q Query) (*Response, error) {
|
||||
return nil, fmt.Errorf("querying via UDP is not supported")
|
||||
}
|
||||
|
||||
func (uc *udpclient) QueryAsChunk(q Query) (*ChunkedResponse, error) {
|
||||
return nil, fmt.Errorf("querying via UDP is not supported")
|
||||
}
|
||||
|
||||
func (uc *udpclient) Ping(timeout time.Duration) (time.Duration, string, error) {
|
||||
return 0, "", nil
|
||||
}
|
||||
-48
@@ -1,48 +0,0 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ConsistencyLevel represent a required replication criteria before a write can
|
||||
// be returned as successful.
|
||||
//
|
||||
// The consistency level is handled in open-source InfluxDB but only applicable to clusters.
|
||||
type ConsistencyLevel int
|
||||
|
||||
const (
|
||||
// ConsistencyLevelAny allows for hinted handoff, potentially no write happened yet.
|
||||
ConsistencyLevelAny ConsistencyLevel = iota
|
||||
|
||||
// ConsistencyLevelOne requires at least one data node acknowledged a write.
|
||||
ConsistencyLevelOne
|
||||
|
||||
// ConsistencyLevelQuorum requires a quorum of data nodes to acknowledge a write.
|
||||
ConsistencyLevelQuorum
|
||||
|
||||
// ConsistencyLevelAll requires all data nodes to acknowledge a write.
|
||||
ConsistencyLevelAll
|
||||
)
|
||||
|
||||
var (
|
||||
// ErrInvalidConsistencyLevel is returned when parsing the string version
|
||||
// of a consistency level.
|
||||
ErrInvalidConsistencyLevel = errors.New("invalid consistency level")
|
||||
)
|
||||
|
||||
// ParseConsistencyLevel converts a consistency level string to the corresponding ConsistencyLevel const.
|
||||
func ParseConsistencyLevel(level string) (ConsistencyLevel, error) {
|
||||
switch strings.ToLower(level) {
|
||||
case "any":
|
||||
return ConsistencyLevelAny, nil
|
||||
case "one":
|
||||
return ConsistencyLevelOne, nil
|
||||
case "quorum":
|
||||
return ConsistencyLevelQuorum, nil
|
||||
case "all":
|
||||
return ConsistencyLevelAll, nil
|
||||
default:
|
||||
return 0, ErrInvalidConsistencyLevel
|
||||
}
|
||||
}
|
||||
-28
@@ -1,28 +0,0 @@
|
||||
// Code generated by "stringer -type=FieldType"; DO NOT EDIT.
|
||||
|
||||
package models
|
||||
|
||||
import "strconv"
|
||||
|
||||
func _() {
|
||||
// An "invalid array index" compiler error signifies that the constant values have changed.
|
||||
// Re-run the stringer command to generate them again.
|
||||
var x [1]struct{}
|
||||
_ = x[Integer-0]
|
||||
_ = x[Float-1]
|
||||
_ = x[Boolean-2]
|
||||
_ = x[String-3]
|
||||
_ = x[Empty-4]
|
||||
_ = x[Unsigned-5]
|
||||
}
|
||||
|
||||
const _FieldType_name = "IntegerFloatBooleanStringEmptyUnsigned"
|
||||
|
||||
var _FieldType_index = [...]uint8{0, 7, 12, 19, 25, 30, 38}
|
||||
|
||||
func (i FieldType) String() string {
|
||||
if i < 0 || i >= FieldType(len(_FieldType_index)-1) {
|
||||
return "FieldType(" + strconv.FormatInt(int64(i), 10) + ")"
|
||||
}
|
||||
return _FieldType_name[_FieldType_index[i]:_FieldType_index[i+1]]
|
||||
}
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
package models
|
||||
|
||||
//go:generate stringer -type=FieldType
|
||||
-32
@@ -1,32 +0,0 @@
|
||||
package models // import "github.com/influxdata/influxdb/models"
|
||||
|
||||
// from stdlib hash/fnv/fnv.go
|
||||
const (
|
||||
prime64 = 1099511628211
|
||||
offset64 = 14695981039346656037
|
||||
)
|
||||
|
||||
// InlineFNV64a is an alloc-free port of the standard library's fnv64a.
|
||||
// See https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function.
|
||||
type InlineFNV64a uint64
|
||||
|
||||
// NewInlineFNV64a returns a new instance of InlineFNV64a.
|
||||
func NewInlineFNV64a() InlineFNV64a {
|
||||
return offset64
|
||||
}
|
||||
|
||||
// Write adds data to the running hash.
|
||||
func (s *InlineFNV64a) Write(data []byte) (int, error) {
|
||||
hash := uint64(*s)
|
||||
for _, c := range data {
|
||||
hash ^= uint64(c)
|
||||
hash *= prime64
|
||||
}
|
||||
*s = InlineFNV64a(hash)
|
||||
return len(data), nil
|
||||
}
|
||||
|
||||
// Sum64 returns the uint64 of the current resulting hash.
|
||||
func (s *InlineFNV64a) Sum64() uint64 {
|
||||
return uint64(*s)
|
||||
}
|
||||
-34
@@ -1,34 +0,0 @@
|
||||
package models // import "github.com/influxdata/influxdb/models"
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"unsafe"
|
||||
)
|
||||
|
||||
// parseIntBytes is a zero-alloc wrapper around strconv.ParseInt.
|
||||
func parseIntBytes(b []byte, base int, bitSize int) (i int64, err error) {
|
||||
s := unsafeBytesToString(b)
|
||||
return strconv.ParseInt(s, base, bitSize)
|
||||
}
|
||||
|
||||
// parseUintBytes is a zero-alloc wrapper around strconv.ParseUint.
|
||||
func parseUintBytes(b []byte, base int, bitSize int) (i uint64, err error) {
|
||||
s := unsafeBytesToString(b)
|
||||
return strconv.ParseUint(s, base, bitSize)
|
||||
}
|
||||
|
||||
// parseFloatBytes is a zero-alloc wrapper around strconv.ParseFloat.
|
||||
func parseFloatBytes(b []byte, bitSize int) (float64, error) {
|
||||
s := unsafeBytesToString(b)
|
||||
return strconv.ParseFloat(s, bitSize)
|
||||
}
|
||||
|
||||
// parseBoolBytes is a zero-alloc wrapper around strconv.ParseBool.
|
||||
func parseBoolBytes(b []byte) (bool, error) {
|
||||
return strconv.ParseBool(unsafeBytesToString(b))
|
||||
}
|
||||
|
||||
// unsafeBytesToString converts a []byte to a string without a heap allocation.
|
||||
func unsafeBytesToString(in []byte) string {
|
||||
return *(*string)(unsafe.Pointer(&in))
|
||||
}
|
||||
-2596
File diff suppressed because it is too large
Load Diff
-62
@@ -1,62 +0,0 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"sort"
|
||||
)
|
||||
|
||||
// Row represents a single row returned from the execution of a statement.
|
||||
type Row struct {
|
||||
Name string `json:"name,omitempty"`
|
||||
Tags map[string]string `json:"tags,omitempty"`
|
||||
Columns []string `json:"columns,omitempty"`
|
||||
Values [][]interface{} `json:"values,omitempty"`
|
||||
Partial bool `json:"partial,omitempty"`
|
||||
}
|
||||
|
||||
// SameSeries returns true if r contains values for the same series as o.
|
||||
func (r *Row) SameSeries(o *Row) bool {
|
||||
return r.tagsHash() == o.tagsHash() && r.Name == o.Name
|
||||
}
|
||||
|
||||
// tagsHash returns a hash of tag key/value pairs.
|
||||
func (r *Row) tagsHash() uint64 {
|
||||
h := NewInlineFNV64a()
|
||||
keys := r.tagsKeys()
|
||||
for _, k := range keys {
|
||||
h.Write([]byte(k))
|
||||
h.Write([]byte(r.Tags[k]))
|
||||
}
|
||||
return h.Sum64()
|
||||
}
|
||||
|
||||
// tagKeys returns a sorted list of tag keys.
|
||||
func (r *Row) tagsKeys() []string {
|
||||
a := make([]string, 0, len(r.Tags))
|
||||
for k := range r.Tags {
|
||||
a = append(a, k)
|
||||
}
|
||||
sort.Strings(a)
|
||||
return a
|
||||
}
|
||||
|
||||
// Rows represents a collection of rows. Rows implements sort.Interface.
|
||||
type Rows []*Row
|
||||
|
||||
// Len implements sort.Interface.
|
||||
func (p Rows) Len() int { return len(p) }
|
||||
|
||||
// Less implements sort.Interface.
|
||||
func (p Rows) Less(i, j int) bool {
|
||||
// Sort by name first.
|
||||
if p[i].Name != p[j].Name {
|
||||
return p[i].Name < p[j].Name
|
||||
}
|
||||
|
||||
// Sort by tag set hash. Tags don't have a meaningful sort order so we
|
||||
// just compute a hash and sort by that instead. This allows the tests
|
||||
// to receive rows in a predictable order every time.
|
||||
return p[i].tagsHash() < p[j].tagsHash()
|
||||
}
|
||||
|
||||
// Swap implements sort.Interface.
|
||||
func (p Rows) Swap(i, j int) { p[i], p[j] = p[j], p[i] }
|
||||
-42
@@ -1,42 +0,0 @@
|
||||
package models
|
||||
|
||||
// Statistic is the representation of a statistic used by the monitoring service.
|
||||
type Statistic struct {
|
||||
Name string `json:"name"`
|
||||
Tags map[string]string `json:"tags"`
|
||||
Values map[string]interface{} `json:"values"`
|
||||
}
|
||||
|
||||
// NewStatistic returns an initialized Statistic.
|
||||
func NewStatistic(name string) Statistic {
|
||||
return Statistic{
|
||||
Name: name,
|
||||
Tags: make(map[string]string),
|
||||
Values: make(map[string]interface{}),
|
||||
}
|
||||
}
|
||||
|
||||
// StatisticTags is a map that can be merged with others without causing
|
||||
// mutations to either map.
|
||||
type StatisticTags map[string]string
|
||||
|
||||
// Merge creates a new map containing the merged contents of tags and t.
|
||||
// If both tags and the receiver map contain the same key, the value in tags
|
||||
// is used in the resulting map.
|
||||
//
|
||||
// Merge always returns a usable map.
|
||||
func (t StatisticTags) Merge(tags map[string]string) map[string]string {
|
||||
// Add everything in tags to the result.
|
||||
out := make(map[string]string, len(tags))
|
||||
for k, v := range tags {
|
||||
out[k] = v
|
||||
}
|
||||
|
||||
// Only add values from t that don't appear in tags.
|
||||
for k, v := range t {
|
||||
if _, ok := tags[k]; !ok {
|
||||
out[k] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
-156
@@ -1,156 +0,0 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// TagKeysSet provides set operations for combining Tags.
|
||||
type TagKeysSet struct {
|
||||
i int
|
||||
keys [2][][]byte
|
||||
tmp [][]byte
|
||||
}
|
||||
|
||||
// Clear removes all the elements of TagKeysSet and ensures all internal
|
||||
// buffers are reset.
|
||||
func (set *TagKeysSet) Clear() {
|
||||
set.clear(set.keys[0])
|
||||
set.clear(set.keys[1])
|
||||
set.clear(set.tmp)
|
||||
set.i = 0
|
||||
set.keys[0] = set.keys[0][:0]
|
||||
}
|
||||
|
||||
func (set *TagKeysSet) clear(b [][]byte) {
|
||||
b = b[:cap(b)]
|
||||
for i := range b {
|
||||
b[i] = nil
|
||||
}
|
||||
}
|
||||
|
||||
// KeysBytes returns the merged keys in lexicographical order.
|
||||
// The slice is valid until the next call to UnionKeys, UnionBytes or Reset.
|
||||
func (set *TagKeysSet) KeysBytes() [][]byte {
|
||||
return set.keys[set.i&1]
|
||||
}
|
||||
|
||||
// Keys returns a copy of the merged keys in lexicographical order.
|
||||
func (set *TagKeysSet) Keys() []string {
|
||||
keys := set.KeysBytes()
|
||||
s := make([]string, 0, len(keys))
|
||||
for i := range keys {
|
||||
s = append(s, string(keys[i]))
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func (set *TagKeysSet) String() string {
|
||||
var s []string
|
||||
for _, k := range set.KeysBytes() {
|
||||
s = append(s, string(k))
|
||||
}
|
||||
return strings.Join(s, ",")
|
||||
}
|
||||
|
||||
// IsSupersetKeys returns true if the TagKeysSet is a superset of all the keys
|
||||
// contained in other.
|
||||
func (set *TagKeysSet) IsSupersetKeys(other Tags) bool {
|
||||
keys := set.keys[set.i&1]
|
||||
i, j := 0, 0
|
||||
for i < len(keys) && j < len(other) {
|
||||
if cmp := bytes.Compare(keys[i], other[j].Key); cmp > 0 {
|
||||
return false
|
||||
} else if cmp == 0 {
|
||||
j++
|
||||
}
|
||||
i++
|
||||
}
|
||||
|
||||
return j == len(other)
|
||||
}
|
||||
|
||||
// IsSupersetBytes returns true if the TagKeysSet is a superset of all the keys
|
||||
// in other.
|
||||
// Other must be lexicographically sorted or the results are undefined.
|
||||
func (set *TagKeysSet) IsSupersetBytes(other [][]byte) bool {
|
||||
keys := set.keys[set.i&1]
|
||||
i, j := 0, 0
|
||||
for i < len(keys) && j < len(other) {
|
||||
if cmp := bytes.Compare(keys[i], other[j]); cmp > 0 {
|
||||
return false
|
||||
} else if cmp == 0 {
|
||||
j++
|
||||
}
|
||||
i++
|
||||
}
|
||||
|
||||
return j == len(other)
|
||||
}
|
||||
|
||||
// UnionKeys updates the set so that it is the union of itself and all the
|
||||
// keys contained in other.
|
||||
func (set *TagKeysSet) UnionKeys(other Tags) {
|
||||
if set.IsSupersetKeys(other) {
|
||||
return
|
||||
}
|
||||
|
||||
if l := len(other); cap(set.tmp) < l {
|
||||
set.tmp = make([][]byte, l)
|
||||
} else {
|
||||
set.tmp = set.tmp[:l]
|
||||
}
|
||||
|
||||
for i := range other {
|
||||
set.tmp[i] = other[i].Key
|
||||
}
|
||||
|
||||
set.merge(set.tmp)
|
||||
}
|
||||
|
||||
// UnionBytes updates the set so that it is the union of itself and all the
|
||||
// keys contained in other.
|
||||
// Other must be lexicographically sorted or the results are undefined.
|
||||
func (set *TagKeysSet) UnionBytes(other [][]byte) {
|
||||
if set.IsSupersetBytes(other) {
|
||||
return
|
||||
}
|
||||
|
||||
set.merge(other)
|
||||
}
|
||||
|
||||
func (set *TagKeysSet) merge(in [][]byte) {
|
||||
keys := set.keys[set.i&1]
|
||||
l := len(keys) + len(in)
|
||||
set.i = (set.i + 1) & 1
|
||||
keya := set.keys[set.i&1]
|
||||
if cap(keya) < l {
|
||||
keya = make([][]byte, 0, l)
|
||||
} else {
|
||||
keya = keya[:0]
|
||||
}
|
||||
|
||||
i, j := 0, 0
|
||||
for i < len(keys) && j < len(in) {
|
||||
ki, kj := keys[i], in[j]
|
||||
if cmp := bytes.Compare(ki, kj); cmp < 0 {
|
||||
i++
|
||||
} else if cmp > 0 {
|
||||
ki = kj
|
||||
j++
|
||||
} else {
|
||||
i++
|
||||
j++
|
||||
}
|
||||
|
||||
keya = append(keya, ki)
|
||||
}
|
||||
|
||||
if i < len(keys) {
|
||||
keya = append(keya, keys[i:]...)
|
||||
} else if j < len(in) {
|
||||
keya = append(keya, in[j:]...)
|
||||
}
|
||||
|
||||
set.keys[set.i&1] = keya
|
||||
}
|
||||
-74
@@ -1,74 +0,0 @@
|
||||
package models
|
||||
|
||||
// Helper time methods since parsing time can easily overflow and we only support a
|
||||
// specific time range.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
// MinNanoTime is the minimum time that can be represented.
|
||||
//
|
||||
// 1677-09-21 00:12:43.145224194 +0000 UTC
|
||||
//
|
||||
// The two lowest minimum integers are used as sentinel values. The
|
||||
// minimum value needs to be used as a value lower than any other value for
|
||||
// comparisons and another separate value is needed to act as a sentinel
|
||||
// default value that is unusable by the user, but usable internally.
|
||||
// Because these two values need to be used for a special purpose, we do
|
||||
// not allow users to write points at these two times.
|
||||
MinNanoTime = int64(math.MinInt64) + 2
|
||||
|
||||
// MaxNanoTime is the maximum time that can be represented.
|
||||
//
|
||||
// 2262-04-11 23:47:16.854775806 +0000 UTC
|
||||
//
|
||||
// The highest time represented by a nanosecond needs to be used for an
|
||||
// exclusive range in the shard group, so the maximum time needs to be one
|
||||
// less than the possible maximum number of nanoseconds representable by an
|
||||
// int64 so that we don't lose a point at that one time.
|
||||
MaxNanoTime = int64(math.MaxInt64) - 1
|
||||
)
|
||||
|
||||
var (
|
||||
minNanoTime = time.Unix(0, MinNanoTime).UTC()
|
||||
maxNanoTime = time.Unix(0, MaxNanoTime).UTC()
|
||||
|
||||
// ErrTimeOutOfRange gets returned when time is out of the representable range using int64 nanoseconds since the epoch.
|
||||
ErrTimeOutOfRange = fmt.Errorf("time outside range %d - %d", MinNanoTime, MaxNanoTime)
|
||||
)
|
||||
|
||||
// SafeCalcTime safely calculates the time given. Will return error if the time is outside the
|
||||
// supported range.
|
||||
func SafeCalcTime(timestamp int64, precision string) (time.Time, error) {
|
||||
mult := GetPrecisionMultiplier(precision)
|
||||
if t, ok := safeSignedMult(timestamp, mult); ok {
|
||||
tme := time.Unix(0, t).UTC()
|
||||
return tme, CheckTime(tme)
|
||||
}
|
||||
|
||||
return time.Time{}, ErrTimeOutOfRange
|
||||
}
|
||||
|
||||
// CheckTime checks that a time is within the safe range.
|
||||
func CheckTime(t time.Time) error {
|
||||
if t.Before(minNanoTime) || t.After(maxNanoTime) {
|
||||
return ErrTimeOutOfRange
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Perform the multiplication and check to make sure it didn't overflow.
|
||||
func safeSignedMult(a, b int64) (int64, bool) {
|
||||
if a == 0 || b == 0 || a == 1 || b == 1 {
|
||||
return a * b, true
|
||||
}
|
||||
if a == MinNanoTime || b == MaxNanoTime {
|
||||
return 0, false
|
||||
}
|
||||
c := a * b
|
||||
return c, c/b == a
|
||||
}
|
||||
-7
@@ -1,7 +0,0 @@
|
||||
//go:build uint || uint64
|
||||
|
||||
package models
|
||||
|
||||
func init() {
|
||||
EnableUintSupport()
|
||||
}
|
||||
-115
@@ -1,115 +0,0 @@
|
||||
// Package escape contains utilities for escaping parts of InfluxQL
|
||||
// and InfluxDB line protocol.
|
||||
package escape // import "github.com/influxdata/influxdb/pkg/escape"
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Codes is a map of bytes to be escaped.
|
||||
var Codes = map[byte][]byte{
|
||||
',': []byte(`\,`),
|
||||
'"': []byte(`\"`),
|
||||
' ': []byte(`\ `),
|
||||
'=': []byte(`\=`),
|
||||
}
|
||||
|
||||
// Bytes escapes characters on the input slice, as defined by Codes.
|
||||
func Bytes(in []byte) []byte {
|
||||
for b, esc := range Codes {
|
||||
in = bytes.Replace(in, []byte{b}, esc, -1)
|
||||
}
|
||||
return in
|
||||
}
|
||||
|
||||
const escapeChars = `," =`
|
||||
|
||||
// IsEscaped returns whether b has any escaped characters,
|
||||
// i.e. whether b seems to have been processed by Bytes.
|
||||
func IsEscaped(b []byte) bool {
|
||||
for len(b) > 0 {
|
||||
i := bytes.IndexByte(b, '\\')
|
||||
if i < 0 {
|
||||
return false
|
||||
}
|
||||
|
||||
if i+1 < len(b) && strings.IndexByte(escapeChars, b[i+1]) >= 0 {
|
||||
return true
|
||||
}
|
||||
b = b[i+1:]
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// AppendUnescaped appends the unescaped version of src to dst
|
||||
// and returns the resulting slice.
|
||||
func AppendUnescaped(dst, src []byte) []byte {
|
||||
var pos int
|
||||
for len(src) > 0 {
|
||||
next := bytes.IndexByte(src[pos:], '\\')
|
||||
if next < 0 || pos+next+1 >= len(src) {
|
||||
return append(dst, src...)
|
||||
}
|
||||
|
||||
if pos+next+1 < len(src) && strings.IndexByte(escapeChars, src[pos+next+1]) >= 0 {
|
||||
if pos+next > 0 {
|
||||
dst = append(dst, src[:pos+next]...)
|
||||
}
|
||||
src = src[pos+next+1:]
|
||||
pos = 0
|
||||
} else {
|
||||
pos += next + 1
|
||||
}
|
||||
}
|
||||
|
||||
return dst
|
||||
}
|
||||
|
||||
// Unescape returns a new slice containing the unescaped version of in.
|
||||
func Unescape(in []byte) []byte {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
if bytes.IndexByte(in, '\\') == -1 {
|
||||
return in
|
||||
}
|
||||
|
||||
i := 0
|
||||
inLen := len(in)
|
||||
|
||||
// The output size will be no more than inLen. Preallocating the
|
||||
// capacity of the output is faster and uses less memory than
|
||||
// letting append() do its own (over)allocation.
|
||||
out := make([]byte, 0, inLen)
|
||||
|
||||
for {
|
||||
if i >= inLen {
|
||||
break
|
||||
}
|
||||
if in[i] == '\\' && i+1 < inLen {
|
||||
switch in[i+1] {
|
||||
case ',':
|
||||
out = append(out, ',')
|
||||
i += 2
|
||||
continue
|
||||
case '"':
|
||||
out = append(out, '"')
|
||||
i += 2
|
||||
continue
|
||||
case ' ':
|
||||
out = append(out, ' ')
|
||||
i += 2
|
||||
continue
|
||||
case '=':
|
||||
out = append(out, '=')
|
||||
i += 2
|
||||
continue
|
||||
}
|
||||
}
|
||||
out = append(out, in[i])
|
||||
i += 1
|
||||
}
|
||||
return out
|
||||
}
|
||||
-21
@@ -1,21 +0,0 @@
|
||||
package escape
|
||||
|
||||
import "strings"
|
||||
|
||||
var (
|
||||
escaper = strings.NewReplacer(`,`, `\,`, `"`, `\"`, ` `, `\ `, `=`, `\=`)
|
||||
unescaper = strings.NewReplacer(`\,`, `,`, `\"`, `"`, `\ `, ` `, `\=`, `=`)
|
||||
)
|
||||
|
||||
// UnescapeString returns unescaped version of in.
|
||||
func UnescapeString(in string) string {
|
||||
if strings.IndexByte(in, '\\') == -1 {
|
||||
return in
|
||||
}
|
||||
return unescaper.Replace(in)
|
||||
}
|
||||
|
||||
// String returns the escaped version of in.
|
||||
func String(in string) string {
|
||||
return escaper.Replace(in)
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
# Test binary, build with `go test -c`
|
||||
*.test
|
||||
|
||||
# Output of the go coverage tool, specifically when used with LiteIDE
|
||||
*.out
|
||||
Generated
Vendored
+22
@@ -0,0 +1,22 @@
|
||||
# line-protocol
|
||||
|
||||
This is an encoder for the influx [line protocol.](https://docs.influxdata.com/influxdb/latest/write_protocols/line_protocol_reference/)
|
||||
|
||||
It has an interface similar to the standard library's `json.Encoder`.
|
||||
|
||||
|
||||
### some caveats.
|
||||
- It is not concurrency-safe. If you want to make multiple calls to `Encoder.Encode` concurrently you have to manage the concurrency yourself.
|
||||
- It can only encode values that are uint64, int64, int, float32, float64, string, or bool.
|
||||
- Ints are converted to int64, float32's to float64.
|
||||
- If UintSupport is not set, uint64s are converted to int64's and if they are larger than the max int64, they get truncated to the max int64 instead of overflowing.
|
||||
|
||||
|
||||
### Example:
|
||||
```go
|
||||
buf := &bytes.Buffer{}
|
||||
serializer := protocol.NewEncoder(buf)
|
||||
serializer.SetMaxLineBytes(1024)
|
||||
serializer.SetFieldTypeSupport(UintSupport)
|
||||
serializer.Encode(e) // where e is something that implements the protocol.Metric interface
|
||||
```
|
||||
+299
@@ -0,0 +1,299 @@
|
||||
package protocol
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"math"
|
||||
"sort"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ErrIsNaN is a field error for when a float field is NaN.
|
||||
var ErrIsNaN = &FieldError{"is NaN"}
|
||||
|
||||
// ErrIsInf is a field error for when a float field is Inf.
|
||||
var ErrIsInf = &FieldError{"is Inf"}
|
||||
|
||||
// Encoder marshals Metrics into influxdb line protocol.
|
||||
// It is not safe for concurrent use, make a new one!
|
||||
// The default behavior when encountering a field error is to ignore the field and move on.
|
||||
// If you wish it to error out on field errors, use Encoder.FailOnFieldErr(true)
|
||||
type Encoder struct {
|
||||
w io.Writer
|
||||
fieldSortOrder FieldSortOrder
|
||||
fieldTypeSupport FieldTypeSupport
|
||||
failOnFieldError bool
|
||||
maxLineBytes int
|
||||
fieldList []*Field
|
||||
header []byte
|
||||
footer []byte
|
||||
pair []byte
|
||||
precision time.Duration
|
||||
}
|
||||
|
||||
// SetMaxLineBytes sets a maximum length for a line, Encode will error if the generated line is longer
|
||||
func (e *Encoder) SetMaxLineBytes(i int) {
|
||||
e.maxLineBytes = i
|
||||
}
|
||||
|
||||
// SetFieldSortOrder sets a sort order for the data.
|
||||
// The options are:
|
||||
// NoSortFields (doesn't sort the fields)
|
||||
// SortFields (sorts the keys in alphabetical order)
|
||||
func (e *Encoder) SetFieldSortOrder(s FieldSortOrder) {
|
||||
e.fieldSortOrder = s
|
||||
}
|
||||
|
||||
// SetFieldTypeSupport sets flags for if the encoder supports certain optional field types such as uint64
|
||||
func (e *Encoder) SetFieldTypeSupport(s FieldTypeSupport) {
|
||||
e.fieldTypeSupport = s
|
||||
}
|
||||
|
||||
// FailOnFieldErr whether or not to fail on a field error or just move on.
|
||||
// The default behavior to move on
|
||||
func (e *Encoder) FailOnFieldErr(s bool) {
|
||||
e.failOnFieldError = s
|
||||
}
|
||||
|
||||
// SetPrecision sets time precision for writes
|
||||
// Default is nanoseconds precision
|
||||
func (e *Encoder) SetPrecision(p time.Duration) {
|
||||
e.precision = p
|
||||
}
|
||||
|
||||
// NewEncoder gives us an encoder that marshals to a writer in influxdb line protocol
|
||||
// as defined by:
|
||||
// https://docs.influxdata.com/influxdb/v1.5/write_protocols/line_protocol_reference/
|
||||
func NewEncoder(w io.Writer) *Encoder {
|
||||
return &Encoder{
|
||||
w: w,
|
||||
header: make([]byte, 0, 128),
|
||||
footer: make([]byte, 0, 128),
|
||||
pair: make([]byte, 0, 128),
|
||||
fieldList: make([]*Field, 0, 16),
|
||||
precision: time.Nanosecond,
|
||||
}
|
||||
}
|
||||
|
||||
// This is here to significantly reduce allocations, wish that we had constant/immutable keyword that applied to
|
||||
// more complex objects
|
||||
var comma = []byte(",")
|
||||
|
||||
// Encode marshals a Metric to the io.Writer in the Encoder
|
||||
func (e *Encoder) Encode(m Metric) (int, error) {
|
||||
err := e.buildHeader(m)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
e.buildFooter(m.Time())
|
||||
|
||||
// here we make a copy of the *fields so we can do an in-place sort
|
||||
e.fieldList = append(e.fieldList[:0], m.FieldList()...)
|
||||
|
||||
if e.fieldSortOrder == SortFields {
|
||||
sort.Slice(e.fieldList, func(i, j int) bool {
|
||||
return e.fieldList[i].Key < e.fieldList[j].Key
|
||||
})
|
||||
}
|
||||
i := 0
|
||||
totalWritten := 0
|
||||
pairsLen := 0
|
||||
firstField := true
|
||||
for _, field := range e.fieldList {
|
||||
err = e.buildFieldPair(field.Key, field.Value)
|
||||
if err != nil {
|
||||
if e.failOnFieldError {
|
||||
return 0, err
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
bytesNeeded := len(e.header) + pairsLen + len(e.pair) + len(e.footer)
|
||||
|
||||
// Additional length needed for field separator `,`
|
||||
if !firstField {
|
||||
bytesNeeded++
|
||||
}
|
||||
|
||||
if e.maxLineBytes > 0 && bytesNeeded > e.maxLineBytes {
|
||||
// Need at least one field per line
|
||||
if firstField {
|
||||
return 0, ErrNeedMoreSpace
|
||||
}
|
||||
|
||||
i, err = e.w.Write(e.footer)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
pairsLen = 0
|
||||
totalWritten += i
|
||||
|
||||
bytesNeeded = len(e.header) + len(e.pair) + len(e.footer)
|
||||
|
||||
if e.maxLineBytes > 0 && bytesNeeded > e.maxLineBytes {
|
||||
return 0, ErrNeedMoreSpace
|
||||
}
|
||||
|
||||
i, err = e.w.Write(e.header)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
totalWritten += i
|
||||
|
||||
i, err = e.w.Write(e.pair)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
totalWritten += i
|
||||
|
||||
pairsLen += len(e.pair)
|
||||
firstField = false
|
||||
continue
|
||||
}
|
||||
|
||||
if firstField {
|
||||
i, err = e.w.Write(e.header)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
totalWritten += i
|
||||
|
||||
} else {
|
||||
i, err = e.w.Write(comma)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
totalWritten += i
|
||||
|
||||
}
|
||||
|
||||
e.w.Write(e.pair)
|
||||
|
||||
pairsLen += len(e.pair)
|
||||
firstField = false
|
||||
}
|
||||
|
||||
if firstField {
|
||||
return 0, ErrNoFields
|
||||
}
|
||||
i, err = e.w.Write(e.footer)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
totalWritten += i
|
||||
return totalWritten, nil
|
||||
|
||||
}
|
||||
|
||||
func (e *Encoder) buildHeader(m Metric) error {
|
||||
e.header = e.header[:0]
|
||||
name := nameEscape(m.Name())
|
||||
if name == "" {
|
||||
return ErrInvalidName
|
||||
}
|
||||
e.header = append(e.header, name...)
|
||||
|
||||
for _, tag := range m.TagList() {
|
||||
key := escape(tag.Key)
|
||||
value := escape(tag.Value)
|
||||
|
||||
// Some keys and values are not encodeable as line protocol, such as
|
||||
// those with a trailing '\' or empty strings.
|
||||
if key == "" || value == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
e.header = append(e.header, ',')
|
||||
e.header = append(e.header, key...)
|
||||
e.header = append(e.header, '=')
|
||||
e.header = append(e.header, value...)
|
||||
}
|
||||
|
||||
e.header = append(e.header, ' ')
|
||||
return nil
|
||||
}
|
||||
|
||||
func (e *Encoder) buildFieldVal(value interface{}) error {
|
||||
switch v := value.(type) {
|
||||
case uint64:
|
||||
if e.fieldTypeSupport&UintSupport != 0 {
|
||||
e.pair = append(strconv.AppendUint(e.pair, v, 10), 'u')
|
||||
} else if v <= uint64(math.MaxInt64) {
|
||||
e.pair = append(strconv.AppendInt(e.pair, int64(v), 10), 'i')
|
||||
} else {
|
||||
e.pair = append(strconv.AppendInt(e.pair, math.MaxInt64, 10), 'i')
|
||||
}
|
||||
case int64:
|
||||
e.pair = append(strconv.AppendInt(e.pair, v, 10), 'i')
|
||||
case int:
|
||||
e.pair = append(strconv.AppendInt(e.pair, int64(v), 10), 'i')
|
||||
case float64:
|
||||
if math.IsNaN(v) {
|
||||
return ErrIsNaN
|
||||
}
|
||||
|
||||
if math.IsInf(v, 0) {
|
||||
return ErrIsInf
|
||||
}
|
||||
|
||||
e.pair = strconv.AppendFloat(e.pair, v, 'f', -1, 64)
|
||||
case float32:
|
||||
v32 := float64(v)
|
||||
if math.IsNaN(v32) {
|
||||
return ErrIsNaN
|
||||
}
|
||||
|
||||
if math.IsInf(v32, 0) {
|
||||
return ErrIsInf
|
||||
}
|
||||
|
||||
e.pair = strconv.AppendFloat(e.pair, v32, 'f', -1, 64)
|
||||
|
||||
case string:
|
||||
e.pair = append(e.pair, '"')
|
||||
e.pair = append(e.pair, stringFieldEscape(v)...)
|
||||
e.pair = append(e.pair, '"')
|
||||
case []byte:
|
||||
e.pair = append(e.pair, '"')
|
||||
stringFieldEscapeBytes(&e.pair, v)
|
||||
e.pair = append(e.pair, '"')
|
||||
case bool:
|
||||
e.pair = strconv.AppendBool(e.pair, v)
|
||||
default:
|
||||
return &FieldError{fmt.Sprintf("invalid value type: %T", v)}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (e *Encoder) buildFieldPair(key string, value interface{}) error {
|
||||
e.pair = e.pair[:0]
|
||||
key = escape(key)
|
||||
// Some keys are not encodeable as line protocol, such as those with a
|
||||
// trailing '\' or empty strings.
|
||||
if key == "" || key[:len(key)-1] == "\\" {
|
||||
return &FieldError{"invalid field key"}
|
||||
}
|
||||
e.pair = append(e.pair, key...)
|
||||
e.pair = append(e.pair, '=')
|
||||
return e.buildFieldVal(value)
|
||||
}
|
||||
|
||||
func (e *Encoder) buildFooter(t time.Time) {
|
||||
e.footer = e.footer[:0]
|
||||
if !t.IsZero() {
|
||||
e.footer = append(e.footer, ' ')
|
||||
switch e.precision {
|
||||
case time.Microsecond:
|
||||
e.footer = strconv.AppendInt(e.footer, t.UnixNano()/1000, 10)
|
||||
case time.Millisecond:
|
||||
e.footer = strconv.AppendInt(e.footer, t.UnixNano()/1000000, 10)
|
||||
case time.Second:
|
||||
e.footer = strconv.AppendInt(e.footer, t.Unix(), 10)
|
||||
default:
|
||||
e.footer = strconv.AppendInt(e.footer, t.UnixNano(), 10)
|
||||
}
|
||||
}
|
||||
e.footer = append(e.footer, '\n')
|
||||
}
|
||||
+264
@@ -0,0 +1,264 @@
|
||||
package protocol
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
"unsafe"
|
||||
)
|
||||
|
||||
const (
|
||||
escapes = "\t\n\f\r ,="
|
||||
nameEscapes = "\t\n\f\r ,"
|
||||
stringFieldEscapes = "\t\n\f\r\\\""
|
||||
)
|
||||
|
||||
var (
|
||||
stringEscaper = strings.NewReplacer(
|
||||
"\t", `\t`,
|
||||
"\n", `\n`,
|
||||
"\f", `\f`,
|
||||
"\r", `\r`,
|
||||
`,`, `\,`,
|
||||
` `, `\ `,
|
||||
`=`, `\=`,
|
||||
)
|
||||
|
||||
nameEscaper = strings.NewReplacer(
|
||||
"\t", `\t`,
|
||||
"\n", `\n`,
|
||||
"\f", `\f`,
|
||||
"\r", `\r`,
|
||||
`,`, `\,`,
|
||||
` `, `\ `,
|
||||
)
|
||||
|
||||
stringFieldEscaper = strings.NewReplacer(
|
||||
"\t", `\t`,
|
||||
"\n", `\n`,
|
||||
"\f", `\f`,
|
||||
"\r", `\r`,
|
||||
`"`, `\"`,
|
||||
`\`, `\\`,
|
||||
)
|
||||
)
|
||||
|
||||
var (
|
||||
unescaper = strings.NewReplacer(
|
||||
`\,`, `,`,
|
||||
`\"`, `"`, // ???
|
||||
`\ `, ` `,
|
||||
`\=`, `=`,
|
||||
)
|
||||
|
||||
nameUnescaper = strings.NewReplacer(
|
||||
`\,`, `,`,
|
||||
`\ `, ` `,
|
||||
)
|
||||
|
||||
stringFieldUnescaper = strings.NewReplacer(
|
||||
`\"`, `"`,
|
||||
`\\`, `\`,
|
||||
)
|
||||
)
|
||||
|
||||
// The various escape functions allocate, I'd like to fix that.
|
||||
// TODO: make escape not allocate
|
||||
|
||||
// Escape a tagkey, tagvalue, or fieldkey
|
||||
func escape(s string) string {
|
||||
if strings.ContainsAny(s, escapes) {
|
||||
return stringEscaper.Replace(s)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Escape a measurement name
|
||||
func nameEscape(s string) string {
|
||||
if strings.ContainsAny(s, nameEscapes) {
|
||||
return nameEscaper.Replace(s)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Escape a string field
|
||||
func stringFieldEscape(s string) string {
|
||||
if strings.ContainsAny(s, stringFieldEscapes) {
|
||||
return stringFieldEscaper.Replace(s)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
const (
|
||||
utf8mask = byte(0x3F)
|
||||
utf8bytex = byte(0x80) // 1000 0000
|
||||
utf8len2 = byte(0xC0) // 1100 0000
|
||||
utf8len3 = byte(0xE0) // 1110 0000
|
||||
utf8len4 = byte(0xF0) // 1111 0000
|
||||
)
|
||||
|
||||
func escapeBytes(dest *[]byte, b []byte) {
|
||||
if bytes.ContainsAny(b, escapes) {
|
||||
var r rune
|
||||
for i, j := 0, 0; i < len(b); i += j {
|
||||
r, j = utf8.DecodeRune(b[i:])
|
||||
switch {
|
||||
case r == '\t':
|
||||
*dest = append(*dest, `\t`...)
|
||||
case r == '\n':
|
||||
*dest = append(*dest, `\n`...)
|
||||
case r == '\f':
|
||||
*dest = append(*dest, `\f`...)
|
||||
case r == '\r':
|
||||
*dest = append(*dest, `\r`...)
|
||||
case r == ',':
|
||||
*dest = append(*dest, `\,`...)
|
||||
case r == ' ':
|
||||
*dest = append(*dest, `\ `...)
|
||||
case r == '=':
|
||||
*dest = append(*dest, `\=`...)
|
||||
case r <= 1<<7-1:
|
||||
*dest = append(*dest, byte(r))
|
||||
case r <= 1<<11-1:
|
||||
*dest = append(*dest, utf8len2|byte(r>>6), utf8bytex|byte(r)&utf8mask)
|
||||
case r <= 1<<16-1:
|
||||
*dest = append(*dest, utf8len3|byte(r>>12), utf8bytex|byte(r>>6)&utf8mask, utf8bytex|byte(r)&utf8mask)
|
||||
default:
|
||||
*dest = append(*dest, utf8len4|byte(r>>18), utf8bytex|byte(r>>12)&utf8mask, utf8bytex|byte(r>>6)&utf8mask, utf8bytex|byte(r)&utf8mask)
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
*dest = append(*dest, b...)
|
||||
}
|
||||
|
||||
// Escape a measurement name
|
||||
func nameEscapeBytes(dest *[]byte, b []byte) {
|
||||
if bytes.ContainsAny(b, nameEscapes) {
|
||||
var r rune
|
||||
for i, j := 0, 0; i < len(b); i += j {
|
||||
r, j = utf8.DecodeRune(b[i:])
|
||||
switch {
|
||||
case r == '\t':
|
||||
*dest = append(*dest, `\t`...)
|
||||
case r == '\n':
|
||||
*dest = append(*dest, `\n`...)
|
||||
case r == '\f':
|
||||
*dest = append(*dest, `\f`...)
|
||||
case r == '\r':
|
||||
*dest = append(*dest, `\r`...)
|
||||
case r == ',':
|
||||
*dest = append(*dest, `\,`...)
|
||||
case r == ' ':
|
||||
*dest = append(*dest, `\ `...)
|
||||
case r == '\\':
|
||||
*dest = append(*dest, `\\`...)
|
||||
case r <= 1<<7-1:
|
||||
*dest = append(*dest, byte(r))
|
||||
case r <= 1<<11-1:
|
||||
*dest = append(*dest, utf8len2|byte(r>>6), utf8bytex|byte(r)&utf8mask)
|
||||
case r <= 1<<16-1:
|
||||
*dest = append(*dest, utf8len3|byte(r>>12), utf8bytex|byte(r>>6)&utf8mask, utf8bytex|byte(r)&utf8mask)
|
||||
default:
|
||||
*dest = append(*dest, utf8len4|byte(r>>18), utf8bytex|byte(r>>12)&utf8mask, utf8bytex|byte(r>>6)&utf8mask, utf8bytex|byte(r)&utf8mask)
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
*dest = append(*dest, b...)
|
||||
}
|
||||
|
||||
func stringFieldEscapeBytes(dest *[]byte, b []byte) {
|
||||
if bytes.ContainsAny(b, stringFieldEscapes) {
|
||||
var r rune
|
||||
for i, j := 0, 0; i < len(b); i += j {
|
||||
r, j = utf8.DecodeRune(b[i:])
|
||||
switch {
|
||||
case r == '\t':
|
||||
*dest = append(*dest, `\t`...)
|
||||
case r == '\n':
|
||||
*dest = append(*dest, `\n`...)
|
||||
case r == '\f':
|
||||
*dest = append(*dest, `\f`...)
|
||||
case r == '\r':
|
||||
*dest = append(*dest, `\r`...)
|
||||
case r == ',':
|
||||
*dest = append(*dest, `\,`...)
|
||||
case r == ' ':
|
||||
*dest = append(*dest, `\ `...)
|
||||
case r == '\\':
|
||||
*dest = append(*dest, `\\`...)
|
||||
case r <= 1<<7-1:
|
||||
*dest = append(*dest, byte(r))
|
||||
case r <= 1<<11-1:
|
||||
*dest = append(*dest, utf8len2|byte(r>>6), utf8bytex|byte(r)&utf8mask)
|
||||
case r <= 1<<16-1:
|
||||
*dest = append(*dest, utf8len3|byte(r>>12), utf8bytex|byte(r>>6)&utf8mask, utf8bytex|byte(r)&utf8mask)
|
||||
default:
|
||||
*dest = append(*dest, utf8len4|byte(r>>18), utf8bytex|byte(r>>12)&utf8mask, utf8bytex|byte(r>>6)&utf8mask, utf8bytex|byte(r)&utf8mask)
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
*dest = append(*dest, b...)
|
||||
}
|
||||
|
||||
func unescape(b []byte) string {
|
||||
if bytes.ContainsAny(b, escapes) {
|
||||
return unescaper.Replace(unsafeBytesToString(b))
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func nameUnescape(b []byte) string {
|
||||
if bytes.ContainsAny(b, nameEscapes) {
|
||||
return nameUnescaper.Replace(unsafeBytesToString(b))
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// unsafeBytesToString converts a []byte to a string without a heap allocation.
|
||||
//
|
||||
// It is unsafe, and is intended to prepare input to short-lived functions
|
||||
// that require strings.
|
||||
func unsafeBytesToString(in []byte) string {
|
||||
src := *(*reflect.SliceHeader)(unsafe.Pointer(&in))
|
||||
dst := reflect.StringHeader{
|
||||
Data: src.Data,
|
||||
Len: src.Len,
|
||||
}
|
||||
s := *(*string)(unsafe.Pointer(&dst))
|
||||
return s
|
||||
}
|
||||
|
||||
// parseIntBytes is a zero-alloc wrapper around strconv.ParseInt.
|
||||
func parseIntBytes(b []byte, base int, bitSize int) (i int64, err error) {
|
||||
s := unsafeBytesToString(b)
|
||||
return strconv.ParseInt(s, base, bitSize)
|
||||
}
|
||||
|
||||
// parseUintBytes is a zero-alloc wrapper around strconv.ParseUint.
|
||||
func parseUintBytes(b []byte, base int, bitSize int) (i uint64, err error) {
|
||||
s := unsafeBytesToString(b)
|
||||
return strconv.ParseUint(s, base, bitSize)
|
||||
}
|
||||
|
||||
// parseFloatBytes is a zero-alloc wrapper around strconv.ParseFloat.
|
||||
func parseFloatBytes(b []byte, bitSize int) (float64, error) {
|
||||
s := unsafeBytesToString(b)
|
||||
return strconv.ParseFloat(s, bitSize)
|
||||
}
|
||||
|
||||
// parseBoolBytes is a zero-alloc wrapper around strconv.ParseBool.
|
||||
func parseBoolBytes(b []byte) (bool, error) {
|
||||
return strconv.ParseBool(unsafeBytesToString(b))
|
||||
}
|
||||
|
||||
func stringFieldUnescape(b []byte) string {
|
||||
if bytes.ContainsAny(b, stringFieldEscapes) {
|
||||
return stringFieldUnescaper.Replace(unsafeBytesToString(b))
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
package protocol
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
// MetricHandler implements the Handler interface and produces Metric.
|
||||
type MetricHandler struct {
|
||||
timePrecision time.Duration
|
||||
timeFunc TimeFunc
|
||||
metric MutableMetric
|
||||
}
|
||||
|
||||
func NewMetricHandler() *MetricHandler {
|
||||
return &MetricHandler{
|
||||
timePrecision: time.Nanosecond,
|
||||
timeFunc: time.Now,
|
||||
}
|
||||
}
|
||||
|
||||
func (h *MetricHandler) SetTimePrecision(p time.Duration) {
|
||||
h.timePrecision = p
|
||||
// When the timestamp is omitted from the metric, the timestamp
|
||||
// comes from the server clock, truncated to the nearest unit of
|
||||
// measurement provided in precision.
|
||||
//
|
||||
// When a timestamp is provided in the metric, precsision is
|
||||
// overloaded to hold the unit of measurement of the timestamp.
|
||||
}
|
||||
|
||||
func (h *MetricHandler) SetTimeFunc(f TimeFunc) {
|
||||
h.timeFunc = f
|
||||
}
|
||||
|
||||
func (h *MetricHandler) Metric() (Metric, error) {
|
||||
if h.metric.Time().IsZero() {
|
||||
h.metric.SetTime(h.timeFunc().Truncate(h.timePrecision))
|
||||
}
|
||||
return h.metric, nil
|
||||
}
|
||||
|
||||
func (h *MetricHandler) SetMeasurement(name []byte) error {
|
||||
var err error
|
||||
h.metric, err = New(nameUnescape(name),
|
||||
nil, nil, time.Time{})
|
||||
return err
|
||||
}
|
||||
|
||||
func (h *MetricHandler) AddTag(key []byte, value []byte) error {
|
||||
tk := unescape(key)
|
||||
tv := unescape(value)
|
||||
h.metric.AddTag(tk, tv)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (h *MetricHandler) AddInt(key []byte, value []byte) error {
|
||||
fk := unescape(key)
|
||||
fv, err := parseIntBytes(bytes.TrimSuffix(value, []byte("i")), 10, 64)
|
||||
if err != nil {
|
||||
if numerr, ok := err.(*strconv.NumError); ok {
|
||||
return numerr.Err
|
||||
}
|
||||
return err
|
||||
}
|
||||
h.metric.AddField(fk, fv)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (h *MetricHandler) AddUint(key []byte, value []byte) error {
|
||||
fk := unescape(key)
|
||||
fv, err := parseUintBytes(bytes.TrimSuffix(value, []byte("u")), 10, 64)
|
||||
if err != nil {
|
||||
if numerr, ok := err.(*strconv.NumError); ok {
|
||||
return numerr.Err
|
||||
}
|
||||
return err
|
||||
}
|
||||
h.metric.AddField(fk, fv)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (h *MetricHandler) AddFloat(key []byte, value []byte) error {
|
||||
fk := unescape(key)
|
||||
fv, err := parseFloatBytes(value, 64)
|
||||
if err != nil {
|
||||
if numerr, ok := err.(*strconv.NumError); ok {
|
||||
return numerr.Err
|
||||
}
|
||||
return err
|
||||
}
|
||||
h.metric.AddField(fk, fv)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (h *MetricHandler) AddString(key []byte, value []byte) error {
|
||||
fk := unescape(key)
|
||||
fv := stringFieldUnescape(value)
|
||||
h.metric.AddField(fk, fv)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (h *MetricHandler) AddBool(key []byte, value []byte) error {
|
||||
fk := unescape(key)
|
||||
fv, err := parseBoolBytes(value)
|
||||
if err != nil {
|
||||
return errors.New("unparseable bool")
|
||||
}
|
||||
h.metric.AddField(fk, fv)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (h *MetricHandler) SetTimestamp(tm []byte) error {
|
||||
v, err := parseIntBytes(tm, 10, 64)
|
||||
if err != nil {
|
||||
if numerr, ok := err.(*strconv.NumError); ok {
|
||||
return numerr.Err
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
//time precision is overloaded to mean time unit here
|
||||
ns := v * int64(h.timePrecision)
|
||||
h.metric.SetTime(time.Unix(0, ns))
|
||||
return nil
|
||||
}
|
||||
+34921
File diff suppressed because it is too large
Load Diff
+549
@@ -0,0 +1,549 @@
|
||||
package protocol
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io"
|
||||
)
|
||||
|
||||
var (
|
||||
ErrNameParse = errors.New("expected measurement name")
|
||||
ErrFieldParse = errors.New("expected field")
|
||||
ErrTagParse = errors.New("expected tag")
|
||||
ErrTimestampParse = errors.New("expected timestamp")
|
||||
ErrParse = errors.New("parse error")
|
||||
EOF = errors.New("EOF")
|
||||
)
|
||||
|
||||
%%{
|
||||
machine LineProtocol;
|
||||
|
||||
action begin {
|
||||
m.pb = m.p
|
||||
}
|
||||
|
||||
action name_error {
|
||||
err = ErrNameParse
|
||||
fhold;
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
|
||||
action field_error {
|
||||
err = ErrFieldParse
|
||||
fhold;
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
|
||||
action tagset_error {
|
||||
err = ErrTagParse
|
||||
fhold;
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
|
||||
action timestamp_error {
|
||||
err = ErrTimestampParse
|
||||
fhold;
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
|
||||
action parse_error {
|
||||
err = ErrParse
|
||||
fhold;
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
|
||||
action align_error {
|
||||
err = ErrParse
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
|
||||
action hold_recover {
|
||||
fhold;
|
||||
fgoto main;
|
||||
}
|
||||
|
||||
action goto_align {
|
||||
fgoto align;
|
||||
}
|
||||
|
||||
action begin_metric {
|
||||
m.beginMetric = true
|
||||
}
|
||||
|
||||
action name {
|
||||
err = m.handler.SetMeasurement(m.text())
|
||||
if err != nil {
|
||||
fhold;
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
}
|
||||
|
||||
action tagkey {
|
||||
m.key = m.text()
|
||||
}
|
||||
|
||||
action tagvalue {
|
||||
err = m.handler.AddTag(m.key, m.text())
|
||||
if err != nil {
|
||||
fhold;
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
}
|
||||
|
||||
action fieldkey {
|
||||
m.key = m.text()
|
||||
}
|
||||
|
||||
action integer {
|
||||
err = m.handler.AddInt(m.key, m.text())
|
||||
if err != nil {
|
||||
fhold;
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
}
|
||||
|
||||
action unsigned {
|
||||
err = m.handler.AddUint(m.key, m.text())
|
||||
if err != nil {
|
||||
fhold;
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
}
|
||||
|
||||
action float {
|
||||
err = m.handler.AddFloat(m.key, m.text())
|
||||
if err != nil {
|
||||
fhold;
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
}
|
||||
|
||||
action bool {
|
||||
err = m.handler.AddBool(m.key, m.text())
|
||||
if err != nil {
|
||||
fhold;
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
}
|
||||
|
||||
action string {
|
||||
err = m.handler.AddString(m.key, m.text())
|
||||
if err != nil {
|
||||
fhold;
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
}
|
||||
|
||||
action timestamp {
|
||||
err = m.handler.SetTimestamp(m.text())
|
||||
if err != nil {
|
||||
fhold;
|
||||
fnext discard_line;
|
||||
fbreak;
|
||||
}
|
||||
}
|
||||
|
||||
action incr_newline {
|
||||
m.lineno++
|
||||
m.sol = m.p
|
||||
m.sol++ // next char will be the first column in the line
|
||||
}
|
||||
|
||||
action eol {
|
||||
m.finishMetric = true
|
||||
fnext align;
|
||||
fbreak;
|
||||
}
|
||||
|
||||
action finish_metric {
|
||||
m.finishMetric = true
|
||||
}
|
||||
|
||||
ws =
|
||||
[\t\v\f ];
|
||||
|
||||
newline =
|
||||
'\r'? '\n' >incr_newline;
|
||||
|
||||
non_zero_digit =
|
||||
[1-9];
|
||||
|
||||
integer =
|
||||
'-'? ( digit | ( non_zero_digit digit* ) );
|
||||
|
||||
unsigned =
|
||||
( digit | ( non_zero_digit digit* ) );
|
||||
|
||||
number =
|
||||
'-'? (digit+ ('.' digit*)? | '.' digit+);
|
||||
|
||||
scientific =
|
||||
number 'e'i ["\-+"]? digit+;
|
||||
|
||||
timestamp =
|
||||
('-'? digit{1,19}) >begin %timestamp;
|
||||
|
||||
fieldkeychar =
|
||||
[^\t\n\f\r ,=\\] | ( '\\' [^\t\n\f\r] );
|
||||
|
||||
fieldkey =
|
||||
fieldkeychar+ >begin %fieldkey;
|
||||
|
||||
fieldfloat =
|
||||
(scientific | number) >begin %float;
|
||||
|
||||
fieldinteger =
|
||||
(integer 'i') >begin %integer;
|
||||
|
||||
fieldunsigned =
|
||||
(unsigned 'u') >begin %unsigned;
|
||||
|
||||
false =
|
||||
"false" | "FALSE" | "False" | "F" | "f";
|
||||
|
||||
true =
|
||||
"true" | "TRUE" | "True" | "T" | "t";
|
||||
|
||||
fieldbool =
|
||||
(true | false) >begin %bool;
|
||||
|
||||
fieldstringchar =
|
||||
[^\f\r\n\\"] | '\\' [\\"] | newline;
|
||||
|
||||
fieldstring =
|
||||
fieldstringchar* >begin %string;
|
||||
|
||||
fieldstringquoted =
|
||||
'"' fieldstring '"';
|
||||
|
||||
fieldvalue = fieldinteger | fieldunsigned | fieldfloat | fieldstringquoted | fieldbool;
|
||||
|
||||
field =
|
||||
fieldkey '=' fieldvalue;
|
||||
|
||||
fieldset =
|
||||
field ( ',' field )*;
|
||||
|
||||
tagchar =
|
||||
[^\t\n\f\r ,=\\] | ( '\\' [^\t\n\f\r\\] ) | '\\\\' %to{ fhold; };
|
||||
|
||||
tagkey =
|
||||
tagchar+ >begin %tagkey;
|
||||
|
||||
tagvalue =
|
||||
tagchar+ >begin %eof(tagvalue) %tagvalue;
|
||||
|
||||
tagset =
|
||||
((',' tagkey '=' tagvalue) $err(tagset_error))*;
|
||||
|
||||
measurement_chars =
|
||||
[^\t\n\f\r ,\\] | ( '\\' [^\t\n\f\r] );
|
||||
|
||||
measurement_start =
|
||||
measurement_chars - '#';
|
||||
|
||||
measurement =
|
||||
(measurement_start measurement_chars*) >begin %eof(name) %name;
|
||||
|
||||
eol_break =
|
||||
newline %to(eol)
|
||||
;
|
||||
|
||||
metric =
|
||||
measurement >err(name_error)
|
||||
tagset
|
||||
ws+ fieldset $err(field_error)
|
||||
(ws+ timestamp)? $err(timestamp_error)
|
||||
;
|
||||
|
||||
line_with_term =
|
||||
ws* metric ws* eol_break
|
||||
;
|
||||
|
||||
line_without_term =
|
||||
ws* metric ws*
|
||||
;
|
||||
|
||||
main :=
|
||||
(line_with_term*
|
||||
(line_with_term | line_without_term?)
|
||||
) >begin_metric %eof(finish_metric)
|
||||
;
|
||||
|
||||
# The discard_line machine discards the current line. Useful for recovering
|
||||
# on the next line when an error occurs.
|
||||
discard_line :=
|
||||
(any -- newline)* newline @goto_align;
|
||||
|
||||
commentline =
|
||||
ws* '#' (any -- newline)* newline;
|
||||
|
||||
emptyline =
|
||||
ws* newline;
|
||||
|
||||
# The align machine scans forward to the start of the next line. This machine
|
||||
# is used to skip over whitespace and comments, keeping this logic out of the
|
||||
# main machine.
|
||||
#
|
||||
# Skip valid lines that don't contain line protocol, any other data will move
|
||||
# control to the main parser via the err action.
|
||||
align :=
|
||||
(emptyline | commentline | ws+)* %err(hold_recover);
|
||||
|
||||
# Series is a machine for matching measurement+tagset
|
||||
series :=
|
||||
(measurement >err(name_error) tagset eol_break?)
|
||||
>begin_metric
|
||||
;
|
||||
}%%
|
||||
|
||||
%% write data;
|
||||
|
||||
type Handler interface {
|
||||
SetMeasurement(name []byte) error
|
||||
AddTag(key []byte, value []byte) error
|
||||
AddInt(key []byte, value []byte) error
|
||||
AddUint(key []byte, value []byte) error
|
||||
AddFloat(key []byte, value []byte) error
|
||||
AddString(key []byte, value []byte) error
|
||||
AddBool(key []byte, value []byte) error
|
||||
SetTimestamp(tm []byte) error
|
||||
}
|
||||
|
||||
type machine struct {
|
||||
data []byte
|
||||
cs int
|
||||
p, pe, eof int
|
||||
pb int
|
||||
lineno int
|
||||
sol int
|
||||
handler Handler
|
||||
initState int
|
||||
key []byte
|
||||
beginMetric bool
|
||||
finishMetric bool
|
||||
}
|
||||
|
||||
func NewMachine(handler Handler) *machine {
|
||||
m := &machine{
|
||||
handler: handler,
|
||||
initState: LineProtocol_en_align,
|
||||
}
|
||||
|
||||
%% access m.;
|
||||
%% variable p m.p;
|
||||
%% variable cs m.cs;
|
||||
%% variable pe m.pe;
|
||||
%% variable eof m.eof;
|
||||
%% variable data m.data;
|
||||
%% write init;
|
||||
|
||||
return m
|
||||
}
|
||||
|
||||
func NewSeriesMachine(handler Handler) *machine {
|
||||
m := &machine{
|
||||
handler: handler,
|
||||
initState: LineProtocol_en_series,
|
||||
}
|
||||
|
||||
%% access m.;
|
||||
%% variable p m.p;
|
||||
%% variable pe m.pe;
|
||||
%% variable eof m.eof;
|
||||
%% variable data m.data;
|
||||
%% write init;
|
||||
|
||||
return m
|
||||
}
|
||||
|
||||
func (m *machine) SetData(data []byte) {
|
||||
m.data = data
|
||||
m.p = 0
|
||||
m.pb = 0
|
||||
m.lineno = 1
|
||||
m.sol = 0
|
||||
m.pe = len(data)
|
||||
m.eof = len(data)
|
||||
m.key = nil
|
||||
m.beginMetric = false
|
||||
m.finishMetric = false
|
||||
|
||||
%% write init;
|
||||
m.cs = m.initState
|
||||
}
|
||||
|
||||
// Next parses the next metric line and returns nil if it was successfully
|
||||
// processed. If the line contains a syntax error an error is returned,
|
||||
// otherwise if the end of file is reached before finding a metric line then
|
||||
// EOF is returned.
|
||||
func (m *machine) Next() error {
|
||||
if m.p == m.pe && m.pe == m.eof {
|
||||
return EOF
|
||||
}
|
||||
|
||||
m.key = nil
|
||||
m.beginMetric = false
|
||||
m.finishMetric = false
|
||||
|
||||
return m.exec()
|
||||
}
|
||||
|
||||
func (m *machine) exec() error {
|
||||
var err error
|
||||
%% write exec;
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// This would indicate an error in the machine that was reported with a
|
||||
// more specific error. We return a generic error but this should
|
||||
// possibly be a panic.
|
||||
if m.cs == %%{ write error; }%% {
|
||||
m.cs = LineProtocol_en_discard_line
|
||||
return ErrParse
|
||||
}
|
||||
|
||||
// If we haven't found a metric line yet and we reached the EOF, report it
|
||||
// now. This happens when the data ends with a comment or whitespace.
|
||||
//
|
||||
// Otherwise we have successfully parsed a metric line, so if we are at
|
||||
// the EOF we will report it the next call.
|
||||
if !m.beginMetric && m.p == m.pe && m.pe == m.eof {
|
||||
return EOF
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Position returns the current byte offset into the data.
|
||||
func (m *machine) Position() int {
|
||||
return m.p
|
||||
}
|
||||
|
||||
// LineOffset returns the byte offset of the current line.
|
||||
func (m *machine) LineOffset() int {
|
||||
return m.sol
|
||||
}
|
||||
|
||||
// LineNumber returns the current line number. Lines are counted based on the
|
||||
// regular expression `\r?\n`.
|
||||
func (m *machine) LineNumber() int {
|
||||
return m.lineno
|
||||
}
|
||||
|
||||
// Column returns the current column.
|
||||
func (m *machine) Column() int {
|
||||
lineOffset := m.p - m.sol
|
||||
return lineOffset + 1
|
||||
}
|
||||
|
||||
func (m *machine) text() []byte {
|
||||
return m.data[m.pb:m.p]
|
||||
}
|
||||
|
||||
type streamMachine struct {
|
||||
machine *machine
|
||||
reader io.Reader
|
||||
}
|
||||
|
||||
func NewStreamMachine(r io.Reader, handler Handler) *streamMachine {
|
||||
m := &streamMachine{
|
||||
machine: NewMachine(handler),
|
||||
reader: r,
|
||||
}
|
||||
|
||||
m.machine.SetData(make([]byte, 1024))
|
||||
m.machine.pe = 0
|
||||
m.machine.eof = -1
|
||||
return m
|
||||
}
|
||||
|
||||
func (m *streamMachine) Next() error {
|
||||
// Check if we are already at EOF, this should only happen if called again
|
||||
// after already returning EOF.
|
||||
if m.machine.p == m.machine.pe && m.machine.pe == m.machine.eof {
|
||||
return EOF
|
||||
}
|
||||
|
||||
copy(m.machine.data, m.machine.data[m.machine.p:])
|
||||
m.machine.pe = m.machine.pe - m.machine.p
|
||||
m.machine.sol = m.machine.sol - m.machine.p
|
||||
m.machine.pb = 0
|
||||
m.machine.p = 0
|
||||
m.machine.eof = -1
|
||||
|
||||
m.machine.key = nil
|
||||
m.machine.beginMetric = false
|
||||
m.machine.finishMetric = false
|
||||
|
||||
for {
|
||||
// Expand the buffer if it is full
|
||||
if m.machine.pe == len(m.machine.data) {
|
||||
expanded := make([]byte, 2 * len(m.machine.data))
|
||||
copy(expanded, m.machine.data)
|
||||
m.machine.data = expanded
|
||||
}
|
||||
|
||||
n, err := m.reader.Read(m.machine.data[m.machine.pe:])
|
||||
if n == 0 && err == io.EOF {
|
||||
m.machine.eof = m.machine.pe
|
||||
} else if err != nil && err != io.EOF {
|
||||
return err
|
||||
}
|
||||
|
||||
m.machine.pe += n
|
||||
|
||||
err = m.machine.exec()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// If we have successfully parsed a full metric line break out
|
||||
if m.machine.finishMetric {
|
||||
break
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Position returns the current byte offset into the data.
|
||||
func (m *streamMachine) Position() int {
|
||||
return m.machine.Position()
|
||||
}
|
||||
|
||||
// LineOffset returns the byte offset of the current line.
|
||||
func (m *streamMachine) LineOffset() int {
|
||||
return m.machine.LineOffset()
|
||||
}
|
||||
|
||||
// LineNumber returns the current line number. Lines are counted based on the
|
||||
// regular expression `\r?\n`.
|
||||
func (m *streamMachine) LineNumber() int {
|
||||
return m.machine.LineNumber()
|
||||
}
|
||||
|
||||
// Column returns the current column.
|
||||
func (m *streamMachine) Column() int {
|
||||
return m.machine.Column()
|
||||
}
|
||||
|
||||
// LineText returns the text of the current line that has been parsed so far.
|
||||
func (m *streamMachine) LineText() string {
|
||||
return string(m.machine.data[0:m.machine.p])
|
||||
}
|
||||
+428
@@ -0,0 +1,428 @@
|
||||
package protocol
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"hash/fnv"
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Tag holds the keys and values for a bunch of Tag k/v pairs.
|
||||
type Tag struct {
|
||||
Key string
|
||||
Value string
|
||||
}
|
||||
|
||||
// Field holds the keys and values for a bunch of Metric Field k/v pairs where Value can be a uint64, int64, int, float32, float64, string, or bool.
|
||||
type Field struct {
|
||||
Key string
|
||||
Value interface{}
|
||||
}
|
||||
|
||||
// Metric is the interface for marshaling, if you implement this interface you can be marshalled into the line protocol. Woot!
|
||||
type Metric interface {
|
||||
Time() time.Time
|
||||
Name() string
|
||||
TagList() []*Tag
|
||||
FieldList() []*Field
|
||||
}
|
||||
|
||||
// MutableMetric represents a metric that can be be modified.
|
||||
type MutableMetric interface {
|
||||
Metric
|
||||
SetTime(time.Time)
|
||||
AddTag(key, value string)
|
||||
AddField(key string, value interface{})
|
||||
}
|
||||
|
||||
// FieldSortOrder is a type for controlling if Fields are sorted
|
||||
type FieldSortOrder int
|
||||
|
||||
const (
|
||||
// NoSortFields tells the Decoder to not sort the fields.
|
||||
NoSortFields FieldSortOrder = iota
|
||||
|
||||
// SortFields tells the Decoder to sort the fields.
|
||||
SortFields
|
||||
)
|
||||
|
||||
// FieldTypeSupport is a type for the parser to understand its type support.
|
||||
type FieldTypeSupport int
|
||||
|
||||
const (
|
||||
// UintSupport means the parser understands uint64s and can store them without having to convert to int64.
|
||||
UintSupport FieldTypeSupport = 1 << iota
|
||||
)
|
||||
|
||||
// MetricError is an error causing a metric to be unserializable.
|
||||
type MetricError struct {
|
||||
s string
|
||||
}
|
||||
|
||||
func (e MetricError) Error() string {
|
||||
return e.s
|
||||
}
|
||||
|
||||
// FieldError is an error causing a field to be unserializable.
|
||||
type FieldError struct {
|
||||
s string
|
||||
}
|
||||
|
||||
func (e FieldError) Error() string {
|
||||
return e.s
|
||||
}
|
||||
|
||||
var (
|
||||
// ErrNeedMoreSpace tells us that the Decoder's io.Reader is full.
|
||||
ErrNeedMoreSpace = &MetricError{"need more space"}
|
||||
|
||||
// ErrInvalidName tells us that the chosen name is invalid.
|
||||
ErrInvalidName = &MetricError{"invalid name"}
|
||||
|
||||
// ErrNoFields tells us that there were no serializable fields in the line/metric.
|
||||
ErrNoFields = &MetricError{"no serializable fields"}
|
||||
)
|
||||
|
||||
type metric struct {
|
||||
name string
|
||||
tags []*Tag
|
||||
fields []*Field
|
||||
tm time.Time
|
||||
}
|
||||
|
||||
// New creates a new metric via maps.
|
||||
func New(
|
||||
name string,
|
||||
tags map[string]string,
|
||||
fields map[string]interface{},
|
||||
tm time.Time,
|
||||
) (MutableMetric, error) {
|
||||
m := &metric{
|
||||
name: name,
|
||||
tags: nil,
|
||||
fields: nil,
|
||||
tm: tm,
|
||||
}
|
||||
|
||||
if len(tags) > 0 {
|
||||
m.tags = make([]*Tag, 0, len(tags))
|
||||
for k, v := range tags {
|
||||
m.tags = append(m.tags,
|
||||
&Tag{Key: k, Value: v})
|
||||
}
|
||||
sort.Slice(m.tags, func(i, j int) bool { return m.tags[i].Key < m.tags[j].Key })
|
||||
}
|
||||
|
||||
if len(fields) > 0 {
|
||||
m.fields = make([]*Field, 0, len(fields))
|
||||
for k, v := range fields {
|
||||
v := convertField(v)
|
||||
if v == nil {
|
||||
continue
|
||||
}
|
||||
m.AddField(k, v)
|
||||
}
|
||||
}
|
||||
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// FromMetric returns a deep copy of the metric with any tracking information
|
||||
// removed.
|
||||
func FromMetric(other Metric) Metric {
|
||||
m := &metric{
|
||||
name: other.Name(),
|
||||
tags: make([]*Tag, len(other.TagList())),
|
||||
fields: make([]*Field, len(other.FieldList())),
|
||||
tm: other.Time(),
|
||||
}
|
||||
|
||||
for i, tag := range other.TagList() {
|
||||
m.tags[i] = &Tag{Key: tag.Key, Value: tag.Value}
|
||||
}
|
||||
|
||||
for i, field := range other.FieldList() {
|
||||
m.fields[i] = &Field{Key: field.Key, Value: field.Value}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func (m *metric) String() string {
|
||||
return fmt.Sprintf("%s %v %v %d", m.name, m.Tags(), m.Fields(), m.tm.UnixNano())
|
||||
}
|
||||
|
||||
func (m *metric) Name() string {
|
||||
return m.name
|
||||
}
|
||||
|
||||
func (m *metric) Tags() map[string]string {
|
||||
tags := make(map[string]string, len(m.tags))
|
||||
for _, tag := range m.tags {
|
||||
tags[tag.Key] = tag.Value
|
||||
}
|
||||
return tags
|
||||
}
|
||||
|
||||
func (m *metric) TagList() []*Tag {
|
||||
return m.tags
|
||||
}
|
||||
|
||||
func (m *metric) Fields() map[string]interface{} {
|
||||
fields := make(map[string]interface{}, len(m.fields))
|
||||
for _, field := range m.fields {
|
||||
fields[field.Key] = field.Value
|
||||
}
|
||||
|
||||
return fields
|
||||
}
|
||||
|
||||
func (m *metric) FieldList() []*Field {
|
||||
return m.fields
|
||||
}
|
||||
|
||||
func (m *metric) Time() time.Time {
|
||||
return m.tm
|
||||
}
|
||||
|
||||
func (m *metric) SetName(name string) {
|
||||
m.name = name
|
||||
}
|
||||
|
||||
func (m *metric) AddPrefix(prefix string) {
|
||||
m.name = prefix + m.name
|
||||
}
|
||||
|
||||
func (m *metric) AddSuffix(suffix string) {
|
||||
m.name = m.name + suffix
|
||||
}
|
||||
|
||||
func (m *metric) AddTag(key, value string) {
|
||||
for i, tag := range m.tags {
|
||||
if key > tag.Key {
|
||||
continue
|
||||
}
|
||||
|
||||
if key == tag.Key {
|
||||
tag.Value = value
|
||||
return
|
||||
}
|
||||
|
||||
m.tags = append(m.tags, nil)
|
||||
copy(m.tags[i+1:], m.tags[i:])
|
||||
m.tags[i] = &Tag{Key: key, Value: value}
|
||||
return
|
||||
}
|
||||
|
||||
m.tags = append(m.tags, &Tag{Key: key, Value: value})
|
||||
}
|
||||
|
||||
func (m *metric) HasTag(key string) bool {
|
||||
for _, tag := range m.tags {
|
||||
if tag.Key == key {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (m *metric) GetTag(key string) (string, bool) {
|
||||
for _, tag := range m.tags {
|
||||
if tag.Key == key {
|
||||
return tag.Value, true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
func (m *metric) RemoveTag(key string) {
|
||||
for i, tag := range m.tags {
|
||||
if tag.Key == key {
|
||||
copy(m.tags[i:], m.tags[i+1:])
|
||||
m.tags[len(m.tags)-1] = nil
|
||||
m.tags = m.tags[:len(m.tags)-1]
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (m *metric) AddField(key string, value interface{}) {
|
||||
for i, field := range m.fields {
|
||||
if key == field.Key {
|
||||
m.fields[i] = &Field{Key: key, Value: convertField(value)}
|
||||
return
|
||||
}
|
||||
}
|
||||
m.fields = append(m.fields, &Field{Key: key, Value: convertField(value)})
|
||||
}
|
||||
|
||||
func (m *metric) HasField(key string) bool {
|
||||
for _, field := range m.fields {
|
||||
if field.Key == key {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (m *metric) GetField(key string) (interface{}, bool) {
|
||||
for _, field := range m.fields {
|
||||
if field.Key == key {
|
||||
return field.Value, true
|
||||
}
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
func (m *metric) RemoveField(key string) {
|
||||
for i, field := range m.fields {
|
||||
if field.Key == key {
|
||||
copy(m.fields[i:], m.fields[i+1:])
|
||||
m.fields[len(m.fields)-1] = nil
|
||||
m.fields = m.fields[:len(m.fields)-1]
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (m *metric) SetTime(t time.Time) {
|
||||
m.tm = t
|
||||
}
|
||||
|
||||
func (m *metric) Copy() Metric {
|
||||
m2 := &metric{
|
||||
name: m.name,
|
||||
tags: make([]*Tag, len(m.tags)),
|
||||
fields: make([]*Field, len(m.fields)),
|
||||
tm: m.tm,
|
||||
}
|
||||
|
||||
for i, tag := range m.tags {
|
||||
m2.tags[i] = &Tag{Key: tag.Key, Value: tag.Value}
|
||||
}
|
||||
|
||||
for i, field := range m.fields {
|
||||
m2.fields[i] = &Field{Key: field.Key, Value: field.Value}
|
||||
}
|
||||
return m2
|
||||
}
|
||||
|
||||
func (m *metric) HashID() uint64 {
|
||||
h := fnv.New64a()
|
||||
h.Write([]byte(m.name))
|
||||
h.Write([]byte("\n"))
|
||||
for _, tag := range m.tags {
|
||||
h.Write([]byte(tag.Key))
|
||||
h.Write([]byte("\n"))
|
||||
h.Write([]byte(tag.Value))
|
||||
h.Write([]byte("\n"))
|
||||
}
|
||||
return h.Sum64()
|
||||
}
|
||||
|
||||
func (m *metric) Accept() {
|
||||
}
|
||||
|
||||
func (m *metric) Reject() {
|
||||
}
|
||||
|
||||
func (m *metric) Drop() {
|
||||
}
|
||||
|
||||
// Convert field to a supported type or nil if unconvertible
|
||||
func convertField(v interface{}) interface{} {
|
||||
switch v := v.(type) {
|
||||
case float64:
|
||||
return v
|
||||
case int64:
|
||||
return v
|
||||
case string:
|
||||
return v
|
||||
case bool:
|
||||
return v
|
||||
case int:
|
||||
return int64(v)
|
||||
case uint:
|
||||
return uint64(v)
|
||||
case uint64:
|
||||
return uint64(v)
|
||||
case []byte:
|
||||
return string(v)
|
||||
case int32:
|
||||
return int64(v)
|
||||
case int16:
|
||||
return int64(v)
|
||||
case int8:
|
||||
return int64(v)
|
||||
case uint32:
|
||||
return uint64(v)
|
||||
case uint16:
|
||||
return uint64(v)
|
||||
case uint8:
|
||||
return uint64(v)
|
||||
case float32:
|
||||
return float64(v)
|
||||
case *float64:
|
||||
if v != nil {
|
||||
return *v
|
||||
}
|
||||
case *int64:
|
||||
if v != nil {
|
||||
return *v
|
||||
}
|
||||
case *string:
|
||||
if v != nil {
|
||||
return *v
|
||||
}
|
||||
case *bool:
|
||||
if v != nil {
|
||||
return *v
|
||||
}
|
||||
case *int:
|
||||
if v != nil {
|
||||
return int64(*v)
|
||||
}
|
||||
case *uint:
|
||||
if v != nil {
|
||||
return uint64(*v)
|
||||
}
|
||||
case *uint64:
|
||||
if v != nil {
|
||||
return uint64(*v)
|
||||
}
|
||||
case *[]byte:
|
||||
if v != nil {
|
||||
return string(*v)
|
||||
}
|
||||
case *int32:
|
||||
if v != nil {
|
||||
return int64(*v)
|
||||
}
|
||||
case *int16:
|
||||
if v != nil {
|
||||
return int64(*v)
|
||||
}
|
||||
case *int8:
|
||||
if v != nil {
|
||||
return int64(*v)
|
||||
}
|
||||
case *uint32:
|
||||
if v != nil {
|
||||
return uint64(*v)
|
||||
}
|
||||
case *uint16:
|
||||
if v != nil {
|
||||
return uint64(*v)
|
||||
}
|
||||
case *uint8:
|
||||
if v != nil {
|
||||
return uint64(*v)
|
||||
}
|
||||
case *float32:
|
||||
if v != nil {
|
||||
return float64(*v)
|
||||
}
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+192
@@ -0,0 +1,192 @@
|
||||
package protocol
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
maxErrorBufferSize = 1024
|
||||
)
|
||||
|
||||
// TimeFunc is used to override the default time for a metric
|
||||
// with no specified timestamp.
|
||||
type TimeFunc func() time.Time
|
||||
|
||||
// ParseError indicates a error in the parsing of the text.
|
||||
type ParseError struct {
|
||||
Offset int
|
||||
LineOffset int
|
||||
LineNumber int
|
||||
Column int
|
||||
msg string
|
||||
buf string
|
||||
}
|
||||
|
||||
func (e *ParseError) Error() string {
|
||||
buffer := e.buf[e.LineOffset:]
|
||||
eol := strings.IndexAny(buffer, "\r\n")
|
||||
if eol >= 0 {
|
||||
buffer = buffer[:eol]
|
||||
}
|
||||
if len(buffer) > maxErrorBufferSize {
|
||||
buffer = buffer[:maxErrorBufferSize] + "..."
|
||||
}
|
||||
return fmt.Sprintf("metric parse error: %s at %d:%d: %q", e.msg, e.LineNumber, e.Column, buffer)
|
||||
}
|
||||
|
||||
// Parser is an InfluxDB Line Protocol parser that implements the
|
||||
// parsers.Parser interface.
|
||||
type Parser struct {
|
||||
DefaultTags map[string]string
|
||||
|
||||
sync.Mutex
|
||||
*machine
|
||||
handler *MetricHandler
|
||||
}
|
||||
|
||||
// NewParser returns a Parser than accepts line protocol
|
||||
func NewParser(handler *MetricHandler) *Parser {
|
||||
return &Parser{
|
||||
machine: NewMachine(handler),
|
||||
handler: handler,
|
||||
}
|
||||
}
|
||||
|
||||
// NewSeriesParser returns a Parser than accepts a measurement and tagset
|
||||
func NewSeriesParser(handler *MetricHandler) *Parser {
|
||||
return &Parser{
|
||||
machine: NewSeriesMachine(handler),
|
||||
handler: handler,
|
||||
}
|
||||
}
|
||||
|
||||
// SetTimeFunc allows default times to be set when no time is specified
|
||||
// for a metric in line-protocol.
|
||||
func (p *Parser) SetTimeFunc(f TimeFunc) {
|
||||
p.handler.SetTimeFunc(f)
|
||||
}
|
||||
|
||||
// Parse interprets line-protocol bytes as many metrics.
|
||||
func (p *Parser) Parse(input []byte) ([]Metric, error) {
|
||||
p.Lock()
|
||||
defer p.Unlock()
|
||||
metrics := make([]Metric, 0)
|
||||
p.machine.SetData(input)
|
||||
|
||||
for {
|
||||
err := p.machine.Next()
|
||||
if err == EOF {
|
||||
break
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return nil, &ParseError{
|
||||
Offset: p.machine.Position(),
|
||||
LineOffset: p.machine.LineOffset(),
|
||||
LineNumber: p.machine.LineNumber(),
|
||||
Column: p.machine.Column(),
|
||||
msg: err.Error(),
|
||||
buf: string(input),
|
||||
}
|
||||
}
|
||||
|
||||
metric, err := p.handler.Metric()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if metric == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
metrics = append(metrics, metric)
|
||||
}
|
||||
|
||||
return metrics, nil
|
||||
}
|
||||
|
||||
// StreamParser is an InfluxDB Line Protocol parser. It is not safe for
|
||||
// concurrent use in multiple goroutines.
|
||||
type StreamParser struct {
|
||||
machine *streamMachine
|
||||
handler *MetricHandler
|
||||
}
|
||||
|
||||
// NewStreamParser parses from a reader and iterates the machine
|
||||
// metric by metric. Not safe for concurrent use in multiple goroutines.
|
||||
func NewStreamParser(r io.Reader) *StreamParser {
|
||||
handler := NewMetricHandler()
|
||||
return &StreamParser{
|
||||
machine: NewStreamMachine(r, handler),
|
||||
handler: handler,
|
||||
}
|
||||
}
|
||||
|
||||
// SetTimeFunc changes the function used to determine the time of metrics
|
||||
// without a timestamp. The default TimeFunc is time.Now. Useful mostly for
|
||||
// testing, or perhaps if you want all metrics to have the same timestamp.
|
||||
func (p *StreamParser) SetTimeFunc(f TimeFunc) {
|
||||
p.handler.SetTimeFunc(f)
|
||||
}
|
||||
|
||||
// SetTimePrecision specifies units for the time stamp.
|
||||
func (p *StreamParser) SetTimePrecision(u time.Duration) {
|
||||
p.handler.SetTimePrecision(u)
|
||||
}
|
||||
|
||||
// Next parses the next item from the stream. You can repeat calls to this
|
||||
// function until it returns EOF.
|
||||
func (p *StreamParser) Next() (Metric, error) {
|
||||
err := p.machine.Next()
|
||||
if err == EOF {
|
||||
return nil, EOF
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return nil, &ParseError{
|
||||
Offset: p.machine.Position(),
|
||||
LineOffset: p.machine.LineOffset(),
|
||||
LineNumber: p.machine.LineNumber(),
|
||||
Column: p.machine.Column(),
|
||||
msg: err.Error(),
|
||||
buf: p.machine.LineText(),
|
||||
}
|
||||
}
|
||||
|
||||
metric, err := p.handler.Metric()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return metric, nil
|
||||
}
|
||||
|
||||
// Position returns the current byte offset into the data.
|
||||
func (p *StreamParser) Position() int {
|
||||
return p.machine.Position()
|
||||
}
|
||||
|
||||
// LineOffset returns the byte offset of the current line.
|
||||
func (p *StreamParser) LineOffset() int {
|
||||
return p.machine.LineOffset()
|
||||
}
|
||||
|
||||
// LineNumber returns the current line number. Lines are counted based on the
|
||||
// regular expression `\r?\n`.
|
||||
func (p *StreamParser) LineNumber() int {
|
||||
return p.machine.LineNumber()
|
||||
}
|
||||
|
||||
// Column returns the current column.
|
||||
func (p *StreamParser) Column() int {
|
||||
return p.machine.Column()
|
||||
}
|
||||
|
||||
// LineText returns the text of the current line that has been parsed so far.
|
||||
func (p *StreamParser) LineText() string {
|
||||
return p.machine.LineText()
|
||||
}
|
||||
+130
@@ -0,0 +1,130 @@
|
||||
package protocol
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Write writes out data to a line protocol encoder. Note: it does no sorting. It assumes you have done your own sorting for tagValues
|
||||
func (e *Encoder) Write(name []byte, ts time.Time, tagKeys, tagVals, fieldKeys [][]byte, fieldVals []interface{}) (int, error) {
|
||||
e.header = e.header[:0]
|
||||
if len(name) == 0 || name[len(name)-1] == byte('\\') {
|
||||
return 0, ErrInvalidName
|
||||
}
|
||||
nameEscapeBytes(&e.header, name)
|
||||
for i := range tagKeys {
|
||||
// Some keys and values are not encodeable as line protocol, such as
|
||||
// those with a trailing '\' or empty strings.
|
||||
if len(tagKeys[i]) == 0 || len(tagVals[i]) == 0 || tagKeys[i][len(tagKeys[i])-1] == byte('\\') {
|
||||
if e.failOnFieldError {
|
||||
return 0, fmt.Errorf("invalid field: key \"%s\", val \"%s\"", tagKeys[i], tagVals[i])
|
||||
}
|
||||
continue
|
||||
}
|
||||
e.header = append(e.header, byte(','))
|
||||
escapeBytes(&e.header, tagKeys[i])
|
||||
e.header = append(e.header, byte('='))
|
||||
escapeBytes(&e.header, tagVals[i])
|
||||
}
|
||||
e.header = append(e.header, byte(' '))
|
||||
e.buildFooter(ts)
|
||||
|
||||
i := 0
|
||||
totalWritten := 0
|
||||
pairsLen := 0
|
||||
firstField := true
|
||||
for i := range fieldKeys {
|
||||
e.pair = e.pair[:0]
|
||||
key := fieldKeys[i]
|
||||
if len(key) == 0 || key[len(key)-1] == byte('\\') {
|
||||
if e.failOnFieldError {
|
||||
return 0, &FieldError{"invalid field key"}
|
||||
}
|
||||
continue
|
||||
}
|
||||
escapeBytes(&e.pair, key)
|
||||
// Some keys are not encodeable as line protocol, such as those with a
|
||||
// trailing '\' or empty strings.
|
||||
e.pair = append(e.pair, byte('='))
|
||||
err := e.buildFieldVal(fieldVals[i])
|
||||
if err != nil {
|
||||
if e.failOnFieldError {
|
||||
return 0, err
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
bytesNeeded := len(e.header) + pairsLen + len(e.pair) + len(e.footer)
|
||||
|
||||
// Additional length needed for field separator `,`
|
||||
if !firstField {
|
||||
bytesNeeded++
|
||||
}
|
||||
|
||||
if e.maxLineBytes > 0 && bytesNeeded > e.maxLineBytes {
|
||||
// Need at least one field per line
|
||||
if firstField {
|
||||
return 0, ErrNeedMoreSpace
|
||||
}
|
||||
|
||||
i, err = e.w.Write(e.footer)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
totalWritten += i
|
||||
|
||||
bytesNeeded = len(e.header) + len(e.pair) + len(e.footer)
|
||||
|
||||
if e.maxLineBytes > 0 && bytesNeeded > e.maxLineBytes {
|
||||
return 0, ErrNeedMoreSpace
|
||||
}
|
||||
|
||||
i, err = e.w.Write(e.header)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
totalWritten += i
|
||||
|
||||
i, err = e.w.Write(e.pair)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
totalWritten += i
|
||||
|
||||
pairsLen += len(e.pair)
|
||||
firstField = false
|
||||
continue
|
||||
}
|
||||
|
||||
if firstField {
|
||||
i, err = e.w.Write(e.header)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
totalWritten += i
|
||||
|
||||
} else {
|
||||
i, err = e.w.Write(comma)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
totalWritten += i
|
||||
|
||||
}
|
||||
|
||||
e.w.Write(e.pair)
|
||||
|
||||
pairsLen += len(e.pair)
|
||||
firstField = false
|
||||
}
|
||||
|
||||
if firstField {
|
||||
return 0, ErrNoFields
|
||||
}
|
||||
i, err := e.w.Write(e.footer)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
totalWritten += i
|
||||
return totalWritten, nil
|
||||
}
|
||||
+191
@@ -0,0 +1,191 @@
|
||||
// Copyright 2012 The Go Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:generate go run gen.go
|
||||
|
||||
// Package publicsuffix provides a public suffix list based on data from
|
||||
// https://publicsuffix.org/
|
||||
//
|
||||
// A public suffix is one under which Internet users can directly register
|
||||
// names. It is related to, but different from, a TLD (top level domain).
|
||||
//
|
||||
// "com" is a TLD (top level domain). Top level means it has no dots.
|
||||
//
|
||||
// "com" is also a public suffix. Amazon and Google have registered different
|
||||
// siblings under that domain: "amazon.com" and "google.com".
|
||||
//
|
||||
// "au" is another TLD, again because it has no dots. But it's not "amazon.au".
|
||||
// Instead, it's "amazon.com.au".
|
||||
//
|
||||
// "com.au" isn't an actual TLD, because it's not at the top level (it has
|
||||
// dots). But it is an eTLD (effective TLD), because that's the branching point
|
||||
// for domain name registrars.
|
||||
//
|
||||
// Another name for "an eTLD" is "a public suffix". Often, what's more of
|
||||
// interest is the eTLD+1, or one more label than the public suffix. For
|
||||
// example, browsers partition read/write access to HTTP cookies according to
|
||||
// the eTLD+1. Web pages served from "amazon.com.au" can't read cookies from
|
||||
// "google.com.au", but web pages served from "maps.google.com" can share
|
||||
// cookies from "www.google.com", so you don't have to sign into Google Maps
|
||||
// separately from signing into Google Web Search. Note that all four of those
|
||||
// domains have 3 labels and 2 dots. The first two domains are each an eTLD+1,
|
||||
// the last two are not (but share the same eTLD+1: "google.com").
|
||||
//
|
||||
// All of these domains have the same eTLD+1:
|
||||
// - "www.books.amazon.co.uk"
|
||||
// - "books.amazon.co.uk"
|
||||
// - "amazon.co.uk"
|
||||
//
|
||||
// Specifically, the eTLD+1 is "amazon.co.uk", because the eTLD is "co.uk".
|
||||
//
|
||||
// There is no closed form algorithm to calculate the eTLD of a domain.
|
||||
// Instead, the calculation is data driven. This package provides a
|
||||
// pre-compiled snapshot of Mozilla's PSL (Public Suffix List) data at
|
||||
// https://publicsuffix.org/
|
||||
package publicsuffix // import "golang.org/x/net/publicsuffix"
|
||||
|
||||
// TODO: specify case sensitivity and leading/trailing dot behavior for
|
||||
// func PublicSuffix and func EffectiveTLDPlusOne.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http/cookiejar"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// List implements the cookiejar.PublicSuffixList interface by calling the
|
||||
// PublicSuffix function.
|
||||
var List cookiejar.PublicSuffixList = list{}
|
||||
|
||||
type list struct{}
|
||||
|
||||
func (list) PublicSuffix(domain string) string {
|
||||
ps, _ := PublicSuffix(domain)
|
||||
return ps
|
||||
}
|
||||
|
||||
func (list) String() string {
|
||||
return version
|
||||
}
|
||||
|
||||
// PublicSuffix returns the public suffix of the domain using a copy of the
|
||||
// publicsuffix.org database compiled into the library.
|
||||
//
|
||||
// icann is whether the public suffix is managed by the Internet Corporation
|
||||
// for Assigned Names and Numbers. If not, the public suffix is either a
|
||||
// privately managed domain (and in practice, not a top level domain) or an
|
||||
// unmanaged top level domain (and not explicitly mentioned in the
|
||||
// publicsuffix.org list). For example, "foo.org" and "foo.co.uk" are ICANN
|
||||
// domains, "foo.dyndns.org" and "foo.blogspot.co.uk" are private domains and
|
||||
// "cromulent" is an unmanaged top level domain.
|
||||
//
|
||||
// Use cases for distinguishing ICANN domains like "foo.com" from private
|
||||
// domains like "foo.appspot.com" can be found at
|
||||
// https://wiki.mozilla.org/Public_Suffix_List/Use_Cases
|
||||
func PublicSuffix(domain string) (publicSuffix string, icann bool) {
|
||||
lo, hi := uint32(0), uint32(numTLD)
|
||||
s, suffix, icannNode, wildcard := domain, len(domain), false, false
|
||||
loop:
|
||||
for {
|
||||
dot := strings.LastIndex(s, ".")
|
||||
if wildcard {
|
||||
icann = icannNode
|
||||
suffix = 1 + dot
|
||||
}
|
||||
if lo == hi {
|
||||
break
|
||||
}
|
||||
f := find(s[1+dot:], lo, hi)
|
||||
if f == notFound {
|
||||
break
|
||||
}
|
||||
|
||||
u := uint32(nodeValue(f) >> (nodesBitsTextOffset + nodesBitsTextLength))
|
||||
icannNode = u&(1<<nodesBitsICANN-1) != 0
|
||||
u >>= nodesBitsICANN
|
||||
u = children[u&(1<<nodesBitsChildren-1)]
|
||||
lo = u & (1<<childrenBitsLo - 1)
|
||||
u >>= childrenBitsLo
|
||||
hi = u & (1<<childrenBitsHi - 1)
|
||||
u >>= childrenBitsHi
|
||||
switch u & (1<<childrenBitsNodeType - 1) {
|
||||
case nodeTypeNormal:
|
||||
suffix = 1 + dot
|
||||
case nodeTypeException:
|
||||
suffix = 1 + len(s)
|
||||
break loop
|
||||
}
|
||||
u >>= childrenBitsNodeType
|
||||
wildcard = u&(1<<childrenBitsWildcard-1) != 0
|
||||
if !wildcard {
|
||||
icann = icannNode
|
||||
}
|
||||
|
||||
if dot == -1 {
|
||||
break
|
||||
}
|
||||
s = s[:dot]
|
||||
}
|
||||
if suffix == len(domain) {
|
||||
// If no rules match, the prevailing rule is "*".
|
||||
return domain[1+strings.LastIndex(domain, "."):], icann
|
||||
}
|
||||
return domain[suffix:], icann
|
||||
}
|
||||
|
||||
const notFound uint32 = 1<<32 - 1
|
||||
|
||||
// find returns the index of the node in the range [lo, hi) whose label equals
|
||||
// label, or notFound if there is no such node. The range is assumed to be in
|
||||
// strictly increasing node label order.
|
||||
func find(label string, lo, hi uint32) uint32 {
|
||||
for lo < hi {
|
||||
mid := lo + (hi-lo)/2
|
||||
s := nodeLabel(mid)
|
||||
if s < label {
|
||||
lo = mid + 1
|
||||
} else if s == label {
|
||||
return mid
|
||||
} else {
|
||||
hi = mid
|
||||
}
|
||||
}
|
||||
return notFound
|
||||
}
|
||||
|
||||
func nodeValue(i uint32) uint64 {
|
||||
off := uint64(i * (nodesBits / 8))
|
||||
return uint64(nodes[off])<<32 |
|
||||
uint64(nodes[off+1])<<24 |
|
||||
uint64(nodes[off+2])<<16 |
|
||||
uint64(nodes[off+3])<<8 |
|
||||
uint64(nodes[off+4])
|
||||
}
|
||||
|
||||
// nodeLabel returns the label for the i'th node.
|
||||
func nodeLabel(i uint32) string {
|
||||
x := nodeValue(i)
|
||||
length := x & (1<<nodesBitsTextLength - 1)
|
||||
x >>= nodesBitsTextLength
|
||||
offset := x & (1<<nodesBitsTextOffset - 1)
|
||||
return text[offset : offset+length]
|
||||
}
|
||||
|
||||
// EffectiveTLDPlusOne returns the effective top level domain plus one more
|
||||
// label. For example, the eTLD+1 for "foo.bar.golang.org" is "golang.org".
|
||||
func EffectiveTLDPlusOne(domain string) (string, error) {
|
||||
if strings.HasPrefix(domain, ".") || strings.HasSuffix(domain, ".") || strings.Contains(domain, "..") {
|
||||
return "", fmt.Errorf("publicsuffix: empty label in domain %q", domain)
|
||||
}
|
||||
|
||||
suffix, _ := PublicSuffix(domain)
|
||||
if len(domain) <= len(suffix) {
|
||||
return "", fmt.Errorf("publicsuffix: cannot derive eTLD+1 for domain %q", domain)
|
||||
}
|
||||
i := len(domain) - len(suffix) - 1
|
||||
if domain[i] != '.' {
|
||||
return "", fmt.Errorf("publicsuffix: invalid public suffix %q for domain %q", suffix, domain)
|
||||
}
|
||||
return domain[1+strings.LastIndex(domain[:i], "."):], nil
|
||||
}
|
||||
+10586
File diff suppressed because it is too large
Load Diff
Vendored
+21
-5
@@ -37,6 +37,10 @@ github.com/containerd/continuity/pathdriver
|
||||
# github.com/davecgh/go-spew v1.1.1
|
||||
## explicit
|
||||
github.com/davecgh/go-spew/spew
|
||||
# github.com/deepmap/oapi-codegen v1.8.2
|
||||
## explicit; go 1.14
|
||||
github.com/deepmap/oapi-codegen/pkg/runtime
|
||||
github.com/deepmap/oapi-codegen/pkg/types
|
||||
# github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f
|
||||
## explicit
|
||||
github.com/dgryski/go-rendezvous
|
||||
@@ -266,11 +270,22 @@ github.com/imdario/mergo
|
||||
# github.com/inconshreveable/mousetrap v1.0.1
|
||||
## explicit; go 1.18
|
||||
github.com/inconshreveable/mousetrap
|
||||
# github.com/influxdata/influxdb v1.10.0
|
||||
## explicit; go 1.18
|
||||
github.com/influxdata/influxdb/client/v2
|
||||
github.com/influxdata/influxdb/models
|
||||
github.com/influxdata/influxdb/pkg/escape
|
||||
# github.com/influxdata/influxdb-client-go/v2 v2.12.2
|
||||
## explicit; go 1.17
|
||||
github.com/influxdata/influxdb-client-go/v2
|
||||
github.com/influxdata/influxdb-client-go/v2/api
|
||||
github.com/influxdata/influxdb-client-go/v2/api/http
|
||||
github.com/influxdata/influxdb-client-go/v2/api/query
|
||||
github.com/influxdata/influxdb-client-go/v2/api/write
|
||||
github.com/influxdata/influxdb-client-go/v2/domain
|
||||
github.com/influxdata/influxdb-client-go/v2/internal/gzip
|
||||
github.com/influxdata/influxdb-client-go/v2/internal/http
|
||||
github.com/influxdata/influxdb-client-go/v2/internal/log
|
||||
github.com/influxdata/influxdb-client-go/v2/internal/write
|
||||
github.com/influxdata/influxdb-client-go/v2/log
|
||||
# github.com/influxdata/line-protocol v0.0.0-20200327222509-2487e7298839
|
||||
## explicit; go 1.13
|
||||
github.com/influxdata/line-protocol
|
||||
# github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa
|
||||
## explicit; go 1.12
|
||||
github.com/jackc/pgerrcode
|
||||
@@ -680,6 +695,7 @@ golang.org/x/net/internal/timeseries
|
||||
golang.org/x/net/ipv4
|
||||
golang.org/x/net/ipv6
|
||||
golang.org/x/net/proxy
|
||||
golang.org/x/net/publicsuffix
|
||||
golang.org/x/net/trace
|
||||
# golang.org/x/sync v0.1.0
|
||||
## explicit
|
||||
|
||||
Reference in New Issue
Block a user