Compare commits

..

36 Commits

Author SHA1 Message Date
Chung-Ting Huang 66e087a825 Release 2019.8.3 2019-08-21 11:15:36 -05:00
Michael Borkenstein 47254113ee Revert "AUTH-1941: Adds initial SSH server implementation"
This reverts commit e9c9bf3cbd.
2019-08-20 17:20:48 -05:00
Chung-Ting Huang 026f2d1cc9 Release 2019.8.2 2019-08-20 16:47:23 -05:00
Austin Cherry e9c9bf3cbd AUTH-1941: Adds initial SSH server implementation 2019-08-20 16:18:37 -05:00
Ashcon Partovi 5da2109811 STOR-519: Add db-connect, a SQL over HTTPS server 2019-08-20 13:13:29 -05:00
Chung-Ting Huang 28f6c2ed7c TUN-2163: Add GrapQLType method to Scope interface 2019-08-11 12:18:54 -05:00
Areg Harutyunyan f6c7f8a96c TUN-2152: Requests with a query in the URL are erroneously escaped 2019-08-06 17:47:00 -05:00
Areg Harutyunyan 1d73c2752e Release 2019.8.1 2019-08-06 16:40:02 -05:00
Areg Harutyunyan 5afbb3fd6d Revert "TUN-1736: Missing headers when passing an invalid path"
This reverts commit 3c93d9b300.
2019-08-06 16:35:41 -05:00
Chung-Ting Huang fd4ab314dc TUN-2111: Implement custom serialization logic for FallibleConfig and OriginConfig 2019-08-06 10:56:16 -05:00
Michael Borkenstein 993a9bc4b9 Release 2019.8.0 2019-08-06 09:49:49 -05:00
Michael Borkenstein 8d1ea7202a Merge branch 'mike/AUTH-1972-delete-token-lockfile' of ssh://bitbucket.cfdata.org:7999/tun/cloudflared 2019-08-06 08:12:02 -05:00
Michael Borkenstein 9adbab96af AUTH-1972: Deletes token lock file if backoff retry attempts exceeded and intercepts signals until lock is released 2019-08-06 08:07:48 -05:00
Chung-Ting Huang bdd70e798a TUN-2110: Implement custom deserialization logic for OriginConfig 2019-08-05 19:28:51 -05:00
Adam Chalmers 5feba7e3a9 TUN-2147: Implemented ScopeUnmarshaler 2019-08-05 18:14:32 -05:00
Adam Chalmers d3d9460f64 TUN-2125: Add PostgresType() to Scope 2019-08-02 10:21:09 -05:00
Nick Vollmar 74f3a55c57 TUN-2117: read group/system-name from CLI, send it to edge 2019-08-01 22:04:05 +00:00
Areg Harutyunyan 3c93d9b300 TUN-1736: Missing headers when passing an invalid path 2019-07-31 16:56:29 +00:00
Nick Vollmar 3c0ea0db65 TUN-1956: Go 1.12 update 2019-07-12 22:09:16 +00:00
Austin Cherry 7c03e2cbd2 Release 2019.7.0 2019-07-11 10:22:37 -05:00
Austin Cherry 8f25704a90 AUTH-1736: Better handling of token revocation
We removed all token validation from cloudflared and now rely on
the edge to do the validation. This is better because the edge is
the only thing that fully knows about token revocation. So if a user
logs out or the application revokes all it's tokens cloudflared will
now handle that process instead of barfing on it.

When we go to fetch a token we will check for the existence of a
lock file. If the lock file exists, we stop and poll every half
second to see if the lock is still there. Once the lock file is
removed, it will restart the function to (hopefully) go pick up
the valid token that was just created.
2019-07-10 21:35:46 +00:00
Areg Harutyunyan 583bad4972 Merge branch 'master' of ssh://stash.cfops.it:7999/tun/cloudflared 2019-07-10 11:46:13 -05:00
Areg Harutyunyan b5f340d64c Merge branch 'master' of github.com:cloudflare/cloudflared 2019-07-10 11:45:57 -05:00
Frank Denis 9cd0b7048a DoH: change the media type to application/dns-message
The media type for DoH was changed from `application/dns-udpwireformat` to `application/dns-message` in May 2018.

Ref: RFC8484 section 6.
2019-07-03 14:13:28 -05:00
Chung-Ting Huang 4858ce79d0 TUN-1977: Validate OriginConfig has valid URL, and use scheme to determine if a HTTPOriginService is expecting HTTP or Unix 2019-07-01 15:31:58 -05:00
Chung-Ting Huang 4090049fff TUN-1981: Write response header & body on proxy error to notify eyeballs of failure category 2019-07-01 20:20:59 +00:00
Adam Chalmers e255a7da26 TUN-2005: Upgrade logrus 2019-06-26 12:25:58 -05:00
Chung-Ting Huang c2a3ac3991 TUN-1982: Load custom origin CA when OriginCAPool is specified 2019-06-21 11:44:26 -05:00
Chung-Ting Huang 2fa09e1cc6 TUN-1976: Pass tunnel hostname through header 2019-06-21 10:43:06 -05:00
Chung-Ting Huang 0a742feb98 TUN-1885: Reconfigure cloudflared on receiving new ClientConfig 2019-06-20 19:07:59 -05:00
Chung-Ting Huang 80a15547e3 TUN-1961: Create EdgeConnectionManager to maintain outbound connections to the edge 2019-06-18 16:37:38 -05:00
Chung-Ting Huang d26a8c5d44 TUN-1893: Proxy requests to the origin based on tunnel hostname 2019-06-13 14:46:22 -05:00
Chung-Ting Huang ca619a97bc TUN-1952: Group ClientConfig fields by the component that uses the config, and return the part of the config that failed to be applied 2019-06-13 14:39:51 -05:00
Areg Harutyunyan 25a04e0c69 Merge branch 'master' of ssh://stash.cfops.it:7999/tun/cloudflared 2019-06-06 13:53:03 +04:00
Johan Bergström d3f9aa2ae7 Build a docker container
Use debian/distroless (glibc) as a base which in general gives
slightly higher performance than alpine (musl) while trading
off container size (about 10mb).

The build phase makes the assumption that any capnproto-files
are pre-processed before building in-docker.
2019-06-06 13:50:13 +04:00
Chung-Ting Huang d32fb8e82c TUN-1913: Define OriginService for each type of origin 2019-06-04 17:02:34 -05:00
528 changed files with 371755 additions and 6298 deletions
+12
View File
@@ -0,0 +1,12 @@
FROM golang:1.12 as builder
WORKDIR /go/src/github.com/cloudflare/cloudflared/
RUN apt-get update && apt-get install -y --no-install-recommends upx
# Run after `apt-get update` to improve rebuild scenarios
COPY . .
RUN make cloudflared
RUN upx --no-progress cloudflared
FROM gcr.io/distroless/base
COPY --from=builder /go/src/github.com/cloudflare/cloudflared/cloudflared /usr/local/bin/
ENTRYPOINT ["cloudflared", "--no-autoupdate"]
CMD ["version"]
Generated
+214 -43
View File
@@ -1,6 +1,14 @@
# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.
[[projects]]
digest = "1:3d3a509c5ba327e8573bb57f9da8430c63a46a06886eb1d2ffc8af4e76f31c72"
name = "cloud.google.com/go"
packages = ["civil"]
pruneopts = "UT"
revision = "cdaaf98f9226c39dc162b8e55083b2fbc67b4674"
version = "v0.43.0"
[[projects]]
digest = "1:9f3b30d9f8e0d7040f729b82dcbc8f0dead820a133b3147ce355fc451f32d761"
name = "github.com/BurntSushi/toml"
@@ -9,6 +17,14 @@
revision = "3012a1dbe2e4bd1391d42b32f0577cb7bbc7f005"
version = "v0.3.1"
[[projects]]
digest = "1:c84a587136cb69cecc11f3dbe9f9001444044c0dba74997b07f7e4c150b07cda"
name = "github.com/DATA-DOG/go-sqlmock"
packages = ["."]
pruneopts = "UT"
revision = "3f9954f6f6697845b082ca57995849ddf614f450"
version = "v1.3.3"
[[projects]]
digest = "1:d6afaeed1502aa28e80a4ed0981d570ad91b2579193404256ce672ed0a609e0d"
name = "github.com/beorn7/perks"
@@ -39,6 +55,14 @@
pruneopts = "UT"
revision = "333127dbecfcc23a8db7d9a4f52785d23aff44a1"
[[projects]]
branch = "master"
digest = "1:05756b73bf7eae03d6f6c5f0ae83de42ce296e7d83336563dd9ce2072106d5e6"
name = "github.com/cloudflare/golz4"
packages = ["."]
pruneopts = "UT"
revision = "ef862a3cdc58a6f1fee4e3af3d44fbe279194cde"
[[projects]]
digest = "1:3f9506ee991cdee1f05bf0cd3e34b5cd922dc00d6a950fb4beb4e07ab1c4d3d1"
name = "github.com/coredns/coredns"
@@ -73,17 +97,12 @@
version = "v1.2.0"
[[projects]]
digest = "1:6a503e232df389d94ebb97dfb22d4ae463b6e2f351660613e11d9e42f57ab6df"
branch = "v2"
digest = "1:ca6ce07b0d28c6044411b9c966cd845233e27c2ff91d08a3a869631f329c9918"
name = "github.com/coreos/go-oidc"
packages = [
"http",
"jose",
"key",
"oauth2",
"oidc",
]
packages = ["."]
pruneopts = "UT"
revision = "a93f71fdfe73d2c0f5413c0565eea0af6523a6df"
revision = "274971e2c94cd6fb17614979e2899edbc81af146"
[[projects]]
digest = "1:1da3a221f0bc090792d3a2a080ff09008427c0e0f0533a4ed6abd8994421da73"
@@ -93,18 +112,6 @@
revision = "95778dfbb74eb7e4dbaf43bf7d71809650ef8076"
version = "v19"
[[projects]]
digest = "1:6fda0d7f5e52b081e075775b1ecebf1ea0c923e7be33604ed0225ae078e701b5"
name = "github.com/coreos/pkg"
packages = [
"health",
"httputil",
"timeutil",
]
pruneopts = "UT"
revision = "97fdf19511ea361ae1c100dd393cc47f8dcfa1e1"
version = "v4"
[[projects]]
digest = "1:ffe9824d294da03b391f44e1ae8281281b4afc1bdaa9588c9097785e3af10cec"
name = "github.com/davecgh/go-spew"
@@ -115,11 +122,15 @@
[[projects]]
branch = "master"
digest = "1:c013ffc6e15f9f898078f9d38441c68b228aa7b899659452170250ccb27f5f1e"
name = "github.com/elgs/gosqljson"
packages = ["."]
digest = "1:27cbe99893a40975358d20736d80adc95ef395d3bf5923f802f3de259533c94a"
name = "github.com/denisenkom/go-mssqldb"
packages = [
".",
"internal/cp",
"internal/querytext",
]
pruneopts = "UT"
revision = "027aa4915315a0b2825c0f025cea347829b974fa"
revision = "11b2859924c1e3c3da056fd4e0976b93883f3545"
[[projects]]
digest = "1:d4268b2a09b1f736633577c4ac93f2a5356c73742fff5344e2451aeec60a7ad0"
@@ -143,6 +154,14 @@
pruneopts = "UT"
revision = "75cf19382434e82df4dd84953f566b8ad23d6e9e"
[[projects]]
digest = "1:296cda2c4a6a7a54964d7b3b0815e2aed215da75217b72033060ffb1c4e4b6ae"
name = "github.com/felixge/httpsnoop"
packages = ["."]
pruneopts = "UT"
revision = "eadd4fad6aac69ae62379194fe0219f3dbc80fd3"
version = "v1.0.0"
[[projects]]
branch = "master"
digest = "1:50a46ab1d5edbbdd55125b4d37f1bf503d0807c26461f9ad7b358d6006641d09"
@@ -158,6 +177,14 @@
pruneopts = "UT"
revision = "ed7bcb39ff10f39ab08e317ce16df282845852fa"
[[projects]]
digest = "1:ec6f9bf5e274c833c911923c9193867f3f18788c461f76f05f62bb1510e0ae65"
name = "github.com/go-sql-driver/mysql"
packages = ["."]
pruneopts = "UT"
revision = "72cd26f257d44c1114970e19afddcd812016007e"
version = "v1.4.1"
[[projects]]
branch = "master"
digest = "1:3e6afc3ed8a72949aa735c00fddc23427dc9384ccfd51cf0d91a412e668da632"
@@ -166,6 +193,14 @@
pruneopts = "UT"
revision = "604e922904d35e97f98a774db7881f049cd8d970"
[[projects]]
branch = "master"
digest = "1:01745416ed734d0dbd8f2a25344536e399ee60319654873da12b107e9e3cb309"
name = "github.com/golang/gddo"
packages = ["httputil/header"]
pruneopts = "UT"
revision = "af0f2af80721261f4d211b2c9563f7b46b2aab06"
[[projects]]
digest = "1:239c4c7fd2159585454003d9be7207167970194216193a8a210b8d29576f19c9"
name = "github.com/golang/protobuf"
@@ -213,15 +248,45 @@
revision = "8e809c8a86450a29b90dcc9efbf062d0fe6d9746"
[[projects]]
digest = "1:75ab90ae3f5d876167e60f493beadfe66f0ed861a710f283fb06c86437a09538"
name = "github.com/jonboulle/clockwork"
packages = ["."]
digest = "1:6c41d4f998a03b6604227ccad36edaed6126c397e5d78709ef4814a1145a6757"
name = "github.com/jmoiron/sqlx"
packages = [
".",
"reflectx",
]
pruneopts = "UT"
revision = "2eee05ed794112d45db504eb05aa693efd2b8b09"
version = "v0.1.0"
revision = "d161d7a76b5661016ad0b085869f77fd410f3e6a"
version = "v1.2.0"
[[projects]]
digest = "1:bc1c0be40c67b6b4aee09d7508d5a2a52c1c116b1fa43806dad2b0d6b4d4003b"
digest = "1:31e761d97c76151dde79e9d28964a812c46efc5baee4085b86f68f0c654450de"
name = "github.com/konsorten/go-windows-terminal-sequences"
packages = ["."]
pruneopts = "UT"
revision = "f55edac94c9bbba5d6182a4be46d86a2c9b5b50e"
version = "v1.0.2"
[[projects]]
digest = "1:177a18252d40bf39a867876456854717780b0b1212fdae231d31e93b6f885d80"
name = "github.com/kshvakov/clickhouse"
packages = [
".",
"lib/binary",
"lib/cityhash102",
"lib/column",
"lib/data",
"lib/leakypool",
"lib/lz4",
"lib/protocol",
"lib/types",
"lib/writebuffer",
]
pruneopts = "UT"
revision = "43e176a8d165376662f4d67203c71cc8a027c86b"
version = "v1.3.9"
[[projects]]
digest = "1:12cb143f2148bf54bcd9fe622abac17325e85eeb1d84b8ec6caf1c80232108fd"
name = "github.com/lib/pq"
packages = [
".",
@@ -229,8 +294,8 @@
"scram",
]
pruneopts = "UT"
revision = "51e2106eed1cea199c802d2a49e91e2491b02056"
version = "v1.1.0"
revision = "3427c32cb71afc948325f299f040e53c1dd78979"
version = "v1.2.0"
[[projects]]
digest = "1:2fa7b0155cd54479a755c629de26f888a918e13f8857a2c442205d825368e084"
@@ -248,6 +313,14 @@
revision = "c2a7a6ca930a4cd0bc33a3f298eb71960732a3a7"
version = "v0.0.7"
[[projects]]
digest = "1:79e87abf06b873987dee86598950f5b51732ac454d5a5cab6445a14330e6c9e3"
name = "github.com/mattn/go-sqlite3"
packages = ["."]
pruneopts = "UT"
revision = "b612a2feea6aa87c6d052d9086572551df06497e"
version = "v1.11.0"
[[projects]]
digest = "1:ff5ebae34cfbf047d505ee150de27e60570e8c394b3b8fdbb720ff6ac71985fc"
name = "github.com/matttproud/golang_protobuf_extensions"
@@ -283,6 +356,22 @@
revision = "af06845cf3004701891bf4fdb884bfe4920b3727"
version = "v1.1.0"
[[projects]]
digest = "1:dd3d50b46f8d3c88a8d3986f2e5344db582c457469a7cb3a0dda953c669cdf9a"
name = "github.com/mitchellh/go-server-timing"
packages = ["."]
pruneopts = "UT"
revision = "1ca01db910d57d64ec1c68f27105614b77558e28"
version = "v1.0.0"
[[projects]]
digest = "1:53bc4cd4914cd7cd52139990d5170d6dc99067ae31c56530621b18b35fc30318"
name = "github.com/mitchellh/mapstructure"
packages = ["."]
pruneopts = "UT"
revision = "3536a929edddb9a5b34bd6861dc4a9647cb459fe"
version = "v1.1.2"
[[projects]]
digest = "1:11e62d6050198055e6cd87ed57e5d8c669e84f839c16e16f192374d913d1a70d"
name = "github.com/opentracing/opentracing-go"
@@ -311,6 +400,17 @@
revision = "792786c7400a136282c1664665ae0a8db921c6c2"
version = "v1.0.0"
[[projects]]
branch = "master"
digest = "1:bd9efe4e0b0f768302a1e2f0c22458149278de533e521206e5ddc71848c269a0"
name = "github.com/pquerna/cachecontrol"
packages = [
".",
"cacheobject",
]
pruneopts = "UT"
revision = "1555304b9b35fdd2b425bccf1a5613677705e7d0"
[[projects]]
digest = "1:c968b29db5d68ec97de404b6d058d5937fa015a141b3b4f7a0d87d5f8226f04c"
name = "github.com/prometheus/client_golang"
@@ -359,24 +459,35 @@
version = "v2.4"
[[projects]]
digest = "1:5f2aaa360f48d1711795bd88c7e45a38f86cf81e4bc01453d20983baa67e2d51"
digest = "1:04457f9f6f3ffc5fea48e71d62f2ca256637dee0a04d710288e27e05c8b41976"
name = "github.com/sirupsen/logrus"
packages = ["."]
pruneopts = "UT"
revision = "f006c2ac4710855cf0f916dd6b77acf6b048dc6e"
version = "v1.0.3"
revision = "839c75faf7f98a33d445d181f3018b5c3409a45e"
version = "v1.4.2"
[[projects]]
digest = "1:f85e109eda8f6080877185d1c39e98dd8795e1780c08beca28304b87fd855a1c"
digest = "1:7e8d267900c7fa7f35129a2a37596e38ed0f11ca746d6d9ba727980ee138f9f6"
name = "github.com/stretchr/testify"
packages = ["assert"]
packages = [
"assert",
"require",
]
pruneopts = "UT"
revision = "12b6f73e6084dad08a7c6e575284b177ecafbc71"
version = "v1.2.1"
[[projects]]
branch = "master"
digest = "1:a84d5ec8b40a827962ea250f2cf03434138ccae9d83fcac12fb49b70c70b80cc"
digest = "1:cac263b8eb2e9ad2327b70e03be1de1227040bfb3be26b49c4b56cbed568afc7"
name = "github.com/xo/dburl"
packages = ["."]
pruneopts = "UT"
revision = "98997a05b24fc069c79fa13b05ccf376c22a326c"
[[projects]]
branch = "master"
digest = "1:1a9391a80fe548ffcf90aee8ae35bfb0b188198b57a3fe2ef5bb099ba4b4f610"
name = "golang.org/x/crypto"
packages = [
"curve25519",
@@ -384,8 +495,10 @@
"ed25519/internal/edwards25519",
"internal/chacha20",
"internal/subtle",
"md4",
"nacl/box",
"nacl/secretbox",
"pbkdf2",
"poly1305",
"salsa20/salsa",
"ssh",
@@ -396,11 +509,12 @@
[[projects]]
branch = "master"
digest = "1:52d140f7ab52e491cc1cbc93e6637aa5e9a7f3beae7545d675b02e52ca9d7290"
digest = "1:6b3dbfffba73ea0d8131ecb5738fe87f6addc4c7b73154b9d58e2693ad6a6256"
name = "golang.org/x/net"
packages = [
"bpf",
"context",
"context/ctxhttp",
"http/httpguts",
"http2",
"http2/hpack",
@@ -416,6 +530,17 @@
pruneopts = "UT"
revision = "1da14a5a36f220ea3f03470682b737b1dfd5de22"
[[projects]]
branch = "master"
digest = "1:8d1c112fb1679fa097e9a9255a786ee47383fa2549a3da71bcb1334a693ebcfe"
name = "golang.org/x/oauth2"
packages = [
".",
"internal",
]
pruneopts = "UT"
revision = "0f29369cfe4552d0e4bcddc57cc75f4d7e672a33"
[[projects]]
digest = "1:39ebcc2b11457b703ae9ee2e8cca0f68df21969c6102cb3b705f76cca0ea0239"
name = "golang.org/x/sync"
@@ -462,6 +587,23 @@
revision = "f21a4dfb5e38f5895301dc265a8def02365cc3d0"
version = "v0.3.0"
[[projects]]
digest = "1:7e8b9c5ae49011b12ae8473834ac1a7bb8ac029ba201270c723e4c280c9e4855"
name = "google.golang.org/appengine"
packages = [
"cloudsql",
"internal",
"internal/base",
"internal/datastore",
"internal/log",
"internal/remote_api",
"internal/urlfetch",
"urlfetch",
]
pruneopts = "UT"
revision = "b2f4a3cf3c67576a2ee09e1fe62656a5086ce880"
version = "v1.6.1"
[[projects]]
branch = "master"
digest = "1:c3076e7defee87de1236f1814beb588f40a75544c60121e6eb38b3b3721783e2"
@@ -511,6 +653,26 @@
revision = "236199dd5f8031d698fb64091194aecd1c3895b2"
version = "v1.20.0"
[[projects]]
branch = "v1"
digest = "1:6f70106e7bc1c803e8a0a4519e09c12d154771acfa2559206e97b033bbd1dd38"
name = "gopkg.in/coreos/go-oidc.v1"
packages = ["jose"]
pruneopts = "UT"
revision = "e860bd55bfa7d7cb35d30d26a167982584f616b0"
[[projects]]
digest = "1:8c05919580be8a5be668709d7e5a69d5cd19b8ee9f23d62ce5b10d3457bf6a13"
name = "gopkg.in/square/go-jose.v2"
packages = [
".",
"cipher",
"json",
]
pruneopts = "UT"
revision = "730df5f748271903322feb182be83b43ebbbe27d"
version = "v2.3.1"
[[projects]]
branch = "altsrc-parse-durations"
digest = "1:0370b1bceda03dbfade3abbde639a43f1113bab711ec760452e5c0dcc0c14787"
@@ -559,6 +721,7 @@
analyzer-name = "dep"
analyzer-version = 1
input-imports = [
"github.com/DATA-DOG/go-sqlmock",
"github.com/cloudflare/brotli-go",
"github.com/cloudflare/golibs/lrucache",
"github.com/coredns/coredns/core/dnsserver",
@@ -568,27 +731,34 @@
"github.com/coredns/coredns/plugin/pkg/dnstest",
"github.com/coredns/coredns/plugin/pkg/rcode",
"github.com/coredns/coredns/request",
"github.com/coreos/go-oidc/jose",
"github.com/coreos/go-oidc/oidc",
"github.com/coreos/go-oidc",
"github.com/coreos/go-systemd/daemon",
"github.com/elgs/gosqljson",
"github.com/denisenkom/go-mssqldb",
"github.com/equinox-io/equinox",
"github.com/facebookgo/grace/gracenet",
"github.com/getsentry/raven-go",
"github.com/go-sql-driver/mysql",
"github.com/golang-collections/collections/queue",
"github.com/google/uuid",
"github.com/gorilla/mux",
"github.com/gorilla/websocket",
"github.com/jmoiron/sqlx",
"github.com/kshvakov/clickhouse",
"github.com/lib/pq",
"github.com/mattn/go-colorable",
"github.com/mattn/go-sqlite3",
"github.com/miekg/dns",
"github.com/mitchellh/go-homedir",
"github.com/mitchellh/go-server-timing",
"github.com/mitchellh/mapstructure",
"github.com/pkg/errors",
"github.com/prometheus/client_golang/prometheus",
"github.com/prometheus/client_golang/prometheus/promhttp",
"github.com/rifflock/lfshook",
"github.com/sirupsen/logrus",
"github.com/stretchr/testify/assert",
"github.com/stretchr/testify/require",
"github.com/xo/dburl",
"golang.org/x/crypto/nacl/box",
"golang.org/x/crypto/ssh",
"golang.org/x/crypto/ssh/terminal",
@@ -603,6 +773,7 @@
"golang.org/x/sys/windows/svc",
"golang.org/x/sys/windows/svc/eventlog",
"golang.org/x/sys/windows/svc/mgr",
"gopkg.in/coreos/go-oidc.v1/jose",
"gopkg.in/urfave/cli.v2",
"gopkg.in/urfave/cli.v2/altsrc",
"zombiezen.com/go/capnproto2",
+37 -7
View File
@@ -6,8 +6,6 @@
name = "github.com/cloudflare/brotli-go"
unused-packages = false
[[constraint]]
name = "github.com/facebookgo/grace"
revision = "75cf19382434e82df4dd84953f566b8ad23d6e9e"
@@ -26,7 +24,7 @@
[[constraint]]
name = "github.com/sirupsen/logrus"
version = "=1.0.3"
version = "=1.4.2"
[[constraint]]
name = "github.com/stretchr/testify"
@@ -70,10 +68,6 @@
name = "github.com/mholt/caddy"
revision = "d3b731e9255b72d4571a5aac125634cf1b6031dc"
[[constraint]]
name = "github.com/coreos/go-oidc"
revision = "a93f71fdfe73d2c0f5413c0565eea0af6523a6df"
[[constraint]]
name = "golang.org/x/crypto"
branch = "master" # master required by github.com/miekg/dns
@@ -85,3 +79,39 @@
[[constraint]]
name = "github.com/google/uuid"
version = "=1.1.1"
[[constraint]]
name = "github.com/mitchellh/mapstructure"
version = "1.1.2"
[[constraint]]
name = "github.com/jmoiron/sqlx"
version = "1.2.0"
[[constraint]]
name = "github.com/go-sql-driver/mysql"
version = "1.4.1"
[[constraint]]
name = "github.com/mattn/go-sqlite3"
version = "1.11.0"
[[constraint]]
branch = "master"
name = "github.com/denisenkom/go-mssqldb"
[[constraint]]
name = "github.com/kshvakov/clickhouse"
version = "1.3.9"
[[constraint]]
name = "github.com/DATA-DOG/go-sqlmock"
version = "1.3.3"
[[constraint]]
branch = "master"
name = "github.com/xo/dburl"
[[constraint]]
branch = "v2"
name = "github.com/coreos/go-oidc"
+4
View File
@@ -31,6 +31,10 @@ clean:
cloudflared: tunnel-deps
go build -v $(VERSION_FLAGS) $(IMPORT_PATH)/cmd/cloudflared
.PHONY: container
container:
docker build -t cloudflare/cloudflared:"$(VERSION)" .
.PHONY: test
test: vet
go test -v -race $(VERSION_FLAGS) ./...
+36
View File
@@ -1,3 +1,39 @@
2019.8.3
- 2019-08-20 STOR-519: Add db-connect, a SQL over HTTPS server
- 2019-08-20 Release 2019.8.2
- 2019-08-20 Revert "AUTH-1941: Adds initial SSH server implementation"
- 2019-08-11 TUN-2163: Add GrapQLType method to Scope interface
- 2019-08-06 TUN-2152: Requests with a query in the URL are erroneously escaped
- 2019-07-18 AUTH-1941: Adds initial SSH server implementation
2019.8.1
- 2019-08-05 TUN-2111: Implement custom serialization logic for FallibleConfig and OriginConfig
- 2019-08-06 Revert "TUN-1736: Missing headers when passing an invalid path"
2019.8.0
- 2019-07-11 TUN-1956: Go 1.12 update
- 2019-07-24 TUN-1736: Missing headers when passing an invalid path
- 2019-07-30 TUN-2117: read group/system-name from CLI, send it to edge
- 2019-08-02 TUN-2125: Add PostgresType() to Scope
- 2019-08-05 TUN-2147: Implemented ScopeUnmarshaler
- 2019-07-31 TUN-2110: Implement custom deserialization logic for OriginConfig
- 2019-07-31 AUTH-1972: Deletes token lock file if backoff retry attempts exceeded and intercepts signals until lock is released
2019.7.0
- 2019-05-28 TUN-1913: Define OriginService for each type of origin
- 2019-04-29 Build a docker container
- 2019-06-12 TUN-1952: Group ClientConfig fields by the component that uses the config, and return the part of the config that failed to be applied
- 2019-06-05 TUN-1893: Proxy requests to the origin based on tunnel hostname
- 2019-06-17 TUN-1961: Create EdgeConnectionManager to maintain outbound connections to the edge
- 2019-06-18 TUN-1885: Reconfigure cloudflared on receiving new ClientConfig
- 2019-06-19 TUN-1976: Pass tunnel hostname through header
- 2019-06-20 TUN-1982: Load custom origin CA when OriginCAPool is specified
- 2019-06-26 TUN-2005: Upgrade logrus
- 2019-06-20 TUN-1981: Write response header & body on proxy error to notify eyeballs of failure category
- 2019-06-20 TUN-1977: Validate OriginConfig has valid URL, and use scheme to determine if a HTTPOriginService is expecting HTTP or Unix
- 2019-06-13 DoH: change the media type to application/dns-message
- 2019-06-26 AUTH-1736: Better handling of token revocation
2019.6.0
- 2019-05-17 TUN-1828: Update declarative tunnel config struct
- 2019-05-29 Handle exit code on err
+85 -23
View File
@@ -4,7 +4,6 @@
package carrier
import (
"errors"
"io"
"net"
"net/http"
@@ -12,7 +11,8 @@ import (
"strings"
"github.com/cloudflare/cloudflared/cmd/cloudflared/token"
"github.com/cloudflare/cloudflared/websocket"
cloudflaredWebsocket "github.com/cloudflare/cloudflared/websocket"
"github.com/gorilla/websocket"
"github.com/sirupsen/logrus"
)
@@ -37,6 +37,13 @@ func (c *StdinoutStream) Write(p []byte) (int, error) {
return os.Stdout.Write(p)
}
// Helper to allow defering the response close with a check that the resp is not nil
func closeRespBody(resp *http.Response) {
if resp != nil {
resp.Body.Close()
}
}
// StartClient will copy the data from stdin/stdout over a WebSocket connection
// to the edge (originURL)
func StartClient(logger *logrus.Logger, stream io.ReadWriter, options *StartOptions) error {
@@ -90,7 +97,7 @@ func serveStream(logger *logrus.Logger, conn io.ReadWriter, options *StartOption
}
defer wsConn.Close()
websocket.Stream(wsConn, conn)
cloudflaredWebsocket.Stream(wsConn, conn)
return nil
}
@@ -98,28 +105,17 @@ func serveStream(logger *logrus.Logger, conn io.ReadWriter, options *StartOption
// createWebsocketStream will create a WebSocket connection to stream data over
// It also handles redirects from Access and will present that flow if
// the token is not present on the request
func createWebsocketStream(options *StartOptions) (*websocket.Conn, error) {
func createWebsocketStream(options *StartOptions) (*cloudflaredWebsocket.Conn, error) {
req, err := http.NewRequest(http.MethodGet, options.OriginURL, nil)
if err != nil {
return nil, err
}
req.Header = options.Headers
wsConn, resp, err := websocket.ClientConnect(req, nil)
if err != nil && resp != nil && resp.StatusCode > 300 {
location, err := resp.Location()
if err != nil {
return nil, err
}
if !strings.Contains(location.String(), "cdn-cgi/access/login") {
return nil, errors.New("not an Access redirect")
}
req, err := buildAccessRequest(options.OriginURL)
if err != nil {
return nil, err
}
wsConn, _, err = websocket.ClientConnect(req, nil)
wsConn, resp, err := cloudflaredWebsocket.ClientConnect(req, nil)
defer closeRespBody(resp)
if err != nil && isAccessResponse(resp) {
wsConn, err = createAccessAuthenticatedStream(options)
if err != nil {
return nil, err
}
@@ -127,12 +123,72 @@ func createWebsocketStream(options *StartOptions) (*websocket.Conn, error) {
return nil, err
}
return &websocket.Conn{Conn: wsConn}, nil
return &cloudflaredWebsocket.Conn{Conn: wsConn}, nil
}
// isAccessResponse checks the http Response to see if the url location
// contains the Access structure.
func isAccessResponse(resp *http.Response) bool {
if resp == nil || resp.StatusCode <= 300 {
return false
}
location, err := resp.Location()
if err != nil || location == nil {
return false
}
if strings.HasPrefix(location.Path, "/cdn-cgi/access/login") {
return true
}
return false
}
// createAccessAuthenticatedStream will try load a token from storage and make
// a connection with the token set on the request. If it still get redirect,
// this probably means the token in storage is invalid (expired/revoked). If that
// happens it deletes the token and runs the connection again, so the user can
// login again and generate a new one.
func createAccessAuthenticatedStream(options *StartOptions) (*websocket.Conn, error) {
wsConn, resp, err := createAccessWebSocketStream(options)
defer closeRespBody(resp)
if err == nil {
return wsConn, nil
}
if !isAccessResponse(resp) {
return nil, err
}
// Access Token is invalid for some reason. Go through regen flow
originReq, err := http.NewRequest(http.MethodGet, options.OriginURL, nil)
if err != nil {
return nil, err
}
if err := token.RemoveTokenIfExists(originReq.URL); err != nil {
return nil, err
}
wsConn, resp, err = createAccessWebSocketStream(options)
defer closeRespBody(resp)
if err != nil {
return nil, err
}
return wsConn, nil
}
// createAccessWebSocketStream builds an Access request and makes a connection
func createAccessWebSocketStream(options *StartOptions) (*websocket.Conn, *http.Response, error) {
req, err := buildAccessRequest(options)
if err != nil {
return nil, nil, err
}
return cloudflaredWebsocket.ClientConnect(req, nil)
}
// buildAccessRequest builds an HTTP request with the Access token set
func buildAccessRequest(originURL string) (*http.Request, error) {
req, err := http.NewRequest(http.MethodGet, originURL, nil)
func buildAccessRequest(options *StartOptions) (*http.Request, error) {
req, err := http.NewRequest(http.MethodGet, options.OriginURL, nil)
if err != nil {
return nil, err
}
@@ -144,11 +200,17 @@ func buildAccessRequest(originURL string) (*http.Request, error) {
// We need to create a new request as FetchToken will modify req (boo mutable)
// as it has to follow redirect on the API and such, so here we init a new one
originRequest, err := http.NewRequest(http.MethodGet, originURL, nil)
originRequest, err := http.NewRequest(http.MethodGet, options.OriginURL, nil)
if err != nil {
return nil, err
}
originRequest.Header.Set("cf-access-token", token)
for k, v := range options.Headers {
if len(v) >= 1 {
originRequest.Header.Set(k, v[0])
}
}
return originRequest, nil
}
+25
View File
@@ -91,6 +91,31 @@ func TestStartServer(t *testing.T) {
assert.Equal(t, string(readBuffer), message)
}
func TestIsAccessResponse(t *testing.T) {
validLocationHeader := http.Header{}
validLocationHeader.Add("location", "https://test.cloudflareaccess.com/cdn-cgi/access/login/blahblah")
invalidLocationHeader := http.Header{}
invalidLocationHeader.Add("location", "https://google.com")
testCases := []struct {
Description string
In *http.Response
ExpectedOut bool
}{
{"nil response", nil, false},
{"redirect with no location", &http.Response{StatusCode: http.StatusPermanentRedirect}, false},
{"200 ok", &http.Response{StatusCode: http.StatusOK}, false},
{"redirect with location", &http.Response{StatusCode: http.StatusPermanentRedirect, Header: validLocationHeader}, true},
{"redirect with invalid location", &http.Response{StatusCode: http.StatusPermanentRedirect, Header: invalidLocationHeader}, false},
}
for i, tc := range testCases {
if isAccessResponse(tc.In) != tc.ExpectedOut {
t.Fatalf("Failed case %d -- %s", i, tc.Description)
}
}
}
func newTestWebSocketServer() *httptest.Server {
upgrader := ws.Upgrader{
ReadBufferSize: 1024,
+12 -1
View File
@@ -1,4 +1,4 @@
pinned_go: &pinned_go go=1.11.5-1
pinned_go: &pinned_go go=1.12.7-1
build_dir: &build_dir /cfsetup_build/src/github.com/cloudflare/cloudflared/
default-flavor: stretch
stretch: &stretch
@@ -93,3 +93,14 @@ stretch: &stretch
- make test
jessie: *stretch
# cfsetup compose
default-stack: test_dbconnect
test_dbconnect:
compose:
up-args:
- --renew-anon-volumes
- --abort-on-container-exit
- --exit-code-from=cloudflared
files:
- dbconnect/integration_test/dbconnect.yaml
+28
View File
@@ -0,0 +1,28 @@
package buildinfo
import (
"runtime"
"github.com/sirupsen/logrus"
)
type BuildInfo struct {
GoOS string `json:"go_os"`
GoVersion string `json:"go_version"`
GoArch string `json:"go_arch"`
CloudflaredVersion string `json:"cloudflared_version"`
}
func GetBuildInfo(cloudflaredVersion string) *BuildInfo {
return &BuildInfo{
GoOS: runtime.GOOS,
GoVersion: runtime.Version(),
GoArch: runtime.GOARCH,
CloudflaredVersion: cloudflaredVersion,
}
}
func (bi *BuildInfo) Log(logger *logrus.Logger) {
logger.Infof("Version %s", bi.CloudflaredVersion)
logger.Infof("GOOS: %s, GOVersion: %s, GoArch: %s", bi.GoOS, bi.GoVersion, bi.GoArch)
}
+124 -13
View File
@@ -1,15 +1,20 @@
package token
import (
"context"
"fmt"
"io/ioutil"
"net/url"
"time"
"os"
"os/signal"
"syscall"
"github.com/cloudflare/cloudflared/cmd/cloudflared/config"
"github.com/cloudflare/cloudflared/cmd/cloudflared/path"
"github.com/cloudflare/cloudflared/cmd/cloudflared/transfer"
"github.com/cloudflare/cloudflared/log"
"github.com/coreos/go-oidc/jose"
"github.com/coreos/go-oidc/oidc"
"github.com/cloudflare/cloudflared/origin"
"gopkg.in/coreos/go-oidc.v1/jose"
)
const (
@@ -18,6 +23,95 @@ const (
var logger = log.CreateLogger()
type lock struct {
lockFilePath string
backoff *origin.BackoffHandler
sigHandler *signalHandler
}
type signalHandler struct {
sigChannel chan os.Signal
signals []os.Signal
}
func (s *signalHandler) register(handler func()) {
s.sigChannel = make(chan os.Signal, 1)
signal.Notify(s.sigChannel, s.signals...)
go func(s *signalHandler) {
for range s.sigChannel {
handler()
}
}(s)
}
func (s *signalHandler) deregister() {
signal.Stop(s.sigChannel)
close(s.sigChannel)
}
func errDeleteTokenFailed(lockFilePath string) error {
return fmt.Errorf("failed to acquire a new Access token. Please try to delete %s", lockFilePath)
}
// newLock will get a new file lock
func newLock(path string) *lock {
lockPath := path + ".lock"
return &lock{
lockFilePath: lockPath,
backoff: &origin.BackoffHandler{MaxRetries: 7},
sigHandler: &signalHandler{
signals: []os.Signal{syscall.SIGINT, syscall.SIGTERM},
},
}
}
func (l *lock) Acquire() error {
// Intercept SIGINT and SIGTERM to release lock before exiting
l.sigHandler.register(func() {
l.deleteLockFile()
os.Exit(0)
})
// Check for a path.lock file
// if the lock file exists; start polling
// if not, create the lock file and go through the normal flow.
// See AUTH-1736 for the reason why we do all this
for isTokenLocked(l.lockFilePath) {
if l.backoff.Backoff(context.Background()) {
continue
}
if err := l.deleteLockFile(); err != nil {
return err
}
}
// Create a lock file so other processes won't also try to get the token at
// the same time
if err := ioutil.WriteFile(l.lockFilePath, []byte{}, 0600); err != nil {
return err
}
return nil
}
func (l *lock) deleteLockFile() error {
if err := os.Remove(l.lockFilePath); err != nil && !os.IsNotExist(err) {
return errDeleteTokenFailed(l.lockFilePath)
}
return nil
}
func (l *lock) Release() error {
defer l.sigHandler.deregister()
return l.deleteLockFile()
}
// isTokenLocked checks to see if there is another process attempting to get the token already
func isTokenLocked(lockFilePath string) bool {
exists, err := config.FileExists(lockFilePath)
return exists && err == nil
}
// FetchToken will either load a stored token or generate a new one
func FetchToken(appURL *url.URL) (string, error) {
if token, err := GetTokenIfExists(appURL); token != "" && err == nil {
@@ -29,6 +123,19 @@ func FetchToken(appURL *url.URL) (string, error) {
return "", err
}
fileLock := newLock(path)
err = fileLock.Acquire()
if err != nil {
return "", err
}
defer fileLock.Release()
// check to see if another process has gotten a token while we waited for the lock
if token, err := GetTokenIfExists(appURL); token != "" && err == nil {
return token, nil
}
// this weird parameter is the resource name (token) and the key/value
// we want to send to the transfer service. the key is token and the value
// is blank (basically just the id generated in the transfer service)
@@ -55,14 +162,18 @@ func GetTokenIfExists(url *url.URL) (string, error) {
return "", err
}
claims, err := token.Claims()
if err != nil {
return "", err
}
ident, err := oidc.IdentityFromClaims(claims)
// AUTH-1404, reauth if the token is about to expire within 15 minutes
if err == nil && ident.ExpiresAt.After(time.Now().Add(time.Minute*15)) {
return token.Encode(), nil
}
return "", err
return token.Encode(), nil
}
// RemoveTokenIfExists removes the a token from local storage if it exists
func RemoveTokenIfExists(url *url.URL) error {
path, err := path.GenerateFilePathFromURL(url, keyName)
if err != nil {
return err
}
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
return err
}
return nil
}
+52
View File
@@ -0,0 +1,52 @@
package token
import (
"os"
"syscall"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestSignalHandler(t *testing.T) {
sigHandler := signalHandler{signals: []os.Signal{syscall.SIGUSR1}}
handlerRan := false
done := make(chan struct{})
timer := time.NewTimer(time.Second)
sigHandler.register(func(){
handlerRan = true
done <- struct{}{}
})
p, err := os.FindProcess(os.Getpid())
require.Nil(t, err)
p.Signal(syscall.SIGUSR1)
// Blocks for up to one second to make sure the handler callback runs before the assert.
select {
case <- done:
assert.True(t, handlerRan)
case <- timer.C:
t.Fail()
}
sigHandler.deregister()
}
func TestSignalHandlerClose(t *testing.T) {
sigHandler := signalHandler{signals: []os.Signal{syscall.SIGUSR1}}
done := make(chan struct{})
timer := time.NewTimer(time.Second)
sigHandler.register(func(){done <- struct{}{}})
sigHandler.deregister()
p, err := os.FindProcess(os.Getpid())
require.Nil(t, err)
p.Signal(syscall.SIGUSR1)
select {
case <- done:
t.Fail()
case <- timer.C:
}
}
+255 -59
View File
@@ -1,26 +1,34 @@
package tunnel
import (
"context"
"fmt"
"io/ioutil"
"net"
"net/url"
"os"
"reflect"
"runtime/trace"
"sync"
"syscall"
"time"
"github.com/getsentry/raven-go"
"golang.org/x/crypto/ssh/terminal"
"github.com/cloudflare/cloudflared/dbconnect"
"github.com/cloudflare/cloudflared/tunnelrpc/pogs"
"github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/supervisor"
"github.com/google/uuid"
"github.com/getsentry/raven-go"
"github.com/cloudflare/cloudflared/cmd/cloudflared/buildinfo"
"github.com/cloudflare/cloudflared/cmd/cloudflared/config"
"github.com/cloudflare/cloudflared/cmd/cloudflared/updater"
"github.com/cloudflare/cloudflared/cmd/sqlgateway"
"github.com/cloudflare/cloudflared/hello"
"github.com/cloudflare/cloudflared/metrics"
"github.com/cloudflare/cloudflared/origin"
"github.com/cloudflare/cloudflared/signal"
"github.com/cloudflare/cloudflared/tlsconfig"
"github.com/cloudflare/cloudflared/tunneldns"
"github.com/cloudflare/cloudflared/websocket"
"github.com/coreos/go-systemd/daemon"
@@ -90,43 +98,7 @@ func Commands() []*cli.Command {
ArgsUsage: " ", // can't be the empty string or we get the default output
Hidden: false,
},
{
Name: "db",
Action: func(c *cli.Context) error {
tags := make(map[string]string)
tags["hostname"] = c.String("hostname")
raven.SetTagsContext(tags)
fmt.Printf("\nSQL Database Password: ")
pass, err := terminal.ReadPassword(int(syscall.Stdin))
if err != nil {
logger.Error(err)
}
go sqlgateway.StartProxy(c, logger, string(pass))
raven.CapturePanic(func() { err = tunnel(c) }, nil)
if err != nil {
raven.CaptureError(err, nil)
}
return err
},
Before: Before,
Usage: "SQL Gateway is an SQL over HTTP reverse proxy",
Flags: []cli.Flag{
&cli.BoolFlag{
Name: "db",
Value: true,
Usage: "Enable the SQL Gateway Proxy",
},
&cli.StringFlag{
Name: "address",
Value: "",
Usage: "Database connection string: db://user:pass",
},
},
Hidden: true,
},
dbConnectCmd(),
}
var subcommands []*cli.Command
@@ -234,9 +206,8 @@ func StartServer(c *cli.Context, version string, shutdownC, graceShutdownC chan
return err
}
buildInfo := origin.GetBuildInfo()
logger.Infof("Build info: %+v", *buildInfo)
logger.Infof("Version %s", version)
buildInfo := buildinfo.GetBuildInfo(version)
buildInfo.Log(logger)
logClientOptions(c)
if c.IsSet("proxy-dns") {
@@ -252,16 +223,6 @@ func StartServer(c *cli.Context, version string, shutdownC, graceShutdownC chan
// Wait for proxy-dns to come up (if used)
<-dnsReadySignal
// update needs to be after DNS proxy is up to resolve equinox server address
if updater.IsAutoupdateEnabled(c) {
logger.Infof("Autoupdate frequency is set to %v", c.Duration("autoupdate-freq"))
wg.Add(1)
go func() {
defer wg.Done()
errC <- updater.Autoupdate(c.Duration("autoupdate-freq"), &listeners, shutdownC)
}()
}
metricsListener, err := listeners.Listen("tcp", c.String("metrics"))
if err != nil {
logger.WithError(err).Error("Error opening metrics server listener")
@@ -279,6 +240,33 @@ func StartServer(c *cli.Context, version string, shutdownC, graceShutdownC chan
go writePidFile(connectedSignal, c.String("pidfile"))
}
cloudflaredID, err := uuid.NewRandom()
if err != nil {
logger.WithError(err).Error("Cannot generate cloudflared ID")
return err
}
ctx, cancel := context.WithCancel(context.Background())
go func() {
<-shutdownC
cancel()
}()
if c.IsSet("use-declarative-tunnels") {
return startDeclarativeTunnel(ctx, c, cloudflaredID, buildInfo, &listeners)
}
// update needs to be after DNS proxy is up to resolve equinox server address
if updater.IsAutoupdateEnabled(c) {
logger.Infof("Autoupdate frequency is set to %v", c.Duration("autoupdate-freq"))
wg.Add(1)
go func() {
defer wg.Done()
autoupdater := updater.NewAutoUpdater(c.Duration("autoupdate-freq"), &listeners)
errC <- autoupdater.Run(ctx)
}()
}
// Serve DNS proxy stand-alone if no hostname or tag or app is going to run
if dnsProxyStandAlone(c) {
connectedSignal.Notify()
@@ -287,6 +275,7 @@ func StartServer(c *cli.Context, version string, shutdownC, graceShutdownC chan
}
if c.IsSet("hello-world") {
logger.Infof("hello-world set")
helloListener, err := hello.CreateTLSListener("127.0.0.1:")
if err != nil {
logger.WithError(err).Error("Cannot start Hello World Server")
@@ -323,7 +312,7 @@ func StartServer(c *cli.Context, version string, shutdownC, graceShutdownC chan
wg.Add(1)
go func() {
defer wg.Done()
errC <- origin.StartTunnelDaemon(tunnelConfig, graceShutdownC, connectedSignal)
errC <- origin.StartTunnelDaemon(ctx, tunnelConfig, connectedSignal, cloudflaredID)
}()
return waitToShutdown(&wg, errC, shutdownC, graceShutdownC, c.Duration("grace-period"))
@@ -348,6 +337,128 @@ func Before(c *cli.Context) error {
return nil
}
func startDeclarativeTunnel(ctx context.Context,
c *cli.Context,
cloudflaredID uuid.UUID,
buildInfo *buildinfo.BuildInfo,
listeners *gracenet.Net,
) error {
reverseProxyOrigin, err := defaultOriginConfig(c)
if err != nil {
logger.WithError(err)
return err
}
reverseProxyConfig, err := pogs.NewReverseProxyConfig(
c.String("hostname"),
reverseProxyOrigin,
c.Uint64("retries"),
c.Duration("proxy-connection-timeout"),
c.Uint64("compression-quality"),
)
if err != nil {
logger.WithError(err).Error("Cannot initialize default client config because reverse proxy config is invalid")
return err
}
defaultClientConfig := &pogs.ClientConfig{
Version: pogs.InitVersion(),
SupervisorConfig: &pogs.SupervisorConfig{
AutoUpdateFrequency: c.Duration("autoupdate-freq"),
MetricsUpdateFrequency: c.Duration("metrics-update-freq"),
GracePeriod: c.Duration("grace-period"),
},
EdgeConnectionConfig: &pogs.EdgeConnectionConfig{
NumHAConnections: uint8(c.Int("ha-connections")),
HeartbeatInterval: c.Duration("heartbeat-interval"),
Timeout: c.Duration("dial-edge-timeout"),
MaxFailedHeartbeats: c.Uint64("heartbeat-count"),
},
DoHProxyConfigs: []*pogs.DoHProxyConfig{},
ReverseProxyConfigs: []*pogs.ReverseProxyConfig{reverseProxyConfig},
}
autoupdater := updater.NewAutoUpdater(defaultClientConfig.SupervisorConfig.AutoUpdateFrequency, listeners)
originCert, err := getOriginCert(c)
if err != nil {
logger.WithError(err).Error("error getting origin cert")
return err
}
toEdgeTLSConfig, err := tlsconfig.CreateTunnelConfig(c)
if err != nil {
logger.WithError(err).Error("unable to create TLS config to connect with edge")
return err
}
tags, err := NewTagSliceFromCLI(c.StringSlice("tag"))
if err != nil {
logger.WithError(err).Error("unable to parse tag")
return err
}
var scope pogs.Scope
if c.IsSet("group") == c.IsSet("system-name") {
err = fmt.Errorf("exactly one of --group or --system-name must be specified")
logger.WithError(err).Error("unable to determine scope")
return err
} else if c.IsSet("group") {
scope = pogs.NewGroup(c.String("group"))
} else {
scope = pogs.NewSystemName(c.String("system-name"))
}
cloudflaredConfig := &connection.CloudflaredConfig{
BuildInfo: buildInfo,
CloudflaredID: cloudflaredID,
Scope: scope,
Tags: tags,
}
serviceDiscoverer, err := serviceDiscoverer(c, logger)
if err != nil {
logger.WithError(err).Error("unable to create service discoverer")
return err
}
supervisor, err := supervisor.NewSupervisor(defaultClientConfig, originCert, toEdgeTLSConfig,
serviceDiscoverer, cloudflaredConfig, autoupdater, updater.SupportAutoUpdate(), logger)
if err != nil {
logger.WithError(err).Error("unable to create Supervisor")
return err
}
return supervisor.Run(ctx)
}
func defaultOriginConfig(c *cli.Context) (pogs.OriginConfig, error) {
if c.IsSet("hello-world") {
return &pogs.HelloWorldOriginConfig{}, nil
}
originConfig := &pogs.HTTPOriginConfig{
TCPKeepAlive: c.Duration("proxy-tcp-keepalive"),
DialDualStack: !c.Bool("proxy-no-happy-eyeballs"),
TLSHandshakeTimeout: c.Duration("proxy-tls-timeout"),
TLSVerify: !c.Bool("no-tls-verify"),
OriginCAPool: c.String("origin-ca-pool"),
OriginServerName: c.String("origin-server-name"),
MaxIdleConnections: c.Uint64("proxy-keepalive-connections"),
IdleConnectionTimeout: c.Duration("proxy-keepalive-timeout"),
ProxyConnectionTimeout: c.Duration("proxy-connection-timeout"),
ExpectContinueTimeout: c.Duration("proxy-expect-continue-timeout"),
ChunkedEncoding: c.Bool("no-chunked-encoding"),
}
if c.IsSet("unix-socket") {
unixSocket, err := config.ValidateUnixSocket(c)
if err != nil {
return nil, errors.Wrap(err, "error validating --unix-socket")
}
originConfig.URLString = unixSocket
}
originAddr, err := config.ValidateUrl(c)
if err != nil {
return nil, errors.Wrap(err, "error validating origin URL")
}
originConfig.URLString = originAddr
return originConfig, nil
}
func waitToShutdown(wg *sync.WaitGroup,
errC chan error,
shutdownC, graceShutdownC chan struct{},
@@ -411,6 +522,60 @@ func addPortIfMissing(uri *url.URL, port int) string {
return fmt.Sprintf("%s:%d", uri.Hostname(), port)
}
func dbConnectCmd() *cli.Command {
cmd := dbconnect.Cmd()
// Append the tunnel commands so users can customize the daemon settings.
cmd.Flags = appendFlags(Flags(), cmd.Flags...)
// Override before to run tunnel validation before dbconnect validation.
cmd.Before = func(c *cli.Context) error {
err := Before(c)
if err == nil {
err = dbconnect.CmdBefore(c)
}
return err
}
// Override action to setup the Proxy, then if successful, start the tunnel daemon.
cmd.Action = func(c *cli.Context) error {
err := dbconnect.CmdAction(c)
if err == nil {
err = tunnel(c)
}
return err
}
return cmd
}
// appendFlags will append extra flags to a slice of flags.
//
// The cli package will panic if two flags exist with the same name,
// so if extraFlags contains a flag that was already defined, modify the
// original flags to use the extra version.
func appendFlags(flags []cli.Flag, extraFlags ...cli.Flag) []cli.Flag {
for _, extra := range extraFlags {
var found bool
// Check if an extra flag overrides an existing flag.
for i, flag := range flags {
if reflect.DeepEqual(extra.Names(), flag.Names()) {
flags[i] = extra
found = true
break
}
}
// Append the extra flag if it has nothing to override.
if !found {
flags = append(flags, extra)
}
}
return flags
}
func tunnelFlags(shouldHide bool) []cli.Flag {
return []cli.Flag{
&cli.StringFlag{
@@ -421,8 +586,8 @@ func tunnelFlags(shouldHide bool) []cli.Flag {
},
altsrc.NewDurationFlag(&cli.DurationFlag{
Name: "autoupdate-freq",
Usage: "Autoupdate frequency. Default is 24h.",
Value: time.Hour * 24,
Usage: fmt.Sprintf("Autoupdate frequency. Default is %v.", updater.DefaultCheckUpdateFreq),
Value: updater.DefaultCheckUpdateFreq,
Hidden: shouldHide,
}),
altsrc.NewBoolFlag(&cli.BoolFlag{
@@ -444,7 +609,7 @@ func tunnelFlags(shouldHide bool) []cli.Flag {
Hidden: true,
}),
altsrc.NewStringFlag(&cli.StringFlag{
Name: "cacert",
Name: tlsconfig.CaCertFlag,
Usage: "Certificate Authority authenticating connections with Cloudflare's edge network.",
EnvVars: []string{"TUNNEL_CACERT"},
Hidden: true,
@@ -463,7 +628,7 @@ func tunnelFlags(shouldHide bool) []cli.Flag {
Hidden: shouldHide,
}),
altsrc.NewStringFlag(&cli.StringFlag{
Name: "origin-ca-pool",
Name: tlsconfig.OriginCAPoolFlag,
Usage: "Path to the CA for the certificate of your origin. This option should be used only if your certificate is not signed by Cloudflare.",
EnvVars: []string{"TUNNEL_ORIGIN_CA_POOL"},
Hidden: shouldHide,
@@ -636,6 +801,18 @@ func tunnelFlags(shouldHide bool) []cli.Flag {
Value: time.Second * 90,
Hidden: shouldHide,
}),
altsrc.NewDurationFlag(&cli.DurationFlag{
Name: "proxy-connection-timeout",
Usage: "HTTP proxy timeout for closing an idle connection",
Value: time.Second * 90,
Hidden: shouldHide,
}),
altsrc.NewDurationFlag(&cli.DurationFlag{
Name: "proxy-expect-continue-timeout",
Usage: "HTTP proxy timeout for closing an idle connection",
Value: time.Second * 90,
Hidden: shouldHide,
}),
altsrc.NewBoolFlag(&cli.BoolFlag{
Name: "proxy-dns",
Usage: "Run a DNS over HTTPS proxy server.",
@@ -695,5 +872,24 @@ func tunnelFlags(shouldHide bool) []cli.Flag {
EnvVars: []string{"TUNNEL_USE_DECLARATIVE"},
Hidden: true,
}),
altsrc.NewStringFlag(&cli.StringFlag{
Name: "system-name",
Usage: "Unique identifier for this cloudflared instance. It can be configured individually in the Declarative Tunnel UI. Mutually exclusive with `--group`.",
EnvVars: []string{"TUNNEL_SYSTEM_NAME"},
Hidden: true,
}),
altsrc.NewStringFlag(&cli.StringFlag{
Name: "group",
Usage: "Name of a group of cloudflared instances, of which this instance should be an identical copy. They can be configured collectively in the Declarative Tunnel UI. Mutually exclusive with `--system-name`.",
EnvVars: []string{"TUNNEL_GROUP"},
Hidden: true,
}),
altsrc.NewDurationFlag(&cli.DurationFlag{
Name: "dial-edge-timeout",
Usage: "Maximum wait time to set up a connection with the edge",
Value: time.Second * 15,
EnvVars: []string{"DIAL_EDGE_TIMEOUT"},
Hidden: true,
}),
}
}
+11 -108
View File
@@ -3,18 +3,18 @@ package tunnel
import (
"context"
"crypto/tls"
"crypto/x509"
"fmt"
"io/ioutil"
"net"
"net/http"
"os"
"path/filepath"
"runtime"
"strings"
"time"
"github.com/cloudflare/cloudflared/cmd/cloudflared/buildinfo"
"github.com/cloudflare/cloudflared/cmd/cloudflared/config"
"github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/origin"
"github.com/cloudflare/cloudflared/tlsconfig"
tunnelpogs "github.com/cloudflare/cloudflared/tunnelrpc/pogs"
@@ -147,7 +147,7 @@ If you don't have a certificate signed by Cloudflare, run the command:
func prepareTunnelConfig(
c *cli.Context,
buildInfo *origin.BuildInfo,
buildInfo *buildinfo.BuildInfo,
version string, logger,
transportLogger *logrus.Logger,
) (*origin.TunnelConfig, error) {
@@ -187,7 +187,7 @@ func prepareTunnelConfig(
}
}
originCertPool, err := loadCertPool(c, logger)
originCertPool, err := tlsconfig.LoadOriginCA(c, logger)
if err != nil {
logger.WithError(err).Error("Error loading cert pool")
return nil, errors.Wrap(err, "Error loading cert pool")
@@ -236,7 +236,7 @@ func prepareTunnelConfig(
return nil, errors.Wrap(err, "unable to connect to the origin")
}
toEdgeTLSConfig, err := createTunnelConfig(c)
toEdgeTLSConfig, err := tlsconfig.CreateTunnelConfig(c)
if err != nil {
logger.WithError(err).Error("unable to create TLS config to connect with edge")
return nil, errors.Wrap(err, "unable to create TLS config to connect with edge")
@@ -274,110 +274,13 @@ func prepareTunnelConfig(
}, nil
}
func loadCertPool(c *cli.Context, logger *logrus.Logger) (*x509.CertPool, error) {
const originCAPoolFlag = "origin-ca-pool"
originCAPoolFilename := c.String(originCAPoolFlag)
var originCustomCAPool []byte
if originCAPoolFilename != "" {
var err error
originCustomCAPool, err = ioutil.ReadFile(originCAPoolFilename)
if err != nil {
return nil, errors.Wrap(err, fmt.Sprintf("unable to read the file %s for --%s", originCAPoolFilename, originCAPoolFlag))
}
func serviceDiscoverer(c *cli.Context, logger *logrus.Logger) (connection.EdgeServiceDiscoverer, error) {
// If --edge is specfied, resolve edge server addresses
if len(c.StringSlice("edge")) > 0 {
return connection.NewEdgeHostnameResolver(c.StringSlice("edge"))
}
originCertPool, err := loadOriginCertPool(originCustomCAPool)
if err != nil {
return nil, errors.Wrap(err, "error loading the certificate pool")
}
// Windows users should be notified that they can use the flag
if runtime.GOOS == "windows" && originCAPoolFilename == "" {
logger.Infof("cloudflared does not support loading the system root certificate pool on Windows. Please use the --%s to specify it", originCAPoolFlag)
}
return originCertPool, nil
}
func loadOriginCertPool(originCAPoolPEM []byte) (*x509.CertPool, error) {
// Get the global pool
certPool, err := loadGlobalCertPool()
if err != nil {
return nil, err
}
// Then, add any custom origin CA pool the user may have passed
if originCAPoolPEM != nil {
if !certPool.AppendCertsFromPEM(originCAPoolPEM) {
logger.Warn("could not append the provided origin CA to the cloudflared certificate pool")
}
}
return certPool, nil
}
func loadGlobalCertPool() (*x509.CertPool, error) {
// First, obtain the system certificate pool
certPool, err := x509.SystemCertPool()
if err != nil {
if runtime.GOOS != "windows" {
logger.WithError(err).Warn("error obtaining the system certificates")
}
certPool = x509.NewCertPool()
}
// Next, append the Cloudflare CAs into the system pool
cfRootCA, err := tlsconfig.GetCloudflareRootCA()
if err != nil {
return nil, errors.Wrap(err, "could not append Cloudflare Root CAs to cloudflared certificate pool")
}
for _, cert := range cfRootCA {
certPool.AddCert(cert)
}
// Finally, add the Hello certificate into the pool (since it's self-signed)
helloCert, err := tlsconfig.GetHelloCertificateX509()
if err != nil {
return nil, errors.Wrap(err, "could not append Hello server certificate to cloudflared certificate pool")
}
certPool.AddCert(helloCert)
return certPool, nil
}
func createTunnelConfig(c *cli.Context) (*tls.Config, error) {
var rootCAs []string
if c.String("cacert") != "" {
rootCAs = append(rootCAs, c.String("cacert"))
}
edgeAddrs := c.StringSlice("edge")
userConfig := &tlsconfig.TLSParameters{RootCAs: rootCAs}
tlsConfig, err := tlsconfig.GetConfig(userConfig)
if err != nil {
return nil, err
}
if tlsConfig.RootCAs == nil {
rootCAPool := x509.NewCertPool()
cfRootCA, err := tlsconfig.GetCloudflareRootCA()
if err != nil {
return nil, errors.Wrap(err, "could not append Cloudflare Root CAs to cloudflared certificate pool")
}
for _, cert := range cfRootCA {
rootCAPool.AddCert(cert)
}
tlsConfig.RootCAs = rootCAPool
tlsConfig.ServerName = "cftunnel.com"
} else if len(edgeAddrs) > 0 {
// Set for development environments and for testing specific origintunneld instances
tlsConfig.ServerName, _, _ = net.SplitHostPort(edgeAddrs[0])
}
if tlsConfig.ServerName == "" && !tlsConfig.InsecureSkipVerify {
return nil, fmt.Errorf("either ServerName or InsecureSkipVerify must be specified in the tls.Config")
}
return tlsConfig, nil
// Otherwise lookup edge server addresses through service discovery
return connection.NewEdgeAddrResolver(logger)
}
func isRunningFromTerminal() bool {
+84 -26
View File
@@ -1,6 +1,7 @@
package updater
import (
"context"
"os"
"runtime"
"time"
@@ -14,6 +15,7 @@ import (
)
const (
DefaultCheckUpdateFreq = time.Hour * 24
appID = "app_idCzgxYerVD"
noUpdateInShellMessage = "cloudflared will not automatically update when run from the shell. To enable auto-updates, run cloudflared as a service: https://developers.cloudflare.com/argo-tunnel/reference/service/"
noUpdateOnWindowsMessage = "cloudflared will not automatically update on Windows systems."
@@ -75,30 +77,6 @@ func Update(_ *cli.Context) error {
return updateOutcome.Error
}
func Autoupdate(freq time.Duration, listeners *gracenet.Net, shutdownC chan struct{}) error {
tickC := time.Tick(freq)
for {
updateOutcome := loggedUpdate()
if updateOutcome.Updated {
os.Args = append(os.Args, "--is-autoupdated=true")
pid, err := listeners.StartProcess()
if err != nil {
logger.WithError(err).Error("Unable to restart server automatically")
return err
}
// stop old process after autoupdate. Otherwise we create a new process
// after each update
logger.Infof("PID of the new process is %d", pid)
return nil
}
select {
case <-tickC:
case <-shutdownC:
return nil
}
}
}
// Checks for an update and applies it if one is available
func loggedUpdate() UpdateOutcome {
updateOutcome := checkForUpdateAndApply()
@@ -112,7 +90,88 @@ func loggedUpdate() UpdateOutcome {
return updateOutcome
}
// AutoUpdater periodically checks for new version of cloudflared.
type AutoUpdater struct {
configurable *configurable
listeners *gracenet.Net
updateConfigChan chan *configurable
}
// AutoUpdaterConfigurable is the attributes of AutoUpdater that can be reconfigured during runtime
type configurable struct {
enabled bool
freq time.Duration
}
func NewAutoUpdater(freq time.Duration, listeners *gracenet.Net) *AutoUpdater {
updaterConfigurable := &configurable{
enabled: true,
freq: freq,
}
if freq == 0 {
updaterConfigurable.enabled = false
updaterConfigurable.freq = DefaultCheckUpdateFreq
}
return &AutoUpdater{
configurable: updaterConfigurable,
listeners: listeners,
updateConfigChan: make(chan *configurable),
}
}
func (a *AutoUpdater) Run(ctx context.Context) error {
ticker := time.NewTicker(a.configurable.freq)
for {
if a.configurable.enabled {
updateOutcome := loggedUpdate()
if updateOutcome.Updated {
os.Args = append(os.Args, "--is-autoupdated=true")
pid, err := a.listeners.StartProcess()
if err != nil {
logger.WithError(err).Error("Unable to restart server automatically")
return err
}
// stop old process after autoupdate. Otherwise we create a new process
// after each update
logger.Infof("PID of the new process is %d", pid)
return nil
}
}
select {
case <-ctx.Done():
return ctx.Err()
case newConfigurable := <-a.updateConfigChan:
ticker.Stop()
a.configurable = newConfigurable
ticker = time.NewTicker(a.configurable.freq)
// Check if there is new version of cloudflared after receiving new AutoUpdaterConfigurable
case <-ticker.C:
}
}
}
// Update is the method to pass new AutoUpdaterConfigurable to a running AutoUpdater. It is safe to be called concurrently
func (a *AutoUpdater) Update(newFreq time.Duration) {
newConfigurable := &configurable{
enabled: true,
freq: newFreq,
}
// A ero duration means autoupdate is disabled
if newFreq == 0 {
newConfigurable.enabled = false
newConfigurable.freq = DefaultCheckUpdateFreq
}
a.updateConfigChan <- newConfigurable
}
func IsAutoupdateEnabled(c *cli.Context) bool {
if !SupportAutoUpdate() {
return false
}
return !c.Bool("no-autoupdate") && c.Duration("autoupdate-freq") != 0
}
func SupportAutoUpdate() bool {
if runtime.GOOS == "windows" {
logger.Info(noUpdateOnWindowsMessage)
return false
@@ -122,8 +181,7 @@ func IsAutoupdateEnabled(c *cli.Context) bool {
logger.Info(noUpdateInShellMessage)
return false
}
return !c.Bool("no-autoupdate") && c.Duration("autoupdate-freq") != 0
return true
}
func isRunningFromTerminal() bool {
+26
View File
@@ -0,0 +1,26 @@
package updater
import (
"context"
"testing"
"github.com/facebookgo/grace/gracenet"
"github.com/stretchr/testify/assert"
)
func TestDisabledAutoUpdater(t *testing.T) {
listeners := &gracenet.Net{}
autoupdater := NewAutoUpdater(0, listeners)
ctx, cancel := context.WithCancel(context.Background())
errC := make(chan error)
go func() {
errC <- autoupdater.Run(ctx)
}()
assert.False(t, autoupdater.configurable.enabled)
assert.Equal(t, DefaultCheckUpdateFreq, autoupdater.configurable.freq)
cancel()
// Make sure that autoupdater terminates after canceling the context
assert.Equal(t, context.Canceled, <-errC)
}
-148
View File
@@ -1,148 +0,0 @@
package sqlgateway
import (
"database/sql"
"encoding/json"
"fmt"
"math/rand"
"net/http"
"strings"
"time"
_ "github.com/lib/pq"
cli "gopkg.in/urfave/cli.v2"
"github.com/elgs/gosqljson"
"github.com/gorilla/mux"
"github.com/sirupsen/logrus"
)
type Message struct {
Connection Connection `json:"connection"`
Command string `json:"command"`
Params []interface{} `json:"params"`
}
type Connection struct {
SSLMode string `json:"sslmode"`
Token string `json:"token"`
}
type Response struct {
Columns []string `json:"columns"`
Rows [][]string `json:"rows"`
Error string `json:"error"`
}
type Proxy struct {
Context *cli.Context
Router *mux.Router
Token string
User string
Password string
Driver string
Database string
Logger *logrus.Logger
}
func StartProxy(c *cli.Context, logger *logrus.Logger, password string) error {
proxy := NewProxy(c, logger, password)
logger.Infof("Starting SQL Gateway Proxy on port %s", strings.Split(c.String("url"), ":")[1])
err := http.ListenAndServe(":"+strings.Split(c.String("url"), ":")[1], proxy.Router)
if err != nil {
return err
}
return nil
}
func randID(n int, c *cli.Context) string {
charBytes := []byte("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890")
b := make([]byte, n)
for i := range b {
b[i] = charBytes[rand.Intn(len(charBytes))]
}
return fmt.Sprintf("%s&%s", c.String("hostname"), b)
}
// db://user@dbname
func parseInfo(input string) (string, string, string) {
p1 := strings.Split(input, "://")
p2 := strings.Split(p1[1], "@")
return p1[0], p2[0], p2[1]
}
func NewProxy(c *cli.Context, logger *logrus.Logger, pass string) *Proxy {
rand.Seed(time.Now().UnixNano())
driver, user, dbname := parseInfo(c.String("address"))
proxy := Proxy{
Context: c,
Router: mux.NewRouter(),
Token: randID(64, c),
Logger: logger,
User: user,
Password: pass,
Database: dbname,
Driver: driver,
}
logger.Info(fmt.Sprintf(`
--------------------
SQL Gateway Proxy
Token: %s
--------------------
`, proxy.Token))
proxy.Router.HandleFunc("/", proxy.proxyRequest).Methods("POST")
return &proxy
}
func (proxy *Proxy) proxyRequest(rw http.ResponseWriter, req *http.Request) {
var message Message
response := Response{}
err := json.NewDecoder(req.Body).Decode(&message)
if err != nil {
proxy.Logger.Error(err)
http.Error(rw, fmt.Sprintf("400 - %s", err.Error()), http.StatusBadRequest)
return
}
if message.Connection.Token != proxy.Token {
proxy.Logger.Error("Invalid token")
http.Error(rw, "400 - Invalid token", http.StatusBadRequest)
return
}
connStr := fmt.Sprintf("user=%s password=%s dbname=%s sslmode=%s", proxy.User, proxy.Password, proxy.Database, message.Connection.SSLMode)
db, err := sql.Open(proxy.Driver, connStr)
defer db.Close()
if err != nil {
proxy.Logger.Error(err)
http.Error(rw, fmt.Sprintf("400 - %s", err.Error()), http.StatusBadRequest)
return
} else {
proxy.Logger.Info("Forwarding SQL: ", message.Command)
rw.Header().Set("Content-Type", "application/json")
headers, data, err := gosqljson.QueryDbToArray(db, "lower", message.Command, message.Params...)
if err != nil {
proxy.Logger.Error(err)
http.Error(rw, fmt.Sprintf("400 - %s", err.Error()), http.StatusBadRequest)
return
} else {
response = Response{headers, data, ""}
}
}
json.NewEncoder(rw).Encode(response)
}
+32 -113
View File
@@ -2,14 +2,14 @@ package connection
import (
"context"
"crypto/tls"
"net"
"sync"
"time"
"github.com/cloudflare/cloudflared/h2mux"
"github.com/cloudflare/cloudflared/tunnelrpc"
"github.com/cloudflare/cloudflared/tunnelrpc/pogs"
tunnelpogs "github.com/cloudflare/cloudflared/tunnelrpc/pogs"
"github.com/google/uuid"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
@@ -17,7 +17,6 @@ import (
)
const (
dialTimeout = 5 * time.Second
openStreamTimeout = 30 * time.Second
)
@@ -29,134 +28,54 @@ func (e dialError) Error() string {
return e.cause.Error()
}
type muxerShutdownError struct{}
func (e muxerShutdownError) Error() string {
return "muxer shutdown"
type Connection struct {
id uuid.UUID
muxer *h2mux.Muxer
}
type ConnectionConfig struct {
TLSConfig *tls.Config
HeartbeatInterval time.Duration
MaxHeartbeats uint64
Logger *logrus.Entry
}
type connectionHandler interface {
serve(ctx context.Context) error
connect(ctx context.Context, parameters *tunnelpogs.ConnectParameters) (*tunnelpogs.ConnectResult, error)
shutdown()
}
type h2muxHandler struct {
muxer *h2mux.Muxer
logger *logrus.Entry
}
type muxedStreamHandler struct {
}
// Implements MuxedStreamHandler interface
func (h *muxedStreamHandler) ServeStream(stream *h2mux.MuxedStream) error {
return nil
}
func (h *h2muxHandler) serve(ctx context.Context) error {
// Serve doesn't return until h2mux is shutdown
if err := h.muxer.Serve(ctx); err != nil {
return err
func newConnection(muxer *h2mux.Muxer, edgeIP *net.TCPAddr) (*Connection, error) {
id, err := uuid.NewRandom()
if err != nil {
return nil, err
}
return muxerShutdownError{}
return &Connection{
id: id,
muxer: muxer,
}, nil
}
func (c *Connection) Serve(ctx context.Context) error {
// Serve doesn't return until h2mux is shutdown
return c.muxer.Serve(ctx)
}
// Connect is used to establish connections with cloudflare's edge network
func (h *h2muxHandler) connect(ctx context.Context, parameters *tunnelpogs.ConnectParameters) (*tunnelpogs.ConnectResult, error) {
func (c *Connection) Connect(ctx context.Context, parameters *tunnelpogs.ConnectParameters, logger *logrus.Entry) (*pogs.ConnectResult, error) {
openStreamCtx, cancel := context.WithTimeout(ctx, openStreamTimeout)
defer cancel()
conn, err := h.newRPConn(openStreamCtx)
rpcConn, err := c.newRPConn(openStreamCtx, logger)
if err != nil {
return nil, errors.Wrap(err, "Failed to create new RPC connection")
return nil, errors.Wrap(err, "cannot create new RPC connection")
}
defer conn.Close()
tsClient := tunnelpogs.TunnelServer_PogsClient{Client: conn.Bootstrap(ctx)}
defer rpcConn.Close()
tsClient := tunnelpogs.TunnelServer_PogsClient{Client: rpcConn.Bootstrap(ctx)}
return tsClient.Connect(ctx, parameters)
}
func (h *h2muxHandler) shutdown() {
h.muxer.Shutdown()
func (c *Connection) Shutdown() {
c.muxer.Shutdown()
}
func (h *h2muxHandler) newRPConn(ctx context.Context) (*rpc.Conn, error) {
stream, err := h.muxer.OpenStream(ctx, []h2mux.Header{
{Name: ":method", Value: "RPC"},
{Name: ":scheme", Value: "capnp"},
{Name: ":path", Value: "*"},
}, nil)
func (c *Connection) newRPConn(ctx context.Context, logger *logrus.Entry) (*rpc.Conn, error) {
stream, err := c.muxer.OpenRPCStream(ctx)
if err != nil {
return nil, err
}
return rpc.NewConn(
tunnelrpc.NewTransportLogger(h.logger.WithField("subsystem", "rpc-register"), rpc.StreamTransport(stream)),
tunnelrpc.ConnLog(h.logger.WithField("subsystem", "rpc-transport")),
tunnelrpc.NewTransportLogger(logger.WithField("rpc", "connect"), rpc.StreamTransport(stream)),
tunnelrpc.ConnLog(logger.WithField("rpc", "connect")),
), nil
}
// NewConnectionHandler returns a connectionHandler, wrapping h2mux to make RPC calls
func newH2MuxHandler(ctx context.Context,
config *ConnectionConfig,
edgeIP *net.TCPAddr,
) (connectionHandler, error) {
// Inherit from parent context so we can cancel (Ctrl-C) while dialing
dialCtx, dialCancel := context.WithTimeout(ctx, dialTimeout)
defer dialCancel()
dialer := net.Dialer{DualStack: true}
plaintextEdgeConn, err := dialer.DialContext(dialCtx, "tcp", edgeIP.String())
if err != nil {
return nil, dialError{cause: errors.Wrap(err, "DialContext error")}
}
edgeConn := tls.Client(plaintextEdgeConn, config.TLSConfig)
edgeConn.SetDeadline(time.Now().Add(dialTimeout))
err = edgeConn.Handshake()
if err != nil {
return nil, dialError{cause: errors.Wrap(err, "Handshake with edge error")}
}
// clear the deadline on the conn; h2mux has its own timeouts
edgeConn.SetDeadline(time.Time{})
// Establish a muxed connection with the edge
// Client mux handshake with agent server
muxer, err := h2mux.Handshake(edgeConn, edgeConn, h2mux.MuxerConfig{
Timeout: dialTimeout,
Handler: &muxedStreamHandler{},
IsClient: true,
HeartbeatInterval: config.HeartbeatInterval,
MaxHeartbeats: config.MaxHeartbeats,
Logger: config.Logger,
})
if err != nil {
return nil, err
}
return &h2muxHandler{
muxer: muxer,
logger: config.Logger,
}, nil
}
// connectionPool is a pool of connection handlers
type connectionPool struct {
sync.Mutex
connectionHandlers []connectionHandler
}
func (cp *connectionPool) put(h connectionHandler) {
cp.Lock()
defer cp.Unlock()
cp.connectionHandlers = append(cp.connectionHandlers, h)
}
func (cp *connectionPool) close() {
cp.Lock()
defer cp.Unlock()
for _, h := range cp.connectionHandlers {
h.shutdown()
}
}
+129 -18
View File
@@ -5,10 +5,11 @@ import (
"crypto/tls"
"fmt"
"net"
"sync"
"time"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"github.com/sirupsen/logrus"
)
const (
@@ -22,6 +23,9 @@ const (
dotServerName = "cloudflare-dns.com"
dotServerAddr = "1.1.1.1:853"
dotTimeout = time.Duration(15 * time.Second)
// SRV record resolution TTL
resolveEdgeAddrTTL = 1 * time.Hour
)
var friendlyDNSErrorLines = []string{
@@ -34,20 +38,65 @@ var friendlyDNSErrorLines = []string{
` https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/`,
}
func ResolveEdgeIPs(logger *log.Logger, addresses []string) ([]*net.TCPAddr, error) {
if len(addresses) > 0 {
var tcpAddrs []*net.TCPAddr
for _, address := range addresses {
// Addresses specified (for testing, usually)
tcpAddr, err := net.ResolveTCPAddr("tcp", address)
if err != nil {
return nil, err
}
tcpAddrs = append(tcpAddrs, tcpAddr)
}
return tcpAddrs, nil
// EdgeServiceDiscoverer is an interface for looking up Cloudflare's edge network addresses
type EdgeServiceDiscoverer interface {
// Addr returns an address to connect to cloudflare's edge network
Addr() *net.TCPAddr
// AvailableAddrs returns the number of unique addresses
AvailableAddrs() uint8
// Refresh rediscover Cloudflare's edge network addresses
Refresh() error
}
// EdgeAddrResolver discovers the addresses of Cloudflare's edge network through SRV record.
// It implements EdgeServiceDiscoverer interface
type EdgeAddrResolver struct {
sync.Mutex
// Addrs to connect to cloudflare's edge network
addrs []*net.TCPAddr
// index of the next element to use in addrs
nextAddrIndex int
logger *logrus.Entry
}
func NewEdgeAddrResolver(logger *logrus.Logger) (EdgeServiceDiscoverer, error) {
r := &EdgeAddrResolver{
logger: logger.WithField("subsystem", " edgeAddrResolver"),
}
// HA service discovery lookup
if err := r.Refresh(); err != nil {
return nil, err
}
return r, nil
}
func (r *EdgeAddrResolver) Addr() *net.TCPAddr {
r.Lock()
defer r.Unlock()
addr := r.addrs[r.nextAddrIndex]
r.nextAddrIndex = (r.nextAddrIndex + 1) % len(r.addrs)
return addr
}
func (r *EdgeAddrResolver) AvailableAddrs() uint8 {
r.Lock()
defer r.Unlock()
return uint8(len(r.addrs))
}
func (r *EdgeAddrResolver) Refresh() error {
newAddrs, err := EdgeDiscovery(r.logger)
if err != nil {
return err
}
r.Lock()
defer r.Unlock()
r.addrs = newAddrs
r.nextAddrIndex = 0
return nil
}
// HA service discovery lookup
func EdgeDiscovery(logger *logrus.Entry) ([]*net.TCPAddr, error) {
_, addrs, err := net.LookupSRV(srvService, srvProto, srvName)
if err != nil {
// Try to fall back to DoT from Cloudflare directly.
@@ -78,7 +127,7 @@ func ResolveEdgeIPs(logger *log.Logger, addresses []string) ([]*net.TCPAddr, err
var resolvedIPsPerCNAME [][]*net.TCPAddr
var lookupErr error
for _, addr := range addrs {
ips, err := ResolveSRVToTCP(addr)
ips, err := resolveSRVToTCP(addr)
if err != nil || len(ips) == 0 {
// don't return early, we might be able to resolve other addresses
lookupErr = err
@@ -86,14 +135,14 @@ func ResolveEdgeIPs(logger *log.Logger, addresses []string) ([]*net.TCPAddr, err
}
resolvedIPsPerCNAME = append(resolvedIPsPerCNAME, ips)
}
ips := FlattenServiceIPs(resolvedIPsPerCNAME)
ips := flattenServiceIPs(resolvedIPsPerCNAME)
if lookupErr == nil && len(ips) == 0 {
return nil, fmt.Errorf("Unknown service discovery error")
}
return ips, lookupErr
}
func ResolveSRVToTCP(srv *net.SRV) ([]*net.TCPAddr, error) {
func resolveSRVToTCP(srv *net.SRV) ([]*net.TCPAddr, error) {
ips, err := net.LookupIP(srv.Target)
if err != nil {
return nil, err
@@ -107,7 +156,7 @@ func ResolveSRVToTCP(srv *net.SRV) ([]*net.TCPAddr, error) {
// FlattenServiceIPs transposes and flattens the input slices such that the
// first element of the n inner slices are the first n elements of the result.
func FlattenServiceIPs(ipsByService [][]*net.TCPAddr) []*net.TCPAddr {
func flattenServiceIPs(ipsByService [][]*net.TCPAddr) []*net.TCPAddr {
var result []*net.TCPAddr
for len(ipsByService) > 0 {
filtered := ipsByService[:0]
@@ -141,3 +190,65 @@ func fallbackResolver(serverName, serverAddress string) *net.Resolver {
},
}
}
// EdgeHostnameResolver discovers the addresses of Cloudflare's edge network via a list of server hostnames.
// It implements EdgeServiceDiscoverer interface, and is used mainly for testing connectivity.
type EdgeHostnameResolver struct {
sync.Mutex
// hostnames of edge servers
hostnames []string
// Addrs to connect to cloudflare's edge network
addrs []*net.TCPAddr
// index of the next element to use in addrs
nextAddrIndex int
}
func NewEdgeHostnameResolver(edgeHostnames []string) (EdgeServiceDiscoverer, error) {
r := &EdgeHostnameResolver{
hostnames: edgeHostnames,
}
if err := r.Refresh(); err != nil {
return nil, err
}
return r, nil
}
func (r *EdgeHostnameResolver) Addr() *net.TCPAddr {
r.Lock()
defer r.Unlock()
addr := r.addrs[r.nextAddrIndex]
r.nextAddrIndex = (r.nextAddrIndex + 1) % len(r.addrs)
return addr
}
func (r *EdgeHostnameResolver) AvailableAddrs() uint8 {
r.Lock()
defer r.Unlock()
return uint8(len(r.addrs))
}
func (r *EdgeHostnameResolver) Refresh() error {
newAddrs, err := ResolveAddrs(r.hostnames)
if err != nil {
return err
}
r.Lock()
defer r.Unlock()
r.addrs = newAddrs
r.nextAddrIndex = 0
return nil
}
// Resolve TCP address given a list of addresses. Address can be a hostname, however, it will return at most one
// of the hostname's IP addresses
func ResolveAddrs(addrs []string) ([]*net.TCPAddr, error) {
var tcpAddrs []*net.TCPAddr
for _, addr := range addrs {
tcpAddr, err := net.ResolveTCPAddr("tcp", addr)
if err != nil {
return nil, err
}
tcpAddrs = append(tcpAddrs, tcpAddr)
}
return tcpAddrs, nil
}
+19 -1
View File
@@ -7,8 +7,26 @@ import (
"github.com/stretchr/testify/assert"
)
type mockEdgeServiceDiscoverer struct {
}
func (mr *mockEdgeServiceDiscoverer) Addr() *net.TCPAddr {
return &net.TCPAddr{
IP: net.ParseIP("127.0.0.1"),
Port: 63102,
}
}
func (mr *mockEdgeServiceDiscoverer) AvailableAddrs() uint8 {
return 1
}
func (mr *mockEdgeServiceDiscoverer) Refresh() error {
return nil
}
func TestFlattenServiceIPs(t *testing.T) {
result := FlattenServiceIPs([][]*net.TCPAddr{
result := flattenServiceIPs([][]*net.TCPAddr{
[]*net.TCPAddr{
&net.TCPAddr{Port: 1},
&net.TCPAddr{Port: 2},
+284
View File
@@ -0,0 +1,284 @@
package connection
import (
"context"
"crypto/tls"
"fmt"
"net"
"sync"
"time"
"github.com/cloudflare/cloudflared/cmd/cloudflared/buildinfo"
"github.com/cloudflare/cloudflared/h2mux"
"github.com/cloudflare/cloudflared/tunnelrpc/pogs"
"github.com/google/uuid"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
)
const (
quickStartLink = "https://developers.cloudflare.com/argo-tunnel/quickstart/"
faqLink = "https://developers.cloudflare.com/argo-tunnel/faq/"
)
// EdgeManager manages connections with the edge
type EdgeManager struct {
// streamHandler handles stream opened by the edge
streamHandler h2mux.MuxedStreamHandler
// TLSConfig is the TLS configuration to connect with edge
tlsConfig *tls.Config
// cloudflaredConfig is the cloudflared configuration that is determined when the process first starts
cloudflaredConfig *CloudflaredConfig
// serviceDiscoverer returns the next edge addr to connect to
serviceDiscoverer EdgeServiceDiscoverer
// state is attributes of ConnectionManager that can change during runtime.
state *edgeManagerState
logger *logrus.Entry
}
// EdgeConnectionManagerConfigurable is the configurable attributes of a EdgeConnectionManager
type EdgeManagerConfigurable struct {
TunnelHostnames []h2mux.TunnelHostname
*pogs.EdgeConnectionConfig
}
type CloudflaredConfig struct {
CloudflaredID uuid.UUID
Tags []pogs.Tag
BuildInfo *buildinfo.BuildInfo
Scope pogs.Scope
}
func NewEdgeManager(
streamHandler h2mux.MuxedStreamHandler,
edgeConnMgrConfigurable *EdgeManagerConfigurable,
userCredential []byte,
tlsConfig *tls.Config,
serviceDiscoverer EdgeServiceDiscoverer,
cloudflaredConfig *CloudflaredConfig,
logger *logrus.Logger,
) *EdgeManager {
return &EdgeManager{
streamHandler: streamHandler,
tlsConfig: tlsConfig,
cloudflaredConfig: cloudflaredConfig,
serviceDiscoverer: serviceDiscoverer,
state: newEdgeConnectionManagerState(edgeConnMgrConfigurable, userCredential),
logger: logger.WithField("subsystem", "connectionManager"),
}
}
func (em *EdgeManager) Run(ctx context.Context) error {
defer em.shutdown()
resolveEdgeIPTicker := time.Tick(resolveEdgeAddrTTL)
for {
select {
case <-ctx.Done():
return errors.Wrap(ctx.Err(), "EdgeConnectionManager terminated")
case <-resolveEdgeIPTicker:
if err := em.serviceDiscoverer.Refresh(); err != nil {
em.logger.WithError(err).Warn("Cannot refresh Cloudflare edge addresses")
}
default:
time.Sleep(1 * time.Second)
}
// Create/delete connection one at a time, so we don't need to adjust for connections that are being created/deleted
// in shouldCreateConnection or shouldReduceConnection calculation
if em.state.shouldCreateConnection(em.serviceDiscoverer.AvailableAddrs()) {
if err := em.newConnection(ctx); err != nil {
em.logger.WithError(err).Error("cannot create new connection")
}
} else if em.state.shouldReduceConnection() {
if err := em.closeConnection(ctx); err != nil {
em.logger.WithError(err).Error("cannot close connection")
}
}
}
}
func (em *EdgeManager) UpdateConfigurable(newConfigurable *EdgeManagerConfigurable) {
em.logger.Infof("New edge connection manager configuration %+v", newConfigurable)
em.state.updateConfigurable(newConfigurable)
}
func (em *EdgeManager) newConnection(ctx context.Context) error {
edgeIP := em.serviceDiscoverer.Addr()
edgeConn, err := em.dialEdge(ctx, edgeIP)
if err != nil {
return errors.Wrap(err, "dial edge error")
}
configurable := em.state.getConfigurable()
// Establish a muxed connection with the edge
// Client mux handshake with agent server
muxer, err := h2mux.Handshake(edgeConn, edgeConn, h2mux.MuxerConfig{
Timeout: configurable.Timeout,
Handler: em.streamHandler,
IsClient: true,
HeartbeatInterval: configurable.HeartbeatInterval,
MaxHeartbeats: configurable.MaxFailedHeartbeats,
Logger: em.logger.WithField("subsystem", "muxer"),
})
if err != nil {
return errors.Wrap(err, "couldn't perform handshake with edge")
}
h2muxConn, err := newConnection(muxer, edgeIP)
if err != nil {
return errors.Wrap(err, "couldn't create h2mux connection")
}
go em.serveConn(ctx, h2muxConn)
connResult, err := h2muxConn.Connect(ctx, &pogs.ConnectParameters{
CloudflaredID: em.cloudflaredConfig.CloudflaredID,
CloudflaredVersion: em.cloudflaredConfig.BuildInfo.CloudflaredVersion,
NumPreviousAttempts: 0,
OriginCert: em.state.getUserCredential(),
Scope: em.cloudflaredConfig.Scope,
Tags: em.cloudflaredConfig.Tags,
}, em.logger)
if err != nil {
h2muxConn.Shutdown()
return errors.Wrap(err, "couldn't connect to edge")
}
if connErr := connResult.Err; connErr != nil {
if !connErr.ShouldRetry {
return errors.Wrap(connErr, em.noRetryMessage())
}
return errors.Wrapf(connErr, "edge responded with RetryAfter=%v", connErr.RetryAfter)
}
em.state.newConnection(h2muxConn)
em.logger.Infof("connected to %s", connResult.ServerInfo.LocationName)
return nil
}
func (em *EdgeManager) closeConnection(ctx context.Context) error {
conn := em.state.getFirstConnection()
if conn == nil {
return fmt.Errorf("no connection to close")
}
conn.Shutdown()
return nil
}
func (em *EdgeManager) serveConn(ctx context.Context, conn *Connection) {
err := conn.Serve(ctx)
em.logger.WithError(err).Warn("Connection closed")
em.state.closeConnection(conn)
}
func (em *EdgeManager) dialEdge(ctx context.Context, edgeIP *net.TCPAddr) (*tls.Conn, error) {
timeout := em.state.getConfigurable().Timeout
// Inherit from parent context so we can cancel (Ctrl-C) while dialing
dialCtx, dialCancel := context.WithTimeout(ctx, timeout)
defer dialCancel()
dialer := net.Dialer{DualStack: true}
edgeConn, err := dialer.DialContext(dialCtx, "tcp", edgeIP.String())
if err != nil {
return nil, dialError{cause: errors.Wrap(err, "DialContext error")}
}
tlsEdgeConn := tls.Client(edgeConn, em.tlsConfig)
tlsEdgeConn.SetDeadline(time.Now().Add(timeout))
if err = tlsEdgeConn.Handshake(); err != nil {
return nil, dialError{cause: errors.Wrap(err, "Handshake with edge error")}
}
// clear the deadline on the conn; h2mux has its own timeouts
tlsEdgeConn.SetDeadline(time.Time{})
return tlsEdgeConn, nil
}
func (em *EdgeManager) noRetryMessage() string {
messageTemplate := "cloudflared could not register an Argo Tunnel on your account. Please confirm the following before trying again:" +
"1. You have Argo Smart Routing enabled in your account, See Enable Argo section of %s." +
"2. Your credential at %s is still valid. See %s."
return fmt.Sprintf(messageTemplate, quickStartLink, em.state.getConfigurable().UserCredentialPath, faqLink)
}
func (em *EdgeManager) shutdown() {
em.state.shutdown()
}
type edgeManagerState struct {
sync.RWMutex
configurable *EdgeManagerConfigurable
userCredential []byte
conns map[uuid.UUID]*Connection
}
func newEdgeConnectionManagerState(configurable *EdgeManagerConfigurable, userCredential []byte) *edgeManagerState {
return &edgeManagerState{
configurable: configurable,
userCredential: userCredential,
conns: make(map[uuid.UUID]*Connection),
}
}
func (ems *edgeManagerState) shouldCreateConnection(availableEdgeAddrs uint8) bool {
ems.RLock()
defer ems.RUnlock()
expectedHAConns := ems.configurable.NumHAConnections
if availableEdgeAddrs < expectedHAConns {
expectedHAConns = availableEdgeAddrs
}
return uint8(len(ems.conns)) < expectedHAConns
}
func (ems *edgeManagerState) shouldReduceConnection() bool {
ems.RLock()
defer ems.RUnlock()
return uint8(len(ems.conns)) > ems.configurable.NumHAConnections
}
func (ems *edgeManagerState) newConnection(conn *Connection) {
ems.Lock()
defer ems.Unlock()
ems.conns[conn.id] = conn
}
func (ems *edgeManagerState) closeConnection(conn *Connection) {
ems.Lock()
defer ems.Unlock()
delete(ems.conns, conn.id)
}
func (ems *edgeManagerState) getFirstConnection() *Connection {
ems.RLock()
defer ems.RUnlock()
for _, conn := range ems.conns {
return conn
}
return nil
}
func (ems *edgeManagerState) shutdown() {
ems.Lock()
defer ems.Unlock()
for _, conn := range ems.conns {
conn.Shutdown()
}
}
func (ems *edgeManagerState) getConfigurable() *EdgeManagerConfigurable {
ems.Lock()
defer ems.Unlock()
return ems.configurable
}
func (ems *edgeManagerState) updateConfigurable(newConfigurable *EdgeManagerConfigurable) {
ems.Lock()
defer ems.Unlock()
ems.configurable = newConfigurable
}
func (ems *edgeManagerState) getUserCredential() []byte {
ems.RLock()
defer ems.RUnlock()
return ems.userCredential
}
+77
View File
@@ -0,0 +1,77 @@
package connection
import (
"testing"
"time"
"github.com/cloudflare/cloudflared/cmd/cloudflared/buildinfo"
"github.com/stretchr/testify/assert"
"github.com/cloudflare/cloudflared/h2mux"
"github.com/cloudflare/cloudflared/tunnelrpc/pogs"
"github.com/google/uuid"
"github.com/sirupsen/logrus"
)
var (
configurable = &EdgeManagerConfigurable{
[]h2mux.TunnelHostname{
"http.example.com",
"ws.example.com",
"hello.example.com",
},
&pogs.EdgeConnectionConfig{
NumHAConnections: 1,
HeartbeatInterval: 1 * time.Second,
Timeout: 5 * time.Second,
MaxFailedHeartbeats: 3,
UserCredentialPath: "/etc/cloudflared/cert.pem",
},
}
cloudflaredConfig = &CloudflaredConfig{
CloudflaredID: uuid.New(),
Tags: []pogs.Tag{
{Name: "pool", Value: "east-6"},
},
BuildInfo: &buildinfo.BuildInfo{
GoOS: "linux",
GoVersion: "1.12",
GoArch: "amd64",
CloudflaredVersion: "2019.6.0",
},
}
)
type mockStreamHandler struct {
}
func (msh *mockStreamHandler) ServeStream(*h2mux.MuxedStream) error {
return nil
}
func mockEdgeManager() *EdgeManager {
return NewEdgeManager(
&mockStreamHandler{},
configurable,
[]byte{},
nil,
&mockEdgeServiceDiscoverer{},
cloudflaredConfig,
logrus.New(),
)
}
func TestUpdateConfigurable(t *testing.T) {
m := mockEdgeManager()
newConfigurable := &EdgeManagerConfigurable{
[]h2mux.TunnelHostname{
"second.example.com",
},
&pogs.EdgeConnectionConfig{
NumHAConnections: 2,
},
}
m.UpdateConfigurable(newConfigurable)
assert.Equal(t, newConfigurable, m.state.getConfigurable())
}
-147
View File
@@ -1,147 +0,0 @@
package connection
import (
"context"
"net"
"time"
tunnelpogs "github.com/cloudflare/cloudflared/tunnelrpc/pogs"
"github.com/google/uuid"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
)
const (
// Waiting time before retrying a failed tunnel connection
reconnectDuration = time.Second * 10
// SRV record resolution TTL
resolveTTL = time.Hour
// Interval between establishing new connection
connectionInterval = time.Second
)
type CloudflaredConfig struct {
ConnectionConfig *ConnectionConfig
OriginCert []byte
Tags []tunnelpogs.Tag
EdgeAddrs []string
HAConnections uint
Logger *logrus.Logger
CloudflaredVersion string
}
// Supervisor is a stateful object that manages connections with the edge
type Supervisor struct {
config *CloudflaredConfig
state *supervisorState
connErrors chan error
}
type supervisorState struct {
// IPs to connect to cloudflare's edge network
edgeIPs []*net.TCPAddr
// index of the next element to use in edgeIPs
nextEdgeIPIndex int
// last time edgeIPs were refreshed
lastResolveTime time.Time
// ID of this cloudflared instance
cloudflaredID uuid.UUID
// connectionPool is a pool of connectionHandlers that can be used to make RPCs
connectionPool *connectionPool
}
func (s *supervisorState) getNextEdgeIP() *net.TCPAddr {
ip := s.edgeIPs[s.nextEdgeIPIndex%len(s.edgeIPs)]
s.nextEdgeIPIndex++
return ip
}
func NewSupervisor(config *CloudflaredConfig) *Supervisor {
return &Supervisor{
config: config,
state: &supervisorState{
connectionPool: &connectionPool{},
},
connErrors: make(chan error),
}
}
func (s *Supervisor) Run(ctx context.Context) error {
logger := s.config.Logger
if err := s.initialize(); err != nil {
logger.WithError(err).Error("Failed to get edge IPs")
return err
}
defer s.state.connectionPool.close()
var currentConnectionCount uint
expectedConnectionCount := s.config.HAConnections
if uint(len(s.state.edgeIPs)) < s.config.HAConnections {
logger.Warnf("You requested %d HA connections but I can give you at most %d.", s.config.HAConnections, len(s.state.edgeIPs))
expectedConnectionCount = uint(len(s.state.edgeIPs))
}
for {
select {
case <-ctx.Done():
return nil
case connErr := <-s.connErrors:
logger.WithError(connErr).Warnf("Connection dropped unexpectedly")
currentConnectionCount--
default:
time.Sleep(5 * time.Second)
}
if currentConnectionCount < expectedConnectionCount {
h, err := newH2MuxHandler(ctx, s.config.ConnectionConfig, s.state.getNextEdgeIP())
if err != nil {
logger.WithError(err).Error("Failed to create new connection handler")
continue
}
go func() {
s.connErrors <- h.serve(ctx)
}()
connResult, err := s.connect(ctx, s.config, s.state.cloudflaredID, h)
if err != nil {
logger.WithError(err).Errorf("Failed to connect to cloudflared's edge network")
h.shutdown()
continue
}
if connErr := connResult.Err; connErr != nil && !connErr.ShouldRetry {
logger.WithError(connErr).Errorf("Server respond with don't retry to connect")
h.shutdown()
return err
}
logger.Infof("Connected to %s", connResult.ServerInfo.LocationName)
s.state.connectionPool.put(h)
currentConnectionCount++
}
}
}
func (s *Supervisor) initialize() error {
edgeIPs, err := ResolveEdgeIPs(s.config.Logger, s.config.EdgeAddrs)
if err != nil {
return errors.Wrapf(err, "Failed to resolve cloudflare edge network address")
}
s.state.edgeIPs = edgeIPs
s.state.lastResolveTime = time.Now()
cloudflaredID, err := uuid.NewRandom()
if err != nil {
return errors.Wrap(err, "Failed to generate cloudflared ID")
}
s.state.cloudflaredID = cloudflaredID
return nil
}
func (s *Supervisor) connect(ctx context.Context,
config *CloudflaredConfig,
cloudflaredID uuid.UUID,
h connectionHandler,
) (*tunnelpogs.ConnectResult, error) {
connectParameters := &tunnelpogs.ConnectParameters{
OriginCert: config.OriginCert,
CloudflaredID: cloudflaredID,
NumPreviousAttempts: 0,
CloudflaredVersion: config.CloudflaredVersion,
}
return h.connect(ctx, connectParameters)
}
+145
View File
@@ -0,0 +1,145 @@
package dbconnect
import (
"context"
"encoding/json"
"fmt"
"net/url"
"strings"
"time"
"unicode"
"unicode/utf8"
)
// Client is an interface to talk to any database.
//
// Currently, the only implementation is SQLClient, but its structure
// should be designed to handle a MongoClient or RedisClient in the future.
type Client interface {
Ping(context.Context) error
Submit(context.Context, *Command) (interface{}, error)
}
// NewClient creates a database client based on its URL scheme.
func NewClient(ctx context.Context, originURL *url.URL) (Client, error) {
return NewSQLClient(ctx, originURL)
}
// Command is a standard, non-vendor format for submitting database commands.
//
// When determining the scope of this struct, refer to the following litmus test:
// Could this (roughly) conform to SQL, Document-based, and Key-value command formats?
type Command struct {
Statement string `json:"statement"`
Arguments Arguments `json:"arguments,omitempty"`
Mode string `json:"mode,omitempty"`
Isolation string `json:"isolation,omitempty"`
Timeout time.Duration `json:"timeout,omitempty"`
}
// Validate enforces the contract of Command: non empty statement (both in length and logic),
// lowercase mode and isolation, non-zero timeout, and valid Arguments.
func (cmd *Command) Validate() error {
if cmd.Statement == "" {
return fmt.Errorf("cannot provide an empty statement")
}
if strings.Map(func(char rune) rune {
if char == ';' || unicode.IsSpace(char) {
return -1
}
return char
}, cmd.Statement) == "" {
return fmt.Errorf("cannot provide a statement with no logic: '%s'", cmd.Statement)
}
cmd.Mode = strings.ToLower(cmd.Mode)
cmd.Isolation = strings.ToLower(cmd.Isolation)
if cmd.Timeout.Nanoseconds() <= 0 {
cmd.Timeout = 24 * time.Hour
}
return cmd.Arguments.Validate()
}
// UnmarshalJSON converts a byte representation of JSON into a Command, which is also validated.
func (cmd *Command) UnmarshalJSON(data []byte) error {
// Alias is required to avoid infinite recursion from the default UnmarshalJSON.
type Alias Command
alias := &struct {
*Alias
}{
Alias: (*Alias)(cmd),
}
err := json.Unmarshal(data, &alias)
if err == nil {
err = cmd.Validate()
}
return err
}
// Arguments is a wrapper for either map-based or array-based Command arguments.
//
// Each field is mutually-exclusive and some Client implementations may not
// support both fields (eg. MySQL does not accept named arguments).
type Arguments struct {
Named map[string]interface{}
Positional []interface{}
}
// Validate enforces the contract of Arguments: non nil, mutually exclusive, and no empty or reserved keys.
func (args *Arguments) Validate() error {
if args.Named == nil {
args.Named = map[string]interface{}{}
}
if args.Positional == nil {
args.Positional = []interface{}{}
}
if len(args.Named) > 0 && len(args.Positional) > 0 {
return fmt.Errorf("both named and positional arguments cannot be specified: %+v and %+v", args.Named, args.Positional)
}
for key := range args.Named {
if key == "" {
return fmt.Errorf("named arguments cannot contain an empty key: %+v", args.Named)
}
if !utf8.ValidString(key) {
return fmt.Errorf("named argument does not conform to UTF-8 encoding: %s", key)
}
if strings.HasPrefix(key, "_") {
return fmt.Errorf("named argument cannot start with a reserved keyword '_': %s", key)
}
if unicode.IsNumber([]rune(key)[0]) {
return fmt.Errorf("named argument cannot start with a number: %s", key)
}
}
return nil
}
// UnmarshalJSON converts a byte representation of JSON into Arguments, which is also validated.
func (args *Arguments) UnmarshalJSON(data []byte) error {
var obj interface{}
err := json.Unmarshal(data, &obj)
if err != nil {
return err
}
named, ok := obj.(map[string]interface{})
if ok {
args.Named = named
} else {
positional, ok := obj.([]interface{})
if ok {
args.Positional = positional
} else {
return fmt.Errorf("arguments must either be an object {\"0\":\"val\"} or an array [\"val\"]: %s", string(data))
}
}
return args.Validate()
}
+183
View File
@@ -0,0 +1,183 @@
package dbconnect
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
)
func TestCommandValidateEmpty(t *testing.T) {
stmts := []string{
"",
";",
" \n\t",
";\n;\t;",
}
for _, stmt := range stmts {
cmd := Command{Statement: stmt}
assert.Error(t, cmd.Validate(), stmt)
}
}
func TestCommandValidateMode(t *testing.T) {
modes := []string{
"",
"query",
"ExEc",
"PREPARE",
}
for _, mode := range modes {
cmd := Command{Statement: "Ok", Mode: mode}
assert.NoError(t, cmd.Validate(), mode)
assert.Equal(t, strings.ToLower(mode), cmd.Mode)
}
}
func TestCommandValidateIsolation(t *testing.T) {
isos := []string{
"",
"default",
"read_committed",
"SNAPshot",
}
for _, iso := range isos {
cmd := Command{Statement: "Ok", Isolation: iso}
assert.NoError(t, cmd.Validate(), iso)
assert.Equal(t, strings.ToLower(iso), cmd.Isolation)
}
}
func TestCommandValidateTimeout(t *testing.T) {
cmd := Command{Statement: "Ok", Timeout: 0}
assert.NoError(t, cmd.Validate())
assert.NotZero(t, cmd.Timeout)
cmd = Command{Statement: "Ok", Timeout: 1 * time.Second}
assert.NoError(t, cmd.Validate())
assert.Equal(t, 1*time.Second, cmd.Timeout)
}
func TestCommandValidateArguments(t *testing.T) {
cmd := Command{Statement: "Ok", Arguments: Arguments{
Named: map[string]interface{}{"key": "val"},
Positional: []interface{}{"val"},
}}
assert.Error(t, cmd.Validate())
}
func TestCommandUnmarshalJSON(t *testing.T) {
strs := []string{
"{\"statement\":\"Ok\"}",
"{\"statement\":\"Ok\",\"arguments\":[0, 3.14, \"apple\"],\"mode\":\"query\"}",
"{\"statement\":\"Ok\",\"isolation\":\"read_uncommitted\",\"timeout\":1000}",
}
for _, str := range strs {
var cmd Command
assert.NoError(t, json.Unmarshal([]byte(str), &cmd), str)
}
strs = []string{
"",
"\"",
"{}",
"{\"argument\":{\"key\":\"val\"}}",
"{\"statement\":[\"Ok\"]}",
}
for _, str := range strs {
var cmd Command
assert.Error(t, json.Unmarshal([]byte(str), &cmd), str)
}
}
func TestArgumentsValidateNotNil(t *testing.T) {
args := Arguments{}
assert.NoError(t, args.Validate())
assert.NotNil(t, args.Named)
assert.NotNil(t, args.Positional)
}
func TestArgumentsValidateMutuallyExclusive(t *testing.T) {
args := []Arguments{
Arguments{},
Arguments{Named: map[string]interface{}{"key": "val"}},
Arguments{Positional: []interface{}{"val"}},
}
for _, arg := range args {
assert.NoError(t, arg.Validate())
assert.False(t, len(arg.Named) > 0 && len(arg.Positional) > 0)
}
args = []Arguments{
Arguments{
Named: map[string]interface{}{"key": "val"},
Positional: []interface{}{"val"},
},
}
for _, arg := range args {
assert.Error(t, arg.Validate())
assert.True(t, len(arg.Named) > 0 && len(arg.Positional) > 0)
}
}
func TestArgumentsValidateKeys(t *testing.T) {
keys := []string{
"",
"_",
"_key",
"1",
"1key",
"\xf0\x28\x8c\xbc", // non-utf8
}
for _, key := range keys {
args := Arguments{Named: map[string]interface{}{key: "val"}}
assert.Error(t, args.Validate(), key)
}
}
func TestArgumentsUnmarshalJSON(t *testing.T) {
strs := []string{
"{}",
"{\"key\":\"val\"}",
"{\"key\":[1, 3.14, {\"key\":\"val\"}]}",
"[]",
"[\"key\",\"val\"]",
"[{}]",
}
for _, str := range strs {
var args Arguments
assert.NoError(t, json.Unmarshal([]byte(str), &args), str)
}
strs = []string{
"",
"\"",
"1",
"\"key\"",
"{\"key\",\"val\"}",
}
for _, str := range strs {
var args Arguments
assert.Error(t, json.Unmarshal([]byte(str), &args), str)
}
}
+157
View File
@@ -0,0 +1,157 @@
package dbconnect
import (
"context"
"log"
"net"
"strconv"
"gopkg.in/urfave/cli.v2"
"gopkg.in/urfave/cli.v2/altsrc"
)
// Cmd is the entrypoint command for dbconnect.
//
// The tunnel package is responsible for appending this to tunnel.Commands().
func Cmd() *cli.Command {
return &cli.Command{
Category: "Database Connect (ALPHA)",
Name: "db-connect",
Usage: "Access your SQL database from Cloudflare Workers or the browser",
ArgsUsage: " ",
Description: `
Creates a connection between your database and the Cloudflare edge.
Now you can execute SQL commands anywhere you can send HTTPS requests.
Connect your database with any of the following commands, you can also try the "playground" without a database:
cloudflared db-connect --hostname sql.mysite.com --url postgres://user:pass@localhost?sslmode=disable \
--auth-domain mysite.cloudflareaccess.com --application-aud my-access-policy-tag
cloudflared db-connect --hostname sql-dev.mysite.com --url mysql://localhost --insecure
cloudflared db-connect --playground
Requests should be authenticated using Cloudflare Access, learn more about how to enable it here:
https://developers.cloudflare.com/access/service-auth/service-token/
`,
Flags: []cli.Flag{
altsrc.NewStringFlag(&cli.StringFlag{
Name: "url",
Usage: "URL to the database (eg. postgres://user:pass@localhost?sslmode=disable)",
EnvVars: []string{"TUNNEL_URL"},
}),
altsrc.NewStringFlag(&cli.StringFlag{
Name: "hostname",
Usage: "Hostname to accept commands over HTTPS (eg. sql.mysite.com)",
EnvVars: []string{"TUNNEL_HOSTNAME"},
}),
altsrc.NewStringFlag(&cli.StringFlag{
Name: "auth-domain",
Usage: "Cloudflare Access authentication domain for your account (eg. mysite.cloudflareaccess.com)",
EnvVars: []string{"TUNNEL_ACCESS_AUTH_DOMAIN"},
}),
altsrc.NewStringFlag(&cli.StringFlag{
Name: "application-aud",
Usage: "Cloudflare Access application \"AUD\" to verify JWTs from requests",
EnvVars: []string{"TUNNEL_ACCESS_APPLICATION_AUD"},
}),
altsrc.NewBoolFlag(&cli.BoolFlag{
Name: "insecure",
Usage: "Disable authentication, the database will be open to the Internet",
Value: false,
EnvVars: []string{"TUNNEL_ACCESS_INSECURE"},
}),
altsrc.NewBoolFlag(&cli.BoolFlag{
Name: "playground",
Usage: "Run a temporary, in-memory SQLite3 database for testing",
Value: false,
EnvVars: []string{"TUNNEL_HELLO_WORLD"},
}),
altsrc.NewStringFlag(&cli.StringFlag{
Name: "loglevel",
Value: "debug", // Make it more verbose than the tunnel default 'info'.
EnvVars: []string{"TUNNEL_LOGLEVEL"},
Hidden: true,
}),
},
Before: CmdBefore,
Action: CmdAction,
Hidden: true,
}
}
// CmdBefore runs some validation checks before running the command.
func CmdBefore(c *cli.Context) error {
// Show the help text is no flags are specified.
if c.NumFlags() == 0 {
return cli.ShowSubcommandHelp(c)
}
// Hello-world and playground are synonymous with each other,
// unset hello-world to prevent tunnel from initializing the hello package.
if c.IsSet("hello-world") {
c.Set("playground", "true")
c.Set("hello-world", "false")
}
// Unix-socket database urls are supported, but the logic is the same as url.
if c.IsSet("unix-socket") {
c.Set("url", c.String("unix-socket"))
c.Set("unix-socket", "")
}
// When playground mode is enabled, run with an in-memory database.
if c.IsSet("playground") {
c.Set("url", "sqlite3::memory:?cache=shared")
c.Set("insecure", strconv.FormatBool(!c.IsSet("auth-domain") && !c.IsSet("application-aud")))
}
// At this point, insecure configurations are valid.
if c.Bool("insecure") {
return nil
}
// Ensure that secure configurations specify a hostname, domain, and tag for JWT validation.
if !c.IsSet("hostname") || !c.IsSet("auth-domain") || !c.IsSet("application-aud") {
log.Fatal("must specify --hostname, --auth-domain, and --application-aud unless you want to run in --insecure mode")
}
return nil
}
// CmdAction starts the Proxy and sets the url in cli.Context to point to the Proxy address.
func CmdAction(c *cli.Context) error {
// STOR-612: sync with context in tunnel daemon.
ctx := context.Background()
var proxy *Proxy
var err error
if c.Bool("insecure") {
proxy, err = NewInsecureProxy(ctx, c.String("url"))
} else {
proxy, err = NewSecureProxy(ctx, c.String("url"), c.String("auth-domain"), c.String("application-aud"))
}
if err != nil {
log.Fatal(err)
return err
}
listenerC := make(chan net.Listener)
defer close(listenerC)
// Since the Proxy should only talk to the tunnel daemon, find the next available
// localhost port and start to listen to requests.
go func() {
err := proxy.Start(ctx, "127.0.0.1:", listenerC)
if err != nil {
log.Fatal(err)
}
}()
// Block until the the Proxy is online, retreive its address, and change the url to point to it.
// This is effectively "handing over" control to the tunnel package so it can run the tunnel daemon.
c.Set("url", "https://"+(<-listenerC).Addr().String())
return nil
}
+27
View File
@@ -0,0 +1,27 @@
package dbconnect
import (
"testing"
"github.com/stretchr/testify/assert"
"gopkg.in/urfave/cli.v2"
)
func TestCmd(t *testing.T) {
tests := [][]string{
{"cloudflared", "db-connect", "--playground"},
{"cloudflared", "db-connect", "--playground", "--hostname", "sql.mysite.com"},
{"cloudflared", "db-connect", "--url", "sqlite3::memory:?cache=shared", "--insecure"},
{"cloudflared", "db-connect", "--url", "sqlite3::memory:?cache=shared", "--hostname", "sql.mysite.com", "--auth-domain", "mysite.cloudflareaccess.com", "--application-aud", "aud"},
}
app := &cli.App{
Name: "cloudflared",
Commands: []*cli.Command{Cmd()},
}
for _, test := range tests {
assert.NoError(t, app.Run(test))
}
}
+78
View File
@@ -0,0 +1,78 @@
# docker-compose -f ./dbconnect/integration_test/dbconnect.yaml up --build --force-recreate --renew-anon-volumes --exit-code-from cloudflared
version: "2.3"
networks:
test-dbconnect-network:
driver: bridge
services:
cloudflared:
build:
context: ../../
dockerfile: dev.Dockerfile
command: go test github.com/cloudflare/cloudflared/dbconnect/integration_test -v
depends_on:
postgres:
condition: service_healthy
mysql:
condition: service_healthy
mssql:
condition: service_healthy
clickhouse:
condition: service_healthy
environment:
DBCONNECT_INTEGRATION_TEST: "true"
POSTGRESQL_URL: postgres://postgres:secret@postgres/db?sslmode=disable
MYSQL_URL: mysql://root:secret@mysql/db?tls=false
MSSQL_URL: mssql://sa:secret12345!@mssql
CLICKHOUSE_URL: clickhouse://clickhouse:9000/db
networks:
- test-dbconnect-network
postgres:
image: postgres:11.4-alpine
environment:
POSTGRES_DB: db
POSTGRES_PASSWORD: secret
healthcheck:
test: ["CMD", "pg_isready", "-U", "postgres"]
start_period: 3s
interval: 1s
timeout: 3s
retries: 10
networks:
- test-dbconnect-network
mysql:
image: mysql:8.0
environment:
MYSQL_DATABASE: db
MYSQL_ROOT_PASSWORD: secret
healthcheck:
test: ["CMD", "mysqladmin", "ping"]
start_period: 3s
interval: 1s
timeout: 3s
retries: 10
networks:
- test-dbconnect-network
mssql:
image: mcr.microsoft.com/mssql/server:2017-CU8-ubuntu
environment:
ACCEPT_EULA: "Y"
SA_PASSWORD: secret12345!
healthcheck:
test: ["CMD", "/opt/mssql-tools/bin/sqlcmd", "-S", "localhost", "-U", "sa", "-P", "secret12345!", "-Q", "SELECT 1"]
start_period: 3s
interval: 1s
timeout: 3s
retries: 10
networks:
- test-dbconnect-network
clickhouse:
image: yandex/clickhouse-server:19.11
healthcheck:
test: ["CMD", "clickhouse-client", "--query", "SELECT 1"]
start_period: 3s
interval: 1s
timeout: 3s
retries: 10
networks:
- test-dbconnect-network
+265
View File
@@ -0,0 +1,265 @@
package dbconnect_test
import (
"context"
"log"
"net/url"
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/cloudflare/cloudflared/dbconnect"
)
func TestIntegrationPostgreSQL(t *testing.T) {
ctx, pq := helperNewSQLClient(t, "POSTGRESQL_URL")
err := pq.Ping(ctx)
assert.NoError(t, err)
_, err = pq.Submit(ctx, &dbconnect.Command{
Statement: "CREATE TABLE t (a TEXT, b UUID, c JSON, d INET[], e SERIAL);",
Mode: "exec",
})
assert.NoError(t, err)
_, err = pq.Submit(ctx, &dbconnect.Command{
Statement: "INSERT INTO t VALUES ($1, $2, $3, $4);",
Mode: "exec",
Arguments: dbconnect.Arguments{
Positional: []interface{}{
"text",
"6b8d686d-bd8e-43bc-b09a-cfcbbe702c10",
"{\"bool\":true,\"array\":[\"a\", 1, 3.14],\"embed\":{\"num\":21}}",
[]string{"1.1.1.1", "1.0.0.1"},
},
},
})
assert.NoError(t, err)
_, err = pq.Submit(ctx, &dbconnect.Command{
Statement: "UPDATE t SET b = NULL;",
Mode: "exec",
})
assert.NoError(t, err)
res, err := pq.Submit(ctx, &dbconnect.Command{
Statement: "SELECT * FROM t;",
Mode: "query",
})
assert.NoError(t, err)
assert.IsType(t, make([]map[string]interface{}, 0), res)
actual := res.([]map[string]interface{})[0]
expected := map[string]interface{}{
"a": "text",
"b": nil,
"c": map[string]interface{}{
"bool": true,
"array": []interface{}{"a", float64(1), 3.14},
"embed": map[string]interface{}{"num": float64(21)},
},
"d": "{1.1.1.1,1.0.0.1}",
"e": int64(1),
}
assert.EqualValues(t, expected, actual)
_, err = pq.Submit(ctx, &dbconnect.Command{
Statement: "DROP TABLE t;",
Mode: "exec",
})
assert.NoError(t, err)
}
func TestIntegrationMySQL(t *testing.T) {
ctx, my := helperNewSQLClient(t, "MYSQL_URL")
err := my.Ping(ctx)
assert.NoError(t, err)
_, err = my.Submit(ctx, &dbconnect.Command{
Statement: "CREATE TABLE t (a CHAR, b TINYINT, c FLOAT, d JSON, e YEAR);",
Mode: "exec",
})
assert.NoError(t, err)
_, err = my.Submit(ctx, &dbconnect.Command{
Statement: "INSERT INTO t VALUES (?, ?, ?, ?, ?);",
Mode: "exec",
Arguments: dbconnect.Arguments{
Positional: []interface{}{
"a",
10,
3.14,
"{\"bool\":true}",
2000,
},
},
})
assert.NoError(t, err)
_, err = my.Submit(ctx, &dbconnect.Command{
Statement: "ALTER TABLE t ADD COLUMN f GEOMETRY;",
Mode: "exec",
})
assert.NoError(t, err)
res, err := my.Submit(ctx, &dbconnect.Command{
Statement: "SELECT * FROM t;",
Mode: "query",
})
assert.NoError(t, err)
assert.IsType(t, make([]map[string]interface{}, 0), res)
actual := res.([]map[string]interface{})[0]
expected := map[string]interface{}{
"a": "a",
"b": float64(10),
"c": 3.14,
"d": map[string]interface{}{"bool": true},
"e": float64(2000),
"f": nil,
}
assert.EqualValues(t, expected, actual)
_, err = my.Submit(ctx, &dbconnect.Command{
Statement: "DROP TABLE t;",
Mode: "exec",
})
assert.NoError(t, err)
}
func TestIntegrationMSSQL(t *testing.T) {
ctx, ms := helperNewSQLClient(t, "MSSQL_URL")
err := ms.Ping(ctx)
assert.NoError(t, err)
_, err = ms.Submit(ctx, &dbconnect.Command{
Statement: "CREATE TABLE t (a BIT, b DECIMAL, c MONEY, d TEXT);",
Mode: "exec"})
assert.NoError(t, err)
_, err = ms.Submit(ctx, &dbconnect.Command{
Statement: "INSERT INTO t VALUES (?, ?, ?, ?);",
Mode: "exec",
Arguments: dbconnect.Arguments{
Positional: []interface{}{
0,
3,
"$0.99",
"text",
},
},
})
assert.NoError(t, err)
_, err = ms.Submit(ctx, &dbconnect.Command{
Statement: "UPDATE t SET d = NULL;",
Mode: "exec",
})
assert.NoError(t, err)
res, err := ms.Submit(ctx, &dbconnect.Command{
Statement: "SELECT * FROM t;",
Mode: "query",
})
assert.NoError(t, err)
assert.IsType(t, make([]map[string]interface{}, 0), res)
actual := res.([]map[string]interface{})[0]
expected := map[string]interface{}{
"a": false,
"b": float64(3),
"c": float64(0.99),
"d": nil,
}
assert.EqualValues(t, expected, actual)
_, err = ms.Submit(ctx, &dbconnect.Command{
Statement: "DROP TABLE t;",
Mode: "exec",
})
assert.NoError(t, err)
}
func TestIntegrationClickhouse(t *testing.T) {
ctx, ch := helperNewSQLClient(t, "CLICKHOUSE_URL")
err := ch.Ping(ctx)
assert.NoError(t, err)
_, err = ch.Submit(ctx, &dbconnect.Command{
Statement: "CREATE TABLE t (a UUID, b String, c Float64, d UInt32, e Int16, f Array(Enum8('a'=1, 'b'=2, 'c'=3))) engine=Memory;",
Mode: "exec",
})
assert.NoError(t, err)
_, err = ch.Submit(ctx, &dbconnect.Command{
Statement: "INSERT INTO t VALUES (?, ?, ?, ?, ?, ?);",
Mode: "exec",
Arguments: dbconnect.Arguments{
Positional: []interface{}{
"ec65f626-6f50-4c86-9628-6314ef1edacd",
"",
3.14,
314,
-144,
[]string{"a", "b", "c"},
},
},
})
assert.NoError(t, err)
res, err := ch.Submit(ctx, &dbconnect.Command{
Statement: "SELECT * FROM t;",
Mode: "query",
})
assert.NoError(t, err)
assert.IsType(t, make([]map[string]interface{}, 0), res)
actual := res.([]map[string]interface{})[0]
expected := map[string]interface{}{
"a": "ec65f626-6f50-4c86-9628-6314ef1edacd",
"b": "",
"c": float64(3.14),
"d": uint32(314),
"e": int16(-144),
"f": []string{"a", "b", "c"},
}
assert.EqualValues(t, expected, actual)
_, err = ch.Submit(ctx, &dbconnect.Command{
Statement: "DROP TABLE t;",
Mode: "exec",
})
assert.NoError(t, err)
}
func helperNewSQLClient(t *testing.T, env string) (context.Context, dbconnect.Client) {
_, ok := os.LookupEnv("DBCONNECT_INTEGRATION_TEST")
if ok {
t.Helper()
} else {
t.SkipNow()
}
val, ok := os.LookupEnv(env)
if !ok {
log.Fatalf("must provide database url as environment variable: %s", env)
}
parsed, err := url.Parse(val)
if err != nil {
log.Fatalf("cannot provide invalid database url: %s=%s", env, val)
}
ctx := context.Background()
client, err := dbconnect.NewSQLClient(ctx, parsed)
if err != nil {
log.Fatalf("could not start test client: %s", err)
}
return ctx, client
}
+274
View File
@@ -0,0 +1,274 @@
package dbconnect
import (
"context"
"encoding/json"
"fmt"
"io"
"net"
"net/http"
"net/url"
"time"
"github.com/cloudflare/cloudflared/hello"
"github.com/cloudflare/cloudflared/validation"
"github.com/gorilla/mux"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
timing "github.com/mitchellh/go-server-timing"
)
// Proxy is an HTTP server that proxies requests to a Client.
type Proxy struct {
client Client
accessValidator *validation.Access
logger *logrus.Logger
}
// NewInsecureProxy creates a Proxy that talks to a Client at an origin.
//
// In insecure mode, the Proxy will allow all Command requests.
func NewInsecureProxy(ctx context.Context, origin string) (*Proxy, error) {
originURL, err := url.Parse(origin)
if err != nil {
return nil, errors.Wrap(err, "must provide a valid database url")
}
client, err := NewClient(ctx, originURL)
if err != nil {
return nil, err
}
err = client.Ping(ctx)
if err != nil {
return nil, errors.Wrap(err, "could not connect to the database")
}
return &Proxy{client, nil, logrus.New()}, nil
}
// NewSecureProxy creates a Proxy that talks to a Client at an origin.
//
// In secure mode, the Proxy will reject any Command requests that are
// not authenticated by Cloudflare Access with a valid JWT.
func NewSecureProxy(ctx context.Context, origin, authDomain, applicationAUD string) (*Proxy, error) {
proxy, err := NewInsecureProxy(ctx, origin)
if err != nil {
return nil, err
}
validator, err := validation.NewAccessValidator(ctx, authDomain, authDomain, applicationAUD)
if err != nil {
return nil, err
}
proxy.accessValidator = validator
return proxy, err
}
// IsInsecure gets whether the Proxy will accept a Command from any source.
func (proxy *Proxy) IsInsecure() bool {
return proxy.accessValidator == nil
}
// IsAllowed checks whether a http.Request is allowed to receive data.
//
// By default, requests must pass through Cloudflare Access for authentication.
// If the proxy is explcitly set to insecure mode, all requests will be allowed.
func (proxy *Proxy) IsAllowed(r *http.Request, verbose ...bool) bool {
if proxy.IsInsecure() {
return true
}
// Access and Tunnel should prevent bad JWTs from even reaching the origin,
// but validate tokens anyway as an abundance of caution.
err := proxy.accessValidator.ValidateRequest(r.Context(), r)
if err == nil {
return true
}
// Warn administrators that invalid JWTs are being rejected. This is indicative
// of either a misconfiguration of the CLI or a massive failure of upstream systems.
if len(verbose) > 0 {
proxy.httpLog(r, err).Error("Failed JWT authentication")
}
return false
}
// Start the Proxy at a given address and notify the listener channel when the server is online.
func (proxy *Proxy) Start(ctx context.Context, addr string, listenerC chan<- net.Listener) error {
// STOR-611: use a seperate listener and consider web socket support.
httpListener, err := hello.CreateTLSListener(addr)
if err != nil {
return errors.Wrapf(err, "could not create listener at %s", addr)
}
errC := make(chan error)
defer close(errC)
// Starts the HTTP server and begins to serve requests.
go func() {
errC <- proxy.httpListen(ctx, httpListener)
}()
// Continually ping the server until it comes online or 10 attempts fail.
go func() {
var err error
for i := 0; i < 10; i++ {
_, err = http.Get("http://" + httpListener.Addr().String())
// Once no error was detected, notify the listener channel and return.
if err == nil {
listenerC <- httpListener
return
}
// Backoff between requests to ping the server.
<-time.After(1 * time.Second)
}
errC <- errors.Wrap(err, "took too long for the http server to start")
}()
return <-errC
}
// httpListen starts the httpServer and blocks until the context closes.
func (proxy *Proxy) httpListen(ctx context.Context, listener net.Listener) error {
httpServer := &http.Server{
Addr: listener.Addr().String(),
Handler: timing.Middleware(proxy.httpRouter(), nil),
ReadTimeout: 10 * time.Second,
WriteTimeout: 60 * time.Second,
IdleTimeout: 60 * time.Second,
}
go func() {
<-ctx.Done()
httpServer.Close()
listener.Close()
}()
return httpServer.Serve(listener)
}
// httpRouter creates a mux.Router for the Proxy.
func (proxy *Proxy) httpRouter() *mux.Router {
router := mux.NewRouter()
router.HandleFunc("/ping", proxy.httpPing()).Methods("GET", "HEAD")
router.HandleFunc("/submit", proxy.httpSubmit()).Methods("POST")
return router
}
// httpPing tests the connection to the database.
//
// By default, this endpoint is unauthenticated to allow for health checks.
// To enable authentication, Cloudflare Access must be enabled on this route.
func (proxy *Proxy) httpPing() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
metric := timing.FromContext(ctx).NewMetric("db").Start()
err := proxy.client.Ping(ctx)
metric.Stop()
if err == nil {
proxy.httpRespond(w, r, http.StatusOK, "")
} else {
proxy.httpRespondErr(w, r, http.StatusInternalServerError, err)
}
}
}
// httpSubmit sends a command to the database and returns its response.
//
// By default, this endpoint will reject requests that do not pass through Cloudflare Access.
// To disable authentication, the --insecure flag must be specified in the command line.
func (proxy *Proxy) httpSubmit() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if !proxy.IsAllowed(r, true) {
proxy.httpRespondErr(w, r, http.StatusForbidden, fmt.Errorf(""))
return
}
var cmd Command
err := json.NewDecoder(r.Body).Decode(&cmd)
if err != nil {
proxy.httpRespondErr(w, r, http.StatusBadRequest, err)
return
}
ctx := r.Context()
metric := timing.FromContext(ctx).NewMetric("db").Start()
data, err := proxy.client.Submit(ctx, &cmd)
metric.Stop()
if err != nil {
proxy.httpRespondErr(w, r, http.StatusUnprocessableEntity, err)
return
}
w.Header().Set("Content-type", "application/json")
err = json.NewEncoder(w).Encode(data)
if err != nil {
proxy.httpRespondErr(w, r, http.StatusInternalServerError, err)
}
}
}
// httpRespond writes a status code and string response to the response writer.
func (proxy *Proxy) httpRespond(w http.ResponseWriter, r *http.Request, status int, message string) {
w.WriteHeader(status)
// Only expose the message detail of the reponse if the request is not HEAD
// and the user is authenticated. For example, this prevents an unauthenticated
// failed health check from accidentally leaking sensitive information about the Client.
if r.Method != http.MethodHead && proxy.IsAllowed(r) {
if message == "" {
message = http.StatusText(status)
}
fmt.Fprint(w, message)
}
}
// httpRespondErr is similar to httpRespond, except it formats errors to be more friendly.
func (proxy *Proxy) httpRespondErr(w http.ResponseWriter, r *http.Request, defaultStatus int, err error) {
status, err := httpError(defaultStatus, err)
proxy.httpRespond(w, r, status, err.Error())
proxy.httpLog(r, err).Warn("Database connect error")
}
// httpLog returns a logrus.Entry that is formatted to output a request Cf-ray.
func (proxy *Proxy) httpLog(r *http.Request, err error) *logrus.Entry {
return proxy.logger.WithContext(r.Context()).WithField("CF-RAY", r.Header.Get("Cf-ray")).WithError(err)
}
// httpError extracts common errors and returns an status code and friendly error.
func httpError(defaultStatus int, err error) (int, error) {
if err == nil {
return http.StatusNotImplemented, fmt.Errorf("error expected but found none")
}
if err == io.EOF {
return http.StatusBadRequest, fmt.Errorf("request body cannot be empty")
}
if err == context.DeadlineExceeded {
return http.StatusRequestTimeout, err
}
_, ok := err.(net.Error)
if ok {
return http.StatusRequestTimeout, err
}
if err == context.Canceled {
// Does not exist in Golang, but would be: http.StatusClientClosedWithoutResponse
return 444, err
}
return defaultStatus, err
}
+238
View File
@@ -0,0 +1,238 @@
package dbconnect
import (
"context"
"fmt"
"io"
"io/ioutil"
"net"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/gorilla/mux"
"github.com/stretchr/testify/assert"
)
func TestNewInsecureProxy(t *testing.T) {
origins := []string{
"",
":/",
"http://localhost",
"tcp://localhost:9000?debug=true",
"mongodb://127.0.0.1",
}
for _, origin := range origins {
proxy, err := NewInsecureProxy(context.Background(), origin)
assert.Error(t, err)
assert.Empty(t, proxy)
}
}
func TestProxyIsAllowed(t *testing.T) {
proxy := helperNewProxy(t)
req := httptest.NewRequest("GET", "https://1.1.1.1/ping", nil)
assert.True(t, proxy.IsAllowed(req))
proxy = helperNewProxy(t, true)
req.Header.Set("Cf-access-jwt-assertion", "xxx")
assert.False(t, proxy.IsAllowed(req))
}
func TestProxyStart(t *testing.T) {
proxy := helperNewProxy(t)
ctx := context.Background()
listenerC := make(chan net.Listener)
err := proxy.Start(ctx, "1.1.1.1:", listenerC)
assert.Error(t, err)
err = proxy.Start(ctx, "127.0.0.1:-1", listenerC)
assert.Error(t, err)
ctx, cancel := context.WithTimeout(ctx, 0)
defer cancel()
err = proxy.Start(ctx, "127.0.0.1:", listenerC)
assert.IsType(t, http.ErrServerClosed, err)
}
func TestProxyHTTPRouter(t *testing.T) {
proxy := helperNewProxy(t)
router := proxy.httpRouter()
tests := []struct {
path string
method string
valid bool
}{
{"", "GET", false},
{"/", "GET", false},
{"/ping", "GET", true},
{"/ping", "HEAD", true},
{"/ping", "POST", false},
{"/submit", "POST", true},
{"/submit", "GET", false},
{"/submit/extra", "POST", false},
}
for _, test := range tests {
match := &mux.RouteMatch{}
ok := router.Match(httptest.NewRequest(test.method, "https://1.1.1.1"+test.path, nil), match)
assert.True(t, ok == test.valid, test.path)
}
}
func TestProxyHTTPPing(t *testing.T) {
proxy := helperNewProxy(t)
server := httptest.NewServer(proxy.httpPing())
defer server.Close()
client := server.Client()
res, err := client.Get(server.URL)
assert.NoError(t, err)
assert.Equal(t, http.StatusOK, res.StatusCode)
assert.Equal(t, int64(2), res.ContentLength)
res, err = client.Head(server.URL)
assert.NoError(t, err)
assert.Equal(t, http.StatusOK, res.StatusCode)
assert.Equal(t, int64(-1), res.ContentLength)
}
func TestProxyHTTPSubmit(t *testing.T) {
proxy := helperNewProxy(t)
server := httptest.NewServer(proxy.httpSubmit())
defer server.Close()
client := server.Client()
tests := []struct {
input string
status int
output string
}{
{"", http.StatusBadRequest, "request body cannot be empty"},
{"{}", http.StatusBadRequest, "cannot provide an empty statement"},
{"{\"statement\":\"Ok\"}", http.StatusUnprocessableEntity, "cannot provide invalid sql mode: ''"},
{"{\"statement\":\"Ok\",\"mode\":\"query\"}", http.StatusUnprocessableEntity, "near \"Ok\": syntax error"},
{"{\"statement\":\"CREATE TABLE t (a INT);\",\"mode\":\"exec\"}", http.StatusOK, "{\"last_insert_id\":0,\"rows_affected\":0}\n"},
}
for _, test := range tests {
res, err := client.Post(server.URL, "application/json", strings.NewReader(test.input))
assert.NoError(t, err)
assert.Equal(t, test.status, res.StatusCode)
if res.StatusCode > http.StatusOK {
assert.Equal(t, "text/plain; charset=utf-8", res.Header.Get("Content-type"))
} else {
assert.Equal(t, "application/json", res.Header.Get("Content-type"))
}
data, err := ioutil.ReadAll(res.Body)
defer res.Body.Close()
str := string(data)
assert.NoError(t, err)
assert.Equal(t, test.output, str)
}
}
func TestProxyHTTPSubmitForbidden(t *testing.T) {
proxy := helperNewProxy(t, true)
server := httptest.NewServer(proxy.httpSubmit())
defer server.Close()
client := server.Client()
res, err := client.Get(server.URL)
assert.NoError(t, err)
assert.Equal(t, http.StatusForbidden, res.StatusCode)
assert.Zero(t, res.ContentLength)
}
func TestProxyHTTPRespond(t *testing.T) {
proxy := helperNewProxy(t)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
proxy.httpRespond(w, r, http.StatusAccepted, "Hello")
}))
defer server.Close()
client := server.Client()
res, err := client.Get(server.URL)
assert.NoError(t, err)
assert.Equal(t, http.StatusAccepted, res.StatusCode)
assert.Equal(t, int64(5), res.ContentLength)
data, err := ioutil.ReadAll(res.Body)
defer res.Body.Close()
assert.Equal(t, []byte("Hello"), data)
}
func TestProxyHTTPRespondForbidden(t *testing.T) {
proxy := helperNewProxy(t, true)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
proxy.httpRespond(w, r, http.StatusAccepted, "Hello")
}))
defer server.Close()
client := server.Client()
res, err := client.Get(server.URL)
assert.NoError(t, err)
assert.Equal(t, http.StatusAccepted, res.StatusCode)
assert.Equal(t, int64(0), res.ContentLength)
}
func TestHTTPError(t *testing.T) {
_, errTimeout := net.DialTimeout("tcp", "127.0.0.1", 0)
assert.Error(t, errTimeout)
tests := []struct {
input error
status int
output error
}{
{nil, http.StatusNotImplemented, fmt.Errorf("error expected but found none")},
{io.EOF, http.StatusBadRequest, fmt.Errorf("request body cannot be empty")},
{context.DeadlineExceeded, http.StatusRequestTimeout, nil},
{context.Canceled, 444, nil},
{errTimeout, http.StatusRequestTimeout, nil},
{fmt.Errorf(""), http.StatusInternalServerError, nil},
}
for _, test := range tests {
status, err := httpError(http.StatusInternalServerError, test.input)
assert.Error(t, err)
assert.Equal(t, test.status, status)
if test.output == nil {
test.output = test.input
}
assert.Equal(t, test.output, err)
}
}
func helperNewProxy(t *testing.T, secure ...bool) *Proxy {
t.Helper()
proxy, err := NewSecureProxy(context.Background(), "file::memory:?cache=shared", "test.cloudflareaccess.com", "")
assert.NoError(t, err)
assert.NotNil(t, proxy)
if len(secure) == 0 {
proxy.accessValidator = nil // Mark as insecure
}
return proxy
}
+318
View File
@@ -0,0 +1,318 @@
package dbconnect
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"net/url"
"reflect"
"strings"
"github.com/jmoiron/sqlx"
"github.com/pkg/errors"
"github.com/xo/dburl"
// SQL drivers self-register with the database/sql package.
// https://github.com/golang/go/wiki/SQLDrivers
_ "github.com/denisenkom/go-mssqldb"
_ "github.com/go-sql-driver/mysql"
_ "github.com/mattn/go-sqlite3"
"github.com/kshvakov/clickhouse"
"github.com/lib/pq"
)
// SQLClient is a Client that talks to a SQL database.
type SQLClient struct {
Dialect string
driver *sqlx.DB
}
// NewSQLClient creates a SQL client based on its URL scheme.
func NewSQLClient(ctx context.Context, originURL *url.URL) (Client, error) {
res, err := dburl.Parse(originURL.String())
if err != nil {
helpText := fmt.Sprintf("supported drivers: %+q, see documentation for more details: %s", sql.Drivers(), "https://godoc.org/github.com/xo/dburl")
return nil, fmt.Errorf("could not parse sql database url '%s': %s\n%s", originURL, err.Error(), helpText)
}
// Establishes the driver, but does not test the connection.
driver, err := sqlx.Open(res.Driver, res.DSN)
if err != nil {
return nil, fmt.Errorf("could not open sql driver %s: %s\n%s", res.Driver, err.Error(), res.DSN)
}
// Closes the driver, will occur when the context finishes.
go func() {
<-ctx.Done()
driver.Close()
}()
return &SQLClient{driver.DriverName(), driver}, nil
}
// Ping verifies a connection to the database is still alive.
func (client *SQLClient) Ping(ctx context.Context) error {
return client.driver.PingContext(ctx)
}
// Submit queries or executes a command to the SQL database.
func (client *SQLClient) Submit(ctx context.Context, cmd *Command) (interface{}, error) {
txx, err := cmd.ValidateSQL(client.Dialect)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(ctx, cmd.Timeout)
defer cancel()
var res interface{}
// Get the next available sql.Conn and submit the Command.
err = sqlConn(ctx, client.driver, txx, func(conn *sql.Conn) error {
stmt := cmd.Statement
args := cmd.Arguments.Positional
if cmd.Mode == "query" {
res, err = sqlQuery(ctx, conn, stmt, args)
} else {
res, err = sqlExec(ctx, conn, stmt, args)
}
return err
})
return res, err
}
// ValidateSQL extends the contract of Command for SQL dialects:
// mode is conformed, arguments are []sql.NamedArg, and isolation is a sql.IsolationLevel.
//
// When the command should not be wrapped in a transaction, *sql.TxOptions and error will both be nil.
func (cmd *Command) ValidateSQL(dialect string) (*sql.TxOptions, error) {
err := cmd.Validate()
if err != nil {
return nil, err
}
mode, err := sqlMode(cmd.Mode)
if err != nil {
return nil, err
}
// Mutates Arguments to only use positional arguments with the type sql.NamedArg.
// This is a required by the sql.Driver before submitting arguments.
cmd.Arguments.sql(dialect)
iso, err := sqlIsolation(cmd.Isolation)
if err != nil {
return nil, err
}
// When isolation is out-of-range, this is indicative that no
// transaction should be executed and sql.TxOptions should be nil.
if iso < sql.LevelDefault {
return nil, nil
}
// In query mode, execute the transaction in read-only, unless it's Microsoft SQL
// which does not support that type of transaction.
readOnly := mode == "query" && dialect != "mssql"
return &sql.TxOptions{Isolation: iso, ReadOnly: readOnly}, nil
}
// sqlConn gets the next available sql.Conn in the connection pool and runs a function to use it.
//
// If the transaction options are nil, run the useIt function outside a transaction.
// This is potentially an unsafe operation if the command does not clean up its state.
func sqlConn(ctx context.Context, driver *sqlx.DB, txx *sql.TxOptions, useIt func(*sql.Conn) error) error {
conn, err := driver.Conn(ctx)
if err != nil {
return err
}
defer conn.Close()
// If transaction options are specified, begin and defer a rollback to catch errors.
var tx *sql.Tx
if txx != nil {
tx, err = conn.BeginTx(ctx, txx)
if err != nil {
return err
}
defer tx.Rollback()
}
err = useIt(conn)
// Check if useIt was successful and a transaction exists before committing.
if err == nil && tx != nil {
err = tx.Commit()
}
return err
}
// sqlQuery queries rows on a sql.Conn and returns an array of result objects.
func sqlQuery(ctx context.Context, conn *sql.Conn, stmt string, args []interface{}) ([]map[string]interface{}, error) {
rows, err := conn.QueryContext(ctx, stmt, args...)
if err == nil {
return sqlRows(rows)
}
return nil, err
}
// sqlExec executes a command on a sql.Conn and returns the result of the operation.
func sqlExec(ctx context.Context, conn *sql.Conn, stmt string, args []interface{}) (sqlResult, error) {
exec, err := conn.ExecContext(ctx, stmt, args...)
if err == nil {
return sqlResultFrom(exec), nil
}
return sqlResult{}, err
}
// sql mutates Arguments to contain a positional []sql.NamedArg.
//
// The actual return type is []interface{} due to the native Golang
// function signatures for sql.Exec and sql.Query being generic.
func (args *Arguments) sql(dialect string) {
result := args.Positional
for i, val := range result {
result[i] = sqlArg("", val, dialect)
}
for key, val := range args.Named {
result = append(result, sqlArg(key, val, dialect))
}
args.Positional = result
args.Named = map[string]interface{}{}
}
// sqlArg creates a sql.NamedArg from a key-value pair and an optional dialect.
//
// Certain dialects will need to wrap objects, such as arrays, to conform its driver requirements.
func sqlArg(key, val interface{}, dialect string) sql.NamedArg {
switch reflect.ValueOf(val).Kind() {
// PostgreSQL and Clickhouse require arrays to be wrapped before
// being inserted into the driver interface.
case reflect.Slice, reflect.Array:
switch dialect {
case "postgres":
val = pq.Array(val)
case "clickhouse":
val = clickhouse.Array(val)
}
}
return sql.Named(fmt.Sprint(key), val)
}
// sqlIsolation tries to match a string to a sql.IsolationLevel.
func sqlIsolation(str string) (sql.IsolationLevel, error) {
if str == "none" {
return sql.IsolationLevel(-1), nil
}
for iso := sql.LevelDefault; ; iso++ {
if iso > sql.LevelLinearizable {
return -1, fmt.Errorf("cannot provide an invalid sql isolation level: '%s'", str)
}
if str == "" || strings.EqualFold(iso.String(), strings.ReplaceAll(str, "_", " ")) {
return iso, nil
}
}
}
// sqlMode tries to match a string to a command mode: 'query' or 'exec' for now.
func sqlMode(str string) (string, error) {
switch str {
case "query", "exec":
return str, nil
default:
return "", fmt.Errorf("cannot provide invalid sql mode: '%s'", str)
}
}
// sqlRows scans through a SQL result set and returns an array of objects.
func sqlRows(rows *sql.Rows) ([]map[string]interface{}, error) {
columns, err := rows.Columns()
if err != nil {
return nil, errors.Wrap(err, "could not extract columns from result")
}
defer rows.Close()
types, err := rows.ColumnTypes()
if err != nil {
// Some drivers do not support type extraction, so fail silently and continue.
types = make([]*sql.ColumnType, len(columns))
}
values := make([]interface{}, len(columns))
pointers := make([]interface{}, len(columns))
var results []map[string]interface{}
for rows.Next() {
for i := range columns {
pointers[i] = &values[i]
}
rows.Scan(pointers...)
// Convert a row, an array of values, into an object where
// each key is the name of its respective column.
entry := make(map[string]interface{})
for i, col := range columns {
entry[col] = sqlValue(values[i], types[i])
}
results = append(results, entry)
}
return results, nil
}
// sqlValue handles special cases where sql.Rows does not return a "human-readable" object.
func sqlValue(val interface{}, col *sql.ColumnType) interface{} {
bytes, ok := val.([]byte)
if ok {
// Opportunistically check for embeded JSON and convert it to a first-class object.
var embeded interface{}
if json.Unmarshal(bytes, &embeded) == nil {
return embeded
}
// STOR-604: investigate a way to coerce PostgreSQL arrays '{a, b, ...}' into JSON.
// Although easy with strings, it becomes more difficult with special types like INET[].
return string(bytes)
}
return val
}
// sqlResult is a thin wrapper around sql.Result.
type sqlResult struct {
LastInsertId int64 `json:"last_insert_id"`
RowsAffected int64 `json:"rows_affected"`
}
// sqlResultFrom converts sql.Result into a JSON-marshable sqlResult.
func sqlResultFrom(res sql.Result) sqlResult {
insertID, errID := res.LastInsertId()
rowsAffected, errRows := res.RowsAffected()
// If an error occurs when extracting the result, it is because the
// driver does not support that specific field. Instead of passing this
// to the user, omit the field in the response.
if errID != nil {
insertID = -1
}
if errRows != nil {
rowsAffected = -1
}
return sqlResult{insertID, rowsAffected}
}
+336
View File
@@ -0,0 +1,336 @@
package dbconnect
import (
"context"
"database/sql"
"fmt"
"net/url"
"strings"
"testing"
"time"
"github.com/kshvakov/clickhouse"
"github.com/lib/pq"
"github.com/DATA-DOG/go-sqlmock"
"github.com/stretchr/testify/assert"
)
func TestNewSQLClient(t *testing.T) {
originURLs := []string{
"postgres://localhost",
"cockroachdb://localhost:1337",
"postgresql://user:pass@127.0.0.1",
"mysql://localhost",
"clickhouse://127.0.0.1:9000/?debug",
"sqlite3::memory:",
"file:test.db?cache=shared",
}
for _, originURL := range originURLs {
origin, _ := url.Parse(originURL)
_, err := NewSQLClient(context.Background(), origin)
assert.NoError(t, err, originURL)
}
originURLs = []string{
"",
"/",
"http://localhost",
"coolthing://user:pass@127.0.0.1",
}
for _, originURL := range originURLs {
origin, _ := url.Parse(originURL)
_, err := NewSQLClient(context.Background(), origin)
assert.Error(t, err, originURL)
}
}
func TestArgumentsSQL(t *testing.T) {
args := []Arguments{
Arguments{
Positional: []interface{}{
"val", 10, 3.14,
},
},
Arguments{
Named: map[string]interface{}{
"key": time.Unix(0, 0),
},
},
}
var nameType sql.NamedArg
for _, arg := range args {
arg.sql("")
for _, named := range arg.Positional {
assert.IsType(t, nameType, named)
}
}
}
func TestSQLArg(t *testing.T) {
tests := []struct {
key interface{}
val interface{}
dialect string
arg sql.NamedArg
}{
{"key", "val", "mssql", sql.Named("key", "val")},
{0, 1, "sqlite3", sql.Named("0", 1)},
{1, []string{"a", "b", "c"}, "postgres", sql.Named("1", pq.Array([]string{"a", "b", "c"}))},
{"in", []uint{0, 1}, "clickhouse", sql.Named("in", clickhouse.Array([]uint{0, 1}))},
{"", time.Unix(0, 0), "", sql.Named("", time.Unix(0, 0))},
}
for _, test := range tests {
arg := sqlArg(test.key, test.val, test.dialect)
assert.Equal(t, test.arg, arg, test.key)
}
}
func TestSQLIsolation(t *testing.T) {
tests := []struct {
str string
iso sql.IsolationLevel
}{
{"", sql.LevelDefault},
{"DEFAULT", sql.LevelDefault},
{"read_UNcommitted", sql.LevelReadUncommitted},
{"serializable", sql.LevelSerializable},
{"none", sql.IsolationLevel(-1)},
{"SNAP shot", -2},
{"blah", -2},
}
for _, test := range tests {
iso, err := sqlIsolation(test.str)
if test.iso < -1 {
assert.Error(t, err, test.str)
} else {
assert.NoError(t, err)
assert.Equal(t, test.iso, iso, test.str)
}
}
}
func TestSQLMode(t *testing.T) {
modes := []string{
"query",
"exec",
}
for _, mode := range modes {
actual, err := sqlMode(mode)
assert.NoError(t, err)
assert.Equal(t, strings.ToLower(mode), actual, mode)
}
modes = []string{
"",
"blah",
}
for _, mode := range modes {
_, err := sqlMode(mode)
assert.Error(t, err)
}
}
func helperRows(mockRows *sqlmock.Rows) *sql.Rows {
db, mock, _ := sqlmock.New()
mock.ExpectQuery("SELECT").WillReturnRows(mockRows)
rows, _ := db.Query("SELECT")
return rows
}
func TestSQLRows(t *testing.T) {
actual, err := sqlRows(helperRows(sqlmock.NewRows(
[]string{"name", "age", "dept"}).
AddRow("alice", 19, "prod")))
expected := []map[string]interface{}{map[string]interface{}{
"name": "alice",
"age": int64(19),
"dept": "prod"}}
assert.NoError(t, err)
assert.ElementsMatch(t, expected, actual)
}
func TestSQLValue(t *testing.T) {
tests := []struct {
input interface{}
output interface{}
}{
{"hello", "hello"},
{1, 1},
{false, false},
{[]byte("random"), "random"},
{[]byte("{\"json\":true}"), map[string]interface{}{"json": true}},
{[]byte("[]"), []interface{}{}},
}
for _, test := range tests {
assert.Equal(t, test.output, sqlValue(test.input, nil), test.input)
}
}
func TestSQLResultFrom(t *testing.T) {
res := sqlResultFrom(sqlmock.NewResult(1, 2))
assert.Equal(t, sqlResult{1, 2}, res)
res = sqlResultFrom(sqlmock.NewErrorResult(fmt.Errorf("")))
assert.Equal(t, sqlResult{-1, -1}, res)
}
func helperSQLite3(t *testing.T) (context.Context, Client) {
t.Helper()
ctx := context.Background()
url, _ := url.Parse("file::memory:?cache=shared")
sqlite3, err := NewSQLClient(ctx, url)
assert.NoError(t, err)
return ctx, sqlite3
}
func TestPing(t *testing.T) {
ctx, sqlite3 := helperSQLite3(t)
err := sqlite3.Ping(ctx)
assert.NoError(t, err)
}
func TestSubmit(t *testing.T) {
ctx, sqlite3 := helperSQLite3(t)
res, err := sqlite3.Submit(ctx, &Command{
Statement: "CREATE TABLE t (a INTEGER, b FLOAT, c TEXT, d BLOB);",
Mode: "exec",
})
assert.NoError(t, err)
assert.Equal(t, sqlResult{0, 0}, res)
res, err = sqlite3.Submit(ctx, &Command{
Statement: "SELECT * FROM t;",
Mode: "query",
})
assert.NoError(t, err)
assert.Empty(t, res)
res, err = sqlite3.Submit(ctx, &Command{
Statement: "INSERT INTO t VALUES (?, ?, ?, ?);",
Mode: "exec",
Arguments: Arguments{
Positional: []interface{}{
1,
3.14,
"text",
"blob",
},
},
})
assert.NoError(t, err)
assert.Equal(t, sqlResult{1, 1}, res)
res, err = sqlite3.Submit(ctx, &Command{
Statement: "UPDATE t SET c = NULL;",
Mode: "exec",
})
assert.NoError(t, err)
assert.Equal(t, sqlResult{1, 1}, res)
res, err = sqlite3.Submit(ctx, &Command{
Statement: "SELECT * FROM t WHERE a = ?;",
Mode: "query",
Arguments: Arguments{
Positional: []interface{}{1},
},
})
assert.NoError(t, err)
assert.Len(t, res, 1)
resf, ok := res.([]map[string]interface{})
assert.True(t, ok)
assert.EqualValues(t, map[string]interface{}{
"a": int64(1),
"b": 3.14,
"c": nil,
"d": "blob",
}, resf[0])
res, err = sqlite3.Submit(ctx, &Command{
Statement: "DROP TABLE t;",
Mode: "exec",
})
assert.NoError(t, err)
assert.Equal(t, sqlResult{1, 1}, res)
}
func TestSubmitTransaction(t *testing.T) {
ctx, sqlite3 := helperSQLite3(t)
res, err := sqlite3.Submit(ctx, &Command{
Statement: "BEGIN;",
Mode: "exec",
})
assert.Error(t, err)
assert.Empty(t, res)
res, err = sqlite3.Submit(ctx, &Command{
Statement: "BEGIN; CREATE TABLE tt (a INT); COMMIT;",
Mode: "exec",
Isolation: "none",
})
assert.NoError(t, err)
assert.Equal(t, sqlResult{0, 0}, res)
rows, err := sqlite3.Submit(ctx, &Command{
Statement: "SELECT * FROM tt;",
Mode: "query",
Isolation: "repeatable_read",
})
assert.NoError(t, err)
assert.Empty(t, rows)
}
func TestSubmitTimeout(t *testing.T) {
ctx, sqlite3 := helperSQLite3(t)
res, err := sqlite3.Submit(ctx, &Command{
Statement: "SELECT * FROM t;",
Mode: "query",
Timeout: 1 * time.Nanosecond,
})
assert.Error(t, err)
assert.Empty(t, res)
}
func TestSubmitMode(t *testing.T) {
ctx, sqlite3 := helperSQLite3(t)
res, err := sqlite3.Submit(ctx, &Command{
Statement: "SELECT * FROM t;",
Mode: "notanoption",
})
assert.Error(t, err)
assert.Empty(t, res)
}
func TestSubmitEmpty(t *testing.T) {
ctx, sqlite3 := helperSQLite3(t)
res, err := sqlite3.Submit(ctx, &Command{
Statement: "; ; ; ;",
Mode: "query",
})
assert.Error(t, err)
assert.Empty(t, res)
}
+4
View File
@@ -0,0 +1,4 @@
FROM golang:1.12 as builder
WORKDIR /go/src/github.com/cloudflare/cloudflared/
RUN apt-get update
COPY . .
+43
View File
@@ -94,6 +94,14 @@ type Header struct {
Name, Value string
}
func RPCHeaders() []Header {
return []Header{
{Name: ":method", Value: "RPC"},
{Name: ":scheme", Value: "capnp"},
{Name: ":path", Value: "*"},
}
}
// Handshake establishes a muxed connection with the peer.
// After the handshake completes, it is possible to open and accept streams.
func Handshake(
@@ -414,6 +422,41 @@ func (m *Muxer) OpenStream(ctx context.Context, headers []Header, body io.Reader
}
}
func (m *Muxer) OpenRPCStream(ctx context.Context) (*MuxedStream, error) {
stream := &MuxedStream{
responseHeadersReceived: make(chan struct{}),
readBuffer: NewSharedBuffer(),
writeBuffer: &bytes.Buffer{},
writeBufferMaxLen: m.config.StreamWriteBufferMaxLen,
writeBufferHasSpace: make(chan struct{}, 1),
receiveWindow: m.config.DefaultWindowSize,
receiveWindowCurrentMax: m.config.DefaultWindowSize,
receiveWindowMax: m.config.MaxWindowSize,
sendWindow: m.config.DefaultWindowSize,
readyList: m.readyList,
writeHeaders: RPCHeaders(),
dictionaries: m.muxReader.dictionaries,
}
select {
// Will be received by mux writer
case <-ctx.Done():
return nil, ErrOpenStreamTimeout
case <-m.abortChan:
return nil, ErrConnectionClosed
case m.newStreamChan <- MuxedStreamRequest{stream: stream, body: nil}:
}
select {
case <-ctx.Done():
return nil, ErrResponseHeadersTimeout
case <-m.abortChan:
return nil, ErrConnectionClosed
case <-stream.responseHeadersReceived:
return stream, nil
}
}
func (m *Muxer) Metrics() *MuxerMetrics {
return m.muxMetricsUpdater.metrics()
}
+31 -1
View File
@@ -68,7 +68,8 @@ type MuxedStream struct {
sentEOF bool
// true if the peer sent us an EOF
receivedEOF bool
// If valid, tunnelHostname is used to identify which origin service is the intended recipient of the request
tunnelHostname TunnelHostname
// Compression-related fields
receivedUseDict bool
method string
@@ -77,6 +78,16 @@ type MuxedStream struct {
dictionaries h2Dictionaries
}
type TunnelHostname string
func (th TunnelHostname) String() string {
return string(th)
}
func (th TunnelHostname) IsSet() bool {
return th != ""
}
func (s *MuxedStream) Read(p []byte) (n int, err error) {
var readBuffer ReadWriteClosedCloser
if s.dictionaries.read != nil {
@@ -185,6 +196,25 @@ func (s *MuxedStream) WriteHeaders(headers []Header) error {
return nil
}
// IsRPCStream returns if the stream is used to transport RPC.
func (s *MuxedStream) IsRPCStream() bool {
rpcHeaders := RPCHeaders()
if len(s.Headers) != len(rpcHeaders) {
return false
}
// The headers order matters, so RPC stream should be opened with OpenRPCStream method and let MuxWriter serializes the headers.
for i, rpcHeader := range rpcHeaders {
if s.Headers[i] != rpcHeader {
return false
}
}
return true
}
func (s *MuxedStream) TunnelHostname() TunnelHostname {
return s.tunnelHostname
}
func (s *MuxedStream) getReceiveWindow() uint32 {
s.writeLock.Lock()
defer s.writeLock.Unlock()
+27
View File
@@ -98,3 +98,30 @@ func TestMuxedStreamEOF(t *testing.T) {
assert.Equal(t, 0, n)
}
}
func TestIsRPCStream(t *testing.T) {
tests := []struct {
stream *MuxedStream
isRPCStream bool
}{
{
stream: &MuxedStream{},
isRPCStream: false,
},
{
stream: &MuxedStream{Headers: RPCHeaders()},
isRPCStream: true,
},
{
stream: &MuxedStream{Headers: []Header{
{Name: ":method", Value: "rpc"},
{Name: ":scheme", Value: "Capnp"},
{Name: ":path", Value: "/"},
}},
isRPCStream: false,
},
}
for _, test := range tests {
assert.Equal(t, test.isRPCStream, test.stream.IsRPCStream())
}
}
+6
View File
@@ -11,6 +11,10 @@ import (
"golang.org/x/net/http2"
)
const (
CloudflaredProxyTunnelHostnameHeader = "cf-cloudflared-proxy-tunnel-hostname"
)
type MuxReader struct {
// f is used to read HTTP2 frames.
f *http2.Framer
@@ -235,6 +239,8 @@ func (r *MuxReader) receiveHeaderData(frame *http2.MetaHeadersFrame) error {
if r.dictionaries.write != nil {
continue
}
case CloudflaredProxyTunnelHostnameHeader:
stream.tunnelHostname = TunnelHostname(header.Value)
}
headers = append(headers, Header{Name: header.Name, Value: header.Value})
}
+107
View File
@@ -0,0 +1,107 @@
package h2mux
import (
"context"
"testing"
"time"
"github.com/stretchr/testify/assert"
)
var (
methodHeader = Header{
Name: ":method",
Value: "GET",
}
schemeHeader = Header{
Name: ":scheme",
Value: "https",
}
pathHeader = Header{
Name: ":path",
Value: "/api/tunnels",
}
tunnelHostnameHeader = Header{
Name: CloudflaredProxyTunnelHostnameHeader,
Value: "tunnel.example.com",
}
respStatusHeader = Header{
Name: ":status",
Value: "200",
}
)
type mockOriginStreamHandler struct {
stream *MuxedStream
}
func (mosh *mockOriginStreamHandler) ServeStream(stream *MuxedStream) error {
mosh.stream = stream
// Echo tunnel hostname in header
stream.WriteHeaders([]Header{respStatusHeader})
return nil
}
func getCloudflaredProxyTunnelHostnameHeader(stream *MuxedStream) string {
for _, header := range stream.Headers {
if header.Name == CloudflaredProxyTunnelHostnameHeader {
return header.Value
}
}
return ""
}
func assertOpenStreamSucceed(t *testing.T, stream *MuxedStream, err error) {
assert.NoError(t, err)
assert.Len(t, stream.Headers, 1)
assert.Equal(t, respStatusHeader, stream.Headers[0])
}
func TestMissingHeaders(t *testing.T) {
originHandler := &mockOriginStreamHandler{}
muxPair := NewDefaultMuxerPair(t, originHandler.ServeStream)
muxPair.Serve(t)
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
reqHeaders := []Header{
{
Name: "content-type",
Value: "application/json",
},
}
// Request doesn't contain CloudflaredProxyTunnelHostnameHeader
stream, err := muxPair.EdgeMux.OpenStream(ctx, reqHeaders, nil)
assertOpenStreamSucceed(t, stream, err)
assert.Empty(t, originHandler.stream.method)
assert.Empty(t, originHandler.stream.path)
assert.False(t, originHandler.stream.TunnelHostname().IsSet())
}
func TestReceiveHeaderData(t *testing.T) {
originHandler := &mockOriginStreamHandler{}
muxPair := NewDefaultMuxerPair(t, originHandler.ServeStream)
muxPair.Serve(t)
reqHeaders := []Header{
methodHeader,
schemeHeader,
pathHeader,
tunnelHostnameHeader,
}
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
reqHeaders = append(reqHeaders, tunnelHostnameHeader)
stream, err := muxPair.EdgeMux.OpenStream(ctx, reqHeaders, nil)
assertOpenStreamSucceed(t, stream, err)
assert.Equal(t, methodHeader.Value, originHandler.stream.method)
assert.Equal(t, pathHeader.Value, originHandler.stream.path)
assert.True(t, originHandler.stream.TunnelHostname().IsSet())
assert.Equal(t, tunnelHostnameHeader.Value, originHandler.stream.TunnelHostname().String())
}
-19
View File
@@ -1,19 +0,0 @@
package origin
import (
"runtime"
)
type BuildInfo struct {
GoOS string `json:"go_os"`
GoVersion string `json:"go_version"`
GoArch string `json:"go_arch"`
}
func GetBuildInfo() *BuildInfo {
return &BuildInfo{
GoOS: runtime.GOOS,
GoVersion: runtime.Version(),
GoArch: runtime.GOARCH,
}
}
+19 -3
View File
@@ -6,6 +6,8 @@ import (
"net"
"time"
"github.com/sirupsen/logrus"
"github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/signal"
@@ -34,6 +36,8 @@ type Supervisor struct {
// currently-connecting tunnels to finish connecting so we can reset backoff timer
nextConnectedIndex int
nextConnectedSignal chan struct{}
logger *logrus.Entry
}
type resolveResult struct {
@@ -51,6 +55,7 @@ func NewSupervisor(config *TunnelConfig) *Supervisor {
config: config,
tunnelErrors: make(chan tunnelError),
tunnelsConnecting: map[int]chan struct{}{},
logger: config.Logger.WithField("subsystem", "supervisor"),
}
}
@@ -124,8 +129,10 @@ func (s *Supervisor) Run(ctx context.Context, connectedSignal *signal.Signal, u
}
func (s *Supervisor) initialize(ctx context.Context, connectedSignal *signal.Signal, u uuid.UUID) error {
logger := s.config.Logger
edgeIPs, err := connection.ResolveEdgeIPs(logger, s.config.EdgeAddrs)
logger := s.logger
edgeIPs, err := s.resolveEdgeIPs()
if err != nil {
logger.Infof("ResolveEdgeIPs err")
return err
@@ -215,6 +222,15 @@ func (s *Supervisor) getEdgeIP(index int) *net.TCPAddr {
return s.edgeIPs[index%len(s.edgeIPs)]
}
func (s *Supervisor) resolveEdgeIPs() ([]*net.TCPAddr, error) {
// If --edge is specfied, resolve edge server addresses
if len(s.config.EdgeAddrs) > 0 {
return connection.ResolveAddrs(s.config.EdgeAddrs)
}
// Otherwise lookup edge server addresses through service discovery
return connection.EdgeDiscovery(s.logger)
}
func (s *Supervisor) refreshEdgeIPs() {
if s.resolverC != nil {
return
@@ -224,7 +240,7 @@ func (s *Supervisor) refreshEdgeIPs() {
}
s.resolverC = make(chan resolveResult)
go func() {
edgeIPs, err := connection.ResolveEdgeIPs(s.config.Logger, s.config.EdgeAddrs)
edgeIPs, err := s.resolveEdgeIPs()
s.resolverC <- resolveResult{edgeIPs: edgeIPs, err: err}
}()
}
+8 -84
View File
@@ -14,9 +14,10 @@ import (
"sync"
"time"
"github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/cmd/cloudflared/buildinfo"
"github.com/cloudflare/cloudflared/h2mux"
"github.com/cloudflare/cloudflared/signal"
"github.com/cloudflare/cloudflared/streamhandler"
"github.com/cloudflare/cloudflared/tunnelrpc"
tunnelpogs "github.com/cloudflare/cloudflared/tunnelrpc/pogs"
"github.com/cloudflare/cloudflared/validation"
@@ -41,7 +42,7 @@ const (
)
type TunnelConfig struct {
BuildInfo *BuildInfo
BuildInfo *buildinfo.BuildInfo
ClientID string
ClientTlsConfig *tls.Config
CloseConnOnce *sync.Once // Used to close connectedSignal no more than once
@@ -139,44 +140,8 @@ func (c *TunnelConfig) RegistrationOptions(connectionID uint8, OriginLocalIP str
}
}
func StartTunnelDaemon(config *TunnelConfig, shutdownC <-chan struct{}, connectedSignal *signal.Signal) error {
ctx, cancel := context.WithCancel(context.Background())
go func() {
<-shutdownC
cancel()
}()
u, err := uuid.NewRandom()
if err != nil {
return err
}
// If a user specified negative HAConnections, we will treat it as requesting 1 connection
if config.HAConnections > 1 {
if config.UseDeclarativeTunnel {
return connection.NewSupervisor(&connection.CloudflaredConfig{
ConnectionConfig: &connection.ConnectionConfig{
TLSConfig: config.TlsConfig,
HeartbeatInterval: config.HeartbeatInterval,
MaxHeartbeats: config.MaxHeartbeats,
Logger: config.Logger.WithField("subsystem", "connection_supervisor"),
},
OriginCert: config.OriginCert,
Tags: config.Tags,
EdgeAddrs: config.EdgeAddrs,
HAConnections: uint(config.HAConnections),
Logger: config.Logger,
CloudflaredVersion: config.ReportedVersion,
}).Run(ctx)
}
return NewSupervisor(config).Run(ctx, connectedSignal, u)
} else {
addrs, err := connection.ResolveEdgeIPs(config.Logger, config.EdgeAddrs)
if err != nil {
return err
}
return ServeTunnelLoop(ctx, config, addrs[0], 0, connectedSignal, u)
}
func StartTunnelDaemon(ctx context.Context, config *TunnelConfig, connectedSignal *signal.Signal, cloudflaredID uuid.UUID) error {
return NewSupervisor(config).Run(ctx, connectedSignal, cloudflaredID)
}
func ServeTunnelLoop(ctx context.Context,
@@ -471,39 +436,6 @@ func LogServerInfo(
metrics.registerServerLocation(uint8ToString(connectionID), serverInfo.LocationName)
}
func H2RequestHeadersToH1Request(h2 []h2mux.Header, h1 *http.Request) error {
for _, header := range h2 {
switch header.Name {
case ":method":
h1.Method = header.Value
case ":scheme":
case ":authority":
// Otherwise the host header will be based on the origin URL
h1.Host = header.Value
case ":path":
u, err := url.Parse(header.Value)
if err != nil {
return fmt.Errorf("unparseable path")
}
resolved := h1.URL.ResolveReference(u)
// prevent escaping base URL
if !strings.HasPrefix(resolved.String(), h1.URL.String()) {
return fmt.Errorf("invalid path")
}
h1.URL = resolved
case "content-length":
contentLength, err := strconv.ParseInt(header.Value, 10, 64)
if err != nil {
return fmt.Errorf("unparseable content length")
}
h1.ContentLength = contentLength
default:
h1.Header.Add(http.CanonicalHeaderKey(header.Name), header.Value)
}
}
return nil
}
func H1ResponseToH2Response(h1 *http.Response) (h2 []h2mux.Header) {
h2 = []h2mux.Header{{Name: ":status", Value: fmt.Sprintf("%d", h1.StatusCode)}}
for headerName, headerValues := range h1.Header {
@@ -514,10 +446,6 @@ func H1ResponseToH2Response(h1 *http.Response) (h2 []h2mux.Header) {
return
}
func FindCfRayHeader(h1 *http.Request) string {
return h1.Header.Get("Cf-Ray")
}
type TunnelHandler struct {
originUrl string
muxer *h2mux.Muxer
@@ -605,8 +533,8 @@ func (h *TunnelHandler) ServeStream(stream *h2mux.MuxedStream) error {
return reqErr
}
cfRay := FindCfRayHeader(req)
lbProbe := isLBProbeRequest(req)
cfRay := streamhandler.FindCfRayHeader(req)
lbProbe := streamhandler.IsLBProbeRequest(req)
h.logRequest(req, cfRay, lbProbe)
var resp *http.Response
@@ -629,7 +557,7 @@ func (h *TunnelHandler) createRequest(stream *h2mux.MuxedStream) (*http.Request,
if err != nil {
return nil, errors.Wrap(err, "Unexpected error from http.NewRequest")
}
err = H2RequestHeadersToH1Request(stream.Headers, req)
err = streamhandler.H2RequestHeadersToH1Request(stream.Headers, req)
if err != nil {
return nil, errors.Wrap(err, "invalid request received")
}
@@ -759,10 +687,6 @@ func uint8ToString(input uint8) string {
return strconv.FormatUint(uint64(input), 10)
}
func isLBProbeRequest(req *http.Request) bool {
return strings.HasPrefix(req.UserAgent(), lbProbeUserAgentPrefix)
}
// Print out the given lines in a nice ASCII box.
func asciiBox(lines []string, padding int) (box []string) {
maxLen := maxLen(lines)
+235
View File
@@ -0,0 +1,235 @@
// Package client defines and implements interface to proxy to HTTP, websocket and hello world origins
package originservice
import (
"bufio"
"crypto/tls"
"fmt"
"io"
"net"
"net/http"
"net/url"
"strconv"
"strings"
"github.com/cloudflare/cloudflared/h2mux"
"github.com/cloudflare/cloudflared/hello"
"github.com/cloudflare/cloudflared/log"
"github.com/cloudflare/cloudflared/websocket"
"github.com/pkg/errors"
)
// OriginService is an interface to proxy requests to different type of origins
type OriginService interface {
Proxy(stream *h2mux.MuxedStream, req *http.Request) (resp *http.Response, err error)
URL() *url.URL
Summary() string
Shutdown()
}
// HTTPService talks to origin using HTTP/HTTPS
type HTTPService struct {
client http.RoundTripper
originURL *url.URL
chunkedEncoding bool
}
func NewHTTPService(transport http.RoundTripper, url *url.URL, chunkedEncoding bool) OriginService {
return &HTTPService{
client: transport,
originURL: url,
chunkedEncoding: chunkedEncoding,
}
}
func (hc *HTTPService) Proxy(stream *h2mux.MuxedStream, req *http.Request) (*http.Response, error) {
// Support for WSGI Servers by switching transfer encoding from chunked to gzip/deflate
if !hc.chunkedEncoding {
req.TransferEncoding = []string{"gzip", "deflate"}
cLength, err := strconv.Atoi(req.Header.Get("Content-Length"))
if err == nil {
req.ContentLength = int64(cLength)
}
}
// Request origin to keep connection alive to improve performance
req.Header.Set("Connection", "keep-alive")
resp, err := hc.client.RoundTrip(req)
if err != nil {
return nil, errors.Wrap(err, "error proxying request to HTTP origin")
}
defer resp.Body.Close()
err = stream.WriteHeaders(h1ResponseToH2Response(resp))
if err != nil {
return nil, errors.Wrap(err, "error writing response header to HTTP origin")
}
if isEventStream(resp) {
writeEventStream(stream, resp.Body)
} else {
// Use CopyBuffer, because Copy only allocates a 32KiB buffer, and cross-stream
// compression generates dictionary on first write
io.CopyBuffer(stream, resp.Body, make([]byte, 512*1024))
}
return resp, nil
}
func (hc *HTTPService) URL() *url.URL {
return hc.originURL
}
func (hc *HTTPService) Summary() string {
return fmt.Sprintf("HTTP service listening on %s", hc.originURL)
}
func (hc *HTTPService) Shutdown() {}
// WebsocketService talks to origin using WS/WSS
type WebsocketService struct {
tlsConfig *tls.Config
originURL *url.URL
shutdownC chan struct{}
}
func NewWebSocketService(tlsConfig *tls.Config, url *url.URL) (OriginService, error) {
listener, err := net.Listen("tcp", "127.0.0.1:")
if err != nil {
return nil, errors.Wrap(err, "cannot start Websocket Proxy Server")
}
shutdownC := make(chan struct{})
go func() {
websocket.StartProxyServer(log.CreateLogger(), listener, url.String(), shutdownC)
}()
return &WebsocketService{
tlsConfig: tlsConfig,
originURL: url,
shutdownC: shutdownC,
}, nil
}
func (wsc *WebsocketService) Proxy(stream *h2mux.MuxedStream, req *http.Request) (*http.Response, error) {
if !websocket.IsWebSocketUpgrade(req) {
return nil, fmt.Errorf("request is not a websocket connection")
}
conn, response, err := websocket.ClientConnect(req, wsc.tlsConfig)
if err != nil {
return nil, err
}
defer conn.Close()
err = stream.WriteHeaders(h1ResponseToH2Response(response))
if err != nil {
return nil, errors.Wrap(err, "error writing response header to websocket origin")
}
// Copy to/from stream to the undelying connection. Use the underlying
// connection because cloudflared doesn't operate on the message themselves
websocket.Stream(conn.UnderlyingConn(), stream)
return response, nil
}
func (wsc *WebsocketService) URL() *url.URL {
return wsc.originURL
}
func (wsc *WebsocketService) Summary() string {
return fmt.Sprintf("Websocket listening on %s", wsc.originURL)
}
func (wsc *WebsocketService) Shutdown() {
close(wsc.shutdownC)
}
// HelloWorldService talks to the hello world example origin
type HelloWorldService struct {
client http.RoundTripper
listener net.Listener
originURL *url.URL
shutdownC chan struct{}
}
func NewHelloWorldService(transport http.RoundTripper) (OriginService, error) {
listener, err := hello.CreateTLSListener("127.0.0.1:")
if err != nil {
return nil, errors.Wrap(err, "cannot start Hello World Server")
}
shutdownC := make(chan struct{})
go func() {
hello.StartHelloWorldServer(log.CreateLogger(), listener, shutdownC)
}()
return &HelloWorldService{
client: transport,
listener: listener,
originURL: &url.URL{
Scheme: "https",
Host: listener.Addr().String(),
},
shutdownC: shutdownC,
}, nil
}
func (hwc *HelloWorldService) Proxy(stream *h2mux.MuxedStream, req *http.Request) (*http.Response, error) {
// Request origin to keep connection alive to improve performance
req.Header.Set("Connection", "keep-alive")
resp, err := hwc.client.RoundTrip(req)
if err != nil {
return nil, errors.Wrap(err, "error proxying request to Hello World origin")
}
defer resp.Body.Close()
err = stream.WriteHeaders(h1ResponseToH2Response(resp))
if err != nil {
return nil, errors.Wrap(err, "error writing response header to Hello World origin")
}
// Use CopyBuffer, because Copy only allocates a 32KiB buffer, and cross-stream
// compression generates dictionary on first write
io.CopyBuffer(stream, resp.Body, make([]byte, 512*1024))
return resp, nil
}
func (hwc *HelloWorldService) URL() *url.URL {
return hwc.originURL
}
func (hwc *HelloWorldService) Summary() string {
return fmt.Sprintf("Hello World service listening on %s", hwc.originURL)
}
func (hwc *HelloWorldService) Shutdown() {
hwc.listener.Close()
}
func isEventStream(resp *http.Response) bool {
// Check if content-type is text/event-stream. We need to check if the header value starts with text/event-stream
// because text/event-stream; charset=UTF-8 is also valid
// Ref: https://tools.ietf.org/html/rfc7231#section-3.1.1.1
for _, contentType := range resp.Header["content-type"] {
if strings.HasPrefix(strings.ToLower(contentType), "text/event-stream") {
return true
}
}
return false
}
func writeEventStream(stream *h2mux.MuxedStream, respBody io.ReadCloser) {
reader := bufio.NewReader(respBody)
for {
line, err := reader.ReadBytes('\n')
if err != nil {
break
}
stream.Write(line)
}
}
func h1ResponseToH2Response(h1 *http.Response) (h2 []h2mux.Header) {
h2 = []h2mux.Header{{Name: ":status", Value: fmt.Sprintf("%d", h1.StatusCode)}}
for headerName, headerValues := range h1.Header {
for _, headerValue := range headerValues {
h2 = append(h2, h2mux.Header{Name: strings.ToLower(headerName), Value: headerValue})
}
}
return
}
+60
View File
@@ -0,0 +1,60 @@
package originservice
import (
"net/http"
"testing"
"github.com/stretchr/testify/assert"
)
func TestIsEventStream(t *testing.T) {
tests := []struct {
resp *http.Response
isEventStream bool
}{
{
resp: &http.Response{},
isEventStream: false,
},
{
// isEventStream checks all headers
resp: &http.Response{
Header: http.Header{
"accept": []string{"text/html"},
"content-type": []string{"text/event-stream"},
},
},
isEventStream: true,
},
{
// Content-Type and text/event-stream are case-insensitive. text/event-stream can be followed by OWS parameter
resp: &http.Response{
Header: http.Header{
"content-type": []string{"Text/event-stream;charset=utf-8"},
},
},
isEventStream: true,
},
{
// Content-Type and text/event-stream are case-insensitive. text/event-stream can be followed by OWS parameter
resp: &http.Response{
Header: http.Header{
"content-type": []string{"appication/json", "text/html", "Text/event-stream;charset=utf-8"},
},
},
isEventStream: true,
},
{
// Not an event stream because the content-type value doesn't start with text/event-stream
resp: &http.Response{
Header: http.Header{
"content-type": []string{" text/event-stream"},
},
},
isEventStream: false,
},
}
for _, test := range tests {
assert.Equal(t, test.isEventStream, isEventStream(test.resp), "Header: %v", test.resp.Header)
}
}
+1 -1
View File
@@ -18,9 +18,9 @@ import (
"github.com/cloudflare/cloudflared/cmd/cloudflared/config"
cfpath "github.com/cloudflare/cloudflared/cmd/cloudflared/path"
"github.com/coreos/go-oidc/jose"
homedir "github.com/mitchellh/go-homedir"
gossh "golang.org/x/crypto/ssh"
"gopkg.in/coreos/go-oidc.v1/jose"
)
const (
+1 -1
View File
@@ -16,8 +16,8 @@ import (
"github.com/cloudflare/cloudflared/cmd/cloudflared/config"
cfpath "github.com/cloudflare/cloudflared/cmd/cloudflared/path"
"github.com/coreos/go-oidc/jose"
"github.com/stretchr/testify/assert"
"gopkg.in/coreos/go-oidc.v1/jose"
)
const (
+66
View File
@@ -0,0 +1,66 @@
package streamhandler
import (
"fmt"
"net/http"
"net/url"
"strconv"
"strings"
"github.com/cloudflare/cloudflared/h2mux"
"github.com/pkg/errors"
)
const (
lbProbeUserAgentPrefix = "Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/;"
)
func FindCfRayHeader(h1 *http.Request) string {
return h1.Header.Get("Cf-Ray")
}
func IsLBProbeRequest(req *http.Request) bool {
return strings.HasPrefix(req.UserAgent(), lbProbeUserAgentPrefix)
}
func createRequest(stream *h2mux.MuxedStream, url *url.URL) (*http.Request, error) {
req, err := http.NewRequest(http.MethodGet, url.String(), h2mux.MuxedStreamReader{MuxedStream: stream})
if err != nil {
return nil, errors.Wrap(err, "unexpected error from http.NewRequest")
}
err = H2RequestHeadersToH1Request(stream.Headers, req)
if err != nil {
return nil, errors.Wrap(err, "invalid request received")
}
return req, nil
}
func H2RequestHeadersToH1Request(h2 []h2mux.Header, h1 *http.Request) error {
for _, header := range h2 {
switch header.Name {
case ":method":
h1.Method = header.Value
case ":scheme":
case ":authority":
// Otherwise the host header will be based on the origin URL
h1.Host = header.Value
case ":path":
// Separate the path and the (optional) query from each other
chunks := strings.SplitN(header.Value, "?", 2)
h1.URL.Path = chunks[0]
if len(chunks) > 1 {
h1.URL.RawQuery = chunks[1]
}
case "content-length":
contentLength, err := strconv.ParseInt(header.Value, 10, 64)
if err != nil {
return fmt.Errorf("unparseable content length")
}
h1.ContentLength = contentLength
default:
h1.Header.Add(http.CanonicalHeaderKey(header.Name), header.Value)
}
}
return nil
}
+104
View File
@@ -0,0 +1,104 @@
package streamhandler
import (
"net/http"
"testing"
"github.com/cloudflare/cloudflared/h2mux"
"github.com/stretchr/testify/assert"
)
func TestH2RequestHeadersToH1Request_RegularHeaders(t *testing.T) {
request, err := http.NewRequest(http.MethodGet, "http://example.com", nil)
assert.NoError(t, err)
headersConversionErr := H2RequestHeadersToH1Request(
[]h2mux.Header{
h2mux.Header{
Name: "Mock header 1",
Value: "Mock value 1",
},
h2mux.Header{
Name: "Mock header 2",
Value: "Mock value 2",
},
},
request,
)
assert.Equal(t, http.Header{
"Mock header 1": []string{"Mock value 1"},
"Mock header 2": []string{"Mock value 2"},
}, request.Header)
assert.NoError(t, headersConversionErr)
}
func TestH2RequestHeadersToH1Request_NoHeaders(t *testing.T) {
request, err := http.NewRequest(http.MethodGet, "http://example.com", nil)
assert.NoError(t, err)
headersConversionErr := H2RequestHeadersToH1Request(
[]h2mux.Header{},
request,
)
assert.Equal(t, http.Header{}, request.Header)
assert.NoError(t, headersConversionErr)
}
func TestH2RequestHeadersToH1Request_InvalidHostPath(t *testing.T) {
request, err := http.NewRequest(http.MethodGet, "http://example.com", nil)
assert.NoError(t, err)
headersConversionErr := H2RequestHeadersToH1Request(
[]h2mux.Header{
h2mux.Header{
Name: ":path",
Value: "//bad_path/",
},
h2mux.Header{
Name: "Mock header",
Value: "Mock value",
},
},
request,
)
assert.Equal(t, http.Header{
"Mock header": []string{"Mock value"},
}, request.Header)
assert.Equal(t, "http://example.com//bad_path/", request.URL.String())
assert.NoError(t, headersConversionErr)
}
func TestH2RequestHeadersToH1Request_HostPathWithQuery(t *testing.T) {
request, err := http.NewRequest(http.MethodGet, "http://example.com/", nil)
assert.NoError(t, err)
headersConversionErr := H2RequestHeadersToH1Request(
[]h2mux.Header{
h2mux.Header{
Name: ":path",
Value: "/?query",
},
h2mux.Header{
Name: "Mock header",
Value: "Mock value",
},
},
request,
)
assert.Equal(t, http.Header{
"Mock header": []string{"Mock value"},
}, request.Header)
assert.Equal(t, "http://example.com/?query", request.URL.String())
assert.NoError(t, headersConversionErr)
}
+183
View File
@@ -0,0 +1,183 @@
package streamhandler
import (
"context"
"fmt"
"net/http"
"strconv"
"github.com/cloudflare/cloudflared/h2mux"
"github.com/cloudflare/cloudflared/tunnelhostnamemapper"
"github.com/cloudflare/cloudflared/tunnelrpc"
"github.com/cloudflare/cloudflared/tunnelrpc/pogs"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
"zombiezen.com/go/capnproto2/rpc"
)
const (
statusPseudoHeader = ":status"
)
type httpErrorStatus struct {
status string
text []byte
}
var (
statusBadRequest = newHTTPErrorStatus(http.StatusBadRequest)
statusNotFound = newHTTPErrorStatus(http.StatusNotFound)
statusBadGateway = newHTTPErrorStatus(http.StatusBadGateway)
)
func newHTTPErrorStatus(status int) *httpErrorStatus {
return &httpErrorStatus{
status: strconv.Itoa(status),
text: []byte(http.StatusText(status)),
}
}
// StreamHandler handles new stream opened by the edge. The streams can be used to proxy requests or make RPC.
type StreamHandler struct {
// newConfigChan is a send-only channel to notify Supervisor of a new ClientConfig
newConfigChan chan<- *pogs.ClientConfig
// useConfigResultChan is a receive-only channel for Supervisor to communicate the result of applying a new ClientConfig
useConfigResultChan <-chan *pogs.UseConfigurationResult
// originMapper maps tunnel hostname to origin service
tunnelHostnameMapper *tunnelhostnamemapper.TunnelHostnameMapper
logger *logrus.Entry
}
// NewStreamHandler creates a new StreamHandler
func NewStreamHandler(newConfigChan chan<- *pogs.ClientConfig,
useConfigResultChan <-chan *pogs.UseConfigurationResult,
logger *logrus.Logger,
) *StreamHandler {
return &StreamHandler{
newConfigChan: newConfigChan,
useConfigResultChan: useConfigResultChan,
tunnelHostnameMapper: tunnelhostnamemapper.NewTunnelHostnameMapper(),
logger: logger.WithField("subsystem", "streamHandler"),
}
}
// UseConfiguration implements ClientService
func (s *StreamHandler) UseConfiguration(ctx context.Context, config *pogs.ClientConfig) (*pogs.UseConfigurationResult, error) {
select {
case <-ctx.Done():
err := fmt.Errorf("Timeout while sending new config to Supervisor")
s.logger.Error(err)
return nil, err
case s.newConfigChan <- config:
}
select {
case <-ctx.Done():
err := fmt.Errorf("Timeout applying new configuration")
s.logger.Error(err)
return nil, err
case result := <-s.useConfigResultChan:
return result, nil
}
}
// UpdateConfig replaces current originmapper mapping with mappings from newConfig
func (s *StreamHandler) UpdateConfig(newConfig []*pogs.ReverseProxyConfig) (failedConfigs []*pogs.FailedConfig) {
// TODO: TUN-1968: Gracefully apply new config
s.tunnelHostnameMapper.DeleteAll()
for _, tunnelConfig := range newConfig {
tunnelHostname := tunnelConfig.TunnelHostname
originSerice, err := tunnelConfig.OriginConfigJSONHandler.OriginConfig.Service()
if err != nil {
s.logger.WithField("tunnelHostname", tunnelHostname).WithError(err).Error("Invalid origin service config")
failedConfigs = append(failedConfigs, &pogs.FailedConfig{
Config: tunnelConfig,
Reason: tunnelConfig.FailReason(err),
})
continue
}
s.tunnelHostnameMapper.Add(tunnelConfig.TunnelHostname, originSerice)
s.logger.WithField("tunnelHostname", tunnelHostname).Infof("New origin service config: %v", originSerice.Summary())
}
return
}
// ServeStream implements MuxedStreamHandler interface
func (s *StreamHandler) ServeStream(stream *h2mux.MuxedStream) error {
if stream.IsRPCStream() {
return s.serveRPC(stream)
}
if err := s.serveRequest(stream); err != nil {
s.logger.Error(err)
return err
}
return nil
}
func (s *StreamHandler) serveRPC(stream *h2mux.MuxedStream) error {
stream.WriteHeaders([]h2mux.Header{{Name: ":status", Value: "200"}})
main := pogs.ClientService_ServerToClient(s)
rpcLogger := s.logger.WithField("subsystem", "clientserver-rpc")
rpcConn := rpc.NewConn(
tunnelrpc.NewTransportLogger(rpcLogger, rpc.StreamTransport(stream)),
rpc.MainInterface(main.Client),
tunnelrpc.ConnLog(s.logger.WithField("subsystem", "clientserver-rpc-transport")),
)
return rpcConn.Wait()
}
func (s *StreamHandler) serveRequest(stream *h2mux.MuxedStream) error {
tunnelHostname := stream.TunnelHostname()
if !tunnelHostname.IsSet() {
s.writeErrorStatus(stream, statusBadRequest)
return fmt.Errorf("stream doesn't have tunnelHostname")
}
originService, ok := s.tunnelHostnameMapper.Get(tunnelHostname)
if !ok {
s.writeErrorStatus(stream, statusNotFound)
return fmt.Errorf("cannot map tunnel hostname %s to origin", tunnelHostname)
}
req, err := createRequest(stream, originService.URL())
if err != nil {
s.writeErrorStatus(stream, statusBadRequest)
return errors.Wrap(err, "cannot create request")
}
logger := s.requestLogger(req, tunnelHostname)
logger.Debugf("Request Headers %+v", req.Header)
resp, err := originService.Proxy(stream, req)
if err != nil {
s.writeErrorStatus(stream, statusBadGateway)
return errors.Wrap(err, "cannot proxy request")
}
logger.WithField("status", resp.Status).Debugf("Response Headers %+v", resp.Header)
return nil
}
func (s *StreamHandler) requestLogger(req *http.Request, tunnelHostname h2mux.TunnelHostname) *logrus.Entry {
cfRay := FindCfRayHeader(req)
lbProbe := IsLBProbeRequest(req)
logger := s.logger.WithField("tunnelHostname", tunnelHostname)
if cfRay != "" {
logger = logger.WithField("CF-RAY", cfRay)
logger.Debugf("%s %s %s", req.Method, req.URL, req.Proto)
} else if lbProbe {
logger.Debugf("Load Balancer health check %s %s %s", req.Method, req.URL, req.Proto)
} else {
logger.Warnf("Requests %v does not have CF-RAY header. Please open a support ticket with Cloudflare.", req)
}
return logger
}
func (s *StreamHandler) writeErrorStatus(stream *h2mux.MuxedStream, status *httpErrorStatus) {
stream.WriteHeaders([]h2mux.Header{
{
Name: statusPseudoHeader,
Value: status.status,
},
})
stream.Write(status.text)
}
+227
View File
@@ -0,0 +1,227 @@
package streamhandler
import (
"context"
"io"
"net"
"net/http"
"net/http/httptest"
"strconv"
"sync"
"testing"
"time"
"github.com/cloudflare/cloudflared/h2mux"
"github.com/cloudflare/cloudflared/tunnelrpc/pogs"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert"
"golang.org/x/sync/errgroup"
)
const (
testOpenStreamTimeout = time.Millisecond * 5000
testHandshakeTimeout = time.Millisecond * 1000
)
var (
testTunnelHostname = h2mux.TunnelHostname("123.cftunnel.com")
baseHeaders = []h2mux.Header{
{Name: ":method", Value: "GET"},
{Name: ":scheme", Value: "http"},
{Name: ":authority", Value: "example.com"},
{Name: ":path", Value: "/"},
}
tunnelHostnameHeader = h2mux.Header{
Name: h2mux.CloudflaredProxyTunnelHostnameHeader,
Value: testTunnelHostname.String(),
}
)
func TestServeRequest(t *testing.T) {
configChan := make(chan *pogs.ClientConfig)
useConfigResultChan := make(chan *pogs.UseConfigurationResult)
streamHandler := NewStreamHandler(configChan, useConfigResultChan, logrus.New())
message := []byte("Hello cloudflared")
httpServer := httptest.NewServer(&mockHTTPHandler{message})
reverseProxyConfigs := []*pogs.ReverseProxyConfig{
{
TunnelHostname: testTunnelHostname,
OriginConfigJSONHandler: &pogs.OriginConfigJSONHandler{
OriginConfig: &pogs.HTTPOriginConfig{
URLString: httpServer.URL,
},
},
},
}
streamHandler.UpdateConfig(reverseProxyConfigs)
muxPair := NewDefaultMuxerPair(t, streamHandler)
muxPair.Serve(t)
ctx, cancel := context.WithTimeout(context.Background(), testOpenStreamTimeout)
defer cancel()
headers := append(baseHeaders, tunnelHostnameHeader)
stream, err := muxPair.EdgeMux.OpenStream(ctx, headers, nil)
assert.NoError(t, err)
assertStatusHeader(t, http.StatusOK, stream.Headers)
assertRespBody(t, message, stream)
}
func TestServeBadRequest(t *testing.T) {
configChan := make(chan *pogs.ClientConfig)
useConfigResultChan := make(chan *pogs.UseConfigurationResult)
streamHandler := NewStreamHandler(configChan, useConfigResultChan, logrus.New())
muxPair := NewDefaultMuxerPair(t, streamHandler)
muxPair.Serve(t)
ctx, cancel := context.WithTimeout(context.Background(), testOpenStreamTimeout)
defer cancel()
// No tunnel hostname header, expect to get 400 Bad Request
stream, err := muxPair.EdgeMux.OpenStream(ctx, baseHeaders, nil)
assert.NoError(t, err)
assertStatusHeader(t, http.StatusBadRequest, stream.Headers)
assertRespBody(t, statusBadRequest.text, stream)
// No mapping for the tunnel hostname, expect to get 404 Not Found
headers := append(baseHeaders, tunnelHostnameHeader)
stream, err = muxPair.EdgeMux.OpenStream(ctx, headers, nil)
assert.NoError(t, err)
assertStatusHeader(t, http.StatusNotFound, stream.Headers)
assertRespBody(t, statusNotFound.text, stream)
// Nothing listening on empty url, so proxy would fail. Expect to get 502 Bad Gateway
reverseProxyConfigs := []*pogs.ReverseProxyConfig{
{
TunnelHostname: testTunnelHostname,
OriginConfigJSONHandler: &pogs.OriginConfigJSONHandler{
OriginConfig: &pogs.HTTPOriginConfig{
URLString: "",
},
},
},
}
streamHandler.UpdateConfig(reverseProxyConfigs)
stream, err = muxPair.EdgeMux.OpenStream(ctx, headers, nil)
assert.NoError(t, err)
assertStatusHeader(t, http.StatusBadGateway, stream.Headers)
assertRespBody(t, statusBadGateway.text, stream)
// Invalid content-length, wouldn't not be able to create a request. Expect to get 400 Bad Request
headers = append(headers, h2mux.Header{
Name: "content-length",
Value: "x",
})
stream, err = muxPair.EdgeMux.OpenStream(ctx, headers, nil)
assert.NoError(t, err)
assertStatusHeader(t, http.StatusBadRequest, stream.Headers)
assertRespBody(t, statusBadRequest.text, stream)
}
func assertStatusHeader(t *testing.T, expectedStatus int, headers []h2mux.Header) {
assert.Equal(t, statusPseudoHeader, headers[0].Name)
assert.Equal(t, strconv.Itoa(expectedStatus), headers[0].Value)
}
func assertRespBody(t *testing.T, expectedRespBody []byte, stream *h2mux.MuxedStream) {
respBody := make([]byte, len(expectedRespBody))
_, err := stream.Read(respBody)
assert.NoError(t, err)
assert.Equal(t, expectedRespBody, respBody)
}
type DefaultMuxerPair struct {
OriginMuxConfig h2mux.MuxerConfig
OriginMux *h2mux.Muxer
OriginConn net.Conn
EdgeMuxConfig h2mux.MuxerConfig
EdgeMux *h2mux.Muxer
EdgeConn net.Conn
doneC chan struct{}
}
func NewDefaultMuxerPair(t assert.TestingT, h h2mux.MuxedStreamHandler) *DefaultMuxerPair {
origin, edge := net.Pipe()
p := &DefaultMuxerPair{
OriginMuxConfig: h2mux.MuxerConfig{
Timeout: testHandshakeTimeout,
Handler: h,
IsClient: true,
Name: "origin",
Logger: logrus.NewEntry(logrus.New()),
DefaultWindowSize: (1 << 8) - 1,
MaxWindowSize: (1 << 15) - 1,
StreamWriteBufferMaxLen: 1024,
},
OriginConn: origin,
EdgeMuxConfig: h2mux.MuxerConfig{
Timeout: testHandshakeTimeout,
IsClient: false,
Name: "edge",
Logger: logrus.NewEntry(logrus.New()),
DefaultWindowSize: (1 << 8) - 1,
MaxWindowSize: (1 << 15) - 1,
StreamWriteBufferMaxLen: 1024,
},
EdgeConn: edge,
doneC: make(chan struct{}),
}
assert.NoError(t, p.Handshake())
return p
}
func (p *DefaultMuxerPair) Handshake() error {
ctx, cancel := context.WithTimeout(context.Background(), testHandshakeTimeout)
defer cancel()
errGroup, _ := errgroup.WithContext(ctx)
errGroup.Go(func() (err error) {
p.EdgeMux, err = h2mux.Handshake(p.EdgeConn, p.EdgeConn, p.EdgeMuxConfig)
return errors.Wrap(err, "edge handshake failure")
})
errGroup.Go(func() (err error) {
p.OriginMux, err = h2mux.Handshake(p.OriginConn, p.OriginConn, p.OriginMuxConfig)
return errors.Wrap(err, "origin handshake failure")
})
return errGroup.Wait()
}
func (p *DefaultMuxerPair) Serve(t assert.TestingT) {
ctx := context.Background()
var wg sync.WaitGroup
wg.Add(2)
go func() {
err := p.EdgeMux.Serve(ctx)
if err != nil && err != io.EOF && err != io.ErrClosedPipe {
t.Errorf("error in edge muxer Serve(): %s", err)
}
p.OriginMux.Shutdown()
wg.Done()
}()
go func() {
err := p.OriginMux.Serve(ctx)
if err != nil && err != io.EOF && err != io.ErrClosedPipe {
t.Errorf("error in origin muxer Serve(): %s", err)
}
p.EdgeMux.Shutdown()
wg.Done()
}()
go func() {
// notify when both muxes have stopped serving
wg.Wait()
close(p.doneC)
}()
}
type mockHTTPHandler struct {
message []byte
}
func (mth *mockHTTPHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
w.Write(mth.message)
}
+179
View File
@@ -0,0 +1,179 @@
package supervisor
import (
"context"
"crypto/tls"
"fmt"
"os"
"os/signal"
"sync"
"syscall"
"golang.org/x/sync/errgroup"
"github.com/cloudflare/cloudflared/cmd/cloudflared/updater"
"github.com/cloudflare/cloudflared/connection"
"github.com/cloudflare/cloudflared/h2mux"
"github.com/cloudflare/cloudflared/streamhandler"
"github.com/cloudflare/cloudflared/tunnelrpc/pogs"
"github.com/sirupsen/logrus"
)
type Supervisor struct {
connManager *connection.EdgeManager
streamHandler *streamhandler.StreamHandler
autoupdater *updater.AutoUpdater
supportAutoupdate bool
newConfigChan <-chan *pogs.ClientConfig
useConfigResultChan chan<- *pogs.UseConfigurationResult
state *state
logger *logrus.Entry
}
func NewSupervisor(
defaultClientConfig *pogs.ClientConfig,
userCredential []byte,
tlsConfig *tls.Config,
serviceDiscoverer connection.EdgeServiceDiscoverer,
cloudflaredConfig *connection.CloudflaredConfig,
autoupdater *updater.AutoUpdater,
supportAutoupdate bool,
logger *logrus.Logger,
) (*Supervisor, error) {
newConfigChan := make(chan *pogs.ClientConfig)
useConfigResultChan := make(chan *pogs.UseConfigurationResult)
streamHandler := streamhandler.NewStreamHandler(newConfigChan, useConfigResultChan, logger)
invalidConfigs := streamHandler.UpdateConfig(defaultClientConfig.ReverseProxyConfigs)
if len(invalidConfigs) > 0 {
for _, invalidConfig := range invalidConfigs {
logger.Errorf("Tunnel %+v is invalid, reason: %s", invalidConfig.Config, invalidConfig.Reason)
}
return nil, fmt.Errorf("At least 1 Tunnel config is invalid")
}
tunnelHostnames := make([]h2mux.TunnelHostname, len(defaultClientConfig.ReverseProxyConfigs))
for i, reverseProxyConfig := range defaultClientConfig.ReverseProxyConfigs {
tunnelHostnames[i] = reverseProxyConfig.TunnelHostname
}
defaultEdgeMgrConfigurable := &connection.EdgeManagerConfigurable{
TunnelHostnames: tunnelHostnames,
EdgeConnectionConfig: defaultClientConfig.EdgeConnectionConfig,
}
return &Supervisor{
connManager: connection.NewEdgeManager(streamHandler, defaultEdgeMgrConfigurable, userCredential, tlsConfig,
serviceDiscoverer, cloudflaredConfig, logger),
streamHandler: streamHandler,
autoupdater: autoupdater,
supportAutoupdate: supportAutoupdate,
newConfigChan: newConfigChan,
useConfigResultChan: useConfigResultChan,
state: newState(defaultClientConfig),
logger: logger.WithField("subsystem", "supervisor"),
}, nil
}
func (s *Supervisor) Run(ctx context.Context) error {
errGroup, groupCtx := errgroup.WithContext(ctx)
errGroup.Go(func() error {
return s.connManager.Run(groupCtx)
})
errGroup.Go(func() error {
return s.listenToNewConfig(groupCtx)
})
errGroup.Go(func() error {
return s.listenToShutdownSignal(groupCtx)
})
if s.supportAutoupdate {
errGroup.Go(func() error {
return s.autoupdater.Run(groupCtx)
})
}
err := errGroup.Wait()
s.logger.Warnf("Supervisor terminated, reason: %v", err)
return err
}
func (s *Supervisor) listenToShutdownSignal(serveCtx context.Context) error {
signals := make(chan os.Signal, 10)
signal.Notify(signals, syscall.SIGTERM, syscall.SIGINT)
defer signal.Stop(signals)
select {
case <-serveCtx.Done():
return serveCtx.Err()
case sig := <-signals:
return fmt.Errorf("received %v signal", sig)
}
}
func (s *Supervisor) listenToNewConfig(ctx context.Context) error {
for {
select {
case <-ctx.Done():
return ctx.Err()
case newConfig := <-s.newConfigChan:
s.useConfigResultChan <- s.notifySubsystemsNewConfig(newConfig)
}
}
}
func (s *Supervisor) notifySubsystemsNewConfig(newConfig *pogs.ClientConfig) *pogs.UseConfigurationResult {
s.logger.Infof("Received configuration %v", newConfig.Version)
if s.state.hasAppliedVersion(newConfig.Version) {
s.logger.Infof("%v has been applied", newConfig.Version)
return &pogs.UseConfigurationResult{
Success: true,
}
}
s.state.updateConfig(newConfig)
var tunnelHostnames []h2mux.TunnelHostname
for _, tunnelConfig := range newConfig.ReverseProxyConfigs {
tunnelHostnames = append(tunnelHostnames, tunnelConfig.TunnelHostname)
}
// Update connManager configurable
s.connManager.UpdateConfigurable(&connection.EdgeManagerConfigurable{
TunnelHostnames: tunnelHostnames,
EdgeConnectionConfig: newConfig.EdgeConnectionConfig,
})
// Update streamHandler tunnelHostnameMapper mapping
failedConfigs := s.streamHandler.UpdateConfig(newConfig.ReverseProxyConfigs)
if s.supportAutoupdate {
s.autoupdater.Update(newConfig.SupervisorConfig.AutoUpdateFrequency)
}
return &pogs.UseConfigurationResult{
Success: len(failedConfigs) == 0,
FailedConfigs: failedConfigs,
}
}
type state struct {
sync.RWMutex
currentConfig *pogs.ClientConfig
}
func newState(currentConfig *pogs.ClientConfig) *state {
return &state{
currentConfig: currentConfig,
}
}
func (s *state) hasAppliedVersion(incomingVersion pogs.Version) bool {
s.RLock()
defer s.RUnlock()
return s.currentConfig.Version.IsNewerOrEqual(incomingVersion)
}
func (s *state) updateConfig(newConfig *pogs.ClientConfig) {
s.Lock()
defer s.Unlock()
s.currentConfig = newConfig
}
+148
View File
@@ -2,10 +2,22 @@ package tlsconfig
import (
"crypto/tls"
"crypto/x509"
"fmt"
"io/ioutil"
"net"
"runtime"
"sync"
"github.com/getsentry/raven-go"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
"gopkg.in/urfave/cli.v2"
)
const (
OriginCAPoolFlag = "origin-ca-pool"
CaCertFlag = "cacert"
)
// CertReloader can load and reload a TLS certificate from a particular filepath.
@@ -51,3 +63,139 @@ func (cr *CertReloader) LoadCert() error {
cr.certificate = &cert
return nil
}
func LoadOriginCA(c *cli.Context, logger *logrus.Logger) (*x509.CertPool, error) {
var originCustomCAPool []byte
originCAPoolFilename := c.String(OriginCAPoolFlag)
if originCAPoolFilename != "" {
var err error
originCustomCAPool, err = ioutil.ReadFile(originCAPoolFilename)
if err != nil {
return nil, errors.Wrap(err, fmt.Sprintf("unable to read the file %s for --%s", originCAPoolFilename, OriginCAPoolFlag))
}
}
originCertPool, err := loadOriginCertPool(originCustomCAPool, logger)
if err != nil {
return nil, errors.Wrap(err, "error loading the certificate pool")
}
// Windows users should be notified that they can use the flag
if runtime.GOOS == "windows" && originCAPoolFilename == "" {
logger.Infof("cloudflared does not support loading the system root certificate pool on Windows. Please use the --%s to specify it", OriginCAPoolFlag)
}
return originCertPool, nil
}
func LoadCustomOriginCA(originCAFilename string) (*x509.CertPool, error) {
// First, obtain the system certificate pool
certPool, err := x509.SystemCertPool()
if err != nil {
certPool = x509.NewCertPool()
}
// Next, append the Cloudflare CAs into the system pool
cfRootCA, err := GetCloudflareRootCA()
if err != nil {
return nil, errors.Wrap(err, "could not append Cloudflare Root CAs to cloudflared certificate pool")
}
for _, cert := range cfRootCA {
certPool.AddCert(cert)
}
if originCAFilename == "" {
return certPool, nil
}
customOriginCA, err := ioutil.ReadFile(originCAFilename)
if err != nil {
return nil, errors.Wrap(err, fmt.Sprintf("unable to read the file %s", originCAFilename))
}
if !certPool.AppendCertsFromPEM(customOriginCA) {
return nil, fmt.Errorf("error appending custom CA to cert pool")
}
return certPool, nil
}
func CreateTunnelConfig(c *cli.Context) (*tls.Config, error) {
var rootCAs []string
if c.String(CaCertFlag) != "" {
rootCAs = append(rootCAs, c.String(CaCertFlag))
}
userConfig := &TLSParameters{RootCAs: rootCAs}
tlsConfig, err := GetConfig(userConfig)
if err != nil {
return nil, err
}
if tlsConfig.RootCAs == nil {
rootCAPool := x509.NewCertPool()
cfRootCA, err := GetCloudflareRootCA()
if err != nil {
return nil, errors.Wrap(err, "could not append Cloudflare Root CAs to cloudflared certificate pool")
}
for _, cert := range cfRootCA {
rootCAPool.AddCert(cert)
}
tlsConfig.RootCAs = rootCAPool
tlsConfig.ServerName = "cftunnel.com"
} else if edgeAddrs := c.StringSlice("edge"); len(edgeAddrs) > 0 {
// Set for development environments and for testing specific origintunneld instances
tlsConfig.ServerName, _, _ = net.SplitHostPort(edgeAddrs[0])
}
if tlsConfig.ServerName == "" && !tlsConfig.InsecureSkipVerify {
return nil, fmt.Errorf("either ServerName or InsecureSkipVerify must be specified in the tls.Config")
}
return tlsConfig, nil
}
func loadOriginCertPool(originCAPoolPEM []byte, logger *logrus.Logger) (*x509.CertPool, error) {
// Get the global pool
certPool, err := loadGlobalCertPool(logger)
if err != nil {
return nil, err
}
// Then, add any custom origin CA pool the user may have passed
if originCAPoolPEM != nil {
if !certPool.AppendCertsFromPEM(originCAPoolPEM) {
logger.Warn("could not append the provided origin CA to the cloudflared certificate pool")
}
}
return certPool, nil
}
func loadGlobalCertPool(logger *logrus.Logger) (*x509.CertPool, error) {
// First, obtain the system certificate pool
certPool, err := x509.SystemCertPool()
if err != nil {
if runtime.GOOS != "windows" { // See https://github.com/golang/go/issues/16736
logger.WithError(err).Warn("error obtaining the system certificates")
}
certPool = x509.NewCertPool()
}
// Next, append the Cloudflare CAs into the system pool
cfRootCA, err := GetCloudflareRootCA()
if err != nil {
return nil, errors.Wrap(err, "could not append Cloudflare Root CAs to cloudflared certificate pool")
}
for _, cert := range cfRootCA {
certPool.AddCert(cert)
}
// Finally, add the Hello certificate into the pool (since it's self-signed)
helloCert, err := GetHelloCertificateX509()
if err != nil {
return nil, errors.Wrap(err, "could not append Hello server certificate to cloudflared certificate pool")
}
certPool.AddCert(helloCert)
return certPool, nil
}
+1 -1
View File
@@ -82,7 +82,7 @@ func (u *UpstreamHTTPS) exchangeWireformat(msg []byte) ([]byte, error) {
return nil, errors.Wrap(err, "failed to create an HTTPS request")
}
req.Header.Add("Content-Type", "application/dns-udpwireformat")
req.Header.Add("Content-Type", "application/dns-message")
req.Host = u.endpoint.Hostname()
resp, err := u.client.Do(req)
@@ -0,0 +1,49 @@
package tunnelhostnamemapper
import (
"sync"
"github.com/cloudflare/cloudflared/h2mux"
"github.com/cloudflare/cloudflared/originservice"
)
// TunnelHostnameMapper maps TunnelHostname to an OriginService
type TunnelHostnameMapper struct {
sync.RWMutex
tunnelHostnameToOrigin map[h2mux.TunnelHostname]originservice.OriginService
}
func NewTunnelHostnameMapper() *TunnelHostnameMapper {
return &TunnelHostnameMapper{
tunnelHostnameToOrigin: make(map[h2mux.TunnelHostname]originservice.OriginService),
}
}
// Get an OriginService given a TunnelHostname
func (om *TunnelHostnameMapper) Get(key h2mux.TunnelHostname) (originservice.OriginService, bool) {
om.RLock()
defer om.RUnlock()
originService, ok := om.tunnelHostnameToOrigin[key]
return originService, ok
}
// Add a mapping. If there is already an OriginService with this key, shutdown the old origin service and replace it
// with the new one
func (om *TunnelHostnameMapper) Add(key h2mux.TunnelHostname, os originservice.OriginService) {
om.Lock()
defer om.Unlock()
if oldOS, ok := om.tunnelHostnameToOrigin[key]; ok {
oldOS.Shutdown()
}
om.tunnelHostnameToOrigin[key] = os
}
// DeleteAll mappings, and shutdown all OriginService
func (om *TunnelHostnameMapper) DeleteAll() {
om.Lock()
defer om.Unlock()
for key, os := range om.tunnelHostnameToOrigin {
os.Shutdown()
delete(om.tunnelHostnameToOrigin, key)
}
}
@@ -0,0 +1,74 @@
package tunnelhostnamemapper
import (
"fmt"
"net/http"
"net/url"
"sync"
"testing"
"github.com/cloudflare/cloudflared/h2mux"
"github.com/cloudflare/cloudflared/originservice"
"github.com/stretchr/testify/assert"
)
const (
routines = 1000
)
func TestTunnelHostnameMapperConcurrentAccess(t *testing.T) {
thm := NewTunnelHostnameMapper()
concurrentOps(t, func(i int) {
// om is empty
os, ok := thm.Get(tunnelHostname(i))
assert.False(t, ok)
assert.Nil(t, os)
})
firstURL, err := url.Parse("https://127.0.0.1:8080")
assert.NoError(t, err)
httpOS := originservice.NewHTTPService(http.DefaultTransport, firstURL, false)
concurrentOps(t, func(i int) {
thm.Add(tunnelHostname(i), httpOS)
})
concurrentOps(t, func(i int) {
os, ok := thm.Get(tunnelHostname(i))
assert.True(t, ok)
assert.Equal(t, httpOS, os)
})
secondURL, err := url.Parse("https://127.0.0.1:8080")
assert.NoError(t, err)
secondHTTPOS := originservice.NewHTTPService(http.DefaultTransport, secondURL, true)
concurrentOps(t, func(i int) {
// Add should httpOS with secondHTTPOS
thm.Add(tunnelHostname(i), secondHTTPOS)
})
concurrentOps(t, func(i int) {
os, ok := thm.Get(tunnelHostname(i))
assert.True(t, ok)
assert.Equal(t, secondHTTPOS, os)
})
thm.DeleteAll()
assert.Empty(t, thm.tunnelHostnameToOrigin)
}
func concurrentOps(t *testing.T, f func(i int)) {
var wg sync.WaitGroup
wg.Add(routines)
for i := 0; i < routines; i++ {
go func(i int) {
f(i)
wg.Done()
}(i)
}
wg.Wait()
}
func tunnelHostname(i int) h2mux.TunnelHostname {
return h2mux.TunnelHostname(fmt.Sprintf("%d.cftunnel.com", i))
}
+499 -207
View File
@@ -2,49 +2,136 @@ package pogs
import (
"context"
"crypto/tls"
"crypto/x509"
"encoding/json"
"fmt"
"net"
"net/http"
"net/url"
"time"
"github.com/cloudflare/cloudflared/h2mux"
"github.com/cloudflare/cloudflared/originservice"
"github.com/cloudflare/cloudflared/tlsconfig"
"github.com/cloudflare/cloudflared/tunnelrpc"
"github.com/pkg/errors"
capnp "zombiezen.com/go/capnproto2"
"zombiezen.com/go/capnproto2/pogs"
"zombiezen.com/go/capnproto2/rpc"
"zombiezen.com/go/capnproto2/server"
)
///
/// Structs
///
// ClientConfig is a collection of FallibleConfig that determines how cloudflared should function
type ClientConfig struct {
Version uint64
AutoUpdateFrequency time.Duration
MetricsUpdateFrequency time.Duration
HeartbeatInterval time.Duration
MaxFailedHeartbeats uint64
GracePeriod time.Duration
DoHProxyConfigs []*DoHProxyConfig
ReverseProxyConfigs []*ReverseProxyConfig
NumHAConnections uint8
Version Version `json:"version"`
SupervisorConfig *SupervisorConfig `json:"supervisor_config"`
EdgeConnectionConfig *EdgeConnectionConfig `json:"edge_connection_config"`
DoHProxyConfigs []*DoHProxyConfig `json:"doh_proxy_configs"`
ReverseProxyConfigs []*ReverseProxyConfig `json:"reverse_proxy_configs"`
}
type UseConfigurationResult struct {
Success bool
ErrorMessage string
// Version type models the version of a ClientConfig
type Version uint64
func InitVersion() Version {
return Version(0)
}
func (v Version) IsNewerOrEqual(comparedVersion Version) bool {
return v >= comparedVersion
}
func (v Version) String() string {
return fmt.Sprintf("Version: %d", v)
}
// FallibleConfig is an interface implemented by configs that cloudflared might not be able to apply
type FallibleConfig interface {
FailReason(err error) string
jsonType() string
}
// SupervisorConfig specifies config of components managed by Supervisor other than ConnectionManager
type SupervisorConfig struct {
AutoUpdateFrequency time.Duration `json:"auto_update_frequency"`
MetricsUpdateFrequency time.Duration `json:"metrics_update_frequency"`
GracePeriod time.Duration `json:"grace_period"`
}
// FailReason impelents FallibleConfig interface for SupervisorConfig
func (sc *SupervisorConfig) FailReason(err error) string {
return fmt.Sprintf("Cannot apply SupervisorConfig, err: %v", err)
}
func (sc *SupervisorConfig) MarshalJSON() ([]byte, error) {
marshaler := make(map[string]SupervisorConfig, 1)
marshaler[sc.jsonType()] = *sc
return json.Marshal(marshaler)
}
func (sc *SupervisorConfig) jsonType() string {
return "supervisor_config"
}
// EdgeConnectionConfig specifies what parameters and how may connections should ConnectionManager establish with edge
type EdgeConnectionConfig struct {
NumHAConnections uint8 `json:"num_ha_connections"`
HeartbeatInterval time.Duration `json:"heartbeat_interval"`
Timeout time.Duration `json:"timeout"`
MaxFailedHeartbeats uint64 `json:"max_failed_heartbeats"`
UserCredentialPath string `json:"user_credential_path"`
}
// FailReason impelents FallibleConfig interface for EdgeConnectionConfig
func (cmc *EdgeConnectionConfig) FailReason(err error) string {
return fmt.Sprintf("Cannot apply EdgeConnectionConfig, err: %v", err)
}
func (cmc *EdgeConnectionConfig) MarshalJSON() ([]byte, error) {
marshaler := make(map[string]EdgeConnectionConfig, 1)
marshaler[cmc.jsonType()] = *cmc
return json.Marshal(marshaler)
}
func (cmc *EdgeConnectionConfig) jsonType() string {
return "edge_connection_config"
}
// DoHProxyConfig is configuration for DNS over HTTPS service
type DoHProxyConfig struct {
ListenHost string
ListenPort uint16
Upstreams []string
ListenHost string `json:"listen_host"`
ListenPort uint16 `json:"listen_port"`
Upstreams []string `json:"upstreams"`
}
// FailReason impelents FallibleConfig interface for DoHProxyConfig
func (dpc *DoHProxyConfig) FailReason(err error) string {
return fmt.Sprintf("Cannot apply DoHProxyConfig, err: %v", err)
}
func (dpc *DoHProxyConfig) MarshalJSON() ([]byte, error) {
marshaler := make(map[string]DoHProxyConfig, 1)
marshaler[dpc.jsonType()] = *dpc
return json.Marshal(marshaler)
}
func (dpc *DoHProxyConfig) jsonType() string {
return "doh_proxy_config"
}
// ReverseProxyConfig how and for what hostnames can this cloudflared proxy
type ReverseProxyConfig struct {
TunnelHostname string
Origin OriginConfig
Retries uint64
ConnectionTimeout time.Duration
CompressionQuality uint64
TunnelHostname h2mux.TunnelHostname `json:"tunnel_hostname"`
OriginConfigJSONHandler *OriginConfigJSONHandler `json:"origin_config"`
Retries uint64 `json:"retries"`
ConnectionTimeout time.Duration `json:"connection_timeout"`
CompressionQuality uint64 `json:"compression_quality"`
}
func NewReverseProxyConfig(
@@ -55,82 +142,175 @@ func NewReverseProxyConfig(
compressionQuality uint64,
) (*ReverseProxyConfig, error) {
if originConfig == nil {
return nil, fmt.Errorf("NewReverseProxyConfig: originConfig was null")
return nil, fmt.Errorf("NewReverseProxyConfig: originConfigUnmarshaler was null")
}
return &ReverseProxyConfig{
TunnelHostname: tunnelHostname,
Origin: originConfig,
Retries: retries,
ConnectionTimeout: connectionTimeout,
CompressionQuality: compressionQuality,
TunnelHostname: h2mux.TunnelHostname(tunnelHostname),
OriginConfigJSONHandler: &OriginConfigJSONHandler{originConfig},
Retries: retries,
ConnectionTimeout: connectionTimeout,
CompressionQuality: compressionQuality,
}, nil
}
// FailReason impelents FallibleConfig interface for ReverseProxyConfig
func (rpc *ReverseProxyConfig) FailReason(err error) string {
return fmt.Sprintf("Cannot apply ReverseProxyConfig, err: %v", err)
}
func (rpc *ReverseProxyConfig) MarshalJSON() ([]byte, error) {
marshaler := make(map[string]ReverseProxyConfig, 1)
marshaler[rpc.jsonType()] = *rpc
return json.Marshal(marshaler)
}
func (rpc *ReverseProxyConfig) jsonType() string {
return "reverse_proxy_config"
}
//go-sumtype:decl OriginConfig
type OriginConfig interface {
isOriginConfig()
// Service returns a OriginService used to proxy to the origin
Service() (originservice.OriginService, error)
// go-sumtype requires at least one unexported method, otherwise it will complain that interface is not sealed
jsonType() string
}
type HTTPOriginConfig struct {
URL OriginAddr `capnp:"url"`
TCPKeepAlive time.Duration `capnp:"tcpKeepAlive"`
DialDualStack bool
TLSHandshakeTimeout time.Duration `capnp:"tlsHandshakeTimeout"`
TLSVerify bool `capnp:"tlsVerify"`
OriginCAPool string
OriginServerName string
MaxIdleConnections uint64
IdleConnectionTimeout time.Duration
ProxyConnectTimeout time.Duration
ExpectContinueTimeout time.Duration
ChunkedEncoding bool
}
type originType int
func (_ *HTTPOriginConfig) isOriginConfig() {}
const (
httpType originType = iota
wsType
helloWorldType
)
type OriginAddr interface {
Addr() string
}
type HTTPURL struct {
URL *url.URL
}
func (ha *HTTPURL) Addr() string {
return ha.URL.String()
}
func (ha *HTTPURL) capnpHTTPURL() *CapnpHTTPURL {
return &CapnpHTTPURL{
URL: ha.URL.String(),
func (ot originType) String() string {
switch ot {
case httpType:
return "Http"
case wsType:
return "WebSocket"
case helloWorldType:
return "HelloWorld"
default:
return "unknown"
}
}
// URL for a HTTP origin, capnp doesn't have native support for URL, so represent it as string
type CapnpHTTPURL struct {
URL string `capnp:"url"`
type HTTPOriginConfig struct {
URLString string `capnp:"urlString" json:"url_string" mapstructure:"url_string"`
TCPKeepAlive time.Duration `capnp:"tcpKeepAlive" json:"tcp_keep_alive" mapstructure:"tcp_keep_alive"`
DialDualStack bool `json:"dial_dual_stack" mapstructure:"dial_dual_stack"`
TLSHandshakeTimeout time.Duration `capnp:"tlsHandshakeTimeout" json:"tls_handshake_timeout" mapstructure:"tls_handshake_timeout"`
TLSVerify bool `capnp:"tlsVerify" json:"tls_verify" mapstructure:"tls_verify"`
OriginCAPool string `json:"origin_ca_pool" mapstructure:"origin_ca_pool"`
OriginServerName string `json:"origin_server_name" mapstructure:"origin_server_name"`
MaxIdleConnections uint64 `json:"max_idle_connections" mapstructure:"max_idle_connections"`
IdleConnectionTimeout time.Duration `json:"idle_connection_timeout" mapstructure:"idle_connection_timeout"`
ProxyConnectionTimeout time.Duration `json:"proxy_connection_timeout" mapstructure:"proxy_connection_timeout"`
ExpectContinueTimeout time.Duration `json:"expect_continue_timeout" mapstructure:"expect_continue_timeout"`
ChunkedEncoding bool `json:"chunked_encoding" mapstructure:"chunked_encoding"`
}
type UnixPath struct {
Path string
func (hc *HTTPOriginConfig) Service() (originservice.OriginService, error) {
rootCAs, err := tlsconfig.LoadCustomOriginCA(hc.OriginCAPool)
if err != nil {
return nil, err
}
dialContext := (&net.Dialer{
Timeout: hc.ProxyConnectionTimeout,
KeepAlive: hc.TCPKeepAlive,
DualStack: hc.DialDualStack,
}).DialContext
transport := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: dialContext,
TLSClientConfig: &tls.Config{
RootCAs: rootCAs,
ServerName: hc.OriginServerName,
InsecureSkipVerify: hc.TLSVerify,
},
TLSHandshakeTimeout: hc.TLSHandshakeTimeout,
MaxIdleConns: int(hc.MaxIdleConnections),
IdleConnTimeout: hc.IdleConnectionTimeout,
ExpectContinueTimeout: hc.ExpectContinueTimeout,
}
url, err := url.Parse(hc.URLString)
if err != nil {
return nil, errors.Wrapf(err, "%s is not a valid URL", hc.URLString)
}
if url.Scheme == "unix" {
transport.DialContext = func(ctx context.Context, _, _ string) (net.Conn, error) {
return dialContext(ctx, "unix", url.Host)
}
}
return originservice.NewHTTPService(transport, url, hc.ChunkedEncoding), nil
}
func (up *UnixPath) Addr() string {
return up.Path
func (_ *HTTPOriginConfig) jsonType() string {
return httpType.String()
}
type WebSocketOriginConfig struct {
URL string `capnp:"url"`
TLSVerify bool `capnp:"tlsVerify"`
OriginCAPool string
OriginServerName string
URLString string `capnp:"urlString" json:"url_string" mapstructure:"url_string"`
TLSVerify bool `capnp:"tlsVerify" json:"tls_verify" mapstructure:"tls_verify"`
OriginCAPool string `json:"origin_ca_pool" mapstructure:"origin_ca_pool"`
OriginServerName string `json:"origin_server_name" mapstructure:"origin_server_name"`
}
func (_ *WebSocketOriginConfig) isOriginConfig() {}
func (wsc *WebSocketOriginConfig) Service() (originservice.OriginService, error) {
rootCAs, err := tlsconfig.LoadCustomOriginCA(wsc.OriginCAPool)
if err != nil {
return nil, err
}
tlsConfig := &tls.Config{
RootCAs: rootCAs,
ServerName: wsc.OriginServerName,
InsecureSkipVerify: wsc.TLSVerify,
}
url, err := url.Parse(wsc.URLString)
if err != nil {
return nil, errors.Wrapf(err, "%s is not a valid URL", wsc.URLString)
}
return originservice.NewWebSocketService(tlsConfig, url)
}
func (_ *WebSocketOriginConfig) jsonType() string {
return wsType.String()
}
type HelloWorldOriginConfig struct{}
func (_ *HelloWorldOriginConfig) isOriginConfig() {}
func (_ *HelloWorldOriginConfig) Service() (originservice.OriginService, error) {
helloCert, err := tlsconfig.GetHelloCertificateX509()
if err != nil {
return nil, errors.Wrap(err, "Cannot get Hello World server certificate")
}
rootCAs := x509.NewCertPool()
rootCAs.AddCert(helloCert)
transport := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
DualStack: true,
}).DialContext,
TLSClientConfig: &tls.Config{
RootCAs: rootCAs,
},
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
}
return originservice.NewHelloWorldService(transport)
}
func (_ *HelloWorldOriginConfig) jsonType() string {
return helloWorldType.String()
}
/*
* Boilerplate to convert between these structs and the primitive structs
@@ -142,18 +322,45 @@ func (_ *HelloWorldOriginConfig) isOriginConfig() {}
*/
func MarshalClientConfig(s tunnelrpc.ClientConfig, p *ClientConfig) error {
s.SetVersion(p.Version)
s.SetAutoUpdateFrequency(p.AutoUpdateFrequency.Nanoseconds())
s.SetMetricsUpdateFrequency(p.MetricsUpdateFrequency.Nanoseconds())
s.SetHeartbeatInterval(p.HeartbeatInterval.Nanoseconds())
s.SetMaxFailedHeartbeats(p.MaxFailedHeartbeats)
s.SetGracePeriod(p.GracePeriod.Nanoseconds())
s.SetNumHAConnections(p.NumHAConnections)
err := marshalDoHProxyConfigs(s, p.DoHProxyConfigs)
s.SetVersion(uint64(p.Version))
supervisorConfig, err := s.NewSupervisorConfig()
if err != nil {
return err
return errors.Wrap(err, "failed to get SupervisorConfig")
}
return marshalReverseProxyConfigs(s, p.ReverseProxyConfigs)
if err = MarshalSupervisorConfig(supervisorConfig, p.SupervisorConfig); err != nil {
return errors.Wrap(err, "MarshalSupervisorConfig error")
}
edgeConnectionConfig, err := s.NewEdgeConnectionConfig()
if err != nil {
return errors.Wrap(err, "failed to get EdgeConnectionConfig")
}
if err := MarshalEdgeConnectionConfig(edgeConnectionConfig, p.EdgeConnectionConfig); err != nil {
return errors.Wrap(err, "MarshalEdgeConnectionConfig error")
}
if err := marshalDoHProxyConfigs(s, p.DoHProxyConfigs); err != nil {
return errors.Wrap(err, "marshalDoHProxyConfigs error")
}
if err := marshalReverseProxyConfigs(s, p.ReverseProxyConfigs); err != nil {
return errors.Wrap(err, "marshalReverseProxyConfigs error")
}
return nil
}
func MarshalSupervisorConfig(s tunnelrpc.SupervisorConfig, p *SupervisorConfig) error {
if err := pogs.Insert(tunnelrpc.SupervisorConfig_TypeID, s.Struct, p); err != nil {
return errors.Wrap(err, "failed to insert SupervisorConfig")
}
return nil
}
func MarshalEdgeConnectionConfig(s tunnelrpc.EdgeConnectionConfig, p *EdgeConnectionConfig) error {
if err := pogs.Insert(tunnelrpc.EdgeConnectionConfig_TypeID, s.Struct, p); err != nil {
return errors.Wrap(err, "failed to insert EdgeConnectionConfig")
}
return nil
}
func marshalDoHProxyConfigs(s tunnelrpc.ClientConfig, dohProxyConfigs []*DoHProxyConfig) error {
@@ -186,23 +393,48 @@ func marshalReverseProxyConfigs(s tunnelrpc.ClientConfig, reverseProxyConfigs []
func UnmarshalClientConfig(s tunnelrpc.ClientConfig) (*ClientConfig, error) {
p := new(ClientConfig)
p.Version = s.Version()
p.AutoUpdateFrequency = time.Duration(s.AutoUpdateFrequency())
p.MetricsUpdateFrequency = time.Duration(s.MetricsUpdateFrequency())
p.HeartbeatInterval = time.Duration(s.HeartbeatInterval())
p.MaxFailedHeartbeats = s.MaxFailedHeartbeats()
p.GracePeriod = time.Duration(s.GracePeriod())
p.NumHAConnections = s.NumHAConnections()
dohProxyConfigs, err := unmarshalDoHProxyConfigs(s)
p.Version = Version(s.Version())
supervisorConfig, err := s.SupervisorConfig()
if err != nil {
return nil, err
return nil, errors.Wrap(err, "failed to get SupervisorConfig")
}
p.DoHProxyConfigs = dohProxyConfigs
reverseProxyConfigs, err := unmarshalReverseProxyConfigs(s)
p.SupervisorConfig, err = UnmarshalSupervisorConfig(supervisorConfig)
if err != nil {
return nil, err
return nil, errors.Wrap(err, "UnmarshalSupervisorConfig error")
}
p.ReverseProxyConfigs = reverseProxyConfigs
edgeConnectionConfig, err := s.EdgeConnectionConfig()
if err != nil {
return nil, errors.Wrap(err, "failed to get ConnectionManagerConfig")
}
p.EdgeConnectionConfig, err = UnmarshalEdgeConnectionConfig(edgeConnectionConfig)
if err != nil {
return nil, errors.Wrap(err, "UnmarshalConnectionManagerConfig error")
}
p.DoHProxyConfigs, err = unmarshalDoHProxyConfigs(s)
if err != nil {
return nil, errors.Wrap(err, "unmarshalDoHProxyConfigs error")
}
p.ReverseProxyConfigs, err = unmarshalReverseProxyConfigs(s)
if err != nil {
return nil, errors.Wrap(err, "unmarshalReverseProxyConfigs error")
}
return p, nil
}
func UnmarshalSupervisorConfig(s tunnelrpc.SupervisorConfig) (*SupervisorConfig, error) {
p := new(SupervisorConfig)
err := pogs.Extract(p, tunnelrpc.SupervisorConfig_TypeID, s.Struct)
return p, err
}
func UnmarshalEdgeConnectionConfig(s tunnelrpc.EdgeConnectionConfig) (*EdgeConnectionConfig, error) {
p := new(EdgeConnectionConfig)
err := pogs.Extract(p, tunnelrpc.EdgeConnectionConfig_TypeID, s.Struct)
return p, err
}
@@ -241,13 +473,38 @@ func unmarshalReverseProxyConfigs(s tunnelrpc.ClientConfig) ([]*ReverseProxyConf
}
func MarshalUseConfigurationResult(s tunnelrpc.UseConfigurationResult, p *UseConfigurationResult) error {
return pogs.Insert(tunnelrpc.UseConfigurationResult_TypeID, s.Struct, p)
capnpList, err := s.NewFailedConfigs(int32(len(p.FailedConfigs)))
if err != nil {
return errors.Wrap(err, "Cannot create new FailedConfigs")
}
for i, unmarshalledFailedConfig := range p.FailedConfigs {
err := MarshalFailedConfig(capnpList.At(i), unmarshalledFailedConfig)
if err != nil {
return errors.Wrapf(err, "Cannot MarshalFailedConfig at index %d", i)
}
}
s.SetSuccess(p.Success)
return nil
}
func UnmarshalUseConfigurationResult(s tunnelrpc.UseConfigurationResult) (*UseConfigurationResult, error) {
p := new(UseConfigurationResult)
err := pogs.Extract(p, tunnelrpc.UseConfigurationResult_TypeID, s.Struct)
return p, err
var failedConfigs []*FailedConfig
marshalledFailedConfigs, err := s.FailedConfigs()
if err != nil {
return nil, errors.Wrap(err, "Cannot get FailedConfigs")
}
for i := 0; i < marshalledFailedConfigs.Len(); i++ {
ss := marshalledFailedConfigs.At(i)
failedConfig, err := UnmarshalFailedConfig(ss)
if err != nil {
return nil, errors.Wrapf(err, "Cannot UnmarshalFailedConfig at index %d", i)
}
failedConfigs = append(failedConfigs, failedConfig)
}
p.FailedConfigs = failedConfigs
p.Success = s.Success()
return p, nil
}
func MarshalDoHProxyConfig(s tunnelrpc.DoHProxyConfig, p *DoHProxyConfig) error {
@@ -261,8 +518,8 @@ func UnmarshalDoHProxyConfig(s tunnelrpc.DoHProxyConfig) (*DoHProxyConfig, error
}
func MarshalReverseProxyConfig(s tunnelrpc.ReverseProxyConfig, p *ReverseProxyConfig) error {
s.SetTunnelHostname(p.TunnelHostname)
switch config := p.Origin.(type) {
s.SetTunnelHostname(p.TunnelHostname.String())
switch config := p.OriginConfigJSONHandler.OriginConfig.(type) {
case *HTTPOriginConfig:
ss, err := s.Origin().NewHttp()
if err != nil {
@@ -302,7 +559,7 @@ func UnmarshalReverseProxyConfig(s tunnelrpc.ReverseProxyConfig) (*ReverseProxyC
if err != nil {
return nil, err
}
p.TunnelHostname = tunnelHostname
p.TunnelHostname = h2mux.TunnelHostname(tunnelHostname)
switch s.Origin().Which() {
case tunnelrpc.ReverseProxyConfig_origin_Which_http:
ss, err := s.Origin().Http()
@@ -313,7 +570,7 @@ func UnmarshalReverseProxyConfig(s tunnelrpc.ReverseProxyConfig) (*ReverseProxyC
if err != nil {
return nil, err
}
p.Origin = config
p.OriginConfigJSONHandler = &OriginConfigJSONHandler{config}
case tunnelrpc.ReverseProxyConfig_origin_Which_websocket:
ss, err := s.Origin().Websocket()
if err != nil {
@@ -323,7 +580,7 @@ func UnmarshalReverseProxyConfig(s tunnelrpc.ReverseProxyConfig) (*ReverseProxyC
if err != nil {
return nil, err
}
p.Origin = config
p.OriginConfigJSONHandler = &OriginConfigJSONHandler{config}
case tunnelrpc.ReverseProxyConfig_origin_Which_helloWorld:
ss, err := s.Origin().HelloWorld()
if err != nil {
@@ -333,7 +590,7 @@ func UnmarshalReverseProxyConfig(s tunnelrpc.ReverseProxyConfig) (*ReverseProxyC
if err != nil {
return nil, err
}
p.Origin = config
p.OriginConfigJSONHandler = &OriginConfigJSONHandler{config}
}
p.Retries = s.Retries()
p.ConnectionTimeout = time.Duration(s.ConnectionTimeout())
@@ -342,115 +599,12 @@ func UnmarshalReverseProxyConfig(s tunnelrpc.ReverseProxyConfig) (*ReverseProxyC
}
func MarshalHTTPOriginConfig(s tunnelrpc.HTTPOriginConfig, p *HTTPOriginConfig) error {
switch originAddr := p.URL.(type) {
case *HTTPURL:
ss, err := s.OriginAddr().NewHttp()
if err != nil {
return err
}
if err := MarshalHTTPURL(ss, originAddr); err != nil {
return err
}
case *UnixPath:
ss, err := s.OriginAddr().NewUnix()
if err != nil {
return err
}
if err := MarshalUnixPath(ss, originAddr); err != nil {
return err
}
default:
return fmt.Errorf("Unknown type for OriginAddr: %T", originAddr)
}
s.SetTcpKeepAlive(p.TCPKeepAlive.Nanoseconds())
s.SetDialDualStack(p.DialDualStack)
s.SetTlsHandshakeTimeout(p.TLSHandshakeTimeout.Nanoseconds())
s.SetTlsVerify(p.TLSVerify)
s.SetOriginCAPool(p.OriginCAPool)
s.SetOriginServerName(p.OriginServerName)
s.SetMaxIdleConnections(p.MaxIdleConnections)
s.SetIdleConnectionTimeout(p.IdleConnectionTimeout.Nanoseconds())
s.SetProxyConnectionTimeout(p.ProxyConnectTimeout.Nanoseconds())
s.SetExpectContinueTimeout(p.ExpectContinueTimeout.Nanoseconds())
s.SetChunkedEncoding(p.ChunkedEncoding)
return nil
return pogs.Insert(tunnelrpc.HTTPOriginConfig_TypeID, s.Struct, p)
}
func UnmarshalHTTPOriginConfig(s tunnelrpc.HTTPOriginConfig) (*HTTPOriginConfig, error) {
p := new(HTTPOriginConfig)
switch s.OriginAddr().Which() {
case tunnelrpc.HTTPOriginConfig_originAddr_Which_http:
ss, err := s.OriginAddr().Http()
if err != nil {
return nil, err
}
originAddr, err := UnmarshalCapnpHTTPURL(ss)
if err != nil {
return nil, err
}
p.URL = originAddr
case tunnelrpc.HTTPOriginConfig_originAddr_Which_unix:
ss, err := s.OriginAddr().Unix()
if err != nil {
return nil, err
}
originAddr, err := UnmarshalUnixPath(ss)
if err != nil {
return nil, err
}
p.URL = originAddr
default:
return nil, fmt.Errorf("Unknown type for OriginAddr: %T", s.OriginAddr().Which())
}
p.TCPKeepAlive = time.Duration(s.TcpKeepAlive())
p.DialDualStack = s.DialDualStack()
p.TLSHandshakeTimeout = time.Duration(s.TlsHandshakeTimeout())
p.TLSVerify = s.TlsVerify()
originCAPool, err := s.OriginCAPool()
if err != nil {
return nil, err
}
p.OriginCAPool = originCAPool
originServerName, err := s.OriginServerName()
if err != nil {
return nil, err
}
p.OriginServerName = originServerName
p.MaxIdleConnections = s.MaxIdleConnections()
p.IdleConnectionTimeout = time.Duration(s.IdleConnectionTimeout())
p.ProxyConnectTimeout = time.Duration(s.ProxyConnectionTimeout())
p.ExpectContinueTimeout = time.Duration(s.ExpectContinueTimeout())
p.ChunkedEncoding = s.ChunkedEncoding()
return p, nil
}
func MarshalHTTPURL(s tunnelrpc.CapnpHTTPURL, p *HTTPURL) error {
return pogs.Insert(tunnelrpc.CapnpHTTPURL_TypeID, s.Struct, p.capnpHTTPURL())
}
func UnmarshalCapnpHTTPURL(s tunnelrpc.CapnpHTTPURL) (*HTTPURL, error) {
p := new(CapnpHTTPURL)
err := pogs.Extract(p, tunnelrpc.CapnpHTTPURL_TypeID, s.Struct)
if err != nil {
return nil, err
}
url, err := url.Parse(p.URL)
if err != nil {
return nil, err
}
return &HTTPURL{
URL: url,
}, nil
}
func MarshalUnixPath(s tunnelrpc.UnixPath, p *UnixPath) error {
err := pogs.Insert(tunnelrpc.UnixPath_TypeID, s.Struct, p)
return err
}
func UnmarshalUnixPath(s tunnelrpc.UnixPath) (*UnixPath, error) {
p := new(UnixPath)
err := pogs.Extract(p, tunnelrpc.UnixPath_TypeID, s.Struct)
err := pogs.Extract(p, tunnelrpc.HTTPOriginConfig_TypeID, s.Struct)
return p, err
}
@@ -505,3 +659,141 @@ func (c *ClientService_PogsClient) UseConfiguration(
}
return UnmarshalUseConfigurationResult(retval)
}
func ClientService_ServerToClient(s ClientService) tunnelrpc.ClientService {
return tunnelrpc.ClientService_ServerToClient(ClientService_PogsImpl{s})
}
type ClientService_PogsImpl struct {
impl ClientService
}
func (i ClientService_PogsImpl) UseConfiguration(p tunnelrpc.ClientService_useConfiguration) error {
config, err := p.Params.ClientServiceConfig()
if err != nil {
return errors.Wrap(err, "Cannot get CloudflaredConfig parameter")
}
pogsConfig, err := UnmarshalClientConfig(config)
if err != nil {
return errors.Wrap(err, "Cannot unmarshal tunnelrpc.CloudflaredConfig to *CloudflaredConfig")
}
server.Ack(p.Options)
userConfigResult, err := i.impl.UseConfiguration(p.Ctx, pogsConfig)
if err != nil {
return err
}
result, err := p.Results.NewResult()
if err != nil {
return err
}
return MarshalUseConfigurationResult(result, userConfigResult)
}
type UseConfigurationResult struct {
Success bool `json:"success"`
FailedConfigs []*FailedConfig `json:"failed_configs"`
}
type FailedConfig struct {
Config FallibleConfig `json:"config"`
Reason string `json:"reason"`
}
func MarshalFailedConfig(s tunnelrpc.FailedConfig, p *FailedConfig) error {
switch config := p.Config.(type) {
case *SupervisorConfig:
ss, err := s.Config().NewSupervisor()
if err != nil {
return err
}
err = MarshalSupervisorConfig(ss, config)
if err != nil {
return err
}
case *EdgeConnectionConfig:
ss, err := s.Config().NewEdgeConnection()
if err != nil {
return err
}
err = MarshalEdgeConnectionConfig(ss, config)
if err != nil {
return err
}
case *DoHProxyConfig:
ss, err := s.Config().NewDoh()
if err != nil {
return err
}
err = MarshalDoHProxyConfig(ss, config)
if err != nil {
return err
}
case *ReverseProxyConfig:
ss, err := s.Config().NewReverseProxy()
if err != nil {
return err
}
err = MarshalReverseProxyConfig(ss, config)
if err != nil {
return err
}
default:
return fmt.Errorf("Unknown type for Config: %T", config)
}
s.SetReason(p.Reason)
return nil
}
func UnmarshalFailedConfig(s tunnelrpc.FailedConfig) (*FailedConfig, error) {
p := new(FailedConfig)
switch s.Config().Which() {
case tunnelrpc.FailedConfig_config_Which_supervisor:
ss, err := s.Config().Supervisor()
if err != nil {
return nil, errors.Wrap(err, "Cannot get SupervisorConfig from Config")
}
config, err := UnmarshalSupervisorConfig(ss)
if err != nil {
return nil, errors.Wrap(err, "Cannot UnmarshalSupervisorConfig")
}
p.Config = config
case tunnelrpc.FailedConfig_config_Which_edgeConnection:
ss, err := s.Config().EdgeConnection()
if err != nil {
return nil, errors.Wrap(err, "Cannot get ConnectionManager from Config")
}
config, err := UnmarshalEdgeConnectionConfig(ss)
if err != nil {
return nil, errors.Wrap(err, "Cannot UnmarshalConnectionManagerConfig")
}
p.Config = config
case tunnelrpc.FailedConfig_config_Which_doh:
ss, err := s.Config().Doh()
if err != nil {
return nil, errors.Wrap(err, "Cannot get Doh from Config")
}
config, err := UnmarshalDoHProxyConfig(ss)
if err != nil {
return nil, errors.Wrap(err, "Cannot UnmarshalDoHProxyConfig")
}
p.Config = config
case tunnelrpc.FailedConfig_config_Which_reverseProxy:
ss, err := s.Config().ReverseProxy()
if err != nil {
return nil, errors.Wrap(err, "Cannot get ReverseProxy from Config")
}
config, err := UnmarshalReverseProxyConfig(ss)
if err != nil {
return nil, errors.Wrap(err, "Cannot UnmarshalReverseProxyConfig")
}
p.Config = config
default:
return nil, fmt.Errorf("Unknown type for FailedConfig: %v", s.Config().Which())
}
reason, err := s.Reason()
if err != nil {
return nil, errors.Wrap(err, "Cannot get Reason")
}
p.Reason = reason
return p, nil
}
+131 -56
View File
@@ -2,7 +2,6 @@ package pogs
import (
"fmt"
"net/url"
"reflect"
"testing"
"time"
@@ -13,6 +12,23 @@ import (
capnp "zombiezen.com/go/capnproto2"
)
// Assert *HTTPOriginConfig implements OriginConfig
var _ OriginConfig = (*HTTPOriginConfig)(nil)
// Assert *WebSocketOriginConfig implements OriginConfig
var _ OriginConfig = (*WebSocketOriginConfig)(nil)
// Assert *HelloWorldOriginConfig implements OriginConfig
var _ OriginConfig = (*HelloWorldOriginConfig)(nil)
func TestVersion(t *testing.T) {
firstVersion := InitVersion()
secondVersion := Version(1)
assert.False(t, firstVersion.IsNewerOrEqual(secondVersion))
assert.True(t, secondVersion.IsNewerOrEqual(firstVersion))
assert.True(t, secondVersion.IsNewerOrEqual(secondVersion))
}
func TestClientConfig(t *testing.T) {
addDoHProxyConfigs := func(c *ClientConfig) {
c.DoHProxyConfigs = []*DoHProxyConfig{
@@ -25,13 +41,13 @@ func TestClientConfig(t *testing.T) {
sampleReverseProxyConfig(func(c *ReverseProxyConfig) {
}),
sampleReverseProxyConfig(func(c *ReverseProxyConfig) {
c.Origin = sampleHTTPOriginConfig()
c.OriginConfigJSONHandler = &OriginConfigJSONHandler{sampleHTTPOriginConfig()}
}),
sampleReverseProxyConfig(func(c *ReverseProxyConfig) {
c.Origin = sampleHTTPOriginUnixPathConfig()
c.OriginConfigJSONHandler = &OriginConfigJSONHandler{sampleHTTPOriginConfigUnixPath()}
}),
sampleReverseProxyConfig(func(c *ReverseProxyConfig) {
c.Origin = sampleWebSocketOriginConfig()
c.OriginConfigJSONHandler = &OriginConfigJSONHandler{sampleWebSocketOriginConfig()}
}),
}
}
@@ -66,8 +82,17 @@ func TestUseConfigurationResult(t *testing.T) {
Success: true,
},
&UseConfigurationResult{
Success: false,
ErrorMessage: "the quick brown fox jumped over the lazy dogs",
Success: false,
FailedConfigs: []*FailedConfig{
{
Config: sampleReverseProxyConfig(),
Reason: "Invalid certificate",
},
{
Config: sampleDoHProxyConfig(),
Reason: "Cannot listen on port 53",
},
},
},
}
for i, testCase := range testCases {
@@ -117,13 +142,13 @@ func TestReverseProxyConfig(t *testing.T) {
testCases := []*ReverseProxyConfig{
sampleReverseProxyConfig(),
sampleReverseProxyConfig(func(c *ReverseProxyConfig) {
c.Origin = sampleHTTPOriginConfig()
c.OriginConfigJSONHandler = &OriginConfigJSONHandler{sampleHTTPOriginConfig()}
}),
sampleReverseProxyConfig(func(c *ReverseProxyConfig) {
c.Origin = sampleHTTPOriginUnixPathConfig()
c.OriginConfigJSONHandler = &OriginConfigJSONHandler{sampleHTTPOriginConfigUnixPath()}
}),
sampleReverseProxyConfig(func(c *ReverseProxyConfig) {
c.Origin = sampleWebSocketOriginConfig()
c.OriginConfigJSONHandler = &OriginConfigJSONHandler{sampleWebSocketOriginConfig()}
}),
}
for i, testCase := range testCases {
@@ -188,18 +213,42 @@ func TestWebSocketOriginConfig(t *testing.T) {
}
}
func TestOriginConfigInvalidURL(t *testing.T) {
invalidConfigs := []OriginConfig{
&HTTPOriginConfig{
// this url doesn't have a scheme
URLString: "127.0.0.1:36192",
},
&WebSocketOriginConfig{
URLString: "127.0.0.1:36192",
},
}
for _, config := range invalidConfigs {
service, err := config.Service()
assert.Error(t, err)
assert.Nil(t, service)
}
}
//////////////////////////////////////////////////////////////////////////////
// Functions to generate sample data for ease of testing
//
// There's one "sample" function per struct type. Each goes like this:
// 1. Initialize an instance of the relevant struct.
// 2. Ensure the instance has no zero-valued fields. (This catches the
// error-case where a field was added, but we forgot to add code to
// marshal/unmarshal this field in CapnProto.)
// 3. Apply one or more "override" functions (which accept a
// pointer-to-struct, so they can mutate the instance).
// sampleClientConfig initializes a new ClientConfig literal,
// applies any number of overrides to it, and returns it.
func sampleClientConfig(overrides ...func(*ClientConfig)) *ClientConfig {
sample := &ClientConfig{
Version: uint64(1337),
AutoUpdateFrequency: 21 * time.Hour,
MetricsUpdateFrequency: 11 * time.Minute,
HeartbeatInterval: 5 * time.Second,
MaxFailedHeartbeats: 9001,
GracePeriod: 31 * time.Second,
NumHAConnections: 49,
Version: Version(1337),
SupervisorConfig: sampleSupervisorConfig(),
EdgeConnectionConfig: sampleEdgeConnectionConfig(),
}
sample.ensureNoZeroFields()
for _, f := range overrides {
@@ -208,11 +257,35 @@ func sampleClientConfig(overrides ...func(*ClientConfig)) *ClientConfig {
return sample
}
func sampleSupervisorConfig() *SupervisorConfig {
sample := &SupervisorConfig{
AutoUpdateFrequency: 21 * time.Hour,
MetricsUpdateFrequency: 11 * time.Minute,
GracePeriod: 31 * time.Second,
}
sample.ensureNoZeroFields()
return sample
}
func sampleEdgeConnectionConfig() *EdgeConnectionConfig {
sample := &EdgeConnectionConfig{
NumHAConnections: 49,
HeartbeatInterval: 5 * time.Second,
Timeout: 9 * time.Second,
MaxFailedHeartbeats: 9001,
UserCredentialPath: "/Users/example/.cloudflared/cert.pem",
}
sample.ensureNoZeroFields()
return sample
}
// sampleDoHProxyConfig initializes a new DoHProxyConfig struct,
// applies any number of overrides to it, and returns it.
func sampleDoHProxyConfig(overrides ...func(*DoHProxyConfig)) *DoHProxyConfig {
sample := &DoHProxyConfig{
ListenHost: "127.0.0.1",
ListenPort: 53,
Upstreams: []string{"https://1.example.com", "https://2.example.com"},
Upstreams: []string{"1.1.1.1", "1.0.0.1"},
}
sample.ensureNoZeroFields()
for _, f := range overrides {
@@ -221,13 +294,15 @@ func sampleDoHProxyConfig(overrides ...func(*DoHProxyConfig)) *DoHProxyConfig {
return sample
}
// sampleReverseProxyConfig initializes a new ReverseProxyConfig struct,
// applies any number of overrides to it, and returns it.
func sampleReverseProxyConfig(overrides ...func(*ReverseProxyConfig)) *ReverseProxyConfig {
sample := &ReverseProxyConfig{
TunnelHostname: "hijk.example.com",
Origin: &HelloWorldOriginConfig{},
Retries: 18,
ConnectionTimeout: 5 * time.Second,
CompressionQuality: 4,
TunnelHostname: "mock-non-lb-tunnel.example.com",
OriginConfigJSONHandler: &OriginConfigJSONHandler{&HelloWorldOriginConfig{}},
Retries: 18,
ConnectionTimeout: 5 * time.Second,
CompressionQuality: 3,
}
sample.ensureNoZeroFields()
for _, f := range overrides {
@@ -238,23 +313,18 @@ func sampleReverseProxyConfig(overrides ...func(*ReverseProxyConfig)) *ReversePr
func sampleHTTPOriginConfig(overrides ...func(*HTTPOriginConfig)) *HTTPOriginConfig {
sample := &HTTPOriginConfig{
URL: &HTTPURL{
URL: &url.URL{
Scheme: "https",
Host: "example.com",
},
},
TCPKeepAlive: 7 * time.Second,
DialDualStack: true,
TLSHandshakeTimeout: 11 * time.Second,
TLSVerify: true,
OriginCAPool: "/etc/cert.pem",
OriginServerName: "secure.example.com",
MaxIdleConnections: 19,
IdleConnectionTimeout: 17 * time.Second,
ProxyConnectTimeout: 15 * time.Second,
ExpectContinueTimeout: 21 * time.Second,
ChunkedEncoding: true,
URLString: "https.example.com",
TCPKeepAlive: 7 * time.Second,
DialDualStack: true,
TLSHandshakeTimeout: 11 * time.Second,
TLSVerify: true,
OriginCAPool: "/etc/cert.pem",
OriginServerName: "secure.example.com",
MaxIdleConnections: 19,
IdleConnectionTimeout: 17 * time.Second,
ProxyConnectionTimeout: 15 * time.Second,
ExpectContinueTimeout: 21 * time.Second,
ChunkedEncoding: true,
}
sample.ensureNoZeroFields()
for _, f := range overrides {
@@ -263,22 +333,20 @@ func sampleHTTPOriginConfig(overrides ...func(*HTTPOriginConfig)) *HTTPOriginCon
return sample
}
func sampleHTTPOriginUnixPathConfig(overrides ...func(*HTTPOriginConfig)) *HTTPOriginConfig {
func sampleHTTPOriginConfigUnixPath(overrides ...func(*HTTPOriginConfig)) *HTTPOriginConfig {
sample := &HTTPOriginConfig{
URL: &UnixPath{
Path: "/var/lib/file.sock",
},
TCPKeepAlive: 7 * time.Second,
DialDualStack: true,
TLSHandshakeTimeout: 11 * time.Second,
TLSVerify: true,
OriginCAPool: "/etc/cert.pem",
OriginServerName: "secure.example.com",
MaxIdleConnections: 19,
IdleConnectionTimeout: 17 * time.Second,
ProxyConnectTimeout: 15 * time.Second,
ExpectContinueTimeout: 21 * time.Second,
ChunkedEncoding: true,
URLString: "unix:/var/lib/file.sock",
TCPKeepAlive: 7 * time.Second,
DialDualStack: true,
TLSHandshakeTimeout: 11 * time.Second,
TLSVerify: true,
OriginCAPool: "/etc/cert.pem",
OriginServerName: "secure.example.com",
MaxIdleConnections: 19,
IdleConnectionTimeout: 17 * time.Second,
ProxyConnectionTimeout: 15 * time.Second,
ExpectContinueTimeout: 21 * time.Second,
ChunkedEncoding: true,
}
sample.ensureNoZeroFields()
for _, f := range overrides {
@@ -289,7 +357,7 @@ func sampleHTTPOriginUnixPathConfig(overrides ...func(*HTTPOriginConfig)) *HTTPO
func sampleWebSocketOriginConfig(overrides ...func(*WebSocketOriginConfig)) *WebSocketOriginConfig {
sample := &WebSocketOriginConfig{
URL: "ssh://example.com",
URLString: "ssh://example.com",
TLSVerify: true,
OriginCAPool: "/etc/cert.pem",
OriginServerName: "secure.example.com",
@@ -305,6 +373,14 @@ func (c *ClientConfig) ensureNoZeroFields() {
ensureNoZeroFieldsInSample(reflect.ValueOf(c), []string{"DoHProxyConfigs", "ReverseProxyConfigs"})
}
func (c *SupervisorConfig) ensureNoZeroFields() {
ensureNoZeroFieldsInSample(reflect.ValueOf(c), []string{})
}
func (c *EdgeConnectionConfig) ensureNoZeroFields() {
ensureNoZeroFieldsInSample(reflect.ValueOf(c), []string{})
}
func (c *DoHProxyConfig) ensureNoZeroFields() {
ensureNoZeroFieldsInSample(reflect.ValueOf(c), []string{})
}
@@ -334,7 +410,6 @@ func (c *WebSocketOriginConfig) ensureNoZeroFields() {
// include a field in the sample value, it won't be covered under tests.
// This check reduces that risk by requiring fields to be either initialized
// or explicitly uninitialized.
// https://bitbucket.cfdata.org/projects/TUN/repos/cloudflared/pull-requests/151/overview?commentId=348012
func ensureNoZeroFieldsInSample(ptrToSampleValue reflect.Value, allowedZeroFieldNames []string) {
sampleValue := ptrToSampleValue.Elem()
structType := ptrToSampleValue.Type().Elem()
+101
View File
@@ -0,0 +1,101 @@
package pogs
import (
"encoding/json"
"fmt"
"github.com/mitchellh/mapstructure"
"github.com/pkg/errors"
)
// ScopeUnmarshaler can marshal a Scope pog from JSON.
type ScopeUnmarshaler struct {
Scope Scope
}
// UnmarshalJSON takes in a JSON string, and attempts to marshal it into a Scope.
// If successful, the Scope member of this ScopeUnmarshaler is set and nil is returned.
// If unsuccessful, returns an error.
func (su *ScopeUnmarshaler) UnmarshalJSON(b []byte) error {
var scopeJSON map[string]interface{}
if err := json.Unmarshal(b, &scopeJSON); err != nil {
return errors.Wrapf(err, "cannot unmarshal %s into scopeJSON", string(b))
}
if group, ok := scopeJSON["group"]; ok {
if val, ok := group.(string); ok {
su.Scope = NewGroup(val)
return nil
}
return fmt.Errorf("JSON should have been a Scope, but the 'group' key contained %v", group)
}
if systemName, ok := scopeJSON["system_name"]; ok {
if val, ok := systemName.(string); ok {
su.Scope = NewSystemName(val)
return nil
}
return fmt.Errorf("JSON should have been a Scope, but the 'system_name' key contained %v", systemName)
}
return fmt.Errorf("JSON should have been an object with one root key, either 'system_name' or 'group'")
}
// OriginConfigJSONHandler is a wrapper to serialize OriginConfig with type information, and deserialize JSON
// into an OriginConfig.
type OriginConfigJSONHandler struct {
OriginConfig OriginConfig
}
func (ocjh *OriginConfigJSONHandler) MarshalJSON() ([]byte, error) {
marshaler := make(map[string]OriginConfig, 1)
marshaler[ocjh.OriginConfig.jsonType()] = ocjh.OriginConfig
return json.Marshal(marshaler)
}
func (ocjh *OriginConfigJSONHandler) UnmarshalJSON(b []byte) error {
var originJSON map[string]interface{}
if err := json.Unmarshal(b, &originJSON); err != nil {
return errors.Wrapf(err, "cannot unmarshal %s into originJSON", string(b))
}
if originConfig, ok := originJSON[httpType.String()]; ok {
httpOriginConfig := &HTTPOriginConfig{}
if err := mapstructure.Decode(originConfig, httpOriginConfig); err != nil {
return errors.Wrapf(err, "cannot decode %+v into HTTPOriginConfig", originConfig)
}
ocjh.OriginConfig = httpOriginConfig
return nil
}
if originConfig, ok := originJSON[wsType.String()]; ok {
wsOriginConfig := &WebSocketOriginConfig{}
if err := mapstructure.Decode(originConfig, wsOriginConfig); err != nil {
return errors.Wrapf(err, "cannot decode %+v into WebSocketOriginConfig", originConfig)
}
ocjh.OriginConfig = wsOriginConfig
return nil
}
if originConfig, ok := originJSON[helloWorldType.String()]; ok {
helloWorldOriginConfig := &HelloWorldOriginConfig{}
if err := mapstructure.Decode(originConfig, helloWorldOriginConfig); err != nil {
return errors.Wrapf(err, "cannot decode %+v into HelloWorldOriginConfig", originConfig)
}
ocjh.OriginConfig = helloWorldOriginConfig
return nil
}
return fmt.Errorf("cannot unmarshal %s into OriginConfig", string(b))
}
// FallibleConfigMarshaler is a wrapper for FallibleConfig to implement custom marshal logic
type FallibleConfigMarshaler struct {
FallibleConfig FallibleConfig
}
func (fcm *FallibleConfigMarshaler) MarshalJSON() ([]byte, error) {
marshaler := make(map[string]FallibleConfig, 1)
marshaler[fcm.FallibleConfig.jsonType()] = fcm.FallibleConfig
return json.Marshal(marshaler)
}
+342
View File
@@ -0,0 +1,342 @@
package pogs
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
)
func TestScopeUnmarshaler_UnmarshalJSON(t *testing.T) {
type fields struct {
Scope Scope
}
type args struct {
b []byte
}
tests := []struct {
name string
fields fields
args args
wantErr bool
wantScope Scope
}{
{
name: "group_successful",
args: args{b: []byte(`{"group": "my-group"}`)},
wantScope: NewGroup("my-group"),
},
{
name: "system_name_successful",
args: args{b: []byte(`{"system_name": "my-computer"}`)},
wantScope: NewSystemName("my-computer"),
},
{
name: "not_a_scope",
args: args{b: []byte(`{"x": "y"}`)},
wantErr: true,
},
{
name: "malformed_group",
args: args{b: []byte(`{"group": ["a", "b"]}`)},
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
su := &ScopeUnmarshaler{
Scope: tt.fields.Scope,
}
err := su.UnmarshalJSON(tt.args.b)
if !tt.wantErr {
if err != nil {
t.Errorf("ScopeUnmarshaler.UnmarshalJSON() error = %v, wantErr %v", err, tt.wantErr)
}
if !eqScope(tt.wantScope, su.Scope) {
t.Errorf("Wanted scope %v but got scope %v", tt.wantScope, su.Scope)
}
}
})
}
}
func TestUnmarshalOrigin(t *testing.T) {
tests := []struct {
jsonLiteral string
exceptedOriginConfig OriginConfig
}{
{
jsonLiteral: `{
"Http":{
"url_string":"https.example.com",
"tcp_keep_alive":7000000000,
"dial_dual_stack":true,
"tls_handshake_timeout":11000000000,
"tls_verify":true,
"origin_ca_pool":"/etc/cert.pem",
"origin_server_name":"secure.example.com",
"max_idle_connections":19,
"idle_connection_timeout":17000000000,
"proxy_connection_timeout":15000000000,
"expect_continue_timeout":21000000000,
"chunked_encoding":true
}
}`,
exceptedOriginConfig: sampleHTTPOriginConfig(),
},
{
jsonLiteral: `{
"WebSocket":{
"url_string":"ssh://example.com",
"tls_verify":true,
"origin_ca_pool":"/etc/cert.pem",
"origin_server_name":"secure.example.com"
}
}`,
exceptedOriginConfig: sampleWebSocketOriginConfig(),
},
{
jsonLiteral: `{
"HelloWorld": {}
}`,
exceptedOriginConfig: &HelloWorldOriginConfig{},
},
}
for _, test := range tests {
originConfigJSON := prettyToValidJSON(test.jsonLiteral)
var OriginConfigJSONHandler OriginConfigJSONHandler
err := json.Unmarshal([]byte(originConfigJSON), &OriginConfigJSONHandler)
assert.NoError(t, err)
assert.Equal(t, test.exceptedOriginConfig, OriginConfigJSONHandler.OriginConfig)
}
}
func TestUnmarshalClientConfig(t *testing.T) {
prettyClientConfigJSON := `{
"version":10,
"supervisor_config":{
"auto_update_frequency":86400000000000,
"metrics_update_frequency":300000000000,
"grace_period":30000000000
},
"edge_connection_config":{
"num_ha_connections":4,
"heartbeat_interval":5000000000,
"timeout":30000000000,
"max_failed_heartbeats":5,
"user_credential_path":"~/.cloudflared/cert.pem"
},
"doh_proxy_configs":[{
"listen_host": "localhost",
"listen_port": 53,
"upstreams": ["https://1.1.1.1/dns-query", "https://1.0.0.1/dns-query"]
}],
"reverse_proxy_configs":[{
"tunnel_hostname":"sdfjadk33.cftunnel.com",
"origin_config":{
"Http":{
"url_string":"https://127.0.0.1:8080",
"tcp_keep_alive":30000000000,
"dial_dual_stack":true,
"tls_handshake_timeout":10000000000,
"tls_verify":true,
"origin_ca_pool":"",
"origin_server_name":"",
"max_idle_connections":100,
"idle_connection_timeout":90000000000,
"proxy_connection_timeout":90000000000,
"expect_continue_timeout":90000000000,
"chunked_encoding":true
}
},
"retries":5,
"connection_timeout":30,
"compression_quality":0
}]
}`
// replace new line and tab
clientConfigJSON := prettyToValidJSON(prettyClientConfigJSON)
var clientConfig ClientConfig
err := json.Unmarshal([]byte(clientConfigJSON), &clientConfig)
assert.NoError(t, err)
assert.Equal(t, Version(10), clientConfig.Version)
supervisorConfig := SupervisorConfig{
AutoUpdateFrequency: time.Hour * 24,
MetricsUpdateFrequency: time.Second * 300,
GracePeriod: time.Second * 30,
}
assert.Equal(t, supervisorConfig, *clientConfig.SupervisorConfig)
edgeConnectionConfig := EdgeConnectionConfig{
NumHAConnections: 4,
HeartbeatInterval: time.Second * 5,
Timeout: time.Second * 30,
MaxFailedHeartbeats: 5,
UserCredentialPath: "~/.cloudflared/cert.pem",
}
assert.Equal(t, edgeConnectionConfig, *clientConfig.EdgeConnectionConfig)
dohProxyConfig := DoHProxyConfig{
ListenHost: "localhost",
ListenPort: 53,
Upstreams: []string{"https://1.1.1.1/dns-query", "https://1.0.0.1/dns-query"},
}
assert.Len(t, clientConfig.DoHProxyConfigs, 1)
assert.Equal(t, dohProxyConfig, *clientConfig.DoHProxyConfigs[0])
reverseProxyConfig := ReverseProxyConfig{
TunnelHostname: "sdfjadk33.cftunnel.com",
OriginConfigJSONHandler: &OriginConfigJSONHandler{
OriginConfig: &HTTPOriginConfig{
URLString: "https://127.0.0.1:8080",
TCPKeepAlive: time.Second * 30,
DialDualStack: true,
TLSHandshakeTimeout: time.Second * 10,
TLSVerify: true,
OriginCAPool: "",
OriginServerName: "",
MaxIdleConnections: 100,
IdleConnectionTimeout: time.Second * 90,
ProxyConnectionTimeout: time.Second * 90,
ExpectContinueTimeout: time.Second * 90,
ChunkedEncoding: true,
},
},
Retries: 5,
ConnectionTimeout: 30,
CompressionQuality: 0,
}
assert.Len(t, clientConfig.ReverseProxyConfigs, 1)
assert.Equal(t, reverseProxyConfig, *clientConfig.ReverseProxyConfigs[0])
}
func TestMarshalFallibleConfig(t *testing.T) {
tests := []struct {
fallibleConfig FallibleConfig
expctedJSONLiteral string
}{
{
fallibleConfig: sampleSupervisorConfig(),
expctedJSONLiteral: `{
"supervisor_config":{
"auto_update_frequency":75600000000000,
"metrics_update_frequency":660000000000,
"grace_period":31000000000
}
}`,
},
{
fallibleConfig: sampleEdgeConnectionConfig(),
expctedJSONLiteral: `{
"edge_connection_config":{
"num_ha_connections":49,
"heartbeat_interval":5000000000,
"timeout":9000000000,
"max_failed_heartbeats":9001,
"user_credential_path":"/Users/example/.cloudflared/cert.pem"
}
}`,
},
{
fallibleConfig: sampleDoHProxyConfig(),
expctedJSONLiteral: `{
"doh_proxy_config":{
"listen_host":"127.0.0.1",
"listen_port":53,
"upstreams":["1.1.1.1","1.0.0.1"]
}
}`,
},
{
fallibleConfig: sampleReverseProxyConfig(func(c *ReverseProxyConfig) {
c.OriginConfigJSONHandler = &OriginConfigJSONHandler{sampleHTTPOriginConfig()}
}),
expctedJSONLiteral: `{
"reverse_proxy_config":{
"tunnel_hostname":"mock-non-lb-tunnel.example.com",
"origin_config":{
"Http":{
"url_string":"https.example.com",
"tcp_keep_alive":7000000000,
"dial_dual_stack":true,
"tls_handshake_timeout":11000000000,
"tls_verify":true,
"origin_ca_pool":"/etc/cert.pem",
"origin_server_name":"secure.example.com",
"max_idle_connections":19,
"idle_connection_timeout":17000000000,
"proxy_connection_timeout":15000000000,
"expect_continue_timeout":21000000000,
"chunked_encoding":true
}
},
"retries":18,
"connection_timeout":5000000000,
"compression_quality":3
}
}`,
},
{
fallibleConfig: sampleReverseProxyConfig(func(c *ReverseProxyConfig) {
c.OriginConfigJSONHandler = &OriginConfigJSONHandler{sampleWebSocketOriginConfig()}
}),
expctedJSONLiteral: `{
"reverse_proxy_config":{
"tunnel_hostname":"mock-non-lb-tunnel.example.com",
"origin_config":{
"WebSocket":{
"url_string":"ssh://example.com",
"tls_verify":true,
"origin_ca_pool":"/etc/cert.pem",
"origin_server_name":"secure.example.com"
}
},
"retries":18,
"connection_timeout":5000000000,
"compression_quality":3
}
}`,
},
{
fallibleConfig: sampleReverseProxyConfig(func(c *ReverseProxyConfig) {
c.OriginConfigJSONHandler = &OriginConfigJSONHandler{&HelloWorldOriginConfig{}}
}),
expctedJSONLiteral: `{
"reverse_proxy_config":{
"tunnel_hostname":"mock-non-lb-tunnel.example.com",
"origin_config":{
"HelloWorld":{}
},
"retries":18,
"connection_timeout":5000000000,
"compression_quality":3
}
}`,
},
}
for _, test := range tests {
b, err := json.Marshal(test.fallibleConfig)
assert.NoError(t, err)
assert.Equal(t, prettyToValidJSON(test.expctedJSONLiteral), string(b))
}
}
type prettyJSON string
func prettyToValidJSON(prettyJSON string) string {
return strings.ReplaceAll(strings.ReplaceAll(prettyJSON, "\n", ""), "\t", "")
}
func eqScope(s1, s2 Scope) bool {
return s1.Value() == s2.Value() && s1.PostgresType() == s2.PostgresType()
}
+145 -21
View File
@@ -2,6 +2,7 @@ package pogs
import (
"context"
"fmt"
"time"
"github.com/cloudflare/cloudflared/tunnelrpc"
@@ -127,52 +128,175 @@ func UnmarshalServerInfo(s tunnelrpc.ServerInfo) (*ServerInfo, error) {
return p, err
}
//go-sumtype:decl Scope
type Scope interface {
Value() string
PostgresType() string
GraphQLType() string
isScope()
}
type SystemName struct {
systemName string
}
func NewSystemName(systemName string) *SystemName {
return &SystemName{systemName: systemName}
}
func (s *SystemName) Value() string { return s.systemName }
func (_ *SystemName) PostgresType() string { return "system_name" }
func (_ *SystemName) GraphQLType() string { return "SYSTEM_NAME" }
func (_ *SystemName) isScope() {}
type Group struct {
group string
}
func NewGroup(group string) *Group {
return &Group{group: group}
}
func (g *Group) Value() string { return g.group }
func (_ *Group) PostgresType() string { return "group" }
func (_ *Group) GraphQLType() string { return "GROUP" }
func (_ *Group) isScope() {}
func MarshalScope(s tunnelrpc.Scope, p Scope) error {
ss := s.Value()
switch scope := p.(type) {
case *SystemName:
ss.SetSystemName(scope.systemName)
case *Group:
ss.SetGroup(scope.group)
default:
return fmt.Errorf("unexpected Scope value: %v", p)
}
return nil
}
func UnmarshalScope(s tunnelrpc.Scope) (Scope, error) {
ss := s.Value()
switch ss.Which() {
case tunnelrpc.Scope_value_Which_systemName:
systemName, err := ss.SystemName()
if err != nil {
return nil, err
}
return NewSystemName(systemName), nil
case tunnelrpc.Scope_value_Which_group:
group, err := ss.Group()
if err != nil {
return nil, err
}
return NewGroup(group), nil
default:
return nil, fmt.Errorf("unexpected Scope tag: %v", ss.Which())
}
}
type ConnectParameters struct {
OriginCert []byte
CloudflaredID uuid.UUID
NumPreviousAttempts uint8
Tags []Tag
CloudflaredVersion string
}
// CapnpConnectParameters is ConnectParameters represented in Cap'n Proto build-in types
type CapnpConnectParameters struct {
OriginCert []byte
CloudflaredID []byte
NumPreviousAttempts uint8
Tags []Tag
CloudflaredVersion string
Scope Scope
}
func MarshalConnectParameters(s tunnelrpc.CapnpConnectParameters, p *ConnectParameters) error {
if err := s.SetOriginCert(p.OriginCert); err != nil {
return err
}
cloudflaredIDBytes, err := p.CloudflaredID.MarshalBinary()
if err != nil {
return err
}
capnpConnectParameters := &CapnpConnectParameters{
OriginCert: p.OriginCert,
CloudflaredID: cloudflaredIDBytes,
NumPreviousAttempts: p.NumPreviousAttempts,
CloudflaredVersion: p.CloudflaredVersion,
if err := s.SetCloudflaredID(cloudflaredIDBytes); err != nil {
return err
}
return pogs.Insert(tunnelrpc.CapnpConnectParameters_TypeID, s.Struct, capnpConnectParameters)
s.SetNumPreviousAttempts(p.NumPreviousAttempts)
if len(p.Tags) > 0 {
tagsCapnpList, err := s.NewTags(int32(len(p.Tags)))
if err != nil {
return err
}
for i, tag := range p.Tags {
tagCapnp := tagsCapnpList.At(i)
if err := tagCapnp.SetName(tag.Name); err != nil {
return err
}
if err := tagCapnp.SetValue(tag.Value); err != nil {
return err
}
}
}
if err := s.SetCloudflaredVersion(p.CloudflaredVersion); err != nil {
return err
}
scope, err := s.NewScope()
if err != nil {
return err
}
return MarshalScope(scope, p.Scope)
}
func UnmarshalConnectParameters(s tunnelrpc.CapnpConnectParameters) (*ConnectParameters, error) {
p := new(CapnpConnectParameters)
err := pogs.Extract(p, tunnelrpc.CapnpConnectParameters_TypeID, s.Struct)
originCert, err := s.OriginCert()
if err != nil {
return nil, err
}
cloudflaredID, err := uuid.FromBytes(p.CloudflaredID)
cloudflaredIDBytes, err := s.CloudflaredID()
if err != nil {
return nil, err
}
cloudflaredID, err := uuid.FromBytes(cloudflaredIDBytes)
if err != nil {
return nil, err
}
tagsCapnpList, err := s.Tags()
if err != nil {
return nil, err
}
var tags []Tag
for i := 0; i < tagsCapnpList.Len(); i++ {
tagCapnp := tagsCapnpList.At(i)
name, err := tagCapnp.Name()
if err != nil {
return nil, err
}
value, err := tagCapnp.Value()
if err != nil {
return nil, err
}
tags = append(tags, Tag{Name: name, Value: value})
}
cloudflaredVersion, err := s.CloudflaredVersion()
if err != nil {
return nil, err
}
scopeCapnp, err := s.Scope()
if err != nil {
return nil, err
}
scope, err := UnmarshalScope(scopeCapnp)
if err != nil {
return nil, err
}
return &ConnectParameters{
OriginCert: p.OriginCert,
OriginCert: originCert,
CloudflaredID: cloudflaredID,
NumPreviousAttempts: p.NumPreviousAttempts,
CloudflaredVersion: p.CloudflaredVersion,
NumPreviousAttempts: s.NumPreviousAttempts(),
Tags: tags,
CloudflaredVersion: cloudflaredVersion,
Scope: scope,
}, nil
}
+97
View File
@@ -0,0 +1,97 @@
package pogs
import (
"reflect"
"testing"
"github.com/cloudflare/cloudflared/tunnelrpc"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
capnp "zombiezen.com/go/capnproto2"
)
// Assert *SystemName implements Scope
var _ Scope = (*SystemName)(nil)
// Assert *Group implements Scope
var _ Scope = (*Group)(nil)
func TestScope(t *testing.T) {
testCases := []Scope{
&SystemName{systemName: "my_system"},
&Group{group: "my_group"},
}
for i, testCase := range testCases {
_, seg, err := capnp.NewMessage(capnp.SingleSegment(nil))
capnpEntity, err := tunnelrpc.NewScope(seg)
if !assert.NoError(t, err) {
t.Fatal("Couldn't initialize a new message")
}
err = MarshalScope(capnpEntity, testCase)
if !assert.NoError(t, err, "testCase index %v failed to marshal", i) {
continue
}
result, err := UnmarshalScope(capnpEntity)
if !assert.NoError(t, err, "testCase index %v failed to unmarshal", i) {
continue
}
assert.Equal(t, testCase, result, "testCase index %v didn't preserve struct through marshalling and unmarshalling", i)
}
}
func TestConnectParameters(t *testing.T) {
testCases := []*ConnectParameters{
sampleConnectParameters(),
sampleConnectParameters(func(c *ConnectParameters) {
c.Scope = &SystemName{systemName: "my_system"}
}),
sampleConnectParameters(func(c *ConnectParameters) {
c.Tags = nil
}),
}
for i, testCase := range testCases {
_, seg, err := capnp.NewMessage(capnp.SingleSegment(nil))
capnpEntity, err := tunnelrpc.NewCapnpConnectParameters(seg)
if !assert.NoError(t, err) {
t.Fatal("Couldn't initialize a new message")
}
err = MarshalConnectParameters(capnpEntity, testCase)
if !assert.NoError(t, err, "testCase index %v failed to marshal", i) {
continue
}
result, err := UnmarshalConnectParameters(capnpEntity)
if !assert.NoError(t, err, "testCase index %v failed to unmarshal", i) {
continue
}
assert.Equal(t, testCase, result, "testCase index %v didn't preserve struct through marshalling and unmarshalling", i)
}
}
func sampleConnectParameters(overrides ...func(*ConnectParameters)) *ConnectParameters {
cloudflaredID, err := uuid.Parse("ED7BA470-8E54-465E-825C-99712043E01C")
if err != nil {
panic(err)
}
sample := &ConnectParameters{
OriginCert: []byte("my-origin-cert"),
CloudflaredID: cloudflaredID,
NumPreviousAttempts: 19,
Tags: []Tag{
Tag{
Name: "provision-method",
Value: "new",
},
},
CloudflaredVersion: "7.0",
Scope: &Group{group: "my_group"},
}
sample.ensureNoZeroFields()
for _, f := range overrides {
f(sample)
}
return sample
}
func (c *ConnectParameters) ensureNoZeroFields() {
ensureNoZeroFieldsInSample(reflect.ValueOf(c), []string{})
}
+66 -45
View File
@@ -57,6 +57,17 @@ struct CapnpConnectParameters {
tags @3 :List(Tag);
# release version of cloudflared
cloudflaredVersion @4 :Text;
# identifier for this cloudflared instance
scope @5 :Scope;
}
struct Scope {
value :union {
# Standalone instance
systemName @0 :Text;
# Associated with a group of identical cloudflared instances
group @1 :Text;
}
}
struct ConnectResult {
@@ -77,35 +88,48 @@ struct ClientConfig {
# to monotonically increase in value. Any configuration supplied to
# useConfiguration() with a smaller `version` should be ignored.
version @0 :UInt64;
# supervisorConfig is configuration for supervisor, the component that manages connection manager,
# autoupdater and metrics server
supervisorConfig @1 :SupervisorConfig;
# edgeConnectionConfig is configuration for connection manager, the componenet that manages connections with the edge
edgeConnectionConfig @2 :EdgeConnectionConfig;
# Configuration for cloudflared to run as a DNS-over-HTTPS proxy.
# cloudflared CLI option: `proxy-dns`
dohProxyConfigs @3 :List(DoHProxyConfig);
# Configuration for cloudflared to run as an HTTP reverse proxy.
reverseProxyConfigs @4 :List(ReverseProxyConfig);
}
struct SupervisorConfig {
# Frequency (in ns) to check Equinox for updates.
# Zero means auto-update is disabled.
# cloudflared CLI option: `autoupdate-freq`
autoUpdateFrequency @1 :Int64;
autoUpdateFrequency @0 :Int64;
# Frequency (in ns) to update connection-based metrics.
# cloudflared CLI option: `metrics-update-freq`
metricsUpdateFrequency @2 :Int64;
# interval (in ns) between heartbeats with the Cloudflare edge
# cloudflared CLI option: `heartbeat-interval`
heartbeatInterval @3 :Int64;
# Minimum number of unacked heartbeats for cloudflared to send before
# closing the connection to the edge.
# cloudflared CLI option: `heartbeat-count`
maxFailedHeartbeats @4 :UInt64;
metricsUpdateFrequency @1 :Int64;
# Time (in ns) to continue serving requests after cloudflared receives its
# first SIGINT/SIGTERM. A second SIGINT/SIGTERM will force cloudflared to
# shutdown immediately. For example, this field can be used to gracefully
# transition traffic to another cloudflared instance.
# cloudflared CLI option: `grace-period`
gracePeriod @5 :Int64;
# Configuration for cloudflared to run as a DNS-over-HTTPS proxy.
# cloudflared CLI option: `proxy-dns`
dohProxyConfigs @6 :List(DoHProxyConfig);
# Configuration for cloudflared to run as an HTTP reverse proxy.
reverseProxyConfigs @7 :List(ReverseProxyConfig);
# Number of persistent connections to keep open between cloudflared and
# the edge.
gracePeriod @2 :Int64;
}
struct EdgeConnectionConfig {
# cloudflared CLI option: `ha-connections`
numHAConnections @8 :UInt8;
numHAConnections @0 :UInt8;
# Interval (in ns) between heartbeats with the Cloudflare edge
# cloudflared CLI option: `heartbeat-interval`
heartbeatInterval @1 :Int64;
# Maximum wait time to connect with the edge.
timeout @2 :Int64;
# Number of unacked heartbeats for cloudflared to send before
# closing the connection to the edge.
# cloudflared CLI option: `heartbeat-count`
maxFailedHeartbeats @3 :UInt64;
# Absolute path of the file containing certificate and token to connect with the edge
userCredentialPath @4 :Text;
}
struct ReverseProxyConfig {
@@ -134,7 +158,7 @@ struct WebSocketOriginConfig {
# cloudflared will start a websocket server that forwards data to this URI
# cloudflared CLI option: `url`
# cloudflared logic: https://github.com/cloudflare/cloudflared/blob/2019.3.2/cmd/cloudflared/tunnel/cmd.go#L304
url @0 :Text;
urlString @0 :Text;
# Whether cloudflared should verify TLS connections to the origin.
# negation of cloudflared CLI option: `no-tls-verify`
tlsVerify @1 :Bool;
@@ -155,25 +179,22 @@ struct WebSocketOriginConfig {
struct HTTPOriginConfig {
# HTTP(S) URL of the origin service.
# cloudflared CLI option: `url`
originAddr :union {
http @0 :CapnpHTTPURL;
unix @1 :UnixPath;
}
urlString @0 :Text;
# the TCP keep-alive period (in ns) for an active network connection.
# Zero means keep-alives are not enabled.
# cloudflared CLI option: `proxy-tcp-keepalive`
tcpKeepAlive @2 :Int64;
tcpKeepAlive @1 :Int64;
# whether cloudflared should use a "happy eyeballs"-compliant procedure
# to connect to origins that resolve to both IPv4 and IPv6 addresses
# negation of cloudflared CLI option: `proxy-no-happy-eyeballs`
dialDualStack @3 :Bool;
dialDualStack @2 :Bool;
# maximum time (in ns) for cloudflared to wait for a TLS handshake
# with the origin. Zero means no timeout.
# cloudflared CLI option: `proxy-tls-timeout`
tlsHandshakeTimeout @4 :Int64;
tlsHandshakeTimeout @3 :Int64;
# Whether cloudflared should verify TLS connections to the origin.
# negation of cloudflared CLI option: `no-tls-verify`
tlsVerify @5 :Bool;
tlsVerify @4 :Bool;
# originCAPool specifies the root CA that cloudflared should use when
# verifying TLS connections to the origin.
# - if tlsVerify is false, originCAPool will be ignored.
@@ -182,39 +203,29 @@ struct HTTPOriginConfig {
# - if tlsVerify is true and originCAPool is non-empty, cloudflared will
# treat it as the filepath to the root CA.
# cloudflared CLI option: `origin-ca-pool`
originCAPool @6 :Text;
originCAPool @5 :Text;
# Hostname to use when verifying TLS connections to the origin.
# cloudflared CLI option: `origin-server-name`
originServerName @7 :Text;
originServerName @6 :Text;
# maximum number of idle (keep-alive) connections for cloudflared to
# keep open with the origin. Zero means no limit.
# cloudflared CLI option: `proxy-keepalive-connections`
maxIdleConnections @8 :UInt64;
maxIdleConnections @7 :UInt64;
# maximum time (in ns) for an idle (keep-alive) connection to remain
# idle before closing itself. Zero means no timeout.
# cloudflared CLI option: `proxy-keepalive-timeout`
idleConnectionTimeout @9 :Int64;
idleConnectionTimeout @8 :Int64;
# maximum amount of time a dial will wait for a connect to complete.
proxyConnectionTimeout @10 :Int64;
proxyConnectionTimeout @9 :Int64;
# The amount of time to wait for origin's first response headers after fully
# writing the request headers if the request has an "Expect: 100-continue" header.
# Zero means no timeout and causes the body to be sent immediately, without
# waiting for the server to approve.
expectContinueTimeout @11 :Int64;
expectContinueTimeout @10 :Int64;
# Whether cloudflared should allow chunked transfer encoding to the
# origin. (This should be disabled for WSGI origins, for example.)
# negation of cloudflared CLI option: `no-chunked-encoding`
chunkedEncoding @12 :Bool;
}
# URL for a HTTP origin, capnp doesn't have native support for URL, so represent it as Text
struct CapnpHTTPURL {
url @0: Text;
}
# Path to a unix socket
struct UnixPath {
path @0: Text;
chunkedEncoding @11 :Bool;
}
# configuration for cloudflared to provide a DNS over HTTPS proxy server
@@ -251,7 +262,17 @@ struct ServerInfo {
struct UseConfigurationResult {
success @0 :Bool;
errorMessage @1 :Text;
failedConfigs @1 :List(FailedConfig);
}
struct FailedConfig {
config :union {
supervisor @0 :SupervisorConfig;
edgeConnection @1 :EdgeConnectionConfig;
doh @2 :DoHProxyConfig;
reverseProxy @3 :ReverseProxyConfig;
}
reason @4 :Text;
}
interface TunnelServer {
+1077 -514
View File
File diff suppressed because it is too large Load Diff
+56 -2
View File
@@ -1,6 +1,7 @@
package validation
import (
"context"
"fmt"
"net"
"net/url"
@@ -9,15 +10,21 @@ import (
"net/http"
"github.com/coreos/go-oidc"
"github.com/pkg/errors"
"golang.org/x/net/idna"
)
const defaultScheme = "http"
const (
defaultScheme = "http"
accessDomain = "cloudflareaccess.com"
accessCertPath = "/cdn-cgi/access/certs"
accessJwtHeader = "Cf-access-jwt-assertion"
)
var (
supportedProtocols = []string{"http", "https", "rdp"}
validationTimeout = time.Duration(30 * time.Second)
validationTimeout = time.Duration(30 * time.Second)
)
func ValidateHostname(hostname string) (string, error) {
@@ -197,3 +204,50 @@ func toggleProtocol(httpProtocol string) string {
return httpProtocol
}
}
// Access checks if a JWT from Cloudflare Access is valid.
type Access struct {
verifier *oidc.IDTokenVerifier
}
func NewAccessValidator(ctx context.Context, domain, issuer, applicationAUD string) (*Access, error) {
domainURL, err := ValidateUrl(domain)
if err != nil {
return nil, err
}
issuerURL, err := ValidateUrl(issuer)
if err != nil {
return nil, err
}
// An issuerURL from Cloudflare Access will always use HTTPS.
issuerURL = strings.Replace(issuerURL, "http:", "https:", 1)
keySet := oidc.NewRemoteKeySet(ctx, domainURL+accessCertPath)
return &Access{oidc.NewVerifier(issuerURL, keySet, &oidc.Config{ClientID: applicationAUD})}, nil
}
func (a *Access) Validate(ctx context.Context, jwt string) error {
token, err := a.verifier.Verify(ctx, jwt)
if err != nil {
return errors.Wrapf(err, "token is invalid: %s", jwt)
}
// Perform extra sanity checks, just to be safe.
if token == nil {
return fmt.Errorf("token is nil: %s", jwt)
}
if !strings.HasSuffix(token.Issuer, accessDomain) {
return fmt.Errorf("token has non-cloudflare issuer of %s: %s", token.Issuer, jwt)
}
return nil
}
func (a *Access) ValidateRequest(ctx context.Context, r *http.Request) error {
return a.Validate(ctx, r.Header.Get(accessJwtHeader))
}
+45 -106
View File
@@ -180,33 +180,6 @@ func TestValidateHTTPService_HTTP2HTTP(t *testing.T) {
}
panic("Shouldn't reach here")
})))
// Integration-style test with a mock server
server, client, err := createMockServerAndClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, hostname, r.Host)
w.WriteHeader(200)
}))
assert.NoError(t, err)
defer server.Close()
assert.Nil(t, ValidateHTTPService(originURL, hostname, client.Transport))
// this will fail if the client follows the 302
redirectServer, redirectClient, err := createMockServerAndClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/followedRedirect" {
t.Fatal("shouldn't have followed the 302")
}
if r.Method == "CONNECT" {
assert.Equal(t, "127.0.0.1:443", r.Host)
} else {
assert.Equal(t, hostname, r.Host)
}
w.Header().Set("Location", "/followedRedirect")
w.WriteHeader(302)
}))
assert.NoError(t, err)
defer redirectServer.Close()
assert.Nil(t, ValidateHTTPService(originURL, hostname, redirectClient.Transport))
}
// Happy path 2: originURL is HTTPS, and HTTPS connections work
@@ -235,36 +208,6 @@ func TestValidateHTTPService_HTTPS2HTTPS(t *testing.T) {
}
panic("Shouldn't reach here")
})))
// Integration-style test with a mock server
server, client, err := createSecureMockServerAndClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == "CONNECT" {
assert.Equal(t, "127.0.0.1:443", r.Host)
} else {
assert.Equal(t, hostname, r.Host)
}
w.WriteHeader(200)
}))
assert.NoError(t, err)
defer server.Close()
assert.Nil(t, ValidateHTTPService(originURL, hostname, client.Transport))
// this will fail if the client follows the 302
redirectServer, redirectClient, err := createSecureMockServerAndClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/followedRedirect" {
t.Fatal("shouldn't have followed the 302")
}
if r.Method == "CONNECT" {
assert.Equal(t, "127.0.0.1:443", r.Host)
} else {
assert.Equal(t, hostname, r.Host)
}
w.Header().Set("Location", "/followedRedirect")
w.WriteHeader(302)
}))
assert.NoError(t, err)
defer redirectServer.Close()
assert.Nil(t, ValidateHTTPService(originURL, hostname, redirectClient.Transport))
}
// Error path 1: originURL is HTTPS, but HTTP connections work
@@ -293,37 +236,6 @@ func TestValidateHTTPService_HTTPS2HTTP(t *testing.T) {
}
panic("Shouldn't reach here")
})))
// Integration-style test with a mock server
server, client, err := createMockServerAndClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == "CONNECT" {
assert.Equal(t, "127.0.0.1:1234", r.Host)
} else {
assert.Equal(t, hostname, r.Host)
}
w.WriteHeader(200)
}))
assert.NoError(t, err)
defer server.Close()
assert.Error(t, ValidateHTTPService(originURL, hostname, client.Transport))
// this will fail if the client follows the 302
redirectServer, redirectClient, err := createMockServerAndClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/followedRedirect" {
t.Fatal("shouldn't have followed the 302")
}
if r.Method == "CONNECT" {
assert.Equal(t, "127.0.0.1:1234", r.Host)
} else {
assert.Equal(t, hostname, r.Host)
}
w.Header().Set("Location", "/followedRedirect")
w.WriteHeader(302)
}))
assert.NoError(t, err)
defer redirectServer.Close()
assert.Error(t, ValidateHTTPService(originURL, hostname, redirectClient.Transport))
}
// Error path 2: originURL is HTTP, but HTTPS connections work
@@ -352,21 +264,11 @@ func TestValidateHTTPService_HTTP2HTTPS(t *testing.T) {
}
panic("Shouldn't reach here")
})))
}
// Integration-style test with a mock server
server, client, err := createSecureMockServerAndClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == "CONNECT" {
assert.Equal(t, "127.0.0.1:1234", r.Host)
} else {
assert.Equal(t, hostname, r.Host)
}
w.WriteHeader(200)
}))
assert.NoError(t, err)
defer server.Close()
assert.Error(t, ValidateHTTPService(originURL, hostname, client.Transport))
// this will fail if the client follows the 302
// Ensure the client does not follow 302 responses
func TestValidateHTTPService_NoFollowRedirects(t *testing.T) {
hostname := "example.com"
redirectServer, redirectClient, err := createSecureMockServerAndClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/followedRedirect" {
t.Fatal("shouldn't have followed the 302")
@@ -381,12 +283,12 @@ func TestValidateHTTPService_HTTP2HTTPS(t *testing.T) {
}))
assert.NoError(t, err)
defer redirectServer.Close()
assert.Error(t, ValidateHTTPService(originURL, hostname, redirectClient.Transport))
assert.NoError(t, ValidateHTTPService(redirectServer.URL, hostname, redirectClient.Transport))
}
// error path 3: origin URL is nonresponsive
// Ensure validation times out when origin URL is nonresponsive
func TestValidateHTTPService_NonResponsiveOrigin(t *testing.T) {
originURL := "https://127.0.0.1/"
originURL := "http://127.0.0.1/"
hostname := "example.com"
oldValidationTimeout := validationTimeout
defer func() {
@@ -394,7 +296,10 @@ func TestValidateHTTPService_NonResponsiveOrigin(t *testing.T) {
}()
validationTimeout = 500 * time.Millisecond
server, client, err := createSecureMockServerAndClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Use createMockServerAndClient, not createSecureMockServerAndClient.
// The latter will bail with HTTP 400 immediately on an http:// request,
// which defeats the purpose of a 'nonresponsive origin' test.
server, client, err := createMockServerAndClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == "CONNECT" {
assert.Equal(t, "127.0.0.1:443", r.Host)
} else {
@@ -409,11 +314,45 @@ func TestValidateHTTPService_NonResponsiveOrigin(t *testing.T) {
defer server.Close()
err = ValidateHTTPService(originURL, hostname, client.Transport)
fmt.Println(err)
if err, ok := err.(net.Error); assert.True(t, ok) {
assert.True(t, err.Timeout())
}
}
func TestNewAccessValidatorOk(t *testing.T) {
ctx := context.Background()
url := "test.cloudflareaccess.com"
access, err := NewAccessValidator(ctx, url, url, "")
assert.NoError(t, err)
assert.NotNil(t, access)
assert.Error(t, access.Validate(ctx, ""))
assert.Error(t, access.Validate(ctx, "invalid"))
req := httptest.NewRequest("GET", "https://test.cloudflareaccess.com", nil)
req.Header.Set(accessJwtHeader, "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c")
assert.Error(t, access.ValidateRequest(ctx, req))
}
func TestNewAccessValidatorErr(t *testing.T) {
ctx := context.Background()
urls := []string{
"",
"tcp://test.cloudflareaccess.com",
"wss://cloudflarenone.com",
}
for _, url := range urls {
access, err := NewAccessValidator(ctx, url, url, "")
assert.Error(t, err, url)
assert.Nil(t, access)
}
}
type testRoundTripper func(req *http.Request) (*http.Response, error)
func (f testRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
@@ -1,4 +1,5 @@
Apache License
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
@@ -178,7 +179,7 @@ Apache License
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
@@ -186,7 +187,7 @@ Apache License
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright {yyyy} {name of copyright owner}
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
+277
View File
@@ -0,0 +1,277 @@
// Copyright 2016 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package civil implements types for civil time, a time-zone-independent
// representation of time that follows the rules of the proleptic
// Gregorian calendar with exactly 24-hour days, 60-minute hours, and 60-second
// minutes.
//
// Because they lack location information, these types do not represent unique
// moments or intervals of time. Use time.Time for that purpose.
package civil
import (
"fmt"
"time"
)
// A Date represents a date (year, month, day).
//
// This type does not include location information, and therefore does not
// describe a unique 24-hour timespan.
type Date struct {
Year int // Year (e.g., 2014).
Month time.Month // Month of the year (January = 1, ...).
Day int // Day of the month, starting at 1.
}
// DateOf returns the Date in which a time occurs in that time's location.
func DateOf(t time.Time) Date {
var d Date
d.Year, d.Month, d.Day = t.Date()
return d
}
// ParseDate parses a string in RFC3339 full-date format and returns the date value it represents.
func ParseDate(s string) (Date, error) {
t, err := time.Parse("2006-01-02", s)
if err != nil {
return Date{}, err
}
return DateOf(t), nil
}
// String returns the date in RFC3339 full-date format.
func (d Date) String() string {
return fmt.Sprintf("%04d-%02d-%02d", d.Year, d.Month, d.Day)
}
// IsValid reports whether the date is valid.
func (d Date) IsValid() bool {
return DateOf(d.In(time.UTC)) == d
}
// In returns the time corresponding to time 00:00:00 of the date in the location.
//
// In is always consistent with time.Date, even when time.Date returns a time
// on a different day. For example, if loc is America/Indiana/Vincennes, then both
// time.Date(1955, time.May, 1, 0, 0, 0, 0, loc)
// and
// civil.Date{Year: 1955, Month: time.May, Day: 1}.In(loc)
// return 23:00:00 on April 30, 1955.
//
// In panics if loc is nil.
func (d Date) In(loc *time.Location) time.Time {
return time.Date(d.Year, d.Month, d.Day, 0, 0, 0, 0, loc)
}
// AddDays returns the date that is n days in the future.
// n can also be negative to go into the past.
func (d Date) AddDays(n int) Date {
return DateOf(d.In(time.UTC).AddDate(0, 0, n))
}
// DaysSince returns the signed number of days between the date and s, not including the end day.
// This is the inverse operation to AddDays.
func (d Date) DaysSince(s Date) (days int) {
// We convert to Unix time so we do not have to worry about leap seconds:
// Unix time increases by exactly 86400 seconds per day.
deltaUnix := d.In(time.UTC).Unix() - s.In(time.UTC).Unix()
return int(deltaUnix / 86400)
}
// Before reports whether d1 occurs before d2.
func (d1 Date) Before(d2 Date) bool {
if d1.Year != d2.Year {
return d1.Year < d2.Year
}
if d1.Month != d2.Month {
return d1.Month < d2.Month
}
return d1.Day < d2.Day
}
// After reports whether d1 occurs after d2.
func (d1 Date) After(d2 Date) bool {
return d2.Before(d1)
}
// MarshalText implements the encoding.TextMarshaler interface.
// The output is the result of d.String().
func (d Date) MarshalText() ([]byte, error) {
return []byte(d.String()), nil
}
// UnmarshalText implements the encoding.TextUnmarshaler interface.
// The date is expected to be a string in a format accepted by ParseDate.
func (d *Date) UnmarshalText(data []byte) error {
var err error
*d, err = ParseDate(string(data))
return err
}
// A Time represents a time with nanosecond precision.
//
// This type does not include location information, and therefore does not
// describe a unique moment in time.
//
// This type exists to represent the TIME type in storage-based APIs like BigQuery.
// Most operations on Times are unlikely to be meaningful. Prefer the DateTime type.
type Time struct {
Hour int // The hour of the day in 24-hour format; range [0-23]
Minute int // The minute of the hour; range [0-59]
Second int // The second of the minute; range [0-59]
Nanosecond int // The nanosecond of the second; range [0-999999999]
}
// TimeOf returns the Time representing the time of day in which a time occurs
// in that time's location. It ignores the date.
func TimeOf(t time.Time) Time {
var tm Time
tm.Hour, tm.Minute, tm.Second = t.Clock()
tm.Nanosecond = t.Nanosecond()
return tm
}
// ParseTime parses a string and returns the time value it represents.
// ParseTime accepts an extended form of the RFC3339 partial-time format. After
// the HH:MM:SS part of the string, an optional fractional part may appear,
// consisting of a decimal point followed by one to nine decimal digits.
// (RFC3339 admits only one digit after the decimal point).
func ParseTime(s string) (Time, error) {
t, err := time.Parse("15:04:05.999999999", s)
if err != nil {
return Time{}, err
}
return TimeOf(t), nil
}
// String returns the date in the format described in ParseTime. If Nanoseconds
// is zero, no fractional part will be generated. Otherwise, the result will
// end with a fractional part consisting of a decimal point and nine digits.
func (t Time) String() string {
s := fmt.Sprintf("%02d:%02d:%02d", t.Hour, t.Minute, t.Second)
if t.Nanosecond == 0 {
return s
}
return s + fmt.Sprintf(".%09d", t.Nanosecond)
}
// IsValid reports whether the time is valid.
func (t Time) IsValid() bool {
// Construct a non-zero time.
tm := time.Date(2, 2, 2, t.Hour, t.Minute, t.Second, t.Nanosecond, time.UTC)
return TimeOf(tm) == t
}
// MarshalText implements the encoding.TextMarshaler interface.
// The output is the result of t.String().
func (t Time) MarshalText() ([]byte, error) {
return []byte(t.String()), nil
}
// UnmarshalText implements the encoding.TextUnmarshaler interface.
// The time is expected to be a string in a format accepted by ParseTime.
func (t *Time) UnmarshalText(data []byte) error {
var err error
*t, err = ParseTime(string(data))
return err
}
// A DateTime represents a date and time.
//
// This type does not include location information, and therefore does not
// describe a unique moment in time.
type DateTime struct {
Date Date
Time Time
}
// Note: We deliberately do not embed Date into DateTime, to avoid promoting AddDays and Sub.
// DateTimeOf returns the DateTime in which a time occurs in that time's location.
func DateTimeOf(t time.Time) DateTime {
return DateTime{
Date: DateOf(t),
Time: TimeOf(t),
}
}
// ParseDateTime parses a string and returns the DateTime it represents.
// ParseDateTime accepts a variant of the RFC3339 date-time format that omits
// the time offset but includes an optional fractional time, as described in
// ParseTime. Informally, the accepted format is
// YYYY-MM-DDTHH:MM:SS[.FFFFFFFFF]
// where the 'T' may be a lower-case 't'.
func ParseDateTime(s string) (DateTime, error) {
t, err := time.Parse("2006-01-02T15:04:05.999999999", s)
if err != nil {
t, err = time.Parse("2006-01-02t15:04:05.999999999", s)
if err != nil {
return DateTime{}, err
}
}
return DateTimeOf(t), nil
}
// String returns the date in the format described in ParseDate.
func (dt DateTime) String() string {
return dt.Date.String() + "T" + dt.Time.String()
}
// IsValid reports whether the datetime is valid.
func (dt DateTime) IsValid() bool {
return dt.Date.IsValid() && dt.Time.IsValid()
}
// In returns the time corresponding to the DateTime in the given location.
//
// If the time is missing or ambigous at the location, In returns the same
// result as time.Date. For example, if loc is America/Indiana/Vincennes, then
// both
// time.Date(1955, time.May, 1, 0, 30, 0, 0, loc)
// and
// civil.DateTime{
// civil.Date{Year: 1955, Month: time.May, Day: 1}},
// civil.Time{Minute: 30}}.In(loc)
// return 23:30:00 on April 30, 1955.
//
// In panics if loc is nil.
func (dt DateTime) In(loc *time.Location) time.Time {
return time.Date(dt.Date.Year, dt.Date.Month, dt.Date.Day, dt.Time.Hour, dt.Time.Minute, dt.Time.Second, dt.Time.Nanosecond, loc)
}
// Before reports whether dt1 occurs before dt2.
func (dt1 DateTime) Before(dt2 DateTime) bool {
return dt1.In(time.UTC).Before(dt2.In(time.UTC))
}
// After reports whether dt1 occurs after dt2.
func (dt1 DateTime) After(dt2 DateTime) bool {
return dt2.Before(dt1)
}
// MarshalText implements the encoding.TextMarshaler interface.
// The output is the result of dt.String().
func (dt DateTime) MarshalText() ([]byte, error) {
return []byte(dt.String()), nil
}
// UnmarshalText implements the encoding.TextUnmarshaler interface.
// The datetime is expected to be a string in a format accepted by ParseDateTime
func (dt *DateTime) UnmarshalText(data []byte) error {
var err error
*dt, err = ParseDateTime(string(data))
return err
}
+3
View File
@@ -0,0 +1,3 @@
/examples/blog/blog
/examples/orders/orders
/examples/basic/basic
+25
View File
@@ -0,0 +1,25 @@
language: go
go_import_path: github.com/DATA-DOG/go-sqlmock
go:
- 1.2.x
- 1.3.x
- 1.4 # has no cover tool for latest releases
- 1.5.x
- 1.6.x
- 1.7.x
- 1.8.x
- 1.9.x
- 1.10.x
- 1.11.x
sudo: false
script:
- go vet
- test -z "$(go fmt ./...)" # fail if not formatted properly
- go test -race -coverprofile=coverage.txt -covermode=atomic
after_success:
- bash <(curl -s https://codecov.io/bash)
+28
View File
@@ -0,0 +1,28 @@
The three clause BSD license (http://en.wikipedia.org/wiki/BSD_licenses)
Copyright (c) 2013-2019, DATA-DOG team
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* The name DataDog.lt may not be used to endorse or promote products
derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL MICHAEL BOSTOCK BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+259
View File
@@ -0,0 +1,259 @@
[![Build Status](https://travis-ci.org/DATA-DOG/go-sqlmock.svg)](https://travis-ci.org/DATA-DOG/go-sqlmock)
[![GoDoc](https://godoc.org/github.com/DATA-DOG/go-sqlmock?status.svg)](https://godoc.org/github.com/DATA-DOG/go-sqlmock)
[![Go Report Card](https://goreportcard.com/badge/github.com/DATA-DOG/go-sqlmock)](https://goreportcard.com/report/github.com/DATA-DOG/go-sqlmock)
[![codecov.io](https://codecov.io/github/DATA-DOG/go-sqlmock/branch/master/graph/badge.svg)](https://codecov.io/github/DATA-DOG/go-sqlmock)
# Sql driver mock for Golang
**sqlmock** is a mock library implementing [sql/driver](https://godoc.org/database/sql/driver). Which has one and only
purpose - to simulate any **sql** driver behavior in tests, without needing a real database connection. It helps to
maintain correct **TDD** workflow.
- this library is now complete and stable. (you may not find new changes for this reason)
- supports concurrency and multiple connections.
- supports **go1.8** Context related feature mocking and Named sql parameters.
- does not require any modifications to your source code.
- the driver allows to mock any sql driver method behavior.
- has strict by default expectation order matching.
- has no third party dependencies.
**NOTE:** in **v1.2.0** **sqlmock.Rows** has changed to struct from interface, if you were using any type references to that
interface, you will need to switch it to a pointer struct type. Also, **sqlmock.Rows** were used to implement **driver.Rows**
interface, which was not required or useful for mocking and was removed. Hope it will not cause issues.
## Install
go get github.com/DATA-DOG/go-sqlmock
## Documentation and Examples
Visit [godoc](http://godoc.org/github.com/DATA-DOG/go-sqlmock) for general examples and public api reference.
See **.travis.yml** for supported **go** versions.
Different use case, is to functionally test with a real database - [go-txdb](https://github.com/DATA-DOG/go-txdb)
all database related actions are isolated within a single transaction so the database can remain in the same state.
See implementation examples:
- [blog API server](https://github.com/DATA-DOG/go-sqlmock/tree/master/examples/blog)
- [the same orders example](https://github.com/DATA-DOG/go-sqlmock/tree/master/examples/orders)
### Something you may want to test, assuming you use the [go-mysql-driver](https://github.com/go-sql-driver/mysql)
``` go
package main
import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
)
func recordStats(db *sql.DB, userID, productID int64) (err error) {
tx, err := db.Begin()
if err != nil {
return
}
defer func() {
switch err {
case nil:
err = tx.Commit()
default:
tx.Rollback()
}
}()
if _, err = tx.Exec("UPDATE products SET views = views + 1"); err != nil {
return
}
if _, err = tx.Exec("INSERT INTO product_viewers (user_id, product_id) VALUES (?, ?)", userID, productID); err != nil {
return
}
return
}
func main() {
// @NOTE: the real connection is not required for tests
db, err := sql.Open("mysql", "root@/blog")
if err != nil {
panic(err)
}
defer db.Close()
if err = recordStats(db, 1 /*some user id*/, 5 /*some product id*/); err != nil {
panic(err)
}
}
```
### Tests with sqlmock
``` go
package main
import (
"fmt"
"testing"
"github.com/DATA-DOG/go-sqlmock"
)
// a successful case
func TestShouldUpdateStats(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("an error '%s' was not expected when opening a stub database connection", err)
}
defer db.Close()
mock.ExpectBegin()
mock.ExpectExec("UPDATE products").WillReturnResult(sqlmock.NewResult(1, 1))
mock.ExpectExec("INSERT INTO product_viewers").WithArgs(2, 3).WillReturnResult(sqlmock.NewResult(1, 1))
mock.ExpectCommit()
// now we execute our method
if err = recordStats(db, 2, 3); err != nil {
t.Errorf("error was not expected while updating stats: %s", err)
}
// we make sure that all expectations were met
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("there were unfulfilled expectations: %s", err)
}
}
// a failing test case
func TestShouldRollbackStatUpdatesOnFailure(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("an error '%s' was not expected when opening a stub database connection", err)
}
defer db.Close()
mock.ExpectBegin()
mock.ExpectExec("UPDATE products").WillReturnResult(sqlmock.NewResult(1, 1))
mock.ExpectExec("INSERT INTO product_viewers").
WithArgs(2, 3).
WillReturnError(fmt.Errorf("some error"))
mock.ExpectRollback()
// now we execute our method
if err = recordStats(db, 2, 3); err == nil {
t.Errorf("was expecting an error, but there was none")
}
// we make sure that all expectations were met
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("there were unfulfilled expectations: %s", err)
}
}
```
## Customize SQL query matching
There were plenty of requests from users regarding SQL query string validation or different matching option.
We have now implemented the `QueryMatcher` interface, which can be passed through an option when calling
`sqlmock.New` or `sqlmock.NewWithDSN`.
This now allows to include some library, which would allow for example to parse and validate `mysql` SQL AST.
And create a custom QueryMatcher in order to validate SQL in sophisticated ways.
By default, **sqlmock** is preserving backward compatibility and default query matcher is `sqlmock.QueryMatcherRegexp`
which uses expected SQL string as a regular expression to match incoming query string. There is an equality matcher:
`QueryMatcherEqual` which will do a full case sensitive match.
In order to customize the QueryMatcher, use the following:
``` go
db, mock, err := sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherEqual))
```
The query matcher can be fully customized based on user needs. **sqlmock** will not
provide a standard sql parsing matchers, since various drivers may not follow the same SQL standard.
## Matching arguments like time.Time
There may be arguments which are of `struct` type and cannot be compared easily by value like `time.Time`. In this case
**sqlmock** provides an [Argument](https://godoc.org/github.com/DATA-DOG/go-sqlmock#Argument) interface which
can be used in more sophisticated matching. Here is a simple example of time argument matching:
``` go
type AnyTime struct{}
// Match satisfies sqlmock.Argument interface
func (a AnyTime) Match(v driver.Value) bool {
_, ok := v.(time.Time)
return ok
}
func TestAnyTimeArgument(t *testing.T) {
t.Parallel()
db, mock, err := New()
if err != nil {
t.Errorf("an error '%s' was not expected when opening a stub database connection", err)
}
defer db.Close()
mock.ExpectExec("INSERT INTO users").
WithArgs("john", AnyTime{}).
WillReturnResult(NewResult(1, 1))
_, err = db.Exec("INSERT INTO users(name, created_at) VALUES (?, ?)", "john", time.Now())
if err != nil {
t.Errorf("error '%s' was not expected, while inserting a row", err)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("there were unfulfilled expectations: %s", err)
}
}
```
It only asserts that argument is of `time.Time` type.
## Run tests
go test -race
## Change Log
- **2019-02-13** - added `go.mod` removed the references and suggestions using `gopkg.in`.
- **2018-12-11** - added expectation of Rows to be closed, while mocking expected query.
- **2018-12-11** - introduced an option to provide **QueryMatcher** in order to customize SQL query matching.
- **2017-09-01** - it is now possible to expect that prepared statement will be closed,
using **ExpectedPrepare.WillBeClosed**.
- **2017-02-09** - implemented support for **go1.8** features. **Rows** interface was changed to struct
but contains all methods as before and should maintain backwards compatibility. **ExpectedQuery.WillReturnRows** may now
accept multiple row sets.
- **2016-11-02** - `db.Prepare()` was not validating expected prepare SQL
query. It should still be validated even if Exec or Query is not
executed on that prepared statement.
- **2016-02-23** - added **sqlmock.AnyArg()** function to provide any kind
of argument matcher.
- **2016-02-23** - convert expected arguments to driver.Value as natural
driver does, the change may affect time.Time comparison and will be
stricter. See [issue](https://github.com/DATA-DOG/go-sqlmock/issues/31).
- **2015-08-27** - **v1** api change, concurrency support, all known issues fixed.
- **2014-08-16** instead of **panic** during reflect type mismatch when comparing query arguments - now return error
- **2014-08-14** added **sqlmock.NewErrorResult** which gives an option to return driver.Result with errors for
interface methods, see [issue](https://github.com/DATA-DOG/go-sqlmock/issues/5)
- **2014-05-29** allow to match arguments in more sophisticated ways, by providing an **sqlmock.Argument** interface
- **2014-04-21** introduce **sqlmock.New()** to open a mock database connection for tests. This method
calls sql.DB.Ping to ensure that connection is open, see [issue](https://github.com/DATA-DOG/go-sqlmock/issues/4).
This way on Close it will surely assert if all expectations are met, even if database was not triggered at all.
The old way is still available, but it is advisable to call db.Ping manually before asserting with db.Close.
- **2014-02-14** RowsFromCSVString is now a part of Rows interface named as FromCSVString.
It has changed to allow more ways to construct rows and to easily extend this API in future.
See [issue 1](https://github.com/DATA-DOG/go-sqlmock/issues/1)
**RowsFromCSVString** is deprecated and will be removed in future
## Contributions
Feel free to open a pull request. Note, if you wish to contribute an extension to public (exported methods or types) -
please open an issue before, to discuss whether these changes can be accepted. All backward incompatible changes are
and will be treated cautiously
## License
The [three clause BSD license](http://en.wikipedia.org/wiki/BSD_licenses)
+24
View File
@@ -0,0 +1,24 @@
package sqlmock
import "database/sql/driver"
// Argument interface allows to match
// any argument in specific way when used with
// ExpectedQuery and ExpectedExec expectations.
type Argument interface {
Match(driver.Value) bool
}
// AnyArg will return an Argument which can
// match any kind of arguments.
//
// Useful for time.Time or similar kinds of arguments.
func AnyArg() Argument {
return anyArgument{}
}
type anyArgument struct{}
func (a anyArgument) Match(_ driver.Value) bool {
return true
}
+81
View File
@@ -0,0 +1,81 @@
package sqlmock
import (
"database/sql"
"database/sql/driver"
"fmt"
"sync"
)
var pool *mockDriver
func init() {
pool = &mockDriver{
conns: make(map[string]*sqlmock),
}
sql.Register("sqlmock", pool)
}
type mockDriver struct {
sync.Mutex
counter int
conns map[string]*sqlmock
}
func (d *mockDriver) Open(dsn string) (driver.Conn, error) {
d.Lock()
defer d.Unlock()
c, ok := d.conns[dsn]
if !ok {
return c, fmt.Errorf("expected a connection to be available, but it is not")
}
c.opened++
return c, nil
}
// New creates sqlmock database connection and a mock to manage expectations.
// Accepts options, like ValueConverterOption, to use a ValueConverter from
// a specific driver.
// Pings db so that all expectations could be
// asserted.
func New(options ...func(*sqlmock) error) (*sql.DB, Sqlmock, error) {
pool.Lock()
dsn := fmt.Sprintf("sqlmock_db_%d", pool.counter)
pool.counter++
smock := &sqlmock{dsn: dsn, drv: pool, ordered: true}
pool.conns[dsn] = smock
pool.Unlock()
return smock.open(options)
}
// NewWithDSN creates sqlmock database connection with a specific DSN
// and a mock to manage expectations.
// Accepts options, like ValueConverterOption, to use a ValueConverter from
// a specific driver.
// Pings db so that all expectations could be asserted.
//
// This method is introduced because of sql abstraction
// libraries, which do not provide a way to initialize
// with sql.DB instance. For example GORM library.
//
// Note, it will error if attempted to create with an
// already used dsn
//
// It is not recommended to use this method, unless you
// really need it and there is no other way around.
func NewWithDSN(dsn string, options ...func(*sqlmock) error) (*sql.DB, Sqlmock, error) {
pool.Lock()
if _, ok := pool.conns[dsn]; ok {
pool.Unlock()
return nil, nil, fmt.Errorf("cannot create a new mock database with the same dsn: %s", dsn)
}
smock := &sqlmock{dsn: dsn, drv: pool, ordered: true}
pool.conns[dsn] = smock
pool.Unlock()
return smock.open(options)
}
+355
View File
@@ -0,0 +1,355 @@
package sqlmock
import (
"database/sql/driver"
"fmt"
"strings"
"sync"
"time"
)
// an expectation interface
type expectation interface {
fulfilled() bool
Lock()
Unlock()
String() string
}
// common expectation struct
// satisfies the expectation interface
type commonExpectation struct {
sync.Mutex
triggered bool
err error
}
func (e *commonExpectation) fulfilled() bool {
return e.triggered
}
// ExpectedClose is used to manage *sql.DB.Close expectation
// returned by *Sqlmock.ExpectClose.
type ExpectedClose struct {
commonExpectation
}
// WillReturnError allows to set an error for *sql.DB.Close action
func (e *ExpectedClose) WillReturnError(err error) *ExpectedClose {
e.err = err
return e
}
// String returns string representation
func (e *ExpectedClose) String() string {
msg := "ExpectedClose => expecting database Close"
if e.err != nil {
msg += fmt.Sprintf(", which should return error: %s", e.err)
}
return msg
}
// ExpectedBegin is used to manage *sql.DB.Begin expectation
// returned by *Sqlmock.ExpectBegin.
type ExpectedBegin struct {
commonExpectation
delay time.Duration
}
// WillReturnError allows to set an error for *sql.DB.Begin action
func (e *ExpectedBegin) WillReturnError(err error) *ExpectedBegin {
e.err = err
return e
}
// String returns string representation
func (e *ExpectedBegin) String() string {
msg := "ExpectedBegin => expecting database transaction Begin"
if e.err != nil {
msg += fmt.Sprintf(", which should return error: %s", e.err)
}
return msg
}
// WillDelayFor allows to specify duration for which it will delay
// result. May be used together with Context
func (e *ExpectedBegin) WillDelayFor(duration time.Duration) *ExpectedBegin {
e.delay = duration
return e
}
// ExpectedCommit is used to manage *sql.Tx.Commit expectation
// returned by *Sqlmock.ExpectCommit.
type ExpectedCommit struct {
commonExpectation
}
// WillReturnError allows to set an error for *sql.Tx.Close action
func (e *ExpectedCommit) WillReturnError(err error) *ExpectedCommit {
e.err = err
return e
}
// String returns string representation
func (e *ExpectedCommit) String() string {
msg := "ExpectedCommit => expecting transaction Commit"
if e.err != nil {
msg += fmt.Sprintf(", which should return error: %s", e.err)
}
return msg
}
// ExpectedRollback is used to manage *sql.Tx.Rollback expectation
// returned by *Sqlmock.ExpectRollback.
type ExpectedRollback struct {
commonExpectation
}
// WillReturnError allows to set an error for *sql.Tx.Rollback action
func (e *ExpectedRollback) WillReturnError(err error) *ExpectedRollback {
e.err = err
return e
}
// String returns string representation
func (e *ExpectedRollback) String() string {
msg := "ExpectedRollback => expecting transaction Rollback"
if e.err != nil {
msg += fmt.Sprintf(", which should return error: %s", e.err)
}
return msg
}
// ExpectedQuery is used to manage *sql.DB.Query, *dql.DB.QueryRow, *sql.Tx.Query,
// *sql.Tx.QueryRow, *sql.Stmt.Query or *sql.Stmt.QueryRow expectations.
// Returned by *Sqlmock.ExpectQuery.
type ExpectedQuery struct {
queryBasedExpectation
rows driver.Rows
delay time.Duration
rowsMustBeClosed bool
rowsWereClosed bool
}
// WithArgs will match given expected args to actual database query arguments.
// if at least one argument does not match, it will return an error. For specific
// arguments an sqlmock.Argument interface can be used to match an argument.
func (e *ExpectedQuery) WithArgs(args ...driver.Value) *ExpectedQuery {
e.args = args
return e
}
// RowsWillBeClosed expects this query rows to be closed.
func (e *ExpectedQuery) RowsWillBeClosed() *ExpectedQuery {
e.rowsMustBeClosed = true
return e
}
// WillReturnError allows to set an error for expected database query
func (e *ExpectedQuery) WillReturnError(err error) *ExpectedQuery {
e.err = err
return e
}
// WillDelayFor allows to specify duration for which it will delay
// result. May be used together with Context
func (e *ExpectedQuery) WillDelayFor(duration time.Duration) *ExpectedQuery {
e.delay = duration
return e
}
// String returns string representation
func (e *ExpectedQuery) String() string {
msg := "ExpectedQuery => expecting Query, QueryContext or QueryRow which:"
msg += "\n - matches sql: '" + e.expectSQL + "'"
if len(e.args) == 0 {
msg += "\n - is without arguments"
} else {
msg += "\n - is with arguments:\n"
for i, arg := range e.args {
msg += fmt.Sprintf(" %d - %+v\n", i, arg)
}
msg = strings.TrimSpace(msg)
}
if e.rows != nil {
msg += fmt.Sprintf("\n - %s", e.rows)
}
if e.err != nil {
msg += fmt.Sprintf("\n - should return error: %s", e.err)
}
return msg
}
// ExpectedExec is used to manage *sql.DB.Exec, *sql.Tx.Exec or *sql.Stmt.Exec expectations.
// Returned by *Sqlmock.ExpectExec.
type ExpectedExec struct {
queryBasedExpectation
result driver.Result
delay time.Duration
}
// WithArgs will match given expected args to actual database exec operation arguments.
// if at least one argument does not match, it will return an error. For specific
// arguments an sqlmock.Argument interface can be used to match an argument.
func (e *ExpectedExec) WithArgs(args ...driver.Value) *ExpectedExec {
e.args = args
return e
}
// WillReturnError allows to set an error for expected database exec action
func (e *ExpectedExec) WillReturnError(err error) *ExpectedExec {
e.err = err
return e
}
// WillDelayFor allows to specify duration for which it will delay
// result. May be used together with Context
func (e *ExpectedExec) WillDelayFor(duration time.Duration) *ExpectedExec {
e.delay = duration
return e
}
// String returns string representation
func (e *ExpectedExec) String() string {
msg := "ExpectedExec => expecting Exec or ExecContext which:"
msg += "\n - matches sql: '" + e.expectSQL + "'"
if len(e.args) == 0 {
msg += "\n - is without arguments"
} else {
msg += "\n - is with arguments:\n"
var margs []string
for i, arg := range e.args {
margs = append(margs, fmt.Sprintf(" %d - %+v", i, arg))
}
msg += strings.Join(margs, "\n")
}
if e.result != nil {
res, _ := e.result.(*result)
msg += "\n - should return Result having:"
msg += fmt.Sprintf("\n LastInsertId: %d", res.insertID)
msg += fmt.Sprintf("\n RowsAffected: %d", res.rowsAffected)
if res.err != nil {
msg += fmt.Sprintf("\n Error: %s", res.err)
}
}
if e.err != nil {
msg += fmt.Sprintf("\n - should return error: %s", e.err)
}
return msg
}
// WillReturnResult arranges for an expected Exec() to return a particular
// result, there is sqlmock.NewResult(lastInsertID int64, affectedRows int64) method
// to build a corresponding result. Or if actions needs to be tested against errors
// sqlmock.NewErrorResult(err error) to return a given error.
func (e *ExpectedExec) WillReturnResult(result driver.Result) *ExpectedExec {
e.result = result
return e
}
// ExpectedPrepare is used to manage *sql.DB.Prepare or *sql.Tx.Prepare expectations.
// Returned by *Sqlmock.ExpectPrepare.
type ExpectedPrepare struct {
commonExpectation
mock *sqlmock
expectSQL string
statement driver.Stmt
closeErr error
mustBeClosed bool
wasClosed bool
delay time.Duration
}
// WillReturnError allows to set an error for the expected *sql.DB.Prepare or *sql.Tx.Prepare action.
func (e *ExpectedPrepare) WillReturnError(err error) *ExpectedPrepare {
e.err = err
return e
}
// WillReturnCloseError allows to set an error for this prepared statement Close action
func (e *ExpectedPrepare) WillReturnCloseError(err error) *ExpectedPrepare {
e.closeErr = err
return e
}
// WillDelayFor allows to specify duration for which it will delay
// result. May be used together with Context
func (e *ExpectedPrepare) WillDelayFor(duration time.Duration) *ExpectedPrepare {
e.delay = duration
return e
}
// WillBeClosed expects this prepared statement to
// be closed.
func (e *ExpectedPrepare) WillBeClosed() *ExpectedPrepare {
e.mustBeClosed = true
return e
}
// ExpectQuery allows to expect Query() or QueryRow() on this prepared statement.
// This method is convenient in order to prevent duplicating sql query string matching.
func (e *ExpectedPrepare) ExpectQuery() *ExpectedQuery {
eq := &ExpectedQuery{}
eq.expectSQL = e.expectSQL
eq.converter = e.mock.converter
e.mock.expected = append(e.mock.expected, eq)
return eq
}
// ExpectExec allows to expect Exec() on this prepared statement.
// This method is convenient in order to prevent duplicating sql query string matching.
func (e *ExpectedPrepare) ExpectExec() *ExpectedExec {
eq := &ExpectedExec{}
eq.expectSQL = e.expectSQL
eq.converter = e.mock.converter
e.mock.expected = append(e.mock.expected, eq)
return eq
}
// String returns string representation
func (e *ExpectedPrepare) String() string {
msg := "ExpectedPrepare => expecting Prepare statement which:"
msg += "\n - matches sql: '" + e.expectSQL + "'"
if e.err != nil {
msg += fmt.Sprintf("\n - should return error: %s", e.err)
}
if e.closeErr != nil {
msg += fmt.Sprintf("\n - should return error on Close: %s", e.closeErr)
}
return msg
}
// query based expectation
// adds a query matching logic
type queryBasedExpectation struct {
commonExpectation
expectSQL string
converter driver.ValueConverter
args []driver.Value
}
func (e *queryBasedExpectation) attemptArgMatch(args []namedValue) (err error) {
// catch panic
defer func() {
if e := recover(); e != nil {
_, ok := e.(error)
if !ok {
err = fmt.Errorf(e.(string))
}
}
}()
err = e.argsMatches(args)
return
}
+52
View File
@@ -0,0 +1,52 @@
// +build !go1.8
package sqlmock
import (
"database/sql/driver"
"fmt"
"reflect"
)
// WillReturnRows specifies the set of resulting rows that will be returned
// by the triggered query
func (e *ExpectedQuery) WillReturnRows(rows *Rows) *ExpectedQuery {
e.rows = &rowSets{sets: []*Rows{rows}, ex: e}
return e
}
func (e *queryBasedExpectation) argsMatches(args []namedValue) error {
if nil == e.args {
return nil
}
if len(args) != len(e.args) {
return fmt.Errorf("expected %d, but got %d arguments", len(e.args), len(args))
}
for k, v := range args {
// custom argument matcher
matcher, ok := e.args[k].(Argument)
if ok {
// @TODO: does it make sense to pass value instead of named value?
if !matcher.Match(v.Value) {
return fmt.Errorf("matcher %T could not match %d argument %T - %+v", matcher, k, args[k], args[k])
}
continue
}
dval := e.args[k]
// convert to driver converter
darg, err := e.converter.ConvertValue(dval)
if err != nil {
return fmt.Errorf("could not convert %d argument %T - %+v to driver value: %s", k, e.args[k], e.args[k], err)
}
if !driver.IsValue(darg) {
return fmt.Errorf("argument %d: non-subset type %T returned from Value", k, darg)
}
if !reflect.DeepEqual(darg, v.Value) {
return fmt.Errorf("argument %d expected [%T - %+v] does not match actual [%T - %+v]", k, darg, darg, v.Value, v.Value)
}
}
return nil
}
+66
View File
@@ -0,0 +1,66 @@
// +build go1.8
package sqlmock
import (
"database/sql"
"database/sql/driver"
"fmt"
"reflect"
)
// WillReturnRows specifies the set of resulting rows that will be returned
// by the triggered query
func (e *ExpectedQuery) WillReturnRows(rows ...*Rows) *ExpectedQuery {
sets := make([]*Rows, len(rows))
for i, r := range rows {
sets[i] = r
}
e.rows = &rowSets{sets: sets, ex: e}
return e
}
func (e *queryBasedExpectation) argsMatches(args []namedValue) error {
if nil == e.args {
return nil
}
if len(args) != len(e.args) {
return fmt.Errorf("expected %d, but got %d arguments", len(e.args), len(args))
}
// @TODO should we assert either all args are named or ordinal?
for k, v := range args {
// custom argument matcher
matcher, ok := e.args[k].(Argument)
if ok {
if !matcher.Match(v.Value) {
return fmt.Errorf("matcher %T could not match %d argument %T - %+v", matcher, k, args[k], args[k])
}
continue
}
dval := e.args[k]
if named, isNamed := dval.(sql.NamedArg); isNamed {
dval = named.Value
if v.Name != named.Name {
return fmt.Errorf("named argument %d: name: \"%s\" does not match expected: \"%s\"", k, v.Name, named.Name)
}
} else if k+1 != v.Ordinal {
return fmt.Errorf("argument %d: ordinal position: %d does not match expected: %d", k, k+1, v.Ordinal)
}
// convert to driver converter
darg, err := e.converter.ConvertValue(dval)
if err != nil {
return fmt.Errorf("could not convert %d argument %T - %+v to driver value: %s", k, e.args[k], e.args[k], err)
}
if !driver.IsValue(darg) {
return fmt.Errorf("argument %d: non-subset type %T returned from Value", k, darg)
}
if !reflect.DeepEqual(darg, v.Value) {
return fmt.Errorf("argument %d expected [%T - %+v] does not match actual [%T - %+v]", k, darg, darg, v.Value, v.Value)
}
}
return nil
}
+1
View File
@@ -0,0 +1 @@
module github.com/DATA-DOG/go-sqlmock
+22
View File
@@ -0,0 +1,22 @@
package sqlmock
import "database/sql/driver"
// ValueConverterOption allows to create a sqlmock connection
// with a custom ValueConverter to support drivers with special data types.
func ValueConverterOption(converter driver.ValueConverter) func(*sqlmock) error {
return func(s *sqlmock) error {
s.converter = converter
return nil
}
}
// QueryMatcherOption allows to customize SQL query matcher
// and match SQL query strings in more sophisticated ways.
// The default QueryMatcher is QueryMatcherRegexp.
func QueryMatcherOption(queryMatcher QueryMatcher) func(*sqlmock) error {
return func(s *sqlmock) error {
s.queryMatcher = queryMatcher
return nil
}
}
+68
View File
@@ -0,0 +1,68 @@
package sqlmock
import (
"fmt"
"regexp"
"strings"
)
var re = regexp.MustCompile("\\s+")
// strip out new lines and trim spaces
func stripQuery(q string) (s string) {
return strings.TrimSpace(re.ReplaceAllString(q, " "))
}
// QueryMatcher is an SQL query string matcher interface,
// which can be used to customize validation of SQL query strings.
// As an example, external library could be used to build
// and validate SQL ast, columns selected.
//
// sqlmock can be customized to implement a different QueryMatcher
// configured through an option when sqlmock.New or sqlmock.NewWithDSN
// is called, default QueryMatcher is QueryMatcherRegexp.
type QueryMatcher interface {
// Match expected SQL query string without whitespace to
// actual SQL.
Match(expectedSQL, actualSQL string) error
}
// QueryMatcherFunc type is an adapter to allow the use of
// ordinary functions as QueryMatcher. If f is a function
// with the appropriate signature, QueryMatcherFunc(f) is a
// QueryMatcher that calls f.
type QueryMatcherFunc func(expectedSQL, actualSQL string) error
// Match implements the QueryMatcher
func (f QueryMatcherFunc) Match(expectedSQL, actualSQL string) error {
return f(expectedSQL, actualSQL)
}
// QueryMatcherRegexp is the default SQL query matcher
// used by sqlmock. It parses expectedSQL to a regular
// expression and attempts to match actualSQL.
var QueryMatcherRegexp QueryMatcher = QueryMatcherFunc(func(expectedSQL, actualSQL string) error {
expect := stripQuery(expectedSQL)
actual := stripQuery(actualSQL)
re, err := regexp.Compile(expect)
if err != nil {
return err
}
if !re.MatchString(actual) {
return fmt.Errorf(`could not match actual sql: "%s" with expected regexp "%s"`, actual, re.String())
}
return nil
})
// QueryMatcherEqual is the SQL query matcher
// which simply tries a case sensitive match of
// expected and actual SQL strings without whitespace.
var QueryMatcherEqual QueryMatcher = QueryMatcherFunc(func(expectedSQL, actualSQL string) error {
expect := stripQuery(expectedSQL)
actual := stripQuery(actualSQL)
if actual != expect {
return fmt.Errorf(`actual sql: "%s" does not equal to expected "%s"`, actual, expect)
}
return nil
})
+39
View File
@@ -0,0 +1,39 @@
package sqlmock
import (
"database/sql/driver"
)
// Result satisfies sql driver Result, which
// holds last insert id and rows affected
// by Exec queries
type result struct {
insertID int64
rowsAffected int64
err error
}
// NewResult creates a new sql driver Result
// for Exec based query mocks.
func NewResult(lastInsertID int64, rowsAffected int64) driver.Result {
return &result{
insertID: lastInsertID,
rowsAffected: rowsAffected,
}
}
// NewErrorResult creates a new sql driver Result
// which returns an error given for both interface methods
func NewErrorResult(err error) driver.Result {
return &result{
err: err,
}
}
func (r *result) LastInsertId() (int64, error) {
return r.insertID, r.err
}
func (r *result) RowsAffected() (int64, error) {
return r.rowsAffected, r.err
}
+176
View File
@@ -0,0 +1,176 @@
package sqlmock
import (
"database/sql/driver"
"encoding/csv"
"fmt"
"io"
"strings"
)
// CSVColumnParser is a function which converts trimmed csv
// column string to a []byte representation. Currently
// transforms NULL to nil
var CSVColumnParser = func(s string) []byte {
switch {
case strings.ToLower(s) == "null":
return nil
}
return []byte(s)
}
type rowSets struct {
sets []*Rows
pos int
ex *ExpectedQuery
}
func (rs *rowSets) Columns() []string {
return rs.sets[rs.pos].cols
}
func (rs *rowSets) Close() error {
rs.ex.rowsWereClosed = true
return rs.sets[rs.pos].closeErr
}
// advances to next row
func (rs *rowSets) Next(dest []driver.Value) error {
r := rs.sets[rs.pos]
r.pos++
if r.pos > len(r.rows) {
return io.EOF // per interface spec
}
for i, col := range r.rows[r.pos-1] {
dest[i] = col
}
return r.nextErr[r.pos-1]
}
// transforms to debuggable printable string
func (rs *rowSets) String() string {
if rs.empty() {
return "with empty rows"
}
msg := "should return rows:\n"
if len(rs.sets) == 1 {
for n, row := range rs.sets[0].rows {
msg += fmt.Sprintf(" row %d - %+v\n", n, row)
}
return strings.TrimSpace(msg)
}
for i, set := range rs.sets {
msg += fmt.Sprintf(" result set: %d\n", i)
for n, row := range set.rows {
msg += fmt.Sprintf(" row %d - %+v\n", n, row)
}
}
return strings.TrimSpace(msg)
}
func (rs *rowSets) empty() bool {
for _, set := range rs.sets {
if len(set.rows) > 0 {
return false
}
}
return true
}
// Rows is a mocked collection of rows to
// return for Query result
type Rows struct {
converter driver.ValueConverter
cols []string
rows [][]driver.Value
pos int
nextErr map[int]error
closeErr error
}
// NewRows allows Rows to be created from a
// sql driver.Value slice or from the CSV string and
// to be used as sql driver.Rows.
// Use Sqlmock.NewRows instead if using a custom converter
func NewRows(columns []string) *Rows {
return &Rows{
cols: columns,
nextErr: make(map[int]error),
converter: driver.DefaultParameterConverter,
}
}
// CloseError allows to set an error
// which will be returned by rows.Close
// function.
//
// The close error will be triggered only in cases
// when rows.Next() EOF was not yet reached, that is
// a default sql library behavior
func (r *Rows) CloseError(err error) *Rows {
r.closeErr = err
return r
}
// RowError allows to set an error
// which will be returned when a given
// row number is read
func (r *Rows) RowError(row int, err error) *Rows {
r.nextErr[row] = err
return r
}
// AddRow composed from database driver.Value slice
// return the same instance to perform subsequent actions.
// Note that the number of values must match the number
// of columns
func (r *Rows) AddRow(values ...driver.Value) *Rows {
if len(values) != len(r.cols) {
panic("Expected number of values to match number of columns")
}
row := make([]driver.Value, len(r.cols))
for i, v := range values {
// Convert user-friendly values (such as int or driver.Valuer)
// to database/sql native value (driver.Value such as int64)
var err error
v, err = r.converter.ConvertValue(v)
if err != nil {
panic(fmt.Errorf(
"row #%d, column #%d (%q) type %T: %s",
len(r.rows)+1, i, r.cols[i], values[i], err,
))
}
row[i] = v
}
r.rows = append(r.rows, row)
return r
}
// FromCSVString build rows from csv string.
// return the same instance to perform subsequent actions.
// Note that the number of values must match the number
// of columns
func (r *Rows) FromCSVString(s string) *Rows {
res := strings.NewReader(strings.TrimSpace(s))
csvReader := csv.NewReader(res)
for {
res, err := csvReader.Read()
if err != nil || res == nil {
break
}
row := make([]driver.Value, len(r.cols))
for i, v := range res {
row[i] = CSVColumnParser(strings.TrimSpace(v))
}
r.rows = append(r.rows, row)
}
return r
}
+20
View File
@@ -0,0 +1,20 @@
// +build go1.8
package sqlmock
import "io"
// Implement the "RowsNextResultSet" interface
func (rs *rowSets) HasNextResultSet() bool {
return rs.pos+1 < len(rs.sets)
}
// Implement the "RowsNextResultSet" interface
func (rs *rowSets) NextResultSet() error {
if !rs.HasNextResultSet() {
return io.EOF
}
rs.pos++
return nil
}
+589
View File
@@ -0,0 +1,589 @@
/*
Package sqlmock is a mock library implementing sql driver. Which has one and only
purpose - to simulate any sql driver behavior in tests, without needing a real
database connection. It helps to maintain correct **TDD** workflow.
It does not require any modifications to your source code in order to test
and mock database operations. Supports concurrency and multiple database mocking.
The driver allows to mock any sql driver method behavior.
*/
package sqlmock
import (
"database/sql"
"database/sql/driver"
"fmt"
"time"
)
// Sqlmock interface serves to create expectations
// for any kind of database action in order to mock
// and test real database behavior.
type Sqlmock interface {
// ExpectClose queues an expectation for this database
// action to be triggered. the *ExpectedClose allows
// to mock database response
ExpectClose() *ExpectedClose
// ExpectationsWereMet checks whether all queued expectations
// were met in order. If any of them was not met - an error is returned.
ExpectationsWereMet() error
// ExpectPrepare expects Prepare() to be called with expectedSQL query.
// the *ExpectedPrepare allows to mock database response.
// Note that you may expect Query() or Exec() on the *ExpectedPrepare
// statement to prevent repeating expectedSQL
ExpectPrepare(expectedSQL string) *ExpectedPrepare
// ExpectQuery expects Query() or QueryRow() to be called with expectedSQL query.
// the *ExpectedQuery allows to mock database response.
ExpectQuery(expectedSQL string) *ExpectedQuery
// ExpectExec expects Exec() to be called with expectedSQL query.
// the *ExpectedExec allows to mock database response
ExpectExec(expectedSQL string) *ExpectedExec
// ExpectBegin expects *sql.DB.Begin to be called.
// the *ExpectedBegin allows to mock database response
ExpectBegin() *ExpectedBegin
// ExpectCommit expects *sql.Tx.Commit to be called.
// the *ExpectedCommit allows to mock database response
ExpectCommit() *ExpectedCommit
// ExpectRollback expects *sql.Tx.Rollback to be called.
// the *ExpectedRollback allows to mock database response
ExpectRollback() *ExpectedRollback
// MatchExpectationsInOrder gives an option whether to match all
// expectations in the order they were set or not.
//
// By default it is set to - true. But if you use goroutines
// to parallelize your query executation, that option may
// be handy.
//
// This option may be turned on anytime during tests. As soon
// as it is switched to false, expectations will be matched
// in any order. Or otherwise if switched to true, any unmatched
// expectations will be expected in order
MatchExpectationsInOrder(bool)
// NewRows allows Rows to be created from a
// sql driver.Value slice or from the CSV string and
// to be used as sql driver.Rows.
NewRows(columns []string) *Rows
}
type sqlmock struct {
ordered bool
dsn string
opened int
drv *mockDriver
converter driver.ValueConverter
queryMatcher QueryMatcher
expected []expectation
}
func (c *sqlmock) open(options []func(*sqlmock) error) (*sql.DB, Sqlmock, error) {
db, err := sql.Open("sqlmock", c.dsn)
if err != nil {
return db, c, err
}
for _, option := range options {
err := option(c)
if err != nil {
return db, c, err
}
}
if c.converter == nil {
c.converter = driver.DefaultParameterConverter
}
if c.queryMatcher == nil {
c.queryMatcher = QueryMatcherRegexp
}
return db, c, db.Ping()
}
func (c *sqlmock) ExpectClose() *ExpectedClose {
e := &ExpectedClose{}
c.expected = append(c.expected, e)
return e
}
func (c *sqlmock) MatchExpectationsInOrder(b bool) {
c.ordered = b
}
// Close a mock database driver connection. It may or may not
// be called depending on the circumstances, but if it is called
// there must be an *ExpectedClose expectation satisfied.
// meets http://golang.org/pkg/database/sql/driver/#Conn interface
func (c *sqlmock) Close() error {
c.drv.Lock()
defer c.drv.Unlock()
c.opened--
if c.opened == 0 {
delete(c.drv.conns, c.dsn)
}
var expected *ExpectedClose
var fulfilled int
var ok bool
for _, next := range c.expected {
next.Lock()
if next.fulfilled() {
next.Unlock()
fulfilled++
continue
}
if expected, ok = next.(*ExpectedClose); ok {
break
}
next.Unlock()
if c.ordered {
return fmt.Errorf("call to database Close, was not expected, next expectation is: %s", next)
}
}
if expected == nil {
msg := "call to database Close was not expected"
if fulfilled == len(c.expected) {
msg = "all expectations were already fulfilled, " + msg
}
return fmt.Errorf(msg)
}
expected.triggered = true
expected.Unlock()
return expected.err
}
func (c *sqlmock) ExpectationsWereMet() error {
for _, e := range c.expected {
e.Lock()
fulfilled := e.fulfilled()
e.Unlock()
if !fulfilled {
return fmt.Errorf("there is a remaining expectation which was not matched: %s", e)
}
// for expected prepared statement check whether it was closed if expected
if prep, ok := e.(*ExpectedPrepare); ok {
if prep.mustBeClosed && !prep.wasClosed {
return fmt.Errorf("expected prepared statement to be closed, but it was not: %s", prep)
}
}
// must check whether all expected queried rows are closed
if query, ok := e.(*ExpectedQuery); ok {
if query.rowsMustBeClosed && !query.rowsWereClosed {
return fmt.Errorf("expected query rows to be closed, but it was not: %s", query)
}
}
}
return nil
}
// Begin meets http://golang.org/pkg/database/sql/driver/#Conn interface
func (c *sqlmock) Begin() (driver.Tx, error) {
ex, err := c.begin()
if ex != nil {
time.Sleep(ex.delay)
}
if err != nil {
return nil, err
}
return c, nil
}
func (c *sqlmock) begin() (*ExpectedBegin, error) {
var expected *ExpectedBegin
var ok bool
var fulfilled int
for _, next := range c.expected {
next.Lock()
if next.fulfilled() {
next.Unlock()
fulfilled++
continue
}
if expected, ok = next.(*ExpectedBegin); ok {
break
}
next.Unlock()
if c.ordered {
return nil, fmt.Errorf("call to database transaction Begin, was not expected, next expectation is: %s", next)
}
}
if expected == nil {
msg := "call to database transaction Begin was not expected"
if fulfilled == len(c.expected) {
msg = "all expectations were already fulfilled, " + msg
}
return nil, fmt.Errorf(msg)
}
expected.triggered = true
expected.Unlock()
return expected, expected.err
}
func (c *sqlmock) ExpectBegin() *ExpectedBegin {
e := &ExpectedBegin{}
c.expected = append(c.expected, e)
return e
}
// Exec meets http://golang.org/pkg/database/sql/driver/#Execer
func (c *sqlmock) Exec(query string, args []driver.Value) (driver.Result, error) {
namedArgs := make([]namedValue, len(args))
for i, v := range args {
namedArgs[i] = namedValue{
Ordinal: i + 1,
Value: v,
}
}
ex, err := c.exec(query, namedArgs)
if ex != nil {
time.Sleep(ex.delay)
}
if err != nil {
return nil, err
}
return ex.result, nil
}
func (c *sqlmock) exec(query string, args []namedValue) (*ExpectedExec, error) {
var expected *ExpectedExec
var fulfilled int
var ok bool
for _, next := range c.expected {
next.Lock()
if next.fulfilled() {
next.Unlock()
fulfilled++
continue
}
if c.ordered {
if expected, ok = next.(*ExpectedExec); ok {
break
}
next.Unlock()
return nil, fmt.Errorf("call to ExecQuery '%s' with args %+v, was not expected, next expectation is: %s", query, args, next)
}
if exec, ok := next.(*ExpectedExec); ok {
if err := c.queryMatcher.Match(exec.expectSQL, query); err != nil {
next.Unlock()
continue
}
if err := exec.attemptArgMatch(args); err == nil {
expected = exec
break
}
}
next.Unlock()
}
if expected == nil {
msg := "call to ExecQuery '%s' with args %+v was not expected"
if fulfilled == len(c.expected) {
msg = "all expectations were already fulfilled, " + msg
}
return nil, fmt.Errorf(msg, query, args)
}
defer expected.Unlock()
if err := c.queryMatcher.Match(expected.expectSQL, query); err != nil {
return nil, fmt.Errorf("ExecQuery: %v", err)
}
if err := expected.argsMatches(args); err != nil {
return nil, fmt.Errorf("ExecQuery '%s', arguments do not match: %s", query, err)
}
expected.triggered = true
if expected.err != nil {
return expected, expected.err // mocked to return error
}
if expected.result == nil {
return nil, fmt.Errorf("ExecQuery '%s' with args %+v, must return a database/sql/driver.Result, but it was not set for expectation %T as %+v", query, args, expected, expected)
}
return expected, nil
}
func (c *sqlmock) ExpectExec(expectedSQL string) *ExpectedExec {
e := &ExpectedExec{}
e.expectSQL = expectedSQL
e.converter = c.converter
c.expected = append(c.expected, e)
return e
}
// Prepare meets http://golang.org/pkg/database/sql/driver/#Conn interface
func (c *sqlmock) Prepare(query string) (driver.Stmt, error) {
ex, err := c.prepare(query)
if ex != nil {
time.Sleep(ex.delay)
}
if err != nil {
return nil, err
}
return &statement{c, ex, query}, nil
}
func (c *sqlmock) prepare(query string) (*ExpectedPrepare, error) {
var expected *ExpectedPrepare
var fulfilled int
var ok bool
for _, next := range c.expected {
next.Lock()
if next.fulfilled() {
next.Unlock()
fulfilled++
continue
}
if c.ordered {
if expected, ok = next.(*ExpectedPrepare); ok {
break
}
next.Unlock()
return nil, fmt.Errorf("call to Prepare statement with query '%s', was not expected, next expectation is: %s", query, next)
}
if pr, ok := next.(*ExpectedPrepare); ok {
if err := c.queryMatcher.Match(pr.expectSQL, query); err == nil {
expected = pr
break
}
}
next.Unlock()
}
if expected == nil {
msg := "call to Prepare '%s' query was not expected"
if fulfilled == len(c.expected) {
msg = "all expectations were already fulfilled, " + msg
}
return nil, fmt.Errorf(msg, query)
}
defer expected.Unlock()
if err := c.queryMatcher.Match(expected.expectSQL, query); err != nil {
return nil, fmt.Errorf("Prepare: %v", err)
}
expected.triggered = true
return expected, expected.err
}
func (c *sqlmock) ExpectPrepare(expectedSQL string) *ExpectedPrepare {
e := &ExpectedPrepare{expectSQL: expectedSQL, mock: c}
c.expected = append(c.expected, e)
return e
}
type namedValue struct {
Name string
Ordinal int
Value driver.Value
}
// Query meets http://golang.org/pkg/database/sql/driver/#Queryer
func (c *sqlmock) Query(query string, args []driver.Value) (driver.Rows, error) {
namedArgs := make([]namedValue, len(args))
for i, v := range args {
namedArgs[i] = namedValue{
Ordinal: i + 1,
Value: v,
}
}
ex, err := c.query(query, namedArgs)
if ex != nil {
time.Sleep(ex.delay)
}
if err != nil {
return nil, err
}
return ex.rows, nil
}
func (c *sqlmock) query(query string, args []namedValue) (*ExpectedQuery, error) {
var expected *ExpectedQuery
var fulfilled int
var ok bool
for _, next := range c.expected {
next.Lock()
if next.fulfilled() {
next.Unlock()
fulfilled++
continue
}
if c.ordered {
if expected, ok = next.(*ExpectedQuery); ok {
break
}
next.Unlock()
return nil, fmt.Errorf("call to Query '%s' with args %+v, was not expected, next expectation is: %s", query, args, next)
}
if qr, ok := next.(*ExpectedQuery); ok {
if err := c.queryMatcher.Match(qr.expectSQL, query); err != nil {
next.Unlock()
continue
}
if err := qr.attemptArgMatch(args); err == nil {
expected = qr
break
}
}
next.Unlock()
}
if expected == nil {
msg := "call to Query '%s' with args %+v was not expected"
if fulfilled == len(c.expected) {
msg = "all expectations were already fulfilled, " + msg
}
return nil, fmt.Errorf(msg, query, args)
}
defer expected.Unlock()
if err := c.queryMatcher.Match(expected.expectSQL, query); err != nil {
return nil, fmt.Errorf("Query: %v", err)
}
if err := expected.argsMatches(args); err != nil {
return nil, fmt.Errorf("Query '%s', arguments do not match: %s", query, err)
}
expected.triggered = true
if expected.err != nil {
return expected, expected.err // mocked to return error
}
if expected.rows == nil {
return nil, fmt.Errorf("Query '%s' with args %+v, must return a database/sql/driver.Rows, but it was not set for expectation %T as %+v", query, args, expected, expected)
}
return expected, nil
}
func (c *sqlmock) ExpectQuery(expectedSQL string) *ExpectedQuery {
e := &ExpectedQuery{}
e.expectSQL = expectedSQL
e.converter = c.converter
c.expected = append(c.expected, e)
return e
}
func (c *sqlmock) ExpectCommit() *ExpectedCommit {
e := &ExpectedCommit{}
c.expected = append(c.expected, e)
return e
}
func (c *sqlmock) ExpectRollback() *ExpectedRollback {
e := &ExpectedRollback{}
c.expected = append(c.expected, e)
return e
}
// Commit meets http://golang.org/pkg/database/sql/driver/#Tx
func (c *sqlmock) Commit() error {
var expected *ExpectedCommit
var fulfilled int
var ok bool
for _, next := range c.expected {
next.Lock()
if next.fulfilled() {
next.Unlock()
fulfilled++
continue
}
if expected, ok = next.(*ExpectedCommit); ok {
break
}
next.Unlock()
if c.ordered {
return fmt.Errorf("call to Commit transaction, was not expected, next expectation is: %s", next)
}
}
if expected == nil {
msg := "call to Commit transaction was not expected"
if fulfilled == len(c.expected) {
msg = "all expectations were already fulfilled, " + msg
}
return fmt.Errorf(msg)
}
expected.triggered = true
expected.Unlock()
return expected.err
}
// Rollback meets http://golang.org/pkg/database/sql/driver/#Tx
func (c *sqlmock) Rollback() error {
var expected *ExpectedRollback
var fulfilled int
var ok bool
for _, next := range c.expected {
next.Lock()
if next.fulfilled() {
next.Unlock()
fulfilled++
continue
}
if expected, ok = next.(*ExpectedRollback); ok {
break
}
next.Unlock()
if c.ordered {
return fmt.Errorf("call to Rollback transaction, was not expected, next expectation is: %s", next)
}
}
if expected == nil {
msg := "call to Rollback transaction was not expected"
if fulfilled == len(c.expected) {
msg = "all expectations were already fulfilled, " + msg
}
return fmt.Errorf(msg)
}
expected.triggered = true
expected.Unlock()
return expected.err
}
// NewRows allows Rows to be created from a
// sql driver.Value slice or from the CSV string and
// to be used as sql driver.Rows.
func (c *sqlmock) NewRows(columns []string) *Rows {
r := NewRows(columns)
r.converter = c.converter
return r
}
+121
View File
@@ -0,0 +1,121 @@
// +build go1.8
package sqlmock
import (
"context"
"database/sql/driver"
"errors"
"time"
)
// ErrCancelled defines an error value, which can be expected in case of
// such cancellation error.
var ErrCancelled = errors.New("canceling query due to user request")
// Implement the "QueryerContext" interface
func (c *sqlmock) QueryContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Rows, error) {
namedArgs := make([]namedValue, len(args))
for i, nv := range args {
namedArgs[i] = namedValue(nv)
}
ex, err := c.query(query, namedArgs)
if ex != nil {
select {
case <-time.After(ex.delay):
if err != nil {
return nil, err
}
return ex.rows, nil
case <-ctx.Done():
return nil, ErrCancelled
}
}
return nil, err
}
// Implement the "ExecerContext" interface
func (c *sqlmock) ExecContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Result, error) {
namedArgs := make([]namedValue, len(args))
for i, nv := range args {
namedArgs[i] = namedValue(nv)
}
ex, err := c.exec(query, namedArgs)
if ex != nil {
select {
case <-time.After(ex.delay):
if err != nil {
return nil, err
}
return ex.result, nil
case <-ctx.Done():
return nil, ErrCancelled
}
}
return nil, err
}
// Implement the "ConnBeginTx" interface
func (c *sqlmock) BeginTx(ctx context.Context, opts driver.TxOptions) (driver.Tx, error) {
ex, err := c.begin()
if ex != nil {
select {
case <-time.After(ex.delay):
if err != nil {
return nil, err
}
return c, nil
case <-ctx.Done():
return nil, ErrCancelled
}
}
return nil, err
}
// Implement the "ConnPrepareContext" interface
func (c *sqlmock) PrepareContext(ctx context.Context, query string) (driver.Stmt, error) {
ex, err := c.prepare(query)
if ex != nil {
select {
case <-time.After(ex.delay):
if err != nil {
return nil, err
}
return &statement{c, ex, query}, nil
case <-ctx.Done():
return nil, ErrCancelled
}
}
return nil, err
}
// Implement the "Pinger" interface
// for now we do not have a Ping expectation
// may be something for the future
func (c *sqlmock) Ping(ctx context.Context) error {
return nil
}
// Implement the "StmtExecContext" interface
func (stmt *statement) ExecContext(ctx context.Context, args []driver.NamedValue) (driver.Result, error) {
return stmt.conn.ExecContext(ctx, stmt.query, args)
}
// Implement the "StmtQueryContext" interface
func (stmt *statement) QueryContext(ctx context.Context, args []driver.NamedValue) (driver.Rows, error) {
return stmt.conn.QueryContext(ctx, stmt.query, args)
}
// @TODO maybe add ExpectedBegin.WithOptions(driver.TxOptions)
// CheckNamedValue meets https://golang.org/pkg/database/sql/driver/#NamedValueChecker
func (c *sqlmock) CheckNamedValue(nv *driver.NamedValue) (err error) {
nv.Value, err = c.converter.ConvertValue(nv.Value)
return err
}
+28
View File
@@ -0,0 +1,28 @@
package sqlmock
import (
"database/sql/driver"
)
type statement struct {
conn *sqlmock
ex *ExpectedPrepare
query string
}
func (stmt *statement) Close() error {
stmt.ex.wasClosed = true
return stmt.ex.closeErr
}
func (stmt *statement) NumInput() int {
return -1
}
func (stmt *statement) Exec(args []driver.Value) (driver.Result, error) {
return stmt.conn.Exec(stmt.query, args)
}
func (stmt *statement) Query(args []driver.Value) (driver.Rows, error) {
return stmt.conn.Query(stmt.query, args)
}
@@ -20,6 +20,3 @@ _cgo_export.*
_testmain.go
*.exe
*.test
*.swp
+27
View File
@@ -0,0 +1,27 @@
Copyright (c) 2013 CloudFlare, Inc.
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this
list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
* Neither the name of the CloudFlare, Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+14
View File
@@ -0,0 +1,14 @@
GCFLAGS :=
LDFLAGS :=
.PHONY: install
install:
@go install -v .
.PHONY: test
test:
@go test -gcflags='$(GCFLAGS)' -ldflags='$(LDFLAGS)' .
.PHONY: bench
bench:
@go test -gcflags='$(GCFLAGS)' -ldflags='$(LDFLAGS)' -bench .
+4
View File
@@ -0,0 +1,4 @@
golz4
=====
Golang interface to LZ4 compression
+4
View File
@@ -0,0 +1,4 @@
// Package lz4 implements compression using lz4.c and lz4hc.c
//
// Copyright (c) 2013 CloudFlare, Inc.
package lz4
+55
View File
@@ -0,0 +1,55 @@
package lz4
// #cgo CFLAGS: -O3
// #include "src/lz4.h"
// #include "src/lz4.c"
import "C"
import (
"errors"
"fmt"
"unsafe"
)
// p gets a char pointer to the first byte of a []byte slice
func p(in []byte) *C.char {
if len(in) == 0 {
return (*C.char)(unsafe.Pointer(nil))
}
return (*C.char)(unsafe.Pointer(&in[0]))
}
// clen gets the length of a []byte slice as a char *
func clen(s []byte) C.int {
return C.int(len(s))
}
// Uncompress with a known output size. len(out) should be equal to
// the length of the uncompressed out.
func Uncompress(in, out []byte) (error) {
if int(C.LZ4_decompress_safe(p(in), p(out), clen(in), clen(out))) < 0 {
return errors.New("Malformed compression stream")
}
return nil
}
// CompressBound calculates the size of the output buffer needed by
// Compress. This is based on the following macro:
//
// #define LZ4_COMPRESSBOUND(isize)
// ((unsigned int)(isize) > (unsigned int)LZ4_MAX_INPUT_SIZE ? 0 : (isize) + ((isize)/255) + 16)
func CompressBound(in []byte) int {
return len(in) + ((len(in) / 255) + 16)
}
// Compress compresses in and puts the content in out. len(out)
// should have enough space for the compressed data (use CompressBound
// to calculate). Returns the number of bytes in the out slice.
func Compress(in, out []byte) (outSize int, err error) {
outSize = int(C.LZ4_compress_limitedOutput(p(in), p(out), clen(in), clen(out)))
if outSize == 0 {
err = fmt.Errorf("insufficient space for compression")
}
return
}
+38
View File
@@ -0,0 +1,38 @@
package lz4
// #cgo CFLAGS: -O3
// #include "src/lz4hc.h"
// #include "src/lz4hc.c"
import "C"
import (
"fmt"
)
// CompressHC compresses in and puts the content in out. len(out)
// should have enough space for the compressed data (use CompressBound
// to calculate). Returns the number of bytes in the out slice. Determines
// the compression level automatically.
func CompressHC(in, out []byte) (int, error) {
// 0 automatically sets the compression level.
return CompressHCLevel(in, out, 0)
}
// CompressHCLevel compresses in at the given compression level and puts the
// content in out. len(out) should have enough space for the compressed data
// (use CompressBound to calculate). Returns the number of bytes in the out
// slice. To automatically choose the compression level, use 0. Otherwise, use
// any value in the inclusive range 1 (worst) through 16 (best). Most
// applications will prefer CompressHC.
func CompressHCLevel(in, out []byte, level int) (outSize int, err error) {
// LZ4HC does not handle empty buffers. Pass through to Compress.
if len(in) == 0 || len(out) == 0 {
return Compress(in, out)
}
outSize = int(C.LZ4_compressHC2_limitedOutput(p(in), p(out), clen(in), clen(out), C.int(level)))
if outSize == 0 {
err = fmt.Errorf("insufficient space for compression")
}
return
}
+143
View File
@@ -0,0 +1,143 @@
CANTO I
IN the midway of this our mortal life,
I found me in a gloomy wood, astray
Gone from the path direct: and e'en to tell
It were no easy task, how savage wild
That forest, how robust and rough its growth,
Which to remember only, my dismay
Renews, in bitterness not far from death.
Yet to discourse of what there good befell,
All else will I relate discover'd there.
How first I enter'd it I scarce can say,
Such sleepy dullness in that instant weigh'd
My senses down, when the true path I left,
But when a mountain's foot I reach'd, where clos'd
The valley, that had pierc'd my heart with dread,
I look'd aloft, and saw his shoulders broad
Already vested with that planet's beam,
Who leads all wanderers safe through every way.
Then was a little respite to the fear,
That in my heart's recesses deep had lain,
All of that night, so pitifully pass'd:
And as a man, with difficult short breath,
Forespent with toiling, 'scap'd from sea to shore,
Turns to the perilous wide waste, and stands
At gaze; e'en so my spirit, that yet fail'd
Struggling with terror, turn'd to view the straits,
That none hath pass'd and liv'd. My weary frame
After short pause recomforted, again
I journey'd on over that lonely steep,
The hinder foot still firmer. Scarce the ascent
Began, when, lo! a panther, nimble, light,
And cover'd with a speckled skin, appear'd,
Nor, when it saw me, vanish'd, rather strove
To check my onward going; that ofttimes
With purpose to retrace my steps I turn'd.
The hour was morning's prime, and on his way
Aloft the sun ascended with those stars,
That with him rose, when Love divine first mov'd
Those its fair works: so that with joyous hope
All things conspir'd to fill me, the gay skin
Of that swift animal, the matin dawn
And the sweet season. Soon that joy was chas'd,
And by new dread succeeded, when in view
A lion came, 'gainst me, as it appear'd,
With his head held aloft and hunger-mad,
That e'en the air was fear-struck. A she-wolf
Was at his heels, who in her leanness seem'd
Full of all wants, and many a land hath made
Disconsolate ere now. She with such fear
O'erwhelmed me, at the sight of her appall'd,
That of the height all hope I lost. As one,
Who with his gain elated, sees the time
When all unwares is gone, he inwardly
Mourns with heart-griping anguish; such was I,
Haunted by that fell beast, never at peace,
Who coming o'er against me, by degrees
Impell'd me where the sun in silence rests.
While to the lower space with backward step
I fell, my ken discern'd the form one of one,
Whose voice seem'd faint through long disuse of speech.
When him in that great desert I espied,
"Have mercy on me!" cried I out aloud,
"Spirit! or living man! what e'er thou be!"
He answer'd: "Now not man, man once I was,
And born of Lombard parents, Mantuana both
By country, when the power of Julius yet
Was scarcely firm. At Rome my life was past
Beneath the mild Augustus, in the time
Of fabled deities and false. A bard
Was I, and made Anchises' upright son
The subject of my song, who came from Troy,
When the flames prey'd on Ilium's haughty towers.
But thou, say wherefore to such perils past
Return'st thou? wherefore not this pleasant mount
Ascendest, cause and source of all delight?"
"And art thou then that Virgil, that well-spring,
From which such copious floods of eloquence
Have issued?" I with front abash'd replied.
"Glory and light of all the tuneful train!
May it avail me that I long with zeal
Have sought thy volume, and with love immense
Have conn'd it o'er. My master thou and guide!
Thou he from whom alone I have deriv'd
That style, which for its beauty into fame
Exalts me. See the beast, from whom I fled.
O save me from her, thou illustrious sage!"
"For every vein and pulse throughout my frame
She hath made tremble." He, soon as he saw
That I was weeping, answer'd, "Thou must needs
Another way pursue, if thou wouldst 'scape
From out that savage wilderness. This beast,
At whom thou criest, her way will suffer none
To pass, and no less hindrance makes than death:
So bad and so accursed in her kind,
That never sated is her ravenous will,
Still after food more craving than before.
To many an animal in wedlock vile
She fastens, and shall yet to many more,
Until that greyhound come, who shall destroy
Her with sharp pain. He will not life support
By earth nor its base metals, but by love,
Wisdom, and virtue, and his land shall be
The land 'twixt either Feltro. In his might
Shall safety to Italia's plains arise,
For whose fair realm, Camilla, virgin pure,
Nisus, Euryalus, and Turnus fell.
He with incessant chase through every town
Shall worry, until he to hell at length
Restore her, thence by envy first let loose.
I for thy profit pond'ring now devise,
That thou mayst follow me, and I thy guide
Will lead thee hence through an eternal space,
Where thou shalt hear despairing shrieks, and see
Spirits of old tormented, who invoke
A second death; and those next view, who dwell
Content in fire, for that they hope to come,
Whene'er the time may be, among the blest,
Into whose regions if thou then desire
T' ascend, a spirit worthier then I
Must lead thee, in whose charge, when I depart,
Thou shalt be left: for that Almighty King,
Who reigns above, a rebel to his law,
Adjudges me, and therefore hath decreed,
That to his city none through me should come.
He in all parts hath sway; there rules, there holds
His citadel and throne. O happy those,
Whom there he chooses!" I to him in few:
"Bard! by that God, whom thou didst not adore,
I do beseech thee (that this ill and worse
I may escape) to lead me, where thou saidst,
That I Saint Peter's gate may view, and those
Who as thou tell'st, are in such dismal plight."
Onward he mov'd, I close his steps pursu'd.
+2
View File
@@ -0,0 +1,2 @@
/bin
/gopath
+16
View File
@@ -0,0 +1,16 @@
language: go
go:
- "1.9"
- "1.10"
install:
- go get -v -t github.com/coreos/go-oidc/...
- go get golang.org/x/tools/cmd/cover
- go get github.com/golang/lint/golint
script:
- ./test
notifications:
email: false
+71
View File
@@ -0,0 +1,71 @@
# How to Contribute
CoreOS projects are [Apache 2.0 licensed](LICENSE) and accept contributions via
GitHub pull requests. This document outlines some of the conventions on
development workflow, commit message formatting, contact points and other
resources to make it easier to get your contribution accepted.
# Certificate of Origin
By contributing to this project you agree to the Developer Certificate of
Origin (DCO). This document was created by the Linux Kernel community and is a
simple statement that you, as a contributor, have the legal right to make the
contribution. See the [DCO](DCO) file for details.
# Email and Chat
The project currently uses the general CoreOS email list and IRC channel:
- Email: [coreos-dev](https://groups.google.com/forum/#!forum/coreos-dev)
- IRC: #[coreos](irc://irc.freenode.org:6667/#coreos) IRC channel on freenode.org
Please avoid emailing maintainers found in the MAINTAINERS file directly. They
are very busy and read the mailing lists.
## Getting Started
- Fork the repository on GitHub
- Read the [README](README.md) for build and test instructions
- Play with the project, submit bugs, submit patches!
## Contribution Flow
This is a rough outline of what a contributor's workflow looks like:
- Create a topic branch from where you want to base your work (usually master).
- Make commits of logical units.
- Make sure your commit messages are in the proper format (see below).
- Push your changes to a topic branch in your fork of the repository.
- Make sure the tests pass, and add any new tests as appropriate.
- Submit a pull request to the original repository.
Thanks for your contributions!
### Format of the Commit Message
We follow a rough convention for commit messages that is designed to answer two
questions: what changed and why. The subject line should feature the what and
the body of the commit should describe the why.
```
scripts: add the test-cluster command
this uses tmux to setup a test cluster that you can easily kill and
start for debugging.
Fixes #38
```
The format can be described more formally as follows:
```
<subsystem>: <what changed>
<BLANK LINE>
<why this change was made>
<BLANK LINE>
<footer>
```
The first line is the subject and should be no longer than 70 characters, the
second line is always blank, and other lines should be wrapped at 80 characters.
This allows the message to be easier to read on GitHub as well as in various
git tools.
+36
View File
@@ -0,0 +1,36 @@
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
660 York Street, Suite 102,
San Francisco, CA 94110 USA
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
+3
View File
@@ -0,0 +1,3 @@
Eric Chiang <ericchiang@google.com> (@ericchiang)
Mike Danese <mikedanese@google.com> (@mikedanese)
Rithu Leena John <rjohn@redhat.com> (@rithujohn191)
+72
View File
@@ -0,0 +1,72 @@
# go-oidc
[![GoDoc](https://godoc.org/github.com/coreos/go-oidc?status.svg)](https://godoc.org/github.com/coreos/go-oidc)
[![Build Status](https://travis-ci.org/coreos/go-oidc.png?branch=master)](https://travis-ci.org/coreos/go-oidc)
## OpenID Connect support for Go
This package enables OpenID Connect support for the [golang.org/x/oauth2](https://godoc.org/golang.org/x/oauth2) package.
```go
provider, err := oidc.NewProvider(ctx, "https://accounts.google.com")
if err != nil {
// handle error
}
// Configure an OpenID Connect aware OAuth2 client.
oauth2Config := oauth2.Config{
ClientID: clientID,
ClientSecret: clientSecret,
RedirectURL: redirectURL,
// Discovery returns the OAuth2 endpoints.
Endpoint: provider.Endpoint(),
// "openid" is a required scope for OpenID Connect flows.
Scopes: []string{oidc.ScopeOpenID, "profile", "email"},
}
```
OAuth2 redirects are unchanged.
```go
func handleRedirect(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, oauth2Config.AuthCodeURL(state), http.StatusFound)
}
```
The on responses, the provider can be used to verify ID Tokens.
```go
var verifier = provider.Verifier(&oidc.Config{ClientID: clientID})
func handleOAuth2Callback(w http.ResponseWriter, r *http.Request) {
// Verify state and errors.
oauth2Token, err := oauth2Config.Exchange(ctx, r.URL.Query().Get("code"))
if err != nil {
// handle error
}
// Extract the ID Token from OAuth2 token.
rawIDToken, ok := oauth2Token.Extra("id_token").(string)
if !ok {
// handle missing token
}
// Parse and verify ID Token payload.
idToken, err := verifier.Verify(ctx, rawIDToken)
if err != nil {
// handle error
}
// Extract custom claims
var claims struct {
Email string `json:"email"`
Verified bool `json:"email_verified"`
}
if err := idToken.Claims(&claims); err != nil {
// handle error
}
}
```
+61
View File
@@ -0,0 +1,61 @@
## CoreOS Community Code of Conduct
### Contributor Code of Conduct
As contributors and maintainers of this project, and in the interest of
fostering an open and welcoming community, we pledge to respect all people who
contribute through reporting issues, posting feature requests, updating
documentation, submitting pull requests or patches, and other activities.
We are committed to making participation in this project a harassment-free
experience for everyone, regardless of level of experience, gender, gender
identity and expression, sexual orientation, disability, personal appearance,
body size, race, ethnicity, age, religion, or nationality.
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery
* Personal attacks
* Trolling or insulting/derogatory comments
* Public or private harassment
* Publishing others' private information, such as physical or electronic addresses, without explicit permission
* Other unethical or unprofessional conduct.
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct. By adopting this Code of Conduct,
project maintainers commit themselves to fairly and consistently applying these
principles to every aspect of managing this project. Project maintainers who do
not follow or enforce the Code of Conduct may be permanently removed from the
project team.
This code of conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community.
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting a project maintainer, Brandon Philips
<brandon.philips@coreos.com>, and/or Rithu John <rithu.john@coreos.com>.
This Code of Conduct is adapted from the Contributor Covenant
(http://contributor-covenant.org), version 1.2.0, available at
http://contributor-covenant.org/version/1/2/0/
### CoreOS Events Code of Conduct
CoreOS events are working conferences intended for professional networking and
collaboration in the CoreOS community. Attendees are expected to behave
according to professional standards and in accordance with their employers
policies on appropriate workplace behavior.
While at CoreOS events or related social networking opportunities, attendees
should not engage in discriminatory or offensive speech or actions including
but not limited to gender, sexuality, race, age, disability, or religion.
Speakers should be especially aware of these concerns.
CoreOS does not condone any statements by speakers contrary to these standards.
CoreOS reserves the right to deny entrance and/or eject from an event (without
refund) any individual found to be engaging in discriminatory or offensive
speech or actions.
Please bring any concerns to the immediate attention of designated on-site
staff, Brandon Philips <brandon.philips@coreos.com>, and/or Rithu John <rithu.john@coreos.com>.
-7
View File
@@ -1,7 +0,0 @@
package http
import "net/http"
type Client interface {
Do(*http.Request) (*http.Response, error)
}

Some files were not shown because too many files have changed in this diff Show More