feat: Add replicator DNS override support for outbound requests. by willholley · Pull Request #5983 · apache/couchdb

willholley · 2026-04-27T15:07:53Z

Overview

This adds a feature to the CouchDB replicator to override the DNS target for specific host
patterns (including wildcards) when making outbound requests. The use case is when requests need to be routed via a transparent SNI proxy e.g. for network egress monitoring. Common approaches to enable host overrides e.g. modifying /etc/hosts are not sufficient because they do not support wildcard routing - this feature avoids having to run a local DNS server such as CoreDNS to direct traffic through the proxy.

There is a new configuration option to specify the overrides:

[replicator]
dns_overrides = host:target, host2:target

The replicator resolves the configured host patterns to the alternative connection targets while
preserving the request URL host (applies to regular requests and session-auth requests).

Note this depends on the connect_to option in ibrowse, which is a custom feature in the CouchDB ibrowse fork.

Testing recommendations

Testing this is with TLS a bit involved as it relies on setting up an SNI proxy. I did it using nginx in docker with the configuration attached to proxy to a cloudant.com database. The proxy was running on a non-standard port (e.g. 8443) so that any replications connecting directly to cloudant.com would fail.

I then set dns_overrides = *.cloudant.com:127.0.0.1 in default.ini and configured a replication from myaccount.cloudant.com:8443/mydb. The test succeeds if the proxy logged the connection and the replication completed.

The feature also works without TLS - you can just use it to direct an arbitrary hostname to your local couchdb, for instance. e.g. if you use dns_overrides = *.cloudant.com:127.0.0.1 and couchdb is running on 127.0.0.1:15984, set up a replication with source or target as http://foo.cloudant.com:15984/db1 and it will be redirected to 127.0.0.1:15984/db1.

nginx.conf.zip

Related Issues or Pull Requests

feat: add connect_to option couchdb-ibrowse#9

Checklist

This is my own work, I did not use AI, LLM's or similar technology
Code is written and works correctly
Changes are covered by tests
Any new configurable parameters are documented in rel/overlay/etc/default.ini
Documentation changes were made in the src/docs folder
Documentation changes were backported (separated PR) to affected branches

nickva

This is very nice! I didn't get to play with it locally just did a quick look-over with some comments first

nickva · 2026-04-29T06:32:50Z

    Body = get_value(body, Params, []),
+
+    % Apply DNS override using connect_to ibrowse option
+    #url{host = Host, protocol = Protocol} = ibrowse_lib:parse_url(Url),


We'd be parsing the url on every request, I wonder if it would work to cache in the httpdb record the host already parsed? Or maybe we could cache the override ssl / connect_to settings in #httpd{}...

I did wonder about the performance overhead but it seems negligible in the grand scheme of things? If you think it beneficial, I can explore it (may need guidance though!).

I looked into this and it seems like the complexity wasn't really worthwhile. The persistent term config cache gives a decent speedup.

nickva · 2026-04-29T06:37:33Z

+    end.
+
+-spec get_overrides() -> [dns_override()].
+get_overrides() ->


It would be nice to have to reparse everything on each resolve. We could use a persistent_term perhaps, but it would add a more code here...

I pushed a commit which adds a cache using a persistent_term.

nickva · 2026-04-29T06:46:07Z

+    resolve_host/1,
+    parse_config/1,
+    match_pattern/2,
+    get_overrides/0


This is exported for testing mostly? I don't think it's used otherwise, if so we could skip exporting

nickva · 2026-05-01T22:03:27Z

+    Opts =
+        case {Proto, OriginalHost} of
+            {https, OrigHost} when is_list(OrigHost) ->
+                case inet:is_ip_address(OrigHost) of


is_ip_address/1 takes an already parsed tuple only

We could use inet:parse_address/1

> inet:parse_address(string:trim("[::0]", both, "[]")). {ok,{0,0,0,0,0,0,0,0}} > inet:parse_address("[::0]"). {error,einval} > inet:parse_address("::0"). {ok,{0,0,0,0,0,0,0,0}} > inet:parse_address("127"). {ok,{0,0,0,127}} > inet:parse_address("1.2.3.4"). {ok,{1,2,3,4}} >inet:parse_address("a.c.d.com"). {error,einval}

Also ibrowse_lib:parse_url{} also return a host type

> ibrowse_lib:parse_url("http://127.0.0.1"). #url{abspath = "http://127.0.0.1",host = "127.0.0.1", port = 80,username = undefined,password = undefined, path = "/",protocol = http,host_type = ipv4_address} > ibrowse_lib:parse_url("http://[::0]"). #url{abspath = "http://[::0]",host = "::0",port = 80, username = undefined,password = undefined,path = "/", protocol = http,host_type = ipv6_address} > ibrowse_lib:parse_url("http://foo.bar.baz.com"). #url{abspath = "http://foo.bar.baz.com", host = "foo.bar.baz.com",port = 80,username = undefined, password = undefined,path = "/",protocol = http, host_type = hostname} >

nickva · 2026-05-01T22:11:32Z

+    case binary:split(Entry, <<":">>) of
+        [Pattern0, Target0] ->
+            Pattern = string:trim(Pattern0),
+            Target = string:trim(Target0),


If the IPv6 is passed in with brackets we should see if ibrowse knows how to connect to a bracketed address. It may have to be stripped of brackets and/or also parsed into an ipv6 address tuple

nickva · 2026-05-01T22:15:36Z

+                {<<"[", _/binary>>, _} ->
+                    invalid_entry_reason(Entry, "IPv6 addresses cannot be used as patterns");
+                _ ->
+                    {true, {Pattern, Target}}


Would *example:127.0.0.1 work or *:127.0.0.1 work?

nickva · 2026-05-01T22:19:19Z

    Url = full_url(HttpDb, Params),
    Body = get_value(body, Params, []),
+
+    % Apply DNS override using connect_to ibrowse option


Some of the "apply dns override" seems to be similar with that we do in auth_session? Wonder if a helper function here work and call that from auth_session, or some common utility library. If they diverge enough, it may not work though.

This adds a feature to the CouchDB replicator to override the DNS target for specific host patterns (including wildcards) when making outbound requests. The use case is when requests need to be routed via a transparent SNI proxy e.g. for network egress monitoring and specifying overrides in /etc/hosts or similar isn't suffient / possible (e.g. due to lack of wildcard support). There is adds a new configuration option to specify the overrides: ``` [replicator] dns_overrides = host:target, host2:target ``` The replicator resolves the configured host patterns to the alternative connection targets while preserving the request URL host (applies to regular requests and session-auth requests).

- Use inet:is_ip_address to detect whether the original target is an IP address. If it is, do not add the SNI header, since this is only valid for hostnames. - Remove unicode support. - Add support for IPv6 targets. This is a little awkward because IPv6 addresses use the same `:` delimiter as our config, so reqiure them to be bracketed. - Clarify documentation / default.ini examples around wildcard support.

Use inet:parse_address/1 to detect valid IP addresses.

Extracts a helper function `couch_replicator_dns:apply_dns_override/2` which applies the `connect_to` ibrowse option and optional SNI header to replication requests in both `couch_replicator_auth_session` and `couch_replicator_httpc`.

Cached the parsed dns_overrides configuration. I did consider adding per-connection caching as well, but this seemed to be of limited benefit in microbenchmarks. Adding the persistent_term cache yields approx a 3x performance improvement in resolution time (amounts to 150ms over 10k resolutions on my machine).

jiahuili430 reviewed Apr 28, 2026

View reviewed changes

Comment thread src/couch_replicator/src/couch_replicator_dns.erl Outdated

jiahuili430 reviewed Apr 28, 2026

View reviewed changes

Comment thread src/couch_replicator/src/couch_replicator_dns.erl Outdated

willholley force-pushed the wh/connect_to branch 3 times, most recently from 6d51f89 to 13796f6 Compare April 28, 2026 20:37

jiahuili430 reviewed Apr 28, 2026

View reviewed changes

Comment thread src/couch_replicator/test/eunit/couch_replicator_dns_tests.erl Outdated

nickva reviewed Apr 29, 2026

View reviewed changes

willholley force-pushed the wh/connect_to branch from da53e78 to 503825c Compare April 29, 2026 16:25

nickva reviewed May 1, 2026

View reviewed changes

willholley added 5 commits May 4, 2026 20:00

add test from PR review

1326666

fix is_ip_address implementation

b742a42

Use inet:parse_address/1 to detect valid IP addresses.

extract common helper for dns_overrides

45fcc9e

Extracts a helper function `couch_replicator_dns:apply_dns_override/2` which applies the `connect_to` ibrowse option and optional SNI header to replication requests in both `couch_replicator_auth_session` and `couch_replicator_httpc`.

willholley force-pushed the wh/connect_to branch from 503825c to 45fcc9e Compare May 5, 2026 06:23

willholley added 2 commits May 5, 2026 13:59

remove unused exports

5947353

willholley marked this pull request as ready for review May 5, 2026 16:27

Conversation

willholley commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Testing recommendations

Related Issues or Pull Requests

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nickva left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

willholley commented Apr 27, 2026 •

edited

Loading