Implement bitcoind-compatible JSON-RPC and REST blockchain methods by dsbaars · Pull Request #795 · libbitcoin/libbitcoin-server

dsbaars · 2026-06-08T17:24:34Z

Summary

The bitcoind compatibility protocol (src/protocols/bitcoind/) was largely stubbed. This PR implements the read-only blockchain surface plus raw-transaction retrieval and broadcast, reusing the existing query layer and the libbitcoin-system bitcoind* JSON serializers. Chain-context fields that the serializers intentionally omit (height, confirmations, next/prev hash, mediantime, blocktime) are injected at the protocol layer, mirroring protocol_native.

Motivation: enabling Bitcoin Core-compatible clients to read chain state and broadcast transactions against a libbitcoin node.

JSON-RPC methods implemented (return live data)

getbestblockhash, getblock (verbosity 0/1/2), getblockchaininfo, getblockcount, getblockhash, getblockheader, getblockfilter, gettxout, getnetworkinfo, getrawtransaction, sendrawtransaction.

getrawtransaction serves any archived (confirmed) transaction by txid. libbitcoin archives all confirmed transactions hash-addressably, i.e. an implicit txindex. Returns raw hex (verbosity 0) or Core-format JSON (txid/hash/size/vsize/weight/vin/vout/hex plus injected in_active_chain/blockhash/confirmations/blocktime/time).
sendrawtransaction deserializes the tx, runs context-free check(), archives it via query.set_code() so the existing protocol_transaction_out_106 can serve it on getdata, broadcasts it onto the shared peer-message bus to announce to peers, and returns the txid. No mempool subsystem is required for this path.

REST endpoints implemented

/rest/chaininfo.json, /rest/block/<hash>, /rest/block/notxdetails/<hash>, /rest/block/spent/<hash>, /rest/blockhashbyheight/<height>, /rest/headers/<count>/<hash>, /rest/blockfilter/basic/<hash>, /rest/blockfilterheaders/basic/<hash>, /rest/blockpart/<hash>/<offset>/<size>. Per-endpoint media type (binary / hex / json) is derived from the URL extension. bitcoind_target parses the Core REST URL scheme into a json-rpc request model that the REST dispatcher routes.

Deliberately left as clean `not_implemented`

getblockstats, getchaintxstats, getchainwork (need aggregate/cumulative indexes), gettxoutsetinfo, verifytxoutset, scantxoutset (need a UTXO-set scan/snapshot), verifychain (full revalidation), pruneblockchain (pruning), savemempool (needs a mempool). These dispatch and return a structured not_implemented error rather than failing.

Notes / design

Shared helpers (median_time_past, inject_block_context, inject_tx_context, header_to_bitcoind, chain_name) are hoisted into include/bitcoin/server/protocols/bitcoind_json.hpp and used by both the RPC and REST units.
getnetworkinfo string fields (subversion, warnings) were being selected as value_t(bool) from bare string literals; fixed.
Signatures were cross-checked against Bitcoin Core's RPCMethod definitions (bitcoin/bitcoin src/rpc). getblockstats hash_or_height now accepts a height or a hash (Core's skip_type_check behavior, via value_t), and getrawtransaction's verbosity param uses Core's canonical name verbosity.

Testing

Built and runtime-tested against an isolated regtest node: all implemented RPC methods and REST endpoints return correct genesis-block data; getrawtransaction (raw + verbose), sendrawtransaction (submit / invalid / unknown paths), and getblock v1/v2 context injection verified.
No new build-system entries required (the bitcoind translation units already existed; the one new file is header-only).

Open questions for maintainers

sendrawtransaction currently archives the tx before broadcasting and performs only context-free check(). Without chaser_transaction there is no contextual (UTXO/policy) validation or eviction of never-mined transactions. This is acceptable for broadcast, but I would welcome guidance on whether to gate it behind a flag or add minimal validation. A TODO marks the spot.
The interface declares getchainwork and verifytxoutset, which are not standard Bitcoin Core RPCs (Core exposes chainwork as a field, and has loadtxoutset/dumptxoutset).
Not yet covered (no current subsystem): getrawmempool/getmempoolinfo/testmempoolaccept and the REST getutxos/mempool/fork endpoints. These require a queryable mempool, which is currently stubbed in chaser_transaction / protocol_transaction_in.

getblock (verbosity 1/2), getblockcount, getblockhash, getblockheader, getblockchaininfo, getblockfilter, gettxout and getnetworkinfo, reusing the libbitcoin-system bitcoind json serializers with a protocol-level context injector (height, confirmations, mediantime, prev/next hash). Aggregate-heavy methods (getchainwork, getchaintxstats, getblockstats, gettxoutsetinfo, scantxoutset, verifychain, verifytxoutset, pruneblockchain, savemempool) remain explicit not_implemented.

Activate the REST path: bitcoind_target parses /rest/block/<hash>.<bin|hex|json> into a json-rpc model, handle_receive_get dispatches via rest_dispatcher_, and handle_get_block serves all three media types through new raw-http senders (send_data/send_hex/send_dom), which the bitcoind base lacked.

Extend the REST interface beyond block: block_hash (blockhashbyheight), block_txs (notxdetails), block_headers, block_part, block_spent_tx_outputs, block_filter, block_filter_headers and chain_information, with their Core REST url patterns parsed in bitcoind_target. Remaining endpoints (get_utxos[_confirmed], mempool[_information], fork_information) need mempool enumeration / deployment status / utxo semantics not yet exposed, so are left unimplemented.

Move median_time_past, inject_block_context, header_to_bitcoind and chain_name out of the duplicated anonymous namespaces in the rpc and rest protocol units into bitcoind_json.hpp, included by both.

A bare string literal selects value_t(boolean_t) over value_t(const string_t&) in the rpc::object_t initializer (const char* -> bool beats the user-defined string conversion), so subversion and warnings serialized as 'true'. Wrap them in std::string. Caught runtime-testing against a regtest node.

getrawtransaction serves any archived (confirmed) tx by txid: raw hex (verbose 0) or verbose JSON via the existing bitcoind_verbose serializer plus a new inject_tx_context helper (in_active_chain/blockhash/ confirmations/blocktime/time). libbitcoin archives all confirmed tx hash-addressable, so this is a built-in txindex. sendrawtransaction deserializes the hex tx, runs context-free check(), archives it via query.set_code() (so the existing protocol_transaction_ out_106 can serve it on getdata), broadcasts it onto the shared peer message bus to announce to peers, and returns the txid. No mempool subsystem required. TODO: contextual (connect) validation before archiving for policy/DoS hardening.

Two issues caught by runtime-testing against a regtest node: - The verbose/maxfeerate params were declared optional<0_u32> but the handlers take double; the rpc dispatcher threw bad_variant_access ("unexpected type") on any numeric arg. Declare as optional<0.0> to match, consistent with getblock's verbosity. - getrawtransaction verbose used bitcoind_verbose(tx), which on a standalone transaction falls back to libbitcoin's plain inputs/outputs form (no txid). Use bitcoind(tx) for Core's txid/hash/size/vsize/ weight/vin/vout/hex fields (same encoding getblock verbosity 2 embeds).

Cross-checked all declared bitcoind signatures against Core's RPCMethod definitions (bitcoin/bitcoin src/rpc). Two fixes: - getblockstats: hash_or_height was declared string_t, but Core accepts a height number OR a block hash (RPCArg::Type::NUM with skip_type_check). A numeric height threw 'unexpected type' at dispatch. Declare as value_t (the dispatcher passes it through untyped), matching Core; verified both a numeric height and a hash now reach the handler. - getrawtransaction: param named 'verbose'; Core's canonical name is 'verbosity' (with 'verbose' as an alias), and getblock already uses 'verbosity'. Rename for named-parameter compatibility; positional dispatch (used by LND/btcwallet) is unaffected.

evoskuil · 2026-06-08T22:29:33Z

Thanks! Just giving this a quick visual scan, and it looks good. A full set of passing functional tests (see electrum and native/block) would help get this merged much more quickly. We should also have acceptance tests (@eynhaender @echennells ?) to verify interface compliance.

Return target (expanded from bits), verificationprogress (confirmed/candidate height), initialblockdownload, and warnings, alongside the existing fields. chainwork and size_on_disk remain omitted, as they require a cumulative-work index and store-size accounting respectively.

Covers getrawtransaction (raw, verbose, coinbase, segwit, unknown txid) and sendrawtransaction (invalid and malformed input; the broadcast path is gated behind BITCOIND_ALLOW_BROADCAST to avoid relaying on mainnet), plus the REST endpoints (block json/hex/bin, notxdetails, spent, blockhashbyheight, headers, blockpart, and the basic filters). Verified against a synced mainnet node. getblockchaininfo's chainwork and size_on_disk assertions are relaxed to match the implementation.

Build an in-process HTTP harness on the bitcoind test fixture (beast POST for json-rpc, GET for REST, replacing the raw-socket placeholder) and add deterministic acceptance tests against the ten-block mock store: getblockcount/getbestblockhash/getblockhash/getblockheader/getblock/getblockchaininfo/gettxout/getrawtransaction/sendrawtransaction/getnetworkinfo, the not_implemented set, and the REST endpoints (chaininfo, block json/hex/bin, notxdetails, spent, blockhashbyheight, headers, blockpart, blockfilter). A witness-store fixture covers segwit getrawtransaction (wtxid != txid, vsize == ceil(weight/4)). These run in CI without a synced node.

dsbaars · 2026-06-09T11:53:50Z

Thanks for the suggestion. I have added both.

Functional tests (endpoints/, against a running node, in the style of the native and electrum suites): test_bitcoind_rpc.py now covers getrawtransaction and sendrawtransaction, and test_bitcoind_rest.py (new) covers the REST surface (chaininfo, block json/hex/bin, notxdetails, spent, blockhashbyheight, headers, blockpart, basic filters). Verified against a node synced past segwit activation: 28 passed, 1 skipped (a broadcast test gated behind an env flag), the rest xfail for the not_implemented methods and for filters when bip158 is off.

Acceptance tests (test/protocols/bitcoind/, in-process, run in CI without a synced node, in the style of native/electrum): the fixture only had a placeholder request helper, so I built an HTTP harness on it (beast POST for json-rpc, GET for REST) and filled in bitcoind_rpc.cpp and bitcoind_rest.cpp. 30 cases assert exact responses for every wired RPC method and REST endpoint against the ten-block mock store, plus a witness-store case for segwit getrawtransaction (reusing the existing block1a/block2a witness mocks).

I also extended getblockchaininfo to return target, verificationprogress, initialblockdownload, and warnings (chainwork and size_on_disk still need a cumulative-work index and store-size accounting). One compatibility note: the system bitcoind(tx) serializer reports size as the stripped size, whereas Core reports the total size for segwit transactions.

evoskuil · 2026-06-09T17:51:41Z

Thanks! Just a clarification on test terminology. We use these terms (sort of informally):

Unit Test - smallest unit possible/practical (e.g. lowest level method/function).
Component Test - aggregate of units (e.g. class or aggregating function/method).
Functional Test - system function (e.g. endpoint communication over a socket, public API).
Acceptance Test - performed over the compiled executable (e.g. customer acceptance).

We target full Unit Test coverage and Component and/or Functional as necessary to ensure expected aggregate behavior. No external dependencies (e.g. Python or other tools), just Boost Test in our pattern. Can be data-driven (which is when we allow loops in a test).

Acceptance is outside of the build (against the executable or compiled lib). Currently we have Python client-server tests in libbitcoin-server but I'm not sure what the status of that is. @eynhaender ?

Boost.Test, in-build, no external dependencies, per the libbitcoin test taxonomy (unit = lowest-level function, component = aggregate over a class). Unit (test/parsers/bitcoind_target.cpp, replacing the stub): cover the pure bitcoind_target() REST path parser across every route and error path (media mapping, missing/invalid target/hash/number, leading-zero, non-basic filter); error and media cases are data-driven. Unit + component (test/protocols/bitcoind/bitcoind_json.cpp, new): pure header_to_bitcoind field mapping, plus chain_name, median_time_past (BIP113), inject_block_context (genesis/middle/tip) and inject_tx_context (confirmed/unknown) against the ten-block mock store via a minimal store+query fixture (no server/socket). Register the new file in Makefile.am and the vs2022/vs2026 vcxproj/filters.

evoskuil

Very nice, minor comments. Suggest @eynhaender look at the python acceptance tests.

Relocate the bitcoind json context helpers out of the inline header (bitcoind_json.hpp) into protocol_bitcoind_rpc as protected static methods, implemented in protocol_bitcoind_rpc_json.cpp and inherited by the rest protocol. Removes them from the broad server namespace and keeps the implementation out of headers. chain_name now derives each network's genesis from system::settings (per the chain::selection enumeration) rather than hardcoded hashes; signet remains a documented stub pending its selection. Source idiom: pass hash_cptr by const& in the rest handlers (native/electrum convention), use to_shared(std::move()) and std::from_chars() in the target parser. Tests: bitcoind_target gains real coverage, bitcoind_json exercises the helpers via a test seam; rest_text() strips line terminators, display hashes are computed once at file scope, header bytes are hashed directly, and the verbose comments are dropped. Register protocol_bitcoind_rpc_json.cpp in Makefile.am and the vs2022/vs2026 project files.

dsbaars · 2026-06-09T22:25:18Z

Thanks for the review, all addressed in 0ef2d05d.

Json helpers moved out of the inline header into protocol_bitcoind_rpc as protected static methods (in protocol_bitcoind_rpc_json.cpp), inherited by rest.
chain_name derives genesis from system::settings per chain::selection; signet stays a documented stub.
Rest handlers take hash_cptr by const&; target parser uses to_shared and std::from_chars.
Tests: rest_text() strips terminators, hashes computed once at file scope, header hashed directly, no wrapping or non data-driven loops.

One deviation: I kept a leading-zero guard before from_chars, since bare from_chars accepts "01" which these fields should reject as non-canonical (matching Core).

Full Boost suite passes. @evoskuil would you mind taking another look when you have a moment?

evoskuil

Sorry to dribble in the reviews, I just had more time just now.

Query stored median_time_past instead of recomputing; use add1/to_half/ floored_subtract/possible_sign_cast and typed literals; drop size_t casts into boost::json; static chain_name lookup; starts_with/trim_right; bool shared_ptr checks. Sweep the same patterns across the bitcoind handlers (fixes a height-as-link mediantime call) and align the rpc tests.

dsbaars · 2026-06-10T11:34:38Z

Thanks, all addressed in 7fe07607.

median_time_past now does a single get_context query of the stored field instead of recomputing over 11 headers.
Arithmetic uses add1 / floored_subtract / possible_sign_cast and typed literals; size_t casts into boost::json dropped (kept on the rpc variant, where the xcode issue requires it).
chain_name uses a static lookup so the genesis is computed once; inject_tx_context relies on get_tx_height alone; nextblockhash uses add1.
starts_with / trim_right, bool checks for shared_ptr, and the constexpr hash_digest test expectation.

While sweeping for the same patterns I found a latent bug: the REST chaininfo handler passed a height where median_time_past now expects a header_link (it compiled via implicit conversion). Fixed, plus the same idiom cleanups across the other handlers and the rpc tests.

Three intentional deviations: I kept a leading-zero guard before from_chars (it accepts "01"); used std::string_view::starts_with since the token is a view; and wrote confirmations as add1(floored_subtract(top, height)) (top - height + 1), since floored_subtract(top, add1(height)) is lower by two.

Full Boost suite passes.

evoskuil · 2026-06-10T20:40:39Z

+    }
+}
+
+boost::json::object protocol_bitcoind_rpc::header_to_bitcoind(


This is just pure object serialization from the type, so should be implemented in system as a serializer.

This is the corresponding impl for the block (header wasn't implemented yet).

https://github.com/libbitcoin/libbitcoin-system/blob/master/include/bitcoin/system/chain/json/block.hpp

https://github.com/libbitcoin/libbitcoin-system/blob/master/src/chain/json/block.cpp

https://github.com/libbitcoin/libbitcoin-system/blob/master/test/chain/json/block.cpp

…c-rest

Query stored median_time_past; confirmations as add1(floored_subtract(top, height)) with no cast or ternary; chain_name as a static lookup comparing hash_digest (signet via constexpr base16_hash; testnet3). REST block_headers streams get_confirmed_headers + get_wire_header into one pre-sized buffer (no per-item copy or masked failure); data/text/json senders; blockpart uses is_lesser, ceilinged_add and std::next; fluent get_height. possible_wide_cast for the rpc value variant; initialblockdownload via is_current.

dsbaars · 2026-06-11T09:35:23Z

Thanks, all addressed in 5cb252d1.

Json/confirmations: median_time_past reads the stored context field; confirmations are add1(floored_subtract(top, height)) with no cast or ternary (per your note); chain_name is a static lookup comparing hash_digest (signet via constexpr base16_hash; testnet3).
REST block_headers: rewritten to get_confirmed_headers + get_wire_header streamed into one pre-sized buffer (no per-item copy, no masked failures).
Idioms: possible_wide_cast/possible_sign_cast over raw casts, is_lesser/ceilinged_add for the blockpart bounds, std::next instead of pointer math, fluent get_height, bool checks for shared_ptr, initialblockdownload via is_current.
Naming: send_text/send_json/to_data/to_text (data/base16, no bin/hex).

Two intentional notes: dropping the confirmations ternary removes the -1 for a non-active-chain block (these handlers serve confirmed blocks); and rest::send_text hides the inherited rpc send_text (different layer, benign).

One deferred item: header_to_bitcoind is pure type-to-json, so it really belongs in libbitcoin-system as a chain/json/header serializer (next to chain/json/block). That is a separate libbitcoin-system PR; until it exists the helper stays in the protocol layer and the headers json branch still loops (it can use value_from once the serializer lands).

evoskuil · 2026-06-11T14:05:58Z

Weekly team meeting on our slack channel #general today at noon NYC time. Open invite on system wiki side menu.

dsbaars · 2026-06-11T14:22:16Z

Would love to join, but about to board a plane to BTC Prague 😅

evoskuil · 2026-06-11T15:24:05Z

+    // Resolve every prevout spent by the block's non-coinbase transactions.
+    chain::output_cptrs spent{};
+    const auto& txs = *block->transactions_ptr();
+    for (size_t tx = 1; tx < txs.size(); ++tx)


For data and text this loop can operate directly over the stream, avoiding vector allocation. For the json this is prob preferrable.

evoskuil · 2026-06-11T15:24:36Z

+    const auto& txs = *block->transactions_ptr();
+    for (size_t tx = 1; tx < txs.size(); ++tx)
+        for (const auto& in: *txs.at(tx)->inputs_ptr())
+            if (const auto out = query.get_output(query.to_output(in->point())))


This condition is suppressing a store failure.

evoskuil · 2026-06-11T15:27:15Z

+    http::read(socket_, buffer, response, ec);
+    BOOST_CHECK_MESSAGE(!ec, ec.message());
+    BOOST_CHECK_EQUAL(response.result(), http::status::ok);
+    return system::data_chunk(response.body().begin(), response.body().end());


system::to_chunk(string)

eynhaender

Ran python acceptance tests against both bitcoind JSON-RPC and REST endpoints and all look good. Implementation also aligns with the existing structure.

dsbaars added 8 commits June 8, 2026 16:28

Hoist shared bitcoind json helpers to a header.

51b7337

Move median_time_past, inject_block_context, header_to_bitcoind and chain_name out of the duplicated anonymous namespaces in the rpc and rest protocol units into bitcoind_json.hpp, included by both.

evoskuil self-requested a review June 8, 2026 22:32

dsbaars added 3 commits June 9, 2026 13:19

evoskuil requested a review from eynhaender June 9, 2026 17:53

evoskuil assigned eynhaender Jun 9, 2026

evoskuil added the enhancement label Jun 9, 2026

evoskuil requested changes Jun 9, 2026

View reviewed changes

evoskuil requested changes Jun 10, 2026

View reviewed changes