Enforce robots.txt Crawl-delay in URLFrontier queues

[dpol1]

Closes #1979. Completes Stage 3: the queue-side politeness work discussed in #867, after explicit Retry-After handling in #1973 and adaptive back-off for headerless rate limits in #1984.

When fetcher.max.crawl.delay.force is enabled, the fetcher caps an excessive robots.txt Crawl-delay locally and continues fetching. Until now, that meant losing the original interval and potentially crawling the host faster than requested. Both fetcher bolts now preserve the requested delay in robots.crawl.delay, and QueueRegulatorBolt forwards it to URLFrontier through setDelay.

Design

• The signal is emitted only when the fetcher actually caps a robots.txt Crawl-delay. Fractional delays are rounded up to whole seconds, so frontier pacing is never more aggressive than robots.txt.
• The signal is rebuilt from the locally parsed robots rules after protocol metadata is merged. Stale metadata or a colliding HTTP response header therefore cannot inject or alter the queue delay.
• Robots pacing is opt-in through urlfrontier.robots.crawl.delay.enabled. CrawlDelayPolicy caps values with urlfrontier.backoff.max.secs, keeps one RPC in flight per host, and coalesces concurrent updates so the latest value eventually wins.
• Successful values are deduplicated for urlfrontier.backoff.decay.secs. A failed RPC retries only a value received while the previous call was in flight; otherwise a later queue signal reasserts it.
• setDelay(key, 0) is deliberately never sent. If a site removes its Crawl-delay, its queue may remain slower than necessary, but the crawler never becomes less polite or erases a frontier-side default.

Configuration contract

The feature fails fast unless the queue and metadata configuration can enforce a per-host delay safely:

partition.url.mode: byHost
fetcher.max.crawl.delay.force: true
urlfrontier.robots.crawl.delay.enabled: true
urlfrontier.max.urls.per.bucket: 1
urlfrontier.backoff.max.secs: 86400

metadata.persist:
  - robots.crawl.delay

robots.crawl.delay must not appear in metadata.transfer, including through wildcards such as robots.*: an outlink must not inherit the delay of its parent host. Custom metadata.transfer.class implementations remain responsible for preserving the same contract.

URLFrontier applies the delay between queue hand-outs; it does not shrink an existing batch or recall URLs already emitted. Requiring one URL per bucket bounds each new hand-out, while already prefetched or concurrent requests may still complete during the transition.

Tests

Coverage includes both fetcher implementations, fractional delay rounding, stale and colliding metadata, opt-in and startup validation, asynchronous last-value-wins serialization, failed and stale callbacks, deduplication, caps, and a Testcontainers test against a real URLFrontier proving that the second URL remains unavailable during the configured delay.

The core and URLFrontier module suites pass with 334 and 55 tests respectively. Configuration, module README, defaults and extension documentation have been updated.

### For all changes

• Is there a issue associated with this PR? Is it referenced in the commit message?
• Does your PR title start with #XXXX where XXXX is the issue number you are trying to resolve?
• Has your PR been rebased against the latest commit within the target branch (typically main)?
• Is your initial contribution a single, squashed commit? (kept as eight atomic commits so the producer, policy, integration, safety validation and documentation changes can be reviewed separately)
• Is the code properly formatted with mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false?

For code changes

• Have you ensured that the full suite of tests is executed via mvn clean verify? (core and URLFrontier module suites pass; full reactor pending)
• Have you written or updated unit tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0? (no new
  dependencies)
• If applicable, have you updated the LICENSE file, including the main LICENSE file? (not applicable)
• If applicable, have you updated the NOTICE file, including the main NOTICE file? (not applicable)

[dpol1]

While reviewing the distributed path, I found that URLFrontier 2.5 does not appear to propagate setDelay or blockQueueUntil when local=false. This is now tracked upstream in crawler-commons/url-frontier#146.

I am moving this PR back to draft while I:

• make host-level robots pacing conservative with max-observed-wins;
• introduce a robots-specific maximum delay;
• clarify or enforce the supported deployment model until the upstream behaviour is resolved;
• update tests and documentation to reflect the revised semantics.

The existing single-node implementation and tests remain valid. Suggestions are welcome here ;-)