feat(cli): support openkb add <URL> with content-type sniffing

[KylinMountain]

Summary

openkb add now accepts http(s) URLs alongside local paths. The fetcher saves the remote document into raw/, then hands the path to the existing add_single_file pipeline — so PageIndex / markitdown / hash dedup / openkb remove all work unchanged.

openkb add https://arxiv.org/abs/2509.11420       # → trafilatura, 3 KB abstract+meta
openkb add https://arxiv.org/pdf/2509.11420       # → urllib, 2.1 MB → PageIndex long-doc
openkb add https://claude.com/blog/the-founders-playbook
openkb add https://cdn.example.com/foo.pdf

Architecture

URL fetching is a peer of converter.py and indexer.py, not bolted onto cli.py:

openkb/
├── cli.py            ← entry / routing (4-line intercept in `add`)
├── converter.py      ← local-file markitdown conversion
├── indexer.py        ← PageIndex long-doc indexing
├── images.py         ← image extraction
├── url_ingest.py     ← (NEW) http(s) URL → raw/ acquisition
└── ...

Design decisions

1. Content-Type drives routing, not URL extension. arxiv.org/pdf/2509.11420 has no .pdf in the URL; cdn.com/foo.pdf has one. Both are PDFs and route the same way — by Content-Type: application/pdf.

2. Magic-byte sniff overrides Content-Type when they disagree. Some CDNs mislabel PDFs as application/octet-stream; some servers serve an HTML interstitial with application/pdf. Reading the first 512 bytes and checking for %PDF- or < wins over the header.

3. HTML goes through trafilatura, not raw save. Live-tested claude.com/blog/...: markitdown on raw HTML yields 25 KB where half is nav/footer/cookie chrome. trafilatura extracts 1.5 KB of clean main content — much better signal for the LLM compiler.

4. No site-specific code. No if arxiv: rewrite abs→pdf. The user picks depth: abs/ URL → abstract + metadata, pdf/ URL → full paper. The arxiv server's own Content-Disposition gives the version-suffixed filename for free.

5. Filename sanitization preserves identifier dots. 2509.11420 is one identifier, not stem.ext — only strip an extension when it matches the target extension we're adding.

6. Chunked PDF write. 50+ MB papers don't sit in RAM; streamed at 64 KB per chunk.

7. Graceful failure with stderr messages, never crash. 404 / DNS / extraction-empty / unsupported type all return None and exit 0 after a clear message.

Files

• openkb/url_ingest.py (NEW, ~190 LOC)
• openkb/cli.py (+4-line intercept at top of add)
• pyproject.toml (+ trafilatura>=2.0)
• tests/test_url_ingest.py (NEW, 31 tests)
• README.md (one-line Quick Start + Commands table update)

Test plan

• 31 new unit tests covering: URL detection, content-type sniffing (magic-byte override 4 cases), filename sanitization (arxiv ID preservation, spaces, parens, length cap, empty fallback), Content-Disposition parsing (RFC 5987 / quoted / unquoted), fetch_url_to_raw integration (PDF chunked write, HTML→trafilatura, short-extraction warning, HTTP 404, DNS error, unsupported type)
• 360 total tests pass (329 prior + 31 new, zero regression)
• Live network smoke-tested against four URL shapes:
  • arxiv.org/abs/<id> → Trading-R1-Financial-Trading-...md (3.3 KB)
  • arxiv.org/pdf/<id> → 2509.11420v1.pdf (2.1 MB, magic %PDF-1.7, version suffix from server CD header)
  • claude.com/blog/... → The-founder-s-playbook-...md (1.5 KB clean)
  • CDN PDF with %20 and (1) → 69fe2a55..._The-Founders-Playbook-..._v3-1.pdf (465 KB)
• CI green

[KylinMountain]

Code review

Found 1 issue.

1. Filename clash silently overwrites prior file in raw/. _extract_html (HTML path) and _download_pdf_chunked (PDF path) both write to raw/<sanitized> without checking whether the target already exists. Two URLs whose sanitization yields the same filename — easy to hit for HTML (any two pages titled "Introduction", "About", "Home" → both → Introduction.md; any two PDFs with same basename across mirrors → same .pdf) — silently overwrite each other. The hash registry still has the first URL's hash mapped to that filename, but the bytes at raw/<filename> are now the second URL's content. State becomes inconsistent: openkb list shows entries that no longer match disk, and openkb remove cleans the wrong logical entry.

   HTML write:

   OpenKB/openkb/url_ingest.py

   Lines 180 to 184 in 829f44f

   	title = (metadata.title if metadata else None) or url
   	filename = _sanitize_filename(title, ".md")
   	target = raw_dir / filename
   	target.write_text(markdown, encoding="utf-8")
   	click.echo(

   PDF write:

   OpenKB/openkb/url_ingest.py

   Lines 232 to 236 in 829f44f

   	url, response.headers.get("Content-Disposition"),
   	)
   	target = raw_dir / filename
   	_download_pdf_chunked(response, head_bytes, target)
   	size_mb = target.stat().st_size / (1024 * 1024)

   Fix: when target.exists(), append _2, _3, … to the stem until you find a free name. ~6-line helper that wraps both call sites.

Out of scope but worth tracking as follow-ups (each scored real but below the strict 80 threshold):

• Redirect-aware filename: _pdf_filename(url, ...) uses the user's original URL rather than response.geturl(). When a URL like https://doi.org/10.xxx/yyy redirects to a publisher CDN and the publisher does NOT send Content-Disposition, the filename is derived from doi.org's path instead of the real basename. arxiv masks this with its CD header so the live smoke test didn't catch it. One-line fix: final_url = response.geturl() or url.

• Orphan raw/ file on dedup-skip: when a URL re-ingest produces a hash already in the registry, add_single_file correctly reports [SKIP] but the just-downloaded file remains in raw/ untracked. For large PDFs this is silent disk waste. Fix: have add_single_file (or the URL caller) unlink the source file when convert_document returns skipped=True AND the path was URL-fetched.

Generated with Claude Code.

[KylinMountain]

Code review (round 2, on fix commit 47e5ec2)

Found 2 issues:

1. Orphan-unlink in URL branch is too broad — fires on pipeline failure, not just dedup-skip. add_single_file returns False from 5 sites: result.skipped (dedup), conversion failure, indexing failure, and both compilation-failure branches. The URL branch unconditionally unlinks the fetched raw file for any False return. If index_long_document succeeds but compile_long_doc fails (transient LLM timeout / rate-limit), the raw PDF is deleted while a dangling PageIndex entry remains — and because the hash is only registered on full success, openkb remove can't clean it up. Retrying re-downloads + may double-index. Suggest distinguishing skip-vs-error (tri-state return, or only unlink when result.skipped was the reason).

OpenKB/openkb/cli.py

Lines 428 to 435 in 47e5ec2

	return
	added = add_single_file(fetched, kb_dir)
	if not added:
	# Duplicate (or failed mid-pipeline) — drop the just-fetched
	# file so raw/ doesn't accumulate orphans across retries.
	fetched.unlink(missing_ok=True)
	return

2. _extract_html success echo prints the pre-collision filename, not the actual saved path. After _unique_path renames on collision (e.g. Introduction_2.md), the echo still uses filename instead of target.name, so the user sees Saved: raw/Introduction.md while the file is actually at raw/Introduction_2.md. The PDF branch (line 269) already uses target.name correctly — same fix needed here.

OpenKB/openkb/url_ingest.py

Lines 207 to 213 in 47e5ec2

	target = _unique_path(raw_dir / filename)
	target.write_text(markdown, encoding="utf-8")
	click.echo(
	f" Extracted: {title!r}\n"
	f" Saved: raw/{filename} ({len(markdown) // 1024 or 1} KB clean markdown)"
	)
	return target

End-to-end flow (fetch_url_to_raw → add_single_file → convert_document → PageIndex/markitdown → wiki) is otherwise structurally sound. URL ingest works for the happy path; the issues above bite on edge cases (LLM API hiccup, second URL with same title).

Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}