fix: log failure reason before COMPLETE and fix misleading SCRAPE ✓ (#1949)

[hafezparast]

Problem

When a crawl fails (anti-bot detection, empty HTML, etc.), users see only [COMPLETE] ✗ with zero explanation — even with verbose=True. The CrawlResult.error_message is correctly set but was never logged to the console.

A second issue: the [SCRAPE] log always emitted ✓ regardless of whether scraping produced any content, which was actively misleading.

Before:

[FETCH]...  ↓ https://example.com  | ✗ | ⏱: 0.52s
[SCRAPE]..  ◆ https://example.com  | ✓ | ⏱: 0.00s   ← misleading ✓
[COMPLETE] ● https://example.com   | ✗ | ⏱: 0.52s   ← no reason shown

After:

[FETCH]...  ↓ https://example.com  | ✗ | ⏱: 0.64s
[SCRAPE]..  ◆ https://example.com  | ✗ | ⏱: 0.00s   ← correct ✗
[ERROR]...  × https://example.com  | Error: Blocked by anti-bot protection: Near-empty content (0 bytes) with HTTP 200
[COMPLETE] ● https://example.com   | ✗ | ⏱: 0.64s   ← user now knows why

Fix

Two-line change in async_webcrawler.py:

1. Before the COMPLETE log: if crawl_result.success=False and error_message is set, emit an [ERROR] log with the reason
2. In aprocess_html: change SCRAPE log from hardcoded success=True to success=bool(cleaned_html)

Test plan

• Reproduced bug: SCRAPE ✓, COMPLETE ✗, no error log — confirmed
• After fix: ERROR log appears before COMPLETE with full reason
• 7 targeted tests — all pass
• Full regression suite: 297 passed, 1 skipped, 0 failures

Fixes #1949

🤖 Generated with Claude Code