Skip to content

fix(fetch): preserve page titles in markdown output #4253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kgarg2468 wants to merge 1 commit into
base: main
Choose a base branch
from
+46 −1
Open

fix(fetch): preserve page titles in markdown output #4253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion src/fetch/src/mcp_server_fetch/server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,15 @@ def extract_content_from_html(html: str) -> str:
content = markdownify.markdownify(
ret["content"],
heading_style=markdownify.ATX,
)
).lstrip()
title = ret.get("title")
if title:
title_markdown = f"# {title.strip()}"
# Readability often omits the document title when the article body does
# not repeat it. Include it once so callers can identify fetched pages,
# while avoiding duplicate headings when the title is already present.
if title_markdown.casefold() not in content[: len(title_markdown) + 32].casefold():
content = f"{title_markdown}\n\n{content}"
return content


Expand Down
37 changes: 37 additions & 0 deletions src/fetch/tests/test_server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,43 @@ def test_simple_html(self):
# readabilipy may extract different parts depending on the content
assert "test paragraph" in result

def test_html_includes_document_title_when_omitted_from_article(self):
"""Test that simplified markdown preserves the page title."""
html = """
<html>
<head><title>What’s new in 2.1.0 (Aug 30, 2023)</title></head>
<body>
<article>
<p>These are the release notes.</p>
</article>
</body>
</html>
"""

result = extract_content_from_html(html)

assert result.startswith("# What’s new in 2.1.0 (Aug 30, 2023)")
assert "These are the release notes." in result

def test_html_does_not_duplicate_existing_title_heading(self):
"""Test that a page title already present as the first heading is not duplicated."""
html = """
<html>
<head><title>Existing Title</title></head>
<body>
<article>
<h1>Existing Title</h1>
<p>Article body.</p>
</article>
</body>
</html>
"""

result = extract_content_from_html(html)

assert result.count("# Existing Title") == 1
assert "Article body." in result

def test_html_with_links(self):
"""Test that links are converted to markdown."""
html = """
Expand Down
Loading