Open Source · Python · CLI · MCP

HTTP-first web content
for LLM pipelines.

Fetch a page, choose what to keep, and inspect the result before passing it to a model. Start over HTTP; add Chromium rendering when the content depends on JavaScript.

Compare real outputs Install and run

HTTP-first no browser required

5 detail levels text, Markdown, or HTML

SSRF guard private addresses blocked by default

MCP ready tool surface included

Features

Fetch, extract, and inspect.

HTTP-First Fetching

HTTPX fetches the page; extraction and Markdown conversion run locally. Chromium rendering is an optional extra for content that needs JavaScript.

HTTPX / optional Chromium

Detail Levels

Five modes let you choose plain text, readable or structured Markdown, full-page Markdown, or raw HTML. Compare the output with the source before choosing a mode.

minimal / readable / structured / full / raw

Batch Fetching

Pass a list of URLs and get back a list of results. Concurrency controlled, errors surfaced per-item, not globally.

concurrent · per-item errors

Bounded Crawl

Follow links from a seed URL with explicit depth, page, and concurrency limits. Crawls stay on the same domain by default.

depth · page-limit · same-domain

Validator Caching

Enable a page cache to reuse results and send ETag or Last-Modified validators when rechecking a page. Cache entries are separated by detail mode.

ETag · Last-Modified · 304

SSRF Guard

Checks URLs and resolved addresses, rejecting private, loopback, and link-local destinations by default. Deployment network controls still matter.

SSRF · pre-connect · IP check

Detail Levels

One source. Five ways to extract it.

Version 0.4.2 accepts detail in Python and --detail in the CLI. These are mode guides; the example gallery shows saved output beside the source pages and commands that produced it.

pulldown get https://pulldown.making-minds.ai/examples/field-notes.html --detail minimal

Title and plain text for a first pass or summary. Check that removing links and structure still leaves enough context for your task.

pulldown get https://pulldown.making-minds.ai/examples/field-notes.html --detail readable --json-output

The default mode chooses an extraction strategy from the page structure. It favors article text on narrative pages and can route listings or dashboards to structured extraction. Inspect meta.routing in the JSON result.

pulldown get https://pulldown.making-minds.ai/examples/field-notes.html --detail structured

Preserves hierarchy and summarizes tables for listings, dashboards, and documentation. Compare the retained headings, links, and rows with the source before using it for a detailed lookup.

pulldown get https://pulldown.making-minds.ai/examples/field-notes.html --detail full

Converts the full page to Markdown, including navigation and footer content. Useful when the relevant text is outside an article body; expect more page chrome.

pulldown get https://pulldown.making-minds.ai/examples/field-notes.html --detail raw

Returns HTML without Markdown extraction. With --render, the returned HTML is the browser's serialized DOM. Use it to inspect the source or apply your own parser.

Mode	Output	Start here for
minimal	Title and plain text	Summaries and triage
readable	Page-aware Markdown	A first pass with routing diagnostics
structured	Hierarchy and summarized tables	Listings, dashboards, and documentation
full	Full-page Markdown	Content outside a clear article body
raw	HTML	Source inspection or custom parsing

Quick Start

Install. Fetch. Done.

Requires Python 3.10 or newer. Begin with the article fixture, then render the JavaScript listing to see when a browser is needed.

Install and fetch

python -m pip install "pulldown==0.4.2"
pulldown get https://pulldown.making-minds.ai/examples/field-notes.html --detail readable
pulldown get https://pulldown.making-minds.ai/examples/field-notes.html --detail structured

Render the JavaScript example

python -m pip install "pulldown[render]==0.4.2"
python -m playwright install chromium
pulldown get https://pulldown.making-minds.ai/examples/live-board.html --render --detail structured

The gallery includes the HTTP-only output for comparison. Installing the render extra and Chromium are separate steps. Rendering requires a canonical URL: redirects, workers, and WebSockets are blocked.

Python API

import asyncio
from pulldown import fetch, Detail

async def main():
    result = await fetch(
        "https://pulldown.making-minds.ai/examples/field-notes.html",
        detail=Detail.structured,
    )
    if not result.ok:
        raise RuntimeError(result.error or result.status_code)
    print(result.content)
    print(result.meta.get("routing"))

asyncio.run(main())

Inspect the example results Read the README

Security

Explicit network and content limits.

URL checks and response limits reduce exposure when accepting a URL from a user or agent. These are network policy guards, not process isolation; set appropriate network and resource limits for the service running them.

SSRF Guard

Private, loopback, link-local, and reserved addresses are rejected by default. Ordinary HTTP fetching checks redirect targets before following them. Browser rendering rejects redirects; use the final canonical URL. Keep private-address access disabled when handling untrusted URLs.

address checks / HTTP redirect validation / explicit override

Size Caps

The default content limit is 10 MiB, configured with max_bytes or --max-bytes. HTTP responses are size-checked; rendering checks the serialized DOM after the browser loads it. This does not cap browser or process memory.

HTTP response cap / rendered DOM limit / configurable threshold

Scheme Validation

Only http:// and https:// URLs are accepted. file://, gopher://, ftp://, and other schemes are rejected before a connection is attempted.

scheme allowlist · early rejection · redirect chain validated

Integrations

Use the CLI, Python API, or MCP server.

CLI

terminal

pulldown get https://pulldown.making-minds.ai/examples/field-notes.html --detail readable --json-output

pulldown crawl https://pulldown.making-minds.ai/examples/field-notes.html --max-depth 1 --max-pages 3 --detail minimal

Python API

async

import asyncio
from pulldown import fetch, Detail

result = asyncio.run(fetch(
    "https://pulldown.making-minds.ai/examples/field-notes.html",
    detail=Detail.readable,
    max_bytes=1_000_000,
))
if not result.ok:
    raise RuntimeError(result.error or result.status_code)
print(result.content)

MCP Server

Claude · Codex · agents

Install the server extra in the environment your host uses:

python -m pip install "pulldown[mcp]==0.4.2"

{
  "mcpServers": {
    "pulldown": {
      "command": "python",
      "args": ["-m", "pulldown.mcp_server"],
      "env": {
        "PULLDOWN_CACHE_DIR": "~/.cache/pulldown"
      }
    }
  }
}

From URL to result

Keep the extraction decision visible.

A successful fetch does not guarantee that the output contains everything your task needs. The result includes status, content, errors, and routing metadata so you can inspect the decision and choose another mode.

Step	What to inspect	Next action
Fetch	HTTP status and result.error	Resolve the fetch failure before extracting content.
Choose a mode	Headings, links, tables, and page chrome	Compare readable, structured, and full with the source.
Inspect routing	result.meta["routing"]	Check the selected strategy and render recommendation.
Render when needed	HTTP-only and rendered outputs	Use --render when JavaScript supplies the missing content.

Compare the five modes making-minds.ai

HTTP-first web contentfor LLM pipelines.