Search, scrape & SEO audit web pages.

Built on ten years of open source

Minifetch is built by the creator and maintainer of url-metadata, an npm package with millions of downloads and tens of thousands of installs every week. Our technical SEO audit isn't a guess about what matters; it's the distilled version of what that package has been doing in production across the open web for a decade. You're getting battle-tested extraction logic with a real track record behind it, not a weekend audit clone.

A web toolkit for AI agents: search, scrape, extract

Most agent work on the web follows the same three steps: find the right pages, fetch them, and pull out the part that matters. Minifetch covers each step with its own API endpoint. Keyword web search returns ranked results as structured JSON, a title, URL and snippet per result. A free preflight check tells your agent which way to fetch a page before it spends anything: the native Minifetch proxy when the site's robots.txt allows it, or the external proxy routes when it doesn't or when the page needs JavaScript to render. Either way, the extract endpoints return exactly what the task needs: structured metadata, links, share previews or clean markdown content, with options to narrow the response and keep token counts down to save you money and compute.

Everything comes back as predictable JSON an agent can act on without scraping HTML itself. Payment is per fetch, by API key or by USDC straight from an agent's own wallet with no account or sign up, and blocked or errored pages are never charged. The same docs humans read are published for agents as llms.txt and SKILL.md, so an agent can discover the toolkit, learn it and start calling it on its own.

Composable by design; built for automation

That starts with how a page gets fetched. Minifetch offers two ways to fetch a web page: the native Minifetch proxy or an external rotating proxy. The native Minifetch proxy is identified, respects robots.txt and is cheapest. The external proxy reaches pages the native one can't, with optional JavaScript rendering for pages that build themselves in the browser. You choose per fetch, by choosing the API route for your use-case, and every response records which proxy fetched the page. Built to be orchestrated by agents and humans who want maximal control.

Our technical SEO page audit isn't a monolith. It's assembled from the same extraction primitives our API toolkit exposes individually: metadata, links, content, previews. Run the full audit when you want the complete picture, or call a single primitive when you don't. That matters when you're iterating on one fix or monitoring a page over time: you fetch just the piece you need, and you get back just that piece — a fraction of the price, and a fraction of the tokens. Same logic, composable parts, pay only for what you actually call.

SEO isn't dying, it's evolving

People have been calling SEO dead for a decade, and the latest obituary blames AI answers for replacing the click. But the underlying job hasn't changed. Whether it's Google's crawler or an LLM pulling context for an answer, every one of them starts the same way: fetch your HTML, parse it, decide whether it's worth surfacing. A clean, well-structured page wins in both worlds. A broken one loses in both. Run an SEO technical audit and you find out which one you're shipping.

Without SEO, there is no GEO or AEO

Generative Engine Optimization and Answer Engine Optimization are the new acronyms, but they sit on the same foundation SEO always has. An LLM can't cite a page it can't crawl, can't summarize a title that isn't there, and won't trust a site whose structured data contradicts itself. Titles, meta tags, canonical tags, json-ld, robots directives, clean markup — the boring technical fundamentals are exactly what the answer engines read too. Get the SEO basics right and you've already done most of the GEO and AEO work.

It's still about helping people find useful information

Strip away every acronym and the goal hasn't moved in twenty years: connect a person to information that actually helps them. Search engines, AI assistants, and answer engines are just different front doors to the same thing. We build tooling for that — for making good pages legible to whatever's reading them — not for gaming a ranking that won't last the quarter.

No black-box scoring

Most technical SEO audit tools hand you a number out of 100 and won't tell you how they got there, which is convenient for them and useless for you. Ours is the opposite. Every check we run is published as a readable SKILL.md file, so you can see exactly what we test, why it matters, and how to fix it. That openness is also what makes SEO audit automation trustworthy: if you're going to pipe audits into an agent or a CI step, you need to be able to inspect the deterministic rules driving them, not just take a score on faith.

Tell us what's broken

This is an MVP and it's improving every week. If a check is wrong, a finding is unclear, or there's a feature you would like, say so. Your feedback is valuable and shapes what gets built next. Contact us →