Search, scrape, extract & SEO audit web pages. Minifetch is a hosted toolkit for developers and AI agents. Always pay-per-fetch, no subscription. This documentation covers the main endpoints available for developers, details about accessing the Minifetch API as well as example data. All API endpoints return responses in JSON format.
The Minifetch API Client has a Quick Start for Javascript or TypeScript if you're looking to add it to your project quickly.
Full API docs are also available for AI agents at llms.txt →
Working in a regulated space? By default, every response's
proxy block records the user agent and whether robots.txt was obeyed, so
how the data was gathered is on the record, per request.
Minifetch is pay-per-fetch. No setup fee, no monthly fee, no charges for blocked or errored pages. Price per fetch are listed on each API endpoint below. First, pick your payment method:
Sign up on the dashboard to get free starter credits worth 25 audits, then top up with your credit card later for as little as $2.
Pay directly from your crypto wallet. Just load your wallet with USDC on Base or Solana and you're ready. No "gas token" (ETH or SOL) required. No Minifetch account setup required.
Then click whichever access method fits your setup:
Visit the Minifetch homepage or hit any API endpoint URL directly using your browser's address bar. The default on the homepage is the full technical SEO audit. Great for trying out the service. If you want full API access, try the other options below.
Sign up, create an API key from the dashboard, and query endpoints from your CLI:
curl "https://minifetch.com/api/v1/run/seo-page-audit?url=https://anthropic.com" \
-H "Authorization: Bearer [your_api_key]"The minifetch-api client is an NPM package purpose-built for Javascript or TypeScript. The simplest way to integrate Minifetch into a Node.js app. Works with both API keys and USDC stablecoin. Recommended for agents and agent builders.
For agents that need to pay at runtime without human sign-up. If you're building an agent yourself, the Minifetch API client above is usually the better starting point.
Coinbase Payments MCP gives AI assistants like Claude Desktop a built-in wallet to discover and pay for x402 services with no code. One command, email sign-in, no private key.
npx @coinbase/payments-mcpThe installer configures your MCP client (Claude Desktop, Claude Code, Gemini CLI, etc). Once it's funded, ask your assistant to search the x402 Bazaar for "Minifetch" — every Minifetch endpoint shares that name and comes back together in one search. For more details point your AI assistant at the SEO Page Audit skill file.
Agentic Wallets
(awal) is Coinbase's wallet infrastructure for autonomous agents: non-custodial, keys isolated,
with programmable spending limits and gasless USDC trades on Base.
Recommended for programmatic and agent-to-agent usage.
npx awal
Once funded, your agent can call Minifetch with any x402-compatible client
(e.g. @x402/fetch) or skip the client entirely by adding awal's
pay-for-service skill.
npx skills add coinbase/agentic-wallet-skillsThe Minifetch API client will handle this for you. All API requests will be made to:
https://minifetch.com
Every endpoint below accepts GET (parameters in the query string)
or POST (parameters as a JSON body with
Content-Type: application/json). Use whichever fits your client;
the response is identical. Over POST, booleans may be real JSON booleans
(e.g. "omitEmpty": true); over the query string they are the
string "true".
The method you call Minifetch with is never forwarded: the target page is always fetched with GET.
curl -X POST "https://minifetch.com/api/v1/extract/url-metadata" \
-H "Authorization: Bearer [your_api_key]" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","fields":"title,description,og","omitEmpty":true}'$0.002
Web Search, AI Agents & RAG Pipelines, Content Discovery, SEO Research
Keyword web search by Ceramic.ai that returns ranked results as structured JSON: a
title, URL, and text snippet per result. The discovery entry point for agents and RAG pipelines:
search to find relevant pages, then go deeper on any result with the free
/preflight/url-check and then the paid /extract endpoints.
Use limit to cap results and descriptionLength
to trim each snippet (0 = titles and URLs only) for desired token efficiency.
| Parameter | Type | Required | Description |
|---|---|---|---|
| query | string | Yes | Keyword(s) to search for. Maximum 50 characters. |
| limit | integer | No |
Number of results to return, 1–10. Default
10. Out-of-range values are clamped to the nearest bound.
|
| descriptionLength | integer | No |
Maximum characters of each result's description snippet,
0–5000. Default 750.
0 returns titles and URLs only. Out-of-range values are clamped
to the nearest bound.
|
https://minifetch.com/api/v1/search/keyword?query=green+tea
https://minifetch.com/api/v1/search/keyword?query=green+tea&limit=5&descriptionLength=0
Free
Call this endpoint before the paid endpoints.
Free check to see if a target URL's robots.txt file allows Minifetch to scrape & extract.
Returns whether the URL is allowed and the crawl-delay Minifetch will observe (defaults
to 1 second so Minifetch never hammers servers or slows it down for real users).
Call this before making requests to the paid /extract and /run/seo-page-audit endpoints.
If the the target URL is blocked and the site is not yours, use the external proxy routes.
If the target URL is blocked and the site is yours or to set a custom crawl-delay, read the tutorial:
How To Allow Minifetch to Fetch Your Pages.
Just edited your robots.txt or crawl-delay? Add &fresh=true to skip Minifetch's
24-hour robots.txt cache and confirm your change registered (check the minifetchCache
object in the response to ensure it was freshly fetched).
Note: this is the one Minifetch endpoint that does not follow redirects.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The target URL to check against robots.txt |
| fresh | boolean | No |
If true, bypasses Minifetch's 24-hour robots.txt cache and re-fetches
robots.txt for this check. Use it right after editing your robots.txt or crawl-delay
(or adding/removing a Minifetch block) to confirm the change took effect. The
minifetchCache object shows whether the result was served from cache
("hit": true) or freshly fetched ("hit": false).
|
https://minifetch.com/api/v1/free/preflight/url-check?url=https://example.com
https://minifetch.com/api/v1/free/preflight/url-check?url=https://example.com&fresh=true
{
"success": true,
"results": [
{
"data": {
"url": "https://example.com",
"allowed": true,
"message": "robots.txt not found, defaulting to allowed",
"crawlDelay": 1,
"proxy": {
"proxyType": "minifetch",
"userAgent": "minifetch/1.0 (+https://minifetch.com/site-owner-faq)"
},
"minifetchCache": {
"hit": false,
"cachedAt": "2026-08-16T18:22:10.000Z",
"expiresAt": "2026-08-17T18:22:10.000Z"
}
}
}
]
}$0.002
Metadata Extraction, SEO Research & Competitive Analysis, AI Agents & Automation Pipelines
Extracts rich structured metadata from a URL: favicons, title, description, canonical, meta tags, Open Graph, Twitter Card, JSON-LD, redirects, headers, performance, headings, images. Every field maps directly to a ranking signal or competitive intelligence data point. Returns everything by default; query params narrow response (pass atomic field names, ex: ?fields=title,description or named groups: network,meta,og,twitter). ?omitEmpty=true trims tokens for AI agents.
Blocked, or the page needs JavaScript? Use the external proxy routes.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The target URL from which to extract metadata |
| fields | string | No |
Comma-separated selection to narrow the response to only the fields you need:
atomic field names (e.g. title), named groups
(network, meta, og, twitter),
and/or meta:<name> for a specific page meta tag.
Returns all fields when omitted, blank, or null; an unknown field returns a 400.
|
| omitEmpty | boolean | No |
If true, drops empty fields (empty string, [], {})
from the response for token efficiency.
|
| includeResponseBody | boolean | No | If set to true, includes the full HTML response body as string in result. |
https://minifetch.com/api/v1/extract/url-metadata?url=https://example.com
https://minifetch.com/api/v1/extract/url-metadata?url=https://example.com&fields=title,description,og&omitEmpty=true
$0.002
Content Extraction, Answer Engine Optimization (AEO), Generative Engine Optimization (GEO), Link Analysis
Returns clean markdown from a page after nav, ads, and scripts are stripped. Can also be used by site owners auditing AI readability: See what survives for AEO, GEO and AI search.
Blocked, or the page needs JavaScript? Use the external proxy routes.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The target URL from which to extract content |
| includeMediaUrls | boolean | No | If set to "true", includes the page's image URLs (url + alt, from <img> tags) as mediaUrls |
https://minifetch.com/api/v1/extract/url-content?url=https://apnews.com/article/time-person-of-year-2025-77ec65c6792bc99ec2ce1919c5f421ea&includeMediaUrls=true
$0.001
Content Extraction, Branding & Identity, Generative Engine Optimization (GEO), Link Analysis, AI Agents & Automation Pipelines
Extracts all fields used for a page's share previews: the lightweight cards that represent the page on social platforms, chat apps, and AI.
Blocked, or the page needs JavaScript? Use the external proxy routes.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The target URL from which to extract the preview |
https://minifetch.com/api/v1/extract/url-preview?url=https://github.com
$0.002
Link Analysis, SEO Research & Competitive Analysis, Sitemap Building & Crawl Planning, Content Extraction
Fetches and extracts all links from your URL categorized by type (internal/external/anchor).
Each link includes anchor text, rel attributes (nofollow, sponsored, ugc, etc.),
title, and target, plus image detection.
Summary stats include the most-linked-to internal pages (with anchor text variants used for each)
and top external domains by link count.
Blocked, or the page needs JavaScript? Use the external proxy routes.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The target URL from which to extract links |
https://minifetch.com/api/v1/extract/url-links?url=https://anthropic.com
| Endpoint | Native | /proxy | /proxy/render |
|---|---|---|---|
| Extract URL Metadata | $0.002 | $0.003 | $0.004 |
| Extract URL Content | $0.002 | $0.003 | $0.004 |
| Extract URL Preview | $0.001 | $0.002 | $0.003 |
| Extract URL Links | $0.002 | $0.003 | $0.004 |
Pages that block Minifetch, JavaScript-rendered pages, Geo-targeted pages
For scraping pages the native Minifetch /extract endpoints can't reach. Add /proxy
before /extract in the route path to fetch through an external rotating proxy
(webscrapingapi.com). Add /proxy/render to also execute the page's JavaScript
in a headless browser before extraction. Available for url-metadata,
url-links, url-preview and url-content.
Same request parameters and response data as the native endpoint, except:
proxy block records it: "proxyType": "managed" and
"robotsTxt": { "allowed": null }. If provenance matters for your
use-case, stay on the Minifetch native endpoints.
minifetchCache is always { "hit": false }.
/proxy/render.
performance and
redirects are not available (requesting either in fields
returns a 400). url is the final URL after redirects and
responseHeaders are the target page's headers as reported by the proxy.
Not sure which to use? Run the free /preflight/url-check first:
allowed means try the cheaper Minifetch native endpoint, blocked means use
/proxy. Use /proxy/render if the content you need is missing.
No charge for blocked or errored pages.
All parameters of the native endpoint, plus:
| Parameter | Type | Required | Description |
|---|---|---|---|
| country | string | No |
Two-letter country code (ISO 3166-1 alpha-2) to fetch from, e.g. us,
de. Omit to let the proxy choose. An invalid code returns a 400.
|
https://minifetch.com/api/v1/proxy/extract/url-metadata?url=https://example.com
https://minifetch.com/api/v1/proxy/render/extract/url-content?url=https://example.com&country=de
The proxy block on a /proxy/render response:
{
"proxyType": "managed",
"provider": "webscrapingapi.com",
"proxyUrl": "https://api.webscrapingapi.com/v2",
"userAgent": "unknown",
"render": true,
"country": "de",
"robotsTxt": {
"allowed": null
}
}$0.01
Technical SEO Page Audits, SEO Research, Pre-publish Page Checks, Site Migrations & Verification, Agent-Orchestrated Site Monitoring
Runs a full technical SEO page audit on a URL. Verify redirects, headers, missing or conflicting canonicals, JSON-LD, indexability and more. Composes our /preflight + /url-metadata + /url-links endpoint data into a structured pass/ warn/ fail report. No black-box scoring; just deterministic, composable signal you can act on or pipe into an agent. Audit rules are documented in the audit results UI and in the audit skill file.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The target URL to run the audit on |
https://minifetch.com/api/v1/run/seo-page-audit?url=https://example.com
Error responses are never charged.
results[0].error.message on a failed fetch is exactly one of the strings below in the response body. The minifetch-api client embeds it in a longer error message, so match with a contains check, not equality.
| error.message | Status | When |
|---|---|---|
invalid url |
400 | missing, malformed or unsupported url |
robots blocked * |
502 | robots.txt disallows the minifetch user agent |
upstream forbidden * |
502 | target returned 403 |
upstream rate limited * |
502 | target returned 429 |
upstream unavailable * |
502 | target returned 503 |
upstream not found |
502 | target returned 404 |
upstream bad request |
502 | target returned 400 |
upstream unauthorized |
502 | target returned 401 |
upstream payment required |
502 | target returned 402 (paywalled) |
upstream error |
502 | any other target 4xx/5xx, or the connection failed |
upstream too many redirects |
502 | redirect chain too long |
dns lookup failed |
502 | domain does not resolve |
queue timeout |
503 | timed out waiting on the site's crawl-delay |
fetch error |
503 | target timed out or the fetch failed |
proxy provider error |
503 | the external proxy provider failed (/proxy/ routes only) |
server error |
500 | unclassified failure |
* the target blocked Minifetch. On the /extract endpoints, retry via the /proxy/ routes.
results[0].error.statusCode, when present, is the target's
HTTP status (ex: 403), not Minifetch's. It is absent when the target never answered
(ex: robots blocked). results[0].data.url is the post-redirect
URL when known, else null.
Minifetch does not sanitize the content it returns. Every field, including titles, descriptions, links, and /url-content bodies, is authored by the remote page and must be treated as untrusted. Output sanitization is the caller's responsibility. The content is inert data and does not execute on its own, but you must sanitize it before use:
Minifetch is protected against server-side request forgery. It fetches only public http and https URLs. Requests to private, internal, loopback, link-local, reserved, and cloud metadata addresses are refused, including DNS rebinding attempts. localhost and private-network URLs will not fetch.
Minifetch offers two ways to fetch a page: the native Minifetch proxy or a rotating external proxy. The native Minifetch proxy is identified, respects robots.txt and is cheapest. The external proxy reaches pages the native one can't. You choose per fetch, by choosing the API route for your use-case. Built to be orchestrated by agents and humans who want maximal control.
| Native Minifetch proxy | External proxy | |
|---|---|---|
| Routes | /extract |
/proxy/extract, /proxy/render/extract |
| Fetches as | Identified minifetch user agent from a fixed IP |
External rotating proxy (webscrapingapi.com), optional country |
| robots.txt | Respected, including crawl-delay | Not checked |
| JavaScript | Not executed | Executed on /proxy/render |
| Cached | Yes | No, every call is a fresh fetch |
| Price | Cheapest | Native price plus a surcharge, see Proxy Routes |
proxy block: which proxy fetched the page, the user agent sent and
whether robots.txt was obeyed. For regulated use-cases where provenance matters,
stay on the native Minifetch proxy.
Not currently available on either proxy: authenticated or logged-in content.
Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the NPM community for 10+ years. During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.
For direct integration with your codebase, consider:
minifetch-apiNEW — the official Minifetch API client for Javascript/ Typescript
npm install minifetch-api --save
url-metadata — the open-source npm package that powers Minifetch
npm install url-metadata --save
Both from the same creator with 4 Million+ downloads over 10 years.
Add them to your project with confidence.
Task-specific step-by-step guides for AI agents using the Minifetch API:
To leave feedback or join the bulk credits waitlist:
use our feedback form.
If you have questions or need assistance:
join our Discord server.
While you're at it, give us a follow on X.com (Twitter):
@minifetch