For SEO research and AI projects
Minifetch provides composable extraction APIs that humans and AI Agents can autonomously discover, orchestrate, and pay for— making web pages simple to access. The API follows ethical scraping practices, respecting robots.txt and using rate limiting to avoid overloading target websites. You can pay-as-you-go with x402 micropayments in USDC, typically for SEO research and AI projects such as AI Agents, RAG pipelines, and building training datasets. There is no Minifetch account setup or monthly fee.
This documentation covers the the main endpoints available for developers. All endpoints return responses in JSON format.
This service uses the x402 protocol for micropayments with USDC on Coinbase's Base network or Solana. You can pay as you go—there is no Minifetch account setup or monthly fee. Prices are listed on each endpoint below. For payment implementation details, see the x402 protocol documentation.
Prefer a traditional credit card & API key? Sign up for the waitlist and we'll let you know when it's ready.
Minifetch can be accessed from a browser, the Minifetch npm client, or via AI tools. Each method handles x402 payments differently. Pick the one that fits your setup.
Visit the Minifetch homepage or hit any API endpoint URL directly in your browser. For paid endpoints the x402 paywall will prompt you to pay with a browser wallet extension like Coinbase Wallet or MetaMask. The preflight URL Check endpoint is always free.
The minifetch-api npm package is a purpose-built TypeScript client that handles x402 payments automatically. It's the simplest way to integrate Minifetch into your Node.js application.
npm install minifetch-api --save
Agentic Wallets (awal)
is Coinbase's dedicated wallet infrastructure for autonomous AI agents. It's the recommended
approach for programmatic and agent-to-agent x402 usage: non-custodial, secured in a TEE,
with programmable spending limits and gasless trading on Base.
npx awal
Once your wallet is funded with USDC on Base, your agent can call Minifetch endpoints
directly using any x402-compatible client (e.g. @x402/fetch) or via the built-in
pay-for-service skill:
npx skills add coinbase/agentic-wallet-skillsThe Coinbase Payments MCP gives AI assistants like Claude Desktop a built-in wallet so they can discover and pay for x402 services—including Minifetch—without any code. Install it with one command and sign in with your email; no private key needed.
npx @coinbase/payments-mcpThe installer will walk you through configuring your MCP client (Claude Desktop, Claude Code, Gemini CLI, etc.). Once installed, ask your AI assistant: "What x402 services are available?" and it will discover Minifetch on the x402 Bazaar automatically.
All API requests should be made to:
https://minifetch.com
Free
Call this endpoint before the extraction endpoints
This free endpoint checks if a URL is allowed to be fetched according to the website's robots.txt file. Use this before making paid requests to avoid spending extra on un-fetchable URLs. This ensures ethical scraping that respects robots.txt and site policies so you can sleep better at night. You can visit the API endpoint directly using your browser, or click the "Check URL" button from the Minifetch.com homepage. See How To Access for programmatic and AI agent options.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The URL to check against robots.txt |
GET https://minifetch.com/api/v1/free/preflight/url-check?url=https://example.com{
"success": true,
"results": [
{
"data": {
"url": "https://example.com",
"allowed": true,
"message": "robots.txt not found, defaulting to allowed",
"crawlDelay": 1
}
}
]
}$0.001 USDC
AI Agents & RAG Pipelines
This endpoint fetches and extracts a light, token-efficient preview of a URL: title, description, and image (only). You can visit the API endpoint directly using your browser. See How To Access for programmatic and AI agent options.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The URL from which to extract the preview |
GET https://minifetch.com/api/v1/x402/extract/url-preview?url=https://github.com{
"success": true,
"results": [
{
"data": {
"requestUrl": "https://github.com", // URL you requested
"url": "https://github.com/", // Last URL in request chain, if there were redirects
"title": "GitHub · Change is constant. GitHub keeps you ahead. · GitHub",
"description": "Join the world's most widely adopted, AI-powered developer platform where millions of developers, businesses, and the largest open source community build software that advances humanity.",
"image": "https://images.ctfassets.net/8aevphvgewt8/4pe4eOtUJ0ARpZRE4fNekf/f52b1f9c52f059a33170229883731ed0/GH-Homepage-Universe-img.png",
"minifetchCache": {
"hit": "false",
"cachedAt": "2026-01-26T00:13:43.490Z",
"expiresAt": "2026-01-26T00:15:43.490Z"
}
}
}
]
}$0.002 USDC
AI Agents & RAG Pipelines
Extracts a clean, LLM-ready, token-efficient content summary as markdown from a URL. Removes ads, nav, scripts. Much more efficient than raw HTML fetches. You can visit the API endpoint directly using your browser. See How To Access for programmatic and AI agent options.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The URL from which to extract content |
| includeMediaUrls | boolean | No | If set to "true", includes image and video URLs in the result |
GET https://minifetch.com/api/v1/x402/extract/url-content?url=https://apnews.com/article/time-person-of-year-2025-77ec65c6792bc99ec2ce1919c5f421ea&includeMediaUrls=true{
"success": true,
"queryParameters": {
"includeMediaUrls": "true"
},
"results": [
{
"data": {
"requestUrl": "https://apnews.com/article/time- ...etc", // URL you requested
"url": "https://apnews.com/article/time- ...etc", // Last URL in request chain, if there were redirects
"summary": "# Time magazine names 'Architects of AI' as its person of the year... etc"
"mediaUrls": [
{
"url": "https://assets.apnews.com/19/66/bc546486408c8595f01753a9fbeb/ap-logo-176-by-208.svg",
"alt": "AP Logo"
},
...etc
],
"minifetchCache": {
"hit": "false",
"cachedAt": "2026-01-26T05:01:37.248Z",
"expiresAt": "2026-01-26T05:03:37.248Z"
}
}
}
]
}$0.002 USDC
SEO, Link Validation & Analysis, Sitemap Building & Crawl Planning
Extracts all links from a URL categorized by type (internal/external/anchor) with SEO metadata. Detects image links, nofollow attributes, and analyzes external domain distribution. You can visit the API endpoint directly using your browser, or click "Options" on the homepage, select "Extract Links" then "Go Fetch!". See How To Access for programmatic and AI agent options.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The URL from which to extract links |
GET https://minifetch.com/api/v1/x402/extract/url-links?url=https://anthropic.com{
"success": true,
"results": [
{
"data": {
"requestUrl": "https://www.anthropic.com", // URL you requested
"url": "https://www.anthropic.com/", // Last URL in request chain, if there were redirects
"links": {
"internal": [
{
"href": "https://www.anthropic.com/",
"text": "",
"rel": []
},
{
"href": "https://www.anthropic.com/research",
"text": "Research",
"rel": []
},
...etc
],
"external": [
{
"href": "http://trust.anthropic.com/",
"text": "Security and compliance",
"rel": []
},
{
"href": "https://claude.com/resources/tutorials",
"text": "Tutorials",
"rel": []
},
...etc
],
"anchors": [
{
"href": "#main",
"text": "Skip to main content"
},
{
"href": "#footer",
"text": "Skip to footer"
},
...etc
],
"summary": {
"totalLinks": 144,
"internalCount": 67,
"externalCount": 77,
"anchorCount": 4,
"nofollowCount": 0,
"uniqueExternalDomains": 10,
"topExternalDomains": [
{
"domain": "claude.com",
"count": 51
},
{
"domain": "claude.ai",
"count": 9
},
...etc
]
},
"minifetchCache": {
"hit": "false",
"cachedAt": "2026-01-24T21:23:21.344Z",
"expiresAt": "2026-01-24T21:25:21.344Z"
}
}
}
}
]
}$0.002 USDC
SEO Research, Content Indexing, Ad Enhancement, Building AI Training Datasets
This endpoint fetches and extracts rich structured metadata from a URL: title, description, og/twitter tags, json-ld, images, headings, response headers, and more. You can visit the API endpoint directly using your browser, or click the "Go Fetch!" button from the Minifetch.com homepage. See How To Access for programmatic and AI agent options.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The URL from which to extract metadata |
| verbosity | string | No | Defaults to "standard". Set to "full" for larger response. |
| includeResponseBody | boolean | No | If set to true, includes the full HTML response body as string in result. |
GET https://minifetch.com/api/v1/x402/extract/url-metadata?url=https://example.com&includeResponseBody=true{
"success": true,
"queryParameters": {
"verbosity": "standard",
"includeResponseBody": true
},
"results": [
{
"data": {
"requestUrl": "https://example.com", // URL you requested
"url": "https://example.com", // Last URL in request chain, if there were redirects
"responseStatusCode": 200,
"responseHeaders": {},
"canonical": "",
"lang": "en",
"hreflang": [],
"charset": "utf-8",
"viewport": "",
"title": "Example Domain",
"favicons": [],
// Additional metadata fields, see "Example Data" section below
}
}
]
}To see example responses, check these results:
Extract URL Preview ($0.001 each) AI
Extract URL Content ($0.002 each) AI
Extract URL Links ($0.002 each) SEOAI
Extract URL Metadata ($0.002 each) SEOAI
Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the community for 10+ years. During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.
For direct integration with your codebase, consider:
3.6 Million+ downloads over 10 years.
Add to your project:
minifetch-apiNEW — the official Minifetch client with x402 payments built in
npm install minifetch-api --save
url-metadata — the open-source npm package (10+ years, 3.6M downloads) that powers Minifetch
npm install url-metadata --save
Minifetch implements several measures to ensure responsible and efficient operation:
These measures help us maintain an ethical scraping approach while providing reliable service.
Minifetch only extracts publicly available metadata and content from pages accessible without authentication and Javascript execution.
What Minifetch does NOT do:
What Minifetch does NOT do currently but may offer in the future as an add-on:
If you have questions about the API or need assistance with implementation, use our feedback form and we'll get back to you.