# Minifetch API Documentation ## Overview Minifetch provides composable extraction APIs for humans and AI Agents-- making web pages simple to access. Designed for SEO research, agentic workflows, RAG pipelines, and AI training datasets, Minifetch is built around a core principle: ethical extraction that identifies itself transparently, respects robots.txt, and never hammers target servers. This makes Minifetch the right choice for developers and teams who need defensible, production-safe data collection. And it's why pages that block aggressive scrapers often work just fine with Minifetch. This documentation covers the the main endpoints available for developers. It also covers details about accessing the Minifetch API as well as example data. All API endpoints return responses in JSON format. The Minifetch API Client has a Quickstart for Javascript/ TypeScript if you're looking to add it to your project quickly: https://www.npmjs.com/package/minifetch-api ### Features #### โœ… Sign up for an account and get your first 125-250 fetches for free. ๐ŸŽ‰๐ŸŽ‰ #### โœ… Always pay-as-you-go at competitive prices. #### โœ… Transparent user-agent identification โ€” site owners know who we are #### โ›” No charge for blocked pages (403 errors). ### Use Cases #### SEO Research & Competitive Analysis #### AI Agents & RAG Pipelines #### Compliance-First Data Collection #### Content Indexing & Link Validation #### Ad Enhancement #### Sitemap Building & Crawl Planning #### Building AI Training Datasets ## Payment This service uses micropayments. Prices for a "fetch" are listed on each API endpoint below. There are two ways to pay. Both of these payment methods work on the Minifetch.com homepage or with the Minifetch API client: https://www.npmjs.com/package/minifetch-api ### 1. Credit Card You can sign up for an account and get started with free credits by visiting the dashboard: https://minifetch.com/dashboard Click the "Sign up" button and verify your email address to create your account. Once you are signed in to your account, each successful "Go Fetch!" will be deducted from your credit balance. After that, top up for as little as $2 with your credit card. ### 2. USDC via x402 on Base or Solana If you do not wish to sign up for an account or you just prefer to pay with USDC stablecoin on Base or Solana network, you can pay with your crypto wallet. For payment implementation details, see the x402 protocol documentation: https://www.x402.org/ Either way, you always pay as you go. There is no Minifetch account setup fee or monthly fee. We do not charge for explicitly blocked pages or pages that error. ## How To Access Minifetch can be accessed from a browser, the Minifetch API client, curl, or AI tools. Pick the one that fits your setup. ### Browser Visit the Minifetch homepage (https://minifetch.com) or hit any API endpoint URL directly in your browser. ### Minifetch API Client [NEW] For developers: the minifetch-api client is an NPM package purpose-built for Javascript/ TypeScript. It handles payments automatically. It's the simplest way to integrate Minifetch into your Node.js application: https://www.npmjs.com/package/minifetch-api ### Curl + API Key For developers: you can sign up for an account, create an API key and query the API endpoints via curl from your CLI. ``` curl "https://minifetch.com/api/v1/extract/url-preview?url=https://anthropic.com" \ -H "Authorization: Bearer [your_api_key]" ``` ### Coinbase Agents Wallet / awal (Recommended for AI Agents) Agentic Wallets (https://docs.cdp.coinbase.com/agentic-wallet/quickstart) is Coinbase's dedicated wallet infrastructure for autonomous AI agents. It's the recommended approach for programmatic and agent-to-agent x402 usage: non-custodial, secured in a TEE, with programmable spending limits and gasless trading on Base. Get started: ``` npx awal ``` Once your wallet is funded with USDC on Base, your agent can call Minifetch endpoints directly using any x402-compatible client (e.g. @x402/fetch) or via the built-in pay-for-service skill: ``` npx skills add coinbase/agentic-wallet-skills ``` ### Coinbase Payments MCP (For AI Assistants) The Coinbase Payments MCP (https://www.npmjs.com/package/@coinbase/payments-mcp) gives AI assistants like Claude Desktop a built-in wallet so they can discover and pay for x402 services โ€” including Minifetch โ€” without any code. Install it with one command and sign in with your email; no private key needed. ``` npx @coinbase/payments-mcp ``` The installer will walk you through configuring your MCP client (Claude Desktop, Claude Code, Gemini CLI, etc). Once installed, ask your AI assistant: "What x402 services are available?" and it will discover Minifetch on the x402 Bazaar automatically. ## Agent Skills (SKILL.md) Task-specific step-by-step guides for AI agents using the Minifetch API: - SEO Audit: https://minifetch.com/skills/seo-audit/SKILL.md - Competitive SEO Analysis: https://minifetch.com/skills/competitive-analysis/SKILL.md - Allow Minifetch via robots.txt: https://minifetch.com/skills/unblock-minifetch/SKILL.md - View all skills: https://minifetch.com/SKILL.md ## Base URL The Minifetch API client will handle this for you. All API requests will be made to: https://minifetch.com ## Endpoints ### URL Check (Free) `GET /api/v1/free/preflight/url-check` #### Price: Free #### Use Case: Call this endpoint before the /extract/ endpoints. #### Description: This free endpoint checks if a URL is allowed to be crawled according to the website's robots.txt file. Use this before making paid requests to the /extract/ endpoints to avoid spending credits on uncrawlable URLs. #### Request Parameters: - url (string, required): The target URL to check against robots.txt #### Example Request: ``` GET https://minifetch.com/api/v1/free/preflight/url-check?url=https://example.com ``` #### Success Response (200 OK): ```json { "success": true, "results": [ { "data": { "url": "https://example.com", "allowed": true, "message": "robots.txt not found, defaulting to allowed", "crawlDelay": 1 } } ] } ``` #### Status Codes: 200 Success 400 Bad Request - Missing or invalid target URL 429 Too Many Requests - Back off and retry, max 5-10 requests per second 503 Service Unavailable - Timeout or fetch error on target URL. Try again later. ### Extract URL Preview (Paid) `GET /api/v1/extract/url-preview` `GET /api/v1/x402/extract/url-preview` #### Price: $0.001 #### Use Cases: AI Agents & RAG Pipelines #### Description: This endpoint fetches and extracts a light, token-efficient preview of a URL: title, description, and image (only). #### Request Parameters: - url (string, required): The target URL from which to extract preview #### Example Request: ``` GET https://minifetch.com/api/v1/extract/url-preview?url=https://github.com ``` #### Success Response (200 OK): ```json { "success": true, "results": [ { "data": { "requestUrl": "https://github.com", // URL you requested "url": "https://github.com/", // Last URL in request chain, if there were redirects "title": "GitHub ยท Change is constant. GitHub keeps you ahead. ยท GitHub", "description": "Join the world's most widely adopted, AI-powered developer platform where millions of developers, businesses, and the largest open source community build software that advances humanity.", "image": "https://images.ctfassets.net/8aevphvgewt8/4pe4eOtUJ0ARpZRE4fNekf/f52b1f9c52f059a33170229883731ed0/GH-Homepage-Universe-img.png", "minifetchCache": { "hit": "false", "cachedAt": "2026-01-26T00:13:43.490Z", "expiresAt": "2026-01-26T00:15:43.490Z" } } } ] } ``` #### Status Codes: 200 Success 400 Bad Request - Missing or invalid target URL 402 Payment Required 429 Too Many Requests - Back off and retry, max 5-10 requests per second 500 Internal Server Error 502 Bad Gateway - 403 block or error, DNS lookup error, etc on target URL 503 Service Unavailable - Timeout or fetch error on target URL. Try again later. ### Extract URL Content (Paid) `GET /api/v1/extract/url-content` `GET /api/v1/x402/extract/url-content` #### Price: $0.002 #### Use Cases: AI Agents & RAG Pipelines #### Description: Extracts a clean, LLM-ready, token-efficient content summary as markdown from a URL. Removes ads, nav, scripts. Much more efficient than raw HTML fetches. #### Request Parameters: - url (string, required): The target URL from which to extract content - includeMediaUrls (boolean, optional): If set to true, includes image and video URLs in the result. #### Example Request: ``` GET https://minifetch.com/api/v1/extract/url-content?url=https://apnews.com/article/time-person-of-year-2025-77ec65c6792bc99ec2ce1919c5f421ea&includeMediaUrls=true ``` #### Success Response (200 OK): ```json { "success": true, "queryParameters": { "includeMediaUrls": "true" }, "results": [ { "data": { "requestUrl": "https://apnews.com/article/time- ...etc", // URL you requested "url": "https://apnews.com/article/time- ...etc", // Last URL in request chain, if there were redirects "summary": "# Time magazine names 'Architects of AI' as its person of the year... etc" "mediaUrls": [ { "url": "https://assets.apnews.com/19/66/bc546486408c8595f01753a9fbeb/ap-logo-176-by-208.svg", "alt": "AP Logo" }, ...etc ], "minifetchCache": { "hit": "false", "cachedAt": "2026-01-26T05:01:37.248Z", "expiresAt": "2026-01-26T05:03:37.248Z" } } } ] } ``` #### Status Codes: 200 Success 400 Bad Request - Missing or invalid target URL 402 Payment Required 429 Too Many Requests - Back off and retry, max 5-10 requests per second 500 Internal Server Error 502 Bad Gateway - 403 block or error, DNS lookup error, etc on target URL 503 Service Unavailable - Timeout or fetch error on target URL. Try again later. ### Extract URL Links (Paid) `GET /api/v1/extract/url-links` `GET /api/v1/x402/extract/url-links` #### Price: $0.002 #### Use Cases: SEO, Link Validation & Analysis, Sitemap Building & Crawl Planning #### Description: Extracts all links from a URL categorized by type (internal/external/anchor) with SEO metadata. Detects image links, nofollow attributes, and analyzes external domain distribution. #### Request Parameters: - url (string, required): The target URL from which to extract links #### Example Request: ``` GET https://minifetch.com/api/v1/extract/url-links?url=https://anthropic.com ``` #### Success Response (200 OK): ```json { "success": true, "results": [ { "data": { "requestUrl": "https://www.anthropic.com", // URL you requested "url": "https://www.anthropic.com/", // Last URL in request chain, if there were redirects "links": { "internal": [ { "href": "https://www.anthropic.com/", "text": "", "rel": [] }, { "href": "https://www.anthropic.com/research", "text": "Research", "rel": [] }, ...etc ], "external": [ { "href": "http://trust.anthropic.com/", "text": "Security and compliance", "rel": [] }, { "href": "https://claude.com/resources/tutorials", "text": "Tutorials", "rel": [] }, ...etc ], "anchors": [ { "href": "#main", "text": "Skip to main content" }, { "href": "#footer", "text": "Skip to footer" }, ...etc ], "summary": { "totalLinks": 144, "internalCount": 67, "externalCount": 77, "anchorCount": 4, "nofollowCount": 0, "uniqueExternalDomains": 10, "topExternalDomains": [ { "domain": "claude.com", "count": 51 }, { "domain": "claude.ai", "count": 9 }, ...etc ] }, "minifetchCache": { "hit": "false", "cachedAt": "2026-01-24T21:23:21.344Z", "expiresAt": "2026-01-24T21:25:21.344Z" } } } } ] } ``` #### Response Fields: - internal (array): Links to pages on the same domain - external (array): Links to pages on different domains - anchors (array): Fragment/anchor links (e.g., #section) - href (string): The absolute URL of the link - text (string): The visible anchor text - rel (array): Rel attributes (nofollow, ugc, sponsored, etc.) - title (string, optional): Title attribute if present - hasImage (boolean, optional): True if link contains an image - imageAlt (string, optional): Alt text of image inside link - summary.totalLinks (number): Total count of internal + external links - summary.uniqueExternalDomains (number): Number of unique external domains linked to - summary.topExternalDomains (array): Top 10 external domains by link count #### Status Codes: 200 Success 400 Bad Request - Missing or invalid target URL 429 Too Many Requests - Back off and retry, max 5-10 requests per second 402 Payment Required 500 Internal Server Error 502 Bad Gateway - 403 block or error, DNS lookup error, etc on target URL 503 Service Unavailable - Timeout or fetch error on target URL. Try again later. ### Extract URL Metadata (Paid) `GET /api/v1/extract/url-metadata` `GET /api/v1/x402/extract/url-metadata` #### Price: $0.002 #### Use Cases: SEO Research, Content Indexing, Ad Enhancement, Building AI Training Datasets #### Description: This endpoint fetches and extracts rich structured metadata from a URL: title, description, meta tags, open graph tags, twitter tags, hreflang, json-ld, citations, images, headings, response headers, and more. The structured output doubles as an instant SEO audit โ€” every field maps directly to a ranking signal or competitive intelligence data point. #### Request Parameters: - url (string, required): The target URL from which to extract metadata - verbosity (string, optional): Defaults to "standard". Set to "full" for larger response. - includeResponseBody (boolean, optional): If set to "true", includes the full HTML response body in the result #### Example Request: ``` GET https://minifetch.com/api/v1/extract/url-metadata?url=https://example.com&includeResponseBody=true ``` #### Success Response (200 OK): ```json { "success": true, "queryParameters": { "verbosity": "standard", "includeResponseBody": true }, "results": [ { "data": { "requestUrl": "https://example.com", // URL you requested "url": "https://example.com", // Last URL in request chain, if there were redirects "responseStatusCode": 200, "responseHeaders": {}, "canonical": "", "lang": "en", "hreflang": [], "charset": "utf-8", "viewport": "", "title": "Example Domain", "favicons": [], // Additional metadata fields, see "Example Data" section below } } ] } ``` #### Status Codes: 200 Success 400 Bad Request - Missing or invalid target URL 429 Too Many Requests - Back off and retry, max 5-10 requests per second 402 Payment Required 500 Internal Server Error 502 Bad Gateway - 403 block or error, DNS lookup error, etc on target URL 503 Service Unavailable - Timeout or fetch error on target URL. Try again later. ## Example Data To see example responses, check these results: ### Extract URL Preview ($0.001) - Github: https://minifetch.com/result/example/preview/github ### Extract URL Content ($0.002 each): - Weather.com: https://minifetch.com/result/example/content/weather - Huffington Post: https://minifetch.com/result/example/content/huffpost - AP News (?includeMediaUrls=true): https://minifetch.com/result/example/content/apnews ### Extract URL Links ($0.002 each): - Amazon Product Page: https://minifetch.com/result/example/links/amazon - Anthropic: https://minifetch.com/result/example/links/anthropic ### Extract URL Metadata ($0.002 each): - GitHub: https://minifetch.com/result/example/metadata/github - eBay Product Page (?verbosity=full): https://minifetch.com/result/example/metadata/ebay - Anthropic (?verbosity=full&includeResponseBody=true): https://minifetch.com/result/example/metadata/anthropic/verbose ## Libraries & SDKs Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the NPM community for 10+ years. During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package: https://www.npmjs.com/package/url-metadata For direct integration with your codebase, consider: - minifetch-api (https://www.npmjs.com/package/minifetch-api): the official Minifetch API client for Javascript/ TypeScript - url-metadata (https://www.npmjs.com/package/url-metadata): the open-source npm package (10+ years, 3.6 Million downloads) that powers Minifetch ## Caching & Rate Limiting Minifetch implements several measures to ensure responsible and efficient operation: - Responses are cached to reduce load on target websites - Look for "minifetchCache: { ... }" in response data for cache details - Crawl delays from robots.txt are strictly observed - Rate limiting is applied per domain to prevent overloading target websites - The API uses a proxy pool for making requests These measures help us maintain an ethical scraping approach while providing reliable service. ## Service Limitations Minifetch only extracts publicly available metadata and content from pages accessible without authentication and Javascript execution. What Minifetch does NOT do: - Ignore robots.txt directives - Access authenticated or logged-in content - Create accounts or log into user sessions - Perform transactional actions (checkout, bidding, purchasing, form submissions) - Bypass paywalls or access restricted content What Minifetch does NOT do currently but may offer in the future as an add-on: - Javascript execution ## Need Help? If you have questions about the API or need assistance with implementation, use our feedback form and we'll get back to you: https://forms.gle/rkMi7T23bHJc8XFw9