# Minifetch API Documentation ## Overview Minifetch is an SEO audit tool that also gives you the underlying data as a portable API you can use with your AI agents and automation workflows. Every fetch is built around a core principle: ethical data extraction that identifies itself transparently, respects robots.txt, and never hammers target servers. It's why sites that block aggressive scrapers often work just fine with Minifetch. The Minifetch API Client has a Quickstart for Javascript or TypeScript if you're looking to add it to your project quickly: https://www.npmjs.com/package/minifetch-api This documentation covers the the main endpoints available for developers. It also covers details about accessing the Minifetch API as well as example data. All API endpoints return responses in JSON format. ### Features #### ✅ Sign up for an account and get free starter credits. 🎉🎉 #### ✅ Always pay-per-fetch at competitive prices. No monthly fees. #### ✅ Transparent user-agent identification — site owners know who we are. #### ⛔ No charge for blocked pages (403 errors). ### Use Cases #### SEO Research & Competitive Analysis #### AI Agents & RAG Pipelines #### Compliance-First Data Collection #### Content Indexing & Link Validation #### Ad Enhancement #### Sitemap Building & Crawl Planning #### Building AI Training Datasets ## Quick Start Minifetch is pay-per-fetch. No setup fee, no monthly fee, no charges for blocked or errored pages. Prices per fetch are listed on each API endpoint below. Pick your payment method: ### 1. Credit Card Sign up on the to get free starter credits, then top up with your credit card: https://minifetch.com/dashboard ### 2. USDC on Base or Solana Pay directly from your crypto wallet via the x402 protocol. No account needed. Now pick whichever access method fits your setup: ### Browser Visit the Minifetch homepage (https://minifetch.com) or hit any API endpoint URL directly. Free to start if you sign up, then top up a with credit card after. You can also use your browser plugin crypto wallet. The default on the homepage is metadata extraction, with the option to extract links from pages as well. Great for trying out the service. If you want full API access, try the other options below. ### Minifetch API Client The minifetch-api client is an NPM package purpose-built for Javascript or TypeScript. The simplest way to integrate Minifetch into a Node.js app. Works with both API keys and USDC. Recommended for agent builders. https://www.npmjs.com/package/minifetch-api ### Curl + API Key Sign up, create an API key from the dashboard and query endpoints from the CLI. https://minifetch.com/dashboard ``` curl "https://minifetch.com/api/v1/extract/url-preview?url=https://anthropic.com" \ -H "Authorization: Bearer [your_api_key]" ``` ### Runtime Agent Access For agents that need to pay at runtime without human sign-up. If you're building an agent yourself, the API Client above is usually the better starting point. Coinbase Payments MCP (https://www.npmjs.com/package/@coinbase/payments-mcp) gives AI assistants like Claude Desktop a built-in wallet to discover and pay for x402 services with no code. One command, email sign-in, no private key. ``` npx @coinbase/payments-mcp ``` The installer configures your MCP client (Claude Desktop, Claude Code, Gemini CLI, etc). The agent can now discover Minifetch endpoints on the x402 Bazaar automatically. Agentic Wallets (https://docs.cdp.coinbase.com/agentic-wallet/welcome) `awal` is Coinbase's wallet infrastructure for autonomous agents: non-custodial, keys isolated, with programmable spending limits and gasless USDC trades on Base. Recommended for programmatic and agent-to-agent usage. ``` npx awal ``` Once funded, your agent can call Minifetch with any x402-compatible client (e.g. @x402/fetch) or skip the client entirely by adding awal's pay-for-service skill. ``` npx skills add coinbase/agentic-wallet-skills ``` ## API Base URL The Minifetch API client will handle this for you. All API requests will be made to: https://minifetch.com ## Endpoints ### URL Check (Free) `GET /api/v1/free/preflight/url-check` #### Description: This free endpoint checks if a URL is allowed to be crawled according to the website's robots.txt file. Use this before making paid requests to the /extract/ endpoints to avoid spending credits on uncrawlable URLs. #### Use Case: Call this endpoint before the /extract/ endpoints. #### Price: Free #### Request Parameters: - url (string, required): The target URL to check against robots.txt #### Example Request: ``` GET https://minifetch.com/api/v1/free/preflight/url-check?url=https://example.com ``` #### Example Response Data: ```json { "success": true, "results": [ { "data": { "url": "https://example.com", "allowed": true, "message": "robots.txt not found, defaulting to allowed", "crawlDelay": 1 } } ] } ``` #### Status Codes: 200 Success 400 Bad Request - Missing or invalid target URL 429 Too Many Requests - Back off and retry, max 5-10 requests per second 503 Service Unavailable - Timeout or fetch error on target URL. Try again later. ### Extract URL Metadata (Paid) `GET /api/v1/extract/url-metadata` `GET /api/v1/x402/extract/url-metadata` #### Description: This endpoint fetches and extracts rich structured metadata from a URL: title, description, meta tags, open graph tags, twitter tags, hreflang, json-ld, citations, images, headings, response headers, and more. The structured output doubles as an instant SEO audit — every field maps directly to a ranking signal or competitive intelligence data point. #### Use Cases: SEO Research, Content Indexing, Ad Enhancement, Building AI Training Datasets #### Price: $0.002 #### Request Parameters: - url (string, required): The target URL from which to extract metadata - verbosity (string, optional): Defaults to "standard". Set to "full" for larger response. - includeResponseBody (boolean, optional): If set to "true", includes the full HTML response body in the result #### Example Request: ``` GET https://minifetch.com/api/v1/extract/url-metadata?url=https://example.com&includeResponseBody=true ``` #### Example Response Data: - GitHub: https://minifetch.com/result/example/metadata/github - eBay Product Page (?verbosity=full): https://minifetch.com/result/example/metadata/ebay - Anthropic (?verbosity=full&includeResponseBody=true): https://minifetch.com/result/example/metadata/anthropic/verbose #### Status Codes: 200 Success 400 Bad Request - Missing or invalid target URL 429 Too Many Requests - Back off and retry, max 5-10 requests per second 402 Payment Required 500 Internal Server Error 502 Bad Gateway - 403 block or error, DNS lookup error, etc on target URL 503 Service Unavailable - Timeout or fetch error on target URL. Try again later. ### Extract URL Links (Paid) `GET /api/v1/extract/url-links` `GET /api/v1/x402/extract/url-links` #### Description: Extracts all links from a URL categorized by type (internal/external/anchor) with SEO metadata. Detects image links, nofollow attributes, and analyzes external domain distribution. #### Use Cases: SEO, Link Validation & Analysis, Sitemap Building & Crawl Planning #### Price: $0.002 #### Request Parameters: - url (string, required): The target URL from which to extract links #### Example Request: ``` GET https://minifetch.com/api/v1/extract/url-links?url=https://anthropic.com ``` #### Example Response Data: - Amazon Product Page: https://minifetch.com/result/example/links/amazon - Anthropic: https://minifetch.com/result/example/links/anthropic #### Status Codes: 200 Success 400 Bad Request - Missing or invalid target URL 429 Too Many Requests - Back off and retry, max 5-10 requests per second 402 Payment Required 500 Internal Server Error 502 Bad Gateway - 403 block or error, DNS lookup error, etc on target URL 503 Service Unavailable - Timeout or fetch error on target URL. Try again later. ### Extract URL Preview (Paid) `GET /api/v1/extract/url-preview` `GET /api/v1/x402/extract/url-preview` #### Description: This endpoint fetches and extracts a light, token-efficient preview of a URL: title, description, and image (only). #### Use Cases: AI Agents & RAG Pipelines #### Price: $0.001 #### Request Parameters: - url (string, required): The target URL from which to extract preview #### Example Request: ``` GET https://minifetch.com/api/v1/extract/url-preview?url=https://github.com ``` #### Example Response Data: - Github: https://minifetch.com/result/example/preview/github #### Status Codes: 200 Success 400 Bad Request - Missing or invalid target URL 402 Payment Required 429 Too Many Requests - Back off and retry, max 5-10 requests per second 500 Internal Server Error 502 Bad Gateway - 403 block or error, DNS lookup error, etc on target URL 503 Service Unavailable - Timeout or fetch error on target URL. Try again later. ### Extract URL Content (Paid) `GET /api/v1/extract/url-content` `GET /api/v1/x402/extract/url-content` #### Description: Extracts a clean, LLM-ready, token-efficient content summary as markdown from a URL. Removes ads, nav, scripts. Much more efficient than raw HTML fetches. #### Use Cases: AI Agents & RAG Pipelines #### Price: $0.002 #### Request Parameters: - url (string, required): The target URL from which to extract content - includeMediaUrls (boolean, optional): If set to true, includes image and video URLs in the result. #### Example Request: ``` GET https://minifetch.com/api/v1/extract/url-content?url=https://apnews.com/article/time-person-of-year-2025-77ec65c6792bc99ec2ce1919c5f421ea&includeMediaUrls=true ``` #### Example Response Data: - Weather.com: https://minifetch.com/result/example/content/weather - Huffington Post: https://minifetch.com/result/example/content/huffpost - AP News (?includeMediaUrls=true): https://minifetch.com/result/example/content/apnews #### Status Codes: 200 Success 400 Bad Request - Missing or invalid target URL 402 Payment Required 429 Too Many Requests - Back off and retry, max 5-10 requests per second 500 Internal Server Error 502 Bad Gateway - 403 block or error, DNS lookup error, etc on target URL 503 Service Unavailable - Timeout or fetch error on target URL. Try again later. ## Libraries & SDKs Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the NPM community for 10+ years. During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package: https://www.npmjs.com/package/url-metadata For direct integration with your codebase, consider: - minifetch-api (https://www.npmjs.com/package/minifetch-api): the official Minifetch API client for Javascript/ TypeScript - url-metadata (https://www.npmjs.com/package/url-metadata): the open-source npm package (10+ years, 3.6 Million downloads) that powers Minifetch Both packages from the same creator with 3.6 Million+ downloads over 10 years. Add them to your project with confidence. ## Agent Skills (SKILL.md) Task-specific step-by-step guides for AI agents using the Minifetch API: - SEO Audit: https://minifetch.com/skills/seo-audit/SKILL.md - Competitive SEO Analysis: https://minifetch.com/skills/competitive-analysis/SKILL.md - Allow Minifetch via robots.txt: https://minifetch.com/skills/unblock-minifetch/SKILL.md - View all skills: https://minifetch.com/SKILL.md ## Caching & Rate Limiting Minifetch implements several measures to ensure responsible and efficient operation: - Responses are cached to reduce load on target websites - Look for "minifetchCache: { ... }" in response data for cache details - Crawl delays from robots.txt are strictly observed - Rate limiting is applied per domain to prevent overloading target websites - The API uses a proxy pool for making requests These measures help us maintain an ethical scraping approach while providing reliable service. ## Service Limitations Minifetch only extracts publicly available metadata and content from pages accessible without authentication and Javascript execution. What Minifetch does NOT do: - Ignore robots.txt directives - Access authenticated or logged-in content - Create accounts or log into user sessions - Perform transactional actions (checkout, bidding, purchasing, form submissions) - Bypass paywalls or access restricted content What Minifetch does NOT do currently but may offer in the future as an add-on: - Javascript execution ## Need Help? If you have questions about the API or need assistance with implementation, use our feedback form and we'll get back to you: https://forms.gle/rkMi7T23bHJc8XFw9