# Minifetch API Documentation

## Overview
Minifetch provides composable extraction APIs that humans and AI Agents can autonomously discover, orchestrate, and pay for— making web pages simple to access.
The API follows ethical scraping practices, respecting robots.txt and using rate limiting to avoid overloading target websites.
You can pay-as-you-go with x402 micropayments in USDC, typically for SEO research and AI projects such as AI Agents or building training datasets.
There is no Minifetch account setup or monthly fee.

### Use Cases
#### SEO Research
#### Content Indexing
#### Ad Enhancement
#### Sitemap Building & Crawl Planning
#### AI Agents & RAG Pipelines
#### Building AI Training Datasets

### Enterprise Use Cases
#### IoT & M2M Payments
#### Telecom Applications

### Features
- No account setup required
- Competitive pay-as-you-go pricing
- Near instant, sub-cent transaction costs on Base network in USDC
- Device-to-API micropayments
- Network data monetization
- Edge compute API access
- Usage-based billing

## Payment
This service uses the x402 protocol for micropayments with USDC on Coinbase's Base network or Solana.
You can pay as you go—there is no Minifetch account setup or monthly fee.

For payment implementation details visit: https://www.x402.org/

### What You'll Need
- A cryptocurrency wallet (browser extension like Coinbase Wallet or MetaMask)
  - Download Coinbase Wallet: https://www.coinbase.com/wallet/articles/getting-started-extension
  - Download Metamask Wallet: https://metamask.io
- USDC on Coinbase's Base network or Solana

### Form Factors

**Browser:**
You can use the Minifetch.com homepage or visit the API endpoints directly in your browser.
It's free to check if a URL is fetchable using the "Check URL" button on the Minifetch.com homepage.
For paid endpoints, you'll be prompted to pay with your browser plugin wallet.

**API Client:**
For programmatic access, we're building an API client library that handles x402 payments for you.
Sign up for the waitlist and we'll notify you when it's ready: https://forms.gle/rkMi7T23bHJc8XFw9

**MCP for AI:**
We're also working on MCP (Model Context Protocol) integration so AI agents can use Minifetch directly.
Stay tuned!

## Base URL
All API requests should be made to: https://minifetch.com

## Endpoints

### URL Check (Free)
`GET /api/v1/free/preflight/url-check`

#### Price: Free
#### Use Case: Call this endpoint before the extraction endpoints

This free endpoint checks if a URL is allowed to be crawled according to the website's robots.txt file.
Use this before making paid requests to avoid spending credits on uncrawlable URLs.
This ensures ethical scraping that respects robots.txt and site policies so you can sleep better at night.
You can visit the API endpoint directly using your browser or click the "Check URL" button from the Minifetch.com homepage.
The Minifetch API client library can also handle payment for you, sign up for our waitlist and we'll notify you when its ready: https://forms.gle/rkMi7T23bHJc8XFw9

#### Request Parameters:
- url (string, required): The URL to check against robots.txt

#### Example Request:
```
GET https://minifetch.com/api/v1/free/preflight/url-check?url=https://example.com
```

#### Success Response (200 OK):
```json
{
  "success": true,
  "results": [
    {
      "data": {
        "url": "https://example.com",
        "allowed": true,
        "message": "robots.txt not found, defaulting to allowed",
        "crawlDelay": 1
      }
    }
  ]
}
```

#### Status Codes:
200 Success
400 Bad Request - Missing or invalid URL
503 Service Unavailable - Rate limiting, timeout, or fetch error. Try again later.

### Extract URL Metadata (Paid)
`GET /api/v1/x402/extract/url-metadata`

#### Price: $0.002 USDC
#### Use Cases: SEO Research, Content Indexing, Ad Enhancement, Building AI Training Datasets

This endpoint fetches and extracts rich structured metadata from a URL: title, description,
og/twitter tags, json-ld, images, headings, response headers, and more.
You can visit the API endpoint directly using your browser or click the "Go Fetch!" button from the Minifetch.com homepage.
The Minifetch API client library can also handle payment for you, sign up for our waitlist and we'll notify you when its ready: https://forms.gle/rkMi7T23bHJc8XFw9

#### Request Parameters:
- url (string, required): The URL from which to extract metadata
- includeResponseBody (boolean, optional): If set to "true", includes the full HTML response body in the result

#### Example Request:
```
GET https://minifetch.com/api/v1/x402/extract/url-metadata?url=https://example.com&includeResponseBody=true
```

#### Success Response (200 OK):
```json
{
  "success": true,
  "queryParameters": {
    "includeResponseBody": true
  },
  "results": [
    {
      "data": {
        "requestUrl": "https://example.com", // URL you requested
        "url": "https://example.com", // Last URL in request chain, if there were redirects
        "responseStatusCode": 200,
        "responseHeaders": {},
        "canonical": "",
        "lang": "en",
        "hreflang": [],
        "charset": "utf-8",
        "viewport": "",
        "title": "Example Domain",
        "favicons": [],
        // Additional metadata fields, see "Example Data" section below
      }
    }
  ]
}
```

#### Status Codes:
200 Success
400 Bad Request - Missing or invalid URL
402 Payment Required
500 Internal Server Error
502 Bad Gateway - Upstream block or error, DNS lookup errors, etc on target URL
503 Service Unavailable - Rate limiting, timeout, or fetch error. Try again later.

### Extract URL Links (Paid)
`GET /api/v1/x402/extract/url-links`

#### Price: $0.002 USDC
#### Use Cases: SEO, Link Validation & Analysis, Sitemap Building & Crawl Planning

Extracts all links from a URL categorized by type (internal/external/anchor) with SEO metadata.
Detects image links, nofollow attributes, and analyzes external domain distribution.
You can visit the API endpoint directly using your browser or click the
"Options" button on the Minifetch.com homepage, select "Extract Links" then "Go Fetch!".
The Minifetch API client library can also handle payment for you, sign up for our waitlist and we'll notify you when its ready: https://forms.gle/rkMi7T23bHJc8XFw9

#### Request Parameters:
- url (string, required): The URL from which to extract links

#### Example Request:
```
GET https://minifetch.com/api/v1/x402/extract/url-links?url=https://anthropic.com
```

#### Success Response (200 OK):
```json
{
  "success": true,
  "results": [
    {
      "data": {
        "requestUrl": "https://www.anthropic.com", // URL you requested
        "url": "https://www.anthropic.com/", // Last URL in request chain, if there were redirects
        "links": {
          "internal": [
            {
              "href": "https://www.anthropic.com/",
              "text": "",
              "rel": []
            },
            {
              "href": "https://www.anthropic.com/research",
              "text": "Research",
              "rel": []
            },
            ...etc
          ],
          "external": [
            {
              "href": "http://trust.anthropic.com/",
              "text": "Security and compliance",
              "rel": []
            },
            {
              "href": "https://claude.com/resources/tutorials",
              "text": "Tutorials",
              "rel": []
            },
            ...etc
          ],
          "anchors": [
            {
              "href": "#main",
              "text": "Skip to main content"
            },
            {
              "href": "#footer",
              "text": "Skip to footer"
            },
            ...etc
          ],
          "summary": {
            "totalLinks": 144,
            "internalCount": 67,
            "externalCount": 77,
            "anchorCount": 4,
            "nofollowCount": 0,
            "uniqueExternalDomains": 10,
            "topExternalDomains": [
              {
                "domain": "claude.com",
                "count": 51
              },
              {
                "domain": "claude.ai",
                "count": 9
              },
              ...etc
            ]
          },
          "minifetchCache": {
            "hit": "false",
            "cachedAt": "2026-01-24T21:23:21.344Z",
            "expiresAt": "2026-01-24T21:25:21.344Z"
          }
        }
      }
    }
  ]
}
```

#### Response Fields:
- internal (array): Links to pages on the same domain
- external (array): Links to pages on different domains
- anchors (array): Fragment/anchor links (e.g., #section)
- href (string): The absolute URL of the link
- text (string): The visible anchor text
- rel (array): Rel attributes (nofollow, ugc, sponsored, etc.)
- title (string, optional): Title attribute if present
- hasImage (boolean, optional): True if link contains an image
- imageAlt (string, optional): Alt text of image inside link
- summary.totalLinks (number): Total count of internal + external links
- summary.uniqueExternalDomains (number): Number of unique external domains linked to
- summary.topExternalDomains (array): Top 10 external domains by link count

#### Status Codes:
200 Success
400 Bad Request - Missing or invalid URL
402 Payment Required
500 Internal Server Error
502 Bad Gateway - Upstream block or error, DNS lookup errors, etc on target URL
503 Service Unavailable - Rate limiting, timeout, or fetch error. Try again later.

### Extract URL Preview (Paid)
`GET /api/v1/x402/extract/url-preview`

#### Price: $0.001 USDC
#### Use Cases: AI Agents & RAG Pipelines

This endpoint fetches and extracts a light, token-efficient preview of a URL: title, description, and image (only).
You can visit the API endpoint directly using your browser.
The Minifetch API client library can also handle payment for you, sign up for our waitlist and we'll notify you when its ready: https://forms.gle/rkMi7T23bHJc8XFw9

#### Request Parameters:
- url (string, required): The URL from which to extract metadata

#### Example Request:
```
GET https://minifetch.com/api/v1/x402/extract/url-preview?url=https://github.com
```

#### Success Response (200 OK):
```json
{
  "success": true,
  "results": [
    {
      "data": {
        "requestUrl": "https://github.com", // URL you requested
        "url": "https://github.com/", // Last URL in request chain, if there were redirects
        "title": "GitHub · Change is constant. GitHub keeps you ahead. · GitHub",
        "description": "Join the world's most widely adopted, AI-powered developer platform where millions of developers, businesses, and the largest open source community build software that advances humanity.",
        "image": "https://images.ctfassets.net/8aevphvgewt8/4pe4eOtUJ0ARpZRE4fNekf/f52b1f9c52f059a33170229883731ed0/GH-Homepage-Universe-img.png",
        "minifetchCache": {
          "hit": "false",
          "cachedAt": "2026-01-26T00:13:43.490Z",
          "expiresAt": "2026-01-26T00:15:43.490Z"
        }
    }
  ]
}
```

#### Status Codes:
200 Success
400 Bad Request - Missing or invalid URL
402 Payment Required
500 Internal Server Error
502 Bad Gateway - Upstream block or error, DNS lookup errors, etc on target URL
503 Service Unavailable - Rate limiting, timeout, or fetch error. Try again later.

### Extract URL Content (Paid)
`GET /api/v1/x402/extract/url-content`

#### Price: $0.002 USDC
#### Use Cases: AI Agents & RAG Pipelines

Extracts a clean, LLM-ready, token-efficient content summary as markdown from a URL.
Removes ads, nav, scripts. Much more efficient than raw HTML fetches.
You can visit the API endpoint directly using your browser.
The Minifetch API client library can also handle payment for you, sign up for our waitlist and we'll notify you when its ready: https://forms.gle/rkMi7T23bHJc8XFw9

#### Request Parameters:
- url (string, required): The URL from which to extract metadata
- includeMediaUrls (boolean, optional): If set to "true", includes image and video URLs in the result.

#### Example Request:
```
GET https://minifetch.com/api/v1/x402/extract/url-content?url=https://apnews.com/article/time-person-of-year-2025-77ec65c6792bc99ec2ce1919c5f421ea&includeMediaUrls=true
```

#### Success Response (200 OK):
```json
{
  "success": true,
  "queryParameters": {
    "includeMediaUrls": "true"
  },
  "results": [
    {
      "data": {
        "requestUrl": "https://apnews.com/article/time- ...etc", // URL you requested
        "url": "https://apnews.com/article/time- ...etc", // Last URL in request chain, if there were redirects
        "summary": "# Time magazine names 'Architects of AI' as its person of the year... etc"
        "mediaUrls": [
          {
            "url": "https://assets.apnews.com/19/66/bc546486408c8595f01753a9fbeb/ap-logo-176-by-208.svg",
            "alt": "AP Logo"
          },
          ...etc
        ],
        "minifetchCache": {
          "hit": "false",
          "cachedAt": "2026-01-26T05:01:37.248Z",
          "expiresAt": "2026-01-26T05:03:37.248Z"
        }
      }
    }
  ]
}
```

#### Status Codes:
200 Success
400 Bad Request - Missing or invalid URL
402 Payment Required
500 Internal Server Error
502 Bad Gateway - Upstream block or error, DNS lookup errors, etc on target URL
503 Service Unavailable - Rate limiting, timeout, or fetch error. Try again later.

## Example Data
To see example responses, check these results:

### Extract URL Metadata ($0.002 each):
- GitHub: https://minifetch.com/example-metadata-github
- Wikipedia: https://minifetch.com/example-metadata-wikipedia
- Anthropic (with response body): https://minifetch.com/example-metadata-anthropic

### Extract URL Links ($0.002 each):
- Anthropic: https://minifetch.com/example-links-anthropic

### Extract URL Preview ($0.002)
- Github: https://minifetch.com/example-preview-github

### Extract URL Content ($0.002 each):
- Weather.com: https://minifetch.com/example-content-weather.com
- Huffington Post: https://minifetch.com/example-content-huffpost
- AP News (incl media urls): https://minifetch.com/example-content-apnews

## Caching & Rate Limiting
Minifetch implements several measures to ensure responsible and efficient operation:

- Responses are cached to reduce load on target websites
- Look for "minifetchCache: { ... }" in response data for cache details
- Crawl delays from robots.txt are strictly observed
- Rate limiting is applied per domain to prevent overloading target websites
- The API uses a proxy pool for making requests

These measures help us maintain an ethical scraping approach while providing reliable service.

## Libraries & SDKs
Minifetch is based on the open-source url-metadata npm package: https://www.npmjs.com/package/url-metadata

For direct integration with your codebase, consider:
- Implementing the Minifetch API client for API access - Coming soon!  Sign up for our waitlist and we'll notify you when its ready: https://forms.gle/rkMi7T23bHJc8XFw9
- Using the url-metadata npm package directly in your Node.js applications