API Docs

Base URL

All API requests should be made to:

Endpoint: URL Check (Free)

This free endpoint checks if a URL is allowed to be fetched according to the website's robots.txt file. Use this before making paid requests to avoid spending extra on un-fetchable URLs.

Request Parameters:

Parameter	Type	Required	Description
url	string	Yes	The URL to check against robots.txt

Example Request:

GET https://minifetch.com/api/v1/free/preflight/url-check?url=https://example.com

Success Response:

{
  "success": true,
  "url": "https://example.com",
  "allowed": true,
  "reason": null,
  "crawlDelay": 1,
  "message": "Access allowed by robots.txt"
}

Error Response:

{
  "success": false,
  "error": "invalid_url",
  "message": "Invalid URL format"
}

Status Codes:

• 200 Success
• 400 Bad Request - Missing or invalid URL

Endpoint: Extract URL Metadata (Paid)

This endpoint fetches and extracts HTML metadata from a specified URL. You'll be prompted to pay $0.01 USDC with your browser plugin wallet on Coinbase's Base network. You can visit the API endpoints directly using your browser or click the "Go Fetch!" button from the Minifetch.com homepage. The Minifetch API client library can also handle payment for you, sign up for our waitlist and we'll notify you when its ready.

Request Parameters:

Parameter	Type	Required	Description
url	string	Yes	The URL from which to extract metadata
includeResponseBody	boolean	No	If set to "true", includes the full HTML response body in the result

Example Request:

GET https://minifetch.com/api/v1/x402/extract/url-metadata?url=https://example.com&includeResponseBody=true

Success Response:

{
  "success": true,
  "metadata": {
    "requestUrl": "https://example.com", // URL you requested
    "url": "https://example.com", // Last URL in request chain, if there were redirects
    "canonical": "",
    "lang": "en",
    "hreflang": [],
    "charset": "utf-8",
    "viewport": "",
    "title": "Example Domain",
    "favicons": [],
    // Additional metadata fields, see "Example Metadata" section below
  }
}

Error Responses:

{
  "success": false,
  "error": "invalid_url",
  "message": "Invalid URL format",
  "url": "https://example.com"
}

{
  "success": false,
  "error": "robots_blocked",
  "message": "URL is blocked by robots.txt",
  "url": "https://example.com/blocked-page"
}

Status Codes:

• 200 Success
• 400 Bad Request - Missing or invalid URL
• 403 Forbidden - URL blocked by robots.txt
• 503 Service Unavailable - Request timeout or fetch error
• 500 Internal Server Error

Example Metadata

To see example metadata responses, check these results:

• GitHub Metadata Example
• Wikipedia Metadata Example
• Anthropic Metadata Example (with response body)

Caching & Rate Limiting

Minifetch implements several measures to ensure responsible and efficient operation:

• Responses are cached to reduce load on target websites
• Crawl delays from robots.txt are strictly observed
• Rate limiting is applied per domain to prevent overloading target websites
• The API uses a proxy pool for making requests

These measures help us maintain an ethical scraping approach while providing reliable service.

Libraries & SDKs

Minifetch is based on the open-source url-metadata npm package that has been serving the community for 9+ years.

For direct integration with your codebase, consider:

• Implementing the Minifetch API client. Sign up for the waitlist and we'll email you when it's ready.
• Using the url-metadata npm package directly in your Node.js applications