# Minifetch API Documentation

## Overview
Minifetch provides a simple API to extract HTML metadata from web pages. The API follows ethical scraping practices, respecting robots.txt and using rate limiting to avoid overloading target websites.

## Base URL
All API requests should be made to: https://minifetch.com

## Endpoints

### URL Check (Free)
`GET /api/v1/free/preflight/url-check`

This free endpoint checks if a URL is allowed to be crawled according to the website's robots.txt file. Use this before making paid requests to avoid spending credits on uncrawlable URLs.

#### Request Parameters:
- url (string, required): The URL to check against robots.txt

#### Example Request:
```
GET https://minifetch.com/api/v1/free/preflight/url-check?url=https://example.com
```

#### Success Response (200 OK):
```json
{
  "success": true,
  "url": "https://example.com",
  "allowed": true,
  "reason": null,
  "crawlDelay": 1,
  "message": "Access allowed by robots.txt"
}
```

#### Error Response (400 Bad Request):
```json
{
  "success": false,
  "error": "invalid_url",
  "message": "Invalid URL format"
}
```

### Extract URL Metadata (Paid)
`GET /api/v1/x402/extract/url-metadata`

This endpoint fetches HTML metadata from a specified URL.
It requires payment via Coinbase's x402 protocol.
From the Minifetch.com homepage you'll be prompted to pay $0.01 USDC cryptocurrency with your browser plugin wallet on Coinbase's Base network each time you click the "Go Fetch!" button, or you can navigate to the API endpoints directly from your browser.
The Minifetch API client library can also handle payment for you, sign up for our waitlist and we'll notify you when its ready: https://forms.gle/rkMi7T23bHJc8XFw9

#### Request Parameters:
- url (string, required): The URL from which to extract metadata
- includeResponseBody (boolean, optional): If set to "true", includes the full HTML response body in the result

#### Example Request:
```
GET https://minifetch.com/api/v1/x402/extract/url-metadata?url=https://example.com&includeResponseBody=true
```

#### Success Response (200 OK):
```json
{
  "success": true,
  "metadata": {
    "requestUrl": "https://example.com", // URL you requested
    "url": "https://example.com", // Last URL in request chain, if there were redirects
    "canonical": "",
    "lang": "en",
    "charset": "utf-8",
    "title": "Example Domain",
    "favicons": [],
    // Additional metadata fields, see "Example Metadata" section below
}
```

#### Error Responses:
- 400 Bad Request (Invalid URL):
```json
{
  "success": false,
  "error": "invalid_url",
  "message": "Invalid URL format",
  "url": "https://example.com"
}
```

- 403 Forbidden (Blocked by robots.txt):
```json
{
  "success": false,
  "error": "robots_blocked",
  "message": "URL is blocked by robots.txt",
  "url": "https://example.com/blocked-page"
}
```

- 503 Service Unavailable (Request timeout):
```json
{
  "success": false,
  "error": "fetch_error",
  "message": "Request timed out after 15000ms",
  "url": "https://example.com"
}
```

## Payment
The metadata extraction endpoint requires payment of $0.01 in USDC cryptocurrency on Coinbase's Base network.
From the Minifetch.com homepage you'll be prompted to pay $0.01 USDC cryptocurrency with your browser plugin wallet on Coinbase's Base network each time you click the "Go Fetch!" button, or you can navigate to the API endpoints directly from your browser.
You can pay as you go, there is no Minifetch account setup or monthly fee.

To use the Minifetch API programmatically, you'll need:
- A cryptocurrency wallet with USDC on Base network
- The Minifetch API client library integrated with your application (Coming soon!  Sign up for our waitlist and we'll notify you when its ready: https://forms.gle/rkMi7T23bHJc8XFw9)

For payment implementation details, see the x402 protocol documentation: https://www.x402.org/

## Example Metadata
To see example metadata responses, check these results:
- GitHub Metadata Example: https://minifetch.com/example-github
- Anthropic Metadata Example (with response body): https://minifetch.com/example-anthropic
- Wikipedia Metadata Example: https://minifetch.com/example-wikipedia

## Caching & Rate Limiting
Minifetch implements several measures to ensure responsible and efficient operation:
- Responses are cached to reduce load on target websites
- Crawl delays from robots.txt are strictly observed
- Rate limiting is applied per domain to prevent overloading target websites
- The API uses a proxy pool for making requests

## Libraries & SDKs
Minifetch is based on the open-source url-metadata npm package: https://www.npmjs.com/package/url-metadata

For direct integration with your codebase, consider:
- Implementing the Minifetch API client for API access (Coming soon!  Sign up for our waitlist and we'll notify you when its ready: https://forms.gle/rkMi7T23bHJc8XFw9)
- Using the url-metadata npm package directly in your Node.js applications