Minifetch provides composable extraction APIs for humans and AI Agents-- making web pages simple to access. Typically for SEO research and AI projects such as AI Agents, RAG pipelines, and building training datasets. The API follows ethical scraping practices, respecting robots.txt and using rate limiting to avoid overloading target websites.
Websites can publish a robots.txt file that says "don't access these pages." We obey those rules. This means Minifetch might return an error that says "blocked by robots.txt", or a simple 502 status error instead of data to indicate your target URL was explicitly blocked. It also means we're operating ethically and transparently by respecting site owners' intellectual property. We do not charge users for explicitly blocked pages like this.
We also offer a free "Check URL" button (and API endpoint) to see if a URL is likely accessible or blocked by robots.txt before spending credits to fetch it. For full policies, check our Terms of Service.
In the browser, we offer a free "Check URL" button (and API endpoint) to see if a URL is likely accessible or blocked by robots.txt before spending credits to fetch it.
We cannot guarantee a URL will be accessible just based on robots.txt. A site may block access for many reasons. Checking robots.txt first is a reasonable step before fetching to ensure success.
In a browser, the "Go Fetch!" button on the homepage extracts data from the target URL you enter into the search box.
If you are signed in to your account, a successful "Go Fetch!" will be deducted from your credit balance. You can sign up for an account and get started for free by visiting the dashboard.
If you do not wish to sign up for an account, you will be prompted to use your browser plugin wallet to pay in USDC cryptocurrency on Coinbase's Base network or Solana. This service uses Coinbase's x402 protocol to facilitate these payments.
Either way you pay as you go. There is no Minifetch account setup or monthly fee. We do not charge for blocked pages or pages that error.
Full API documentation can be found here. We also provide API docs in LLMs.txt format here.
For coders, the Minifetch API client has a good README for getting setup quickly in your Javascript/ Typescript project.
We don't process those. Here's why:
Not directly. The "Check URL" button checks robots.txt for the URL you enter, so bit.ly links will check bit.ly's rules, not the final destination page.
Solution: Expand the shortened URL first (paste it in your browser, copy the final URL), then use the final URL in Minifetch.
Minifetch only extracts publicly available metadata and content from pages accessible without authentication and Javascript execution.
What Minifetch does NOT do:
What Minifetch does NOT do currently but may offer in the future as an add-on:
Yes! If you have premium content or APIs, we can work with you to:
This lets site owners monetize access without managing payment infrastructure. Contact us thru our feedback form to discuss partnership opportunities.
Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the NPM community for 10+ years. During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.
minifetch-apiNEW — the official Minifetch API client for Javascript/ Typescript
npm install minifetch-api --save
url-metadata — the open-source npm package (10+ years, 3.6M downloads) that powers Minifetch
npm install url-metadata --save
3.6 Million+ downloads over 10 years.
Add it to your project.
If you have questions or need assistance:
join our Discord server.
To leave feedback or join the bulk credits waitlist:
use our feedback form.
While you're at it, give us a follow on X.com (Twitter):
@minifetch