User FAQ

Trying out Minifetch?

Here's what you need to know.

What does Minifetch actually do?

Minifetch is a hosted service that extracts data from URLs. Users choose either rich, structured metadata or LLM-ready, token-efficient content summaries as markdown. Users pay per-URL with x402 micropayments, typically for SEO research or AI applications. There is no Minifetch account setup or monthly fee.

Metadata extraction samples ($0.002 each):

Content summary extraction samples ($0.002 each):

What does "We ❤️ robots.txt 🤖" mean?

Websites can publish a robots.txt file that says "don't access these pages." We obey those rules. This means some pages might return "blocked by robots.txt" instead of data - but it also means we're operating ethically and transparently by respecting site owners' intellectual property.

For our users, we offer a free "Check URL" button to see if a URL is likely accessible or blocked by robots.txt before spending credits to fetch it.

What does the "Check URL" button do?

For our users, we offer a free "Check URL" button to see if a URL is likely accessible or blocked by robots.txt before spending credits to fetch it.

We cannot guarantee a URL will be accessible just based on robots.txt. A site may block access for many reasons. Checking robots.txt first is a reasonable step before fetching to ensure success.

How do I pay for a "Go Fetch!"?

The "Go Fetch!" button on the homepage extracts the data from the URL you enter into the search box. When you click "Go Fetch!" you'll be prompted to use your browser plugin wallet to pay in USDC cryptocurrency on Coinbase's Base network to pay. You can pay as you go, there is no Minifetch account setup or monthly fee. This service uses Coinbase's x402 protocol to facilitate these payments.

Are there API Docs?

Full API documentation can be found here.

We also provide API docs in LLMs.txt format here for use by LLMs.

What about "noindex/nofollow" robots meta tags on individual pages?

We don't process those. Here's why:

Robots meta tags in html pages tell search engines what to index
We're not a search engine - we're an on-demand fetch service
We work like a browser visiting a page, not a crawler indexing your site
We respect robots.txt (crawling rules), but page-level noindex/nofollow meta tags control indexing, which we don't do

Can I check shortened URLs (bit.ly, t.co, etc.)?

Not directly. The "Check URL" button checks robots.txt for the URL you enter, so bit.ly links will check bit.ly's rules, not the final destination page.

Solution: Expand the shortened URL first (paste it in your browser, copy the final URL), then use the final URL in Minifetch.

How is Minifetch different from other scraping services?

We always respect robots.txt (no sneaking around blocks)
We respect rate-limits and cache to minimize server load
We identify ourselves clearly with user-agent strings & ip address combo
We use pay-per-request to prevent spam and abuse
We respond to site owner & user feedback

Can I monetize my content through Minifetch?

Yes! If you have premium content or APIs, we can work with you to:

Set custom pricing for your URLs (via x402 protocol)
Share revenue when users access your content through Minifetch
Control access levels and pricing tiers

This lets site owners monetize access without managing payment infrastructure. Contact us thru our feedback form to discuss partnership opportunities.

Is Minifetch open-source?

Yes, Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the community for 9+ years.

During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.

3.3 Million+ downloads over 9 years!
Docs are here. Add to your project:

npm install url-metadata --save

Can I contact Minifetch?

Sure, just use our feedback form and we'll get back to you.