Site Owner FAQ

How do I verify it's really Minifetch.com?

Real Minifetch.com requests come from the following IP address: 138.68.38.1 AND we identify ourselves with this user-agent: minifetch/1.0 (+https://minifetch.com/site-owner-faq)
User-agents can be spoofed, so the IP address is your best verification.

Does Minifetch respect my robots.txt?

Yes, absolutely. We honor all robots.txt directives including crawl-delay and disallow rules. We follow both wildcard (*) rules that apply to all bots AND any specific rules for our user agent.

We offer a free "Check URL" button (and API endpoint) for users to see if a URL is accessible or blocked by robots.txt before spending credits to fetch it. Website owners can also use this feature to check if Minifetch is allowed to fetch pages on their site(s). We cache that response so our users aren't abusing your robots.txt file, either.

If you don't want us fetching something on your site, you can either:

- Block all robots (including us):
User-agent: *
Disallow: /my-private-page
- Block only minifetch:
User-agent: minifetch
Disallow: /my-private-page
- To explicitly allow minifetch while blocking other bots:
User-agent: minifetch
Allow: /

User-agent: *
Disallow: /
- Or to allow us with a crawl delay:
User-agent: minifetch
Crawl-delay: 5
Allow: /
Why allow Minifetch?

• Compliance-first: We check your robots.txt before accessing your content
• Transparent identity: We clearly identify ourselves via user agent
• Cost barrier: Micropayments deter abuse and spam
• Caching: We cache results to minimize repeat requests

What does Minifetch actually do?

Minifetch provides composable extraction APIs for humans and AI Agents-- making web pages simple to access. Typically for SEO research and AI projects such as AI Agents, RAG pipelines, and building training datasets. The API follows ethical scraping practices, respecting robots.txt and using rate limiting to avoid overloading target websites.

Explore the API →

Install the Minifetch API client →

How is Minifetch different from other scraping services?

• We always respect robots.txt (no sneaking around blocks)
• Rate-limit and cache to minimize server load
• Identify ourselves clearly in user-agent strings
• Use pay-per-request to prevent spam and abuse
• Respond to site owner & user feedback

What are Minifetch's service limitations?

Minifetch only extracts publicly available metadata and content from pages accessible without authentication and Javascript execution.

What Minifetch does NOT do:

• Ignore robots.txt directives
• Access authenticated or logged-in content
• Create accounts or log into user sessions
• Perform transactional actions (checkout, bidding, purchasing, form submissions)
• Bypass paywalls or access restricted content

What Minifetch does NOT do currently but may offer in the future as an add-on:

• Javascript execution

What about "noindex/nofollow" robots meta tags on individual pages?

We don't process those. Here's why:

• Robots meta tags in html pages tell search engines what to index
• We're not a search engine - we're an on-demand fetch service
• We work like a browser visiting a page, not a crawler indexing your site
• We respect robots.txt (crawling rules), but page-level noindex/nofollow meta tags control indexing, which we don't do

Can I monetize my content through Minifetch?

Yes! If you have premium content or APIs, we can work with you to:

• Set custom pricing for your URLs (via x402 protocol)
• Share revenue when users access your content through Minifetch
• Control access levels and pricing tiers

This lets site owners monetize access without managing payment infrastructure. Contact us thru our Feedback form to discuss partnership opportunities.

Is Minifetch open-source?

Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the NPM community for 10+ years. During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.

Can I contact Minifetch?

If you have questions or need assistance:
join our Discord server.

To leave feedback or join the bulk credits waitlist:
use our feedback form.

While you're at it, give us a follow on X.com (Twitter):
@minifetch