Site Owner FAQ

Noticed Minifetch.com in your logs?

Here's what you need to know.

How do I verify it's really Minifetch.com?

Real Minifetch.com requests come from the following IP address: 138.68.38.1
AND we identify ourselves with this user-agent: minifetch/1.0 (+https://minifetch.com/site-owner-faq)
User-agents can be spoofed, so the IP address is your best verification.

Does Minifetch respect my robots.txt?

Yes, absolutely. We honor all robots.txt directives including crawl-delay and disallow rules. We follow both wildcard (*) rules that apply to all bots AND any specific rules for our user agent. If you don't want us visiting something on your site, you can either:

- Block all robots (including us):
User-agent: *
Disallow: /my-private-page

- Block only minifetch:
User-agent: minifetch
Disallow: /my-private-page

For our users, we offer a free "Check URL" button to see if a URL is accessible or blocked by robots.txt before spending credits to fetch it. We cache that response for a while so our users aren't abusing your robots.txt file, either.

What does Minifetch actually do?

Minifetch is a hosted service that extracts HTML metadata from web pages (title, description, meta tags, Open Graph tags, JSON-LD, headings, image urls, etc). There is also an option to return the entire scraped content of the page's HTML. Users pay undefined per-URL, typically for SEO research or AI applications. There is no Minifetch account setup or monthly fee.

Sample metadata:

How is Minifetch different from other scraping services?

We always respect robots.txt (no sneaking around blocks)
Rate-limit and cache to minimize server load
Identify ourselves clearly in user-agent strings
Use pay-per-request to prevent spam and abuse
Respond to site owner & user feedback

What about "noindex/nofollow" robots meta tags on individual pages?

We don't process those. Here's why:

Robots meta tags in html pages tell search engines what to index
We're not a search engine - we're an on-demand fetch service
We work like a browser visiting a page, not a crawler indexing your site
We respect robots.txt (crawling rules), but page-level noindex/nofollow meta tags control indexing, which we don't do

Can I monetize my content through Minifetch?

Yes! If you have premium content or APIs, we can work with you to:

Set custom pricing for your URLs (via x402 protocol)
Share revenue when users access your content through Minifetch
Control access levels and pricing tiers

This lets site owners monetize metadata access without managing payment infrastructure. Contact us thru our Feedback form to discuss partnership opportunities.

Is Minifetch open-source?

Yes, Minifetch.com is the hosted version of a free and open-source npm package called url-metadata that has been serving the community for 9+ years.

During that time some users expressed a preference for a hosted service for various reasons, mostly technical hurdles with coding or accidentally triggering blocks from websites when using the open-source package.

3.3 Million+ downloads over 9 years!
Docs are here. Add to your project:

npm install url-metadata --save

Can I contact Minifetch?

Sure, just use our feedback form and we'll get back to you.