Content Gap Finder

It crawls a bounded set of pages on the competitor domain you enter — found through their sitemap, or through links on their homepage when no sitemap exists — and does the same on your domain. From each page it reads the title, the H1, the H2 and H3 headings and the meta description, because those are where a page declares what it is about. It then builds a vocabulary of the topics each site publishes and subtracts yours from theirs. The output is the phrases that appear across the competitor's pages and nowhere in the pages we crawled on yours.

No, and it deliberately does not claim to. Knowing what a domain ranks for requires SERP-tracking or clickstream data from a paid provider, and nothing here has that. What this tool observes is what a competitor chose to publish, which is a different and often more actionable thing — publishing intent is visible in their headings, whether or not those pages currently rank. Treat the output as "topics they cover and you do not", because that is precisely what was measured.

Up to 30 per domain by default, with a hard ceiling of 40, and the whole job runs against a wall-clock budget. Pages are discovered from the sitemap where one is available, sampled across the file rather than taken from the top, so a news-ordered sitemap does not give you thirty pages from the same week. The crawl is deliberately small: it is polite to the site being crawled and it keeps the report fast.

Yes. robots.txt is the first thing requested on each domain, and any URL its Disallow rules cover is dropped before it is ever fetched. The crawler identifies itself with a descriptive user agent that names the tool and links back to this site, so any administrator reading their logs can see exactly what visited and why.

The vocabulary is built from real phrases in real headings, so it inherits whatever the competitor actually wrote — including navigation labels and product names. Nothing is cleaned up by inventing better-looking terms, because that would mean showing you something the crawl did not find. Read the list as raw evidence and let the clustered view do the interpreting.

It cannot. The clustering step is only allowed to return phrases copied character for character from the crawled list, and anything it returns that is not in that list is discarded before you see it. It groups and explains; it never contributes a topic.

Not all of them deserve a page. Work down the list and keep the topics that a customer of yours would plausibly search for and that you could genuinely write something useful about. Discard the ones that are competitor-specific product names or industry jargon nobody searches. Then take the survivors into the keyword research tool to see the real autocomplete demand around each one before committing to a brief.

You can crawl any publicly accessible site — the requests are the same read-only GETs a browser makes. Private, local and internal addresses are refused. Comparing two of your own properties is a legitimate use and is a quick way to find topics one site covers that a sister site has never touched.