SEO, SEM, social media, GEO, content & website services · Six focused SEO applications.Help & support ↗
Technical SEO Answer library

What is crawling in SEO?

Coverage problems are often described as indexing problems, but the cause is earlier. A page that is blocked, orphaned or rendered empty never reaches the index. Diagnosing the crawl stage first saves weeks of content rewrites.

Reviewed 3 min read By Aver, founder and SEO lead

The essentials

Discovery, fetching and rendering are separate steps and can fail independently.

  • Robots.txt blocks fetching but does not remove an already indexed URL.
  • Internal links are still the main way crawlers find new pages.

How it works in practice

This is how we would act on “What is crawling in SEO” in practice, starting with the checks that block everything else.

  • Check robots.txt and meta robots for the URL and its template.
  • Confirm the URL is linked from at least one crawlable internal page.
  • Use URL Inspection to see the crawled and rendered HTML.
  • For large sites, analyse server logs to see what crawlers actually request.

What to measure

Measure “What is crawling in SEO” with one first-party source per number, and record the baseline date.

  • Crawl requests by response code in the server log.
  • Pages discovered but not indexed in Search Console.
  • Time between publication and first crawl for new pages.

Where teams go wrong

None of these “What is crawling in SEO” mistakes looks dramatic on the day it is made; the cost appears later.

  • Disallowing a page in robots.txt and expecting it to be removed from the index.
  • Relying on the sitemap while leaving important pages orphaned.
  • Changing crawl rules during a migration without testing.

Where this fits

This page belongs to the Technical SEO section of the Answers library and answers “What is crawling in SEO” on its own.

Frequently asked questions

What is the difference between crawling and indexing?
Crawling is fetching the content; indexing is storing and understanding it so it can be shown for queries. A crawled page can still be excluded from the index, and a blocked page cannot be crawled at all.
How do I improve crawl budget?
Remove or block low-value URLs that crawlers waste time on, fix slow responses and server errors, keep sitemaps accurate and make important pages reachable within a few clicks. On small sites, crawl budget is rarely the real problem.
Does Google crawl every page on my site?
No. It prioritises by perceived importance, freshness and internal links, and it may leave infinite filters or duplicate parameter URLs uncrawled. That is normal; the goal is to make the pages you care about easy to reach.

Sources we checked

These are primary or first-party documents. Where the page describes our own method, it is marked as our experience rather than presented as an official rule.

Something inaccurate or out of date? Tell us and we will correct it.