How Search Engines Work: Crawling

Imagine the entire web as an enormous library. Every webpage is a book, and search engines are the librarians, with Google being the smartest one. The librarians remember every book they have read. Whenever someone asks a question, they use their impressions to recommend books that might help. “Crawling” is the process in which a librarian comes to read your “book.” If the librarian has never seen it, the book will not be recommended, no matter how well it is written.

For example, you generally would not want search engines to see a website that is still under development or a “mirror” site used specifically by engineers to test new features.

The answer is: none of them. That's right. Technically, every example above is a different URL. Although it is uncommon, each one “could” represent a different page. During crawling, search engines therefore treat all seven as separate URLs.

This includes both internal and external links. A crawler can discover new URLs through links on a page and add them to the crawl queue.

Rather than finding new URLs one link at a time and adding them to the queue, a sitemap tells a crawler which URLs exist on a website all at once, making it a more efficient method.

Major search engines such as Google and Bing provide their own webmaster tools, allowing site administrators to submit URLs or sitemaps. Doing this is essentially like telling Google directly: “Hey! I have one URL—or a whole bunch of them—over here. Add it, or them, to your crawl queue!”

The crawler receives what it wanted and moves to the next step.

The crawler is directed to another URL and repeats the same steps. This is known as a redirect.

The crawler does not receive what it wanted because the URL is incorrect. Perhaps the file does not exist on the server, or perhaps the server has determined that the crawler does not have permission to request the page.

The crawler does not receive what it wanted, but this time the problem is with the server, not necessarily the URL. In this situation, the crawler puts the URL back into the crawl queue in Step 1 and tries again later.

In this situation, the crawler's request fails to reach the server for some reason, such as a firewall. The crawler receives no response and does not even know whether anything exists at the URL.

They are HTTP status codes. HTTP is the “language” that crawlers and servers use to communicate. After receiving a request, a server uses a three-digit code to tell the crawler the status of the requested URL.

• “2XX” — Codes beginning with 2 mean that the server can process the request.
• “3XX” — Codes beginning with 3 indicate a redirect.
• “4XX” — Codes beginning with 4 indicate a problem with the request, such as the common 404 error meaning that a page does not exist.
• “5XX” — Codes beginning with 5 indicate that the server encountered an error while processing the request, such as a Carrefour server becoming overloaded when everyone rushes to buy a Nintendo Switch.

If you are an SEO practitioner or web engineer, pay attention to this section. JavaScript makes SEO more complicated. Some links can be clicked by ordinary users but are ignored by search engines. Here are several common examples:

• <a href="/good-link">crawlable</a>
• <a href="javascript: changePage('bad-link')">not crawlable</a>
• <a onclick="changePage('bad-link')">not crawlable</a>
• <span onclick="changePage('bad-link')">not crawlable</span>
• <a href="/good-link" onclick="changePage('good-link')">crawlable</a>

This corresponds to Step 2 in the previous section. For pages that should be visited, the communication channel between the crawler and the server must remain open. This is not hard to understand: if you want a page to appear in search results, it should not be password-protected, placed behind a firewall, or configured to prohibit search engine visits.

By the same logic, we should use appropriate methods to keep crawlers away from pages that we do not want search engines to crawl. Even a search engine as large as Google does not have unlimited resources for crawling the entire web. Proactively telling crawlers which pages they do not need to crawl is a “friendly” gesture that helps them find the pages worth discovering on your site as efficiently as possible.

This corresponds to Step 1 in the previous section. Pages that should be visited need to be discoverable through internal links. Imagine an important page with no links pointing to it. How would Google know it exists? Even if Google did know, it would not consider the page important.

If something exists at a URL, say that it exists; if nothing exists there, say that it does not. With some strange server configurations, a URL may return a page saying that the webpage does not exist while simultaneously returning a successful HTTP code (2XX). In SEO, this situation is called a Soft 404 Error.

I mentioned redirects earlier. Redirects are not inherently wrong, but too many of them can become a problem. Here are two issues redirects can cause:

• Redirect Chain: A crawler has URL A. After visiting it, the crawler is told that the page has moved to URL B. At URL B, it is told that the page has moved to URL C, then D, E, and so on. A few redirects are fine, but after too many, the crawler may simply throw up its hands and stop.
• Redirect Loop: The server at URL A tells the crawler that the content is on page B, while the server at page B tells the crawler that the content is at URL A, creating an infinite loop.

When a crawler is about to visit a URL, tell it that it is not allowed to enter. Webmasters can set up a firewall, use password protection, or define rules for crawlers through robots.txt.

robots.txt is a text file that must be placed in the root directory. For example, this site's robots.txt is located at https://darrenhuang.com/robots.txt. In this file, we can define rules for search engine crawlers and disallow them from visiting certain files.

For example, if robots.txt contains the line Disallow: /secret-weapon, a search engine crawler that sees a URL containing /secret-weapon in its crawl queue will turn around and leave.

Because this is an introductory article, you only need to understand that the robots.txt file can restrict the paths crawlers are allowed to visit. For detailed rules and syntax, see Google's documentation. If you think you already understand robots.txt quite well, try Digital Engine's robots.txt challenge!

We can also tell a crawler, while it processes a page, “Do not place the links you see here into the crawl queue.”

Webmasters can accomplish this with a Robots Meta Tag, an X-Robots-Tag, or the link's rel="nofollow" attribute.

• Robots Meta Tag: Adding <meta name="robots" content="nofollow" /> to a website's <head> section can prevent search engines from adding any links found on that page to the crawl queue.
• X-Robots-Tag: Adding X-Robots-Tag: nofollow to the server response can prevent search engines from adding any links found on that page to the crawl queue.
• The link's rel="nofollow" attribute: Adding the attribute to a link—for example, <a href="/dont-crawl" rel="nofollow">do not crawl</a>—can prevent search engines from adding that particular link to the crawl queue.

In practice, the three methods above are rarely used for this purpose. If you do not want crawlers to visit one of your own pages, you would usually restrict it in robots.txt. nofollow is more closely related to backlinks, but because the directive was so easily misused, Google later began treating it as a hint rather than strictly obeying it.

For a deeper look at the details of this syntax, see these Google documents: “Link rel attributes” and “Robots Meta Tags and X-Robots-Tag.”

First, thank you for reading this far XD. As an SEO, I would love to tell you that crawling is incredibly important and that terrible things will happen if you do not optimize it. In reality, if your website is just a simple blog or company site, there is a good chance you do not need to worry about how search engines work behind the scenes.

As mentioned above, the web is enormous, and crawlers do not spend unlimited resources on any one website. But search engines such as Google are no slouches. They have their own systems for optimizing crawler efficiency—which links to crawl, which not to crawl, how frequently to crawl them, and so on. These are generally not issues that a small website needs to worry about.

Robots.txt can block search engine crawlers, but it cannot block ordinary users. Putting Disallow: /private-page.html in robots.txt will only make people curious about what embarrassing secret might be hiding at that URL. 😉

If you really need to put something private online, password protection is still the best approach!

E-commerce sites often use what is known as faceted navigation. Imagine the men's clothing category page on an e-commerce site, with a panel on the left that lets visitors filter products by “size,” “brand,” “color,” and so on. Every condition you add changes the URL, like this:

• /mens-clothing Men's clothing
• /mens-clothing?size=XL Men's clothing in XL
• /mens-clothing?size=XL&brand=nike Nike men's clothing in XL
• /mens-clothing?size=XL&brand=nike&color=red Red Nike men's clothing in XL

The example above has only three filter criteria—size, brand, and color—but real e-commerce platforms offer many more. Imagine that every criterion has multiple options, some filters can be selected more than once or left unselected, and even the order after the ? can change—for example, ?brand=nike&size=XL vs. ?size=XL&brand=nike... You may already see where this is going. That's right: all those combinations can create an enormous number of URLs for the men's clothing category alone. To a crawler, every one of those URLs is a “separate” URL, and the content on them is not especially strong!

Crawlers discover URLs in three main ways, and manual submission is only one of them. Even if you do not submit a page, Google can still find its link through the other methods. Submission can indeed get a page crawled and indexed more quickly, but it has nothing to do with rankings.

On a healthy website, Google can usually discover a webpage shortly after it is published even if you do not submit the URL, so there is no particular need to do so. If you find that new articles consistently take several days to be indexed, you can try the submission tool. More importantly, though, that pattern is a reason to check whether the website has another problem.

No. Whether a webpage's URL is https://example.com/seo-friendly or https://example.com/?p=3828482, it is still just a URL. An ugly URL does not affect the crawler's ability to crawl it. Other debates—whether a URL ends in .html, contains Chinese characters, or uses a hyphen “-” instead of an underscore “_”—do not matter here either. A URL is a URL, and its format does not affect search engine crawling. Whether a pretty URL affects rankings is beyond the scope of this article!

Fun fact: The previous paragraph is not entirely correct. One HTTP status code is 414 URI Too Long, which means that the URL is too long for the server or crawler to process. The limit can be as high as 2,000 characters, however, so it is not something a normal website needs to worry about. (Stack Overflow discussion)