What's the first thing that comes to mind when you think about ranking a website?
Content? Or perhaps backlinks?
Both factors are crucial for a website's ranking in search results. But these are not the only ones.
In fact, two other factors play an important role in search engine optimization – crawlability and indexability. Yet most website owners have never even heard of them.
At the same time, even minor indexing or crawlability issues can cause your website to lose its rankings, regardless of how great your content or how many backlinks you have.
Indexability refers to a website's ability to be indexed by search engines. Only indexable pages can appear in search results.
In order to index a website, search engines like Google need to:
Only then can the page appear in the search results.
If web crawlers are unable to do their job and cannot crawl a page on your website, that page cannot be properly indexed. Without indexing, that page on your website cannot attract search traffic because it remains invisible to search engines.
Indexability allows search engines to display your website's pages in the SERPs. Without indexing, no organic search traffic can be directed to your site.
However, it's worth noting that sometimes it makes more sense to make certain pages non-indexable. For example, non-indexable pages are preferred for landing pages, low-quality pages, or "thank you for signing up" pages. After all, you don't want these pages appearing in search results.
However, if you want the URLs to be ranked, you should ensure that these pages are both crawlable and indexable.
To be considered "indexable", the page must meet the following criteria:
For a website to be indexed, search engine crawlers must first be able to access and crawl its content. One element that directly affects a website's "crawlability" is the robots.txt file.
For those who are more familiar with the "technical" side of SEO For those unfamiliar with robots.txt files, a robots.txt file is a small file located in the root directory of your website.
Your task is to give instructions to the web crawlers and tell them which pages of your website should not be crawled. Otherwise, the bots will crawl every page they can find on the website.
There are cases where such "restrictions" make sense. However, you should absolutely ensure that you are not unknowingly blocking the very pages you want to have indexed.
A "noindex" tag is an on-page directive that instructs the search engine not to index a specific page. Essentially, it prevents a page from appearing in the SERPs.
Sometimes it is advisable not to index certain pages – such as landing pages, registration pages, low-quality pages or "thank you" pages – and to keep them "private".
However, if you misuse this instruction or, for example, forget to remove it from the page that is to be indexed, you will lose organic search traffic.
You can crawl your website with Ahrefs Webmaster Tools to ensure that none of the pages you want to index have a "noindex" tag.
A canonical tag (rel="canonical") in a page code defines the primary version for duplicate, near-duplicate, and similar pages. This means that only canonical pages can be indexed, while their non-canonical versions are excluded.
If you do not add the canonical tag to a page, Google relies on signals such as internal and external backlinks, the XML sitemap, and redirects to decide which URL "represents" this cluster in its index.
This means that if you want your page to appear in search results, it must not declare any other page as its canonical version. Good SEO practice is to have self-referencing canonical tags for indexable pages.
Here's what you can do to help Google index your website's pages:
A sitemap is essentially a list (usually in XML format) that contains all the pages of your website. The sitemap tells Google which pages it wants to index on your site.
Before submitting a sitemap, you should check that it reflects the current state of your website and fix any errors, such as broken links, orphaned pages, etc. You can submit your sitemaps to Google using the Google Search Console tool.
If you only want to request the indexing or re-indexing of a few individual URLs, you can use the URL Inspection tool in Google Search Console.
The URL Inspection tool is a diagnostic tool in Google Search Console that provides insights into the version of your website indexed by Google and allows you to test URLs live and view the rendered version of the page. You can also submit individual pages to Google.
If you want to request that a new URL be indexed—or that updated URLs be recrawled and re-indexed—you can ask Google to do so using the "Request indexing" button. If your live URL contains no errors, Google will add your page to the crawl queue.
However, please note that the number of URLs you can request to be indexed per day is limited.
Another way to promote crawling and thus speed up the indexing of the website is internal linking.
When browsing the internet, Google's crawlers follow a path formed by links. So, if one page of your website links to another, the crawlers follow the existing path – and discover new content.
Pages without internal links are called "orphaned pages".
For this reason, it's important to set up internal links to and from your most important pages. Admittedly, this usually happens automatically when you create a well-organized website.
However, it can't hurt to check if there are any orphaned pages on your website – that is, pages that have no internal links pointing to them. You can do this with the Site Audit tool.
In this context, you should ensure that these internal links pointing to indexable pages are not tagged with a "nofollow" tag, as Google will not crawl nofollow links.
It's very simple. Go to Google or another search engine and type in...
website: …and then enter your website address. You should see the results, showing how many pages of your website were indexed.
Most webmasters know that to rank a website well, they need strong and relevant content and backlinks that increase their website's authority. What they don't know is that their efforts are in vain if search engine crawlers can't crawl and index their websites. HTML code plays a crucial role in this.
Therefore, you shouldn't just focus on adding and optimizing pages for relevant keywords and building links, but also constantly monitor whether web crawlers can access your website and report what they find to the search engine. If web crawlers can't crawl the content, it won't be included in the search index. It's therefore crucial to design your website's HTML code in a way that allows it to be optimally processed by search engines.
© 2012-2025, MIK Group GmbH | General Terms and Conditions | Imprint | Privacy policy
