Website Indexing is the IMPORTANT step to get the website or webpage on the search engine. If Google indexes the website, then you are ensured that the website will be seen on the search engine provided rest of the SEO conditions are fulfilled. If the website is not indexed then the chances of getting found on Google are zero. The process of getting ranked on a Search Engine, i.e., SEO – Search Engine Optimization is quite complex, exhaustive and time-consuming and Indexing is the preliminary step to getting found on Google.
It takes a lot of time for indexing by search engines to happen naturally. To avoid this delay, it is necessary for the website to get indexed quickly.
This blog is meant to provide the various steps to get indexed
- The Entire website
- In some cases, only certain web pages – that is when new pages are added to the website.
- There are situations wherein some of the pages do not get indexed quickly. To evaluate the reasons and suggest solutions to enable quick indexing of the webpage.
Let us now get into the process of understanding Indexing and then proceed to further evaluate the steps required to index with the help of Google Search Console.
- How does Google index the webpage or the entire website?
Indexing is preceded by Crawling. Crawling is similar to the method used by a delivery person. The delivery person tries to locate the address based on the address mentioned on the item to be delivered.
In a similar manner, there are millions and millions of websites and Google has its own software also called Googlebot that crawls through these websites. Googlebot crawls through the hyperlink as the hyperlink is the link or address of the website to discover, to find relevant fresh content.
Google crawls the page to evaluate the credentials of the webpage and then accordingly if the webpage is as per the set criteria of Google, it gets indexed. That means the webpage is included in the database of Google for that related category.
The inclusion of the webpage in the database is Indexing.
If I may use the analogy of Index in the textbooks. That is if you check the last few pages of any text books, the index features the display of the important terms and the page numbers and they appear in alphabetical order. This index helps the reader quickly get to that page for the information reader is seeking.
Similarly, Google indexes the web pages so that it can display the relevant webpage for a search query quickly. Hence, it is vital to get the webpage indexed so that Google can then display the webpage quickly for a search query.
Indexing is not the only point that the webpage appears on Google. Ranking on the first page or rather ranking first on the first page is important and that is the eventual end result of a proper SEO process.
Indexing is similar to getting qualified for any important event and ranking is getting the top rank in the event.
As mentioned above, one can say the SEO process can be of three steps and those are Crawling, Indexing, and Ranking.
Thus, Google crawls, indexes the webpage and then displays it for a search query. This process happens in milliseconds.
Google displays results that are relevant to the search query.
- How does Google decide that the page content matches the search query?
Google uses its own software – ranking algorithm to show this result.
- What are the criteria to get the webpage crawled, indexed and then ranked?
The page content has to be unique, relevant and should be fresh. Quality is important as Google will display only those pages that are of good quality.
The following factors cover the major part of the search engine optimization process.
Relevance – The relevance of the webpage for the related search query is a very important factor. There are many web pages that match the search query. Here Google using its ranking algorithm – software – arranges and displays the relevant page in a manner that the page with the highest relevance is shown at the top.
Search results displayed are changing regularly, and some of the websites are able to maintain their ranking. To maintain that ranking, it is necessary to update content on the webpages so that relevancy is intact, freshness in the content shows, and exclusivity of the content helps to improve the engagement of the viewer.
META TAGS – This includes Meta Title and Meta Description – In the Meta Title – To write the primary keyword along with another significant keyword in Meta Title. In the Meta Description – Write page description interspersed with keywords. Importantly the title length and description length should not exceed the requisite number of characters.
Heading TAGS – Assigning H1 to the heading and then subsequently assigning H2 to H6 to other sub-headers in the content of the page.
Image Optimization – Image SEO is vital to improving Core Web Vitals metrics. Large-size images will result in reducing the loading speed of the website. Writing a suitable title and writing the Alt tag is always significant for a good SEO of the image.
- What is to be written in the alt tag of the image for the search engine?
Write the image description in text format in the alt tag. Also, one should reduce the size of the image without affecting the quality of the image. Short Pixel compresses the image without compromising on the quality of the image.
Internal Linking – Internal linking helps in navigating the website for the user as well as for the search engine. Search Engine crawls the website through internal links. Thus, internal linking is a great way to get the website indexed. User clicks on the links on the webpages and browses the webpages through internal linking, it naturally helps in improving the engagement metrics.
Now since all the vital points of SEO are ticked, let us try to figure out how to check whether the website is indexed.
- How to know whether a website or webpage is already indexed?
In the chrome browser, type site:mydomainname and click enter, the result shows as can be seen in the following Image. In the image below, it shows 18 URLs are indexed.
Image1- Google displaying the indexed URLs.
The procedure can be done for a single URL as well to check the status of the URL for indexing. Write the complete URL address in the browser site:mydomainname/URL and you can check the index result. If the page is indexed or if the page is not indexed the result will be displayed as shown in the images below.
Image2- URL is indexed
Image3- URL is not yet indexed
Click here to watch a video from Google about the URL Inspection Tool – Google Search Console Training.
Alternatively, another method to know whether the web page is indexed or not is to check in Google Search Console.
Google Search Console provides detailed information for the pages indexed and not indexed in its dashboard. Google Search Console can be used by signing up for it using the GMAIL address.
Google Search Console and Google Analytics are two super tools that are provided by Google for free. These tools are very helpful to provide valuable data that helps in the SEO process.
Google Analytics provides statistics about visitors, time spent on the website, which pages are being visited, geographical location and much more significant information. Google Search Console helps in submitting the sitemap of the website. The core web vitals metrics are displayed by Google Search Console.
This information can be monitored, analysed, and then accordingly one can make changes wherever required to improve the ranking.
Let us see how Google Search Console is helpful in indexing the website.
Check the image below that shows the number of pages indexed and not indexed in Google Search Console after signing in.
Image4 – Pages indexed and not indexed in Google Search Console
- How to check whether the web page is indexed in Google Search Console?
You can type the URL of the website in the Google Search Console at the top bar with a search icon and click enter. Check the image below.
Image6- Type the URL to be indexed or the domain name in the bar in the marked slot at the top
If the page is indexed it will be shown as in the following image.
Image7- URL is indexed as shown in Google Search Console
However, if the URL is not indexed, it will show the message that the URL is not on Google. Then click on REQUEST INDEXING as shown in the image below–
Image8- Click on Request Indexing if the URL is not on Google
This method is the best practice to implement if a new page is added to the website. It is in a way a message to request Google that a new page is added to the website and to have a look at that page and index it.
Images below show the steps taken by Google to index the pages.
Image9- 1st screenshot seen after clicking on request indexing
Image11- 3rd 2nd screenshot after clicking on request indexing
Further, to get the webpage indexed click on the TEST LIVE URL.
Image12- Click on Test Live URL
The image below displays the screenshot of an URL that has issues and those issues need to be addressed and fixed to get the URLs indexed. In this example the issue of non-responsiveness of the website to the device is the reason for non-indexing – check the image below. The issue can be fixed by making changes accordingly and resubmitting for indexing.
Image14- the screenshot of an URL that has issues
When further inspected Google Search Console also provides the details of the error once clicked on the View Tested Page.
Google Search Console displays the reasons for non-indexing the page. Then these reasons or errors are to be fixed and rectified to get the page indexed. Check the image below.
Image16- Google Search Console displays the errors/reasons for non-indexing of the webpage
Below, I have listed some of the possible reasons (as seen in the image above) that block the indexing along with solutions to repair and make it eligible for indexing.
1 To check the robots.txt file.
One of the reasons can be a crawl block in the robots.txt file.
Let me explain further. Type www.yourdomainname.com/robots.txt in the browser and click enter. The following code is visible.
User-agent: *
Disallow: /
Sitemap: https://www.yourdomainname.com/sitemap_index.xml
The above code is the instruction to Google Crawler to not index pages of the website. The * represents the crawler of the various search engines – Google, Bing and other search engines. There is no need to write User-agent: Googlebot or User-agent: Bingbot and similar other search engine crawlers.
In robots.txt, there is also a provision to block a few pages of the website. In the robots.txt you can write the name of the file or files as shown in the following code.
User-agent: *
Disallow: /page.html
Sitemap: https://www.yourdomainname.com/sitemap_index.xml
If you want the crawler to index the website remove the forward slash (/) in front of Disallow: Check the image below
Image17- showing robots.txt code for indexing of all the webpages
2 Remove the noindex tag and ensure that the page to be indexed does not have a “nofollow” tag.
Apart from robots.txt, there are a few other ways that block the page.
There may be another possibility that must have been the reason for the non-indexing of the webpage and that is due to noindex tag and nofollow tag as shown in the Google Search Console
What is noindex tag?
Noindex tag blocks the page from indexing.
What is a nofollow tag?
In a webpage with a nofollow tag, Googlebot does not index the specific page. It is a message for pages such as contact pages, or privacy page that generally are not required to be indexed and hence one can use noindex, nofollow tag. However, if the page is to be indexed, then remove the “nofollow” tag.
Image18- noindex tag and nofollow tag is shown in the image.
In an html-css website remove the noindex, nofollow tags and upload the website. Type the URL in the Google Search Console in the inspection bar at the top for inspection.
In a WordPress website and if you are using the Yoast plugin for SEO, you can go to the dashboard and click on the edit of the related webpage. Next, go to the advanced section and select Yes for Allow search engine to show this Page in search results and select Yes for Should search engines follow links on this Page. Please refer to the image below.
Image19- Select the “Yes” option for indexing the webpage in the dashboard of the WordPress website.
3 Ensure that the webpage is included in the Sitemap
It is always necessary to include webpage to be indexed in the sitemap and submit the sitemap in the Google Search Console.
Sitemap is the best way to inform Google about the web pages of the website. It is the perfect way for Google to find the web pages of the website. If the website is 5 pages or of 500 pages submitting the sitemap is necessary. In case there is are large number of pages, then Sitemap becomes that much more important and a perfect mode to inform Google about the various pages of the website. Hence, submitting the Sitemap to Google in Search Console will be a solid step in proceeding to index the web pages of the website.
4 Remove the unnecessary canonical tag –
What is a Canonical tag?
Canonical Tag informs Google that a particular page is the preferred version of the page and that has to be indexed. However, sometimes inadvertently or by mistake it happens that a canonical tag indicates to Google the non-existing URL of the page and consequently Google does not index either of the page. Hence it becomes necessary to remove this not required Canonical Tag.
However, it is always necessary to know which is the appropriate version of the page that is to be indexed and then proceed to remove the canonical tag that is not required.
5 Ensure that there is no Orphan Page. Make sure that all pages that are to be indexed are internally linked. A page without any internal links is called an Orphan page on the website. Search Engines will not crawl and Users will also not be able to view the page if it is not internally linked. Internal linking of the page will ensure that Google will be crawling all the pages of the website.
6 How to fix the issue – Page with redirect
- Check and ensure the URL mentioned in the XML sitemap is the version that is to be indexed.
- Check and ensure that the redirect rule is setup accurately
- Check and ensure the source URL and target URL are mentioned as required.
- Check and ensure the Internal linking of the website has the correct URL
- Check and ensure that there is no broken link.
7 Not found (404) error
What is Not found (404) error in Google Search Console?
The related page is not existing and hence Google Search Console is showing this error.
How to fix the Not found (404) error?
Check and ensure that the XML sitemap file is updated if the webpage is deleted from the website. Remove the URL of the deleted page from the XML sitemap and reupload it in Google Search Console.
If you intend to redirect the URL to a new URL, a 301redirect rule has to be created. This can be done in .htaccess in an HTML website and using a plugin if the website is built using CMS such as WordPress, or Drupal.
8 Crawled – currently not indexed – Crawled but currently not indexed means that Google has crawled the page however it has not indexed the page.
How to fix Crawled but currently not indexed?
The specific page can then be tested live for errors if any. If there are errors, they are to be removed and if there are no errors the next step is to request indexing.
9 Discovered – currently not indexed – This means that Google knows that the URL is existing, however, it has not crawled and thus it is not indexed.
How to resolve this issue of discovered but currently not indexed?
Improve the quality of the content. Ensure there is proper internal linking. Google’s primary focus is on the content to improve user experience. Similarly, internal linking helps the crawler to crawl and reach all the pages and get those pages indexed.
Some more PRO TIPS –
Remove the web pages that have no relevance or redirect them to the nearest relevant page. Similarly, build quality backlinks for all the important pages on the website. This will enhance its value and Google will index it faster.
Google ranks for the user experience and this User experience is Page experience. It is the happy experience or fruitful experience of the User and this experience is possible if the core web vitals of the website are appropriate.
Conclusion: It is necessary to emphasize that indexing is not ranking. But indexing is the most important step towards a successful SEO campaign. You can use the above methods to index your website and share your experience or queries in the comments section below.