In the coming weeks I will cover website page indexing. If you are following along on the free website audit template, this check is on tab 2 labeled crawling/indexing.
This week we will dive into that annoying “crawled – currently not indexed” message that you might be seeing in Google Search Console.
If you are not checking in GSC for indexing – you should! 😳 This should be done along side a review of your GSC crawl stats to help you understand how Google is crawling and indexing your website.
When you are in Google Search Console you will want to click on PAGES in the left sidebar. This will give you a snapshot of your page indexing status.
Once you land on the page you may see something similar to the image below. The purple arrows show the areas you should look at. The top left is a dropdown and the bottom right arrows reflects the number of crawled – currently not indexed pages I have.
You can see below that I have 793 pages not indexed and 500 indexed.
But this is not necessarily something to worry about.
Let’s take a look at the top left dropdown. In there are 3 options:
- “all known pages” is the default and will show all URLs that Google has discovered, whether you submitted them or not
- “all submitted pages” are the pages you submit via a sitemap
- “unsubmitted pages only” are URLs not in the sitemap that Google discovered in other ways like links
You will notice that the total of not indexed pages for all submitted and unsubmitted will equal the all known pages value!
In the example below – under the all known pages filter we can see 319 pages that have been crawled but are currently not indexed.
If I change that dropdown to all submitted pages, there are only 9.
For this tutorial, click the dropdown in the upper left and select “all submitted pages”. This will allow you to see the pages you probably want indexed but aren’t.
Then click “crawled – currently not indexed”.
Now you will see a list of the pages that have been crawled but Google is saying are not indexed.
Below are my pages that have this error.
From here you will want to do a couple of checks to confirm that GSC is reporting these pages correctly. GSC can sometimes give incorrect information, so you want to be sure these pages are in fact not indexed.
Here are a few steps to check.
Check 1
Go to Google search and enter site:bluecollarbrain.com/program/automotive/ (of course you will use YOUR url) in the search box. this will show you if the page is indexed or not.
As you can see below, the page is not indexed as noted in GSC
If you find your URL is indexed, then you can ignore that URL and move on to the next.
It will eventually remove itself from the report.
Check 2
If you find the page is not indexed using the site:url check, then click the magnifying glass that shows when you hover over the URL.
This will inspect the URL to provide more information. After further inspection I can get clarification that the page is not indexed.
Before I do anything else, I will go to the page to ensure that there is content and it has value.
If Google is not indexing it – there might be a reason.
- Can you add any content, elements, or images to make it better?
- Is it being seen as low quality due to lack of sources, data, etc.?
- Do you have any internal links to the page from other pages?
- Are there other pages that have similar content?
- Are you publishing too much content at one time?
- Is the page overly promotional or spammy?
- Do you have technical issues that are preventing rendering?
If you feel confident it should be indexed, then click the Test Live URL button to make sure there are no issues that would prevent indexing. In my case all looks good.
At this point, if you feel confident your page should be indexed, go ahead and click the REQUEST INDEXING text.
Keep track in a spreadsheet and check back in a day or two!
UPDATE 4/22: This page was finally indexed after doing a submit request.
Now go back to the crawled – currently not indexed area in pages and repeat as needed.
If you would like, you can also change the upper left dropdown to all known pages. This will show a lot more pages but can help you see if Google is picking up pages they should not be.
Below are some common pages you might see that you do not need to worry about.
These are typically pages you would not want to be indexed.
- URLs with /feed/ on the end
- URLs with /sitemap.xml
- URLs with utm tracking appended
- URLs with dates in them, Google may realize it is outdated content
- URLs with /search in them
- URLs with /tag/ or /categories/ in them – if you want these indexed you need to offer value
Checking your page indexing should be part of your workflow. I try to do a review once a month just to make sure I do not find any unexpected issues.