skip to Main Content
Digital Marketing

Once a page on your website has reached a certain age, you probably don’t care much about how often it gets crawled or indexed. However, this is certainly something you should be thinking about, no matter the age of a certain page. You need to ensure your page can be crawled and indexed because this is what shows up in search engines. If there is an issue here, then your website might not even show up!

There are obviously a lot of different reasons why a webpage is not ranking the way you hoped and one of the reasons (which is actually more common then you think) is that there is a crawl error. Google (and other search engines) have a lot of pages in their index and a lot more that they need to re-index, so you need to ensure you are staying competitive and optimising for indexation.

Have a look below at some of the best tips to help you optimise the pages on your website to improve craw speed and indexation, this will in turn hopefully ensure you get the ranking you deserve. If these tactics fail, try getting help from a Sydney SEO firm.


Check Crawl Status

The first place you should look is at your crawl status with Google Search Console. This is recommended to be done every month, because an issue here could be the result of an even bigger issue. When you are on your Google Search Console, go to the “Coverage” section under “Index”. Here you will find all the errors on your website, including those pages with a crawl issue.

Other common issues that you could find here include; server errors, 404 errors, robots.txt errors and DNS errors. Click on one of the errors to see the exact pages that are being impacted by this. It is important that you resolve as many of these issues as you can to ensure your site is indexed and crawled properly and in a timely fashion.


Submit a Sitemap

To ensure your pages get indexed, it is a good idea to submit a sitemap to each search engine (i.e. to Google Search Console and Bing Webmaster Tools). If you are having difficulty creating your sitemap, the good news is that there are a lot of different sitemap generator tools to assist you!


Content Updates


By updating the content on your existing pages regularly you are showing to search engines that there is something new on this page for them to look at. Search engines are more likely to crawl your website at a higher rate if there is a continuous stream of new content.

Search engines love new content and it is one of the most well known ranking factors. By producing content regularly, it sends the message that your website is improving and evolving with the times and therefore should be indexed and crawled more.


Noindex Tags Pages

Sometimes it does make sense to have a few pages with the “noindex” tag. However, you should be careful about who you say cannot index the page. People usually don’t like scrapping tools like Screaming Frog to be able to access their site because it slows the overall speed down.

There is good news if you are using WordPress, there is a Yoast plugin which makes it easy to switch between index and noindex of a page. If not, you can always perform this action manually when looking at the backend of your website.


Remove Duplicate Content

Having duplicate content on your website is an issue for a number of different reasons, one of them being indexation and crawling. When you have a large amount of duplicate content on your website it slows down the process of crawling and negatively impact s your crawl budget.

So if you have lots of pages that are similar it can be a good idea to have some “noindex” tags put on, which means you can have only the most important pages indexed. As well as this, it could be a good idea to take a look at the meta data for each page which ensures search engines don’t mistake the pages as duplicates.


Mobile Friendly Pages

By now you should already know about Google’s mobile first indexing. Therefore by creating pages that are mobile friendly you will be helping to get your page indexed faster. Some of the things you should think about doing to get your webpage mobile friendly include; ensuring responsive web design, tagging pages with AMP, compressing videos and images, limiting the number of UI elements.

It is important that once you are happy with your page, you test it using Google’s very own Mobile Friendly Test tool. This tool will give you the yes or no for mobile friendliness, as well as giving you pointers to help make your page mobile friendly (if it isn’t already).