• Sorry, but if you want audience growth, you need to take technical SEO seriously
• It’s not that bad — there are just four key questions 
• I. Is your site being crawled properly?
• 2. Is all your content getting indexed
• 3. Does your site have an efficient structure?
• 4. Do users have a positive experience?
• A checklist to get you started

This guide will show how free tools can help spot problems in your site’s technical structure, limiting your appearance on Google Search Results Pages. This is not a technical manual, simply a guide for non-specialists who want to know if they’ve got a problem with Google and what to ask development teams to look into. 

You can’t have a smart content strategy without good technical SEO

The teams I work with tend to ignore technical SEO completely. I understand the reasons — the topic is nerdy and full of jargon, involves spreadsheets and requires more of an engineering than a content mindset. 

There is nothing in there of natural appeal to content producers. And the SEO industry does very little to make it more palatable. However, there are other ways of looking at the topic. 

I’m rather late to this party; I spent two decades running websites before I started to dabble in SEO. AI is changing the world of Search very rapidly, but without a reasonable feel for technical SEO, you can’t be sure of the following: 

1. Traffic from search is as strong as you deserve. 
2. Your content strategy is smart and consistent
3. Your website meets the minimum expectations of users.

When you think about it this way, technical SEO becomes more interesting to non-technical thinkers.

The four questions of technical SEO

In essence, you need very good answers to the following four questions: 

1. Discovery: How completely does Google crawl my site? 
2. Visibility: Does Google index all the content I want to be seen? 
3. Structure: Is my site set up so that it helps both Google’s bots and users?
4. User Experience: Am I being ignored because my site is slow and clunky? 

The primary tool for answering these questions is Google Search Console. This free app helps site owners monitor the technical aspects of Search Engine Optimisation and draw up keyword strategies. 

To get started, you will need to ask the verified owner of your site if it is connected to GSC and, if not, request the connection. They will also need to authorise your access to GSC. 

1. Discovery: How to check your site is being crawled properly

Google’s automated “bots” crawl the web, following links from page to page and discovering new and updated content. Google then indexes that content using its automated cataloguing system. 

So, if everything’s automated, why worry? 

While Google’s bots will automatically crawl sites, not all sites are crawled equally effectively. Smaller sites (fewer than 100 pages) are particularly at risk.  

Crawl Budget Limitations: Google doesn’t crawl every page on every site instantly. Each website has a limited “crawl budget”. Bots allocate limited time and resources to crawl your site. Complex or poorly structured sites can waste this budget.

Discoverability Issues: Not all pages are automatically found. Complicated navigation structures can hide pages. And orphaned pages with no internal links might never get discovered. 

Tests to check if you’re properly set up for Google’s crawlers

Are you using robots.txt correctly?

The robots.txt file is what the crawlers look at before they start. It guides the bots on what needs crawling and what it should ignore. Smaller sites tend to have a smaller ‘crawl budget’, which means the bots don’t spend much time looking for your content. So, it’s important to check the file is set up correctly.

1. Check if you have a robots.txt file by typing your domain address plus ‘/robots.txt’ into a browser. 
2. Most robots.txt files are very short, and it’s a good idea to signal to Google what’s most important by adding ‘Allow’ directions, as I have for runciblecontent.com: 

User-agent: *

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

Disallow: /wp-content/uploads/wpforms/

Allow: /blog/

Allow: /about-runcible/

Allow: /contact-information/

Allow: /insights/

Allow: /strategy/

Allow: /audiencedevelopment/

Allow: /digital-content-skills/

Sitemap: https://runciblecontent.com/sitemap.xml

Sitemap: https://runciblecontent.com/sitemap.rss

If your site is built on a modular content management system, you may be able to do this yourself. Otherwise, ask your dev team to help here.

Visibility: Have you got any Google crawling issues?

Go to ‘Settings’ in GSC and then select ‘Crawl Stats’. Don’t be put off by all the jargon; you just need to do a couple of visual checks:

What’s the trend in total crawl requests?

There will be bumps associated with new content if you are not a regular poster, but, in general, is this stable or, preferably, rising? 

What’s the trend in Average Response Time?

A good number here would be under 200ms, and if the trend is falling, it is even better, as it indicates that Google is finding your site easier to crawl.  

You want to see rising crawl stats indicating growing interest by Google in your content and a low, stable or falling response rate, indicating Google finds it easy to index your site. Anything over 2 seconds, and you’ve got a real problem.

How to fix crawling issues

You may need to tighten up your robots.txt file by disallowing the parts of your site that don’t need indexing. Or there may be a more fundamental problem with site speed and structure.

2. Have you got any Google indexing issues?

Googlebot looks at a series of factors when it indexes pages: Content quality, Relevance, Keywords, Site structure, and User experience signals. But just because Google crawls a page doesn’t mean it indexes it. There are several reasons why this happens.

Excluded by ‘no index’ tag. This means someone, quite possibly a producer using the Content Management System, has specifically requested that the page not be indexed. Sometimes this happens by accident so it is worth checking pages in this category.

Not found 404. There are several reasons for 404 errors being generated:

• Deleted Pages: The page was intentionally removed, but other sites or internal links still reference it.
• Broken Links: Links pointing to a page that doesn’t exist (e.g., typos in URLs or outdated links).
• Moved Pages Without Redirects: Pages moved to a new URL without setting up a proper 301 redirect.
• Incorrect URLs: Visitors or crawlers accessing incorrect or misspelt URLs.
• Temporary Errors: Sometimes, something slips at Google’s end.

All these are wasteful of the crawl budget, and it is worth periodically tidying things up.

Crawled – currently not indexed. This is the category you should pay the most attention to. Googlebot has crawled the page, fetched its content, and analyzed it but has decided not to include it in the search index. This happens for several reasons, typically related to the page’s content, technical setup, or perceived importance.

1. Low-quality, particularly short, stale or unoriginal content. Solution: Add value to the page by improving its content quality, length, and relevance. And update outdated content with fresh, accurate information.

2. Low Perceived Value: Google may judge that the page doesn’t offer significant value or answer search queries effectively. This may be because people who find the page don’t spend long on it or because there are few external links. 

Solution: Optimise the page for user intent, targeting specific keywords and ensuring it satisfies common search queries. Build backlinks and improve internal linking to signal the page’s importance.

3. Crawl Prioritization. Google might decide not to index the page because it considers other pages on your site more important or relevant. If your website has many pages and a limited crawl budget, some pages may be crawled but not indexed. Solution: Improve your site’s overall quality signals to increase your crawl budget. Prioritize important pages in your sitemap and internal linking structure.

4. Technical Issues: Pages with slow server response times may not be considered for indexing. Critical resources (e.g., CSS or JavaScript) are blocked from Googlebot, making it difficult to render the page properly. Solution: Improve page load times and ensure critical resources are accessible to Googlebot.

5. Spam or Over-Optimization Signals: Pages with spammy content or keyword stuffing may be flagged as low-quality. Aggressive optimization techniques (e.g., excessive use of keywords) can reduce trustworthiness. Solution: Ensure all pages adhere to Google’s spam guidelines.

6. Delayed Indexing Decision: New or recently updated pages are sometimes subject to delays. Solution: Submit the URL via the URL Inspection Tool in GCS to request expedited crawling and indexing.

3. How to analyse your site structure

One of the most fundamental factors Google looks at is your site structure. This determines how easy it is to crawl and helps Google understand what you think are the most important areas of the site and, crucially, whether you have depth of expertise around topics. We’re talking here about things like your navigation structure, the depth of your site, and the pattern of internal links. 

How to visualise your site structure

To get a visual fix, try using Sitebulb, free for small sites, and look at the ‘crawl map’. Here’s the one for Runcible Content: 

Crawl Map Analysis

I’m not pretending this site is the last word in optimisation but it does have the following merits:

Narrow — GSC shows that just 40 pages have been indexed (32 visible on this map), meaning I’m not expecting the Googlebot to do much.

Shallow — Depth is just three nodes, represented by three colours. No page is more than two clicks away from the homepage. The problem with deep sites is that you may find Google loses interest at a depth of 4 levels or above. 

Clustered — The labels show groups of pages around ‘Insights’ (my blog) and three themes: Audience Growth, Content Strategy, and Digital Skills Training. I’m following what many SEO consultants regard as best practices: group related content into clusters around the topics you want to be known for.

Organisations tend to prioritise information about the organisation rather than what it can do to answer users’ questions; crawl maps can help you see if you have fallen into this trap. Google strongly recommends that you put users’ interests first because that, after all, is what users want.

The importance of an XML sitemap

The Googlebot uses your XML sitemap to understand the structure of your site. It isn’t the end of the world if you don’t have one. But it will mean that pages that don’t have many links (internal or external) will tend to get ignored. 

Have you got an XML sitemap?

Just type ‘sitemap.xml’ after your site URL to see. If you think some key things are missing, ask the dev team to help you create and upload a more complete one.

Are you linking efficiently within your site?

In GSC click on ‘links’ and look at ‘internal links’. This will show you how many internal links each page has. If a page has none, then the risk is it will not be crawled and never show on a Google Results Page. 

There are lots of reasons why internal linking is important but Google advises that links should feel ‘natural’ and warns site owners not to over-link. A minimum of a couple is a good benchmark for smaller sites. 

A much-favoured content tactic is to create ‘content clusters’, which consist of a topic page and several sub-topic pages linking back and forth. You would aim for at least five inward links for topic pages. 

4. How to check you meet users’ minimum expectations

Even though we are notoriously impatient online, we tend to give our slow-loading websites a free pass. And mobile sites are a particular blindspot. 

Google thinks mobile speed is one of the most critical aspects of user experience. So you must take this seriously — most sites appear to meet the standards. 

Google recommends that displaying your web pages take at most 2.5 seconds. This is partly based on research that shows that 53% of mobile users abandon sites that take longer than 3 seconds to load.

Google’s Core Web Vitals tests

To test your mobile speed, use PageSpeed Insights and look at ‘Performance’. Google uses data on how users experience your site, not just the homepage, and 

the scorecard gives the results of three tests you need to pass the CWV tests:  

Largest Contentful Paint (LCP): The time it takes for your webpage’s most important part to appear. To pass, you need a figure of 2.5 seconds or lower.

Interaction to Next Paint (INP): An average measure of how quickly your site responds to all interactions, e.g. clicking on a dropdown menu, scrolling, or tapping a button. A good measure would be under 200ms.

Cumulative Layout Shift (CLS): How much the page content moves around while loading, e.g. ads or images suddenly pushing text around after the page loads.

What are the major causes of speed issues?

Two challenges affect the majority of sites: 

1. Image Problems affect 70-75% of sites and directly impact LCP. Oversized banners and heavy carousels are often the culprits. You can fix these yourself.
2. JavaScript and Render-Blocking Resources affect 60-65% of sites and will need help from your development team. The detailed diagnostics in the GSC performance tab provide a prioritised list of what needs fixing. 

The other Core Web Vitals tests are also important, and many sites fail the accessibility tests. These generally require technical remedies.