Discover how to identify and correct common technical SEO problems on your website.
A guide to finding and fixing technical SEO issues, including soft 404s, redirect chains, robots.txt, noindex, canonical tags, and duplicate titles.
Key Takeaways
- Technical SEO affects how search engines crawl, index, and serve pages, but fixing issues only improves ranking chances rather than guaranteeing first place.
- Common problems include missing canonical tags, duplicate titles, absent meta descriptions, broken links, JavaScript-heavy pages, and robots.txt or noindex blocking.
- Soft 404s and redirect chains are important technical issues that can mislead search engines even when a page appears normal to users.
- The article uses a local practice site and tools like curl or browser developer tools to help readers inspect HTTP responses and identify real technical SEO problems.
Meta description: Learn how to find and fix common technical SEO problems on a practice site, including soft 404s, redirect chains, robots.txt, noindex, canonical tags and titles.
Tags: technical SEO, site audit, robots.txt, canonical tag
Featured image: Abstract friendly illustration of a website page with a magnifying glass, arrows, tags and a small shoe icon, no logos, no text, no real people.
This tutorial is for site owners, developers and editors who want to find technical SEO problems that keep search engines from reaching or reading their pages. It uses a small practice site so you can see each issue in a real page response, then check the same areas on your own site.
What technical SEO actually controls
Google uses crawlers to discover addresses on the internet. When a crawler finds a page, it can crawl the page and process its text, images and videos. If Google decides the page is worth indexing, it may add the page to the index, the large database of web pages that Google can serve in search results when a query is relevant. Google's guide to how Search works calls these steps crawling, indexing and serving.
Crawling does not guarantee indexing, and indexing does not guarantee placement in results. The guide also says Google does not guarantee any of the three stages, even for a page that follows all of its rules. Fixing technical problems will not necessarily make a site rank first on Google. It improves the chances of ranking higher because each removed obstacle makes the site easier to understand. Technical SEO still works with regular SEO: keywords, backlinks, engagement on social media and other content signals.
In my opinion, technical SEO is harder to apply than regular SEO. It often requires developer knowledge, and changes can feel uncomfortable without a technical background. A technical problem can stay unnoticed for months because it is not a design problem. You may only find it through specific checks, for example with Google PageSpeed Insights.
Speed is also important. A slow site can have weak SEO because most people will not wait for a page to load. This tutorial does not focus on page speed topics such as Largest Contentful Paint, Cumulative Layout Shift or server speed. It focuses on the behind-the-scenes issues that stop search engines from reaching pages.
How common these problems are
The numbers below come from audits run with the site-auditing tool built by the author. Each website is counted once, using its first audit, and the data comes from October 25th. Crawls check many pages on one site, with a median of 10 pages in these samples. Single-page audits check one page, usually the home page.
| Problem | How often | Sample |
|---|---|---|
| At least one page with no canonical tag | 64.2% of sites | 53 crawled sites, Aug 2025 to Oct 2026 |
| Two or more pages with exactly the same title | 61.8% of sites | 34 crawled sites with two or more pages that loaded |
| At least one page with no meta description | 49.1% of sites | 53 crawled sites |
| At least one link leading to a page that returns 404 | 18.9% of sites | 53 crawled sites |
| Page built mostly with JavaScript | 17.7% of pages | 277 single-page audits, May to Oct 2026 |
| At least one page with no title | 13.2% of sites | 53 crawled sites |
| A link that search engines may not be able to follow | 8.3% of pages | 278 single-page audits |
| A robots.txt file with errors, or one that failed to load | 7.9% of sites | 252 single-page audits, Jul to Oct 2026 |
| Page blocked by noindex or by robots.txt | 4.8% of pages | 251 single-page audits, Jul to Oct 2026 |
These problems are technical, but they do not weigh the same. A missing canonical tag is common, but Google does not require one, so a page without it is not automatically broken. It matters when the same or very similar content is reachable at more than one URL. Duplicate titles are easy to create while writing code. Missing meta descriptions matter because the description helps search engines and AI tools such as ChatGPT understand the page briefly. In Google search, the description usually appears under the actual link to the site.
Soft 404s, redirect chains and wrong canonical tags are not in the table because these audits do not record them.
Set up a safe practice site
You do not need to be an experienced programmer. You need a web browser such as Chrome, Firefox, Edge or Safari. In Safari, turn on Show features for web developers under Settings > Advanced. You also need a terminal: Terminal on macOS, PowerShell on Windows, or any terminal on Linux.
You need curl, a small program that downloads web pages. It comes with macOS, most Linux distributions, Windows 10 version 1803 and later, and Windows 11. You need Node.js to run the practice site. The code was tested with Node.js 26.8 and curl 8.7.
On Windows PowerShell, type curl.exe instead of curl, because curl is a shortcut to a different command there. Replace | grep -i "text" with | Select-String "text", and replace /dev/null with NUL.
To avoid criticizing a real website, the author built a tiny practice site for a made-up shoe store called Acme Shoes. Every page has one problem built in, so you can inspect it without risking a real site.
- Create a new folder.
- Add a file called server.js.
- Paste the provided code into that file.
- Start the site with Node.js.
- Open the local pages at localhost:8080.
The server listens on port 8080 and serves pages such as /products/blue-runner, /products/red-runner, /old-sale, /sale, /about, /blog/choosing-trail-shoes, /catalog, /api/products and /robots.txt.
Check status codes and find soft 404s
Start with the HTTP response. A page can look like a normal result while its content says the item is missing. The practice site has /products/red-runner with a message that the product could not be found, but it still answers 200 OK. That mismatch is a soft 404: the page is present, but the server does not clearly signal that the requested content is gone.
Use curl or browser developer tools to inspect the status code and headers. If the product is truly missing, the response should match that state. If a replacement product exists, send the old URL to a relevant page rather than leaving it to show a generic not-found message with a success code.
Shorten redirect chains
The practice site includes a chain: /old-sale redirects to /sale-2025, which redirects to /sale. Redirect chains add steps. Each step can fail, can delay the final page, and can make it harder for search engines to understand the intended target. Fix the chain by pointing /old-sale directly to the final target, /sale, while keeping the sale page reachable and relevant.
Make sure robots.txt is not blocking important pages
robots.txt helps you tell search engines what they may crawl. If it contains errors, or if a path is blocked, you can stop search engines from reaching pages you want them to see. The practice site has a robots.txt file with User-agent: * and Disallow: /blog. That rule blocks every path that starts with /blog, so blog pages may become unreachable to crawlers even if they are important. Check the file for syntax errors and unintended path blocks.
Find accidental noindex rules
A page can be kept out of search results by a noindex rule, even if it looks normal in the browser. The practice site has /about with an X-Robots-Tag header set to noindex. That instruction is not visible in the page content, so only header inspection reveals it.
There are two related but different controls. A noindex rule asks search engines to keep a page out of their results. A robots.txt rule only stops them from crawling it. If you want a page excluded from results, noindex can be the right tool. If you want it indexed but not crawled, that combination does not work as expected because search engines cannot see the noindex signal without crawling the page.
Use canonical tags correctly
A canonical tag lives in the head of an HTML page and tells search engines which URL is the preferred original version when similar content appears at multiple addresses. It is not required, but it becomes important when duplicate or very similar pages compete.
The practice site has /blog/choosing-trail-shoes with a canonical tag pointing to the home page. That sends the wrong signal: the blog post asks search engines to treat the home page as the main version. A self-referential canonical for that blog post would make the intended URL clearer.
Fix missing and duplicate titles and meta descriptions
Each page should say what it is about. The home page has a clear title, Acme Shoes: Running Shoes for Every Runner, and a description about lightweight running shoes for road and trail, with free returns. The product page /products/blue-runner has only the vague title Acme Shoes and no meta description.
The sale page has a specific title, Summer Sale: Up to 40% Off Running Shoes | Acme Shoes, and a description saying to save up to 40% on last season's running shoes until August 31. The blog page has the title How to Choose Trail Running Shoes | Acme Shoes and a description about grip, drop and fit. The about page has the description How a two-person workshop became a running shoe brand. The catalog page has the title All Running Shoes | Acme Shoes and the description Browse every Acme running shoe.
Duplicate titles can confuse search engines because several pages appear to cover the same subject. Missing meta descriptions remove a brief summary that helps search engines and AI tools understand the page. Give each page a unique title and a short description that matches its content.
Make links crawlable
Search engines follow links to discover pages. The practice site has a Summer sale element that uses an onclick handler to move the browser to /sale. That may work for a human who clicks it, but it is not a normal link for a crawler to follow. Replace click handlers with a real anchor link so important pages are discoverable through HTML. If search engines cannot follow a link to a good article, you may have invested time in content that relatively few people will see.
Check whether content needs JavaScript
The practice site has /catalog with a body that shows Loading... and a script that fetches /api/products before filling the page. The JSON endpoint returns Blue Runner, Trail Grip 2 and City Glide. Google can run JavaScript, so a JavaScript-built page can still be indexed. But the content appears only after the script loads and runs. That is slower for visitors and adds a step that can fail. Not every crawler runs JavaScript. For content that matters most, server-side rendering is a safer approach.
Can you check a page in five minutes?
A quick pass can reveal many of these issues. Look at the title and meta description. Check the canonical tag. Inspect the HTTP status and headers for soft 404s, noindex and X-Robots-Tag. Read the robots.txt file for accidental blocks. Trace redirects to see whether they chain. Test whether key content appears without JavaScript. Verify that important links are normal anchors.
Common mistakes
Do not treat a canonical tag as a universal requirement. It is most useful when duplicate content exists. Do not confuse noindex with robots.txt: noindex excludes a page from results, while robots.txt blocks crawling. Do not leave important content behind JavaScript when server-side rendering is possible. Do not use onclick handlers as navigation links. Do not copy page titles across templates. Do not leave soft 404 responses that look like successful pages. Do not forget that fixing technical SEO improves eligibility and understanding, not guaranteed ranking.
Comments (0)