Your $300,000 website scores 100 on Lighthouse, but AI search still cannot read it.

Founders and marketing leads at AI startups that spent $100,000 to $350,000 or more on a new website are the audience for this finding: a weekly AI audit can expose the plain files and stated facts that search engines and ChatGPT-style answer engines depend on. After the pages load, forms pass basic tests and launch posts go out, teams often stop reviewing the files behind the site.

Six months later, buyers may ask ChatGPT about the category, and the startup can be absent from the answer or the answer can describe a product it no longer sells. If someone runs a speed test, the site can still score in the 90s. The account comes from the author’s work with AI companies and from an audit of their own site. That site reached 97 to 100 on Lighthouse. The weekly AI audit still surfaced three plain problems: a 2018 sitemap, a robots.txt file that sent search engines to a university’s sitemap, and a contact email hidden from crawlers. In other words, a site that behaves well for people can still be unclear to machines.

Why costly launches can still leave machine files broken

The budget usually pays for design, build and handover. The boring files are created once and left alone. Agencies are paid to deliver a site that works and looks right, and the contract ends when the site is live. The sitemap and robots.txt files may come from a template, a plugin or a copy of another project.

Marketing owns the copy. Engineering owns the product. The agency owns nothing after handover. The files that connect the site to search engines and AI sit in that gap. Meanwhile, the site changes during the first year: teams add pages, rename products, change positioning and hire new people. The structured data and meta descriptions can still describe launch day.

Founders often judge a site by whether it works. When they run a test, it is usually a speed test such as Lighthouse or PageSpeed Insights. Those scores can measure speed and technical basics, but they do not check whether search engines and AI can read the site correctly. They will not reveal a sitemap listing pages that no longer exist, or show that ChatGPT is quoting last year’s tagline.

What the weekly AI audit found on an SEO adviser’s site

The author’s site is a small consulting site, fast and clean, built by someone who advises on marketing and SEO for a living. The audit read the code, ran Lighthouse, and listed 14 findings ranked by impact. Six apply to almost any startup site.

1. A sitemap that stopped in 2018

A sitemap.xml is the list a site gives to search engines so they know which URLs should be crawled. The author’s sitemap had dates from 2018. It listed a domain with a typo, a duplicate of the homepage, links to page sections that search engines ignore, and a file that no longer existed. The pages that mattered now were missing. Search engines use the sitemap to decide what to crawl, so a wrong sitemap can send them to dead pages and hide the pages the site wants found.

The check is simple: open yourdomain.com/sitemap.xml, noting that most sitemap URLs fall there but the exact location depends on the build. Are the dates recent? Is every important page there? Are test pages, old campaigns or staging URLs listed when they should not be?

2. A robots.txt file pointing to another site

A robots.txt file tells crawlers which parts of a site they may visit and where the sitemap is. The author’s file had been copied from a university department’s site years ago. It pointed search engines to that department’s sitemap, blocked folders that did not exist on the author’s site, and blocked a few crawlers for no clear reason. It did not block AI crawlers, which was good, but that was luck, not a decision.

Because robots.txt is one of the first files a crawler reads, a bad file can distort everything after it. The check is to open yourdomain.com/robots.txt and confirm that the sitemap line points to the site’s own domain and that it does not block important paths. A stray "Disallow: /" can block the whole site and is often left over from a staging setup.

3. A contact email hidden from machines

The email links on the page used Cloudflare email-protection links saved into the HTML. Visitors could still click them, but crawlers and AI engines saw a scrambled link instead of an email address. The links would also break if that Cloudflare feature were ever turned off.

If a machine cannot read how to contact a company, it cannot pass that information on. Some teams hide their email on purpose to reduce spam, which can be a fair choice, but it should be a choice. The check is to view the page source, not the browser inspector, and search for the email address or contact page link. If it is not plain text, a crawler cannot find it either.

4. Best content living on other sites

The author publishes one article a week, but all of it lived on outside publications. The author’s own site linked out to those articles, had no Article structured data and had no article pages of its own. Structured data is code, usually schema.org markup in JSON-LD, that labels what a page is about, such as a person, company, product or article, so machines do not have to guess.

Content on other platforms builds their domain, not yours. Startups do the same thing with blogs on Medium or Substack, announcements only on LinkedIn, and docs on a separate platform. The fix is to publish the best content on the site’s own domain first, then add a canonical link on the copies that points back to the original. The author states that the article follows that rule.

5. Old facts inside structured data

The structured data on the author’s site listed job details that no longer matched current work. Visitors rarely see this code, but AI answer engines do. AI answer engines repeat the facts a site gives them. For a startup, that can mean an old tagline, old product names, past pricing, a founder who left, or a funding stage from two rounds ago sitting in structured data, the About page, the footer or the press kit.

The check is to ask a developer for the structured data on the homepage and About page, or view the page source and search for "application/ld+json". Then read it, along with the meta description, as if an AI were deciding how to describe the company.

6. An FAQ written for ranking, not readers

The FAQ answers repeated the same key phrases two to four times each. It read as if it had been written for a ranking algorithm. Repeated phrases make answers harder to read and harder to quote. AI answer engines tend to use short, direct answers that state one fact clearly. The check is to read the FAQ out loud. If an answer repeats the main keyword more than once, rewrite it in two or three plain sentences.

The same gaps appear on Webflow, Framer, WordPress and custom builds

Every platform creates some of these files automatically, which helps, but it is also why teams stop looking at them. Each platform’s defaults leave a different gap. Platform features change often, so the notes here reflect how each platform worked at the time of the source text; check current documentation before changing anything.

Webflow

Webflow is a common choice for funded startups because designers can build and edit without engineers. Its sitemap is generated automatically and includes published static pages and CMS items unless they are excluded. That can list old campaign pages, thank-you pages, test CMS items and utility pages.

Every Webflow site also has a yoursite.webflow.io address. If indexing of that subdomain is not turned off, search engines can find a second copy of the site. The robots.txt file sits in the SEO settings as a text box and is often empty, copied from another project, or left with rules from before launch. Structured data is manual: Webflow does not add schema automatically. It goes in through custom code, and on CMS templates the fields must be mapped, or every post can end up with the same markup.

Check which pages are excluded from the sitemap, whether the webflow.io subdomain is set not to be indexed, and what structured data the CMS templates output.

Framer

Framer is popular with early-stage teams because a site can move from design to live in days. That speed makes it easy to skip settings. The site can look finished long before each page has its own title, description and social image, so pages can end up sharing the same title and description.

Framer sites also have a Framer-hosted address. Only the main domain should be meant for indexing. Structured data is manual, like Webflow, and goes in through custom code. CMS pages use templates, so if the template’s SEO fields are not set up, every article or case study can get generic metadata. Check the title, description and social image on each important page, not just the homepage, and what structured data each page outputs.

WordPress

WordPress still runs a large share of company blogs and marketing sites, often with many years of plugins and themes layered on top. It creates its own sitemap, and SEO plugins such as Yoast or Rank Math create another. Sites can end up with both, or with the wrong one submitted to Google Search Console.

WordPress also has a setting that asks search engines not to index the site. It is often turned on during the build and forgotten at launch. Plugins and themes can add tag archives, author pages, attachment pages and theme template pages to the sitemap. On another WordPress site the author works on, theme template pages were listed next to real articles. A caching plugin can also be installed without serving cached pages, slowing the site for both people and crawlers.

Check that there is one sitemap, that robots.txt points to it, that the discourage search engines setting is off, and which page types are listed in the sitemap.

Custom builds: Next.js, Astro and similar

Startups with strong engineering teams often build the marketing site in the same framework as the product. In frameworks such as Next.js or Astro, nothing exists unless someone adds it. The frameworks offer ways to generate a sitemap and robots.txt from code, but a developer has to set them up. Many sites have a hand-written sitemap from launch that was never updated; that is what happened on the author’s site.

Content that needs JavaScript may be invisible. If the main text only appears after JavaScript runs in the browser, some crawlers will not see it. Google renders JavaScript, but many AI crawlers read only the raw HTML. Hosting platforms also create preview URLs for each change. If those preview deployments are public and not blocked, search engines can find them.

Check the page source to confirm that the main text is present, that the sitemap is generated from real pages, and that preview URLs are not indexed.

The source also names HubSpot CMS, Wix and Squarespace as additional platforms in the list.

What founders should take from the audit

The files that matter most are plain and easy to forget: sitemap.xml, robots.txt, structured data, and the facts the site states about the company. A high Lighthouse or PageSpeed score can show that a site is fast, but it cannot show whether AI answer engines and search engines are reading the right version of the company.

AI answer engines repeat what a site says about a company; outdated facts become outdated answers. A weekly AI audit reports problems while a person decides the fixes. The source says it catches these issues early and costs very little. For readers, the next step is not another speed test alone. It is a short ownership check: confirm that the sitemap is current, the robots.txt points to the right domain, contact information is readable in source, structured data matches today’s facts, and the most important content belongs to the site’s own domain.