SEO & AI Visibility

A while back, while setting up an early version of SiteVitals, I hit a problem that shows how every day is a school day on the Internet. I'd written a robots.txt file by hand. I knew exactly what was in it. I wanted certain AI crawlers to be able to reach the site, so I'd explicitly allowed them. Then I went to test it, and the bots I'd just allowed couldn't get in.

Beyond Google, a new audience is crawling your site: AI. From GPTBot to Perplexity, AI agents make decisions based on signals traditional SEO tools ignore. I’ve rebuilt the SiteVitals SEO checker to bridge this gap—adding deep schema validation, AI crawler audits, and llms.txt support. Here is a look at the technical hurdles we hit, the three layers of validation we built, and why your site might pass a Google test but fail an AI readiness check.