earlyternAll tools on Apify

All tools SEO and ad research

SEO Site Audit Tool: Website Crawler & Broken Link Checker

Crawl any website and audit every page for technical SEO issues: titles, meta, H1, canonicals, noindex, broken links, redirects, orphan pages, sitemaps.

What it does

Crawl a whole website and get a technical SEO audit of every page in one table: title and meta description (missing, too short, too long, duplicated), H1 headings, canonical tags, noindex, word count, images without alt text, structured data, response time, redirects and broken links. Each website also gets a site summary: pages per issue, every broken link with how many pages carry it, redirect chains, duplicate titles, orphan pages, pages missing from the sitemap, and which AI crawlers the site blocks. Audit 1 website or 100 in one run.

  • Crawls the site like a search engine: starts at the address you give, follows internal links, then reads the XML sitemap (also gzip and sitemap indexes) and audits the pages only the sitemap lists.
  • Audits every page for 21 common technical SEO issues (full list below) and gives the raw values too, so you can apply your own rules.
  • Checks every link once: internal links, links to files (PDF, images) and links to other websites, and reports each broken one (404, 410, 5xx, dead domains) on the pages that carry it.
  • Compares the crawl with the sitemap: orphan pages (in the sitemap, linked from nowhere), sitemap addresses the crawl never reached, and indexable pages missing from the sitemap.
  • AI search readiness: lists the well-known AI crawlers the site's robots.txt blocks, and whether it has an llms.txt file.
  • Follows redirects step by step and reports redirect chains (two or more hops).
  • Fast and light: plain HTTP requests, no browser, no proxy.
  • Polite by design: always obeys robots.txt (including Crawl-delay), waits between requests to each website, slows down when a site asks it to, and never logs in.

Who it is for

  • SEO agencies and freelancers: run a first audit for a new client, then schedule it monthly and compare.
  • Website owners and marketers: find broken links, missing titles and descriptions before they cost traffic.
  • Developers and QA teams: check a site after a migration or a release for 404s, redirect chains and noindex tags left by mistake.
  • Content teams: spot thin pages, duplicate titles and pages no other page links to.

Sample output

One result row, as you download it (long values shortened).

FieldExample
recordTypepage
sitehttps://crawlee.dev/
urlhttps://crawlee.dev/blog
statusCode200
depth1
redirectChainempty
titleCrawlee Blog - learn how to build better scrapers | Crawlee for JavaScript · Bu…
titleLength107
metaDescriptionGuides and tutorials on using Crawlee, the most reliable open-source web scrapi…
metaDescriptionLength151
h1empty
h1Count0
canonicalhttps://crawlee.dev/blog
indexabletrue
wordCount1573

Each row has 9 more fields.

Common questions

Does it run JavaScript? No. It reads the HTML the server sends, as most search engine crawlers do first. On sites that build their pages with JavaScript in the browser, pages may look empty (no title, few words); that is a real SEO risk worth knowing about.

Why was my site not crawled? The site summary row says why in message: robots.txt does not allow crawlers, the site could not be reached, or it belongs to a large platform that does not allow automated access (social networks, search engines, large marketplaces), which this Actor never crawls.

Why do some links count as "not checked"? Links blocked by the target site's robots.txt, links to large platforms, links that answer 401, 403 or 429, links over the link limits (see Limits) and links left when the run ran short of time are not checked. They are counted in linksNotChecked, never reported as broken.

Are 403 answers broken links? No. 401, 403 and 429 usually mean a login wall, bot protection or a rate limit, so such links are counted as not checked, never reported as broken.

Can I audit a staging site? Only if it is on the public internet with a normal domain name. The Actor does not open private network addresses, IP addresses or non-standard ports, and does not log in.