Screaming Frog SEO Spider

What Is Screaming Frog SEO Spider?

Most software categories have a de facto standard tool — the one professionals reach for first, the one that ends up in onboarding guides, the one used by organisations that could afford anything. In technical SEO crawling, that tool is Screaming Frog SEO Spider, built by a bootstrapped company in Henley-on-Thames, Oxfordshire, and priced at £199 per user per year.

The bluntness of that claim is justified by the evidence. Apple, Google, Disney, and NASA are documented users. Screaming Frog has won UK Search Awards Best SEO Software for five consecutive years. The tool ships to agencies and in-house SEO teams in 190+ countries from a company that has never taken external funding, never raised a priced round, and never migrated to a mandatory cloud subscription.

Screaming Frog Limited was incorporated in 2010 (Companies House 07277243) in Henley-on-Thames. The SEO Spider is the flagship product, available for Windows, macOS, and Linux. The company's business model is simple: a permanent free tier capped at 500 URLs, and an annual licence at £199 per user that removes all limits and unlocks advanced features. That has been the model for over a decade. It has not changed.

The architecture choice that distinguishes Screaming Frog from cloud-based competitors is also its simplest: the application runs on your machine. All crawl data is processed locally. No site content is transmitted to Screaming Frog's servers. For agencies handling client site data under GDPR, and for in-house teams at regulated organisations, this is a structural compliance advantage, not a marketing claim.

Key Features

Technical Crawl Depth: More Configuration Than Competitors Bundle

A standard Screaming Frog crawl surfaces broken links, redirect chains, HTTP status codes (4xx, 5xx), canonical tag issues, page title and meta description problems, heading structure errors, orphaned pages, and crawl depth issues. This baseline is available on every SEO crawling tool. What separates Screaming Frog is the depth of configuration available when you need it.

Custom extraction using XPath, CSS selectors, and regular expressions allows any data point from any page to be pulled into the crawl report. Teams auditing structured data implementations, custom CMS fields, or non-standard meta tags use this to extract exactly what they need without post-processing. No other tool in the category gives this level of extraction flexibility in the core product.

Hreflang auditing is thorough and practical: the tool validates hreflang tag syntax, identifies missing reciprocal tags, flags conflicting annotations, and exports the full hreflang matrix for review. For multilingual and international SEO work, this replaces dedicated standalone checkers.

Crawl comparison (diff mode) lets you load two crawl files and see exactly what changed between them — new URLs found, pages that changed status, titles that were updated. This is particularly valuable for tracking the impact of site migrations or development deployments.

JavaScript Rendering via Integrated Chromium

Screaming Frog includes an integrated Chromium engine that fully renders JavaScript before crawling. This means React, Angular, Vue, and other single-page application frameworks are audited as the search engine sees them, not as the raw HTML source.

JS rendering is toggleable per crawl, with configurable options for wait times and rendering behaviour. The practical implication is that content injected by JavaScript — including dynamically generated meta tags, lazy-loaded content, and client-side-rendered navigation — is visible in the crawl. Cloud-based site audit tools sometimes approximate this; Screaming Frog's local Chromium instance does it directly.

API Integrations: Third-Party Data Overlaid on Crawl Results

The paid licence unlocks a set of API integrations that no other standalone crawler matches in breadth. Google Analytics 4 metrics (sessions, bounce rate, conversions) can be pulled alongside crawl data, so you see which broken pages have traffic. Google Search Console data — impressions, clicks, average position — overlays on crawl results. The Google Search Console URL Inspection API can be run in bulk against crawled URLs, returning indexability status for each one at scale.

Third-party SEO data from Majestic (Trust Flow, Citation Flow), Ahrefs (Domain Rating, backlink count), and Moz (Domain Authority) can all be appended to crawl results as additional columns. PageSpeed Insights scores from Google's API run against crawled pages. This combination — your own crawl data enriched with ranking signals, link metrics, traffic data, and Core Web Vitals — in a single exportable spreadsheet is genuinely difficult to replicate outside of expensive enterprise platforms.

AI Content Generation at Crawl Scale

AI-assisted content generation runs during crawl time, configurable to use OpenAI, Google Gemini, Anthropic Claude, or a local Ollama model. It operates against any columns in the crawl data — crawled page headings and body content pass to the LLM, which returns generated meta descriptions or titles at scale. Agency teams doing large-site on-page optimisation can save significant hours on metadata generation that would otherwise be done page by page.

Accessibility Auditing via AXE

The built-in AXE accessibility engine runs WCAG compliance checks alongside the SEO crawl. Accessibility issues — missing alt attributes, inadequate colour contrast, unlabelled form elements — appear in the crawl report alongside SEO findings. For teams handling public sector clients (subject to WCAG requirements in the UK and EU), or for organisations with accessibility obligations, this integration removes the need for a separate accessibility scanning tool.

Pricing

Screaming Frog's pricing model is transparent and consistent with its history. The free tier crawls up to 500 URLs per session, includes core crawl features, JS rendering, and GA4/GSC/PageSpeed integrations, but does not allow saving crawls or scheduling. It is a permanent free tier — useful for small one-off audits without commitment.

The paid licence costs £199 per user per year, which works out to approximately £17 per month. A single annual payment covers unlimited URL crawls, crawl saving and scheduling, custom extraction, crawl comparison, near-duplicate detection, hreflang auditing, accessibility auditing, AI content generation, and the full set of third-party API integrations.

Bulk licence discounts apply from 5 licences, making agency-wide deployment more economical. Screaming Frog does not offer a monthly subscription; the annual licence is the only paid option.

At £199/year, Screaming Frog costs less than two months of an Ahrefs or Semrush plan for 12 months of unlimited crawling. For teams that need full SEO suites, Ahrefs and Semrush serve different purposes — keyword research, backlinks, rank tracking — but for pure technical crawl depth the value comparison is not close. European SEO teams evaluating Surfer or Mangools for content and keyword work will typically run Screaming Frog alongside rather than instead of those tools.

EU Compliance & Privacy

Screaming Frog's compliance picture is unusual in the seo-tools category, and worth understanding precisely.

The crawl data — the content of the sites you crawl — never leaves your machine. Screaming Frog SEO Spider is a desktop application with no server-side component for crawling. When you audit a client's website, the HTML, metadata, and structured data processed by the tool exist only on your local machine. Screaming Frog Limited cannot access it. This is the cleanest possible data handling posture for target site content, and it is structural rather than policy-based.

Screaming Frog's own user and licence registration data is stored in UK and US infrastructure. The company is incorporated in the UK (Oxfordshire, Companies House 07277243) and is subject to UK GDPR — functionally equivalent to EU GDPR post-Brexit but not within EU jurisdiction. This is the honest limitation for EU users: your licence account data is UK-held, and the company is not EU-incorporated.

A Data Processing Agreement is available on request. The DPO contact is [email protected]. Given the desktop-native architecture, the DPA covers licence account data only — which is a much smaller scope than cloud-based tools where all crawl results are stored server-side.

For EU teams, the practical compliance question is: does crawling through Screaming Frog require any GDPR consideration? The answer is that the crawl itself does not involve data transfer to Screaming Frog. The licence account (name, email, licence key) is UK-held. That is a narrow and manageable exposure compared to cloud-based SEO platforms where all crawl results, including client site content, are stored on third-party servers.

Who It's Best For

If you do technical SEO audits regularly — for clients, for an in-house site, or for a platform you maintain — Screaming Frog is the tool to own. The depth of crawl configuration, the JS rendering, the API overlay integrations, and the crawl comparison functionality cover the technical audit use case more thoroughly than any competing tool at any price point.

If you are an SEO agency managing multiple clients, the local data architecture is a compliance advantage: client site content stays on your machines. The annual licence per user is predictable cost. Bulk discounts reduce agency-wide cost. Custom extraction handles the client-specific data requirements that generic reports miss.

If you are new to SEO or need keyword research, rank tracking, or backlink analysis as your primary need, Screaming Frog is not the right starting point. It does not do those things. Ahrefs, Semrush, Sistrix, or Seobility serve the full-suite requirement; Screaming Frog sits alongside them as the technical audit specialist.

If you need continuous cloud-based monitoring — crawls running on a schedule you can check from a browser, alerting on site changes, dashboards shared across a remote team — Screaming Frog's desktop architecture is a constraint. Sitebulb Cloud or DeepCrawl serve that use case. Screaming Frog's scheduling feature requires a machine to be running, which means a local server or VPS rather than a true managed cloud service.

The Verdict

Screaming Frog SEO Spider earns its position as the industry standard for technical SEO crawling by being more configurable and more affordable than any direct competitor. The £199/year price for unlimited crawling, the JS rendering depth, the custom extraction capability, and the API integrations — GA4, GSC URL Inspection in bulk, Ahrefs, Majestic, Moz, PageSpeed — represent a feature density that enterprise-priced tools cannot consistently match.

The bootstrapped independence is meaningful: 15 years of consistent pricing, no forced cloud migration, no VC-driven feature bloat or price restructuring. In a software category full of SaaS price hikes and feature consolidation into higher tiers, Screaming Frog's stability is a real product attribute.

The limitations are structural and honest: desktop-only means no cloud monitoring without a running machine; no keyword research or backlink database means it supplements rather than replaces a full SEO suite; the learning curve is steep for beginners. These are the right trade-offs for a tool that is genuinely best-in-class at what it does.

Frequently Asked Questions

Does Screaming Frog send crawl data to the cloud?

No. Screaming Frog SEO Spider is a desktop application. All crawl data — the content of the websites you crawl — is processed and stored locally on your machine. Screaming Frog Limited has no server-side access to crawled site content. This is a structural privacy advantage for agencies handling client site data under GDPR.

What is the difference between the free and paid licence?

The free tier crawls up to 500 URLs per session and does not allow saving crawls or scheduling. The paid licence (£199 per user per year) removes all URL limits and adds crawl saving, scheduling, crawl comparison, custom extraction via XPath/CSS/regex, near-duplicate detection, hreflang auditing, accessibility auditing (AXE/WCAG), AI content generation, and the full API integrations — Ahrefs, Majestic, Moz, and Google Search Console URL Inspection in bulk.

How does Screaming Frog compare to Ahrefs Site Audit?

Screaming Frog is a dedicated desktop crawler; Ahrefs Site Audit is a cloud-based component of a full SEO suite. For raw technical crawl depth — custom extraction, JS rendering via a local Chromium instance, crawl comparison, hreflang auditing — Screaming Frog is generally more configurable. Ahrefs provides keyword research, backlink analysis, and rank tracking that Screaming Frog does not. Most professional SEO practitioners use both: Screaming Frog for technical audits, Ahrefs or a similar suite for keyword and backlink intelligence.

Does Screaming Frog handle JavaScript-rendered sites?

Yes. The paid licence (and free tier) includes an integrated Chromium rendering engine that executes JavaScript before crawling. React, Angular, Vue, and other SPA frameworks are audited as the browser renders them. Rendering behaviour is configurable per crawl — wait times, JavaScript execution options — giving control over how the crawler handles asynchronous content loading.

Is Screaming Frog GDPR compliant?

The crawl itself is GDPR-neutral from a data transfer perspective: crawl data is processed locally and never sent to Screaming Frog's servers. Screaming Frog Limited is a UK company under UK GDPR (not EU GDPR, but functionally equivalent post-Brexit). User and licence account data is held in UK and US infrastructure. A Data Processing Agreement is available on request at [email protected].