Screaming Frog SEO Spider: The Complete 2026 Guide

Screaming Frog SEO Spider is a desktop crawler. Point it at a site and it fetches every reachable page the way a search engine would, then organizes what it finds into 33 tabs of SEO data: titles, meta descriptions, headings, response codes, canonicals, directives, images, structured data, JavaScript differences, links, and more. Each tab has its own filters, so "Page Titles: Over 60 Characters" or "Canonicals: Non-Indexable Canonical" is one click away. A crawl of a typical small business site takes a few minutes; the master export alone carries about 75 columns per URL.

Reach for it whenever you need the truth about a site rather than a summary of it. Site audits, pre-launch and post-launch checks, migration verification, redirect mapping, finding orphan pages, validating structured data, and building complete URL inventories are all core jobs. It's the evidence layer: dashboards tell you traffic changed, Screaming Frog tells you what on the site could explain it.

The five modes

The Mode menu switches how the tool acquires URLs. Spider is the default: give it a start URL and it discovers the site by following links. List takes a fixed set of URLs you paste or upload and audits exactly those, which is the right mode for checking a redirect map, a sitemap file, or a batch of landing pages. SERP works with title and description data for snippet preview work without crawling.

Compare runs a crawl-over-crawl diff. Load two crawls of the same site and it reports what changed between them across 15 change detection dimensions, which turns a monthly recrawl into a regression report. APIs pulls connected API data (Search Console, GA4, and so on) for a URL list without crawling the pages at all, useful when you want fresh metrics but the pages themselves haven't changed.

Crawl configuration

The SEO Spider consolidates what used to be a dozen separate dialogs into one unified Crawl Config window (Cmd+; on macOS) with a searchable left navigation, a layout it has had since v19. Every item under the Configuration menu opens a pane of this same window, so once you learn the layout you can move through the whole configuration in one sitting.

The Spider section is where most crawls are shaped. It has six panes: Crawl, Extraction, Limits, Rendering, Advanced, and Preferences.

Crawl

The Crawl pane decides what the spider follows and what it stores. Each resource type (images, CSS, JavaScript, SWF, and so on) has separate Crawl and Store toggles, and there are switches for internal and external links, subdomains, nofollow links, and XML sitemap discovery. Storing a resource type means it shows up in the tabs and exports; crawling it means its own response is checked.

For audits, crawl and store everything. Disabling images or JavaScript makes a crawl faster, but it also means broken image references and script errors go unseen. Turn on sitemap discovery so the crawl can compare what the sitemap declares against what actually exists, which is where orphan page detection starts.

Extraction

The Extraction pane controls which on-page elements are pulled out of each page: page details, directives, structured data formats (JSON-LD, microdata, RDFa), and HTML storage. The options worth calling out are Store HTML and Store Rendered HTML, which keep both the raw source and the post-JavaScript version of every page so you can diff them later, plus forms, accessibility data, and PDF properties.

Turn on structured data extraction with validation for any site that depends on rich results. Store both HTML versions on JavaScript-heavy sites; the raw-versus-rendered comparison is often the fastest way to prove that content or links only exist after JavaScript runs.

Limits

Limits caps the crawl: total URLs, crawl depth, folder depth, number of query strings, redirect hops to follow, and URL length. There's also a per-URL-path table that applies a regex-based cap to specific sections of a site, which is the polite way to sample a huge blog archive or faceted category without crawling all of it.

For most sites, leave limits open so the crawl is complete. Use them deliberately on large e-commerce sites where faceted navigation can generate millions of URL combinations; capping query strings at a low number keeps the crawl focused on real pages.

Rendering

Rendering has three modes: Text Only (fast, raw HTML only), Old AJAX Crawling Scheme (legacy), and JavaScript (a headless Chrome renders each page the way Google does). JavaScript rendering is slower but it's the only way to see what a modern framework site actually serves.

With JavaScript selected, the pane expands with rendered page screenshots, JavaScript error reporting, flattened shadow DOM and iframes, the AJAX timeout, and viewport presets including Googlebot Smartphone. Screenshots cost disk space but make client conversations much easier: you can show exactly what the crawler saw.

Advanced

The Advanced pane governs how strictly the crawl mirrors a search engine. Options include respecting noindex, canonicals, and next/prev; HSTS handling; cookie storage; retrying 5xx responses; soft 404 detection with an editable phrase list; HTML validation; and carbon rating calculation.

The respect toggles matter because they change what appears in your reports. With "respect noindex" on, noindexed pages drop out of the crawl views, which matches what Google keeps but hides pages you may want to inspect. For audits, most teams leave the respect options off so everything is visible, then use the indexability columns to filter.

Preferences

Preferences holds every threshold behind the tool's built-in issue filters: page title width (30 to 60 characters, 200 to 561 pixels), meta description limits (70 to 155 characters, 400 to 985 pixels), maximum URL and H1/H2 lengths, the "high crawl depth" cutoff (3 by default), the low content word count (200 words), and the maximum image size (100 KB). The non-descriptive anchor text list and soft 404 phrase list are editable here too.

These defaults are sensible, but they're opinions, not laws. If a site's vertical routinely runs long titles that perform fine, adjust the threshold rather than reporting hundreds of false alarms. Whatever you choose, keep it consistent across recrawls so trend comparisons stay honest.

Content analysis

The Content section of the config drives four analyses that most people never switch on: content area targeting, duplicate detection, spelling and grammar, and embeddings.

Content area

The Content Area pane restricts text analysis to the main content of each page by excluding navigation, footer, and any HTML classes or IDs you specify. Without this, duplicate detection and word counts include the header and footer that repeat on every page, which inflates similarity scores and buries real findings.

Set this per site. Two minutes spent identifying the nav and footer selectors makes every downstream content metric more accurate.

Duplicates

The Duplicates pane enables near-duplicate detection, with a similarity threshold (90 percent by default) and an option to check indexable pages only. Exact duplicates are always detected via page hash; near duplicates require this setting plus a post-crawl analysis run (covered under Crawl analysis below).

Near-duplicate detection is one of the highest-value settings for local SEO and e-commerce work, where templated location pages and product variants quietly cannibalize each other. Pair it with the content area exclusions above or the scores will be skewed by shared page furniture.

Spelling and grammar

Spelling and grammar checking runs across 39 languages. The settings tab picks the language and toggles each check; the rules tab lists every grammar rule individually so you can disable the ones that generate noise for a particular site, and a custom dictionary handles brand names and industry jargon.

This is an easy add-on deliverable. Clients rarely expect their SEO crawl to catch typos on money pages, and it costs nothing extra once enabled.

Embeddings

New territory for a desktop crawler: the Embeddings pane generates a vector embedding for each page's content during the crawl. Two thresholds drive the analysis: semantic similarity (0.95 default) flags pages that say nearly the same thing even when the words differ, and low relevance (0.4 default) flags pages that drift from the site's overall topic. Filter rules let you scope which URLs get embedded.

Embeddings power two features covered later in this guide: the Semantic Search tab, which lets you search a crawl by meaning, and semantic similarity filters in the Content tab. For content-heavy sites, this catches cannibalization that word-level duplicate detection misses.

Crawl scope and behavior

Seven panes control where the crawl goes and how it presents itself to the server.

robots.txt

The robots.txt pane sets whether the crawl respects, ignores, or ignores-but-reports robots.txt rules, and lets you supply a custom robots.txt to test against. The custom option is the underused one: you can validate a proposed robots.txt change against a real crawl before anyone touches production.

URL rewriting

URL Rewriting strips query parameters or applies regex replacements to URLs before they're crawled. The classic use is removing tracking parameters (utm_source and friends) so the same page isn't crawled once per campaign link. Test rewrites in the dialog before running; a bad regex here silently merges URLs that should stay distinct.

CDNs

The CDNs pane lists domains that should be treated as internal even though they sit on a different hostname. If a site serves images from a CDN domain, add it here; otherwise every image shows up as external and drops out of the image audits.

Include and exclude

Include and Exclude take regex patterns that scope the crawl. Include restricts the crawl to matching URLs; Exclude removes matching URLs from it. Both panes have a test field so you can paste a URL and confirm the pattern behaves before crawling. Exclude wins when both match.

Use Include to audit one section of a large site (a /blog/ directory, one country folder) and Exclude to skip infinite URL spaces like calendar pages or logout links. Always run a pattern through the test field first; regex mistakes are the most common reason a crawl comes back mysteriously small.

Speed

Speed sets the maximum threads and, optionally, a cap on URLs per second. More threads means faster crawls and more load on the target server. For sites you don't control, stay conservative; a crawl that degrades a client's site is a bad first impression that no report recovers from. Slow down further for shared hosting and small business sites.

User-agent

The User-Agent pane picks what the crawler calls itself, with presets for Screaming Frog's own agent, Googlebot variants, Bingbot, and browsers. Crawling as Googlebot reveals whether the server treats Google differently from everyone else, which is how you catch cloaking, bot-specific redirects, and overzealous firewalls. Some security layers block unfamiliar agents entirely; switching the user-agent is the first fix to try when a crawl returns nothing but 403s.

HTTP header

The HTTP Header pane adds or overrides request headers. Common uses: an Accept-Language header to test language-based redirects on international sites, and custom headers a staging environment requires before it will serve pages.

Custom intelligence

Four panes under Configuration > Custom let you pull data the standard tabs don't cover. Each supports up to 100 slots.

Custom search

Custom Search checks every crawled page for text or regex patterns you define, in the source or the rendered HTML, and reports which pages contain (or don't contain) each one. Typical uses: verifying an analytics tag exists on every page, finding leftover staging URLs in the source, or locating every mention of a discontinued product.

Custom extraction

Custom Extraction pulls structured values out of pages using CSS selectors, XPath, or regex, up to 100 extractors per crawl. This is how you turn the crawler into a site-specific data collector: prices, SKUs, author names, publish dates, phone numbers, review counts, anything in the HTML becomes a column in your export.

Custom link positions

Link position classification labels every link as navigation, content, sidebar, or footer based on where it sits in the page. The Custom Link Positions pane lets you adjust the selectors that define those zones, with a test field to confirm them, so the labels stay accurate on unusual templates. Accurate link positions matter because a link from body content carries different weight, and different intent, than one repeated in a footer; the link exports carry this classification on every row.

Custom JavaScript

Custom JavaScript runs snippets of your own JavaScript on every rendered page during the crawl and records what they return. It requires JavaScript rendering mode, and the pane carries a privacy note worth reading, since snippets can send page content to external services if you tell them to.

The bundled snippet library is the fast path: around 30 presets covering AI-assisted tasks (generate image alt text, classify page intent, detect language and sentiment via ChatGPT, Gemini, or a local Ollama model), AlsoAsked question research, extracting people names, image classification, passage embeddings, saving page content, and utilities like auto-accepting cookie banners, scrolling the page, or listing the top five words. A User tab stores your own snippets alongside them.

API integrations

Configuration > API Access connects external data sources so their metrics join onto every crawled URL. Eleven providers are available, and each connection is stored per configuration profile.

Google Search Console

The Search Console connection is the single most valuable one for audit work, and it's free. Once connected, every URL gains clicks, impressions, CTR, and average position.

The Search Analytics tab controls the date range, device, country, and search type filters, caps rows at 100,000, and can pull URLs from GSC that the crawl did not find, which surfaces pages Google knows about but your site no longer links to.

The URL Inspection tab goes further: it queries Google's own index status per URL, including coverage state, the Google-selected canonical versus the declared one, rich result validity, and the filter that pays for the setup time by itself, "Indexable URL Not Indexed." The API allows 2,000 URLs per property per day; the multi-property option spreads a large site across several GSC properties to raise that ceiling.

Google Analytics

GA4 joins sessions, engagement, and key events onto each URL, with a metrics tree to pick exactly which numbers you want (Sessions is mandatory). The practical win is orphan detection in reverse: pages receiving GA traffic that the crawl never found are pages users reach but your internal linking has abandoned.

A Universal Analytics connection is still present for anyone with historical UA data, though UA itself stopped processing new data in 2023.

PageSpeed Insights

The PageSpeed connection adds Core Web Vitals (LCP, INP, CLS) and supporting metrics like FCP and TTFB to every URL: field values from CrUX where available, lab values from Lighthouse (INP is field-only; TBT is its lab proxy), plus about 20 opportunity audits. The account tab takes a free API key, but check the Source dropdown: version 24 can run Lighthouse locally instead of calling the remote API, which means no key, no quota, and no rate limits on deep crawls. The metrics tab picks device (mobile or desktop) and which metric groups to store.

Link metrics: Majestic, Ahrefs, Moz

Three backlink providers connect the same way: authenticate, and each crawled URL gains external link metrics, such as referring domains, Trust Flow and Citation Flow (Majestic), DR and UR (Ahrefs, via OAuth sign-in), or DA and PA (Moz). All three are paid subscriptions. Use them when the engagement actually calls for link data, not by default; they add API cost to every crawl they're enabled on.

AI providers: OpenAI, Gemini, Ollama, Anthropic

Version 24 ships four AI connections. Each provider dialog has an account tab for the API key, a prompt configuration tab where you attach up to 100 prompts that run against every crawled page, and an advanced tab for endpoint and rate limit settings. Prompts can classify page intent, judge content quality, check whether a page targets the city it claims to, flag thin or doorway-style copy, or anything else you can phrase as a question about a page.

Ollama is the one to notice: it points at a model running locally on your own machine, so per-page AI analysis costs nothing and no site content leaves your computer. For agencies with confidentiality-sensitive clients, that combination is hard to beat.

Authentication

The Authentication section lets the spider crawl sites behind logins. Standards-based covers HTTP basic and digest authentication, the browser-popup kind common on staging servers.

Forms-based opens an embedded browser where you log in to the site yourself; the spider then crawls with your session cookies. This handles normal web login forms, including most CMS and membership logins. Profiles store authentication setups for export and reuse, so a scheduled or command-line crawl can authenticate without a human present.

Segments

Segments split a crawl into named groups, by URL pattern or other rules, so every tab and report can be filtered to one slice of the site: blog versus product pages, one location folder versus another, templated versus editorial content. Issues can then be counted and trended per segment, which is how you tell a client "the problem is confined to the /shop/ templates" instead of handing over a sitewide list.

Set segments up before a big crawl rather than after; they're the difference between an audit that reads like an inventory and one that reads like a diagnosis.

Crawl analysis

Some metrics can't be computed while the crawl is still running because they need the full link graph or the full content set: Link Score (an internal PageRank-style value per URL), near-duplicate similarity, and semantic similarity from embeddings, among others. The Crawl Analysis config pane selects which of these to compute, and the Crawl Analysis menu runs it after the crawl finishes.

If a column you expected is empty after a crawl, this is the first thing to check. Running crawl analysis is a single click and usually takes seconds to minutes.

Profiles and scheduling

Everything configured above can be saved as a profile, a .seospiderconfig file, from the Configuration > Profiles menu. Profiles are the backbone of consistent audit work: build one "deep dive" profile with everything enabled and one lighter "routine recrawl" profile, and every crawl of every site runs on identical settings. Profiles travel between machines and feed the command-line interface, so a saved profile is also an automation building block.

The File menu holds crawl management: open and save crawls, recent crawls, and the scheduler.

File > Scheduling opens a full in-app scheduler: a task table where each entry pairs a start URL with a configuration profile, an interval, optional API connections, and export actions, plus import/export of tasks and a run history. A monthly or biweekly recrawl can live entirely inside the app with no external cron job.

Application-level settings live in the app menu under Settings: storage mode (memory versus database, where database mode is what makes very large crawls and crawl comparison possible), memory allocation, proxy, language, and the embedded browser. These are per-machine settings rather than per-crawl ones.

Generating sitemaps

Screaming Frog doesn't just audit sitemaps, it builds them. The Sitemaps menu offers XML sitemap and image sitemap generation from the current crawl.

The export dialog has seven tabs. Pages picks which URLs qualify (indexability, response codes, paginated pages, PDFs, and so on).

Last Modified, Priority, and Change Frequency control the optional metadata: lastmod from the server response or a fixed date, priority by crawl depth, and change frequency calculated from last modified or set manually. Google ignores priority and changefreq these days, so feel free to leave them out; lastmod is the one that still matters.

Images controls image sitemap inclusion, with a regex option for CDN-hosted images; Hreflang includes alternate language annotations for international sites.

The Segments tab generates separate sitemaps per segment, which turns a fixed sitemap into a shippable deliverable: one clean sitemap per site section, built from verified crawl data rather than whatever the CMS emits.

Reading the results

The right-hand pane is where a finished crawl becomes an audit. The Overview tab (visible in the main window screenshot at the top of this guide) lists every tab and filter with live URL counts, so it doubles as a table of contents for the whole crawl.

The Issues tab reorganizes the same data by problem instead of by element: every triggered issue with a priority, a plain-language description, and how many URLs it affects. It's the fastest route from "crawl finished" to "here's what we found," and the matching Issues export gives you one spreadsheet of everything.

Site Structure shows crawl depth distribution and the site's shape. Pages more than three clicks from the home page get crawled and ranked less reliably; this chart makes that argument visually for clients.

Segments mirrors your segment definitions with per-segment counts, Response Times shows the speed distribution of server responses across the crawl, and API shows the connection status and data volume for each connected integration, which is where you confirm the GSC or GA4 join actually happened.

Spelling & Grammar aggregates language findings across the site, with the most common errors ranked.

Semantic Search is the newest addition: with embeddings enabled, you can search the crawl by meaning rather than keyword. Ask for "pages about emergency plumbing pricing" and it returns the pages closest in meaning, whether or not they use those words. For content inventories and cannibalization work on large sites, this changes how fast you can move.

Below the main table, per-URL tabs give the forensics on any selected page: inlinks and outlinks, image details, resources, a pixel-accurate SERP snippet preview, the rendered page screenshot, the Chrome console log, and the stored source.

Exports, bulk exports, and reports

Three menus move data out of the tool. Export buttons on each tab save what you're looking at. The Bulk Export menu goes deeper, with around 150 exports including every link edge on the site with anchor text and link position (a small site can easily produce tens of thousands of link rows), all image references, structured data by validation status, accessibility violations, and the AI and custom results.

The Reports menu holds 62 prebuilt reports: redirect and canonical chains, orphan pages from sitemap and analytics comparison, insecure content, hreflang issues, PageSpeed summaries, a cookie inventory, and the crawl overview. Redirect chains and orphan pages are the two most teams use weekly.

Visualisations

The Visualisations menu renders the crawl as interactive diagrams: crawl tree and directory tree graphs, force-directed diagrams (including 3D variants), anchor text and body text word clouds, and a content cluster diagram built from the embedding data. The force-directed crawl diagram is the one clients remember; a site with a healthy structure looks like a tidy dandelion, and a site with a crawl problem looks like a hairball, no explanation needed.

The MCP server

Version 24 includes an MCP (Model Context Protocol) server, available from its own menu with start, stop, and configure entries. Once running, an AI assistant such as Claude can drive the tool directly: start crawls with a chosen configuration file, monitor progress, query individual URLs, pull filters and reports, and run exports, all without a human clicking through the interface.

This is the automation endgame: the same saved configuration profile that standardizes your manual crawls becomes a machine-runnable audit that an assistant can execute, summarize, and file. Combined with the in-app scheduler, the crawl-collect-report loop can run with almost no hands on keyboard.

A recommended 2026 configuration

Build one deep dive profile, save it as a .seospiderconfig, and use it for every first crawl and every scheduled recrawl. The point is that nothing important reports zero because it was never measured.

• —Spider > Crawl: crawl and store all resource types, follow internal and external links, discover XML sitemaps.
• —Spider > Extraction: structured data on (JSON-LD, microdata) with validation; Store HTML and Store Rendered HTML both on.
• —Spider > Rendering: JavaScript rendering with page screenshots and JavaScript error reporting, Googlebot Smartphone viewport.
• —Spider > Advanced: respect options off for audits (see everything, filter later), soft 404 detection on, retry 5xx on.
• —Spider > Preferences: defaults, adjusted per site only when a threshold provably misfires, then kept stable across recrawls.
• —Content: content area set to the site's actual template (exclude nav and footer selectors); near duplicates on at 90 percent; spelling and grammar on in the site's language with a brand dictionary; embeddings on for semantic similarity, low relevance, and Semantic Search.
• —Scope: exclude patterns for known URL traps, tested in the dialog; speed kept polite for sites you don't host.
• —Custom: extraction slots for the site's name, address, phone, and schema fields; a custom search slot verifying the analytics tag on every page.
• —API Access: Search Console with URL Inspection enabled, GA4, and PageSpeed Insights using the local Lighthouse source (no key or quota). Link metric and paid AI providers stay off unless the engagement requires them; if per-page AI prompts are useful, prefer local Ollama first.
• —Crawl Analysis: all post-crawl calculations on, run immediately after each crawl.
• —Segments: defined per site before the first big crawl, mirrored into the sitemap export when sitemaps are a deliverable.
• —After the crawl: read the Issues tab first, confirm the API tab shows the joins landed, run crawl analysis, then export Issues plus the Internal:All master sheet as the audit backbone.

Schedule the recrawl in File > Scheduling against this same profile, and use Mode > Compare against the previous crawl to turn each recrawl into a change report rather than a fresh pile of data.

The vocabulary that makes the tool click

Questions everyone asks