An XML sitemap is a list of URLs you want search engines to discover. It is not a certificate that every URL is indexed, or even a promise that every submitted page will be crawled. Google’s sitemap overview calls it a discovery aid. The practical job of an audit is therefore not to celebrate a green “submitted” label; it is to compare the sitemap against the site’s actual canonical, accessible and useful pages.
A good audit can uncover a missing high-value guide, an old redirect left in a file, stale modification dates, an unreachable sitemap child or an entire section accidentally absent after a migration. It can also show when a supposed “sitemap problem” is really a canonicalization or content problem. The workflow below is designed for a CMS site with one or more sitemap files, but the same logic applies to a large publisher. It favors evidence and repeatable checks over a made-up SEO score.
First, define what the sitemap should contain
Build a small inventory of the site before reading XML. List the preferred protocol and host, the sections intended to be public, the current canonical URL for each important page, and any language or media variants. A sitemap should generally list the URLs you want to appear in search, not every route that responds with a page. Google’s build-a-sitemap guidance advises using fully qualified, canonical URLs and keeping files within its documented limits.
This “desired set” is a working editorial decision. A product page removed intentionally should not be counted as missing; a newly published guide that should be discoverable might be. Obtain the set from the CMS publication list or another trusted export, then spot-check it against the live site. Do not assume every URL in an analytics export ought to be submitted. Site search pages, temporary previews and duplicate filtered routes often have a different purpose.
Define the time window, too. After a move or major template launch, audit recent changes and the largest sections first. For a routine monthly review, sample established sections as well as new URLs. If a CMS generates the sitemap automatically, write down which settings or plugins control inclusion; otherwise an apparent correction may be overwritten on the next generation cycle.
Step 1: Fetch the index and every child file
Find the sitemap location from the site’s robots.txt, CMS settings and the site’s known sitemap URL. It may be a single XML file or a sitemap index pointing to multiple child files. Fetch each referenced child using the same environment that a public crawler can access. Record HTTP status, final URL after redirects, content type, file size, and whether the response is valid XML rather than an HTML error page.
A sitemap index that loads while one child fails is only partly healthy. The first fix is to restore the missing child or remove the stale reference, not to resubmit the healthy index repeatedly. The sitemaps.org protocol defines the XML structure, including urlset, loc and optional lastmod fields; an XML parser is more reliable than a visual scan for malformed nesting or escaped characters.
Check whether the file declares the correct namespace and whether URLs are properly escaped. A stray ampersand can break XML; an absolute URL with the wrong host can be valid XML yet wrong for the website. If a file is compressed, check the uncompressed size and parse the actual output. Google’s published ceiling is 50,000 URLs or 50 MB uncompressed per sitemap; split larger collections into child sitemaps and an index. A smaller section-based split can also make diagnosis faster.
Step 2: Reconcile sitemap URLs with canonical pages
Export the XML loc values and compare them to your desired set. Classify each URL as present and correct, missing, or present but misleading. “Misleading” means a listed URL points to a redirect, returns an error, is blocked from crawling, has a noindex directive, or names a different canonical page. Check the actual response and rendered canonical signals rather than deciding from a filename.
The critical comparison is between the listed URL and the preferred canonical URL. Google’s canonicalization guidance says a sitemap can indicate the URLs you prefer, but it is a weaker signal than redirects or rel=canonical. Contradictory signals make diagnosis harder. If the XML lists HTTP while pages canonicalize to HTTPS, update generation to the final HTTPS URLs. Do not keep the redirecting versions just because they once received traffic.
Not every canonical discrepancy requires deleting a useful page. A print view or translated page might have a deliberate relationship to another URL, and multilingual annotations require their own correct implementation. Confirm the underlying intent before changing directives. In particular, don’t apply noindex simply to “fix” a Search Console report without checking whether the page was meant to rank. Canonical, indexability and user value are separate decisions.
For image and video sitemaps, audit the page URL and media references together. A broken media URL, blocked asset or reused placeholder has a different remedy from a missing HTML page. If you have a small site, a full URL comparison is manageable; if you have thousands, sample every section and test all newly changed/high-priority URLs first.
Step 3: Test status, indexability and crawl access
For each priority loc, request the page and note the final status. A successful 200 response is necessary but not sufficient: a soft error page may still say “not found” to readers, and a 200 page can carry noindex. Also inspect the canonical link, robots directives, and whether the page is linked from the site. A sitemap can help discovery of an orphaned page, but it does not replace crawlable internal links.
Keep the problems distinct. A robots.txt block prevents crawling of that path, while a noindex directive is a page-level indexing instruction a crawler must access to see. If both exist, changing only one setting may not solve the intended issue. Use the site’s robots.txt audit checklist as a handoff target, then add the real website link after publication.
Server failures and redirect loops deserve immediate technical attention. An accidental 404 in a sitemap is usually a stale entry; an important page removed in error may require restoring content and links. If a URL is intentionally retired, remove it from the sitemap and decide whether a relevant successor warrants a redirect. Do not point all deleted pages to a homepage to make a status check turn green.
Step 4: Audit lastmod without faking freshness
The lastmod field is optional. When used, it should describe the page’s meaningful last modification, not the time a sitemap generator ran. Google’s lastmod guidance says it may use this value when it is consistently accurate; updating every date every day is not an indexing shortcut. Changes to main content, important structured data or significant links are stronger candidates than a footer timestamp.
Take a sample of 20 recently edited pages and compare the CMS edit history, actual content change and sitemap lastmod value. If the dates diverge, fix the generator rule or omit the field until it can be trusted. Do not fabricate history by resetting all values during an audit. The same Google post notes that the old sitemap ping endpoint was deprecated; use supported discovery and submission mechanisms instead.
Step 5: Compare Search Console with the live files
Open the Sitemaps report in Google Search Console for the correct verified property. Confirm the submitted index address, last read date and fetch/parse errors, then inspect child problems if available. A submitted sitemap can be accepted while individual URLs remain excluded or undiscovered. Submission and indexing are different states; do not present “success” as full coverage.
Use URL Inspection for a small sample of important pages when authenticated. Compare Google’s selected canonical with the declared canonical and the URL in the sitemap. A discrepancy is a diagnostic lead, not proof that the sitemap alone caused exclusion. Search Console data can lag, so retain a dated audit record and retest after meaningful changes. Use manual Request Indexing sparingly for the highest-priority new or materially changed URLs, if authorized; use the sitemap for the rest and avoid repeated requests for unchanged pages.
If the Sitemaps report shows “Couldn’t fetch,” verify the public endpoint, DNS and server logs before changing the XML content. If it shows parse errors, inspect the exact child file and byte encoding. If it reports an unexpected URL count, compare today’s live file to the version Google last processed. This sequence keeps you from solving yesterday’s cached symptom instead of today’s issue.
The PRIORITY matrix: fix the right URLs first
The following editorial matrix is an original triage tool, not a Google ranking factor or official Search Console metric. Give each issue a business importance from 1 to 3, a search-discovery impact from 1 to 3, and a confidence level from 1 to 3 based on direct evidence. Multiply them for a rough work-queue score, then check the underlying issue before making changes. Do not publish this as keyword difficulty or predicted traffic.
Example calculation
Suppose a new resource guide is absent from every sitemap: importance 3, discovery impact 2, confidence 3 after a CMS/XML comparison. Work-queue score = 3 × 2 × 3 = 18. A known temporary preview accidentally listed: importance 1, impact 2, confidence 3, score 6. Investigate the guide first, but still remove the preview. The numbers are illustrative editorial priorities, not search-volume measurements or evidence that Google would index one page.
Decision matrix
- Missing, canonical, public page: repair the generator’s inclusion rule; verify internal links; submit the updated sitemap if needed.
- Listed URL redirects: replace it with the final canonical URL; check inbound internal links and redirect rules.
- Listed URL is noindex or blocked: confirm intent first; remove it if exclusion is deliberate, or fix the conflicting directive if it should rank.
- Listed URL returns error: restore the page if it is still wanted; otherwise remove it and handle relevant successors.
- lastmod changes without meaningful edits: fix the timestamp source or omit the field until it is dependable.
- Child sitemap cannot be fetched: restore the endpoint or remove the index reference, then confirm with live fetch and Search Console.
This matrix helps keep a team from treating all rows in a crawl export alike. A missing flagship guide and an intentionally excluded preview should not receive equal effort just because both appear in an error report.
A practical XML sitemap audit checklist
- Confirm the preferred host, protocol and property; gather an editorial list of pages intended for search.
- Locate the sitemap index or file via CMS settings and robots.txt; record the current URL.
- Fetch the index and every child publicly; log status, redirects and final MIME/content.
- Parse XML with a real parser; validate namespace, escaping, absolute loc URLs and encoding.
- Enforce documented file limits; split oversized files into child sitemaps.
- Deduplicate loc values and group them by content type, template and language.
- Compare loc values with the canonical publication export; flag missing and stale entries.
- Test priority URLs for final status, canonical, noindex, robots access and useful content.
- Check that important pages have crawlable internal links, not only sitemap entries.
- Sample lastmod against actual meaningful changes; fix generator errors without mass-refreshing dates.
- Review Search Console’s Sitemaps report for fetch, processing and child-file issues.
- Inspect a few representative URLs for Google’s selected canonical and indexing status.
- Assign an owner and fix date to every issue; rerun the same sample after deployment.
- Submit a corrected sitemap when necessary; log any rare manual indexing requests without duplicating them.
What a completed audit looks like
Your deliverable is a compact table with URL, expected state, observed state, evidence, owner, fix and recheck date. For example: “/guides/example/ — public canonical — missing from guide sitemap — CMS export plus live XML — content operations — restore inclusion — next crawl check.” That is actionable. “Sitemap health 97%” without a defined denominator or a list of exceptions is not.
You can calculate priority coverage as (priority canonical URLs correctly listed ÷ priority canonical URLs expected) × 100. If 36 of 40 sampled priority URLs are correctly listed, the sample coverage is 90%; it is not a whole-site score and says nothing by itself about ranking or indexing. Record the sample design. Pair the figure with the four missing URLs and their owners.
Repeat the audit after a migration, major CMS change or publishing-system update. Otherwise set a cadence proportionate to your site. Fix the process that generated the problem, not just the exported XML. The best sitemap is a faithful, maintained reflection of important canonical pages that readers can reach through a usable website.
Key takeaways
- XML sitemaps help discovery but neither guarantee crawling nor indexing.
- Compare live XML with canonical, public, linked pages; “submitted successfully” is not an audit.
- Keep lastmod honest and prioritize fixes with explicit evidence.
- Log fixes and verify them after deployment instead of resubmitting unchanged URLs.
FAQ
Should every page on a site be in an XML sitemap?
No. Include the canonical, public URLs you actually want search engines to consider. A CMS may generate files automatically, but its defaults still need review.
Will submitting a sitemap make Google index all its URLs?
No. It is a discovery signal. Indexing depends on crawl access, canonicalization, content and other systems outside the sitemap.
How often should I update lastmod?
Only when a page changes meaningfully and the value can be accurate. Do not rewrite it merely because the file was regenerated.
Should I request indexing for every corrected URL?
No. When authenticated, reserve manual requests for a few highest-priority new or materially changed pages, keep a request log, and let the sitemap and crawlable links serve the rest.
