A sitemap is not meant to help search engines discover a site. It serves to indicate what to explore first, and especially what not to explore. This distinction radically changes the way we build and maintain the file.
XML Sitemap and Crawl Budget Management: The Angle That Guides Omit
On a site with a few dozen pages, the XML sitemap remains a convenience. On a site with several thousand URLs, it becomes a direct lever for reducing crawl waste. Recent log analyses show that a significant portion of Google’s crawl budget can be consumed by out-of-structure URLs: filter pages, parameter variations, old paginations, or orphan pages.
The sitemap acts here as a declarative filter. By listing only canonical, indexable, and truly useful URLs, we signal to the bots what deserves their time. Conversely, a sitemap that lists 301 redirects, soft 404s, or URLs blocked by robots.txt causes Googlebot to spend its budget in the wrong places.
We recommend systematically cross-referencing the sitemap content with server logs. If URLs present in the file are never crawled, or if Googlebot massively explores URLs absent from the sitemap, the file is out of sync with the reality of the site. It is on the Tekniko site that we find a structured approach to this coherence between sitemap and actual architecture.
Why Inflated Sitemaps Harm Indexing
A sitemap containing thousands of non-indexable URLs sends a contradictory signal. Google interprets the file as a list of pages that the webmaster deems worthy of indexing. If half of them return 3xx, 4xx codes, or noindex tags, the reliability of the file drops.
Googlebot then reduces the frequency of checking the sitemap itself. Newly added URLs take longer to be discovered. The effect is measurable in the Search Console coverage report: an increasing gap between submitted URLs and indexed URLs almost always signals a quality issue with the file, not a content issue.
Lastmod Tag in the Sitemap: When Freshness Becomes a Trap
The lastmod tag is the most poorly used of the XML sitemap metadata. Its function is simple: to indicate the date of the last substantial modification of a page. In practice, most CMSs update it with every save, even for a corrected comma or a modified widget in the sidebar.
Google has explicitly hardened its stance on this point. Lastmod values updated without real content change lead bots to ignore this tag for the entire sitemap. The file then loses its ability to signal real updates, slowing down the re-indexing of pages that have actually been modified.
How to Use Lastmod Reliably
- Only trigger the lastmod update when there is a change in main content (text, media, structured data), never for template or menu changes
- Compare the lastmod dates in the sitemap with the actual modification dates in the database, and correct discrepancies via a script or CMS hook
- Completely remove lastmod if the site cannot guarantee its reliability, rather than leaving artificial dates
A sitemap without lastmod remains usable. A sitemap with misleading lastmod becomes counterproductive.
Specialized Sitemaps: Video, Images, and News
The classic XML sitemap covers HTML pages. For rich content, specific extensions of the sitemaps.org protocol allow for the declaration of additional metadata that Google cannot always extract from HTML alone.
The video sitemap remains relevant for sites hosting their own files. It allows for the declaration of title, description, duration, thumbnail URL, and video file URL. Without this metadata, Google relies solely on Schema markup on the page, which is often incomplete or absent.
The image sitemap only provides a benefit when visuals are loaded dynamically (JavaScript, conditional lazy loading) and Googlebot does not discover them through standard rendering. For a static site with standard img tags, the addition is unnecessary.
The news sitemap concerns exclusively sites registered in Google News. URLs must be less than two days old. Including older articles does not trigger a penalty, but Google ignores them, diluting the relevance of the file.
Sitemap Submission and Robots.txt: The Declaration Mechanics
Two declaration methods coexist. The first, historical, involves adding a Sitemap: https://example.com/sitemap.xml directive in the robots.txt file. The second goes through the Search Console or Bing Webmaster Tools.
We observe that the declaration via robots.txt is often the only one in place on sites that have never configured a Search Console. It works, but provides no feedback on how Google processes the file. Submission via the Search Console remains the preferred method, as it exposes parsing errors, the number of discovered URLs, and indexing status.
Sitemap Index for Large Sites
The sitemaps.org protocol imposes a limit of 50,000 URLs per file and a maximum uncompressed size. For sites exceeding this threshold, a sitemap index points to several child files. Segmenting by content type (articles, categories, products) facilitates diagnosis: if a child file shows a low indexing rate, the problem is isolated to a specific type of page.
- Segment by directory or content type rather than alphabetically or chronologically
- Name files explicitly (sitemap-products.xml, sitemap-blog.xml) to simplify analysis in the Search Console
- Ensure that each child file contains only URLs relevant to its scope, without duplicates between files
A well-segmented sitemap transforms the Search Console into a diagnostic tool by content type. A monolithic sitemap of 40,000 URLs drowns signals in the mass and complicates any targeted intervention.