Sitemaps and canonical URLs: one piece of content, one address

What it is

A canonical URL is the address a site declares as preferred among several duplicates. RFC 6596 defines the link relation canonical for this purpose: the target must identify content that is identical to, or a superset of, the referring document. It is expressed with <link rel="canonical" href="…"> in HTML, or with an HTTP header Link: <…>; rel="canonical" for non-HTML documents. Google's documentation on consolidating duplicate URLs classifies redirects and rel="canonical" annotations as strong signals and inclusion in a sitemap as a weak signal. It notes that the methods reinforce each other when combined, and it recommends absolute rather than relative URLs in the link element; otherwise, if a test site is crawled by accident, its address becomes the canonical one.

The sitemap complements this. The sitemaps.org protocol describes an XML file with one <url> entry per address, containing <loc> (required) and, optionally, <lastmod>, <changefreq> and <priority>. A single file may contain at most 50,000 URLs and be at most 50 MB uncompressed; larger sites split their URLs across several files and reference them from a sitemap index file.

Why it matters

The same text under several addresses (with and without www, HTTP and HTTPS, tracking parameters, JSON and Markdown twins) dilutes signals and wastes crawl budget. Search engines then choose a canonical URL themselves, and it is not necessarily the one you want. Agents that read a site programmatically need the same answers: which address is the reference, and what has changed since the last visit.

How to apply

• Use one host and HTTPS, and redirect all variants permanently, preserving path and query.
• On every indexable HTML page, set a self-referencing canonical that is absolute and uses HTTPS.
• Full-text twins without HTML carry a Link header pointing to the HTML page; partial views (search results, excerpts) get noindex instead of being canonicalized.
• Paginated and filtered pages get their own canonical only if their content is self-contained and finite; otherwise use noindex.
• Generate the sitemap from the database, with real modification times; leave out search results, duplicates and hidden content; state its location in robots.txt.

Pitfalls

A canonical that points to a redirecting or missing address. Different canonicals on the HTTP and HTTPS versions. Canonicals pointing to a staging host. Invented lastmod values, or values reset on every build: a date unrelated to an actual content change carries no information. Trying to hide duplicates via robots.txt: the canonical is then never read.