Sitemap, robots and crawl policy
What is in the sitemap, where each URL's last-modified date comes from, which URLs are deliberately excluded, and the crawl policy.
Crawl policy
/robots.txt allows everything and points at the sitemap index:
User-agent: *
Allow: /
Sitemap: https://gabegiro.com/sitemap-index.xmlThere is no rule targeting AI crawlers specifically. They are welcome, which is the same position the llms.txt files take. There is no crawl-delay directive and no rate limiting worth mentioning; this is a static site.
The sitemap
/sitemap-index.xml is the entry point, and it references the URL set. Every indexable page is listed, each with a <lastmod>.
Where lastmod comes from
This is the part most sites get wrong, so it is worth stating precisely. <lastmod> is only a useful re-crawl signal if it is accurate, and it stops being useful the moment it is set to the build date for every URL.
| Page type | Source of lastmod |
|---|---|
| Blog posts | The post's own updated date, falling back to its publication date |
| Case studies | Same |
| Documentation pages | The page's updated date |
| Static pages | A hand-maintained map of URL to date, bumped in the same change that alters the page |
| Listing pages | The later of the listing page's own date and the newest item it contains |
The listing rule means /blog/ never claims to be older than its newest post, and never claims a change when nothing changed.
What is deliberately excluded
Two categories of URL exist but are kept out of the sitemap rather than advertised:
Redirect stubs. Several pages were consolidated to canonical URLs. The old URLs still resolve, as a courtesy to inbound links and search results that have not caught up, but they are noindex pages that immediately forward. Advertising them in the sitemap would point crawlers at a stale URL that then tells them to go elsewhere, which is a wasted crawl and a confusing signal.
Unlisted pages. A few pages are built so they can be linked directly but are not meant to be discovered: work in progress, and a link page meant for a QR code on a business card. These are noindex and excluded.
Both exclusions are maintained as explicit lists rather than inferred, so removing a page from the sitemap is a deliberate act that shows up in a diff.
If you are crawling this site
The sitemap is the complete list of what is meant to be indexed. If a URL is not in it, either it is one of the two categories above or something is broken, and in the second case I would like to hear about it through the contact page.
Last updated .