Creating a Text Sitemap and robots.txt With Eleventy

A sitemap is a comprehensive list of a web site’s URLs. Sitemaps help search engines crawl the site’s content and are particularly useful if the site is new or has few external links to it.

Many tutorials recommend creating an XML sitemap that includes extra attributes about each URL:

1. lastmod is the date/time the content at the URL was last updated. It doesn’t replace the If-Modified-Since HTML header that’s used by search engines to check if the content has changed since they last crawled the site, so it’s redundant.
2. changefreq is a hint for search engines about how often a URL’s content changes. It is a hint and not a command, so provides little to no value.
3. priority describes the URL’s priority relative to other URLs on the site. Priority isn’t used by search engines, as Google’s Search Console Help calls out: Google does not currently consume the priority attribute in sitemaps.

Simplify

Given that the extra attributes are redundant, provide no value and aren’t consumed by search engines, then all that matters is the URLs. Sitemaps don’t have to be XML, so we can list the URLs in a plain text file following these guidelines:

• The text file must have one URL per line. The URLs cannot contain embedded new lines.
• URLs are fully specified, including the protocol.
• The text file should contain no information other than the list of URLs.
• The text file should contain no header or footer information.
• We can name the text file whatever we like.

Whatever we like

If we can name the text file whatever we like, how do search engines find it? Using the Sitemap directive in the site’s robots.txt file.

Here’s what we need to generate our sitemap and robots.txt file using Eleventy.

_data/site.json to define our site’s base URL and the name of our sitemap file.

{
  "baseUrl": "https://example.com",
  "robots": "/robots.txt",
  "siteMap": "/sitemap.txt"
}

robots.txt.njk as the template for our robots.txt file.

---
eleventyExcludeFromCollections: true
eleventyComputed:
  permalink: "{{ site.robots }}"
---
Sitemap: {{ site.baseUrl + site.siteMap }}

User-agent: *
Disallow:

sitemap.11ty.js as the template for our sitemap.

class SiteMap {
  data() {
    return {
      eleventyExcludeFromCollections: true,
      permalink: function (data) {
        return data.site.siteMap;
      },
    };
  }

  render({ collections: { all: content }, site: { baseUrl } }) {
    const urls = content.map((c) => `${baseUrl}${c.url}`);
    urls.sort();
    return urls.join("\n");
  }
}

module.exports = SiteMap;