How We Built a Dynamic llms.txt with Hugo

The conversation around how Large Language Models (LLMs) interpret website content is growing. We wanted a way to guide them that was as robust and automated as our sitemap generation. The solution was to create a dynamic llms.txt file directly within our Hugo project, with a process that is automatic, maintainable, and content-driven. Our approach generates the llms.txt file automatically and adds a reference to it in robots.txt, much like the standard Sitemap: definition.

Here’s a step-by-step breakdown of how we did it.

Step 1: Controlling Content with Frontmatter

We needed granular control over what content appears in the llms.txt file. We achieved this using two frontmatter parameters for each page:

sitemap_exclude: We already used sitemap_exclude: true to prevent certain pages from appearing in our sitemap.xml. We reused this same logic to exclude these pages from llms.txt as well, ensuring consistency.
```
---
title: "An old, irrelevant blog post"
sitemap_exclude: true
---
```
ai_description: A meta description is for humans, but a description for an LLM should be more factual and context-rich. We added ai_description to write descriptions specifically for AI, and our template only includes pages that have this parameter defined.
```
--- 
title: "Server-side GTM implementation workflow" 
ai_description: "This document visually outlines the complete workflow for implementing ..." 
---
```

Step 2: The Automation Magic in `robots.txt`

The entire process is triggered from within our robots.txt template (layouts/robots.txt). The primary reason for this approach is to gain access to Hugo’s page-aware variables like .Permalink. This allows us to store our llms.txt template in the /assets directory and have Hugo process it correctly, which wouldn’t be possible if it were in /static. Here is the code snippet from our robots.txt template:

{{/* Generate and link to llms.txt */}}
{{- $llmsGoTXT := resources.Get "assets/llms.go.txt" -}}
{{- if and $llmsGoTXT .Site.Params.robots.llmsTXT -}}
  {{- $llmsTXT := $llmsGoTXT | resources.ExecuteAsTemplate "llms.txt" . -}}
  llms-txt: {{ $llmsTXT.Permalink }}
{{- end -}}

This code does two key things:

It uses resources.ExecuteAsTemplate to dynamically generate the llms.txt file from a source template.
It then adds a line to the final robots.txt file: llms-txt: https://example.com/llms.txt, pointing crawlers to the file’s location, similar to how the Sitemap: directive works.

Step 3: The `llms.go.txt` Resource Template

The template for our llms.txt file is located at /assets/llms.go.txt. It contains the logic for building the file’s structure. Here is how it filters our pages using the frontmatter parameters we defined:

{{/* Loop through pages, respecting both sitemap_exclude and ai_description */}}
{{- range (where (sort .Site.Pages "Weight") "Params.sitemap_exclude" "ne" true) -}}
    {{- if .Params.ai_description -}}
        - [{{ .Title }}]({{ .Permalink }}): {{ .Params.ai_description }}
    {{- end -}}
{{- end -}}

Step 4: Highlighting Expertise with Custom Data

Finally, to enrich our llms.txt with more context, we used Hugo’s data files to add information about our team and partners. The template ranges over files like /data/team.toml and /data/partners.toml to add sections for “Core team” and “Partners,” emphasizing the human expertise behind our content.

The Result

This Hugo-native approach gives us a powerful, automated system. By reusing existing frontmatter like sitemap_exclude and adding a dedicated ai_description, we have precise control over the content. The generation process via robots.txt is efficient and ensures all our links are correctly rendered, resulting in a low-maintenance, high-impact llms.txt file.

Go home