Creating and hosting sitemaps in Elixir/Phoenix

Recently, we implemented sitemap generation for ElixirDrops, so I thought I’d share our approach.

Since we don’t store sitemaps in git and hosting providers usually have ephemeral filesystems, our sitemap is removed on every deploy. Therefore, we need to regenerate it on each deploy. But after that initial creation, we don’t regenerate it entirely whenever content changes.

Even as the world shifts from search engines to LLMs, good SEO still matters. Why miss out on free organic traffic?

Smart Update Strategy

The key insight is avoiding full regeneration for every change. Check if the sitemap exists, then either update incrementally or generate from scratch:

def generate(item) do
  sitemap_path = get_sitemap_path()
  
  if File.exists?(sitemap_path) do
    update_sitemap_with_item(item, sitemap_path)
  else
    generate_full_sitemap(sitemap_path)
  end
end

defp get_sitemap_path do
  Path.join([:code.priv_dir(:my_app), "static", "sitemap.xml"])
end

defp update_sitemap_with_item(item, path) do
  with {:ok, content} <- File.read(path),
       updated_content <- update_or_append_item(content, item),
       :ok <- File.write(path, updated_content) do
    {:ok, path}
  end
end

defp update_or_append_item(content, item) do
  item_url = url(~p"/items/#{item.id}")
  item_xml = item_to_xml(item)
  
  if String.contains?(content, item_url) do
    # Update existing entry
    pattern = ~r(<url>\s*<loc>#{Regex.escape(item_url)}</loc>.*?</url>)s
    String.replace(content, pattern, item_xml)
  else
    # Append new entry
    String.replace(content, "</urlset>", "#{item_xml}\n</urlset>")
  end
end

Core Sitemap Generation

Now let’s look at the actual XML generation that follows the sitemap protocol:

defmodule MyApp.Sitemap do
  use MyAppWeb, :verified_routes
  
  def generate_sitemap_content(items) do
    """
    <?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url>
      <loc>#{escape_xml(url(~p"/"))}</loc>
      <changefreq>daily</changefreq>
      <priority>1.0</priority>
    </url>
    #{generate_items(items)}
    </urlset>
    """
  end
  
  defp generate_items(items) do
    Enum.map_join(items, "\n", &item_to_xml/1)
  end
  
  defp item_to_xml(item) do
    """
    <url>
      <loc>#{escape_xml(url(~p"/items/#{item.id}"))}</loc>
      <lastmod>#{format_date(item.updated_at)}</lastmod>
      <changefreq>monthly</changefreq>
      <priority>0.8</priority>
    </url>
    """
  end
  
  defp format_date(datetime) do
    datetime
    |> NaiveDateTime.to_date()
    |> Date.to_iso8601()
  end
end

Essential XML Escaping

Never forget to escape special characters.

defp escape_xml(string) do
  string
  |> String.replace("&", "&")
  |> String.replace("<", "&lt;")
  |> String.replace(">", "&gt;")
  |> String.replace("\"", "&quot;")
  |> String.replace("'", "&apos;")
end

XML validators will reject malformed sitemaps. Test yours at xml-sitemaps.com/validate-xml-sitemap.html or Google Search Console.

Static File Hosting

Make your sitemap accessible by adding it to Phoenix static paths:

# lib/my_app_web.ex
def static_paths do
  ~w(assets fonts images favicon.ico robots.txt sitemap.xml)
end

Place the generated sitemap in priv/static/sitemap.xml for automatic serving. Don’t forget to add it to .gitignore!

Background Processing with Oban

Never block web requests - use Oban for async generation:

defmodule MyApp.SitemapWorker do
  use Oban.Worker, queue: :seo
  
  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"item_id" => item_id}}) do
    case MyApp.Items.get_item(item_id) do
      %Item{} = item -> MyApp.Sitemap.generate(item)
      nil -> {:error, "Item not found"}
    end
  end

  def perform(%Oban.Job{args: %{}}) do
    MyApp.Sitemap.generate_full()
  end
end

# Enqueue updates after changes
%{"item_id" => item.id}
|> MyApp.SitemapWorker.new()
|> Oban.insert()

Handling Ephemeral Filesystems

Since platforms like Fly.io don’t persist filesystem changes between deploys, regenerate on startup:

defmodule MyApp.Release do
  def generate_sitemap do
    Application.ensure_all_started(:my_app)

    %{}
    |> MyApp.SitemapWorker.new(schedule_in: 60)
    |> Oban.insert()
  end
end

# In your deployment script, e.g. `rel/overlays/bin/migrate`
./my_app eval 'MyApp.Release.migrate && MyApp.Release.generate_sitemap'