Skip to content

Fix robots.txt, sitemaps, and noindex canonical tags - #14101

Open
kevmoo wants to merge 4 commits into
mainfrom
gsc-seo
Open

kevmoo wants to merge 4 commits into
mainfrom
gsc-seo

Conversation

@kevmoo

@kevmoo kevmoo commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Publishes /robots.txt on flutter.dev, omits canonical tags on noindex
pages, and populates sitemap lastmod dates from Git history.

  • Shared robots.txt and Git <lastmod>: Serves /robots.txt on
    flutter.dev (Sitemap: on production, Disallow: / on staging) and
    derives flutter.dev/sitemap.xml <lastmod> dates from Git history (25
    distinct commit dates instead of a shared build timestamp), failing closed on
    shallow production clones.
  • Canonical and noindex consistency: Adds
    <meta name="robots" content="noindex"> support to DefaultLayout, omits
    default <link rel="canonical"> when noindex or canonical: false (404,
    /search, /search-all), and resolves relative canonical overrides against
    site.url.
  • Sitemaps and /flutter-for index: Excludes 404.html, /blog/feed.xml,
    /search, and /search-all from sitemap.xml (703 docs and 397 www
    URLs, both sitemap.xsd-valid), filters noindex pages from TocLayout, and
    enables layout: toc on /flutter-for.

Presubmit checklist

  • If you are unwilling, or unable, to sign the CLA, even for a tiny, one-word PR, please file an issue instead of a PR.
  • If this PR is not meant to land until a future stable release, mark it as draft with an explanation.
  • This PR follows the Google Developer Documentation Style Guidelines—for example, it doesn't use i.e. or e.g., and it avoids I and we (first-person pronouns).
  • This PR uses semantic line breaks
    of 80 characters or fewer.

- Promote `RobotsTxtOutput` and Git `<lastmod>` computation (`LastModifiedLoader`
  with fail-closed shallow-clone validation on production builds) to
  `package:site_shared/seo.dart` and wire both into `sites/docs` and `sites/www`
  so `https://flutter.dev/robots.txt` serves `200` `text/plain` with
  `Sitemap: https://flutter.dev/sitemap.xml` and `flutter.dev/sitemap.xml`
  emits Git modification dates.
- Suppress default `<link rel="canonical">` when `noindex` or `canonical: false`
  is set, resolve relative `canonical:` overrides against `site.url`, and render
  `<meta name="robots" content="noindex">` in `DefaultLayout`.
- Add `noindex: true` and `sitemap: false` on `404`, `search`, and `search-all`
  routes across both sites; exclude `noindex` pages from `sitemap.xml` and
  `TocLayout`; mark `BlogAtomFeedOutput` and `RobotsTxtOutput`
  `jaspr-sitemap-data: false` alongside `--sitemap-exclude=\.(md|txt|xml|json)$`;
  and configure `flutter-for/index.md` with `layout: toc`.
- Fix broken redirects, single-segment globs, 2-to-3 hop chains, cache globs,
  and XML `X-Robots-Tag: noindex` headers across `sites/docs/firebase.json`,
  `sites/www/firebase.json`, and `sites/blog_redirects/firebase.json`.
- Expand 1,197 unrendered `{{site.*}}` URLs in `skipTemplateRendering: true`
  release notes and update internal redirecting links across `sites/docs` and
  `sites/www/content/blog`.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request centralizes and standardizes server-side SEO utilities across the website workspaces. It introduces a shared site_shared/seo.dart library containing GitLastModifiedIndex, LastModifiedLoader, and RobotsTxtOutput to manage robots.txt generation, canonical URL resolution, and Git-based last-modified metadata. It also updates sitemap exclusions for pages marked with noindex (such as 404 and search pages), refactors layout files to utilize these shared utilities, updates numerous redirect configurations in firebase.json files, and adds a comprehensive test suite in seo_canonical_robots_test.dart. No review comments were provided, so there is no feedback to evaluate.

@flutter-website-bot

flutter-website-bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Staged preview of the updated docs.flutter.dev site (updated for commit f5e1743):

https://flutter-docs-prod--docs-pr14101-gsc-seo-gz31wssx.web.app

@flutter-website-bot

flutter-website-bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Staged preview of the updated flutter.dev site (updated for commit f5e1743):

https://flutter-dev-230821--www-pr14101-gsc-seo-xociidyy.web.app

Narrows this branch to Jaspr SEO metadata, robots.txt generation,
noindex canonical suppression, and Git-based sitemap lastmod dates.
@kevmoo kevmoo changed the title Fix robots.txt, sitemaps, noindex canonicals, and redirects Fix robots.txt, sitemaps, and noindex canonical tags Nov 19, 2026
- Resolve relative `FilesystemLoader.directory` paths against `projectRoot`
  instead of `path.current` in `GitLastModifiedIndex._resolvePagePaths`.
- Normalize paths with `path.normalize(path.absolute(...))` instead of
  `path.canonicalize(...)` so mixed-case content filenames retain their Git
  index and GitHub source-URL casing on Windows.
- Short-circuit `loadProjectLastModifiedDates` before running
  `rev-parse --is-shallow-repository` when `rev-parse --show-toplevel` fails.
- Add `@TestOn('vm')` and `library;` to `sites/www/test/seo_canonical_robots_test.dart`
  so `dart test -p chrome` in `sites/www` skips server-only `dart:io` and
  `package:jaspr/server.dart` tests cleanly.
@parlough parlough self-assigned this Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants