Repository navigation
Conversation
- Promote `RobotsTxtOutput` and Git `<lastmod>` computation (`LastModifiedLoader` with fail-closed shallow-clone validation on production builds) to `package:site_shared/seo.dart` and wire both into `sites/docs` and `sites/www` so `https://flutter.dev/robots.txt` serves `200` `text/plain` with `Sitemap: https://flutter.dev/sitemap.xml` and `flutter.dev/sitemap.xml` emits Git modification dates. - Suppress default `<link rel="canonical">` when `noindex` or `canonical: false` is set, resolve relative `canonical:` overrides against `site.url`, and render `<meta name="robots" content="noindex">` in `DefaultLayout`. - Add `noindex: true` and `sitemap: false` on `404`, `search`, and `search-all` routes across both sites; exclude `noindex` pages from `sitemap.xml` and `TocLayout`; mark `BlogAtomFeedOutput` and `RobotsTxtOutput` `jaspr-sitemap-data: false` alongside `--sitemap-exclude=\.(md|txt|xml|json)$`; and configure `flutter-for/index.md` with `layout: toc`. - Fix broken redirects, single-segment globs, 2-to-3 hop chains, cache globs, and XML `X-Robots-Tag: noindex` headers across `sites/docs/firebase.json`, `sites/www/firebase.json`, and `sites/blog_redirects/firebase.json`. - Expand 1,197 unrendered `{{site.*}}` URLs in `skipTemplateRendering: true` release notes and update internal redirecting links across `sites/docs` and `sites/www/content/blog`.
There was a problem hiding this comment.
Code Review
This pull request centralizes and standardizes server-side SEO utilities across the website workspaces. It introduces a shared site_shared/seo.dart library containing GitLastModifiedIndex, LastModifiedLoader, and RobotsTxtOutput to manage robots.txt generation, canonical URL resolution, and Git-based last-modified metadata. It also updates sitemap exclusions for pages marked with noindex (such as 404 and search pages), refactors layout files to utilize these shared utilities, updates numerous redirect configurations in firebase.json files, and adds a comprehensive test suite in seo_canonical_robots_test.dart. No review comments were provided, so there is no feedback to evaluate.
|
Staged preview of the updated docs.flutter.dev site (updated for commit f5e1743): https://flutter-docs-prod--docs-pr14101-gsc-seo-gz31wssx.web.app |
|
Staged preview of the updated flutter.dev site (updated for commit f5e1743): https://flutter-dev-230821--www-pr14101-gsc-seo-xociidyy.web.app |
Narrows this branch to Jaspr SEO metadata, robots.txt generation, noindex canonical suppression, and Git-based sitemap lastmod dates.
- Resolve relative `FilesystemLoader.directory` paths against `projectRoot` instead of `path.current` in `GitLastModifiedIndex._resolvePagePaths`. - Normalize paths with `path.normalize(path.absolute(...))` instead of `path.canonicalize(...)` so mixed-case content filenames retain their Git index and GitHub source-URL casing on Windows. - Short-circuit `loadProjectLastModifiedDates` before running `rev-parse --is-shallow-repository` when `rev-parse --show-toplevel` fails.
- Add `@TestOn('vm')` and `library;` to `sites/www/test/seo_canonical_robots_test.dart`
so `dart test -p chrome` in `sites/www` skips server-only `dart:io` and
`package:jaspr/server.dart` tests cleanly.
Publishes
/robots.txtonflutter.dev, omits canonical tags onnoindexpages, and populates sitemap
lastmoddates from Git history.robots.txtand Git<lastmod>: Serves/robots.txtonflutter.dev(Sitemap:on production,Disallow: /on staging) andderives
flutter.dev/sitemap.xml<lastmod>dates from Git history (25distinct commit dates instead of a shared build timestamp), failing closed on
shallow production clones.
noindexconsistency: Adds<meta name="robots" content="noindex">support toDefaultLayout, omitsdefault
<link rel="canonical">whennoindexorcanonical: false(404,/search,/search-all), and resolves relativecanonicaloverrides againstsite.url./flutter-forindex: Excludes404.html,/blog/feed.xml,/search, and/search-allfromsitemap.xml(703docsand397wwwURLs, both
sitemap.xsd-valid), filtersnoindexpages fromTocLayout, andenables
layout: tocon/flutter-for.Presubmit checklist
of 80 characters or fewer.