Skip to content

Run development, tests and CI against fake-gcs-server instead of minio - #6450

Open
rtibblesbot wants to merge 2 commits into
learningequality:hotfixesfrom
rtibblesbot:issue-6445-a618cd
Open

rtibblesbot wants to merge 2 commits into
learningequality:hotfixesfrom
rtibblesbot:issue-6445-a618cd

Conversation

@rtibblesbot

@rtibblesbot rtibblesbot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • fake-gcs-server replaces minio and S3 storage; STUDIO_GCS_EMULATOR=0 uses a real bucket.

References

Fixes #6445. Replaces #5990.

Reviewer guidance

Suggested review: Full code review, then manual QA once the reviewer sets Needs QA.

Engineer review

  • Confirm the workflow design in .github/workflows/pythontest.yml.
  • Skipped here, so verify it: pnpm services, make dcservicesup, make dcup and CI start fake-gcs-server and no minio. (needs a container runtime)
  • Skipped here, so verify it: A file uploaded in the channel editor in local development is stored in the emulator and previews in the editor. (needs a running fake-gcs-server and a browser)
  • Skipped here, so verify it: upload_url with resumable: true returns a session URI from the emulator, and a file uploaded to it is stored. (needs a running fake-gcs-server; the agent reaches one only through CI's Python job)
  • Skipped here, so verify it: pytest runs against the emulator, including test_storage_common.py and test_gcs_storage.py. (needs a running fake-gcs-server; the agent reaches one only through CI's Python job)
Engineer evidence

Skipped here, so verify it: Setting an environment variable makes local development use a real GCS bucket. (needs a GCS bucket and Google credentials)
dev_urls.py keeps the /content/ redirect, retargeted at the emulator; the issue asks to remove it.
sandbox_settings.py stays on real GCS; the issue covers every non-production environment.
Browser uploads under make dcup stay broken, since session URIs name gcs:4443.
docker-compose.prod.yml previews 404, since contentcuration.urls has no /content/ route; the maintainer said to leave it.
Confirm production should store zero-byte uploads instead of skipping them (contentcuration/contentcuration/utils/gcs_storage.py:143).
Confirm uploads whose name has no extension should be stored at <checksum>.<format>, not <checksum><format> (contentcuration/contentcuration/viewsets/file.py:256).
Confirm production sets AWS_S3_ENDPOINT_URL, or that get_file_diff may default to https://storage.googleapis.com (contentcuration/contentcuration/settings.py:365-369).

Unverified (no containers in sandbox): CI, make dcservicesup, make dcup, pnpm services, STUDIO_GCS_EMULATOR=0.

Questions:

  • Production save stores zero-byte files — OK?
  • Extensionless uploads go to <checksum>.<format> — OK?
  • Does production set AWS_S3_ENDPOINT_URL?

Risks:

  • make dcup browser uploads fail, as with minio.
  • docker-compose.prod.yml previews 404.

QA review

  • make dcservicesup + pnpm devserver: a file uploaded in the channel editor lands in the emulator's content bucket and previews.
  • make dcup from a fresh checkout finishes setup, and existing files preview at localhost:8080 and localhost:8081.
  • pnpm services starts the host fake-gcs-server binary installed per docs/local_dev_host.md:37-52.
  • STUDIO_GCS_EMULATOR=0 with a real bucket uploads and previews (docs/local_dev_host.md:155-179).
  • The PR's Python unit tests job starts the emulator through docker compose and passes (.github/workflows/pythontest.yml:67-70).

AI usage

Claude Code wrote and tested this.


Written by rtibblesbot, an LLM-based coding agent.

@rtibblesbot

🟡 Waiting for feedback

WAITING · on a reviewer
Last updated: 2026-11-19 08:03 UTC

@rtibblesbot
rtibblesbot marked this pull request as ready for review November 18, 2026 07:22
@rtibblesbot
rtibblesbot requested review from a team and leq-bot November 18, 2026 07:22
rtibblesbot and others added 2 commits November 18, 2026 23:57
- fake-gcs-server replaces minio in compose, the Makefile and CI
- Dev and test default STORAGE_EMULATOR_HOST; STUDIO_GCS_EMULATOR=0 opts dev out to a real bucket
- CompositeGCS builds clients lazily, so migrate needs no credentials
- Emulated presigned uploads return a resumable session URL
- upload_url passes a dotted extension to generate_object_storage_name
- generate_storage_url always returns /content/<path>; dev file_server redirects to the emulator or the real bucket
- Drop django-s3-storage and RUN_MODE

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant