ci: finish switch day on Forgejo: artifact actions, two sized runners, front tests within memory #574

Merged

admin-amaurycoudr merged 5 commits from ci/forgejo-runner-sizing into main 2026-10-22 16:02:37 +00:00

Conversation 3 Commits 5 Files changed 24 +437 -179

admin-amaurycoudr commented 2026-10-22 12:48:36 +00:00

Owner

Copy link

Stacked on #573: merge it first, then rebase this onto main.

Summary

Two runners on the CI VPS (15 GiB, 4 CPUs), each taking one job at a time. ADR 0050 supersedes 0048.

 CI VPS
-└── runner (docker)         3 GiB, swap allowed, 1 GiB shm, 4 CPUs   every job
+├── runner (docker)         6 GiB, no swap, 2 GiB shm, 4 CPUs   checks, front-test, e2e,
+│                           + ci-pnpm-store volume              build-workspace, preview-screenshots, renovate
+└── runner-deploy (deploy)  2 GiB, no swap, 2 CPUs             plan, reuse-image, build-image, preview,
+                                                               production, comment, preview-close/reap,
+                                                               backup, audit

• Option 1, the limits. front-test peaks near 4 GiB (measured in a container, memory.peak 3.9 GiB with --maxWorkers=1), so GitLab's 3 GiB was below its working set. That fits run 3's symptoms: 2.999 of 3 GiB, the host swapping, 25 tests timing out and a dynamic import failing. --memory-swap now equals --memory, so a job at its cap is OOM-killed at once instead of swapping until its tests time out. /dev/shm goes to 2 GiB; it is tmpfs charged to the same cap.
  • Kept --maxWorkers=1. I have no evidence that the MSW cause is not real. GitLab pipeline 315's symptoms (whole files failing, a setup file that would not load) match memory pressure as well as they match MSW. Once a few front-test runs pass at 6 GiB, trying --maxWorkers=2 is the next step.
  • Kept TASK_CONCURRENCY=2. checks with a cold Turbo cache took 34 s at concurrency 4 and 36 s at 2, both peaking at 4.3 to 4.5 GiB. Vitest's workers dominate that peak, not Turbo's slots, so the higher setting bought nothing measurable.
• Option 2, two jobs at once. Forgejo has no job-level lock, so each runner's capacity: 1 is the lock. Every job that reaches the production VPS runs on deploy, so two deploys never overlap. Every browser job runs on docker, so two browser jobs never share the CPUs. A deploy no longer waits behind a pull request's gate. runner-lanes.test.mjs enforces this: every job runs on a label a runner offers, deploys and browser jobs each stay on one capacity-1 runner, and the caps fit an 8 GiB budget with no swap.
• Option 3, cheaper setup.
  • The pnpm store moves to a volume. actions/cache is gone from ci.yml and the store sits on the ci-pnpm-store volume at /pnpm-store. GitLab rejected this because pnpm copies across mounts. I measured again: copying costs about 2 s an install, against a restore and a gzip upload on every lockfile change. The numbers are under Evidence.
  • Gate jobs skip on main. On a push to main whose pull request passed the gate, checks, front-test and e2e now skip at job level instead of starting three containers that only echo. No status is required on a push. reuse-image accepts success or skipped from them.
  • Not done: a prebuilt CI image. It would save corepack enable and the Playwright image pull, which is cached anyway. That is a few seconds, not worth an image to rebuild on every Node or Playwright bump. Left as a follow-up.

Memory budget, with two jobs running:

	GiB
Job caps (gate 6 + deploy 2)	8
GitLab, until switch-day step 12	~3
e2e's Postgres + Mailpit (outside the cap), an image build in the daemon	~1.5
Forgejo, Traefik, runners, dockerd, OS	~1.5
Total of 15	~14

After step 12, the 3 GiB GitLab frees stays as page cache and margin.

Evidence

Measured locally in node:24.21.0-bookworm at --cpus=4, with this repo's lockfile (store 1.3 GB, 40 785 files):

Step	Time
Store in the workspace: gzip restore (actions/cache without zstd, which the image lacks)	5.0 s, plus the download
Store in the workspace: install, hardlinked	1.8 s
Store in the workspace: gzip save on a new lockfile	24.7 s (57 s on the VPS)
Store on a volume: install, copied	3.6 to 4.1 s, nothing to restore or save
zstd save / restore, for comparison	1.2 s / 1.1 s, after a 2.5 s apt install zstd in every job

Peak memory (memory.peak), same container:

• front-test, --maxWorkers=1: 3.9 GiB, above the old 3 GiB cap.
• checks: 4.3 GiB at TASK_CONCURRENCY=4 and 4.5 GiB at 2, with Vitest seeing the Mac VM's 15 CPUs. On the VPS it sees 4, so the real peak is lower.

The local front-test run itself was not a clean signal: Chromium dropped its connection, and the Docker VM had only 7.75 GiB with other stacks using about 3.3 GiB.

runner-lanes.test.mjs
  before: fails, there is no runner-deploy.yaml and every job runs on the one runner
  after:  5 pass; moving `preview` back to `docker` fails "the deploys run on runner.yaml and runner-deploy.yaml"

Gate: pnpm lint, typecheck, test (test:infra 59 pass), build, format:check all green.

Merge Danger

Door: two-way

Revert the commit, then rerun stages 3 and 4. up.sh will then render the single runner again. The deploy runner stays registered in Forgejo until you delete it under Site administration → Actions → Runners.

Blast Radius: CI

• Until the apply below: the jobs labeled deploy find no runner and wait. Merging this without applying blocks every deploy, and plan, and so every run. Apply right after merging.
• Job-level skips on main are untested on Forgejo. If reuse-image were skipped despite gate jobs that skipped, production would not deploy on a merge. Check the first push to main after this lands.
• Shared pnpm store. Like the Turbo volume, a same-repository pull request's job could tamper with it; it is already root on the host's daemon.
• CPU overlap. Both runners' CPU caps add up to 6 of 4 CPUs. A docker build in the daemon during front-test could slow it. Watch the first runs that overlap.

Apply on the server (the owner)

1. Merge #573, then this. Check out main.
2. Wait until no job runs: Site administration → Actions → Runners shows the runner idle. up.sh restarts a runner whose config changed, which kills its job.
3. infra/forgejo/vps/wizard.sh. Answer "Redo it?" with yes for stage 3 (copies the stack and the new runner-deploy.yaml; the secret key and internal token are kept) and stage 4 (up.sh: registers forgejo-vps-deploy-runner, renders both configs, restarts runner, starts runner-deploy). Answer no for every other stage.
4. Check that sudo docker compose -f /opt/forgejo/compose.yaml ps lists runner and runner-deploy running. Check that Site administration → Actions → Runners lists both idle, with labels docker,ubuntu-latest and deploy.
5. Check that sudo grep -E 'capacity|memory|volume' /opt/forgejo/runner/config.yaml /opt/forgejo/runner-deploy/config.yaml shows --memory=6g --memory-swap=6g, ci-pnpm-store, and --memory=2g --memory-swap=2g.

These steps are also in infra/forgejo/vps/README.md, "Apply a change to the runners". The first gate run fills ci-pnpm-store, which takes about 15 s more than a warm install.

Stacked on #573: merge it first, then rebase this onto `main`. ## Summary Two runners on the CI VPS (15 GiB, 4 CPUs), each taking one job at a time. ADR 0050 supersedes 0048. ```diff CI VPS -└── runner (docker) 3 GiB, swap allowed, 1 GiB shm, 4 CPUs every job +├── runner (docker) 6 GiB, no swap, 2 GiB shm, 4 CPUs checks, front-test, e2e, +│ + ci-pnpm-store volume build-workspace, preview-screenshots, renovate +└── runner-deploy (deploy) 2 GiB, no swap, 2 CPUs plan, reuse-image, build-image, preview, + production, comment, preview-close/reap, + backup, audit ``` - **Option 1, the limits.** front-test peaks near 4 GiB (measured in a container, `memory.peak` 3.9 GiB with `--maxWorkers=1`), so GitLab's 3 GiB was below its working set. That fits run 3's symptoms: 2.999 of 3 GiB, the host swapping, 25 tests timing out and a dynamic import failing. `--memory-swap` now equals `--memory`, so a job at its cap is OOM-killed at once instead of swapping until its tests time out. `/dev/shm` goes to 2 GiB; it is tmpfs charged to the same cap. - **Kept `--maxWorkers=1`.** I have no evidence that the MSW cause is not real. GitLab pipeline 315's symptoms (whole files failing, a setup file that would not load) match memory pressure as well as they match MSW. Once a few front-test runs pass at 6 GiB, trying `--maxWorkers=2` is the next step. - **Kept `TASK_CONCURRENCY=2`.** checks with a cold Turbo cache took 34 s at concurrency 4 and 36 s at 2, both peaking at 4.3 to 4.5 GiB. Vitest's workers dominate that peak, not Turbo's slots, so the higher setting bought nothing measurable. - **Option 2, two jobs at once.** Forgejo has no job-level lock, so each runner's `capacity: 1` is the lock. Every job that reaches the production VPS runs on `deploy`, so two deploys never overlap. Every browser job runs on `docker`, so two browser jobs never share the CPUs. A deploy no longer waits behind a pull request's gate. `runner-lanes.test.mjs` enforces this: every job runs on a label a runner offers, deploys and browser jobs each stay on one capacity-1 runner, and the caps fit an 8 GiB budget with no swap. - **Option 3, cheaper setup.** - **The pnpm store moves to a volume.** `actions/cache` is gone from ci.yml and the store sits on the `ci-pnpm-store` volume at `/pnpm-store`. GitLab rejected this because pnpm copies across mounts. I measured again: copying costs about 2 s an install, against a restore and a gzip upload on every lockfile change. The numbers are under Evidence. - **Gate jobs skip on main.** On a push to main whose pull request passed the gate, checks, front-test and e2e now skip at job level instead of starting three containers that only echo. No status is required on a push. `reuse-image` accepts `success` or `skipped` from them. - **Not done: a prebuilt CI image.** It would save `corepack enable` and the Playwright image pull, which is cached anyway. That is a few seconds, not worth an image to rebuild on every Node or Playwright bump. Left as a follow-up. **Memory budget, with two jobs running:** | | GiB | | --- | --- | | Job caps (gate 6 + deploy 2) | 8 | | GitLab, until switch-day step 12 | ~3 | | e2e's Postgres + Mailpit (outside the cap), an image build in the daemon | ~1.5 | | Forgejo, Traefik, runners, dockerd, OS | ~1.5 | | **Total of 15** | **~14** | After step 12, the 3 GiB GitLab frees stays as page cache and margin. ## Evidence Measured locally in `node:24.21.0-bookworm` at `--cpus=4`, with this repo's lockfile (store 1.3 GB, 40 785 files): | Step | Time | | --- | --- | | Store in the workspace: gzip restore (`actions/cache` without zstd, which the image lacks) | 5.0 s, plus the download | | Store in the workspace: install, hardlinked | 1.8 s | | Store in the workspace: gzip save on a new lockfile | 24.7 s (57 s on the VPS) | | Store on a volume: install, copied | 3.6 to 4.1 s, nothing to restore or save | | zstd save / restore, for comparison | 1.2 s / 1.1 s, after a 2.5 s `apt install zstd` in every job | Peak memory (`memory.peak`), same container: - front-test, `--maxWorkers=1`: **3.9 GiB**, above the old 3 GiB cap. - checks: 4.3 GiB at `TASK_CONCURRENCY=4` and 4.5 GiB at 2, with Vitest seeing the Mac VM's 15 CPUs. On the VPS it sees 4, so the real peak is lower. The local front-test run itself was not a clean signal: Chromium dropped its connection, and the Docker VM had only 7.75 GiB with other stacks using about 3.3 GiB. ```text runner-lanes.test.mjs before: fails, there is no runner-deploy.yaml and every job runs on the one runner after: 5 pass; moving `preview` back to `docker` fails "the deploys run on runner.yaml and runner-deploy.yaml" ``` Gate: `pnpm lint`, `typecheck`, `test` (test:infra 59 pass), `build`, `format:check` all green. ## Merge Danger **Door:** two-way Revert the commit, then rerun stages 3 and 4. `up.sh` will then render the single runner again. The deploy runner stays registered in Forgejo until you delete it under Site administration → Actions → Runners. **Blast Radius:** CI - **Until the apply below:** the jobs labeled `deploy` find no runner and wait. Merging this without applying blocks every deploy, and `plan`, and so every run. Apply right after merging. - **Job-level skips on main are untested on Forgejo.** If `reuse-image` were skipped despite gate jobs that skipped, production would not deploy on a merge. Check the first push to main after this lands. - **Shared pnpm store.** Like the Turbo volume, a same-repository pull request's job could tamper with it; it is already root on the host's daemon. - **CPU overlap.** Both runners' CPU caps add up to 6 of 4 CPUs. A `docker build` in the daemon during front-test could slow it. Watch the first runs that overlap. ### Apply on the server (the owner) 1. Merge #573, then this. Check out `main`. 2. Wait until no job runs: Site administration → Actions → Runners shows the runner idle. `up.sh` restarts a runner whose config changed, which kills its job. 3. `infra/forgejo/vps/wizard.sh`. Answer "Redo it?" with **yes** for stage 3 (copies the stack and the new `runner-deploy.yaml`; the secret key and internal token are kept) and stage 4 (`up.sh`: registers `forgejo-vps-deploy-runner`, renders both configs, restarts `runner`, starts `runner-deploy`). Answer **no** for every other stage. 4. Check that `sudo docker compose -f /opt/forgejo/compose.yaml ps` lists `runner` and `runner-deploy` running. Check that Site administration → Actions → Runners lists both idle, with labels `docker,ubuntu-latest` and `deploy`. 5. Check that `sudo grep -E 'capacity|memory|volume' /opt/forgejo/runner/config.yaml /opt/forgejo/runner-deploy/config.yaml` shows `--memory=6g --memory-swap=6g`, `ci-pnpm-store`, and `--memory=2g --memory-swap=2g`. These steps are also in `infra/forgejo/vps/README.md`, "Apply a change to the runners". The first gate run fills `ci-pnpm-store`, which takes about 15 s more than a warm install.

admin-amaurycoudr added 3 commits 2026-10-22 12:48:36 +00:00

docs(repo): name Forgejo Actions in the ci commit scope ... e6e1162f4e

The ci scope's description still said GitLab CI; the gate and deploys now run as Forgejo
Actions workflows in .forgejo/, as proposed in !68.

Refs #499

ci: upload and download artifacts with Forgejo's patched v4 actions ... 50d5acdf09

The first Forgejo run failed in checks and comment: actions/upload-artifact@v4 and
download-artifact@v4 resolve on data.forgejo.org to GitHub's code, which refuses any host but
github.com (GHESNotSupportedError). code.forgejo.org/forgejo's copies drop that check.

Refs #499

ci: run the gate and the deploys on two runners sized to the CI VPS ... f39a5abfb1

The CI VPS has 15 GiB and 4 CPUs. front-test failed on Forgejo run 3 at GitLab's 3 GiB cap,
swapping until its tests timed out; it peaks near 4 GiB.

- A gate runner (label docker) takes the gate, browser and install jobs: 6 GiB, 2 GiB /dev/shm,
  4 CPUs. A deploy runner (label deploy) takes every job that reaches the production VPS and the
  light ones: 2 GiB, 2 CPUs. Each runs one job at a time, so deploys never overlap and browser
  jobs never share the CPUs. --memory-swap equals --memory, so a capped job fails fast.
- The pnpm store lives on the ci-pnpm-store volume instead of actions/cache: no restore and no
  gzip upload, for about 2 s more per install since pnpm copies across mounts.
- On a push to main whose pull request passed the gate, the gate jobs skip at job level instead
  of starting three containers that only echo.
- runner-lanes.test.mjs checks every workflow job against the two runners and the memory budget.
- ADR 0050 supersedes 0048 with the two runners.

admin-amaurycoudr commented 2026-10-22 12:48:43 +00:00

Author

Owner

Copy link

Order of apply: this pull request's own run needs the deploy runner, since plan now runs on the deploy label. The new runners also serve main's current workflows: the gate runner keeps the docker label and the actions cache. So apply the steps from this branch first (check out ci/forgejo-runner-sizing instead of main in step 1), let this run go green, then merge #573 and this one. Until then, this run waits for a runner.

Order of apply: this pull request's own run needs the deploy runner, since plan now runs on the deploy label. The new runners also serve main's current workflows: the gate runner keeps the docker label and the actions cache. So apply the steps from this branch first (check out ci/forgejo-runner-sizing instead of main in step 1), let this run go green, then merge #573 and this one. Until then, this run waits for a runner.

chess-bot commented 2026-10-22 13:08:09 +00:00

Member

Copy link

Run at e3a051d

The run

Failed tests

No test failed.

Engine coverage

Statements: 99.48%, shown and never enforced.

Screenshots

Before is production, after is this pull request's preview.

Desktop

Page	Before	After
Home
Local game
Room

Mobile

Page	Before	After
Home
Local game
Room

<!-- chess-run --> ### Run at e3a051d [The run](https://git.core-project.space/amaurycoudr/chess-project/actions/runs/15) #### Failed tests No test failed. #### Engine coverage Statements: **99.48%**, shown and never enforced. #### Screenshots Before is production, after is this pull request's preview. ##### Desktop | Page | Before | After | | --- | --- | --- | | Home | <img src="https://git.core-project.space/attachments/46c0e8e1-85d4-457c-b0a4-7d97abdeeb5d" width="400"> | <img src="https://git.core-project.space/attachments/6441f987-80e4-4163-8795-13b418fdfaa1" width="400"> | | Local game | <img src="https://git.core-project.space/attachments/1819c7aa-5c28-4a15-8cb6-2266768cdc59" width="400"> | <img src="https://git.core-project.space/attachments/e2c67f3c-fed3-4c05-a08a-d792e1eb4a37" width="400"> | | Room | <img src="https://git.core-project.space/attachments/bef4e96c-9124-4b9c-90f6-3cb334cc4271" width="400"> | <img src="https://git.core-project.space/attachments/d3999d70-9d8f-46d3-886a-fc489776f201" width="400"> | ##### Mobile | Page | Before | After | | --- | --- | --- | | Home | <img src="https://git.core-project.space/attachments/4a90157b-957c-430c-b3e7-866b13cde5fe" width="200"> | <img src="https://git.core-project.space/attachments/b78da279-9124-4d25-8a66-1bc6cff6a67e" width="200"> | | Local game | <img src="https://git.core-project.space/attachments/652c0b35-9326-44af-a01f-fa2168066dc0" width="200"> | <img src="https://git.core-project.space/attachments/eb956d3e-9c4f-44a0-baa3-16bb20c667da" width="200"> | | Room | <img src="https://git.core-project.space/attachments/24c4bb14-59ee-4e79-b227-c02ca6be13ac" width="200"> | <img src="https://git.core-project.space/attachments/cf4d155a-7dfc-47c7-b90e-d9f12d25f389" width="200"> |

before-home-desktop.png

71 KiB

before-home-mobile.png

146 KiB

before-local-desktop.png

68 KiB

before-local-mobile.png

172 KiB

before-room-desktop.png

67 KiB

before-room-mobile.png

168 KiB

after-home-desktop.png

71 KiB

after-home-mobile.png

146 KiB

after-local-desktop.png

68 KiB

after-local-mobile.png

172 KiB

after-room-desktop.png

67 KiB

after-room-mobile.png

168 KiB

admin-amaurycoudr referenced this pull request 2026-10-22 15:09:58 +00:00

fix(front): run the browser tests on as many pages as --maxWorkers, collecting each one's garbage #575

admin-amaurycoudr added 1 commit 2026-10-22 15:17:49 +00:00

fix(front): run the browser tests on as many pages as --maxWorkers, collecting each one's garbage ... 0d561a208f

Vitest 4.1 hands --maxWorkers to its node pools only, so the browser project opened a page per CPU
whatever the run asked, and each page kept the files it had run as garbage until V8 ran short. In
CI that grew past the job's cap: at 3 GiB the tests starved and timed out, at 6 GiB the kernel
killed Chromium mid-run (Forgejo run 5).

- A plugin hands the run's --maxWorkers to the browser project, and a test pins it.
- A browser command collects the page's garbage before each file.
- MSW's worker starts at the file's first hook rather than on import.

Measured in a container: the whole front suite peaks at about 2.0 GiB, and all 82 files pass;
before, the run stopped after 23 files at 2.4 GiB.

admin-amaurycoudr changed title from ci: run the gate and the deploys on two runners sized to the CI VPS to ci: finish switch day on Forgejo: artifact actions, two sized runners, front tests within memory 2026-10-22 15:18:05 +00:00

admin-amaurycoudr commented 2026-10-22 15:18:06 +00:00

Author

Owner

Copy link

This pull request now combines #573 and #575, which depend on each other: #573's Forgejo artifact actions, this one's two runners (ADR 0050), and #575's front test memory fix (the browser opens as many pages as --maxWorkers and collects each page's garbage). Four commits, gate green locally (front: 82/82 files, 1631/1631 tests). Its merge on main is switch-day step 11, the first production deploy from Forgejo.

This pull request now combines #573 and #575, which depend on each other: #573's Forgejo artifact actions, this one's two runners (ADR 0050), and #575's front test memory fix (the browser opens as many pages as --maxWorkers and collects each page's garbage). Four commits, gate green locally (front: 82/82 files, 1631/1631 tests). Its merge on main is switch-day step 11, the first production deploy from Forgejo.

admin-amaurycoudr referenced this pull request 2026-10-22 15:18:06 +00:00

ci: upload artifacts with Forgejo's actions, and name Forgejo Actions in the ci scope #573

admin-amaurycoudr referenced this pull request 2026-10-22 15:18:06 +00:00

fix(front): run the browser tests on as many pages as --maxWorkers, collecting each one's garbage #575

admin-amaurycoudr added 1 commit 2026-10-22 15:37:55 +00:00

ci: point the runners at Forgejo's public address, so jobs name the registry by it ... e3a051d828

The runners reached Forgejo at http://forgejo:3000, which jobs then saw as their server URL. The
image jobs log in and name images after it, but they go through the host's Docker daemon, which
cannot resolve forgejo: run 8's reuse-image failed its docker login, and production could never
have pulled forgejo:3000/... either.

Both runners now connect to https://<FORGEJO_HOST>/ and, like their job containers, resolve that
name to the host gateway, so the traffic stays on the box through Traefik. Checked on the CI VPS:
a container on the forgejo network gets 200 from /api/healthz through 172.17.0.1 with a valid
certificate, and the host gets the registry's 401 at /v2/.

admin-amaurycoudr merged commit e3a051d828 into main 2026-10-22 16:02:37 +00:00

admin-amaurycoudr deleted branch ci/forgejo-runner-sizing 2026-10-22 16:02:37 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something isn't working

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

question

Further information is requested

  

spec

Parent spec written by to-spec

  

wayfinder:grilling

  

wayfinder:map

  

wayfinder:prototype

  

wayfinder:research

  

wayfinder:task

  

wontfix

This will not be worked on

No labels

bug

documentation

duplicate

enhancement

good first issue

help wanted

invalid

question

spec

wayfinder:grilling

wayfinder:map

wayfinder:prototype

wayfinder:research

wayfinder:task

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

amaurycoudr/chess-project!574

No description provided.