amaurycoudr/chess-project
2
0
Fork 0
Code Issues 5 Pull requests 2 Projects Releases Packages Activity Actions

ci: run the gate and the deploys on two runners sized to the CI VPS #574

Open
admin-amaurycoudr wants to merge 3 commits from ci/forgejo-runner-sizing into main
pull from: ci/forgejo-runner-sizing
merge into: amaurycoudr:main
Conversation 2 Commits 3 Files changed 20 +353 -173
admin-amaurycoudr commented 2026-10-22 12:48:36 +00:00
Owner
Copy link

Stacked on #573: merge it first, then rebase this onto main.

Summary

Two runners on the CI VPS (15 GiB, 4 CPUs), each taking one job at a time. ADR 0050 supersedes 0048.

 CI VPS
-└── runner (docker)         3 GiB, swap allowed, 1 GiB shm, 4 CPUs   every job
+├── runner (docker)         6 GiB, no swap, 2 GiB shm, 4 CPUs   checks, front-test, e2e,
+│                           + ci-pnpm-store volume              build-workspace, preview-screenshots, renovate
+└── runner-deploy (deploy)  2 GiB, no swap, 2 CPUs             plan, reuse-image, build-image, preview,
+                                                               production, comment, preview-close/reap,
+                                                               backup, audit
  • Option 1, the limits. front-test peaks near 4 GiB (measured in a container, memory.peak 3.9 GiB with --maxWorkers=1), so GitLab's 3 GiB was below its working set. That fits run 3's symptoms: 2.999 of 3 GiB, the host swapping, 25 tests timing out and a dynamic import failing. --memory-swap now equals --memory, so a job at its cap is OOM-killed at once instead of swapping until its tests time out. /dev/shm goes to 2 GiB; it is tmpfs charged to the same cap.
    • Kept --maxWorkers=1. I have no evidence that the MSW cause is not real. GitLab pipeline 315's symptoms (whole files failing, a setup file that would not load) match memory pressure as well as they match MSW. Once a few front-test runs pass at 6 GiB, trying --maxWorkers=2 is the next step.
    • Kept TASK_CONCURRENCY=2. checks with a cold Turbo cache took 34 s at concurrency 4 and 36 s at 2, both peaking at 4.3 to 4.5 GiB. Vitest's workers dominate that peak, not Turbo's slots, so the higher setting bought nothing measurable.
  • Option 2, two jobs at once. Forgejo has no job-level lock, so each runner's capacity: 1 is the lock. Every job that reaches the production VPS runs on deploy, so two deploys never overlap. Every browser job runs on docker, so two browser jobs never share the CPUs. A deploy no longer waits behind a pull request's gate. runner-lanes.test.mjs enforces this: every job runs on a label a runner offers, deploys and browser jobs each stay on one capacity-1 runner, and the caps fit an 8 GiB budget with no swap.
  • Option 3, cheaper setup.
    • The pnpm store moves to a volume. actions/cache is gone from ci.yml and the store sits on the ci-pnpm-store volume at /pnpm-store. GitLab rejected this because pnpm copies across mounts. I measured again: copying costs about 2 s an install, against a restore and a gzip upload on every lockfile change. The numbers are under Evidence.
    • Gate jobs skip on main. On a push to main whose pull request passed the gate, checks, front-test and e2e now skip at job level instead of starting three containers that only echo. No status is required on a push. reuse-image accepts success or skipped from them.
    • Not done: a prebuilt CI image. It would save corepack enable and the Playwright image pull, which is cached anyway. That is a few seconds, not worth an image to rebuild on every Node or Playwright bump. Left as a follow-up.

Memory budget, with two jobs running:

GiB
Job caps (gate 6 + deploy 2) 8
GitLab, until switch-day step 12 ~3
e2e's Postgres + Mailpit (outside the cap), an image build in the daemon ~1.5
Forgejo, Traefik, runners, dockerd, OS ~1.5
Total of 15 ~14

After step 12, the 3 GiB GitLab frees stays as page cache and margin.

Evidence

Measured locally in node:24.21.0-bookworm at --cpus=4, with this repo's lockfile (store 1.3 GB, 40 785 files):

Step Time
Store in the workspace: gzip restore (actions/cache without zstd, which the image lacks) 5.0 s, plus the download
Store in the workspace: install, hardlinked 1.8 s
Store in the workspace: gzip save on a new lockfile 24.7 s (57 s on the VPS)
Store on a volume: install, copied 3.6 to 4.1 s, nothing to restore or save
zstd save / restore, for comparison 1.2 s / 1.1 s, after a 2.5 s apt install zstd in every job

Peak memory (memory.peak), same container:

  • front-test, --maxWorkers=1: 3.9 GiB, above the old 3 GiB cap.
  • checks: 4.3 GiB at TASK_CONCURRENCY=4 and 4.5 GiB at 2, with Vitest seeing the Mac VM's 15 CPUs. On the VPS it sees 4, so the real peak is lower.

The local front-test run itself was not a clean signal: Chromium dropped its connection, and the Docker VM had only 7.75 GiB with other stacks using about 3.3 GiB.

runner-lanes.test.mjs
  before: fails, there is no runner-deploy.yaml and every job runs on the one runner
  after:  5 pass; moving `preview` back to `docker` fails "the deploys run on runner.yaml and runner-deploy.yaml"

Gate: pnpm lint, typecheck, test (test:infra 59 pass), build, format:check all green.

Merge Danger

Door: two-way

Revert the commit, then rerun stages 3 and 4. up.sh will then render the single runner again. The deploy runner stays registered in Forgejo until you delete it under Site administration → Actions → Runners.

Blast Radius: CI

  • Until the apply below: the jobs labeled deploy find no runner and wait. Merging this without applying blocks every deploy, and plan, and so every run. Apply right after merging.
  • Job-level skips on main are untested on Forgejo. If reuse-image were skipped despite gate jobs that skipped, production would not deploy on a merge. Check the first push to main after this lands.
  • Shared pnpm store. Like the Turbo volume, a same-repository pull request's job could tamper with it; it is already root on the host's daemon.
  • CPU overlap. Both runners' CPU caps add up to 6 of 4 CPUs. A docker build in the daemon during front-test could slow it. Watch the first runs that overlap.

Apply on the server (the owner)

  1. Merge #573, then this. Check out main.
  2. Wait until no job runs: Site administration → Actions → Runners shows the runner idle. up.sh restarts a runner whose config changed, which kills its job.
  3. infra/forgejo/vps/wizard.sh. Answer "Redo it?" with yes for stage 3 (copies the stack and the new runner-deploy.yaml; the secret key and internal token are kept) and stage 4 (up.sh: registers forgejo-vps-deploy-runner, renders both configs, restarts runner, starts runner-deploy). Answer no for every other stage.
  4. Check that sudo docker compose -f /opt/forgejo/compose.yaml ps lists runner and runner-deploy running. Check that Site administration → Actions → Runners lists both idle, with labels docker,ubuntu-latest and deploy.
  5. Check that sudo grep -E 'capacity|memory|volume' /opt/forgejo/runner/config.yaml /opt/forgejo/runner-deploy/config.yaml shows --memory=6g --memory-swap=6g, ci-pnpm-store, and --memory=2g --memory-swap=2g.

These steps are also in infra/forgejo/vps/README.md, "Apply a change to the runners". The first gate run fills ci-pnpm-store, which takes about 15 s more than a warm install.

Stacked on #573: merge it first, then rebase this onto `main`. ## Summary Two runners on the CI VPS (15 GiB, 4 CPUs), each taking one job at a time. ADR 0050 supersedes 0048. ```diff CI VPS -└── runner (docker) 3 GiB, swap allowed, 1 GiB shm, 4 CPUs every job +├── runner (docker) 6 GiB, no swap, 2 GiB shm, 4 CPUs checks, front-test, e2e, +│ + ci-pnpm-store volume build-workspace, preview-screenshots, renovate +└── runner-deploy (deploy) 2 GiB, no swap, 2 CPUs plan, reuse-image, build-image, preview, + production, comment, preview-close/reap, + backup, audit ``` - **Option 1, the limits.** front-test peaks near 4 GiB (measured in a container, `memory.peak` 3.9 GiB with `--maxWorkers=1`), so GitLab's 3 GiB was below its working set. That fits run 3's symptoms: 2.999 of 3 GiB, the host swapping, 25 tests timing out and a dynamic import failing. `--memory-swap` now equals `--memory`, so a job at its cap is OOM-killed at once instead of swapping until its tests time out. `/dev/shm` goes to 2 GiB; it is tmpfs charged to the same cap. - **Kept `--maxWorkers=1`.** I have no evidence that the MSW cause is not real. GitLab pipeline 315's symptoms (whole files failing, a setup file that would not load) match memory pressure as well as they match MSW. Once a few front-test runs pass at 6 GiB, trying `--maxWorkers=2` is the next step. - **Kept `TASK_CONCURRENCY=2`.** checks with a cold Turbo cache took 34 s at concurrency 4 and 36 s at 2, both peaking at 4.3 to 4.5 GiB. Vitest's workers dominate that peak, not Turbo's slots, so the higher setting bought nothing measurable. - **Option 2, two jobs at once.** Forgejo has no job-level lock, so each runner's `capacity: 1` is the lock. Every job that reaches the production VPS runs on `deploy`, so two deploys never overlap. Every browser job runs on `docker`, so two browser jobs never share the CPUs. A deploy no longer waits behind a pull request's gate. `runner-lanes.test.mjs` enforces this: every job runs on a label a runner offers, deploys and browser jobs each stay on one capacity-1 runner, and the caps fit an 8 GiB budget with no swap. - **Option 3, cheaper setup.** - **The pnpm store moves to a volume.** `actions/cache` is gone from ci.yml and the store sits on the `ci-pnpm-store` volume at `/pnpm-store`. GitLab rejected this because pnpm copies across mounts. I measured again: copying costs about 2 s an install, against a restore and a gzip upload on every lockfile change. The numbers are under Evidence. - **Gate jobs skip on main.** On a push to main whose pull request passed the gate, checks, front-test and e2e now skip at job level instead of starting three containers that only echo. No status is required on a push. `reuse-image` accepts `success` or `skipped` from them. - **Not done: a prebuilt CI image.** It would save `corepack enable` and the Playwright image pull, which is cached anyway. That is a few seconds, not worth an image to rebuild on every Node or Playwright bump. Left as a follow-up. **Memory budget, with two jobs running:** | | GiB | | --- | --- | | Job caps (gate 6 + deploy 2) | 8 | | GitLab, until switch-day step 12 | ~3 | | e2e's Postgres + Mailpit (outside the cap), an image build in the daemon | ~1.5 | | Forgejo, Traefik, runners, dockerd, OS | ~1.5 | | **Total of 15** | **~14** | After step 12, the 3 GiB GitLab frees stays as page cache and margin. ## Evidence Measured locally in `node:24.21.0-bookworm` at `--cpus=4`, with this repo's lockfile (store 1.3 GB, 40 785 files): | Step | Time | | --- | --- | | Store in the workspace: gzip restore (`actions/cache` without zstd, which the image lacks) | 5.0 s, plus the download | | Store in the workspace: install, hardlinked | 1.8 s | | Store in the workspace: gzip save on a new lockfile | 24.7 s (57 s on the VPS) | | Store on a volume: install, copied | 3.6 to 4.1 s, nothing to restore or save | | zstd save / restore, for comparison | 1.2 s / 1.1 s, after a 2.5 s `apt install zstd` in every job | Peak memory (`memory.peak`), same container: - front-test, `--maxWorkers=1`: **3.9 GiB**, above the old 3 GiB cap. - checks: 4.3 GiB at `TASK_CONCURRENCY=4` and 4.5 GiB at 2, with Vitest seeing the Mac VM's 15 CPUs. On the VPS it sees 4, so the real peak is lower. The local front-test run itself was not a clean signal: Chromium dropped its connection, and the Docker VM had only 7.75 GiB with other stacks using about 3.3 GiB. ```text runner-lanes.test.mjs before: fails, there is no runner-deploy.yaml and every job runs on the one runner after: 5 pass; moving `preview` back to `docker` fails "the deploys run on runner.yaml and runner-deploy.yaml" ``` Gate: `pnpm lint`, `typecheck`, `test` (test:infra 59 pass), `build`, `format:check` all green. ## Merge Danger **Door:** two-way Revert the commit, then rerun stages 3 and 4. `up.sh` will then render the single runner again. The deploy runner stays registered in Forgejo until you delete it under Site administration → Actions → Runners. **Blast Radius:** CI - **Until the apply below:** the jobs labeled `deploy` find no runner and wait. Merging this without applying blocks every deploy, and `plan`, and so every run. Apply right after merging. - **Job-level skips on main are untested on Forgejo.** If `reuse-image` were skipped despite gate jobs that skipped, production would not deploy on a merge. Check the first push to main after this lands. - **Shared pnpm store.** Like the Turbo volume, a same-repository pull request's job could tamper with it; it is already root on the host's daemon. - **CPU overlap.** Both runners' CPU caps add up to 6 of 4 CPUs. A `docker build` in the daemon during front-test could slow it. Watch the first runs that overlap. ### Apply on the server (the owner) 1. Merge #573, then this. Check out `main`. 2. Wait until no job runs: Site administration → Actions → Runners shows the runner idle. `up.sh` restarts a runner whose config changed, which kills its job. 3. `infra/forgejo/vps/wizard.sh`. Answer "Redo it?" with **yes** for stage 3 (copies the stack and the new `runner-deploy.yaml`; the secret key and internal token are kept) and stage 4 (`up.sh`: registers `forgejo-vps-deploy-runner`, renders both configs, restarts `runner`, starts `runner-deploy`). Answer **no** for every other stage. 4. Check that `sudo docker compose -f /opt/forgejo/compose.yaml ps` lists `runner` and `runner-deploy` running. Check that Site administration → Actions → Runners lists both idle, with labels `docker,ubuntu-latest` and `deploy`. 5. Check that `sudo grep -E 'capacity|memory|volume' /opt/forgejo/runner/config.yaml /opt/forgejo/runner-deploy/config.yaml` shows `--memory=6g --memory-swap=6g`, `ci-pnpm-store`, and `--memory=2g --memory-swap=2g`. These steps are also in `infra/forgejo/vps/README.md`, "Apply a change to the runners". The first gate run fills `ci-pnpm-store`, which takes about 15 s more than a warm install.
docs(repo): name Forgejo Actions in the ci commit scope
Some checks failed
ci / plan (pull_request) Successful in 5s
Details
ci / checks (pull_request) Failing after 48s
Details
ci / front-test (pull_request) Successful in 5s
Details
ci / e2e (pull_request) Successful in 5s
Details
ci / reuse-image (pull_request) Has been skipped
Details
ci / build-workspace (pull_request) Has been skipped
Details
ci / build-image (pull_request) Has been skipped
Details
ci / preview (pull_request) Has been skipped
Details
ci / preview-screenshots (pull_request) Has been skipped
Details
ci / production (pull_request) Has been skipped
Details
ci / comment (pull_request) Failing after 4s
Details
e6e1162f4e 
The ci scope's description still said GitLab CI; the gate and deploys now run as Forgejo
Actions workflows in .forgejo/, as proposed in !68.

Refs #499
ci: upload and download artifacts with Forgejo's patched v4 actions
Some checks failed
audit / audit (pull_request) Successful in 8s
Details
ci / plan (pull_request) Successful in 3s
Details
ci / checks (pull_request) Successful in 1m37s
Details
ci / e2e (pull_request) Successful in 42s
Details
ci / front-test (pull_request) Failing after 9m26s
Details
ci / reuse-image (pull_request) Has been skipped
Details
ci / build-workspace (pull_request) Has been skipped
Details
ci / build-image (pull_request) Has been skipped
Details
ci / preview (pull_request) Has been skipped
Details
ci / preview-screenshots (pull_request) Has been skipped
Details
ci / production (pull_request) Has been skipped
Details
ci / comment (pull_request) Successful in 3s
Details
50d5acdf09 
The first Forgejo run failed in checks and comment: actions/upload-artifact@v4 and
download-artifact@v4 resolve on data.forgejo.org to GitHub's code, which refuses any host but
github.com (GHESNotSupportedError). code.forgejo.org/forgejo's copies drop that check.

Refs #499
ci: run the gate and the deploys on two runners sized to the CI VPS
Some checks failed
audit / audit (pull_request) Successful in 8s
Details
ci / plan (pull_request) Successful in 3s
Details
ci / checks (pull_request) Successful in 33s
Details
ci / front-test (pull_request) Failing after 4m49s
Details
ci / e2e (pull_request) Successful in 38s
Details
ci / reuse-image (pull_request) Has been skipped
Details
ci / build-workspace (pull_request) Has been skipped
Details
ci / build-image (pull_request) Has been skipped
Details
ci / preview (pull_request) Has been skipped
Details
ci / preview-screenshots (pull_request) Has been skipped
Details
ci / production (pull_request) Has been skipped
Details
ci / comment (pull_request) Successful in 4s
Details
f39a5abfb1 
The CI VPS has 15 GiB and 4 CPUs. front-test failed on Forgejo run 3 at GitLab's 3 GiB cap,
swapping until its tests timed out; it peaks near 4 GiB.

- A gate runner (label docker) takes the gate, browser and install jobs: 6 GiB, 2 GiB /dev/shm,
  4 CPUs. A deploy runner (label deploy) takes every job that reaches the production VPS and the
  light ones: 2 GiB, 2 CPUs. Each runs one job at a time, so deploys never overlap and browser
  jobs never share the CPUs. --memory-swap equals --memory, so a capped job fails fast.
- The pnpm store lives on the ci-pnpm-store volume instead of actions/cache: no restore and no
  gzip upload, for about 2 s more per install since pnpm copies across mounts.
- On a push to main whose pull request passed the gate, the gate jobs skip at job level instead
  of starting three containers that only echo.
- runner-lanes.test.mjs checks every workflow job against the two runners and the memory budget.
- ADR 0050 supersedes 0048 with the two runners.
admin-amaurycoudr commented 2026-10-22 12:48:43 +00:00
Author
Owner
Copy link

Order of apply: this pull request's own run needs the deploy runner, since plan now runs on the deploy label. The new runners also serve main's current workflows: the gate runner keeps the docker label and the actions cache. So apply the steps from this branch first (check out ci/forgejo-runner-sizing instead of main in step 1), let this run go green, then merge #573 and this one. Until then, this run waits for a runner.

Order of apply: this pull request's own run needs the deploy runner, since plan now runs on the deploy label. The new runners also serve main's current workflows: the gate runner keeps the docker label and the actions cache. So apply the steps from this branch first (check out ci/forgejo-runner-sizing instead of main in step 1), let this run go green, then merge #573 and this one. Until then, this run waits for a runner.
chess-bot commented 2026-10-22 13:08:09 +00:00
Member
Copy link

Run at f39a5ab

The run

Failed tests

No test failed.

front-test failed outside any test it reports; its log says why.

Engine coverage

Statements: 99.48%, shown and never enforced.

Screenshots

No screenshots: no preview was deployed, or capturing it failed.

<!-- chess-run --> ### Run at f39a5ab [The run](http://forgejo:3000/amaurycoudr/chess-project/actions/runs/5) #### Failed tests No test failed. front-test failed outside any test it reports; its log says why. #### Engine coverage Statements: **99.48%**, shown and never enforced. #### Screenshots No screenshots: no preview was deployed, or capturing it failed.
Some checks failed
audit / audit (pull_request) Successful in 8s
Details
ci / plan (pull_request) Successful in 3s
Details
ci / checks (pull_request) Successful in 33s
Required
Details
ci / front-test (pull_request) Failing after 4m49s
Required
Details
ci / e2e (pull_request) Successful in 38s
Required
Details
ci / reuse-image (pull_request) Has been skipped
Details
ci / build-workspace (pull_request) Has been skipped
Details
ci / build-image (pull_request) Has been skipped
Details
ci / preview (pull_request) Has been skipped
Details
ci / preview-screenshots (pull_request) Has been skipped
Details
ci / production (pull_request) Has been skipped
Details
ci / comment (pull_request) Successful in 4s
Details
Some required checks were not successful.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin ci/forgejo-runner-sizing:ci/forgejo-runner-sizing
git switch ci/forgejo-runner-sizing
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something isn't working

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
question
Further information is requested

  
spec
Parent spec written by to-spec

  
wayfinder:grilling

   
wayfinder:map

   
wayfinder:prototype

   
wayfinder:research

   
wayfinder:task

   
wontfix
This will not be worked on

No labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
spec
wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wayfinder:task
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
amaurycoudr/chess-project!574
No description provided.