Adding a mirrored upstream repo with CI build

The pattern for “I want a Docker image of an upstream project that doesn’t publish one.” Used when:

The upstream is on GitHub (or any HTTPS git host) and ships source but no container image
You want our CI to build it weekly so we always have a fresh :latest
We don’t want to add a Dockerfile to /docker/* (that repo is for compose + ingress only — see Homelab Conventions)

The mechanism: bachp/git-mirror (running on bots/) pulls from upstream every 5 min and force-pushes to a project under gitlab.mdbook.me/mirrors/. That project runs a CI pipeline defined out-of-tree at mikayla/scripts:ci/<name>.yml, which builds and publishes to registry.mdbook.me/mirrors/<name>:latest. Compose services in /docker/* then reference that image.

Canonical worked example: mirrors/bambu-studio-api — see Bambu Lab 3D printing stack for how the resulting image is consumed downstream.

Step-by-step

1. Create the GitLab project under `mirrors/`

In gitlab.mdbook.me:

Path: mirrors/<name> — pick a short, lowercase name. Match upstream where reasonable; rename only if upstream’s name collides with something already under mirrors/.
Description (this is critical, not cosmetic): the upstream HTTPS clone URL, plain. No #branch suffix — bachp/git-mirror reads this field and feeds it directly to git clone --mirror, and a #branch fragment is treated as part of the URL, GitHub redirects, and git aborts with unable to update url base from redirection. If you want a single branch, mirror the whole repo and let downstream consumers checkout by name.
Visibility: private (we don’t redistribute mirrored upstream code).

Trap: a project with description: null blows up the entire mirror sync — the tool fails to enumerate the group with Unable to parse response as JSON (invalid type: null, expected a string at line 1 column 28) and exits before touching any project. Always set a description. One broken project hides every other project’s status.

2. Add `gitbot` as Maintainer

Project information → Members → Invite member → gitbot → Maintainer.

bachp/git-mirror runs git push -f --mirror as gitbot. Without Maintainer the push fails with HTTP 403 and you get a confusing pile of “every ref rejected” errors that look like the project itself is broken.

3. Allow force-push on the default branch

Settings → Repository → Protected branches → main → Allowed to force push: ✅ Maintainers.

--mirror is one atomic push from gitlab-shell’s perspective. If the protected default branch rejects the force push, every tag, branch, and refs/pull/*/head in the same push gets pre-receive hook declined, making it look like everything’s broken when only main is the problem.

Don’t fully unprotect — these are read-only upstream mirrors and we still want to block accidental human pushes. The force-push allowance is just for the mirror service account.

4. Write the CI pipeline file in `mikayla/scripts:ci/<name>.yml`

The pipeline lives in a separate repo (mikayla/scripts) so the mirror project’s branch state stays a clean copy of upstream. Use kaniko (not DinD — see Why kaniko, not DinD below).

A minimal template (adapt from ci/bambu-studio-api.yml):

variables:
  FF_NETWORK_PER_BUILD: "true"
  DOCKERFILE_PATH: "Dockerfile"   # change if upstream's Dockerfile is in a subdir or named differently
 
stages:
  - build
  - notification
 
build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.23.2-debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker
    - printf '{"auths":{"%s":{"username":"%s","password":"%s"}}}' "$CI_REGISTRY" "$CI_REGISTRY_USER" "$CI_REGISTRY_PASSWORD" > /kaniko/.docker/config.json
    - >
      /kaniko/executor
      --context "$CI_PROJECT_DIR"
      --dockerfile "$CI_PROJECT_DIR/$DOCKERFILE_PATH"
      --destination "$CI_REGISTRY_IMAGE:latest"
      --destination "$CI_REGISTRY_IMAGE:upstream-$CI_COMMIT_SHORT_SHA"
      --snapshot-mode=redo
      --use-new-run
      --single-snapshot
      --cleanup
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
    - if: '$CI_PIPELINE_SOURCE == "web"'
 
success_notification:
  stage: notification
  image: alpine:3.20
  script:
    - apk add curl git
    - wget https://gitlab.mdbook.me/mikayla/scripts/-/raw/master/ci-discord-hook.sh
    - chmod +x ci-discord-hook.sh
    - /bin/sh ci-discord-hook.sh success $WEBHOOK_URL
  when: on_success
 
failure_notification:
  stage: notification
  image: alpine:3.20
  script:
    - apk add curl git
    - wget https://gitlab.mdbook.me/mikayla/scripts/-/raw/master/ci-discord-hook.sh
    - chmod +x ci-discord-hook.sh
    - /bin/sh ci-discord-hook.sh failure $WEBHOOK_URL
  when: on_failure

Commit and push to mikayla/scripts.

The rules: block above triggers a build on:

Schedule (cron) — the primary path
Push to default branch — i.e. when the mirror lands a real upstream change
Manual run from the web UI — for ad-hoc rebuilds

5. Wire the CI config path into the mirror project

In the mirror project (mirrors/<name>):

Settings → CI/CD → General pipelines → CI/CD configuration file → set to ci/<name>.yml@mikayla/scripts.

Trap: GitLab’s CI/CD settings page has multiple sections, each with its own Save changes button. It is very easy to type into the field, see the value sitting there, and navigate away — losing the change silently. After saving, navigate away and come back; the value should still be populated.

6. Disable Auto DevOps

Settings → CI/CD → Auto DevOps → uncheck “Default to Auto DevOps pipeline”.

If left on, GitLab runs herokuish + a Postgres sidecar on every push and ignores the configured ci_config_path. The symptom is a successful but bizarre pipeline that has nothing to do with what you wrote — herokuish trying to detect the project type and falling back to defaults.

7. Schedule the rebuild cron

Settings → CI/CD → Pipeline schedules → New schedule.

Description: Weekly upstream rebuild (or whatever cadence you picked)
Interval pattern: 0 4 * * 0 for Sunday 4am, or whatever fits
Target branch: default branch (main for most repos)
Active: ✅

8. First build

Two choices:

Wait for the next mirror sync (≤5 min) — if upstream has any changes the cron pulls and the push triggers a build via the $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH rule.
Force a build by hitting Pipelines → Run pipeline → main. This uses the $CI_PIPELINE_SOURCE == "web" rule.

Watch the build in the Pipelines view. First builds frequently surface upstream-Dockerfile assumptions that don’t match the runner environment — adjust DOCKERFILE_PATH, --build-arg, or kaniko flags as needed.

9. Reference the image in compose

Once :latest is published, add the service to the relevant box’s /docker/<box>/docker-compose.yaml:

your-service:
    image: registry.mdbook.me/mirrors/<name>:latest
    container_name: your-service
    restart: unless-stopped
    # ... mounts, env, etc.

Per the homelab convention: never add a Dockerfile under /docker/* to “build it locally instead.” The whole point of the mirror+CI pattern is to keep /docker/* for compose+ingress only.

10. Document the deployment

Add a page under Stacks describing the new service and its dependencies. If the new service depends on the mirrored image, link to this page from there for the build pipeline details — don’t repeat the wiring instructions in every stack doc.

Why kaniko, not DinD

Two reasons that have bitten this exact pattern:

Disk pressure. DinD inflates the entire build context inside the runner’s docker-in-docker socket. For any image whose build pulls a large artifact (AppImages, language runtimes, model weights), the runner’s root partition runs out of space mid-build. Kaniko streams layers directly to the registry — much less held on disk at any point.
No privileged container needed. DinD requires privileged: true on the runner; kaniko runs as a regular container. Lower blast radius, easier to audit.

Recommended kaniko flags for the homelab:

Flag	Why
`--snapshot-mode=redo`	Faster than the default `full` mode, equally safe for typical builds
`--use-new-run`	New layer-construction codepath; faster and lower memory
`--single-snapshot`	One snapshot at the end instead of per-RUN; massive speedup for builds with many RUN lines
`--cleanup`	Releases temp files between layers; helps on tight-disk runners

Pin the kaniko image (v1.23.2-debug at time of writing). The -debug tag ships /busybox/sh and basic GNU utils, which the script needs for the auth-config heredoc.

Quick troubleshooting

Symptom	Cause	Fix
`pre-receive hook declined` on every ref in mirror logs	Default branch is force-push-protected	Step 3 above
Mirror enumeration fails with `invalid type: null`	Some project under `mirrors/` has `description: null`	Set descriptions on all `mirrors/*` projects
Pipeline runs herokuish + postgres unexpectedly	Auto DevOps still on, or `ci_config_path` not saved	Step 5 + step 6
`unable to update url base from redirection` in mirror logs	Description has `#branch` fragment, GitHub redirects	Strip the fragment, mirror the whole repo
`Pipeline cannot be run. Missing CI config file`	The `ci_config_path` field was typed but Save was not clicked	Re-save the section, refresh, verify the value persists
Kaniko OOM / `no space left on device`	Runner disk too small for context+layers	Either expand the runner’s root LV, or move build to a fatter runner — kaniko helps a lot but won’t save you on a tiny runner
First build can’t find Dockerfile	Upstream’s Dockerfile is in a subdir or non-standard name	Set `DOCKERFILE_PATH` in pipeline `variables:`
Mirror catch-up push silently skips most refs	GitLab’s `push_repository_pipeline_creation_limit` (default 4)	Admin → Settings → CI/CD → raise limit (~100). See Mirror catch-up pushes silently skip pipeline creation discussion in handoff.md

Where the pieces live

Mirror service: bots/docker-compose.yaml → git-mirror (bachp/git-mirror)
CI pipeline definitions: mikayla/scripts:ci/<name>.yml (one file per mirror)
Discord webhook helper: mikayla/scripts:ci-discord-hook.sh
GitLab projects: gitlab.mdbook.me/mirrors/
Built images: registry.mdbook.me/mirrors/<name>
Operational gotchas, raw error messages, and historical context: /docker/handoff.md → “git-mirror — projects under mirrors/ need force-push allowed on their default branch” and the related sub-sections

mdbook wiki

Explorer

Adding a mirrored upstream repo with CI build

Step-by-step

1. Create the GitLab project under `mirrors/`

2. Add `gitbot` as Maintainer

3. Allow force-push on the default branch

4. Write the CI pipeline file in `mikayla/scripts:ci/<name>.yml`

5. Wire the CI config path into the mirror project

6. Disable Auto DevOps

7. Schedule the rebuild cron

8. First build

9. Reference the image in compose

10. Document the deployment

Why kaniko, not DinD

Quick troubleshooting

Where the pieces live

Graph View

Table of Contents

mdbook wiki

Explorer

Adding a mirrored upstream repo with CI build

Step-by-step

1. Create the GitLab project under mirrors/

2. Add gitbot as Maintainer

3. Allow force-push on the default branch

4. Write the CI pipeline file in mikayla/scripts:ci/<name>.yml

5. Wire the CI config path into the mirror project

6. Disable Auto DevOps

7. Schedule the rebuild cron

8. First build

9. Reference the image in compose

10. Document the deployment

Why kaniko, not DinD

Quick troubleshooting

Where the pieces live

Graph View

Table of Contents

1. Create the GitLab project under `mirrors/`

2. Add `gitbot` as Maintainer

4. Write the CI pipeline file in `mikayla/scripts:ci/<name>.yml`