Skip to main content
A repository commits one file, .outerlayer/setup.sh. It prepares the repository’s checkout, for example by installing dependencies. The runner runs it in every container, microVM and sandbox build, before the build’s command. After the scripts pass, the runner saves the build’s disk. A later build starts from that saved state, so it does not install everything again.

Write the script

The script is a shell script. Put the commands that prepare a checkout in it, such as npm ci or pip install -r requirements.txt.
Follow these rules:
  • Commit the file with its executable bit set. The runner then runs the file itself, and its first line chooses the shell. Without the bit, the runner runs the file with /bin/sh, which is dash on Ubuntu. A script that needs bash fails there.
  • Use set -e. The script’s exit status is the status of its last command. A script without set -e can pass after a failed command, and the runner then saves a broken state.
  • Write commands for an unprivileged user. The script cannot install system packages. Put those in the Dockerfile of the recipe, the repository’s devcontainer file. See Where setup that needs root belongs.
  • Write the script to run twice without harm. It runs in every build, even a build that starts from a saved state. There, the run takes seconds, because the work is already done.
  • In a container build, write files into the build user’s home or a checkout. The saved state holds only those, so a file in /tmp is not in the next build. A microVM build’s saved state holds its whole build disk.
  • In a checkout, write only files that git ignores, such as node_modules. A build that starts from a saved state removes the other untracked files.
The runner runs each repository’s script in that repository’s checkout, as the build user. It runs the primary repository’s script first. The scripts run after the clone, the runtime install and the recipe’s lifecycle commands. The script reaches only the hosts that runner.build.allowHosts allows, as every other command of the build does.

When the script fails

A script that exits with a code other than 0 ends the build at stage provision. A script that runs past runner.build.setupMinutes does the same. The default limit is 30 minutes for each script. The build’s command does not start, and the runner saves no state. The item shows the reason. It starts with setup_failed or setup_timed_out, and names the repository, then the exit code or the limit, then the last lines the script printed. To read more of the output, run outerlayer work logs --item <n>. It prints the last 200 lines of the provision log.

In host checks

Host checks run in a new container or microVM, so nothing the build installed is there. The runner runs each script again in that environment, before the first check. A check can then use what the script installed, such as node_modules. The script runs from the branch the build pushed, with the same time limit. The check environment never starts from a saved state, so the script does all its work there. A script that fails there fails every host check of the build.

Repositories with no script

When a repository has no .outerlayer/setup.sh, the build runs without it. The runner sets OUTERLAYER_MISSING_SETUP_SCRIPTS for the build’s command. Its value is a JSON list of the repositories that have no script, such as ["acme/app"]. The outerlayer-build skill reads it. The agent writes the script from the commands it ran to prepare the checkout, and the build’s pull request adds the file. A build in which no setup script ran saves no state.

Git hooks in a build

A build credits the member who asked for it as co-author of each commit. To add that line, the runner puts git hooks of its own in front of the repository’s hooks while the build’s command runs. Each one runs the repository’s hook of the same name, so your hooks still run. Two cases need a change in the repository:
  • Hooks that are not installed. A repository that uses husky or lefthook must install its hooks in the build. Otherwise each commit and push prints a line that names the hooks folder. Install them in the setup script.
  • lefthook install in the build’s command. It fails with permission denied, because it writes into the runner’s hooks folder. This happens when a prepare script of package.json runs it. Use this command there:
env -u GIT_CONFIG_COUNT clears the runner’s hooks setting for this one command. It also clears every other git setting that the build’s variables pass in the same way. The command also works in the setup script. husky needs no change.

The saved state

A saved state is a copy of what the setup scripts prepared. The runner keeps saved states on its own disk. A container build’s state holds the build user’s home and the checkouts. A microVM build’s state holds the whole build disk. The runner saves a state after a build whose setup scripts all pass, when the build did not start from one. A later build uses the state when all of these match:
  • The factory and its repositories.
  • The size of the build disk, runner.build.disk.
  • The content of each repository’s setup script and version files, on the default branch.
The runner reads the script and the version files from each repository’s default branch, never from the branch a build works on. So a build cannot change the state that later builds use. A build never changes the state it starts from. It works on a copy. When a build starts from a saved state, its log says so and gives the state’s age.

When a build starts from a fresh disk

A build starts from a fresh disk, runs the whole script, and saves a new state in these cases:
  • No saved state matches.
  • The saved state is seven days old or older.
  • A setup script or a version file changed on the default branch. The runner’s log names the file.
To discard a saved state sooner, change the setup script on the default branch. A change to a comment line is enough.

Builds that save no state

These builds run the script and save no state:
  • A build whose image comes from a recipe. Only a build on the universal image saves a state.
  • A build on a managed runner.
  • A build on a runner with builtin:sandbox hooks.
  • A build on a runner whose runner.housekeeping.savedStateMax is off.

What an agent job starts from

A rerun of a build, and an agent job, start from the build disk that the item’s previous job left. An agent job answers a comment or a failed check on the pull request of a build. Neither starts from a saved state. The runner keeps one disk for each work item, on its own host. It keeps the disk when a container or microVM job ends, whatever the outcome of the job’s command. A newer disk replaces the older one. Two kinds of job keep no disk, and the older disk stays:
  • A job that fails at provision.
  • A build that the runner ends for passing its disk limit.
The kept disk holds what the previous job left:
  • Untracked and ignored files in each checkout, such as installed dependencies and build outputs.
  • Files outside the checkouts, such as caches and the home folder.
  • For a microVM job, the Docker images that the job pulled or built.
It does not hold memory or running processes. A service or a database starts again. When a job starts from a kept disk, the runner does this first:
  1. It discards the changes to tracked files in each checkout.
  2. It checks out the head of the item’s branch. Each tracked file then matches the branch. A rerun checks out the default branch when the remote has no such branch.
  3. It runs the setup scripts again.
Commits that the earlier job made and did not push are not on the branch. The runner keeps them in the checkout at the ref refs/outerlayer/kept/<commit>, and its log names that ref.

With more than one runner

A runner keeps an item’s agent session and its unfinished work on its own host. What it keeps of the build disk depends on its hooks: Another runner has none of this. So an item’s next job goes first to the runner that ran its last job. When another runner of your own claims the item, the gateway refuses the claim with work_item_held_for_runner. That runner logs the refusal and takes other work. The first runner takes the item at its next poll. The wait is twice the poll interval of the first runner. It is at least 60 seconds and at most 10 minutes. After the wait, any runner takes the item, and the job starts with a new session and without what the first runner keeps. The gateway does not wait when the first runner cannot take the job:
  • It sent no heartbeat for three of its poll intervals.
  • All its slots are in use.
  • It does not take the item’s queue or repository.
  • It reports that it takes no work, for example while it drains or updates.
A build that you start from a fresh disk does not wait.

When a job starts without the kept disk

A job starts as a first build does, from a saved state or a fresh disk, in these cases:
  • The job is the first build of the item.
  • The job runs on another host than the one that holds the disk.
  • The build image changed. A runner update can change it.
  • runner.build.disk changed.
  • The kept disk cannot be used. For example, a file that the earlier job left untracked is tracked on the branch.
  • A member asked for a fresh disk.
The job never fails for one of these reasons. The job’s card in the Activity section of the item page gives the reason. See The Overview tab. Two more rules apply on a managed runner:
  • It keeps no disk for a build that held a build variable. A command can write the value to the disk.
  • The disk is on one machine. A job that another machine takes starts without it. An agent job waits up to 1 minute for the machine that holds the disk.

Start a rerun from a fresh disk

A damaged node_modules or a full disk can break each rerun. To start a build without the kept disk, tick Start from a fresh disk in the rerun dialog. Or run outerlayer work build --fresh. The build then starts from a saved state or a fresh disk. When it ends, its disk replaces the kept one. The choice applies to a build only. An agent job on an open pull request has no fresh-disk choice. A request for one is refused as fresh_disk_unavailable when the item’s open pull request waits on an agent.

Continue the session of the previous build

A rerun of a build starts a new agent session. The agent reads the issue and the code again, on the kept disk. After a build that you stopped, or that ended at its time limit, the item page offers Resume at its foot. Resume builds again in the session of that build. After any other build, open the rerun dialog and tick Continue the session of the last build. That box and Start from a fresh disk cannot both be ticked. To have the agent go on in the session of the item’s previous build, run outerlayer work build --item <number> --continue. Use it after you stop a build to redirect it, or after a build ends at its time limit. Add --instructions to say what to change. The continued session starts with the files as the earlier job left them. A job that a member stops, or that ends at its time limit, leaves work it did not push. So does a job whose agent ends before it opens a pull request. The runner keeps that work with the session: the edits the job did not commit, and the commits it did not push. A job that continues the session gets both back. Each checkout is at the commit the earlier job was at, and the edits are not committed. This also applies to an agent job on an open pull request. The runner keeps this work on its own host, with the session. It does not push it, and it does not send it to OuterLayer. Ignored files, such as installed dependencies, are not part of it: the setup scripts and the kept disk supply those. The work is not put back in two cases, and the runner’s log says which:
  • The remote branch has commits that the work was not made on. The checkout stays at the head of the branch. The earlier files are in the checkout at the ref refs/outerlayer/kept/<commit>.
  • The work of one checkout is larger than 64 MiB. The runner keeps none for that checkout.
Three things are not put back as they were:
  • Which edits were staged. All edits come back unstaged.
  • A merge or a rebase that the job did not finish. The checkout is at the commit the job was at, and the merge or rebase is not in progress. Its commits are parents of the commit at refs/outerlayer/kept/<commit>. So are a stash, and the commits of another local branch.
  • A git repository inside a checkout. Its files are not part of the work.
A job that starts a new session never gets this work. It starts from the branch as pushed. These limits apply:
  • The rerun must run on the host that ran the previous build. A runner keeps the session of a container or microVM build on its own host.
  • When the runner has no session to continue, the build starts a new session. The runner’s log says why.
  • --continue and --fresh cannot go together.
  • An agent job on an open pull request continues the session already. The option changes nothing there.
A session that went wrong stays wrong when you continue it. After a build that failed, a new session is often the better choice.

Disk

Saved states and kept disks use the disk of the runner’s host. These settings under runner.housekeeping limit them: The runner also removes these:
  • A saved state that is seven days old.
  • The kept disk of an item whose pull request merged or closed.
  • Kept disks, then its oldest saved states, when free disk is low. It does this before it stops claiming work.
A microVM build saves no state when its build disk uses more than savedStateMax. It also saves none when the copy would leave less than runner.housekeeping.minFreeDisk free. A build with a large build directory needs a higher savedStateMax. On a Mac, a container build’s saved states and kept disks use the disk of the VM that runs Docker. outerlayer runner init --vm adds room for savedStateMax to the disk of the Lima VM it makes, and none for kept disks. On Docker Desktop, raise the disk limit in its settings, or set the limits lower.

Keep secrets out of a saved state

A file that a build writes stays in the saved state and on the kept disk. For example, an ~/.npmrc with a token from a build variable stays. Do not write a secret into a file when you can avoid it.
Both stay on the runner’s host, and only builds of the same factory use them. A kept disk goes only to jobs of the same work item. To keep a secret out of the build, use a destination on a runner of your own. The build then holds no secret, so no file can.

On a managed runner

On a managed runner, every step before the build’s command runs without the factory’s build variables and without the build’s gh login. That covers the runtime install, the recipe’s lifecycle commands and the setup scripts. The build’s command gets both. So a setup script there cannot read a build variable, such as a registry token for private packages. It also cannot call gh. Such a script fails, and the runner’s log names the limit. A proxied build variable works in a setup script. The script holds a stand-in, and the runner adds the real value to each request to the service. In the host checks, the setup script does get the build variables. Do not write a script that does different work when a variable is set.