OuterLayer evidence check run. It reads two things side by side: whether OuterLayer’s own checks pass, and where a person’s review stands. OuterLayer writes both only when Post session summaries on pull requests is on, in Settings → General. It is off by default. See Connect a repository.
The verdict is one of these:
- pass: every check shown passed.
- flagged: something needs attention.
- waiting: nothing failed, but a result has not arrived. The comment says whether it waits on a required result or session evidence.
.outerlayer/policy.yaml, and the validator files in .outerlayer/validators/. Both are read from the pull request’s base branch, so a change takes effect when it merges. A few keys below are read from the default branch instead, and say so. When a factory names a context source, the files come from that repository, and a change applies at the next evaluation. See Share instructions across repositories.
The policy file
.outerlayer/policy.yaml:
To let agent sessions replace a recorded criteria list, set
criteria: replace: anyone.
Every key is optional. An empty file sets nothing, and every key takes its default.
An unknown key or an invalid value voids the whole file for that evaluation. The comment shows a policy error row. Your overrides drop out, and every key takes its default. The validator files still apply. Fix the file on the base branch.
extends is not a key. OuterLayer has no preset to extend: every check comes from a validator file. A file that sets extends, including extends: outerlayer:recommended@v1, gets a policy error row that names the fix. Run outerlayer init --template default, which writes .outerlayer/validators/code-review.yaml if it is missing, then delete the extends line.
The OuterLayer evidence check
The title of theOuterLayer evidence check says what decided it. With merge_gate: on-flag:
A person’s “no” blocks a GitHub merge under
merge_gate: on-flag. A missing review does not. With merge_gate: none, every completed result is neutral and blocks nothing, but the titles are the same.
A pull request nobody ran through OuterLayer is never evaluated and gets no comment. It can still get a neutral OuterLayer evidence check titled “Not run through OuterLayer”, which says nothing was checked and never blocks a merge. It gets that check when the base branch’s policy has review: required (the default) or a merge_gate other than none, and Post session summaries on pull requests is on. Otherwise OuterLayer writes nothing to it.
Every check comes from a file in .outerlayer/validators/. OuterLayer defines none of its own, so a repository with no validator files has nothing checked, with or without a policy file.
The code review check
outerlayer init writes a policy file and one check, .outerlayer/validators/code-review.yaml. So does outerlayer init --template default. Each file is written only when it is missing, and is yours to edit. Without its comments, it reads:
code-review for the work item. After a review, emit its report as an artifact, run outerlayer sync, and copy the report’s deep link from the evidence comment. Then record the verdict with that link as its proof:
--result fail when the review left something unfixed. A session, CI or a person can record it. See Record a check.
The row says a result is recorded, not what the review found. While a host still holds the item’s claim, a missing result reads as waiting. Afterwards it reads as not proven.
Level it under validators:, for example code-review: off, or delete the file.
A level for code-review-ran, the id this check had when the product defined it, is ignored while no validator file declares that id.
A policy file written for the retired acceptance-criteria check keeps loading. Its level is ignored, and a validator file of yours may take that id. A report bound with --for acceptance-criteria is not a check any more: the item page links it at the top of its Criteria tab.
No check reads a CI result unless you declare one; the item page shows GitHub’s own status instead. To put a CI result in the verdict, see Checks that run in CI.
Some rows have no level and cannot be turned off: policy errors, an unreadable policy, a person’s pass or fail on the item, rows a linked issue asks for, and criterion-unproven.
Recorded criteria in the verdict
An item’s recorded criteria count toward the verdict of its open and merged pull requests. A pull request closed without merging is not flagged.- An unproven criterion is a recorded criterion whose declared proof is not attached. The row is
criterion-unprovenand it always counts toward the verdict; no policy key lowers it. A team that does not want a proof kind does not declare it. A criterion that declares no proof never produces one. - No recorded list produces
criteria-missing, whichcriteria.missinglevels.warnflags the verdict.inforecords the fact without flagging it.offdrops it. Any other value is a policy error.
- Approve stays disabled on the item page, though the item still reads Ready for review.
- A runner releases the build as incomplete.
- The GitHub check fails when
merge_gate: on-flagis set.
criteria.missing to info or off, or record criteria with outerlayer emit criteria.
Custom validators
Each file in.outerlayer/validators/ declares one check. Only .yaml and .yml files are read, and only the first twenty by name. More files raise a policy error row.
Scoping with when
A file scoped on the issue renders no row when no issue is linked.
Requirements
require names one requirement. To accept any of several, list them under any:
Checks that run in CI
run declares a check whose result comes from outerlayer emit:
where: ci check takes no command. OuterLayer never runs it. Record the result with outerlayer emit e2e --result pass --item 412, from CI or a shell. See Record a check.
The emit name criteria is reserved, because outerlayer emit criteria records acceptance criteria. Choose another name.
Checks that run on your host
where: host declares a command your own host runs after a build:
command is required for where: host and not allowed for where: ci. It is a shell line. Exit status 0 is a pass and any other status is a fail.
After a build succeeds, the host runs each command in a new environment that holds no OuterLayer key. Only a result recorded by a host satisfies the check: a pass recorded under the same name by an item key, the build session or a person is ignored. The runner reference describes the steps, the time limit and each failure reason.
What a host check can read
Before a command runs, the host writes a folder and setsOUTERLAYER_EVAL_INPUT to its path. The folder is read-only: a command cannot write into it or change a file in it. Read the files and call nothing. The environment holds no OuterLayer key, so a command cannot ask OuterLayer for this data.
The transcripts come from the build environment, where the agent ran. The host copies them out before it stops that environment, and removes secrets from them before the command can read them. The check environment is a new one and never held them.
activity.json is one JSON object. Its commands and edits come from the same code the evidence rules use, so a script and a session.ran alternative see the same commands, in the same order. The host reads the transcripts in full, whatever capture tier your tenant stores. When your tenant stores less than full, a rule sees fewer commands than the script does.
The folder is present for every
where: host command, at the same path.
Reserved ids
commits-from-sessions, red-then-green, no-test-tampering and tests-after-last-edit are reserved. A validators: entry naming one is ignored. A custom validator file cannot use one as its id or name it in require.
When a file fails to load
A validator file loads whole or not at all. These drop it:- an unknown key, a bad level, or both
requireandrun - a
where: hostcheck with nocommand, or acommandon awhere: cicheck - a
require.validatornaming no file in the directory, or a reserved id - an
emitted:name no file declares withrun - a reserved emit name or id