Your CI might be green for the wrong reason

A broken expect printed a red x. My test runner was searching the output for FAIL.

So for a while, a failing test could still leave CI green. The suite guards Sprout, the language whose core I froze at v0.1.0, and it had more ways to pass when it shouldn’t: a test file with zero tests in it, and a sort check that can’t tell the same list from a copy.

The red x that grep never saw

Sprout has two kinds of test files. Plain scripts print FAIL when a check breaks. Files built on the test / expect framework print a red x instead. The runner, src/tests/run.sh, used to search every file’s output for FAIL, so a broken expect matched nothing and its file counted as a pass.

The roadmap gives it one line. The fix is listed there as “closing the hole where a failing expect (prints x, not FAIL) slipped past the grep”, and it splits the gate by filename. A *_test.sprout file runs through sprout test and is judged by its exit code, and everything else still goes through grep:

case "$f" in
  *_test.sprout)
    if out="$("$bin" test "$f" 2>&1)"; then
      echo "ok:   $f (sprout test)"
  ...
  *)
    if out="$("$bin" run "$f" 2>&1)" && ! printf '%s' "$out" | grep -qiE "sprout error|FAIL"; then

The exit code of sprout test comes from one C function. I’ve trimmed it to the lines that decide pass or fail.

static int test_report(void) {
  if (g_tpass + g_tfail == 0) return 0;
  printf("\n  " C_BOLD "%d passed" C_RESET, g_tpass);
  ...
  return g_tfail > 0 ? 1 : 0;
}

grep only knew about FAIL, and the expect files never print it. An exit code doesn’t care which format a file prints in.

When zero tests run, sprout test exits 0

Look at the first line of test_report again. If no tests ran, it returns 0, which the runner reads as success. Delete every test block from a _test.sprout file and CI passes it.

The scripts have the same gap. contracts.sprout ends with show "contracts: all checks ran", but run.sh only checks the exit code and that sprout error and FAIL are absent from the output. Nothing requires that closing line, so an empty script passes the same gate. The fix I’d make: test_report returns 1 when zero tests ran, and the runner insists on each script’s closing line.

The gate was never made to prove it could fail. The cheapest proof is to flip one expected value, push, and watch CI go red. If it stays green, the suite isn’t protecting anything, and I’d rather find that out from a test I broke on purpose.

The sort check that would pass a copy

The contract tests use the plain-script style. Each promise is one check call, labelled like the spec, so a failure names the promise it broke. Here is the helper and the two checks on sort:

task check(label, got, want):
    when got != want:
        show "FAIL", label, "- got", got, "wanted", want
...
make same = sort(xs)
check("sort returns the SAME list (a reference)", same, xs)
add(same, 0)
check("...so mutating the result mutates the original", xs, sort(xs))

The labels promise reference semantics, and the checks can’t see references. check compares with !=, and equality in Sprout is deep. If sort sorted xs in place but returned a sorted copy, same would still equal xs, so the first check passes. The second passes either way, because sort(xs) on the right sorts xs before the comparison runs.

The question that exposes this is one you can ask of any test: what wrong implementation would still pass it? Here the answer is exactly the bug the label names. The same file already has the right pattern for list aliases (mutate b, observe a), so a copy fails it. My rewrite applies that pattern to sort. It’s not in the repo yet:

make same = sort(xs)
add(same, 0)
check("sort returns the SAME list (a reference)", last(xs), 0)

If sort returns a copy, the 0 lands in the copy and never reaches xs. Then last(xs) is something else, and the check prints FAIL.

What the v0.1.0 freeze promises

My first freeze, at v0.0.13, held for exactly one version. v0.1.0 is the one that counts, the point where, in the README’s words, “the core stops moving and libraries can build on it.” That makes the freeze a list of promises about observable behaviour: lists and maps are shared references, copy(x) is a deep snapshot, sort returns the list it changed, modulo takes the dividend’s sign, and equality is deep. Each is a labelled line in src/tests/contracts.sprout, and CI runs it on every pull request.

Error kinds are promises too. A caught error carries a kind string that libraries branch on, and the README makes the set add-only (part 2 covers why). A new kind can be added later, but an existing one can’t be renamed or removed.

The process around the freeze is written down. The roadmap’s rule is one batch, one version, one review, with green CI before merge, and CONTRIBUTING.md asks for a test with every fix. In v0.1.22 that review caught a 12–55× slowdown on missing map keys before the release went out.

Three operating systems, release flags included

Every pull request runs the same build and suite on three operating systems. The matrix below is trimmed to the build and test steps.

    strategy:
      fail-fast: false
      matrix:
        os: [ ubuntu-latest, macos-latest, windows-latest ]
    ...
      - name: Build (Linux / macOS)
        if: runner.os != 'Windows'
        run: cc -O2 -Wall -o src/sprout src/sprout.c -lm

      - name: Build (Windows, MinGW)
        if: runner.os == 'Windows'
        run: gcc -O2 -Wall -s -Wl,--stack,67108864 -o src/sprout.exe src/sprout.c -lm -lurlmon
    ...
      - name: Run test suite
        run: bash src/tests/run.sh

With fail-fast: false, a Windows failure doesn’t cancel Linux and macOS, so one run shows every broken platform. The Windows build is effectively a different program, with MinGW, a 64 MB stack and -lurlmon, and the release workflow uses the same compiler flags. After the 51 files in src/tests/, run.sh runs all 16 programs in examples/, and any sprout error fails the build. CI also formats every example and test file twice and diffs the results, and a separate job runs the suite under AddressSanitizer, then again with the collector running on every statement (part 3).

Pushing a v* tag starts release.yml, which builds all three binaries and runs run.sh on each under the comment “never ship a binary that fails its tests”. The publish job has needs: build, so one red platform blocks the release. Two gaps remain. The release reruns the suite but not CI’s bundle, formatter or ASan steps, and no workflow compares the VERSION file, the SPROUT_VERSION macro and the tag.

The README still calls xs[-1] an error

Tests check the interpreter, and nothing checks the docs. At the commit this post describes, the two disagree in at least three places.

The README’s “Decided edge cases” says: “xs[-1] is an error — use last(xs).” Its “Not in the core” list rules out negative indexing and user-defined types, and the wiki repeats both. Classes shipped in v0.1.7 and negative indexing in v0.1.21. This is the index code now.

int len = c.list ? c.list->n : 0; long long i = (long long)ix.num, at = i < 0 ? i + len : i;   /* xs[-1] is the last item */

The freeze survived that change, since it only turned an error into a feature. The docs didn’t, and the tests don’t pin the new behaviour either: the suite reads negative slices like xs[-2:] but never a plain xs[-1]. The header of freeze_test.sprout still lists xs[-1] among errors that “are verified by hand.”

Memory drifted the same way. wiki/architecture.md says “For now, memory is never freed,” while wiki/glossary.md describes the mark-sweep collector that frees lists, maps, closures and strings, and links to that same architecture page.

The error-kind table, “frozen at v0.1.0”, lists math, type, index, io, fail, name and error. error_kind_test.sprout pins math, type, index and error, another file pins fail, and nothing asserts io or name. The interpreter also raises two kinds the table never mentions:

  fail_kind(line, "key", msg);
...
    if(!s[0]) fail_kind(call->line,"value","code needs at least one character, but the text is empty.");

Adding kinds is allowed, but a library author reading the table can’t know to branch on key or value. All three docs were right when I wrote them; nothing told me when they stopped being right. The fixes are tests: one for xs[-1], one expect error per documented kind, a CI check that every kind string in the C source appears in the README table, and a step that runs every fenced sprout block in the README and wiki. Sprout’s CI doesn’t run the docs’ examples yet, and it’s the next gate I’d add.

A checklist for anyone promising stable behaviour

1. Write each promise as “given this input, you observe this result”, with one test labelled after it.
2. For every test, name a wrong implementation that would still pass it.
3. Gate on exit codes, and treat zero tests run as a failure.
4. Break one test on purpose and watch CI go red.
5. Assert errors and their kinds, and keep enum-like strings add-only.
6. Run CI on every OS you ship, with the release’s compiler flags and fail-fast: false.
7. Run the docs’ examples in CI, and publish only when every platform passes.

I broke sort on purpose in a local build, making it sort in place but return a copy. contracts.sprout still printed “contracts: all checks ran” with no FAIL. With the check rewritten to compare identity, the same broken build prints FAIL sort returns the SAME list (a reference) - got 5 wanted 0, and the real build stays clean. Until that rewrite lands in the repo, CI would wave this bug through.