Failures and troubleshooting
Every way a steep test fails, what it prints, and what to do about the usual problems.
The output on this page is real. Each is from a test in failures_test.go, run with go test.
| Failure | Stops the test? | Recorded? |
|---|---|---|
A failed Assert | No | Yes, as failure:n |
A failed Require | Yes | Yes |
| A wait that times out | Yes | Yes |
| A settle that never settles | Yes | Yes |
| A snapshot that differs | With Require | Yes |
| A missing snapshot | With Require | Yes |
| A method the harness can't do | Yes | Yes |
| A condition that locates nothing | Yes | Yes |
| An output that doesn't parse | Yes, at the start | No |
| A hook's error | Yes | Yes |
| A program that fails | Yes | Yes |
| A test that panics | Yes | Yes |
go test -timeout | Kills the process | No |
A failed Assert
Assert reports the failure and the test carries on, so you see every problem in a run.
--- FAIL: TestFailAssert
failures_test.go:32: expected output to satisfy contains ["Genmaicha"]
output:
╭──────────────────────────────────────────────────────────────────────╮
│ The Tea Shop │
│ │
│ Sencha green 1m │
│ Matcha green 0m │
│ Gyokuro green 2m │
│ Oolong oolong 3m │
│ Assam black 4m │
│ Darjeeling black 3m │
│ Pu-erh dark 5m │
│ │
│ Pick a tea. │
│ license tea-8f3a91c2d7 │
╰──────────────────────────────────────────────────────────────────────╯
↑/↓ choose / filter enter brew q quit
failures_test.go:33: expected output to satisfy contains ["Hojicha"]
output:
╭──────────────────────────────────────────────────────────────────────╮
│ The Tea Shop │
│ │
│ Sencha green 1m │
│ Matcha green 0m │
│ Gyokuro green 2m │
│ Oolong oolong 3m │
│ Assam black 4m │
│ Darjeeling black 3m │
│ Pu-erh dark 5m │
│ │
│ Pick a tea. │
│ license tea-8f3a91c2d7 │
╰──────────────────────────────────────────────────────────────────────╯
↑/↓ choose / filter enter brew q quit
harness.go:233: steep: export: wrote ../../docs/public/demos/fail-assert.webp
harness.go:233: steep: export: wrote ../../docs/public/casts/fail-assert.cast
FAIL
FAIL getsteep.dev/steep/_examples/docs-demos
FAILA failed Require
Require reports the same failure and stops, so the later steps don't run.
--- FAIL: TestFailRequire
failures_test.go:41: expected output to satisfy contains ["Genmaicha"]
output:
╭──────────────────────────────────────────────────────────────────────╮
│ The Tea Shop │
│ │
│ Sencha green 1m │
│ Matcha green 0m │
│ Gyokuro green 2m │
│ Oolong oolong 3m │
│ Assam black 4m │
│ Darjeeling black 3m │
│ Pu-erh dark 5m │
│ │
│ Pick a tea. │
│ license tea-8f3a91c2d7 │
╰──────────────────────────────────────────────────────────────────────╯
↑/↓ choose / filter enter brew q quit
harness.go:233: steep: export: wrote ../../docs/public/demos/fail-require.webp
harness.go:233: steep: export: wrote ../../docs/public/casts/fail-require.cast
FAIL
FAIL getsteep.dev/steep/_examples/docs-demos
FAILA wait that times out
The message says what the wait wanted, and shows the screen as it was. Compare the two. The default timeout is 2s, so a slow program can hit this, and STEEP_TIMEOUT raises it.
--- FAIL: TestFailWait
failures_test.go:50: Wait("Genmaicha"): reel: wait for contains ["Genmaicha"]: steep: timeout
last output:
╭──────────────────────────────────────────────────────────────────────╮
│ The Tea Shop │
│ │
│ Sencha green 1m │
│ Matcha green 0m │
│ Gyokuro green 2m │
│ Oolong oolong 3m │
│ Assam black 4m │
│ Darjeeling black 3m │
│ Pu-erh dark 5m │
│ │
│ Pick a tea. │
│ license tea-8f3a91c2d7 │
╰──────────────────────────────────────────────────────────────────────╯
↑/↓ choose / filter enter brew q quit
harness.go:233: steep: export: wrote ../../docs/public/demos/fail-wait.webp
harness.go:233: steep: export: wrote ../../docs/public/casts/fail-wait.cast
FAIL
FAIL getsteep.dev/steep/_examples/docs-demos
FAILA settle that never settles
The screen was still changing when the timeout came, usually because of a spinner or a clock. Use Wait or WaitSettleMessages. The (for 100ms) is the quiet period it was looking for.
--- FAIL: TestFailSettle
failures_test.go:73: reel: wait for settle: steep: timeout (for 100ms)
last output:
frame x
harness.go:233: steep: export: /tmp/go-test is removed when the test ends; pass -artifacts to go test, or set STEEP_EXPORT_DIR or WithArtifacts
harness.go:233: steep: export: wrote /tmp/go-test/TestFailSettle_failed.gif
harness.go:233: steep: export: wrote /tmp/go-test/TestFailSettle_failed.cast
FAIL
FAIL getsteep.dev/steep/_examples/docs-demos
FAILA snapshot that differs
The diff goes from the file to the screen, so - is what was expected and + is what the test saw. If the change is right, update the snapshot.
--- FAIL: TestFailSnapshot
failures_test.go:79: snapshot "../../docs/fixtures/TestFailSnapshot.snap" does not match:
--- ../../docs/fixtures/TestFailSnapshot.snap
+++ actual
@@ -7,9 +7,9 @@
│ Oolong oolong 3m │
│ Assam black 4m │
│ Darjeeling black 3m │
-│ Genmaicha dark 5m │
+│ Pu-erh dark 5m │
│ │
-│ Choose one. │
+│ Pick a tea. │
│ license tea-8f3a91c2d7 │
╰──────────────────────────────────────────────────────────────────────╯
↑/↓ choose / filter enter brew q quit
harness.go:233: steep: export: wrote ../../docs/public/demos/fail-snapshot.webp
harness.go:233: steep: export: wrote ../../docs/public/casts/fail-snapshot.cast
FAIL
FAIL getsteep.dev/steep/_examples/docs-demos
FAILA missing snapshot
There's no file to compare with, which is a failure so that CI can't quietly pass without one. Create it with UPDATE_SNAPSHOTS=true, and commit it.
--- FAIL: TestFailSnapshotMissing
failures_test.go:85: snapshot "/tmp/go-test/TestFailSnapshotMissing.snap" does not exist; set UPDATE_SNAPSHOTS=true to create it
harness.go:233: steep: export: /tmp/go-test is removed when the test ends; pass -artifacts to go test, or set STEEP_EXPORT_DIR or WithArtifacts
harness.go:233: steep: export: wrote /tmp/go-test/TestFailSnapshotMissing_failed.gif
harness.go:233: steep: export: wrote /tmp/go-test/TestFailSnapshotMissing_failed.cast
FAIL
FAIL getsteep.dev/steep/_examples/docs-demos
FAILA method the harness can't do
An exec harness has no program to quit, so this fails the test and says so. Its failure is Fatalf, so the test stops.
--- FAIL: TestFailExecProgram
failures_test.go:91: steep: Harness.QuitProgram needs a tea.Program, and this harness runs a command (see NewExecHarness)
harness.go:233: steep: export: /tmp/go-test is removed when the test ends; pass -artifacts to go test, or set STEEP_EXPORT_DIR or WithArtifacts
harness.go:233: steep: export: wrote /tmp/go-test/TestFailExecProgram_failed.gif
harness.go:233: steep: export: wrote /tmp/go-test/TestFailExecProgram_failed.cast
FAIL
FAIL getsteep.dev/steep/_examples/docs-demos
FAILA condition that locates nothing
Locate needs a cell. reel.NotContains and reel.ScreenFunc hold without pointing at one. Clicking off the screen, as with too large an Offset, fails in the same way.
--- FAIL: TestFailLocate
failures_test.go:97: steep: not contains ["Genmaicha"] holds but locates nothing
harness.go:233: steep: export: /tmp/go-test is removed when the test ends; pass -artifacts to go test, or set STEEP_EXPORT_DIR or WithArtifacts
harness.go:233: steep: export: wrote /tmp/go-test/TestFailLocate_failed.gif
harness.go:233: steep: export: wrote /tmp/go-test/TestFailLocate_failed.cast
FAIL
FAIL getsteep.dev/steep/_examples/docs-demos
FAILAn output that doesn't parse
An entry in STEEP_EXPORT that doesn't parse fails the test when the harness starts, so a typo can't lose you a recording. The rest of the variables are more forgiving: a value that doesn't parse is ignored.
--- FAIL: TestFailBadExport
options.go:522: steep: STEEP_EXPORT: reel: output "out.gif?bogus=1": unknown parameter "bogus"
FAIL
FAIL getsteep.dev/steep/_examples/docs-demos
FAILA hook's error
An error from a hook fails the test immediately.
A program that fails
WaitFinished and FinalModel fail with the program failed: if the program returned an error other than tea.ErrProgramKilled, or if it panicked. So does Close, so a failed program fails the test even if nothing waited for it.
Bubble Tea catches a panic in Update, View, or a command, and prints Caught panic: and a stack to the test's output, so look there for where it happened.
A test that panics
The cleanup still runs, so the recording is written, and test panicked is added to the result. go test prints the panic and its stack as usual.
go test -timeout
go test kills the whole process when its timeout passes, before any cleanups run. Nothing is written, and the output shows only the stacks of the running goroutines. Keep the harness's timeout well below it.
Troubleshooting
A test fails, or is flaky
It passes alone and fails in CI
Most flaky tests check something before the program has drawn it.
Assertright after an input. The program hasn't redrawn yet. UseWait. See Assert vs Require vs Wait.- The timeout is too short. The default is 2s, and a loaded CI machine can take longer. Set
STEEP_TIMEOUT=30son the job, and see if the failures go away. Sleepinstead of a wait. It only works if the machine is fast enough. Wait for what you expect.WaitSettlereturned too early. If the program does something slow before redrawing, the screen looks settled. Wait for the text.
A wait times out, but the text is on the screen
Look at the last output in the failure.
- The text is styled in pieces. It doesn't matter for
Wait, which checks the screen's text. - The text is wrapped or cut off. It has to be on the screen in one piece. Check the terminal's size, or use
WithWindowSize. - It appeared and went away. Waits see the current screen. If a message shows briefly, wait for what follows it, or open the recording.
- A wide character. Text is compared as the screen has it. Check for a character that looks like the one you typed.
A key seems to be ignored
- The program wasn't ready. A key sent before the program has set up its screen may go to the wrong place.
Waitfor the screen first. - The name is wrong. An unknown key name fails the test, but
Type("\n")andKey("enter")aren't always the same to a program. UseKey. - A mode can't encode it. Combinations such as
ctrl+shift+aneed the kitty keyboard protocol, and fail without it.
A mouse click does nothing
Your program must ask for the mouse, with view.MouseMode, or events are dropped. See Mouse. Also check the coordinates, which start at 0. Locate avoids having to know them.
Snapshots fail in CI but pass locally
- Colors. A JSON snapshot's colors depend on the color profile. Set
WithColorexplicitly. - The environment. A program under
NewExecHarnesssees the host's environment, includingTERMandNO_COLOR. - Times, paths, and names. Anything from the machine ends up in the snapshot. Pass in fixed values, or use
snapshot.WithTransform. - Line endings. Check
.gitattributeson Windows runners. Snapshots are compared with LF.
The snapshot's numbers changed
Each snapshot in a test takes the next number, so a new one in the middle renumbers the rest. Name them with a suffix.
Nothing was recorded
Go through these in order:
- Is there a policy? Without
STEEP_EXPORT_MODEorWithExport*, nothing's written. - Did the test fail?
failuremode writes nothing for a passing test. - Was it removed? Look for
is removed when the test endsin the output. Pass-artifacts, or setSTEEP_EXPORT_DIR. - Are you looking in the right directory? A relative
STEEP_EXPORT_DIRis relative to each test's package directory. The test logs each path it wrote, with-vor on failure. - Was it killed? A
go test -timeoutwrites nothing. See the cases.
An export failed
ffmpeg, reel.ErrRawOutput, and exports reel refuses
See reel's troubleshooting, which these come from. A profile with effects.Redact can't also write .cast.
A mark is missing
A profile looked for a mark the recording doesn't have. See Missing marks.
The recording is too fast to follow
Recordings play in real time, and a test takes milliseconds. See How recording works.
The colors look wrong
The default is a 256-color terminal, so Bubble Tea downsamples 24-bit colors. Use steep.WithColor(steep.ColorTrueColor).
Icons or emoji are boxes
reel embeds a font, but not every glyph. See Fonts to add a Nerd Font, or STEEP_RENDERER_FONT_FAMILY to change it for one run.
Getting help
Open an issue or ask on Discord. A .cast of the failure helps a lot, if there's nothing secret in it.
Last updated on