pytest INTERNALERROR: A Standalone test_*.py Script Crashed the Whole Suite
>> The symptom: pytest at repo root aborted with INTERNALERROR before a single real test ran.>> The cause: 18 files named test_*.py were standalone scripts meant to run with python <file>.py directly, never pytest tests. Six called sys.exit() at import time. pytest's collector can't catch SystemExit — it isn't an Exception — so importing any one of them took the whole session down.>> The fix: conftest.py's collect_ignore / collect_ignore_glob, naming every offending file explicitly.>> The proof: 418 passed, 1 skipped, clean twice in a row.
The symptom: zero tests ran, and the error wasn't about any of them
Running pytest from the repo root should have run 418 real tests. Instead it stopped immediately with an INTERNALERROR, no pass count, no fail count, nothing that looked like a normal test failure.
No test in the 418-test suite had failed. The suite had never been reached. Whatever broke, broke before collection finished — the phase where pytest just walks the directory tree looking for files and functions to run, not the phase where it runs anything.
Think of collection as pytest reading the table of contents before it reads the book. If page one of the table of contents itself explodes, you never find out whether chapter four has a typo. The 418 real tests were chapter four. They were fine. Nobody got there.
Why: 18 files that were never pytest tests at all
pytest's default discovery rule is simple: any file matching test_*.py or *_test.py gets imported and scanned for test_* functions or Test* classes. It doesn't check whether the file wants to be a test file — it just matches the name.
This repo had 18 files matching that glob (13 at repo root, 2 under chief-home/test/) that were something else entirely: standalone diagnostic and one-shot scripts, meant to be run directly with python test_whatever.py, documented as such in their own module docstrings. None of them defined a test_* function or Test* class. None guarded their top-level work behind if __name__ == "__main__":. Their names collided with pytest's glob by accident, not by design.
What these files actually were
test_adb_connect.py, test_shush.py, test_restart_gate.py and 15 more — hand-run diagnostics with top-level code and a docstring saying "run me directly," never imported by anything.
What pytest assumed they were
418 members of one collectible test session, because the filename matched test_*.py. pytest imports first, decides what's runnable second.
The part that made it fatal, not just noisy
A standalone script that pytest mistakenly imports usually just contributes zero tests and maybe a warning. That's not what happened here, because six of these eighteen files called sys.exit() (or raised SystemExit directly) at import time, as part of their own normal "run me and I'll exit with a status code" design.
SystemExit is not a subclass of Exception — it inherits from BaseException directly, on purpose, so that a bare except Exception: block never accidentally swallows a deliberate process exit. pytest's collector wraps each file import in exactly that kind of handler. It can catch an ImportError, a SyntaxError, almost anything that inherits from Exception. It cannot catch SystemExit. So instead of "file skipped, one warning logged," the entire pytest process itself terminated mid-collection.
pytest imports test_update_gate.py during collection
-> module-level code runs immediately (no __main__ guard)
-> that code calls sys.exit(1) / raises SystemExit
-> SystemExit is BaseException, not Exception
-> pytest's collector except-block doesn't match it
-> the whole pytest process unwinds -> INTERNALERROR
-> 0 of 418 real tests ever ranA seventeenth file, test_update_gate.py, failed a different way: its unguarded top-level code deleted a temp file a second time and raised a plain collection error instead of a clean SystemExit. Same root cause — top-level code with side effects running on pytest's schedule, not the script author's — different crash shape.
What pytest actually touched before the crash
The fix: tell pytest to stop looking, don't fix the scripts
These 18 files were correct as standalone scripts. Renaming them, adding __main__ guards, or restructuring them to be pytest-safe would have been real work for zero benefit — they still needed to run exactly the way their own docstrings described. The bug wasn't in the scripts. It was in pytest's default assumption that a matching filename means "collect this."
conftest.py already runs before collection starts, so it's the right place to tell pytest which matches to skip. Two hooks do it: collect_ignore for an explicit file list, collect_ignore_glob for a whole directory pattern.
No behavior changed in any of the excluded scripts. Every one of them still runs standalone, exactly as its own docstring says, with python test_whatever.py. The only thing that changed is that pytest's collector no longer imports them as a side effect of matching their filename.
Proof it actually worked
The fix landed as commit b41bd17 ("Fix pytest collection crashes from standalone test_*.py scripts"), and the same exclusion list had to be re-applied on master separately in e6d3d40 after a branch history gap — the exact symptom (INTERNALERROR before any test ran) recurred until the second commit landed, which is its own reminder that a conftest fix like this needs to travel with every branch it protects, not just the one it was written on.
Two other real repos this same failure shows up in the wild for: any project that keeps a folder of one-off diagnostic scripts named like tests, and any project that lets a CI job's exit-code convention (sys.exit(0) on success, sys.exit(1) on failure) leak into a file pytest also happens to be scanning.
The general rule, not just this repo's rule: if a Python file is meant to be run directly and not imported, name it so pytest's default globs don't match it (or wrap all top-level work in if __name__ == "__main__":), and if you can't rename it, tell conftest.py to skip it explicitly. A file that calls sys.exit() at import time sitting in a directory pytest scans is a landmine for the next person who just wants to run pytest and see their real test count.