Skip to content

The hooks other scenarios never reach

pytest 8.1.1 - 9.1.1 - Assertion comparison, deselection, string conditions and the debugger.

These 20 releases produced an identical flow, so they share one page. Identical as far as this scenario can measure - a scenario exercising more hooks may tell them apart.

A plain test run reaches about three quarters of pytest's hooks. The rest need provoking: an assertion that compares rather than merely fails, a deselected test, a skipif written as a string, the assertion-pass hook switched on, and a debugger session.

This scenario provokes all of them in one run, which is why it observes more hooks than any other. pytest.ini enables enable_assertion_pass_hook, and autopdb.py supplies a Pdb that continues the moment it is entered - so the debugger hooks fire without the run hanging.

Three hooks remain out of reach even here. pytest_cmdline_parse runs before any plugin exists to watch it. pytest_report_to_serializable and pytest_report_from_serializable are never called by pytest itself - they belong to plugins that ship reports between processes, such as xdist.

How this was produced

  • Source: scenarios/edge-cases/ - the project this diagram was traced from
  • Invocation: pytest . -k not deselect_me --pdb --pdbcls=autopdb:AutoPdb
  • Traced against: pytest 9.1.1, pluggy 1.6.0, Python 3.12.3
  • Covers: 20 releases - pytest 8.1.1 - 9.1.1, which produced an identical flow
  • Observed: 214 hook calls, 42 distinct hooks

The project is autopdb.py, conftest.py, pytest.ini, test_edges.py. Browse it in the repository rather than inline here - scenarios grow directory trees, and GitHub renders them better than a collapsed block can.

The whole run

Phases left to right, steps top to bottom. Colour says which phase a hook belongs to; how dark a step is says how often it was called. Click a column heading to jump to that phase in full detail, or any hook to open its pytest documentation. Why it looks like this

%3 cluster_phase_startup STARTUP AND CONFIGURATION cluster_startup_n1 pytest_load_initial_conftests cluster_startup_n3 pytest_configure historic cluster_startup_n5 pytest_sessionstart cluster_phase_collection COLLECTION cluster_collection_n1 pytest_collection firstresult cluster_phase_runtest THE RUN-TEST PROTOCOL cluster_runtest_n1 pytest_runtest_protocol firstresult · 4x in run cluster_phase_finish SESSION FINISH cluster_finish_n1 pytest_sessionfinish startup_n2 pytest_plugin_registered historic · x2 startup_n4 pytest_plugin_registered historic · x40 startup_n2->startup_n4 startup_n6 pytest_plugin_registered historic · x39 startup_n4->startup_n6 startup_n7 pytest_report_header startup_n6->startup_n7 collection_n2 pytest_collectstart 3x in run collection_n3 pytest_make_collect_report firstresult · 3x in run collection_n2->collection_n3 collection_n4 pytest_collectreport 3x in run collection_n3->collection_n4 collection_n5 pytest_collectstart 3x in run collection_n4->collection_n5 collection_n6 pytest_make_collect_report firstresult · 3x in run collection_n5->collection_n6 collection_n7 pytest_collectstart 3x in run collection_n6->collection_n7 collection_n8 pytest_make_collect_report firstresult · 3x in run collection_n7->collection_n8 collection_n9 pytest_itemcollected x5 collection_n8->collection_n9 collection_n10 pytest_collectreport x2 collection_n9->collection_n10 collection_n11 pytest_collection_modifyitems collection_n10->collection_n11 collection_n12 pytest_collection_finish collection_n11->collection_n12 collection_n13 pytest_warning_recorded historic collection_n12->collection_n13 runtest_n2 pytest_runtest_logstart 4x in run runtest_n3 pytest_runtest_setup 4x in run runtest_n2->runtest_n3 runtest_n4 pytest_runtest_makereport firstresult · 12x in run runtest_n3->runtest_n4 runtest_n5 pytest_runtest_logreport 12x in run runtest_n4->runtest_n5 runtest_n6 pytest_runtest_call 4x in run runtest_n5->runtest_n6 runtest_n7 pytest_runtest_makereport firstresult · 12x in run runtest_n6->runtest_n7 runtest_n8 pytest_runtest_logreport 12x in run runtest_n7->runtest_n8 runtest_n9 pytest_runtest_teardown 4x in run runtest_n8->runtest_n9 runtest_n10 pytest_runtest_makereport firstresult · 12x in run runtest_n9->runtest_n10 runtest_n11 pytest_runtest_logreport 12x in run runtest_n10->runtest_n11 runtest_n12 pytest_runtest_logfinish 4x in run runtest_n11->runtest_n12 finish_n2 pytest_terminal_summary finish_n3 pytest_unconfigure finish_n2->finish_n3

Startup and configuration

Everything before collection begins: initial conftests are loaded, plugins register, and the session is configured.

%3 cluster_n1 pytest_load_initial_conftests cluster_n3 pytest_configure historic cluster_n5 pytest_sessionstart n2 pytest_plugin_registered historic · x2 n4 pytest_plugin_registered historic · x40 n2->n4 n6 pytest_plugin_registered historic · x39 n4->n6 n7 pytest_report_header n6->n7

Collection

Finding tests. Note that pytest_make_collect_report recurses - directories contain directories contain files.

%3 cluster_n1 pytest_collection firstresult cluster_n3 pytest_make_collect_report firstresult · 3x in run cluster_n7 pytest_make_collect_report firstresult · 3x in run cluster_n19 pytest_collect_file 6x in run cluster_n22 pytest_make_collect_report firstresult · 3x in run cluster_n24 pytest_pycollect_makeitem firstresult · x3 cluster_n27 pytest_pycollect_makeitem firstresult · x2 cluster_n31 pytest_collection_modifyitems cluster_n33 pytest_collection_finish n2 pytest_collectstart 3x in run n4 pytest_collect_directory firstresult n2->n4 n5 pytest_collectreport 3x in run n4->n5 n6 pytest_collectstart 3x in run n5->n6 n8 pytest_ignore_collect firstresult · x2 n6->n8 n9 pytest_collect_file 6x in run n8->n9 n10 pytest_ignore_collect firstresult · 7x in run n9->n10 n11 pytest_collect_file 6x in run n10->n11 n12 pytest_ignore_collect firstresult · 7x in run n11->n12 n13 pytest_collect_file 6x in run n12->n13 n14 pytest_ignore_collect firstresult · 7x in run n13->n14 n15 pytest_collect_file 6x in run n14->n15 n16 pytest_ignore_collect firstresult · 7x in run n15->n16 n17 pytest_collect_file 6x in run n16->n17 n18 pytest_ignore_collect firstresult · 7x in run n17->n18 n20 pytest_pycollect_makemodule firstresult n18->n20 n21 pytest_collectstart 3x in run n20->n21 n23 pytest_pycollect_makeitem firstresult · x3 n21->n23 n25 pytest_generate_tests 5x in run n23->n25 n26 pytest_pycollect_makeitem firstresult · 9x in run n25->n26 n28 pytest_generate_tests 5x in run n26->n28 n29 pytest_itemcollected x5 n28->n29 n30 pytest_collectreport x2 n29->n30 n32 pytest_deselected n30->n32 n34 pytest_report_collectionfinish n32->n34 n35 pytest_warning_recorded historic n34->n35

The run-test protocol

pytest_runtestloop runs this once per collected test. Note the rhythm: setup, call and teardown are each followed by their own makereport and logreport pair - three reports per test, not one.

%3 cluster_n1 pytest_runtest_protocol firstresult · 4x in run cluster_n5 pytest_runtest_logreport 12x in run cluster_n7 pytest_runtest_call 4x in run cluster_n8 pytest_pyfunc_call firstresult · 4x in run cluster_n12 pytest_runtest_logreport 12x in run cluster_n16 pytest_runtest_logreport 12x in run n2 pytest_runtest_logstart 4x in run n3 pytest_runtest_setup 4x in run n2->n3 n4 pytest_runtest_makereport firstresult · 12x in run n3->n4 n6 pytest_report_teststatus firstresult · 13x in run n4->n6 n9 pytest_assertrepr_compare 2x in run n6->n9 n10 pytest_assertion_pass 2x in run n9->n10 n11 pytest_runtest_makereport firstresult · 12x in run n10->n11 n13 pytest_report_teststatus firstresult · 13x in run n11->n13 n14 pytest_runtest_teardown 4x in run n13->n14 n15 pytest_runtest_makereport firstresult · 12x in run n14->n15 n17 pytest_report_teststatus firstresult · 13x in run n15->n17 n18 pytest_runtest_logfinish 4x in run n17->n18

The path 1 of 4 took.

The others differed:

  • 1 test also called pytest_enter_pdb, pytest_exception_interact, pytest_leave_pdb and skipped pytest_assertion_pass
  • 1 test also called pytest_markeval_namespace and skipped pytest_assertion_pass, pytest_assertrepr_compare
  • 1 test skipped pytest_assertrepr_compare

Session finish

Summary reporting and teardown of the session.

%3 cluster_n1 pytest_sessionfinish cluster_n2 pytest_terminal_summary n3 pytest_report_teststatus firstresult · 13x in run n4 pytest_unconfigure n3->n4

Every hook observed

Hook Calls Semantics Implemented by, in call order
pytest_assertion_pass 2 conftest.py
pytest_assertrepr_compare 2 _pytest.assertion
pytest_cmdline_main 1 firstresult _pytest.setupplan [tryfirst]
_pytest.setuponly [tryfirst]
_pytest.mark [tryfirst]
_pytest.helpconfig
_pytest.fixtures
_pytest.main
1
pytest_collect_directory 1 firstresult _pytest.python
_pytest.main
pytest_collect_file 6 _pytest.doctest
_pytest.python
pytest_collection 1 firstresult _pytest.logging.LoggingPlugin (logging-plugin) [wrapper, tryfirst]
_pytest.warnings [wrapper, tryfirst]
_pytest.config.Config (pytestconfig) [wrapper]
_pytest.terminal.TerminalReporter
_pytest.assertion
_pytest.main
pytest_collection_finish 1 _pytest.fixtures.FixtureManager (funcmanage)
_pytest.terminal.TerminalReporter
2
pytest_collection_modifyitems 1 _pytest.fixtures.FixtureManager (funcmanage)
_pytest.main
_pytest.mark
pytest_collectreport 3 _pytest.main.Session.pytest_runtest_logreport (session) [tryfirst]
_pytest.terminal.TerminalReporter
pytest_collectstart 3 _pytest.main.Session [tryfirst]
pytest_configure 1 historic _pytest.subtests
_pytest.faulthandler
_pytest.warnings
_pytest.threadexception
_pytest.unraisableexception
_pytest.junitxml
_pytest.assertion
_pytest.tmpdir
_pytest.legacypath
_pytest.skipping
_pytest.unittest
_pytest.debugging
_pytest.terminal
_pytest.python
_pytest.mark
_pytest.config.PytestPluginManager (<anonymous>)
_pytest.pastebin [trylast]
_pytest.logging [trylast]
3
pytest_deselected 1 _pytest.terminal.TerminalReporter
pytest_enter_pdb 1 _pytest.faulthandler [tryfirst]
pytest_exception_interact 1 _pytest.faulthandler [tryfirst]
_pytest.debugging.PdbInvoke
pytest_generate_tests 5 _pytest.fixtures.FixtureManager (funcmanage)
_pytest.python
pytest_ignore_collect 7 firstresult _pytest.main
pytest_itemcollected 5 -
pytest_leave_pdb 1 -
pytest_load_initial_conftests 1 _pytest.warnings [wrapper]
_pytest.capture [wrapper]
_pytest.legacypath [tryfirst]
_pytest.config.Config (pytestconfig) [trylast]
4
pytest_make_collect_report 3 firstresult _pytest.fixtures.FixtureManager (funcmanage) [wrapper]
_pytest.capture.CaptureManager [wrapper]
_pytest.runner
5
pytest_markeval_namespace 1 -
pytest_plugin_registered 83 historic _pytest.legacypath
_pytest.terminal.TerminalReporter
_pytest.fixtures.FixtureManager (funcmanage)
pytest_pycollect_makeitem 9 firstresult _pytest.unittest
_pytest.python [trylast]
pytest_pycollect_makemodule 1 firstresult _pytest.python
pytest_pyfunc_call 4 firstresult _pytest.python [trylast]
pytest_report_collectionfinish 1 -
pytest_report_header 1 _pytest.terminal.TerminalReporter
_pytest.helpconfig
pytest_report_teststatus 13 firstresult _pytest.subtests [tryfirst]
_pytest.skipping
_pytest.runner
_pytest.terminal [trylast]
6
pytest_runtest_call 4 _pytest.logging.LoggingPlugin (logging-plugin) [wrapper]
_pytest.capture.CaptureManager [wrapper]
_pytest.skipping [wrapper]
_pytest.runner
_pytest.unraisableexception [trylast]
_pytest.threadexception [trylast]
pytest_runtest_logfinish 4 _pytest.logging.LoggingPlugin (logging-plugin)
7
pytest_runtest_logreport 12 _pytest.main.Session [tryfirst]
_pytest.logging.LoggingPlugin (logging-plugin)
_pytest.terminal.TerminalReporter
pytest_runtest_logstart 4 _pytest.logging.LoggingPlugin (logging-plugin)
_pytest.terminal.TerminalReporter
pytest_runtest_makereport 12 firstresult _pytest.tmpdir [wrapper, tryfirst]
_pytest.skipping [wrapper]
_pytest.unittest [tryfirst]
_pytest.runner
pytest_runtest_protocol 4 firstresult _pytest.warnings [wrapper, tryfirst]
_pytest.assertion [wrapper, tryfirst]
_pytest.unittest [wrapper]
_pytest.faulthandler [wrapper, trylast]
_pytest.runner
pytest_runtest_setup 4 _pytest.logging.LoggingPlugin (logging-plugin) [wrapper]
_pytest.capture.CaptureManager [wrapper]
_pytest.skipping [tryfirst]
_pytest.runner
_pytest.unraisableexception [trylast]
_pytest.threadexception [trylast]
pytest_runtest_teardown 4 _pytest.logging.LoggingPlugin (logging-plugin) [wrapper]
_pytest.capture.CaptureManager [wrapper]
_pytest.runner
_pytest.unraisableexception [trylast]
_pytest.threadexception [trylast]
pytest_runtestloop 1 firstresult _pytest.logging.LoggingPlugin (logging-plugin) [wrapper]
_pytest.terminal.TerminalReporter [wrapper]
_pytest.main
8
pytest_sessionfinish 1 _pytest.logging.LoggingPlugin (logging-plugin) [wrapper, tryfirst]
_pytest.terminal.TerminalReporter [wrapper]
_pytest.warnings [wrapper]
_pytest.assertion
_pytest.tmpdir
_pytest.runner
pytest_sessionstart 1 _pytest.logging.LoggingPlugin (logging-plugin) [wrapper, tryfirst]
_pytest.fixtures
_pytest.runner
_pytest.terminal.TerminalReporter [trylast]
pytest_terminal_summary 1 _pytest.terminal.TerminalReporter [wrapper]
_pytest.warnings [wrapper]
_pytest.pastebin
_pytest.runner
pytest_unconfigure 1 _pytest.logging.LoggingPlugin (logging-plugin)
_pytest.terminal.TerminalReporter
_pytest.faulthandler
_pytest.unraisableexception
_pytest.doctest
_pytest.junitxml
_pytest.pastebin
_pytest.mark
9
pytest_warning_recorded 1 historic _pytest.terminal.TerminalReporter

9 of these changed implementer somewhere in pytest 8.1.1 - 9.1.1 without changing the flow - pytest moves implementations between its own plugins. The table shows the newest; what changed along the way is below.

  1. 8.2.0 gained _pytest.fixtures and lost _pytest.python ↩

  2. 9.1.0 gained _pytest.fixtures.FixtureManager (funcmanage) ↩

  3. 8.4.0 gained _pytest.threadexception, _pytest.unraisableexception; 8.4.1 gained _pytest.unittest; 9.0.0 gained _pytest.subtests; 9.1.0 gained _pytest.assertion ↩

  4. 8.4.0 lost _pytest.python_path ↩

  5. 9.1.0 gained _pytest.fixtures.FixtureManager (funcmanage) ↩

  6. 9.0.0 gained _pytest.subtests ↩

  7. 8.3.0 lost _pytest.terminal.TerminalReporter ↩

  8. 8.3.0 gained _pytest.terminal.TerminalReporter ↩

  9. 8.4.0 lost _pytest.python_path; 9.1.0 gained _pytest.unraisableexception ↩