Skip to content

Distributed across workers

pytest 8.0.0 - 8.0.2 - The controller and its workers, which see completely different runs.

These 3 releases produced an identical flow, so they share one page. Identical as far as this scenario can measure - a scenario exercising more hooks may tell them apart.

Run under pytest-xdist, a session is several processes, and they do not see the same thing at all. The process you launched - the controller - never collects a test and never runs one. It starts workers, hands out work, and receives results. Each worker collects the entire suite for itself, then runs whatever it was given.

That is why this page has a diagram per process rather than one. The controller reaches around 25 hooks, a worker around 41, and the overlap between them is small.

The scheduler changes what you see, so the command is printed with every diagram below. This capture uses --dist each, which gives every worker the whole suite: the workers are then identical, and the page draws them once. Under a splitting scheduler - loadfile, loadscope, load, worksteal - workers differ by the tests they happened to draw, and this page would show one section per distinct worker flow. The controller differs too: those four modes all produce one controller flow, and each produces another.

This scenario also reaches the last two hooks pytest declares but never calls itself: pytest_report_to_serializable in the workers, where a result is packed up to be sent, and pytest_report_from_serializable in the controller, where it is unpacked. And it brings twelve hooks of xdist's own, which pytest knows nothing about - they are described here because hookspecs are read from the running plugin manager rather than from pytest's source.

How this was produced

  • Source: scenarios/xdist/ - the project this diagram was traced from
  • Invocation: pytest . -q -n 2 --dist each
  • Traced against: pytest 8.0.2, pluggy 1.6.0, Python 3.12.3
  • Covers: 3 releases - pytest 8.0.0 - 8.0.2, which produced an identical flow
  • Observed: 534 hook calls, 26 distinct hooks

The project is test_alpha.py, test_beta.py, test_delta.py, test_gamma.py, test_outcomes.py. Browse it in the repository rather than inline here - scenarios grow directory trees, and GitHub renders them better than a collapsed block can.

The controller

Command: pytest . -q -n 2 --dist each

Phases left to right, steps top to bottom. Colour says which phase a hook belongs to; how dark a step is says how often it was called. Click a column heading to jump to that phase in full detail, or any hook to open its pytest documentation. Why it looks like this

%3 cluster_phase_startup STARTUP AND CONFIGURATION cluster_startup_n1 pytest_load_initial_conftests cluster_startup_n3 pytest_configure historic cluster_startup_n5 pytest_sessionstart cluster_phase_collection COLLECTION cluster_phase_runtest THE RUN-TEST PROTOCOL cluster_runtest_n1 pytest_runtestloop firstresult cluster_phase_finish SESSION FINISH cluster_finish_n1 pytest_sessionfinish startup_n2 pytest_plugin_registered historic · 93x in run startup_n4 pytest_plugin_registered historic · x43 startup_n2->startup_n4 startup_n6 pytest_plugin_registered historic · x43 startup_n4->startup_n6 startup_n7 pytest_xdist_setupnodes startup_n6->startup_n7 startup_n8 pytest_xdist_newgateway 2x in run startup_n7->startup_n8 startup_n9 pytest_plugin_registered historic · 93x in run startup_n8->startup_n9 startup_n10 pytest_configure_node 2x in run startup_n9->startup_n10 startup_n11 pytest_xdist_getremotemodule firstresult · 2x in run startup_n10->startup_n11 startup_n12 pytest_xdist_newgateway 2x in run startup_n11->startup_n12 startup_n13 pytest_plugin_registered historic · 93x in run startup_n12->startup_n13 startup_n14 pytest_configure_node 2x in run startup_n13->startup_n14 startup_n15 pytest_xdist_getremotemodule firstresult · 2x in run startup_n14->startup_n15 collection_n1 pytest_collection firstresult runtest_n2 pytest_xdist_make_scheduler firstresult runtest_n3 pytest_testnodeready 2x in run runtest_n2->runtest_n3 runtest_n4 pytest_xdist_node_collection_finished 2x in run runtest_n3->runtest_n4 runtest_n5 pytest_testnodeready 2x in run runtest_n4->runtest_n5 runtest_n6 pytest_xdist_node_collection_finished 2x in run runtest_n5->runtest_n6 runtest_n7 pytest_runtest_logstart x2 runtest_n6->runtest_n7 runtest_n8 pytest_report_from_serializable firstresult · x2 runtest_n7->runtest_n8 runtest_n9 pytest_runtest_logreport x2 runtest_n8->runtest_n9 runtest_n10 pytest_report_from_serializable firstresult · 112x in run runtest_n9->runtest_n10 runtest_n11 pytest_runtest_logreport 112x in run runtest_n10->runtest_n11 runtest_n12 pytest_report_from_serializable firstresult · 112x in run runtest_n11->runtest_n12 runtest_n13 pytest_runtest_logreport 112x in run runtest_n12->runtest_n13 runtest_n14 pytest_report_from_serializable firstresult · x4 runtest_n13->runtest_n14 runtest_n15 pytest_runtest_logreport 112x in run runtest_n14->runtest_n15 runtest_n16 pytest_runtest_logfinish 38x in run runtest_n15->runtest_n16 runtest_n17 244 further steps of 4 hooks runtest_n16->runtest_n17 runtest_n18 pytest_testnodedown 2x in run runtest_n17->runtest_n18 runtest_n19 pytest_runtest_logreport 112x in run runtest_n18->runtest_n19 runtest_n20 pytest_runtest_logfinish 38x in run runtest_n19->runtest_n20 runtest_n21 pytest_testnodedown 2x in run runtest_n20->runtest_n21 finish_n2 pytest_terminal_summary finish_n3 pytest_unconfigure finish_n2->finish_n3

Startup and configuration (controller)

Everything before collection begins: initial conftests are loaded, plugins register, and the session is configured.

Collection (controller)

Finding tests. Note that pytest_make_collect_report recurses - directories contain directories contain files.

The run-test protocol (controller)

pytest_runtestloop runs this once per collected test. Note the rhythm: setup, call and teardown are each followed by their own makereport and logreport pair - three reports per test, not one.

A dashed box stands in for a long stretch that only reorders the few hooks it names - under xdist, results arriving from workers in whatever order they happened to finish. Every hook is still listed below.

Session finish (controller)

Summary reporting and teardown of the session.

Every hook observed (controller)

Hook Calls Semantics Implemented by, in call order
pytest_addhooks 1 historic xdist.plugin (xdist)
pytest_addoption 2 historic xdist.plugin (xdist)
xdist.looponfail (xdist.looponfail)
pytest_cmdline_main 1 firstresult xdist.plugin (xdist) [tryfirst]
_pytest.setupplan [tryfirst]
_pytest.setuponly [tryfirst]
_pytest.mark [tryfirst]
xdist.looponfail (xdist.looponfail)
_pytest.python
_pytest.helpconfig
_pytest.main
pytest_cmdline_preparse 1 -
pytest_collection 1 firstresult _pytest.logging.LoggingPlugin (logging-plugin) [wrapper, tryfirst]
_pytest.warnings [wrapper, tryfirst]
_pytest.config.Config (pytestconfig) [wrapper]
xdist.dsession.DSession
_pytest.terminal.TerminalReporter
_pytest.assertion
_pytest.main
pytest_configure 1 historic _pytest.faulthandler
_pytest.warnings
_pytest.junitxml
_pytest.tmpdir
_pytest.legacypath
_pytest.skipping
_pytest.debugging
_pytest.terminal
_pytest.python
_pytest.mark
_pytest.config.PytestPluginManager (<anonymous>)
_pytest.pastebin [trylast]
_pytest.logging [trylast]
xdist.plugin (xdist) [trylast]
pytest_configure_node 2 -
pytest_load_initial_conftests 1 _pytest.warnings [wrapper]
_pytest.capture [wrapper]
_pytest.python_path [tryfirst]
_pytest.legacypath [tryfirst]
_pytest.config.Config (pytestconfig) [trylast]
pytest_plugin_registered 93 historic _pytest.legacypath
_pytest.terminal.TerminalReporter
_pytest.fixtures.FixtureManager (funcmanage)
pytest_report_from_serializable 112 firstresult _pytest.reports
pytest_report_teststatus 114 firstresult _pytest.skipping
_pytest.runner
_pytest.terminal [trylast]
pytest_runtest_logfinish 38 _pytest.logging.LoggingPlugin (logging-plugin)
_pytest.terminal.TerminalReporter
pytest_runtest_logreport 112 _pytest.main.Session [tryfirst]
_pytest.logging.LoggingPlugin (logging-plugin)
_pytest.terminal.TerminalReporter
pytest_runtest_logstart 38 _pytest.logging.LoggingPlugin (logging-plugin)
_pytest.terminal.TerminalReporter
pytest_runtestloop 1 firstresult _pytest.logging.LoggingPlugin (logging-plugin) [wrapper]
xdist.dsession.DSession
_pytest.main
pytest_sessionfinish 1 _pytest.logging.LoggingPlugin (logging-plugin) [wrapper, tryfirst]
_pytest.terminal.TerminalReporter [wrapper]
_pytest.warnings [wrapper]
xdist.dsession.DSession
_pytest.assertion
_pytest.tmpdir
_pytest.runner
pytest_sessionstart 1 _pytest.logging.LoggingPlugin (logging-plugin) [wrapper, tryfirst]
_pytest.fixtures
_pytest.runner
_pytest.terminal.TerminalReporter [trylast]
xdist.dsession.DSession [trylast]
pytest_terminal_summary 1 _pytest.terminal.TerminalReporter [wrapper]
_pytest.warnings [wrapper]
xdist.dsession.DSession
_pytest.pastebin
_pytest.runner
pytest_testnodedown 2 xdist.dsession.TerminalDistReporter
pytest_testnodeready 2 xdist.dsession.TerminalDistReporter
pytest_unconfigure 1 _pytest.logging.LoggingPlugin (logging-plugin)
_pytest.terminal.TerminalReporter
_pytest.faulthandler
_pytest.doctest
_pytest.junitxml
_pytest.pastebin
_pytest.mark
_pytest.python_path [trylast]
pytest_xdist_getremotemodule 2 firstresult xdist.workermanage.WorkerController.RemoteHook (<anonymous>) [trylast]
pytest_xdist_make_scheduler 1 firstresult xdist.dsession.DSession [trylast]
pytest_xdist_newgateway 2 xdist.dsession.TerminalDistReporter
pytest_xdist_node_collection_finished 2 -
pytest_xdist_setupnodes 1 xdist.dsession.TerminalDistReporter

Every worker

Both workers produced this flow. gw0 is drawn.

Command: none. xdist starts a worker over execnet and hands it a configuration, so it has no command line of its own.

Phases left to right, steps top to bottom. Colour says which phase a hook belongs to; how dark a step is says how often it was called. Click a column heading to jump to that phase in full detail, or any hook to open its pytest documentation. Why it looks like this

%3 cluster_phase_startup STARTUP AND CONFIGURATION cluster_startup_n1 pytest_load_initial_conftests cluster_startup_n3 pytest_configure historic cluster_startup_n5 pytest_sessionstart cluster_phase_collection COLLECTION cluster_collection_n1 pytest_collection firstresult cluster_phase_runtest THE RUN-TEST PROTOCOL cluster_runtest_n1 pytest_runtest_protocol firstresult · 19x in run cluster_phase_finish SESSION FINISH cluster_finish_n1 pytest_sessionfinish startup_n2 pytest_plugin_registered historic · 90x in run startup_n4 pytest_plugin_registered historic · x42 startup_n2->startup_n4 startup_n6 pytest_plugin_registered historic · x42 startup_n4->startup_n6 collection_n2 pytest_collectstart 7x in run collection_n3 pytest_make_collect_report firstresult · 7x in run collection_n2->collection_n3 collection_n4 pytest_collectreport 7x in run collection_n3->collection_n4 collection_n5 pytest_collectstart 7x in run collection_n4->collection_n5 collection_n6 pytest_make_collect_report firstresult · 7x in run collection_n5->collection_n6 collection_n7 pytest_collectstart 7x in run collection_n6->collection_n7 collection_n8 pytest_make_collect_report firstresult · 7x in run collection_n7->collection_n8 collection_n9 pytest_itemcollected x4 collection_n8->collection_n9 collection_n10 29 further steps of 4 hooks collection_n9->collection_n10 collection_n11 pytest_collection_modifyitems collection_n10->collection_n11 collection_n12 pytest_collection_finish collection_n11->collection_n12 runtest_n2 pytest_runtest_logstart 19x in run runtest_n3 pytest_runtest_setup 19x in run runtest_n2->runtest_n3 runtest_n4 pytest_runtest_makereport firstresult · 56x in run runtest_n3->runtest_n4 runtest_n5 pytest_runtest_logreport 56x in run runtest_n4->runtest_n5 runtest_n6 pytest_runtest_call 18x in run runtest_n5->runtest_n6 runtest_n7 pytest_runtest_makereport firstresult · 56x in run runtest_n6->runtest_n7 runtest_n8 pytest_runtest_logreport 56x in run runtest_n7->runtest_n8 runtest_n9 pytest_runtest_teardown 19x in run runtest_n8->runtest_n9 runtest_n10 pytest_runtest_makereport firstresult · 56x in run runtest_n9->runtest_n10 runtest_n11 pytest_runtest_logreport 56x in run runtest_n10->runtest_n11 runtest_n12 pytest_runtest_logfinish 19x in run runtest_n11->runtest_n12 finish_n2 pytest_terminal_summary finish_n3 pytest_unconfigure finish_n2->finish_n3

Startup and configuration (gw0)

Everything before collection begins: initial conftests are loaded, plugins register, and the session is configured.

Collection (gw0)

Finding tests. Note that pytest_make_collect_report recurses - directories contain directories contain files.

%3 cluster_n1 pytest_collection firstresult cluster_n3 pytest_make_collect_report firstresult · 7x in run cluster_n7 pytest_make_collect_report firstresult · 7x in run cluster_n13 pytest_collect_file 7x in run cluster_n16 pytest_collect_file 7x in run cluster_n19 pytest_collect_file 7x in run cluster_n22 pytest_collect_file 7x in run cluster_n25 pytest_collect_file 7x in run cluster_n28 pytest_make_collect_report firstresult · 7x in run cluster_n30 pytest_pycollect_makeitem firstresult · 26x in run cluster_n31 pytest_generate_tests 11x in run cluster_n33 pytest_pycollect_makeitem firstresult · 26x in run cluster_n38 pytest_collection_finish n2 pytest_collectstart 7x in run n4 pytest_collect_directory firstresult n2->n4 n5 pytest_collectreport 7x in run n4->n5 n6 pytest_collectstart 7x in run n5->n6 n8 pytest_ignore_collect firstresult · 7x in run n6->n8 n9 pytest_collect_file 7x in run n8->n9 n10 pytest_ignore_collect firstresult · 7x in run n9->n10 n11 pytest_collect_file 7x in run n10->n11 n12 pytest_ignore_collect firstresult · 7x in run n11->n12 n14 pytest_pycollect_makemodule firstresult · 5x in run n12->n14 n15 pytest_ignore_collect firstresult · 7x in run n14->n15 n17 pytest_pycollect_makemodule firstresult · 5x in run n15->n17 n18 pytest_ignore_collect firstresult · 7x in run n17->n18 n20 pytest_pycollect_makemodule firstresult · 5x in run n18->n20 n21 pytest_ignore_collect firstresult · 7x in run n20->n21 n23 pytest_pycollect_makemodule firstresult · 5x in run n21->n23 n24 pytest_ignore_collect firstresult · 7x in run n23->n24 n26 pytest_pycollect_makemodule firstresult · 5x in run n24->n26 n27 pytest_collectstart 7x in run n26->n27 n29 pytest_pycollect_makeitem firstresult · x3 n27->n29 n32 pytest_make_parametrize_id firstresult · x3 n29->n32 n34 pytest_generate_tests 11x in run n32->n34 n35 pytest_itemcollected x4 n34->n35 n36 29 further steps of 4 hooks pytest_collectreport pytest_collectstart pytest_itemcollected pytest_make_collect_report n35->n36 n37 pytest_collection_modifyitems n36->n37 n39 pytest_report_collectionfinish n37->n39

A dashed box stands in for a long stretch that only reorders the few hooks it names - under xdist, results arriving from workers in whatever order they happened to finish. Every hook is still listed below.

The run-test protocol (gw0)

pytest_runtestloop runs this once per collected test. Note the rhythm: setup, call and teardown are each followed by their own makereport and logreport pair - three reports per test, not one.

The path 15 of 19 took.

The others differed:

  • 1 test also called pytest_assertrepr_compare, pytest_exception_interact and skipped pytest_fixture_post_finalizer, pytest_fixture_setup
  • 1 test skipped pytest_fixture_post_finalizer, pytest_fixture_setup, pytest_pyfunc_call, pytest_runtest_call
  • 1 test skipped pytest_fixture_setup

Session finish (gw0)

Summary reporting and teardown of the session.

Every hook observed (gw0)

Hook Calls Semantics Implemented by, in call order
pytest_addhooks 1 historic xdist.plugin (xdist)
pytest_addoption 2 historic xdist.plugin (xdist)
xdist.looponfail (xdist.looponfail)
pytest_assertrepr_compare 1 _pytest.assertion
pytest_cmdline_main 1 firstresult xdist.plugin (xdist) [tryfirst]
_pytest.setupplan [tryfirst]
_pytest.setuponly [tryfirst]
_pytest.mark [tryfirst]
xdist.looponfail (xdist.looponfail)
_pytest.python
_pytest.helpconfig
_pytest.main
pytest_cmdline_preparse 1 -
pytest_collect_directory 1 firstresult _pytest.python
_pytest.main
pytest_collect_file 7 _pytest.doctest
_pytest.python
pytest_collection 1 firstresult _pytest.logging.LoggingPlugin (logging-plugin) [wrapper, tryfirst]
_pytest.warnings [wrapper, tryfirst]
_pytest.config.Config (pytestconfig) [wrapper]
_pytest.terminal.TerminalReporter
__channelexec__.WorkerInteractor (<anonymous>)
_pytest.assertion
_pytest.main
pytest_collection_finish 1 _pytest.terminal.TerminalReporter
__channelexec__.WorkerInteractor (<anonymous>)
pytest_collection_modifyitems 1 _pytest.fixtures.FixtureManager (funcmanage)
__channelexec__.WorkerInteractor (<anonymous>)
_pytest.main
_pytest.mark
pytest_collectreport 7 _pytest.main.Session.pytest_runtest_logreport (session) [tryfirst]
_pytest.terminal.TerminalReporter
__channelexec__.WorkerInteractor (<anonymous>)
pytest_collectstart 7 _pytest.main.Session [tryfirst]
pytest_configure 1 historic _pytest.faulthandler
_pytest.warnings
_pytest.junitxml
_pytest.tmpdir
_pytest.legacypath
_pytest.skipping
_pytest.debugging
_pytest.terminal
_pytest.python
_pytest.mark
_pytest.config.PytestPluginManager (<anonymous>)
_pytest.pastebin [trylast]
_pytest.logging [trylast]
xdist.plugin (xdist) [trylast]
pytest_exception_interact 1 _pytest.faulthandler [tryfirst]
pytest_fixture_post_finalizer 24 _pytest.setuponly
pytest_fixture_setup 17 firstresult _pytest.setuponly [wrapper]
_pytest.setupplan [tryfirst]
_pytest.fixtures
pytest_generate_tests 11 _pytest.fixtures.FixtureManager (funcmanage)
_pytest.python
pytest_ignore_collect 7 firstresult _pytest.main
pytest_itemcollected 19 -
pytest_load_initial_conftests 1 _pytest.warnings [wrapper]
_pytest.capture [wrapper]
_pytest.python_path [tryfirst]
_pytest.legacypath [tryfirst]
_pytest.config.Config (pytestconfig) [trylast]
pytest_make_collect_report 7 firstresult _pytest.capture.CaptureManager [wrapper]
_pytest.runner
pytest_make_parametrize_id 12 firstresult -
pytest_plugin_registered 90 historic _pytest.legacypath
_pytest.terminal.TerminalReporter
_pytest.fixtures.FixtureManager (funcmanage)
pytest_pycollect_makeitem 26 firstresult _pytest.unittest
_pytest.python [trylast]
pytest_pycollect_makemodule 5 firstresult _pytest.python
pytest_pyfunc_call 18 firstresult _pytest.python [trylast]
pytest_report_collectionfinish 1 -
pytest_report_teststatus 57 firstresult _pytest.skipping
_pytest.runner
_pytest.terminal [trylast]
pytest_report_to_serializable 56 firstresult _pytest.reports
pytest_runtest_call 18 _pytest.threadexception [wrapper, tryfirst]
_pytest.unraisableexception [wrapper, tryfirst]
_pytest.logging.LoggingPlugin (logging-plugin) [wrapper]
_pytest.capture.CaptureManager [wrapper]
_pytest.skipping [wrapper]
_pytest.runner
pytest_runtest_logfinish 19 _pytest.logging.LoggingPlugin (logging-plugin)
_pytest.terminal.TerminalReporter
__channelexec__.WorkerInteractor (<anonymous>)
pytest_runtest_logreport 56 _pytest.main.Session [tryfirst]
_pytest.logging.LoggingPlugin (logging-plugin)
_pytest.terminal.TerminalReporter
__channelexec__.WorkerInteractor (<anonymous>)
pytest_runtest_logstart 19 _pytest.logging.LoggingPlugin (logging-plugin)
_pytest.terminal.TerminalReporter
__channelexec__.WorkerInteractor (<anonymous>)
pytest_runtest_makereport 56 firstresult _pytest.tmpdir [wrapper, tryfirst]
_pytest.skipping [wrapper]
_pytest.unittest [tryfirst]
_pytest.runner
pytest_runtest_protocol 19 firstresult _pytest.warnings [wrapper, tryfirst]
_pytest.assertion [wrapper, tryfirst]
_pytest.unittest [wrapper]
_pytest.faulthandler [wrapper, trylast]
_pytest.runner
pytest_runtest_setup 19 _pytest.unraisableexception [wrapper, tryfirst]
_pytest.logging.LoggingPlugin (logging-plugin) [wrapper]
_pytest.capture.CaptureManager [wrapper]
_pytest.threadexception [wrapper, trylast]
_pytest.skipping [tryfirst]
_pytest.runner
_pytest.nose [trylast]
pytest_runtest_teardown 19 _pytest.threadexception [wrapper, tryfirst]
_pytest.unraisableexception [wrapper, tryfirst]
_pytest.logging.LoggingPlugin (logging-plugin) [wrapper]
_pytest.capture.CaptureManager [wrapper]
_pytest.runner
pytest_runtestloop 1 firstresult _pytest.logging.LoggingPlugin (logging-plugin) [wrapper]
__channelexec__.WorkerInteractor (<anonymous>)
_pytest.main
pytest_sessionfinish 1 _pytest.logging.LoggingPlugin (logging-plugin) [wrapper, tryfirst]
_pytest.terminal.TerminalReporter [wrapper]
__channelexec__.WorkerInteractor (<anonymous>) [hookwrapper]
_pytest.warnings [wrapper]
_pytest.assertion
_pytest.tmpdir
_pytest.runner
pytest_sessionstart 1 _pytest.logging.LoggingPlugin (logging-plugin) [wrapper, tryfirst]
__channelexec__.WorkerInteractor (<anonymous>)
_pytest.fixtures
_pytest.runner
_pytest.terminal.TerminalReporter [trylast]
pytest_terminal_summary 1 _pytest.terminal.TerminalReporter [wrapper]
_pytest.warnings [wrapper]
_pytest.pastebin
_pytest.runner
pytest_unconfigure 1 _pytest.logging.LoggingPlugin (logging-plugin)
_pytest.terminal.TerminalReporter
_pytest.faulthandler
_pytest.doctest
_pytest.junitxml
_pytest.pastebin
_pytest.mark
_pytest.python_path [trylast]