Periodic Executors

PyMongo implements a PeriodicExecutor for two purposes: as the background thread for Monitor, and to regularly check if there are OP_KILL_CURSORS messages that must be sent to the server.

Monitoring

For each server in the topology, Topology launches a monitor thread. This thread must not prevent the topology from being freed, so it weakrefs the topology. Furthermore, it uses a weakref callback to close itself promptly when the topology is freed.

Solid lines represent strong references, dashed lines weak ones:

../_images/periodic-executor-refs.png

See Stopping Executors below for an explanation of _EXECUTORS.

Killing Cursors

An incompletely iterated Cursor on the client represents an open cursor object on the server. In code like this, we lose a reference to the cursor before finishing iteration:

for doc in collection.find():
    raise Exception()

We try to send an OP_KILL_CURSORS to the server to tell it to clean up the server-side cursor. But we must not take any locks directly from the cursor’s destructor (see PYTHON-799), so we cannot safely use the PyMongo data structures required to send a message. The solution is to add the cursor’s id to an array on the MongoClient without taking any locks.

Each client has a PeriodicExecutor devoted to checking the array for cursor ids. Any it sees are the result of cursors that were freed while the server-side cursor was still open. The executor can safely take the locks it needs in order to send the OP_KILL_CURSORS message.

Stopping Executors

Just as Cursor must not take any locks from its destructor, neither can MongoClient and Topology. Thus, although the client calls close() on its kill-cursors thread, and the topology calls close() on all its monitor threads, the close() method cannot actually call wake() on the executor, since wake() takes a lock.

Instead, executors wake very frequently to check if self.close is set, and if so they exit.

A thread can log spurious errors if it wakes late in the Python interpreter’s shutdown sequence, so we try to join threads before then. Each periodic executor (either a monitor or a kill-cursors thread) adds a weakref to itself to a set called _EXECUTORS, in the periodic_executor module.

An exit handler runs on shutdown and tells all executors to stop, then tries (with a short timeout) to join all executor threads.