Python 3.7 behavior change, GIL, and threading in C/C++ with Python

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Python 3.7 behavior change, GIL, and threading in C/C++ with Python

Marcus D. Hanwell-2
Hi,

I have seen several fixes for Python 3.7, but have been tracking down why Tomviz deadlocks on when using vtkPythonScopeGilEnsurer, and it appears to be related to the Python specific flags we are using and a behavior change in Python 3.7. This may mainly be a question for David Gobbi and/or Utkarsh. but I send it to the list so that others have a chance to see it, and it gets indexed should others hit this issue.

We use VTK_PYTHON_FULL_THREADSAFE set to ON, and VTK_NO_PYTHON_THREADS set to OFF. This means that when vtkPythonInterpreter::Initialize is called that the following code will be called:

#ifdef VTK_PYTHON_FULL_THREADSAFE
    int threadInit = PyEval_ThreadsInitialized();
    PyEval_InitThreads(); // safe to call this multiple time
    if (!threadInit)
    {
      PyEval_SaveThread(); // release GIL
    }
#endif

This was effectively always calling PyEval_SaveThread() before Python 3.7. but this is now never being called as starting with Python 3.7 Py_Initialize will initialize the GIL, see https://docs.python.org/3/c-api/init.html#c.PyEval_InitThreads for details.

It looks like this check was done to ensure that PyEval_SaveThread would only be called once, now that PyEval_InitThreads doesn't need to be called, moving PyEval_SaveThread outside of the conditional gets rid of the deadlock for me. It looks like something needs to be changed for 3.7+, but I am not clear on how necessary this logic is.

This reminds me that VTK_PYTHON_FULL_THREADSAFE is used in VTK, but the CMake option is in ParaView only as far as I can see. Should that be moved down into VTK, it seems odd.

Thanks,

Marcus

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python 3.7 behavior change, GIL, and threading in C/C++ with Python

Sebastien Jourdain via vtk-developers
On Mon, Nov 26, 2018 at 11:59 AM Marcus D. Hanwell <[hidden email]> wrote:
Hi,

I have seen several fixes for Python 3.7, but have been tracking down why Tomviz deadlocks on when using vtkPythonScopeGilEnsurer, and it appears to be related to the Python specific flags we are using and a behavior change in Python 3.7. This may mainly be a question for David Gobbi and/or Utkarsh. but I send it to the list so that others have a chance to see it, and it gets indexed should others hit this issue.

We use VTK_PYTHON_FULL_THREADSAFE set to ON, and VTK_NO_PYTHON_THREADS set to OFF. This means that when vtkPythonInterpreter::Initialize is called that the following code will be called:

#ifdef VTK_PYTHON_FULL_THREADSAFE
    int threadInit = PyEval_ThreadsInitialized();
    PyEval_InitThreads(); // safe to call this multiple time
    if (!threadInit)
    {
      PyEval_SaveThread(); // release GIL
    }
#endif

This was effectively always calling PyEval_SaveThread() before Python 3.7. but this is now never being called as starting with Python 3.7 Py_Initialize will initialize the GIL, see https://docs.python.org/3/c-api/init.html#c.PyEval_InitThreads for details.

It looks like this check was done to ensure that PyEval_SaveThread would only be called once, now that PyEval_InitThreads doesn't need to be called, moving PyEval_SaveThread outside of the conditional gets rid of the deadlock for me. It looks like something needs to be changed for 3.7+, but I am not clear on how necessary this logic is.

I submitted https://gitlab.kitware.com/vtk/vtk/merge_requests/4921 for review, it seems to check out in VTK, and a related ParaView branch moving VTK forward. 

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers