Quantcast

VTK, Python 3 and the six module - install error on six.pyc

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

VTK, Python 3 and the six module - install error on six.pyc

Marcus D. Hanwell-2
Hi,

I have been looking into this in the context of Tomviz, which uses
Python 3, but was able to reproduce this in a vanilla VTK build tree.
If you use Python 3 everything installs fine, but if you turn on
Module_SixPython then I get the following install error:

-- Installing: /usr/local/vtk/lib/cmake/vtk-7.1/Modules/SixPython.cmake
-- Installing: /usr/local/vtk/lib/python3.5/site-packages/six.py
CMake Error at ThirdParty/SixPython/cmake_install.cmake:40 (file):
 file INSTALL cannot find "/home/marcus/build/vtk/Wrapping/Python/six.pyc".
Call Stack (most recent call first):
 cmake_install.cmake:104 (include)

A quick grep turned up nothing obvious, but it would seem that other
Python modules are correctly finding the pyc files in the __pycache__
directories, but not so with six. Any pointers on where this logic
might be, and why it might be breaking down for this very simple
module.

Thanks,

Marcus
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: VTK, Python 3 and the six module - install error on six.pyc

Marcus D. Hanwell-2
On Mon, May 15, 2017 at 8:09 PM, Marcus D. Hanwell
<[hidden email]> wrote:

> I have been looking into this in the context of Tomviz, which uses
> Python 3, but was able to reproduce this in a vanilla VTK build tree.
> If you use Python 3 everything installs fine, but if you turn on
> Module_SixPython then I get the following install error:
>
> -- Installing: /usr/local/vtk/lib/cmake/vtk-7.1/Modules/SixPython.cmake
> -- Installing: /usr/local/vtk/lib/python3.5/site-packages/six.py
> CMake Error at ThirdParty/SixPython/cmake_install.cmake:40 (file):
>  file INSTALL cannot find "/home/marcus/build/vtk/Wrapping/Python/six.pyc".
> Call Stack (most recent call first):
>  cmake_install.cmake:104 (include)
>
> A quick grep turned up nothing obvious, but it would seem that other
> Python modules are correctly finding the pyc files in the __pycache__
> directories, but not so with six. Any pointers on where this logic
> might be, and why it might be breaking down for this very simple
> module.
>
Isn't it always the way, but I guess the major issue is that only a
few modules use vtk_module_python_package, and that CMake function
seems to have no idea about __pycache__ in Python 3, and is only just
starting to be used there with AutobahnPython and SixPython. I am
guessing when they were added make install wasn't tested with Python
3.

Aron, it looks like you added the Autobahn to Python 3, any chance
that the install target could be fixed? It seems like the
vtk_module_python_package function needs to be extended to deal with
__pycache__ when using Python 3 but people more knowledgeable in
Python-fu may have other ideas.

Thanks,

Marcus
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: VTK, Python 3 and the six module - install error on six.pyc

Aron Helser

On Mon, May 15, 2017 at 8:22 PM, Marcus D. Hanwell <[hidden email]> wrote:
On Mon, May 15, 2017 at 8:09 PM, Marcus D. Hanwell
<[hidden email]> wrote:
> I have been looking into this in the context of Tomviz, which uses
> Python 3, but was able to reproduce this in a vanilla VTK build tree.
> If you use Python 3 everything installs fine, but if you turn on
> Module_SixPython then I get the following install error:
>
> -- Installing: /usr/local/vtk/lib/cmake/vtk-7.1/Modules/SixPython.cmake
> -- Installing: /usr/local/vtk/lib/python3.5/site-packages/six.py
> CMake Error at ThirdParty/SixPython/cmake_install.cmake:40 (file):
>  file INSTALL cannot find "/home/marcus/build/vtk/Wrapping/Python/six.pyc".
> Call Stack (most recent call first):
>  cmake_install.cmake:104 (include)
>
> A quick grep turned up nothing obvious, but it would seem that other
> Python modules are correctly finding the pyc files in the __pycache__
> directories, but not so with six. Any pointers on where this logic
> might be, and why it might be breaking down for this very simple
> module.
>
Isn't it always the way, but I guess the major issue is that only a
few modules use vtk_module_python_package, and that CMake function
seems to have no idea about __pycache__ in Python 3, and is only just
starting to be used there with AutobahnPython and SixPython. I am
guessing when they were added make install wasn't tested with Python
3.

Aron, it looks like you added the Autobahn to Python 3, any chance
that the install target could be fixed? It seems like the
vtk_module_python_package function needs to be extended to deal with
__pycache__ when using Python 3 but people more knowledgeable in
Python-fu may have other ideas.

Yes, I updated Autobahn to use the third-party update mechanism, and used
the cmake files that already existed. I never tested the 'make install' functionality
with Python 3. I would expect the ParaView superbuild to do that, but I don't 
really know. I've not touched the vtk cmake infrastructure yet - I'll go ask for help.
I don't see '__pycache__' handled anywhere in vtk/paraview .cmake files....


Thanks,

Marcus


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: VTK, Python 3 and the six module - install error on six.pyc

Ben Boeckel
On Tue, May 16, 2017 at 11:11:50 -0400, Aron Helser wrote:
> Yes, I updated Autobahn to use the third-party update mechanism, and used
> the cmake files that already existed. I never tested the 'make install'
> functionality
> with Python 3. I would expect the ParaView superbuild to do that, but I
> don't

ParaView does other things for its Python install rules, so I'm not
surprised that VTK on its own is a bit different. Maybe ParaView *only*
installs the `.py` files and after installation does the `.pyc`
generation?

> really know. I've not touched the vtk cmake infrastructure yet - I'll go
> ask for help.
> I don't see '__pycache__' handled anywhere in vtk/paraview .cmake files....

That's the problem ;) . Python3 now stores `.pyc`, `.pyo`, and other
files under a `__pycache__` directory rather than right beside the `.py`
files they are generated from.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: VTK, Python 3 and the six module - install error on six.pyc

Marcus D. Hanwell-2
On Tue, May 16, 2017 at 11:24 AM, Ben Boeckel <[hidden email]> wrote:

> On Tue, May 16, 2017 at 11:11:50 -0400, Aron Helser wrote:
>> Yes, I updated Autobahn to use the third-party update mechanism, and used
>> the cmake files that already existed. I never tested the 'make install'
>> functionality
>> with Python 3. I would expect the ParaView superbuild to do that, but I
>> don't
>
> ParaView does other things for its Python install rules, so I'm not
> surprised that VTK on its own is a bit different. Maybe ParaView *only*
> installs the `.py` files and after installation does the `.pyc`
> generation?

I originally discovered this issue when doing a make install from
ParaView with the web stuff turned on using Python 3. Perhaps this
combination is not tested?
>
>> really know. I've not touched the vtk cmake infrastructure yet - I'll go
>> ask for help.
>> I don't see '__pycache__' handled anywhere in vtk/paraview .cmake files....
>
> That's the problem ;) . Python3 now stores `.pyc`, `.pyo`, and other
> files under a `__pycache__` directory rather than right beside the `.py`
> files they are generated from.
>
Yes, and the third party appears to be missing this. I found it in
building an updated ParaView dependency for Tomviz where we need
ParaView built against Python 3 and using the web flags in order to do
the export to a web viewer. If this isn't tested it would be nice to
get a test added.

I am out on travel right now, but can supply the flags for VTK and/or
ParaView. It is a VTK issue that ParaView seems to inherit when
turning on these modules recently ported to Python 3 (aside from the
make install step).
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: VTK, Python 3 and the six module - install error on six.pyc

Ben Boeckel
On Tue, May 16, 2017 at 11:43:25 -0400, Marcus D. Hanwell wrote:
> I originally discovered this issue when doing a make install from
> ParaView with the web stuff turned on using Python 3. Perhaps this
> combination is not tested?

Ah, that is true. The superbuild is Python2-only right now.

> Yes, and the third party appears to be missing this. I found it in
> building an updated ParaView dependency for Tomviz where we need
> ParaView built against Python 3 and using the web flags in order to do
> the export to a web viewer. If this isn't tested it would be nice to
> get a test added.
>
> I am out on travel right now, but can supply the flags for VTK and/or
> ParaView. It is a VTK issue that ParaView seems to inherit when
> turning on these modules recently ported to Python 3 (aside from the
> make install step).

Aron and I had some off-list messages and have found the culprit
(vtk_module_python_module). I believe he is working on a fix.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: VTK, Python 3 and the six module - install error on six.pyc

Aron Helser
I had to add a file glob to get the .pyc files, since they now include the interpreter and python version in the filename.
Here's a working fix to vtk/CMake/vtkModuleMacrosPython.cmake. I'll submit a MR tomorrow.

       # add install rules.
       if (NOT _no_install AND NOT VTK_INSTALL_NO_RUNTIME)
-        install(FILES "${VTK_BUILD_PYTHON_MODULE_DIR}/${_name}"
-                      "${VTK_BUILD_PYTHON_MODULE_DIR}/${_name_we}.pyc"
-                      "${VTK_BUILD_PYTHON_MODULE_DIR}/${_name_we}.pyo"
-                DESTINATION "${VTK_INSTALL_PYTHON_MODULE_DIR}"
-                COMPONENT "Runtime")
+        if(VTK_PYTHON_VERSION VERSION_LESS 3)
+          install(FILES "${VTK_BUILD_PYTHON_MODULE_DIR}/${_name}"
+                        "${VTK_BUILD_PYTHON_MODULE_DIR}/${_name_we}.pyc"
+                        "${VTK_BUILD_PYTHON_MODULE_DIR}/${_name_we}.pyo"
+                  DESTINATION "${VTK_INSTALL_PYTHON_MODULE_DIR}"
+                  COMPONENT "Runtime")
+        else()
+          # python 3 uses a different directory for .pyc files, and .pyo files are gone.
+          install(FILES "${VTK_BUILD_PYTHON_MODULE_DIR}/${_name}"
+                  DESTINATION "${VTK_INSTALL_PYTHON_MODULE_DIR}"
+                  COMPONENT "Runtime")
+          file(GLOB file_matches "${VTK_BUILD_PYTHON_MODULE_DIR}/__pycache__/${_name_we}.*.pyc")
+          install(FILES ${file_matches}
+                  DESTINATION "${VTK_INSTALL_PYTHON_MODULE_DIR}/__pycache__"
+                  COMPONENT "Runtime")
+        endif()
       endif()

Aron

On Tue, May 16, 2017 at 1:58 PM, Ben Boeckel <[hidden email]> wrote:
On Tue, May 16, 2017 at 11:43:25 -0400, Marcus D. Hanwell wrote:
> I originally discovered this issue when doing a make install from
> ParaView with the web stuff turned on using Python 3. Perhaps this
> combination is not tested?

Ah, that is true. The superbuild is Python2-only right now.

> Yes, and the third party appears to be missing this. I found it in
> building an updated ParaView dependency for Tomviz where we need
> ParaView built against Python 3 and using the web flags in order to do
> the export to a web viewer. If this isn't tested it would be nice to
> get a test added.
>
> I am out on travel right now, but can supply the flags for VTK and/or
> ParaView. It is a VTK issue that ParaView seems to inherit when
> turning on these modules recently ported to Python 3 (aside from the
> make install step).

Aron and I had some off-list messages and have found the culprit
(vtk_module_python_module). I believe he is working on a fix.

--Ben


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: VTK, Python 3 and the six module - install error on six.pyc

Ben Boeckel
On Tue, May 16, 2017 at 22:15:28 -0400, Aron Helser wrote:
> I had to add a file glob to get the .pyc files, since they now include the
> interpreter and python version in the filename.

But we have the version of Python we're using, so we could at least
factor that bit out of the glob. I don't think we really support
anything other than cpython either…

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: VTK, Python 3 and the six module - install error on six.pyc

Marcus D. Hanwell-2
On Wed, May 17, 2017 at 11:04 AM, Ben Boeckel <[hidden email]> wrote:
> On Tue, May 16, 2017 at 22:15:28 -0400, Aron Helser wrote:
>> I had to add a file glob to get the .pyc files, since they now include the
>> interpreter and python version in the filename.
>
> But we have the version of Python we're using, so we could at least
> factor that bit out of the glob. I don't think we really support
> anything other than cpython either…
>
I am not sure I see the benefit of narrowing it down further, but I
would be happy to test a merge request out once ready. We can't build
Tomviz with ParaView/VTK as it stands right now, so I am eager to see
this regression resolved for installation.

Thanks,

Marcus
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Loading...