Python Wrapping fails in master build.

classic Classic list List threaded Threaded
37 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Python Wrapping fails in master build.

Andrew Maclean-3
This seems to be restricted to Windows and has been happening for a while.

When building VTK (Release or Debug) the builds often stop with errors like this:
------------------------
Wrapping\Python\CMakeFiles\vtkCommonCorePythonD.dir\vtkTypeUInt64ArrayPython.cxx.obj
[158/2588] Linking CXX shared library bin\vtkCommonCorePython37D-9.0.dll
   Creating library lib\vtkCommonCorePython37D-9.0.lib and object lib\vtkCommonCorePython37D-9.0.exp
[159/2587] Python Wrapping - generating vtkInitialValueProblemSolverPython.cxx
FAILED: Wrapping/Python/vtkInitialValueProblemSolverPython.cxx 
cmd.exe /C "cd /D 
------------------------ 
 Where "Wrapping/Python/vtkInitialValueProblemSolverPython.cxx" can be any other file name in the Wrapping/Python/ folder.
I'm using VS2017 and Ninja and have tried running Ninja with a single thread to no avail. 

I see this issue on two windows machines and the only solution is to delete the offending file manually, run the build again, and if a a fail happens again, delete that file and rebuild.

Any ideas?

Regards
   Andrew

--
___________________________________________
Andrew J. P. Maclean

___________________________________________

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

David Gobbi
I've never seen this with VS2010 or VS2015, but I can try VS2017 this weekend.  Seems strange that there isn't any additional error text.  I also use ninja for building on Windows.

 - David

On Thu, Sep 20, 2018 at 11:26 PM Andrew Maclean <[hidden email]> wrote:
This seems to be restricted to Windows and has been happening for a while.

When building VTK (Release or Debug) the builds often stop with errors like this:
------------------------
Wrapping\Python\CMakeFiles\vtkCommonCorePythonD.dir\vtkTypeUInt64ArrayPython.cxx.obj
[158/2588] Linking CXX shared library bin\vtkCommonCorePython37D-9.0.dll
   Creating library lib\vtkCommonCorePython37D-9.0.lib and object lib\vtkCommonCorePython37D-9.0.exp
[159/2587] Python Wrapping - generating vtkInitialValueProblemSolverPython.cxx
FAILED: Wrapping/Python/vtkInitialValueProblemSolverPython.cxx 
cmd.exe /C "cd /D 
------------------------ 
 Where "Wrapping/Python/vtkInitialValueProblemSolverPython.cxx" can be any other file name in the Wrapping/Python/ folder.
I'm using VS2017 and Ninja and have tried running Ninja with a single thread to no avail. 

I see this issue on two windows machines and the only solution is to delete the offending file manually, run the build again, and if a a fail happens again, delete that file and rebuild.

Any ideas?

Regards
   Andrew

--
___________________________________________
Andrew J. P. Maclean

___________________________________________

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

David Gobbi
Hi Andrew,

I ran a VS2017 build and didn't see any errors.  Can you send me your cmake cache file?

 - David

On Fri, Sep 21, 2018 at 6:53 AM David Gobbi <[hidden email]> wrote:
I've never seen this with VS2010 or VS2015, but I can try VS2017 this weekend.  Seems strange that there isn't any additional error text.  I also use ninja for building on Windows.

 - David

On Thu, Sep 20, 2018 at 11:26 PM Andrew Maclean <[hidden email]> wrote:
This seems to be restricted to Windows and has been happening for a while.

When building VTK (Release or Debug) the builds often stop with errors like this:
------------------------
Wrapping\Python\CMakeFiles\vtkCommonCorePythonD.dir\vtkTypeUInt64ArrayPython.cxx.obj
[158/2588] Linking CXX shared library bin\vtkCommonCorePython37D-9.0.dll
   Creating library lib\vtkCommonCorePython37D-9.0.lib and object lib\vtkCommonCorePython37D-9.0.exp
[159/2587] Python Wrapping - generating vtkInitialValueProblemSolverPython.cxx
FAILED: Wrapping/Python/vtkInitialValueProblemSolverPython.cxx 
cmd.exe /C "cd /D 
------------------------ 
 Where "Wrapping/Python/vtkInitialValueProblemSolverPython.cxx" can be any other file name in the Wrapping/Python/ folder.
I'm using VS2017 and Ninja and have tried running Ninja with a single thread to no avail. 

I see this issue on two windows machines and the only solution is to delete the offending file manually, run the build again, and if a a fail happens again, delete that file and rebuild.

Any ideas?

Regards
   Andrew

--
___________________________________________
Andrew J. P. Maclean

___________________________________________

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

Andrew Maclean-3
David,
  Just ran a build now. Both debug and release builds failed on both machines at:
FAILED: Wrapping/Python/vtkInteractorStyle3DPython.cxx
If I delete this file and rebuild then it will build OK, however wrapping may fail on another file.

The cmake cache for the debug build is attached along with the logs for the release and debug builds(VTK-*.zip). Also attached is the checkout log for VTK (VTK.zip).

Regards
   Andrew

On Sat, Sep 22, 2018 at 10:27 PM David Gobbi <[hidden email]> wrote:
Hi Andrew,

I ran a VS2017 build and didn't see any errors.  Can you send me your cmake cache file?

 - David

On Fri, Sep 21, 2018 at 6:53 AM David Gobbi <[hidden email]> wrote:
I've never seen this with VS2010 or VS2015, but I can try VS2017 this weekend.  Seems strange that there isn't any additional error text.  I also use ninja for building on Windows.

 - David

On Thu, Sep 20, 2018 at 11:26 PM Andrew Maclean <[hidden email]> wrote:
This seems to be restricted to Windows and has been happening for a while.

When building VTK (Release or Debug) the builds often stop with errors like this:
------------------------
Wrapping\Python\CMakeFiles\vtkCommonCorePythonD.dir\vtkTypeUInt64ArrayPython.cxx.obj
[158/2588] Linking CXX shared library bin\vtkCommonCorePython37D-9.0.dll
   Creating library lib\vtkCommonCorePython37D-9.0.lib and object lib\vtkCommonCorePython37D-9.0.exp
[159/2587] Python Wrapping - generating vtkInitialValueProblemSolverPython.cxx
FAILED: Wrapping/Python/vtkInitialValueProblemSolverPython.cxx 
cmd.exe /C "cd /D 
------------------------ 
 Where "Wrapping/Python/vtkInitialValueProblemSolverPython.cxx" can be any other file name in the Wrapping/Python/ folder.
I'm using VS2017 and Ninja and have tried running Ninja with a single thread to no avail. 

I see this issue on two windows machines and the only solution is to delete the offending file manually, run the build again, and if a a fail happens again, delete that file and rebuild.

Any ideas?

Regards
   Andrew

--
___________________________________________
Andrew J. P. Maclean

___________________________________________


--
___________________________________________
Andrew J. P. Maclean

___________________________________________

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers


CMakeCache.zip (23K) Download Attachment
VTK-Debug.zip (1K) Download Attachment
VTK-Release.zip (1K) Download Attachment
VTK.zip (700 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

David Gobbi
You're using Python 3.7?  There was a regression in the Python 3.7.0 release that causes the 'vtkpython' executable to fail:
I wasn't planning to move to Python 3.7 until Python 3.7.1 is released in a couple weeks.  Are you able to run the vtkpython.exe that you build?

The build problem you're reporting is probably unrelated to the use of 3.7, though. I'm just grasping at straws because I still haven't been able to reproduce your issue on my own system, and there really aren't any good clues as to what is going wrong.  My only guess is that something is causing vtkWrapPython.exe to return with a non-zero exit code, but without printing an error.  Alternatively, Windows is silently killing the vtkWrapPython.exe process for some reason.

I'm hoping that we'll hear from VS2017 users regarding whether they've succeeded/failed to build the Python wrappers.

 - David

On Sat, Sep 22, 2018 at 3:03 PM Andrew Maclean <[hidden email]> wrote:
David,
  Just ran a build now. Both debug and release builds failed on both machines at:
FAILED: Wrapping/Python/vtkInteractorStyle3DPython.cxx
If I delete this file and rebuild then it will build OK, however wrapping may fail on another file.

The cmake cache for the debug build is attached along with the logs for the release and debug builds(VTK-*.zip). Also attached is the checkout log for VTK (VTK.zip).

Regards
   Andrew

On Sat, Sep 22, 2018 at 10:27 PM David Gobbi <[hidden email]> wrote:
Hi Andrew,

I ran a VS2017 build and didn't see any errors.  Can you send me your cmake cache file?

 - David

On Fri, Sep 21, 2018 at 6:53 AM David Gobbi <[hidden email]> wrote:
I've never seen this with VS2010 or VS2015, but I can try VS2017 this weekend.  Seems strange that there isn't any additional error text.  I also use ninja for building on Windows.

 - David

On Thu, Sep 20, 2018 at 11:26 PM Andrew Maclean <[hidden email]> wrote:
This seems to be restricted to Windows and has been happening for a while.

When building VTK (Release or Debug) the builds often stop with errors like this:
------------------------
Wrapping\Python\CMakeFiles\vtkCommonCorePythonD.dir\vtkTypeUInt64ArrayPython.cxx.obj
[158/2588] Linking CXX shared library bin\vtkCommonCorePython37D-9.0.dll
   Creating library lib\vtkCommonCorePython37D-9.0.lib and object lib\vtkCommonCorePython37D-9.0.exp
[159/2587] Python Wrapping - generating vtkInitialValueProblemSolverPython.cxx
FAILED: Wrapping/Python/vtkInitialValueProblemSolverPython.cxx 
cmd.exe /C "cd /D 
------------------------ 
 Where "Wrapping/Python/vtkInitialValueProblemSolverPython.cxx" can be any other file name in the Wrapping/Python/ folder.
I'm using VS2017 and Ninja and have tried running Ninja with a single thread to no avail. 

I see this issue on two windows machines and the only solution is to delete the offending file manually, run the build again, and if a a fail happens again, delete that file and rebuild.

Any ideas?

Regards
   Andrew

--
___________________________________________
Andrew J. P. Maclean

___________________________________________


--
___________________________________________
Andrew J. P. Maclean

___________________________________________

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

Andrew Maclean-3
David,

   Tanks once again for looking at this.

Yes I'm using Python 3.7 and vtkpython doesn't work. I have just recently upgraded.
However all the vtk python code runs Ok using PyCharm. So there is definitely nothing wrong with the wrapping, just with vtkpython. Thanks for the heads up here, I thought it was something I was doing! I'll switch to Python 3.7.1 as soon as it becomes available.

You may be able to replicate the wrapping error in a day or so when there are more commits to the VTK repository e.g when something changes in Rendering or Common. I was seeing the wrapping issue in Python 3.6.5/6 so it is unrelated to the Python version.

I have two approaches which work:

A. Just delete the cxx file that caused the error, and rebuild using ninja. You may or may not get a build failure on a different file again. It that's the case just repeat the process. This is the fastest solution.

B. Delete all the *.cxx files from Wrapping/Python/ and run CMake/ninja again. This rebuilds all the relevant python libs and dlls. This is much slower as some 5100 items have to be rebuilt.

I suspect it is never seen on the testing machines because I think everything is deleted and then rebuilt.

Regards
   Andrew


On Sun, Sep 23, 2018 at 7:38 AM David Gobbi <[hidden email]> wrote:
You're using Python 3.7?  There was a regression in the Python 3.7.0 release that causes the 'vtkpython' executable to fail:
I wasn't planning to move to Python 3.7 until Python 3.7.1 is released in a couple weeks.  Are you able to run the vtkpython.exe that you build?

The build problem you're reporting is probably unrelated to the use of 3.7, though. I'm just grasping at straws because I still haven't been able to reproduce your issue on my own system, and there really aren't any good clues as to what is going wrong.  My only guess is that something is causing vtkWrapPython.exe to return with a non-zero exit code, but without printing an error.  Alternatively, Windows is silently killing the vtkWrapPython.exe process for some reason.

I'm hoping that we'll hear from VS2017 users regarding whether they've succeeded/failed to build the Python wrappers.

 - David

On Sat, Sep 22, 2018 at 3:03 PM Andrew Maclean <[hidden email]> wrote:
David,
  Just ran a build now. Both debug and release builds failed on both machines at:
FAILED: Wrapping/Python/vtkInteractorStyle3DPython.cxx
If I delete this file and rebuild then it will build OK, however wrapping may fail on another file.

The cmake cache for the debug build is attached along with the logs for the release and debug builds(VTK-*.zip). Also attached is the checkout log for VTK (VTK.zip).

Regards
   Andrew

On Sat, Sep 22, 2018 at 10:27 PM David Gobbi <[hidden email]> wrote:
Hi Andrew,

I ran a VS2017 build and didn't see any errors.  Can you send me your cmake cache file?

 - David

On Fri, Sep 21, 2018 at 6:53 AM David Gobbi <[hidden email]> wrote:
I've never seen this with VS2010 or VS2015, but I can try VS2017 this weekend.  Seems strange that there isn't any additional error text.  I also use ninja for building on Windows.

 - David

On Thu, Sep 20, 2018 at 11:26 PM Andrew Maclean <[hidden email]> wrote:
This seems to be restricted to Windows and has been happening for a while.

When building VTK (Release or Debug) the builds often stop with errors like this:
------------------------
Wrapping\Python\CMakeFiles\vtkCommonCorePythonD.dir\vtkTypeUInt64ArrayPython.cxx.obj
[158/2588] Linking CXX shared library bin\vtkCommonCorePython37D-9.0.dll
   Creating library lib\vtkCommonCorePython37D-9.0.lib and object lib\vtkCommonCorePython37D-9.0.exp
[159/2587] Python Wrapping - generating vtkInitialValueProblemSolverPython.cxx
FAILED: Wrapping/Python/vtkInitialValueProblemSolverPython.cxx 
cmd.exe /C "cd /D 
------------------------ 
 Where "Wrapping/Python/vtkInitialValueProblemSolverPython.cxx" can be any other file name in the Wrapping/Python/ folder.
I'm using VS2017 and Ninja and have tried running Ninja with a single thread to no avail. 

I see this issue on two windows machines and the only solution is to delete the offending file manually, run the build again, and if a a fail happens again, delete that file and rebuild.

Any ideas?

Regards
   Andrew

--
___________________________________________
Andrew J. P. Maclean

___________________________________________


--
___________________________________________
Andrew J. P. Maclean

___________________________________________


--
___________________________________________
Andrew J. P. Maclean

___________________________________________

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

Ben Boeckel
On Sun, Sep 23, 2018 at 08:53:44 +1000, Andrew Maclean wrote:
> I suspect it is never seen on the testing machines because I think
> everything is deleted and then rebuilt.

Buildbots do incremental builds for MR builds, so they could be
potential repoducers.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

David Gobbi
Hi Andrew,

I've merged a patch that has a minuscule chance of fixing your wrapper issue:
This is a patch to vtkWrapPython that closes the output *Python.cxx file prior to exit (of course, the Windows loader should close all open files on exit automatically, which isn't an excuse for the missing "fclose()" in the code, but it does suggest that this patch will have no net effect).  The patch also fixes an instance where an error could occur without an error message being printed.


With regards to incremental builds, there are deficiencies in the dependency tracking for wrappers, and even though I don't think they are related to the build problems that you are seeing, they are worth mentioning:

When vtkWrapPython reads a header for wrapping, it must also handle all the #include statements so that it can look for macros and definitions much like a C++ compiler would.  In other words, each file is #included should be a dependency, but most build systems can only handle such dependencies for rules that call the C++ compiler itself, and not for custom commands like vtkWrapPython.  

There actually are two backend-specific ways to handle such dependencies in cmake:
(1) For the "Makefile" generator, cmake can generate the dependencies on the first build but the dependency list becomes out-of-date on subsequent builds.
(2) For the "Ninja" generator, and only for very recent versions of cmake, it would be possible to add a customized dependency-generation rule, but I haven't looked into it in much detail (it would be a lot of work to implement, and wouldn't work on any other backends).

However, the kinds of incremental build changes that cause wrapper builds to fail are rare, and the build errors that you saw are not indicative of a dependency problem.

 - David


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

Andrew Maclean-3
Awesome, I'll try it out now and over the next few days and let you know how it goes.

Andrew

On Tue, Sep 25, 2018 at 6:41 AM David Gobbi <[hidden email]> wrote:
Hi Andrew,

I've merged a patch that has a minuscule chance of fixing your wrapper issue:
This is a patch to vtkWrapPython that closes the output *Python.cxx file prior to exit (of course, the Windows loader should close all open files on exit automatically, which isn't an excuse for the missing "fclose()" in the code, but it does suggest that this patch will have no net effect).  The patch also fixes an instance where an error could occur without an error message being printed.


With regards to incremental builds, there are deficiencies in the dependency tracking for wrappers, and even though I don't think they are related to the build problems that you are seeing, they are worth mentioning:

When vtkWrapPython reads a header for wrapping, it must also handle all the #include statements so that it can look for macros and definitions much like a C++ compiler would.  In other words, each file is #included should be a dependency, but most build systems can only handle such dependencies for rules that call the C++ compiler itself, and not for custom commands like vtkWrapPython.  

There actually are two backend-specific ways to handle such dependencies in cmake:
(1) For the "Makefile" generator, cmake can generate the dependencies on the first build but the dependency list becomes out-of-date on subsequent builds.
(2) For the "Ninja" generator, and only for very recent versions of cmake, it would be possible to add a customized dependency-generation rule, but I haven't looked into it in much detail (it would be a lot of work to implement, and wouldn't work on any other backends).

However, the kinds of incremental build changes that cause wrapper builds to fail are rare, and the build errors that you saw are not indicative of a dependency problem.

 - David



--
___________________________________________
Andrew J. P. Maclean

___________________________________________

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

Andrew Maclean-3
Had one wrapping failure in the Debug build:
----------
[398/4943] Python Wrapping - generating vtkMatrix3x3Python.cxx
FAILED: Wrapping/Python/vtkMatrix3x3Python.cxx 
cmd.exe /C "cd /D D:\Users\amaclean\Development\Kitware\build\VTK-Debug\Wrapping\Python && D:\Users\amaclean\Development\Kitware\build\VTK-Debug\bin\vtkWrapPython-9.0.exe @D:/Users/amaclean/Development/Kitware/build/VTK-Debug/Wrapping/Python/vtkCommonMathPython.Debug.args -o D:/Users/amaclean/Development/Kitware/build/VTK-Debug/Wrapping/Python/vtkMatrix3x3Python.cxx D:/Users/amaclean/Development/Kitware/src/VTK/Common/Math/vtkMatrix3x3.h"
Error opening output file D:/Users/amaclean/Development/Kitware/build/VTK-Debug/Wrapping/Python/vtkMatrix3x3Python.cxx
----------
and one in the  Release build:
----------
FAILED: Wrapping/Python/vtkFunctionSetPython.cxx 
cmd.exe /C "cd /D D:\Users\amaclean\Development\Kitware\build\VTK-Release\Wrapping\Python && D:\Users\amaclean\Development\Kitware\build\VTK-Release\bin\vtkWrapPython-9.0.exe @D:/Users/amaclean/Development/Kitware/build/VTK-Release/Wrapping/Python/vtkCommonMathPython.Release.args -o D:/Users/amaclean/Development/Kitware/build/VTK-Release/Wrapping/Python/vtkFunctionSetPython.cxx D:/Users/amaclean/Development/Kitware/src/VTK/Common/Math/vtkFunctionSet.h"
Error opening output file D:/Users/amaclean/Development/Kitware/build/VTK-Release/Wrapping/Python/vtkFunctionSetPython.cxx
----------

A subsequent of ninja fixed these failures.

Let's see what happens over the next few days.

Andrew



On Tue, Sep 25, 2018 at 8:18 AM Andrew Maclean <[hidden email]> wrote:
Awesome, I'll try it out now and over the next few days and let you know how it goes.

Andrew

On Tue, Sep 25, 2018 at 6:41 AM David Gobbi <[hidden email]> wrote:
Hi Andrew,

I've merged a patch that has a minuscule chance of fixing your wrapper issue:
This is a patch to vtkWrapPython that closes the output *Python.cxx file prior to exit (of course, the Windows loader should close all open files on exit automatically, which isn't an excuse for the missing "fclose()" in the code, but it does suggest that this patch will have no net effect).  The patch also fixes an instance where an error could occur without an error message being printed.


With regards to incremental builds, there are deficiencies in the dependency tracking for wrappers, and even though I don't think they are related to the build problems that you are seeing, they are worth mentioning:

When vtkWrapPython reads a header for wrapping, it must also handle all the #include statements so that it can look for macros and definitions much like a C++ compiler would.  In other words, each file is #included should be a dependency, but most build systems can only handle such dependencies for rules that call the C++ compiler itself, and not for custom commands like vtkWrapPython.  

There actually are two backend-specific ways to handle such dependencies in cmake:
(1) For the "Makefile" generator, cmake can generate the dependencies on the first build but the dependency list becomes out-of-date on subsequent builds.
(2) For the "Ninja" generator, and only for very recent versions of cmake, it would be possible to add a customized dependency-generation rule, but I haven't looked into it in much detail (it would be a lot of work to implement, and wouldn't work on any other backends).

However, the kinds of incremental build changes that cause wrapper builds to fail are rare, and the build errors that you saw are not indicative of a dependency problem.

 - David



--
___________________________________________
Andrew J. P. Maclean

___________________________________________


--
___________________________________________
Andrew J. P. Maclean

___________________________________________

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

Ben Boeckel
On Tue, Sep 25, 2018 at 09:05:01 +1000, Andrew Maclean wrote:

> Had one wrapping failure in the Debug build:
> ----------
> [398/4943] Python Wrapping - generating vtkMatrix3x3Python.cxx
> FAILED: Wrapping/Python/vtkMatrix3x3Python.cxx
> cmd.exe /C "cd /D
> D:\Users\amaclean\Development\Kitware\build\VTK-Debug\Wrapping\Python &&
> D:\Users\amaclean\Development\Kitware\build\VTK-Debug\bin\vtkWrapPython-9.0.exe
> @D:/Users/amaclean/Development/Kitware/build/VTK-Debug/Wrapping/Python/vtkCommonMathPython.Debug.args
> -o
> D:/Users/amaclean/Development/Kitware/build/VTK-Debug/Wrapping/Python/vtkMatrix3x3Python.cxx
> D:/Users/amaclean/Development/Kitware/src/VTK/Common/Math/vtkMatrix3x3.h"
> Error opening output file
> D:/Users/amaclean/Development/Kitware/build/VTK-Debug/Wrapping/Python/vtkMatrix3x3Python.cxx
> ----------
> and one in the  Release build:
> ----------
> FAILED: Wrapping/Python/vtkFunctionSetPython.cxx
> cmd.exe /C "cd /D
> D:\Users\amaclean\Development\Kitware\build\VTK-Release\Wrapping\Python &&
> D:\Users\amaclean\Development\Kitware\build\VTK-Release\bin\vtkWrapPython-9.0.exe
> @D:/Users/amaclean/Development/Kitware/build/VTK-Release/Wrapping/Python/vtkCommonMathPython.Release.args
> -o
> D:/Users/amaclean/Development/Kitware/build/VTK-Release/Wrapping/Python/vtkFunctionSetPython.cxx
> D:/Users/amaclean/Development/Kitware/src/VTK/Common/Math/vtkFunctionSet.h"
> Error opening output file
> D:/Users/amaclean/Development/Kitware/build/VTK-Release/Wrapping/Python/vtkFunctionSetPython.cxx
> ----------
>
> A subsequent of ninja fixed these failures.

I wonder if the file isn't being flushed before it exits? Could that
cause an issue?

It also seems to have an issue *opening* the output file. Is it
possible to extract the error message somehow for when this occurs?
Maybe an AV is getting in our way?

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

David Gobbi
On Tue, Sep 25, 2018 at 6:18 AM Ben Boeckel <[hidden email]> wrote:
I wonder if the file isn't being flushed before it exits? Could that
cause an issue?

That was what I was thinking, too.  If so, the newly-added fclose()
might fix the issue, since fclose() flushes the file.
 
It also seems to have an issue *opening* the output file. Is it
possible to extract the error message somehow for when this occurs?

We could patch vtkWrapPython.c to print the numerical value of errno
on failure, and messages for the most likely values (EACCES, ENOENT). 

Maybe an AV is getting in our way?

An AV?


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

David Gobbi
In reply to this post by Ben Boeckel
On Tue, Sep 25, 2018 at 6:18 AM Ben Boeckel <[hidden email]> wrote:

Maybe an AV is getting in our way?

Oh, you man Anti-Virus software, i.e. maybe Windows Defender is poking the files produced by vtkWrapPython to make sure they're safe.  Andrew, if the problem persists, can you turn off any anti-virus software during the build to see if that helps?

 - David


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

Ben Boeckel
In reply to this post by David Gobbi
On Tue, Sep 25, 2018 at 07:07:01 -0600, David Gobbi wrote:
> We could patch vtkWrapPython.c to print the numerical value of errno
> on failure, and messages for the most likely values (EACCES, ENOENT).

`perror` could be used to print the error message (there's a GNU
extension to get the string directly, but since we need `perror` on
Windows anyways, not much use in using it.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

Andrew Maclean-3
I suspect it is the Ant-Virus Software. I just ran builds on two machines with McAfee scanning turned off and there were no problems.

On Wed, Sep 26, 2018 at 6:21 AM Ben Boeckel <[hidden email]> wrote:
On Tue, Sep 25, 2018 at 07:07:01 -0600, David Gobbi wrote:
> We could patch vtkWrapPython.c to print the numerical value of errno
> on failure, and messages for the most likely values (EACCES, ENOENT).

`perror` could be used to print the error message (there's a GNU
extension to get the string directly, but since we need `perror` on
Windows anyways, not much use in using it.

--Ben


--
___________________________________________
Andrew J. P. Maclean

___________________________________________

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

David Gobbi
On Tue, Sep 25, 2018 at 4:03 PM Andrew Maclean <[hidden email]> wrote:
I suspect it is the Ant-Virus Software. I just ran builds on two machines with McAfee scanning turned off and there were no problems.

You can probably tell McAfee to ignore certain directories.  But before you do that, I'd still like to see what the error number is, just in case it is reasonable for vtkWrapPython to retry if it's unable to open its output file, but I don't think that I can add an error-reporting patch tonight.

 - David

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

Andrew Maclean-3
That sounds like a good idea. It does seem odd that it can't open the output file and the fact that it is not reproducible wrt it being the same file.


Andrew Maclean


On 26 Sep 2018 08:55, "David Gobbi" <[hidden email]> wrote:
On Tue, Sep 25, 2018 at 4:03 PM Andrew Maclean <[hidden email]> wrote:
I suspect it is the Ant-Virus Software. I just ran builds on two machines with McAfee scanning turned off and there were no problems.

You can probably tell McAfee to ignore certain directories.  But before you do that, I'd still like to see what the error number is, just in case it is reasonable for vtkWrapPython to retry if it's unable to open its output file, but I don't think that I can add an error-reporting patch tonight.

 - David


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

David Gobbi
On Tue, 25 Sep 2018, 19:13 Andrew Maclean, <[hidden email]> wrote:
That sounds like a good idea. It does seem odd that it can't open the output file and the fact that it is not reproducible wrt it being the same file.

Doesn't seem odd, though, if McAffee is marching through the files and temporarily locking each one to scan it.  The collisions between vtkWrapPython and McAffee would be expected to be fairly random in this scenario.

 - David

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

Andras Lasso

It is probably worth asking CMake developers about what could be a good approach to address this.

 

A couple of years ago we had similar problems in CMake compile-tests randomly failing (https://cmake.org/Bug/view.php?id=12957).

 

Retrying a file operation can have many non-trivial side effects so it should be avoided. As far as I know, there is no such workaround anywhere in CMake, so it should be possible to avoid it in vtkWrapPython, too.

 

Andras

 

From: vtk-developers <[hidden email]> On Behalf Of David Gobbi
Sent: Tuesday, September 25, 2018 9:26 PM
To: Andrew Maclean <[hidden email]>
Cc: VTK Developers <[hidden email]>
Subject: Re: [vtk-developers] Python Wrapping fails in master build.

 

On Tue, 25 Sep 2018, 19:13 Andrew Maclean, <[hidden email]> wrote:

That sounds like a good idea. It does seem odd that it can't open the output file and the fact that it is not reproducible wrt it being the same file.

 

Doesn't seem odd, though, if McAffee is marching through the files and temporarily locking each one to scan it.  The collisions between vtkWrapPython and McAffee would be expected to be fairly random in this scenario.

 

 - David


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Python Wrapping fails in master build.

David Gobbi
Hi Andras, it's good to hear that someone else has seen problems similar to what Andrew is seeing.

On Tue, Sep 25, 2018 at 7:37 PM Andras Lasso <[hidden email]> wrote:

It is probably worth asking CMake developers about what could be a good approach to address this.


Brad, have there been any new discoveries about why compiler tests and custom-command outputs fail to open their output files on Windows as described in this cmake issue? (https://cmake.org/Bug/view.php?id=12957).  This is something that Andrew is seeing on his machine, that I haven't been able to reproduce on my own machine yet.

Retrying a file operation can have many non-trivial side effects so it should be avoided. As far as I know, there is no such workaround anywhere in CMake, so it should be possible to avoid it in vtkWrapPython, too.


This isn't always true.  On Linux when fread fails with errno=EINTR, retrying is the correct response, and there are a few places where this is done in the CMake code base.  The errno=EINTR condition is very short lived.  It would be nice if the failure on Windows was something similar (i.e. instantly retryable) but I suspect it isn't.  I strongly suspect the failure is a sharing violation, for which the correct response is usually to write to a different file (which is not ideal).

  David

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

12