[ANNOUNCE] New module system landing

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

[ANNOUNCE] New module system landing

VTK - Dev mailing list
Hi all,

The new module system is in a landable state right now. We're planning
on merging it Friday (4 Jan 2019). The buildbots will be updated shortly
afterwards. Once they've been updated, `Do: test` will only work for
rebased merge requests (since the cache entries don't line up). In
addition, after the update, buildbots will default to building all
modules (turning off those which don't build).

The CMake API is now much more regular (some internal APIs have an
oddity or two we can clean up). Keyword arguments, erroring out on
unrecognized arguments, proper error messages for missing required
arguments, etc. All API functions also have a documentation block above
them in Markdown. In the future, this will be extracted and added to the
Doxygen output for VTK. In addition, no global variables[1] are used to
control the build; everything is passed in via arguments to the relevant
functions. There is one slight exception to this for APIs meant to be
called within another context. For example, `vtk_module_add_module` uses
the `LIBRARY_DESTINATION` variable passed to `vtk_module_build` for its
installation. The testing CMake APIs have a few similar expectations.

Previous discussion has occured on this thread on this list:

    https://markmail.org/thread/cu2wyqpvporqvps6

The merge request is here:

    https://gitlab.kitware.com/vtk/vtk/merge_requests/5020

ParaView's update for the new build setup will be landing in the
following days.

Changes for building VTK itself:

  - CMake 3.8 is now required. This is primarily to support
    `target_compile_features(cxx_std_11)` for having *consumers* of VTK
    also have (at least) C++11 enabled. Building with kits requires
    3.12 for object library features. Some modules require newer CMake
    versions for builtin CUDA language support (though they don't seem
    to be guarded by checks at the moment).
  - Module names have been changed. For example, `vtkCommonCore` is now
    `VTK::CommonCore`. This allows for one name to work both inside and
    outside VTK for `target_link_libraries`. Cache variable names use
    the module name.
  - Changed cache variables:
    * Module and group flags:
      - `Module_<MMM>` -> `VTK_MODULE_ENABLE_<MMM>`
      - `VTK_Group_<MMM>` -> `VTK_GROUP_ENABLE_<MMM>`
      - Values are no longer boolean, but instead 5-state:
        * `YES`: Must be built
        * `NO`: Must not be built (turns off modules requiring it;
          errors if required by a `YES` module)
        * `WANT`: Build if possible
        * `DONT_WANT`: Build only if necessary
        * `DEFAULT`: Use group (for modules) and `VTK_BUILD_ALL_MODULES`
          instead
      - `VTK_USE_SYSTEM_<MMM>` -> `VTK_MODULE_USE_EXTERNAL_<MMM>`
        Third party modules which don't support an external version no
        longer have an option available.
  - Turning off Python forces Python modules off.
  - Turning off Java forces Java modules off.
  - Turning off MPI forces MPI modules off.
  - Configuring VTK is now faster. An "all modules" build with Python
    wrapping enabled:

      cmake .  4.68s user 0.84s system 98% cpu 5.578 total

    Doing this on `master` on the same machine and Python *off*:

      cmake .  25.44s user 19.79s system 99% cpu 45.297 total

    This is not a scientific comparison; the module set is as close as I
    could make it, but if anything, `master` has fewer modules enabled
    (find modules not working, Python being disabled, some modules are
    new due to splitting, etc.).
  - CMake output is now minimal. Here is the complete output from
    reconfiguring VTK:

    % cmake .
    -- HDF5: Using hdf5 compiler wrapper to determine C configuration
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /home/boeckb/code/depot/group-kitware/vtk/build

    Not sure where the HDF5 output is coming from; might be a bug in
    `FindHDF5`.
  - The VTKm submodule has moved one directory lower (it is now
    ThirdParty/vtkm/vtkvtkm/vtk-m so that the settings for a local VTKm
    are localized).

For development:

  - Adding a module is similar to before. See `vtk.module` files in the
    source tree.
  - All files should be included in the source listing for modules.
    Using `CLASSES` is the same as doing `<NAME>.cxx` as a source and
    `<NAME>.h` as a public header. Private headers should use
    `PRIVATE_HEADERS` (e.g., object factory files).
  - To hide a module from a build (e.g., a Windows-only module), the
    `CONDITION` argument should be used in the `vtk.module` file. This
    is done for the Python and Java modules.
  - The object factory API has also been updated to the same patterns as
    the new module system (output variables rather than directory
    properties). See existing code for usage.
  - The new CMake API only works when controlled via the
    `vtk_module_scan` and `vtk_module_build` functions. Global and
    target properties are used to store information, so
    `vtk_module_add_module` doesn't work with just a simple
    `add_subdirectory`).
  - Due to kits being a thing, the backing target for a module may have
    different names based of CMake flags. Basically, CMake's `target_`
    commands need to be given `CommonCore`, `VTK::CommonCore`, or
    `CommonCore-objects` based on the build settings. To simplify this,
    there are functions like `vtk_module_include` or `vtk_module_link`
    for handling modules. Similarly for properties, there is
    `vtk_module_set_property`.
  - Prefer imported targets rather than `INCLUDE_DIRS` and `LIBRARIES`
    variables. This ensures that usage requirements are properly done
    and that VTK is relocatable once installed.
  - Public cache variables should be placed as close to their usage as
    possible. For example, the OpenGL settings are now in
    `Utilities/OpenGL` rather than the top level.

For consumers of VTK:

  - `find_package(VTK COMPONENTS CommonCore)` should be used.
    `OPTIONAL_COMPONENTS` is also supported.
  - Install tree is largely the same layout (library names are still
    `vtkCommonCore-X.Y`).
  - For object factories to work, `vtk_module_autoinit` should be used.
    This generates the proper `-D` flags for getting initialization to
    work.
  - Language wrapping APIs is now *much* simpler. Here is the call to
    wrap VTK's modules in Python:

      vtk_module_wrap_python(
        MODULES         ${vtk_modules}
        INSTALL_EXPORT  VTK
        PYTHON_PACKAGE  "vtkmodules"
        MODULE_DESTINATION  "${VTK_PYTHON_SITE_PACKAGES_SUFFIX}"
        CMAKE_DESTINATION   "${vtk_cmake_destination}"
        WRAPPED_MODULES vtk_python_wrapped_modules
        TARGET          VTK::vtkpythonmodules)

    and Java:

      vtk_module_wrap_java(
        JAVA_OUTPUT     "${CMAKE_CURRENT_BINARY_DIR}/src/vtk"
        MODULES         ${vtk_modules}
        WRAPPED_MODULES vtk_java_wrapped_modules)

  - No more PythonD libraries. VTK modules now import their dependent
    modules automatically rather than linking to the backing
    implementations directly (mainly a packager detail, but important).

Things I've noticed on this adventure:

  - To default a module to being built, add it to a group that is on by
    default (currently just `StandAlone` and `Rendering`). Excess
    dependencies is not the way to do this.
  - There are a few modules with debug code which uses modules not in
    the dependency list. Debug code should constrain itself to the same
    dependencies the rest of the code uses (or have just a comment in
    the `vtk.module` file as to its use).

Current state:

  - Fewer than 25 (out of 2000+) tests failing on a build everything[2]
    build with Python enabled.
  - Python optional linking is currently disabled. On Linux, linking
    errors cropped up and macOS had runtime errors appear. Investigation
    is needed.
  - Some examples disabled (Medical and Modelling are ported, but
    *their* tests fail right now). Those with `UNMAINTAINED.md` files
    are *not* ported.
  - The Medical and Modelling examples are disabled. They run tests
    during which fail. Investigation is nedded.
  - Debug points need to be added. However, I'd like to see where
    they're commonly necessary before adding them everywhere they
    *might* be useful. Places I can forsee it being nice to have:
    * Why is this test module not enabled?
    * Why is this module built?
    * Why is this module not built?
  - Remote modules have not been ported nor are the download/build
    mechanisms wired up.
  - There seems to be some issue with finding GLX on some Ubuntu
    installs with custom nvidia driver installations. It's likely
    something wrong with FindOpenGL, but more information needs to be
    gathered.

Near future tasks:

  - The new module system now supports optional test dependencies.
    Moving some test dependencies to be optional and only building the
    parts of the test suite that can would help improve how much of the
    test suite runs in smaller builds.
  - Excess data. There is data that nothing uses (input and baselines).
    This should be sorted through and we should figure out whether they
    were missed when removing old tests or new tests should be written
    to use the data.
  - Updating WikiExamples to use the new system.

Further future tasks:

  - Update Python code to `from vtkmodules.MMM import XXX` instead of
    `from vtk import XXX`. This should vastly improve Python test times.
  - Reducing the number of `try_compile` performed. This will immensely
    help with initial configure times. One idea is to port third party
    libraries over to using KWIML for builtin types rather than checking
    their sizes and signedness at configure time. Upstreamable solutions
    are preferred however.

Third party packages in VTK have also been updated. In general, they no
longer add (visible) cache entries at all and their `try_compile` tests
have been minimized. Here's the current listing of internal third party
libraries in VTK:

  diy2-3.5.0 (I)            libproj 4.9.3       tiff 4.0.6
  doubleconversion 3.1.1    libxml2 2.9.8       utf8 2.3.4 (I)
  eigen 3.3.5               lz4 1.8.3           verdict 1.2.0 (IC)
  exodusII 7.15f (I)        lzma 5.2.4          vpic ???
  expat 2.2.6               mpi4py 2.0.0        vtk-m <master> (I)
  freetype 2.9.1            netcdf 4.6.1        xdmf2 1.2.11 (I)
  gl2ps 1.4.0 (P)           ogg 1.3.3           xdmf3 1.2.11 (I)
  glew 2.1.0                pegtl 2.7.1 (I)     zfp 0.5.4 (I)
  hdf5 1.10.3               png 1.6.35          zlib 1.2.11
  jpeg 2.0.0                pugixml 1.9         kwiml <master> (I)
  jsoncpp 1.8.4             sqlite 3.25.2       kwsys <master> (I)
  kissfft ??? (I)           theora 1.1.1        metaio <master> (I)
  libharu 2.4.0 (P)

  - `C` means it needs converted to use `update.sh` yet
  - `I` means external versions are not supported
  - `P` means upstream needs to merge some PRs before an external
    version would actually work (links to PRs in the relevant CMake
    file)

CDash will now be showing all warnings again (I cleaned out the regular
expressions since third party libraries have been uplifted). I've also
pushed many GCC 8 warning fixes upstream, so MSVC should be the only one
with oodles of warnings yet. When adding exclusions back, please include
comments as to why they are there and, if applicable, a link to an issue
about actually addressing them.

--Ben

[1]There are a few global variables that are read, but they only control
*defaults* to cache variables, not the variable itself (e.g., defaulting
StandAlone and Rendering groups to be built).

[2]Some modules were excluded from this testing on some platforms due to
a lack of dependencies on the testing machines:

Linux:
    VTK::FiltersOpenTurns     VTK::RenderingOculus
    VTK::IOADIOS              VTK::RenderingOpenVR
    VTK::IOPDAL               VTK::RenderingOptiX
    VTK::RenderingOSPRay
macOS:
    VTK::FiltersOpenTurns     VTK::RenderingOSPRay
    VTK::IOADIOS              VTK::RenderingOculus
    VTK::IOMySQL              VTK::RenderingOpenVR
    VTK::IOPDAL               VTK::RenderingOptiX
    VTK::IOPostgreSQL         VTK::mpi
    VTK::Java
Windows:
    VTK::DomainsMicroscopy    VTK::IOPostgreSQL
    VTK::FiltersOpenTurns     VTK::InfovisBoost
    VTK::FiltersReebGraph     VTK::InfovisBoostGraphAlgorithms
    VTK::GUISupportMFC        VTK::Java
    VTK::IOADIOS              VTK::RenderingExternal
    VTK::IOFFMPEG             VTK::RenderingFreeTypeFontConfig
    VTK::IOGDAL               VTK::RenderingOSPRay
    VTK::IOIODBC              VTK::RenderingOculus
    VTK::IOLAS                VTK::RenderingOpenVR
    VTK::IOPDAL               VTK::RenderingOptiX
    VTK::IOPDAL               VTK::xdmf3

Code itself was not touched except where errors were otherwise found.
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

VTK - Dev mailing list
On Wed, Jan 02, 2019 at 14:23:53 -0500, Ben Boeckel wrote:
> The new module system is in a landable state right now. We're planning
> on merging it Friday (4 Jan 2019).

This has been delayed until at least Monday afternoon.

> The buildbots will be updated shortly afterwards.

The change is pending for this. Buildbots will now build "all" modules
by default and we'll turn off those that don't build on the machines.
This means that new modules in merge requests will be tested by default
as well. Some post-merge work to clean up the set of test exclusions and
disabling modules will be necessary.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

Bill Lorensen
I'm not sure if I asked this before? Can the old and new module system coexist?

On Thu, Jan 3, 2019, 12:16 PM Ben Boeckel via vtk-developers <[hidden email] wrote:
On Wed, Jan 02, 2019 at 14:23:53 -0500, Ben Boeckel wrote:
> The new module system is in a landable state right now. We're planning
> on merging it Friday (4 Jan 2019).

This has been delayed until at least Monday afternoon.

> The buildbots will be updated shortly afterwards.

The change is pending for this. Buildbots will now build "all" modules
by default and we'll turn off those that don't build on the machines.
This means that new modules in merge requests will be tested by default
as well. Some post-merge work to clean up the set of test exclusions and
disabling modules will be necessary.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

VTK - Dev mailing list
On Thu, Jan 03, 2019 at 15:23:18 -0800, Bill Lorensen wrote:
> I'm not sure if I asked this before? Can the old and new module system
> coexist?

From VTK's side, not really (i.e., it can't provide both at the same
time). There are just core incompatibilities between the two (the
current one is allergic to imported targets and the new one strongly
prefers them).

For projects providing VTK modules, it won't be easy, but should be
possible.

For projects just consuming VTK, supporting VTK 8.2 and 9.0 at the same
time is possible with a few if blocks and variables for abstraction. I
plan on getting to WikiExamples after ParaView has its merge.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

VTK - Dev mailing list
In reply to this post by VTK - Dev mailing list
On Thu, Jan 03, 2019 at 15:16:22 -0500, Ben Boeckel wrote:
> On Wed, Jan 02, 2019 at 14:23:53 -0500, Ben Boeckel wrote:
> > The new module system is in a landable state right now. We're planning
> > on merging it Friday (4 Jan 2019).
>
> This has been delayed until at least Monday afternoon.

It has now been merged.

Buildbots will be a little rocky until their module set is minimized to
those they can actually build. I hope to make decent progress on that
today.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

VTK - Dev mailing list
In reply to this post by VTK - Dev mailing list
On Wed, Jan 02, 2019 at 14:23:53 -0500, Ben Boeckel wrote:
>       - `Module_<MMM>` -> `VTK_MODULE_ENABLE_<MMM>`
>       - `VTK_USE_SYSTEM_<MMM>` -> `VTK_MODULE_USE_EXTERNAL_<MMM>`

So after some fiddling around with Buildbot, it turns out that putting
`::` in cache variable names makes it really hard to pass them on the
command line. The cache variables have `::` replaced with `_`.

The buildbot cleanup work is ongoing in:

    https://gitlab.kitware.com/vtk/vtk/merge_requests/5039

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

VTK - Dev mailing list
On Tue, Jan 08, 2019 at 11:56:48 -0500, Ben Boeckel wrote:

> On Wed, Jan 02, 2019 at 14:23:53 -0500, Ben Boeckel wrote:
> >       - `Module_<MMM>` -> `VTK_MODULE_ENABLE_<MMM>`
> >       - `VTK_USE_SYSTEM_<MMM>` -> `VTK_MODULE_USE_EXTERNAL_<MMM>`
>
> So after some fiddling around with Buildbot, it turns out that putting
> `::` in cache variable names makes it really hard to pass them on the
> command line. The cache variables have `::` replaced with `_`.
>
> The buildbot cleanup work is ongoing in:
>
>     https://gitlab.kitware.com/vtk/vtk/merge_requests/5039

This MR has now been merged. The buildbots are building code at least.
I'll work on cleaning up the warnings in the coming days with third
party patches and sending the relevant ones upstream.

Current state of test failures:

  - taanab and adora: CMake is warning about the inability to make
    fully correct rpath variables for targets. There may be a way to fix
    this, but it may be up to doing a suppression as well.
  - duma: it should be using OSMesa, but is not.
  - example tests: Examples aren't downloading example data properly.
  - various machines: since freetype has been updated, it seems that
    some new baselines will need to be added. I'll do that as well, but
    feel free to grab some if you want.
  - mun: static build failure with jsoncpp symbols. Will look into this
    tomorrow.
  - mun: shared+kits Python tests are failing. I suspect a missing PATH
    entry or the like. Will investgate tomorrow.

Buildbots are now building all modules allowed by their configuration.
Here is a list of modules that are explicitly disabled on machines
(usually due to lacking CUDA, Boost, or other dependency) that would
otherwise build due to the configuration[1]:

    1         'VTK_MODULE_ENABLE_VTK_GUISupportMFC:STRING': 'NO',
    1         'VTK_MODULE_ENABLE_VTK_IOOpenTurns:STRING': 'NO',
    4         'VTK_MODULE_ENABLE_VTK_RenderingExternal:STRING': 'NO',
    5         'VTK_MODULE_ENABLE_VTK_FiltersReebGraph:STRING': 'NO',
    5         'VTK_MODULE_ENABLE_VTK_InfovisBoostGraphAlgorithms:STRING': 'NO',
    5         'VTK_MODULE_ENABLE_VTK_IOADIOS:STRING': 'NO',
    5         'VTK_MODULE_ENABLE_VTK_xdmf3:STRING': 'NO',
    6         'VTK_MODULE_ENABLE_VTK_IOLAS:STRING': 'NO',
    6         'VTK_MODULE_ENABLE_VTK_RenderingFreeTypeFontConfig:STRING': 'NO',
    7         'VTK_MODULE_ENABLE_VTK_DomainsMicroscopy:STRING': 'NO',
    7         'VTK_MODULE_ENABLE_VTK_InfovisBoost:STRING': 'NO',
    7         'VTK_MODULE_ENABLE_VTK_IOFFMPEG:STRING': 'NO',
    7         'VTK_MODULE_ENABLE_VTK_IOGDAL:STRING': 'NO',
    7         'VTK_MODULE_ENABLE_VTK_IOODBC:STRING': 'NO',
    8         'VTK_MODULE_ENABLE_VTK_FiltersOpenTurns:STRING': 'NO',
    8         'VTK_MODULE_ENABLE_VTK_IOMySQL:STRING': 'NO',
    8         'VTK_MODULE_ENABLE_VTK_IOPDAL:STRING': 'NO',
    8         'VTK_MODULE_ENABLE_VTK_IOPostgreSQL:STRING': 'NO',
    9         'VTK_MODULE_ENABLE_VTK_RenderingOpenVR:STRING': 'NO',
    9         'VTK_MODULE_ENABLE_VTK_RenderingOptiX:STRING': 'NO',

Note that there are 9 machines, so OpenVR and OptiX are currently completely
untested on the buildbots. If these modules should be more widely
tested, feel free to open an issue for the module and which
machines/configurations should be building it and we can try and get the
dependencies installed.

Turnaround times for builds for a from-scratch build:

                configure   build   test    total   prior total
adora:          1:44        19:09  55:55    76:48   29:46
bigmac:         2:22        17:17   5:49    25:28   30:57
dejagore:       <busy with vtk-m>                   54:18
duma:           0:49        14:27  12:00    27:16   17:48
eeloo:
  +extdeps:     1:42        30:22   2:55    34:59   37:23
  +nogl:        0:49        13:45   0:13    14:47   17:54
luigi:          1:40        33:41   5:03    40:24   42:26
mun:
  (no kits):    3:15        12:38   5:38    21:31   35:15
  static:       3:20        <fails>                 39:45
  +kits:        3:19        8:29    3:55    15:43   34:55
taanab:         2:21        11:50   1:59    16:10   16:34
trey:           2:30        33:50   8:10    44:30   43:09

Note that builds are (generally) building more modules and tests than
before. From this analysis, I'm seeing that the eeloo+nogl builder has
270 tests versus the master's 807 tests, but this may be due to
dependency specifications being different. mun without kits is also
missing ~800 or so tests. adora's abysmal test time is due to timeouts
on the machine; I am seeing some X-related errors and they all seem to
be Python tests as well. dejagore is currently busy with `vtk-m`
testing, but I'm headed out for the day, so I'll leave it for tomorrow.

--Ben

[1]For example, GUISupportMFC is only explicitly disabled on Windows
machines since it "hides" on non-Windows builds. Other modules are also
masked by things like `VTK_USE_MPI`, `VTK_WRAP_PYTHON`, or platform
checks.
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

VTK - Dev mailing list
On Tue, Jan 08, 2019 at 18:11:54 -0500, Ben Boeckel wrote:
> Turnaround times for builds for a from-scratch build:

Filling this out with `dejagore`'s timing:

>                 configure   build   test    total   prior total
> adora:          1:44        19:09  55:55    76:48   29:46
> bigmac:         2:22        17:17   5:49    25:28   30:57
dejagore:         2:38        32:38   9:35    44:51   54:18

> duma:           0:49        14:27  12:00    27:16   17:48
> eeloo:
>   +extdeps:     1:42        30:22   2:55    34:59   37:23
>   +nogl:        0:49        13:45   0:13    14:47   17:54
> luigi:          1:40        33:41   5:03    40:24   42:26
> mun:
>   (no kits):    3:15        12:38   5:38    21:31   35:15
>   static:       3:20        <fails>                 39:45
>   +kits:        3:19        8:29    3:55    15:43   34:55
> taanab:         2:21        11:50   1:59    16:10   16:34
> trey:           2:30        33:50   8:10    44:30   43:09

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

VTK - Dev mailing list
In reply to this post by VTK - Dev mailing list
On Tue, Jan 08, 2019 at 18:11:54 -0500, Ben Boeckel wrote:
> Current state of test failures:
>
>   - taanab and adora: CMake is warning about the inability to make
>     fully correct rpath variables for targets. There may be a way to fix
>     this, but it may be up to doing a suppression as well.

Once things have settled down a bit, I'll look into this more.

>   - duma: it should be using OSMesa, but is not.

How to get this working properly is under discussion:

    https://gitlab.kitware.com/vtk/vtk/merge_requests/5044

>   - example tests: Examples aren't downloading example data properly.

This has been fixed.

>   - various machines: since freetype has been updated, it seems that
>     some new baselines will need to be added. I'll do that as well, but
>     feel free to grab some if you want.

New baselines have been added.

>   - mun: static build failure with jsoncpp symbols. Will look into this
>     tomorrow.

Fixed.

>   - mun: shared+kits Python tests are failing. I suspect a missing PATH
>     entry or the like. Will investgate tomorrow.

Still not working. Needs more investigation.

The state of testing is as follows:

  - Tests failing on multiple machines. These seem worth looking into.
      * VTK::IOGDALCxx-TestGDALRasterReader
            scalar array count mismatch
      * VTK::RenderingExternalCxx-TestGLUTRenderWindow
            fails to get an OpenGL 3.2 context on some machines
      * VTK::RenderingFreeTypeFontConfigCxx-TestSystemFontRendering
            trey isn't rendering the Times fonts as italic
      * VTK::IOExportPDFCxx-TestPDFTransformedText
            Not really sure what's going on here. There's no test image
            for comparison.
      * VTK::IOParallelLSDynaCxx-MPI-PLSDynaReader
            Looks to be a scaling issue, but the geometry looks OK.
      * VTK::IOParallelXMLPython-MPI-testParallelXMLWriters
            Assertion about the number of triangles failing.
      * VTK::RenderingMatplotlibCxx-TestGL2PSMathTextOutput-VerifyRasterizedPNG
            Image size mismatch.
      * VTK::RenderingMatplotlibCxx-TestRenderString
            Image size mismatch.
      * VTK::IOVPICCxx-TestVPICReader
            Failing on Windows due to it expecting `\` path separators
            but being passed `/` path separators. I tried to fix it, but
            it wasn't sufficient. Needs more debugging.
      * VTK::RenderingCoreCxx-TestFollowerPicking
            Currently disabled on taanab (it was previously) due to
            error messages:
                vtkInteractorStyleJoystickCamera (0x1e0e030): Timer start failed
      * Example tests on Windows
            Needs prefixes filled out as mentioned here:
                https://public.kitware.com/pipermail/vtk-developers/2019-January/036703.html
  - The static+python builder has had Python disabled for now
        https://gitlab.kitware.com/vtk/vtk/issues/17478
  - bigmac has a number of tests which end up with a pixel or two off
    for the test image size compared to the baselines. Not sure why this
    is. Most, but not all, of them are volume rendering tests.
  - adora is still failing to do XQueryExtension which seems…odd. Still
    need to look into it more.

All third party warnings have been suppressed for now (though some may
slip through yet). I'll deal with those in the future.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

Sean McBride
In reply to this post by VTK - Dev mailing list
On Wed, 2 Jan 2019 14:23:53 -0500, Ben Boeckel via vtk-developers said:

>  - Changed cache variables:
>    * Module and group flags:
>      - `Module_<MMM>` -> `VTK_MODULE_ENABLE_<MMM>`
>      - `VTK_Group_<MMM>` -> `VTK_GROUP_ENABLE_<MMM>`

I'll updated my buildbot scripts with the above (and below) renames.

>      - Values are no longer boolean, but instead 5-state:
>        * `YES`: Must be built
>        * `NO`: Must not be built (turns off modules requiring it;
>          errors if required by a `YES` module)
>        * `WANT`: Build if possible
>        * `DONT_WANT`: Build only if necessary
>        * `DEFAULT`: Use group (for modules) and `VTK_BUILD_ALL_MODULES`
>          instead

So they are strings I guess?

>      - `VTK_USE_SYSTEM_<MMM>` -> `VTK_MODULE_USE_EXTERNAL_<MMM>`
>        Third party modules which don't support an external version no
>        longer have an option available.

This new string is so long that in ccmake I only see:

 VTK_MODULE_USE_EXTERNAL_VTK_do  *OFF                                                                                                                                            
 VTK_MODULE_USE_EXTERNAL_VTK_ei  *OFF                                                                                                                                            
 VTK_MODULE_USE_EXTERNAL_VTK_ex  *OFF                                                                                                                                            
 VTK_MODULE_USE_EXTERNAL_VTK_fr  *OFF                                                                                                                                            

Is there anyway to resize those columns?

Cheers,

--
____________________________________________________________
Sean McBride, B. Eng                 [hidden email]
Rogue Research                        www.rogue-research.com
Mac Software Developer              Montréal, Québec, Canada


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

VTK - Dev mailing list
On Tue, Jan 15, 2019 at 16:11:10 -0500, Sean McBride wrote:
> I'll updated my buildbot scripts with the above (and below) renames.

Thanks.

> So they are strings I guess?

Yes.

> This new string is so long that in ccmake I only see:
>
>  VTK_MODULE_USE_EXTERNAL_VTK_do  *OFF
>  VTK_MODULE_USE_EXTERNAL_VTK_ei  *OFF
>  VTK_MODULE_USE_EXTERNAL_VTK_ex  *OFF
>  VTK_MODULE_USE_EXTERNAL_VTK_fr  *OFF
>
> Is there anyway to resize those columns?

Unfortunately, no :( . I've filed a CMake issue about it:

    https://gitlab.kitware.com/cmake/cmake/issues/18809

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

VTK - Dev mailing list
In reply to this post by VTK - Dev mailing list
On Thu, Jan 10, 2019 at 17:07:12 -0500, Ben Boeckel wrote:
> >   - mun: shared+kits Python tests are failing. I suspect a missing PATH
> >     entry or the like. Will investgate tomorrow.
>
> Still not working. Needs more investigation.

Actually turned out to be some mismatches between setters and getters in
the module system itself. MR for fixing it is here:

    https://gitlab.kitware.com/vtk/vtk/merge_requests/5077

Now all `mun` builds are just failing examples (TBB information needs
forwarding through `vtk-config.cmake`) and 2 other failures, one for the
VPIC thing and an LSDyna failure other machines are seeing.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

Sean McBride
In reply to this post by VTK - Dev mailing list
Ben,

If I create a fresh bin directory and change only VTK_FORBID_DOWNLOADS to NO, then configure and build, I get an error:

-------------------
CMake Error at /Applications/CMake.app/Contents/share/cmake-3.13/Modules/ExternalData.cmake:1121 (message):
 

  Object
  SHA512=d46218c565aa140b706c01b41bce9179106a55875f669bb1a5c2227efa677e9a8066cd5b28ee2be806ebec93d7ab72040562491a963c3bf1c3606db239097b13
  not found at:

    (No ExternalData_URL_TEMPLATES given)
-------------------

So I set BUILD_TESTING to off, but that didn't help.

Then I noticed a VTK_BUILD_TESTING, is that new?  Turning it off fixed it.  So it's clearly different from BUILD_TESTING.  The setting description doesn't help:

BUILD_TESTING: Build the testing tree.                                                                                                                                            
VTK_BUILD_TESTING: Build all modules by default                                                                                                                                    

Is that last description even right?  Why have both these settings anyway?  Should setting VTK_FORBID_DOWNLOADS to NO force testing to NO also?

Cheers,

--
____________________________________________________________
Sean McBride, B. Eng                 [hidden email]
Rogue Research                        www.rogue-research.com
Mac Software Developer              Montréal, Québec, Canada


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

VTK - Dev mailing list
On Thu, Jan 17, 2019 at 12:27:33 -0500, Sean McBride wrote:
> So I set BUILD_TESTING to off, but that didn't help.

`BUILD_TESTING` is a CTest variable which makes `add_test` a no-op.

> Then I noticed a VTK_BUILD_TESTING, is that new?  Turning it off fixed
> it.  So it's clearly different from BUILD_TESTING.  The setting
> description doesn't help:
>
> BUILD_TESTING: Build the testing tree
> VTK_BUILD_TESTING: Build all modules by default

Yeah, the description looks like copy-pasta. It has 3 settings:

  ON: if a module is on, build its tests
  WANT: if a module is on, try to turn on test dependencies and build
        tests if those are all available
  OFF: no module tests

How about "Build module testing directories"?

> Is that last description even right?  Why have both these settings
> anyway?  Should setting VTK_FORBID_DOWNLOADS to NO force testing to NO
> also?

I wasn't aware of `VTK_FORBID_DOWNLOADS`. It can be taken into account
before passing it via `ENABLE_TESTS` to `vtk_module_scan`.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

Sean McBride
On Thu, 17 Jan 2019 12:57:49 -0500, Ben Boeckel said:

>> BUILD_TESTING: Build the testing tree
>> VTK_BUILD_TESTING: Build all modules by default
>
>Yeah, the description looks like copy-pasta. It has 3 settings:
>
>  ON: if a module is on, build its tests
>  WANT: if a module is on, try to turn on test dependencies and build
>        tests if those are all available
>  OFF: no module tests
>
>How about "Build module testing directories"?

Sure.

>> Is that last description even right?  Why have both these settings
>> anyway?  Should setting VTK_FORBID_DOWNLOADS to NO force testing to NO
>> also?
>
>I wasn't aware of `VTK_FORBID_DOWNLOADS`. It can be taken into account
>before passing it via `ENABLE_TESTS` to `vtk_module_scan`.

Thanks.


Next, I'm trying to build ITK master against VTK master and get:

 CMake Error at Modules/Bridge/VtkGlue/CMakeLists.txt:38 (vtk_module_config):
   Unknown CMake command "vtk_module_config".

I'm gonna guess this too is related to this 'new module system' stuff, yes?  Does ITK need to change?  Or is VTK missing some backward compatibility something?

Cheers,

--
____________________________________________________________
Sean McBride, B. Eng                 [hidden email]
Rogue Research                        www.rogue-research.com
Mac Software Developer              Montréal, Québec, Canada


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

VTK - Dev mailing list
On Thu, Jan 17, 2019 at 13:52:22 -0500, Sean McBride wrote:

> On Thu, 17 Jan 2019 12:57:49 -0500, Ben Boeckel said:
> >How about "Build module testing directories"?
>
> Sure.
>
> >> Is that last description even right?  Why have both these settings
> >> anyway?  Should setting VTK_FORBID_DOWNLOADS to NO force testing to NO
> >> also?
> >
> >I wasn't aware of `VTK_FORBID_DOWNLOADS`. It can be taken into account
> >before passing it via `ENABLE_TESTS` to `vtk_module_scan`.
>
> Thanks.

MR up shortly.

> Next, I'm trying to build ITK master against VTK master and get:
>
>  CMake Error at Modules/Bridge/VtkGlue/CMakeLists.txt:38 (vtk_module_config):
>    Unknown CMake command "vtk_module_config".
>
> I'm gonna guess this too is related to this 'new module system' stuff,
> yes?  Does ITK need to change?  Or is VTK missing some backward
> compatibility something?

There's no provided backwards compat bridge. Looking, I suspect a block
like:

    elseif (VTK_VERSION VERSION_LESS 8.90)

should wrap the existing `else` and a new block be added.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] New module system landing

Bill Lorensen
Sean, I'm working on getting the VTKExamples working with both old and new VTK modules. I've got a working solution for individual examples, e.g. https://lorensen.github.io/VTKExamples/site/Cxx/GeometricObjects/Cube/#cmakeliststxt

Next I'll tackle building the whole suite.

Bill

On Thu, Jan 17, 2019, 11:07 AM Ben Boeckel via vtk-developers <[hidden email] wrote:
On Thu, Jan 17, 2019 at 13:52:22 -0500, Sean McBride wrote:
> On Thu, 17 Jan 2019 12:57:49 -0500, Ben Boeckel said:
> >How about "Build module testing directories"?
>
> Sure.
>
> >> Is that last description even right?  Why have both these settings
> >> anyway?  Should setting VTK_FORBID_DOWNLOADS to NO force testing to NO
> >> also?
> >
> >I wasn't aware of `VTK_FORBID_DOWNLOADS`. It can be taken into account
> >before passing it via `ENABLE_TESTS` to `vtk_module_scan`.
>
> Thanks.

MR up shortly.

> Next, I'm trying to build ITK master against VTK master and get:
>
>  CMake Error at Modules/Bridge/VtkGlue/CMakeLists.txt:38 (vtk_module_config):
>    Unknown CMake command "vtk_module_config".
>
> I'm gonna guess this too is related to this 'new module system' stuff,
> yes?  Does ITK need to change?  Or is VTK missing some backward
> compatibility something?

There's no provided backwards compat bridge. Looking, I suspect a block
like:

    elseif (VTK_VERSION VERSION_LESS 8.90)

should wrap the existing `else` and a new block be added.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers