Speeding up `import vtk`

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Speeding up `import vtk`

Utkarsh Ayachit
Folks,

This is in reference to this: https://gitlab.kitware.com/vtk/vtk/issues/16780

`import vtk` is slow because it imports all of VTK. It isn't the best
practice, and has no workaround -- given the current implementation
where all the modules are imported in vtk/__init__.py itself.

I have an MR in progress that addresses this issue:
https://gitlab.kitware.com/vtk/vtk/merge_requests/1921

This MR does the following:
* vtk/__init__.py no longer imports all of VTK.
* vtk/all.py is a new module that imports all of VTK.

This does break old scripts, but provides an easy workaround. For
users who want to keep previous behavior, they can simply do the
following:

   from vtk import all as vtk

Thoughts? If this looks reasonable to everyone, I'll update the MR to
fix all tests accordingly.

Thanks
Utkarsh
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

David Gobbi
Hi Utkarsh,

As usual, I'm a stickler for backwards compatibility and would rather go in the other direction,

  import vtkminimal as vtk

 - David

On Thu, Sep 8, 2016 at 10:55 AM, Utkarsh Ayachit <[hidden email]> wrote:
Folks,

This is in reference to this: https://gitlab.kitware.com/vtk/vtk/issues/16780

`import vtk` is slow because it imports all of VTK. It isn't the best
practice, and has no workaround -- given the current implementation
where all the modules are imported in vtk/__init__.py itself.

I have an MR in progress that addresses this issue:
https://gitlab.kitware.com/vtk/vtk/merge_requests/1921

This MR does the following:
* vtk/__init__.py no longer imports all of VTK.
* vtk/all.py is a new module that imports all of VTK.

This does break old scripts, but provides an easy workaround. For
users who want to keep previous behavior, they can simply do the
following:

   from vtk import all as vtk

Thoughts? If this looks reasonable to everyone, I'll update the MR to
fix all tests accordingly.

Thanks
Utkarsh

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Utkarsh Ayachit
The problem with that is it will have to be an entirely new package.
Further complication, this package cannot use any of *.py files for
VTK modules or others from the vtk package since as soon as it does
that vtk/__init__.py will import everything. So we'll need to
duplicate all the *.py files too.

On Thu, Sep 8, 2016 at 1:14 PM, David Gobbi <[hidden email]> wrote:

> Hi Utkarsh,
>
> As usual, I'm a stickler for backwards compatibility and would rather go in
> the other direction,
>
>   import vtkminimal as vtk
>
>  - David
>
>
> On Thu, Sep 8, 2016 at 10:55 AM, Utkarsh Ayachit
> <[hidden email]> wrote:
>>
>> Folks,
>>
>> This is in reference to this:
>> https://gitlab.kitware.com/vtk/vtk/issues/16780
>>
>> `import vtk` is slow because it imports all of VTK. It isn't the best
>> practice, and has no workaround -- given the current implementation
>> where all the modules are imported in vtk/__init__.py itself.
>>
>> I have an MR in progress that addresses this issue:
>> https://gitlab.kitware.com/vtk/vtk/merge_requests/1921
>>
>> This MR does the following:
>> * vtk/__init__.py no longer imports all of VTK.
>> * vtk/all.py is a new module that imports all of VTK.
>>
>> This does break old scripts, but provides an easy workaround. For
>> users who want to keep previous behavior, they can simply do the
>> following:
>>
>>    from vtk import all as vtk
>>
>> Thoughts? If this looks reasonable to everyone, I'll update the MR to
>> fix all tests accordingly.
>>
>> Thanks
>> Utkarsh
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

David Gobbi
There could be a way to change the behavior of "import vtk", e.g.

import vtk_settings
vtk_settings.modules = [ ... ] # whitelist of desired modules?
import vtk

The .py files for the VTK modules are actually a bit silly, since they just waste time moving classes from the .pyd module to the .py module.  We should build vtkCommonCore.pyd directly (instead of building vtkCommonCorePython.pyd).

 - David


On Thu, Sep 8, 2016 at 11:24 AM, Utkarsh Ayachit <[hidden email]> wrote:
The problem with that is it will have to be an entirely new package.
Further complication, this package cannot use any of *.py files for
VTK modules or others from the vtk package since as soon as it does
that vtk/__init__.py will import everything. So we'll need to
duplicate all the *.py files too.

On Thu, Sep 8, 2016 at 1:14 PM, David Gobbi <[hidden email]> wrote:
> Hi Utkarsh,
>
> As usual, I'm a stickler for backwards compatibility and would rather go in
> the other direction,
>
>   import vtkminimal as vtk
>
>  - David
>
>
> On Thu, Sep 8, 2016 at 10:55 AM, Utkarsh Ayachit
> <[hidden email]> wrote:
>>
>> Folks,
>>
>> This is in reference to this:
>> https://gitlab.kitware.com/vtk/vtk/issues/16780
>>
>> `import vtk` is slow because it imports all of VTK. It isn't the best
>> practice, and has no workaround -- given the current implementation
>> where all the modules are imported in vtk/__init__.py itself.
>>
>> I have an MR in progress that addresses this issue:
>> https://gitlab.kitware.com/vtk/vtk/merge_requests/1921
>>
>> This MR does the following:
>> * vtk/__init__.py no longer imports all of VTK.
>> * vtk/all.py is a new module that imports all of VTK.
>>
>> This does break old scripts, but provides an easy workaround. For
>> users who want to keep previous behavior, they can simply do the
>> following:
>>
>>    from vtk import all as vtk
>>
>> Thoughts? If this looks reasonable to everyone, I'll update the MR to
>> fix all tests accordingly.
>>
>> Thanks
>> Utkarsh


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Matt McCormick-2
In ITK, we use lazy loading. When a class is requested, its module is
loaded. When a module is loaded, all the dependent modules are loaded
(the module dependencies are exported from the build system). This
greatly reduces import times. It will be released in ITK 4.10.1. The
source tree has the details :-)

HTH,
Matt

On Thu, Sep 8, 2016 at 1:39 PM, David Gobbi <[hidden email]> wrote:

> There could be a way to change the behavior of "import vtk", e.g.
>
> import vtk_settings
> vtk_settings.modules = [ ... ] # whitelist of desired modules?
> import vtk
>
> The .py files for the VTK modules are actually a bit silly, since they just
> waste time moving classes from the .pyd module to the .py module.  We should
> build vtkCommonCore.pyd directly (instead of building
> vtkCommonCorePython.pyd).
>
>  - David
>
>
> On Thu, Sep 8, 2016 at 11:24 AM, Utkarsh Ayachit
> <[hidden email]> wrote:
>>
>> The problem with that is it will have to be an entirely new package.
>> Further complication, this package cannot use any of *.py files for
>> VTK modules or others from the vtk package since as soon as it does
>> that vtk/__init__.py will import everything. So we'll need to
>> duplicate all the *.py files too.
>>
>> On Thu, Sep 8, 2016 at 1:14 PM, David Gobbi <[hidden email]> wrote:
>> > Hi Utkarsh,
>> >
>> > As usual, I'm a stickler for backwards compatibility and would rather go
>> > in
>> > the other direction,
>> >
>> >   import vtkminimal as vtk
>> >
>> >  - David
>> >
>> >
>> > On Thu, Sep 8, 2016 at 10:55 AM, Utkarsh Ayachit
>> > <[hidden email]> wrote:
>> >>
>> >> Folks,
>> >>
>> >> This is in reference to this:
>> >> https://gitlab.kitware.com/vtk/vtk/issues/16780
>> >>
>> >> `import vtk` is slow because it imports all of VTK. It isn't the best
>> >> practice, and has no workaround -- given the current implementation
>> >> where all the modules are imported in vtk/__init__.py itself.
>> >>
>> >> I have an MR in progress that addresses this issue:
>> >> https://gitlab.kitware.com/vtk/vtk/merge_requests/1921
>> >>
>> >> This MR does the following:
>> >> * vtk/__init__.py no longer imports all of VTK.
>> >> * vtk/all.py is a new module that imports all of VTK.
>> >>
>> >> This does break old scripts, but provides an easy workaround. For
>> >> users who want to keep previous behavior, they can simply do the
>> >> following:
>> >>
>> >>    from vtk import all as vtk
>> >>
>> >> Thoughts? If this looks reasonable to everyone, I'll update the MR to
>> >> fix all tests accordingly.
>> >>
>> >> Thanks
>> >> Utkarsh
>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Search the list archives at: http://markmail.org/search/?q=vtk-developers
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/vtk-developers
>
>
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Utkarsh Ayachit
@Matt, Any links to the code? I have no idea where to start looking :).

@David, I like that, that seems like a reasonable workaround and
simple enough. I don't hear anything else, I'll cook something up for
the same.


On Thu, Sep 8, 2016 at 1:45 PM, Matt McCormick
<[hidden email]> wrote:

> In ITK, we use lazy loading. When a class is requested, its module is
> loaded. When a module is loaded, all the dependent modules are loaded
> (the module dependencies are exported from the build system). This
> greatly reduces import times. It will be released in ITK 4.10.1. The
> source tree has the details :-)
>
> HTH,
> Matt
>
> On Thu, Sep 8, 2016 at 1:39 PM, David Gobbi <[hidden email]> wrote:
>> There could be a way to change the behavior of "import vtk", e.g.
>>
>> import vtk_settings
>> vtk_settings.modules = [ ... ] # whitelist of desired modules?
>> import vtk
>>
>> The .py files for the VTK modules are actually a bit silly, since they just
>> waste time moving classes from the .pyd module to the .py module.  We should
>> build vtkCommonCore.pyd directly (instead of building
>> vtkCommonCorePython.pyd).
>>
>>  - David
>>
>>
>> On Thu, Sep 8, 2016 at 11:24 AM, Utkarsh Ayachit
>> <[hidden email]> wrote:
>>>
>>> The problem with that is it will have to be an entirely new package.
>>> Further complication, this package cannot use any of *.py files for
>>> VTK modules or others from the vtk package since as soon as it does
>>> that vtk/__init__.py will import everything. So we'll need to
>>> duplicate all the *.py files too.
>>>
>>> On Thu, Sep 8, 2016 at 1:14 PM, David Gobbi <[hidden email]> wrote:
>>> > Hi Utkarsh,
>>> >
>>> > As usual, I'm a stickler for backwards compatibility and would rather go
>>> > in
>>> > the other direction,
>>> >
>>> >   import vtkminimal as vtk
>>> >
>>> >  - David
>>> >
>>> >
>>> > On Thu, Sep 8, 2016 at 10:55 AM, Utkarsh Ayachit
>>> > <[hidden email]> wrote:
>>> >>
>>> >> Folks,
>>> >>
>>> >> This is in reference to this:
>>> >> https://gitlab.kitware.com/vtk/vtk/issues/16780
>>> >>
>>> >> `import vtk` is slow because it imports all of VTK. It isn't the best
>>> >> practice, and has no workaround -- given the current implementation
>>> >> where all the modules are imported in vtk/__init__.py itself.
>>> >>
>>> >> I have an MR in progress that addresses this issue:
>>> >> https://gitlab.kitware.com/vtk/vtk/merge_requests/1921
>>> >>
>>> >> This MR does the following:
>>> >> * vtk/__init__.py no longer imports all of VTK.
>>> >> * vtk/all.py is a new module that imports all of VTK.
>>> >>
>>> >> This does break old scripts, but provides an easy workaround. For
>>> >> users who want to keep previous behavior, they can simply do the
>>> >> following:
>>> >>
>>> >>    from vtk import all as vtk
>>> >>
>>> >> Thoughts? If this looks reasonable to everyone, I'll update the MR to
>>> >> fix all tests accordingly.
>>> >>
>>> >> Thanks
>>> >> Utkarsh
>>
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Search the list archives at: http://markmail.org/search/?q=vtk-developers
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/vtk-developers
>>
>>
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Matt McCormick-2
On Thu, Sep 8, 2016 at 2:27 PM, Utkarsh Ayachit
<[hidden email]> wrote:
> @Matt, Any links to the code? I have no idea where to start looking :).

There are different components, but here are some starting points:

  https://github.com/InsightSoftwareConsortium/ITK/blob/22cd834098516f0afbc669a10133463d9f7c890d/Wrapping/Generators/Python/itkLazy.py#L20-L48

  https://github.com/InsightSoftwareConsortium/ITK/blob/22cd834098516f0afbc669a10133463d9f7c890d/Wrapping/Generators/Python/itkBase.py#L34-L46
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Ben Boeckel
In reply to this post by David Gobbi
[ Yes, this is a necro on this thread, but I'd like to see this happen. ]

On Thu, Sep 08, 2016 at 11:14:21 -0600, David Gobbi wrote:
> As usual, I'm a stickler for backwards compatibility and would rather go in
> the other direction,

How about fixing `vtk` for VTK9? I don't like having it be a separate
module because that means we have to ship things twice and the only
difference is the `__init__.py` file.

Currently `import vtk` (the obvious thing to do) is basically like doing
a `find_package(VTK REQUIRED)` that doesn't support `COMPONENTS`. You
get everything built, but you also have no idea if what you need is
actually present until you get your `AttributeError`.

I'd rather see scripts explicitly request `vtk.FiltersParallel` (or
similar) if they need it rather than just hoping it dumped the right
things into the global namespace. If we want to keep having this
pattern, ParaView's `paraview.simple` module provides everything needed
to "hit the ground running" (though I don't see it as a long-term
maintainable way to use VTK or ParaView via Python). VTK could also have
`vtk.simple` which does this. The problem with having it at the root
level is that it denies any advanced usage of the module which does
request exactly what is required.

With all the other big changes coming in VTK9, I think it would be a
good time to tackle this.

--Ben

P.S. For those curious, this came up because we recently had one module
have more "load commands" than the loader on macOS supports
(vtkPVAnimation) which triggered an `ImportError`. Because `import vtk`
loads all modules, this meant that any loading of `import vtk` was
broken just because this one was, even tests which don't care about it
at all.
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

David Gobbi
Hi Ben,

I like Matt's comments on lazy-loading, it would allow us to keep backwards compatibility while speeding up load times. Python provides a way to override getattr() for modules:

 - David

On Tue, Sep 19, 2017 at 8:29 AM, Ben Boeckel <[hidden email]> wrote:
[ Yes, this is a necro on this thread, but I'd like to see this happen. ]

On Thu, Sep 08, 2016 at 11:14:21 -0600, David Gobbi wrote:
> As usual, I'm a stickler for backwards compatibility and would rather go in
> the other direction,

How about fixing `vtk` for VTK9? I don't like having it be a separate
module because that means we have to ship things twice and the only
difference is the `__init__.py` file.

Currently `import vtk` (the obvious thing to do) is basically like doing
a `find_package(VTK REQUIRED)` that doesn't support `COMPONENTS`. You
get everything built, but you also have no idea if what you need is
actually present until you get your `AttributeError`.

I'd rather see scripts explicitly request `vtk.FiltersParallel` (or
similar) if they need it rather than just hoping it dumped the right
things into the global namespace. If we want to keep having this
pattern, ParaView's `paraview.simple` module provides everything needed
to "hit the ground running" (though I don't see it as a long-term
maintainable way to use VTK or ParaView via Python). VTK could also have
`vtk.simple` which does this. The problem with having it at the root
level is that it denies any advanced usage of the module which does
request exactly what is required.

With all the other big changes coming in VTK9, I think it would be a
good time to tackle this.

--Ben

P.S. For those curious, this came up because we recently had one module
have more "load commands" than the loader on macOS supports
(vtkPVAnimation) which triggered an `ImportError`. Because `import vtk`
loads all modules, this meant that any loading of `import vtk` was
broken just because this one was, even tests which don't care about it
at all.


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Jean-Christophe Fillion-Robin
Hi, 

Lazy loading implemented in ITK python has been working well. Since we have knowledge of the VTK dependency graph, I suggest to use a similar approach.

Also building VTK with VTK_ENABLE_KITS enabled greatly reduces the number of libraries and could avoid to implement such optimization.

Matt (cc'ed) should be able to provide feedback.

Hth
Jc

On Wed, Sep 20, 2017 at 9:18 AM, David Gobbi <[hidden email]> wrote:
Hi Ben,

I like Matt's comments on lazy-loading, it would allow us to keep backwards compatibility while speeding up load times. Python provides a way to override getattr() for modules:

 - David

On Tue, Sep 19, 2017 at 8:29 AM, Ben Boeckel <[hidden email]> wrote:
[ Yes, this is a necro on this thread, but I'd like to see this happen. ]

On Thu, Sep 08, 2016 at 11:14:21 -0600, David Gobbi wrote:
> As usual, I'm a stickler for backwards compatibility and would rather go in
> the other direction,

How about fixing `vtk` for VTK9? I don't like having it be a separate
module because that means we have to ship things twice and the only
difference is the `__init__.py` file.

Currently `import vtk` (the obvious thing to do) is basically like doing
a `find_package(VTK REQUIRED)` that doesn't support `COMPONENTS`. You
get everything built, but you also have no idea if what you need is
actually present until you get your `AttributeError`.

I'd rather see scripts explicitly request `vtk.FiltersParallel` (or
similar) if they need it rather than just hoping it dumped the right
things into the global namespace. If we want to keep having this
pattern, ParaView's `paraview.simple` module provides everything needed
to "hit the ground running" (though I don't see it as a long-term
maintainable way to use VTK or ParaView via Python). VTK could also have
`vtk.simple` which does this. The problem with having it at the root
level is that it denies any advanced usage of the module which does
request exactly what is required.

With all the other big changes coming in VTK9, I think it would be a
good time to tackle this.

--Ben

P.S. For those curious, this came up because we recently had one module
have more "load commands" than the loader on macOS supports
(vtkPVAnimation) which triggered an `ImportError`. Because `import vtk`
loads all modules, this meant that any loading of `import vtk` was
broken just because this one was, even tests which don't care about it
at all.


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers





--
+1 919 869 8849

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Matt McCormick-2
Hi,

Yes, the lazy loading has been working very well for ITK. In addition
to the previous links to the implementation, a few other things to be
aware of are

- The dependencies are stored via CMake's configure_file on .py files
- There is an option to disable lazy loading in the itkConfig module,
which can be imported before "itk"
- An import callback can be configured to monitor / debug the process

VTK_ENABLE_KITS may actually be a little bit slower in the case of
lazy loading because more unused symbols have to be loaded, but that
performance assumption needs to be tested.

HTH,
Matt

On Thu, Sep 21, 2017 at 7:10 PM, Jean-Christophe Fillion-Robin
<[hidden email]> wrote:

> Hi,
>
> Lazy loading implemented in ITK python has been working well. Since we have
> knowledge of the VTK dependency graph, I suggest to use a similar approach.
>
> Also building VTK with VTK_ENABLE_KITS enabled greatly reduces the number of
> libraries and could avoid to implement such optimization.
>
> Matt (cc'ed) should be able to provide feedback.
>
> Hth
> Jc
>
> On Wed, Sep 20, 2017 at 9:18 AM, David Gobbi <[hidden email]> wrote:
>>
>> Hi Ben,
>>
>> I like Matt's comments on lazy-loading, it would allow us to keep
>> backwards compatibility while speeding up load times. Python provides a way
>> to override getattr() for modules:
>> https://stackoverflow.com/questions/2447353/getattr-on-a-module
>>
>>  - David
>>
>> On Tue, Sep 19, 2017 at 8:29 AM, Ben Boeckel <[hidden email]>
>> wrote:
>>>
>>> [ Yes, this is a necro on this thread, but I'd like to see this happen. ]
>>>
>>> On Thu, Sep 08, 2016 at 11:14:21 -0600, David Gobbi wrote:
>>> > As usual, I'm a stickler for backwards compatibility and would rather
>>> > go in
>>> > the other direction,
>>>
>>> How about fixing `vtk` for VTK9? I don't like having it be a separate
>>> module because that means we have to ship things twice and the only
>>> difference is the `__init__.py` file.
>>>
>>> Currently `import vtk` (the obvious thing to do) is basically like doing
>>> a `find_package(VTK REQUIRED)` that doesn't support `COMPONENTS`. You
>>> get everything built, but you also have no idea if what you need is
>>> actually present until you get your `AttributeError`.
>>>
>>> I'd rather see scripts explicitly request `vtk.FiltersParallel` (or
>>> similar) if they need it rather than just hoping it dumped the right
>>> things into the global namespace. If we want to keep having this
>>> pattern, ParaView's `paraview.simple` module provides everything needed
>>> to "hit the ground running" (though I don't see it as a long-term
>>> maintainable way to use VTK or ParaView via Python). VTK could also have
>>> `vtk.simple` which does this. The problem with having it at the root
>>> level is that it denies any advanced usage of the module which does
>>> request exactly what is required.
>>>
>>> With all the other big changes coming in VTK9, I think it would be a
>>> good time to tackle this.
>>>
>>> --Ben
>>>
>>> P.S. For those curious, this came up because we recently had one module
>>> have more "load commands" than the loader on macOS supports
>>> (vtkPVAnimation) which triggered an `ImportError`. Because `import vtk`
>>> loads all modules, this meant that any loading of `import vtk` was
>>> broken just because this one was, even tests which don't care about it
>>> at all.
>>
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Search the list archives at: http://markmail.org/search/?q=vtk-developers
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/vtk-developers
>>
>>
>
>
>
> --
> +1 919 869 8849
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Utkarsh Ayachit
In reply to this post by Ben Boeckel
I went the route of simpler implementation that better handles all
forms of importing vtk including `import vtk` and `from vtk import *`
etc. while still providing me with a package that doesn't import all
the modules.

The implementation is here:
https://gitlab.kitware.com/vtk/vtk/merge_requests/3674

In summary:
* a new `vtkmodules` package replaces the old `vtk` package which has
an empty __init__.py file, thus doesn't import anything by default.
* `vtkmodules.all`  module imports all vtk modules
* vtk.py is a new module that acts as a trampoline, forwarding to the
`vtkmodules` package but also imports everything from the all module.

I'd like to move forward with this change, if there are no objections.
It's 100% backwards compatible.

Utkarsh
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Prabhu Ramachandran-3
On 12/11/17 1:00 PM, Utkarsh Ayachit wrote:
The implementation is here:
https://gitlab.kitware.com/vtk/vtk/merge_requests/3674

In summary:
* a new `vtkmodules` package replaces the old `vtk` package which has
an empty __init__.py file, thus doesn't import anything by default.
* `vtkmodules.all`  module imports all vtk modules
Would `vtkm` be a shorter name rather than `vtkmodules`?

cheers,
Prabhu

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Ben Boeckel
On Tue, Dec 12, 2017 at 15:06:15 -0500, Prabhu Ramachandran wrote:
> Would `vtkm` be a shorter name rather than `vtkmodules`?

`vtkm` is a different project already (though I don't know of its status
with Python wrapping outside of the VTK filters which use it).

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Utkarsh Ayachit
On Tue, Dec 12, 2017 at 3:18 PM, Ben Boeckel <[hidden email]> wrote:
> On Tue, Dec 12, 2017 at 15:06:15 -0500, Prabhu Ramachandran wrote:
>> Would `vtkm` be a shorter name rather than `vtkmodules`?
>
> `vtkm` is a different project already (though I don't know of its status
> with Python wrapping outside of the VTK filters which use it).

Yea, alas `vtkm` won't be a good option.
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Jean-Christophe Fillion-Robin
Hi Utkarsh,

I would prefer to have a more generic way of configuring the import behavior.

The idea would be to have two top-level packages:  vtk and vtkConfig

To skip  loading of all module, you would do:

-------------------
import vtkConfig
vtkConfig.VTK_MODULE_IMPORT = vtkConfig.EXPLICIT   #  Default would be vtkConfig.ALL.  Later we could also have vtkConfig.LAZY

import vtk
-------------------

This would allow to keep things coherent for our user. Naming the package "vtkmodules" seem unconventional.

It would also have the advantage of support addition of new option.

For example:

vtkConfig.RENDERING_METHOD = vtkConfig.OFFSCREEN  #  or vtkConfig.MESA, ....

Jc


On Tue, Dec 12, 2017 at 4:55 PM, Utkarsh Ayachit <[hidden email]> wrote:
On Tue, Dec 12, 2017 at 3:18 PM, Ben Boeckel <[hidden email]> wrote:
> On Tue, Dec 12, 2017 at 15:06:15 -0500, Prabhu Ramachandran wrote:
>> Would `vtkm` be a shorter name rather than `vtkmodules`?
>
> `vtkm` is a different project already (though I don't know of its status
> with Python wrapping outside of the VTK filters which use it).

Yea, alas `vtkm` won't be a good option.
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers




--
+1 919 869 8849

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Utkarsh Ayachit
JC,

I am not a huge fan of this option since the `vtk`package will not
provide a consistent interface based on how the package was imported.
It will make it impossible to write scripts that use `import vtk` and
work reliably. e..g say I have a module (outside VTK) that relies on
`import vtk` to have all necessary symbols. Now my script will fail,
if some other module that the user imported before importing my module
changed the nature of the `vtk` package. Thus, the only way to write
reliable scripts is to import internal modules as needed, and that
defeats the purpose.

The future config you envision could currently be easily provided by
factory or other singletons in VTK library itself that control the
default render window created, for example, and should indeed be
supported there since it's useful for C++ users of VTK too.

Utkarsh

On Fri, Dec 15, 2017 at 12:04 AM, Jean-Christophe Fillion-Robin
<[hidden email]> wrote:

> Hi Utkarsh,
>
> I would prefer to have a more generic way of configuring the import
> behavior.
>
> The idea would be to have two top-level packages:  vtk and vtkConfig
>
> To skip  loading of all module, you would do:
>
> -------------------
> import vtkConfig
> vtkConfig.VTK_MODULE_IMPORT = vtkConfig.EXPLICIT   #  Default would be
> vtkConfig.ALL.  Later we could also have vtkConfig.LAZY
>
> import vtk
> -------------------
>
> This would allow to keep things coherent for our user. Naming the package
> "vtkmodules" seem unconventional.
>
> It would also have the advantage of support addition of new option.
>
> For example:
>
> vtkConfig.RENDERING_METHOD = vtkConfig.OFFSCREEN  #  or vtkConfig.MESA, ....
>
> Jc
>
>
> On Tue, Dec 12, 2017 at 4:55 PM, Utkarsh Ayachit
> <[hidden email]> wrote:
>>
>> On Tue, Dec 12, 2017 at 3:18 PM, Ben Boeckel <[hidden email]>
>> wrote:
>> > On Tue, Dec 12, 2017 at 15:06:15 -0500, Prabhu Ramachandran wrote:
>> >> Would `vtkm` be a shorter name rather than `vtkmodules`?
>> >
>> > `vtkm` is a different project already (though I don't know of its status
>> > with Python wrapping outside of the VTK filters which use it).
>>
>> Yea, alas `vtkm` won't be a good option.
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Search the list archives at: http://markmail.org/search/?q=vtk-developers
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/vtk-developers
>>
>
>
>
> --
> +1 919 869 8849
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Jean-Christophe Fillion-Robin
Hi Utkarsh,

Good point.

Disabling import using vtkConfig is not a good idea for the reason you mentioned.

That said, I still like the idea of implementing lazy loading by default and having vtkConfig to disable it.

> currently be easily provided by factory or other singletons in VTK library itself that control the default render window created

Good point.

Would it roughly work like this:

  from vtk import vtkSomeFactory
  vtkSomeFactory.SetDefaultClass("vtkNameOfSomeClass")

or like this:

  from vtkmodules import vtkSomeFactory
  vtkSomeFactory.SetDefaultClass("vtkNameOfSomeClass")

  import vtk


Based on the proposed implementation for vtk.py [1], looks like the second case is correct.




On Fri, Dec 15, 2017 at 8:55 AM, Utkarsh Ayachit <[hidden email]> wrote:
JC,

I am not a huge fan of this option since the `vtk`package will not
provide a consistent interface based on how the package was imported.
It will make it impossible to write scripts that use `import vtk` and
work reliably. e..g say I have a module (outside VTK) that relies on
`import vtk` to have all necessary symbols. Now my script will fail,
if some other module that the user imported before importing my module
changed the nature of the `vtk` package. Thus, the only way to write
reliable scripts is to import internal modules as needed, and that
defeats the purpose.

The future config you envision could currently be easily provided by
factory or other singletons in VTK library itself that control the
default render window created, for example, and should indeed be
supported there since it's useful for C++ users of VTK too.

Utkarsh

On Fri, Dec 15, 2017 at 12:04 AM, Jean-Christophe Fillion-Robin
<[hidden email]> wrote:
> Hi Utkarsh,
>
> I would prefer to have a more generic way of configuring the import
> behavior.
>
> The idea would be to have two top-level packages:  vtk and vtkConfig
>
> To skip  loading of all module, you would do:
>
> -------------------
> import vtkConfig
> vtkConfig.VTK_MODULE_IMPORT = vtkConfig.EXPLICIT   #  Default would be
> vtkConfig.ALL.  Later we could also have vtkConfig.LAZY
>
> import vtk
> -------------------
>
> This would allow to keep things coherent for our user. Naming the package
> "vtkmodules" seem unconventional.
>
> It would also have the advantage of support addition of new option.
>
> For example:
>
> vtkConfig.RENDERING_METHOD = vtkConfig.OFFSCREEN  #  or vtkConfig.MESA, ....
>
> Jc
>
>
> On Tue, Dec 12, 2017 at 4:55 PM, Utkarsh Ayachit
> <[hidden email]> wrote:
>>
>> On Tue, Dec 12, 2017 at 3:18 PM, Ben Boeckel <[hidden email]>
>> wrote:
>> > On Tue, Dec 12, 2017 at 15:06:15 -0500, Prabhu Ramachandran wrote:
>> >> Would `vtkm` be a shorter name rather than `vtkmodules`?
>> >
>> > `vtkm` is a different project already (though I don't know of its status
>> > with Python wrapping outside of the VTK filters which use it).
>>
>> Yea, alas `vtkm` won't be a good option.
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Search the list archives at: http://markmail.org/search/?q=vtk-developers
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/vtk-developers
>>
>
>
>
> --
> <a href="tel:%2B1%20919%20869%208849" value="+19198698849">+1 919 869 8849



--
+1 919 869 8849

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Ben Boeckel
On Fri, Dec 15, 2017 at 11:30:27 -0500, Jean-Christophe Fillion-Robin wrote:
> That said, I still like the idea of implementing lazy loading by default
> and having vtkConfig to disable it.

That whole lazy loading thing looks finicky to me. Personally, I'd
rather just deprecate the old behavior of bringing everything into
scope on import, but `vtkmodules` is a good middleground.

>   from vtkmodules import vtkSomeFactory
>   vtkSomeFactory.SetDefaultClass("vtkNameOfSomeClass")

This would be `from vtkmodules.vtkSomeModule import vtkSomeFactory`. The
point of vtkmodules is that the top-level package *doesn't* import
everything.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up `import vtk`

Utkarsh Ayachit
>>   from vtkmodules import vtkSomeFactory
>>   vtkSomeFactory.SetDefaultClass("vtkNameOfSomeClass")
>
> This would be `from vtkmodules.vtkSomeModule import vtkSomeFactory`. The
> point of vtkmodules is that the top-level package *doesn't* import
> everything.

You can also:
> from vtk import vtkSomeFactory
>  vtkSomeFactory.SetDefaultClass("vtkNameOfSomeClass")

Utkarsh
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

12