More zero-copy array support for Python

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

More zero-copy array support for Python

David Gobbi
Hi All,

Yesterday I merged the wrapping of the vtkSOADataArrays, and also added a new wrapper hint to assist with VTK's zero-copy methods.

Zero-copy is when an array uses an existing block of memory.  In the Python wrappers, this generally means making a VTK array from a numpy array or vice-versa.  Or making a VTK image from a PIL image.  Or using memmap to share a VTK data set among multiple processes.

The new wrapper hint VTK_ZEROCOPY can be applied to method parameters that are used for zero-copy operations:

  void vtkFloatArray::SetArray(VTK_ZEROCOPY float *ptr, vtkIdType n, int save);

This hint allows the Python wrappers to do the zero-copy magic with SetArray:

  import numpy
  import vtk
  c = numpy.array([1,2,3], 'f')
  a = vtk.vtkFloatArray()
  a.SetArray(c, c.size, True)
  a.array = c  # keep reference to "c"

Astute VTKers will note that this was already possible with SetVoidArray(), as used by the vtk.numpy_support module, but SetArray() adds type checking.

Also, SetArray() works with the newly-wrapped vtkSOADataArrays, so perhaps in the future numpy_support.numpy_to_vtk() can be expanded to support SOA arrays:

  c0 = numpy.array([1,2,3], 'f')
  c1 = numpy.array([4,5,6], 'f')
  a = vtk.vtkSOADataArrayTemplate[c0.dtype]()
  a.SetNumberOfComponents(2)
  a.SetArray(0, c0, c0.size, True, True)
  a.SetArray(1, c1, c1.size, True, True)

In the future this could be done with  a = vtk.numpy_support.numpy_to_vtk( (c0,c1) ).

I haven't yet implemented the mechanism to go in the other direction, i.e. to create a numpy array from a VTK SOA array.

Also: vtkSOADataArrayTemplate is now wrapped, but vtkAOSDataArrayTemplate is not.  Currently the wrappers think that vtkDataArray is the immediate superclass of vtkFloatArray etc.  This could be changed, if there are use cases for directly instantiating vtkAOSDataArrayTemplate.

Cheers,
 - David

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: More zero-copy array support for Python

David Lonie-2
On Fri, Jun 16, 2017 at 8:37 AM, David Gobbi <[hidden email]> wrote:
> Hi All,
>
> Yesterday I merged the wrapping of the vtkSOADataArrays, and also added a
> new wrapper hint to assist with VTK's zero-copy methods.

This is really cool David, thanks for adding this!

> Also: vtkSOADataArrayTemplate is now wrapped, but vtkAOSDataArrayTemplate is
> not.  Currently the wrappers think that vtkDataArray is the immediate
> superclass of vtkFloatArray etc.  This could be changed, if there are use
> cases for directly instantiating vtkAOSDataArrayTemplate.

It's probably time to remove the workarounds that trick the wrappers
into thinking DataArray is the superclass of the AOSArray subclasses.
That's mainly a leftover from when the wrappers couldn't handle
templates. I suppose if it's working for SOA arrays, it should work
for AOS, too.

Dave
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: More zero-copy array support for Python

Berk Geveci-2
In reply to this post by David Gobbi
This is fantastic David!!!

One word of caution to people using numpy array -> VTK array route:

c = numpy.array([1,2,3], 'f')
a = vtk.vtkFloatArray()
a.SetArray(c, c.size, True)
a.array = c  # keep reference to "c"

This is safe only as long as you keep the a object around. If you do something like this, you are in trouble:

pd = vtk.vtkPointData()
pd.AddArray(a)
del a

In this case, the Python object goes away and therefore releases the reference to c. In the dataset_adapter module, I implemented a way around this issue. I suggest taking a look at that which is in the numpyTovtkDataArray() method.

Best,
-berk



On Fri, Jun 16, 2017 at 8:37 AM, David Gobbi <[hidden email]> wrote:
Hi All,

Yesterday I merged the wrapping of the vtkSOADataArrays, and also added a new wrapper hint to assist with VTK's zero-copy methods.

Zero-copy is when an array uses an existing block of memory.  In the Python wrappers, this generally means making a VTK array from a numpy array or vice-versa.  Or making a VTK image from a PIL image.  Or using memmap to share a VTK data set among multiple processes.

The new wrapper hint VTK_ZEROCOPY can be applied to method parameters that are used for zero-copy operations:

  void vtkFloatArray::SetArray(VTK_ZEROCOPY float *ptr, vtkIdType n, int save);

This hint allows the Python wrappers to do the zero-copy magic with SetArray:

  import numpy
  import vtk
  c = numpy.array([1,2,3], 'f')
  a = vtk.vtkFloatArray()
  a.SetArray(c, c.size, True)
  a.array = c  # keep reference to "c"

Astute VTKers will note that this was already possible with SetVoidArray(), as used by the vtk.numpy_support module, but SetArray() adds type checking.

Also, SetArray() works with the newly-wrapped vtkSOADataArrays, so perhaps in the future numpy_support.numpy_to_vtk() can be expanded to support SOA arrays:

  c0 = numpy.array([1,2,3], 'f')
  c1 = numpy.array([4,5,6], 'f')
  a = vtk.vtkSOADataArrayTemplate[c0.dtype]()
  a.SetNumberOfComponents(2)
  a.SetArray(0, c0, c0.size, True, True)
  a.SetArray(1, c1, c1.size, True, True)

In the future this could be done with  a = vtk.numpy_support.numpy_to_vtk( (c0,c1) ).

I haven't yet implemented the mechanism to go in the other direction, i.e. to create a numpy array from a VTK SOA array.

Also: vtkSOADataArrayTemplate is now wrapped, but vtkAOSDataArrayTemplate is not.  Currently the wrappers think that vtkDataArray is the immediate superclass of vtkFloatArray etc.  This could be changed, if there are use cases for directly instantiating vtkAOSDataArrayTemplate.

Cheers,
 - David

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers




_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: More zero-copy array support for Python

David Gobbi
Hi Berk,

That shouldn't be the case.  When a vtk-python object destructs, its attributes aren't deleted until a future GC cycle determines that the vtk-c++ object has destructed.  E.g.:

c = numpy.array([1,2,3], 'f')
a = vtk.vtkFloatArray()
a.SetArray(c, c.size, True)
a.array = c

pd = vtk.vtkPointData()
pd.AddArray(a)
del a
del c

a = pd.GetArray(0)
a.array
-> array([ 1.,  2.,  3.], dtype=float32)

This behavior is tested in Common/Core/Testing/Python/TestGhost.py.  But I notice that numpy_to_vtk() doesn't use this trick...

There could also be other isues that I'm not aware of, so I can look at this in more depth.

 - David 



On Fri, Jun 16, 2017 at 11:18 AM, Berk Geveci <[hidden email]> wrote:
This is fantastic David!!!

One word of caution to people using numpy array -> VTK array route:

c = numpy.array([1,2,3], 'f')
a = vtk.vtkFloatArray()
a.SetArray(c, c.size, True)
a.array = c  # keep reference to "c"

This is safe only as long as you keep the a object around. If you do something like this, you are in trouble:

pd = vtk.vtkPointData()
pd.AddArray(a)
del a

In this case, the Python object goes away and therefore releases the reference to c. In the dataset_adapter module, I implemented a way around this issue. I suggest taking a look at that which is in the numpyTovtkDataArray() method.

Best,
-berk



On Fri, Jun 16, 2017 at 8:37 AM, David Gobbi <[hidden email]> wrote:
Hi All,

Yesterday I merged the wrapping of the vtkSOADataArrays, and also added a new wrapper hint to assist with VTK's zero-copy methods.

Zero-copy is when an array uses an existing block of memory.  In the Python wrappers, this generally means making a VTK array from a numpy array or vice-versa.  Or making a VTK image from a PIL image.  Or using memmap to share a VTK data set among multiple processes.

The new wrapper hint VTK_ZEROCOPY can be applied to method parameters that are used for zero-copy operations:

  void vtkFloatArray::SetArray(VTK_ZEROCOPY float *ptr, vtkIdType n, int save);

This hint allows the Python wrappers to do the zero-copy magic with SetArray:

  import numpy
  import vtk
  c = numpy.array([1,2,3], 'f')
  a = vtk.vtkFloatArray()
  a.SetArray(c, c.size, True)
  a.array = c  # keep reference to "c"

Astute VTKers will note that this was already possible with SetVoidArray(), as used by the vtk.numpy_support module, but SetArray() adds type checking.

Also, SetArray() works with the newly-wrapped vtkSOADataArrays, so perhaps in the future numpy_support.numpy_to_vtk() can be expanded to support SOA arrays:

  c0 = numpy.array([1,2,3], 'f')
  c1 = numpy.array([4,5,6], 'f')
  a = vtk.vtkSOADataArrayTemplate[c0.dtype]()
  a.SetNumberOfComponents(2)
  a.SetArray(0, c0, c0.size, True, True)
  a.SetArray(1, c1, c1.size, True, True)

In the future this could be done with  a = vtk.numpy_support.numpy_to_vtk( (c0,c1) ).

I haven't yet implemented the mechanism to go in the other direction, i.e. to create a numpy array from a VTK SOA array.

Also: vtkSOADataArrayTemplate is now wrapped, but vtkAOSDataArrayTemplate is not.  Currently the wrappers think that vtkDataArray is the immediate superclass of vtkFloatArray etc.  This could be changed, if there are use cases for directly instantiating vtkAOSDataArrayTemplate.

Cheers,
 - David



_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: More zero-copy array support for Python

Berk Geveci-2
Interesting. Is this something relatively recent? I know that this didn't work in the past.

On Fri, Jun 16, 2017 at 3:12 PM, David Gobbi <[hidden email]> wrote:
Hi Berk,

That shouldn't be the case.  When a vtk-python object destructs, its attributes aren't deleted until a future GC cycle determines that the vtk-c++ object has destructed.  E.g.:

c = numpy.array([1,2,3], 'f')
a = vtk.vtkFloatArray()
a.SetArray(c, c.size, True)
a.array = c

pd = vtk.vtkPointData()
pd.AddArray(a)
del a
del c

a = pd.GetArray(0)
a.array
-> array([ 1.,  2.,  3.], dtype=float32)

This behavior is tested in Common/Core/Testing/Python/TestGhost.py.  But I notice that numpy_to_vtk() doesn't use this trick...

There could also be other isues that I'm not aware of, so I can look at this in more depth.

 - David 



On Fri, Jun 16, 2017 at 11:18 AM, Berk Geveci <[hidden email]> wrote:
This is fantastic David!!!

One word of caution to people using numpy array -> VTK array route:

c = numpy.array([1,2,3], 'f')
a = vtk.vtkFloatArray()
a.SetArray(c, c.size, True)
a.array = c  # keep reference to "c"

This is safe only as long as you keep the a object around. If you do something like this, you are in trouble:

pd = vtk.vtkPointData()
pd.AddArray(a)
del a

In this case, the Python object goes away and therefore releases the reference to c. In the dataset_adapter module, I implemented a way around this issue. I suggest taking a look at that which is in the numpyTovtkDataArray() method.

Best,
-berk



On Fri, Jun 16, 2017 at 8:37 AM, David Gobbi <[hidden email]> wrote:
Hi All,

Yesterday I merged the wrapping of the vtkSOADataArrays, and also added a new wrapper hint to assist with VTK's zero-copy methods.

Zero-copy is when an array uses an existing block of memory.  In the Python wrappers, this generally means making a VTK array from a numpy array or vice-versa.  Or making a VTK image from a PIL image.  Or using memmap to share a VTK data set among multiple processes.

The new wrapper hint VTK_ZEROCOPY can be applied to method parameters that are used for zero-copy operations:

  void vtkFloatArray::SetArray(VTK_ZEROCOPY float *ptr, vtkIdType n, int save);

This hint allows the Python wrappers to do the zero-copy magic with SetArray:

  import numpy
  import vtk
  c = numpy.array([1,2,3], 'f')
  a = vtk.vtkFloatArray()
  a.SetArray(c, c.size, True)
  a.array = c  # keep reference to "c"

Astute VTKers will note that this was already possible with SetVoidArray(), as used by the vtk.numpy_support module, but SetArray() adds type checking.

Also, SetArray() works with the newly-wrapped vtkSOADataArrays, so perhaps in the future numpy_support.numpy_to_vtk() can be expanded to support SOA arrays:

  c0 = numpy.array([1,2,3], 'f')
  c1 = numpy.array([4,5,6], 'f')
  a = vtk.vtkSOADataArrayTemplate[c0.dtype]()
  a.SetNumberOfComponents(2)
  a.SetArray(0, c0, c0.size, True, True)
  a.SetArray(1, c1, c1.size, True, True)

In the future this could be done with  a = vtk.numpy_support.numpy_to_vtk( (c0,c1) ).

I haven't yet implemented the mechanism to go in the other direction, i.e. to create a numpy array from a VTK SOA array.

Also: vtkSOADataArrayTemplate is now wrapped, but vtkAOSDataArrayTemplate is not.  Currently the wrappers think that vtkDataArray is the immediate superclass of vtkFloatArray etc.  This could be changed, if there are use cases for directly instantiating vtkAOSDataArrayTemplate.

Cheers,
 - David




_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: More zero-copy array support for Python

David Gobbi
It's a feature that was added in Oct 2010, so it probably pre-dates the dataset_adapter.



On Fri, Jun 16, 2017 at 1:53 PM, Berk Geveci <[hidden email]> wrote:
Interesting. Is this something relatively recent? I know that this didn't work in the past.

On Fri, Jun 16, 2017 at 3:12 PM, David Gobbi <[hidden email]> wrote:
Hi Berk,

That shouldn't be the case.  When a vtk-python object destructs, its attributes aren't deleted until a future GC cycle determines that the vtk-c++ object has destructed.  E.g.:

c = numpy.array([1,2,3], 'f')
a = vtk.vtkFloatArray()
a.SetArray(c, c.size, True)
a.array = c

pd = vtk.vtkPointData()
pd.AddArray(a)
del a
del c

a = pd.GetArray(0)
a.array
-> array([ 1.,  2.,  3.], dtype=float32)

This behavior is tested in Common/Core/Testing/Python/TestGhost.py.  But I notice that numpy_to_vtk() doesn't use this trick...

There could also be other isues that I'm not aware of, so I can look at this in more depth.

 - David 



On Fri, Jun 16, 2017 at 11:18 AM, Berk Geveci <[hidden email]> wrote:
This is fantastic David!!!

One word of caution to people using numpy array -> VTK array route:

c = numpy.array([1,2,3], 'f')
a = vtk.vtkFloatArray()
a.SetArray(c, c.size, True)
a.array = c  # keep reference to "c"

This is safe only as long as you keep the a object around. If you do something like this, you are in trouble:

pd = vtk.vtkPointData()
pd.AddArray(a)
del a

In this case, the Python object goes away and therefore releases the reference to c. In the dataset_adapter module, I implemented a way around this issue. I suggest taking a look at that which is in the numpyTovtkDataArray() method.

Best,
-berk



On Fri, Jun 16, 2017 at 8:37 AM, David Gobbi <[hidden email]> wrote:
Hi All,

Yesterday I merged the wrapping of the vtkSOADataArrays, and also added a new wrapper hint to assist with VTK's zero-copy methods.

Zero-copy is when an array uses an existing block of memory.  In the Python wrappers, this generally means making a VTK array from a numpy array or vice-versa.  Or making a VTK image from a PIL image.  Or using memmap to share a VTK data set among multiple processes.

The new wrapper hint VTK_ZEROCOPY can be applied to method parameters that are used for zero-copy operations:

  void vtkFloatArray::SetArray(VTK_ZEROCOPY float *ptr, vtkIdType n, int save);

This hint allows the Python wrappers to do the zero-copy magic with SetArray:

  import numpy
  import vtk
  c = numpy.array([1,2,3], 'f')
  a = vtk.vtkFloatArray()
  a.SetArray(c, c.size, True)
  a.array = c  # keep reference to "c"

Astute VTKers will note that this was already possible with SetVoidArray(), as used by the vtk.numpy_support module, but SetArray() adds type checking.

Also, SetArray() works with the newly-wrapped vtkSOADataArrays, so perhaps in the future numpy_support.numpy_to_vtk() can be expanded to support SOA arrays:

  c0 = numpy.array([1,2,3], 'f')
  c1 = numpy.array([4,5,6], 'f')
  a = vtk.vtkSOADataArrayTemplate[c0.dtype]()
  a.SetNumberOfComponents(2)
  a.SetArray(0, c0, c0.size, True, True)
  a.SetArray(1, c1, c1.size, True, True)

In the future this could be done with  a = vtk.numpy_support.numpy_to_vtk( (c0,c1) ).

I haven't yet implemented the mechanism to go in the other direction, i.e. to create a numpy array from a VTK SOA array.

Also: vtkSOADataArrayTemplate is now wrapped, but vtkAOSDataArrayTemplate is not.  Currently the wrappers think that vtkDataArray is the immediate superclass of vtkFloatArray etc.  This could be changed, if there are use cases for directly instantiating vtkAOSDataArrayTemplate.

Cheers,
 - David





_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/vtk-developers

Loading...