Added support for std::vector to the wrappers

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Added support for std::vector to the wrappers

David Gobbi
Hi Folks,

Yesterday I merged a patch that allows std::vector<T> to be used from the Python wrappers, where T can be 'std::string' or any numeric type from 'signed char' to 'float' (including typedefs to those types).

This means that if you add a C++ method to VTK that takes std::vector<std::string>, then in Python you can pass a list or any other Python sequence and the wrappers will convert it to a vector for you.

Method examples that can accept any Python sequence:
    void SetStrings(std::vector<std::string> strings);
    void SetStrings(const std::vector<std::string>& strings);

Method example that takes a Python mutable sequence (the C++ method can write back to the container, since this is pass-by-reference):
    void GetStrings(std::vector<std::string>& strings);

Method examples that return a vector (which is converted to a Python tuple):
    const std::vector<std::string>& GetStrings();
    std::vector<std::string>& GetStrings();
    std::vector<std::string> GetStrings();

Note that std::vector is not 'wrapped' per se.  Instead, what is going on here is that the wrappers are converting to/from Python container types. My primary goal was convenience rather than efficiency, we can always add a wrapped std::vector type at a later date without sacrificing the convenience of automatic conversion.

Chances are good that I'll be able to add std::vectors of VTK objects (smart or dumb pointers) and vtkVariant before VTK 9, but only if someone adds methods to VTK that take parameters of those types (as far as I know, there aren't any yet).

See the following VTK gitlab issue for additional information:

The following is a list of VTK methods that take std::vector that have been wrapped due to this change (it is a short list, since people have historically avoided using std::vector as part of the VTK public API due to the obvious reason that they didn't want to exclude their methods from wrapping):

vtkActor::ProcessSelectorPixelBuffers
vtkAMRInformation::GetNumBlocks
vtkCompositePolyDataMapper2::ProcessSelectorPixelBuffers
vtkMapper::ProcessSelectorPixelBuffers
vtkMultiProcessStream::GetRawData
vtkMultiProcessStream::SetRawData
vtkOpenGLGlyph3DHelper::GlyphRender
vtkOpenGLIndexBufferObject::AppendTriangleIndexBuffer
vtkOpenGLIndexBufferObject::AppendLineIndexBuffer
vtkOpenGLIndexBufferObject::AppendTriangleLineIndexBuffer
vtkOpenGLIndexBufferObject::AppendPointIndexBuffer
vtkOpenGLIndexBufferObject::AppendStripIndexBuffer
vtkOpenGLIndexBufferObject::AppendEdgeFlagIndexBuffer
vtkOpenGLMoleculeMapper::ProcessSelectorPixelBuffers
vtkOpenGLPointGaussianMapper::ProcessSelectorPixelBuffers
vtkOpenGLPolyDataMapper::HandleAppleBug
vtkOpenGLPolyDataMapper::ProcessSelectorPixelBuffers
vtkOpenGLVertexBufferObject::SetShift
vtkOpenGLVertexBufferObject::SetScale
vtkOpenGLVertexBufferObject::GetShift
vtkOpenGLVertexBufferObject::GetScale
vtkOpenGLVertexBufferObject::GetPackedVBO
vtkParallelAMRUtilities::DistributeProcessInformation
vtkProp::ProcessSelectorPixelBuffers
vtkResourceFileLocator::Locate
vtkShadowMapPass::ShadowMapTransforms
vtkShadowMapPass::GetShadowMapTextureUnits
vtkSparseArray::GetUniqueCoordinates
vtkUnicodeString::utf16_str

Cheers,
 - David

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Added support for std::vector to the wrappers

Sebastien Jourdain via vtk-developers
Hi David

Since std::vector types are unlikely to be small, like coordinate and bound arrays, that sounds horribly inefficient; especially when large quantities of strings and objects are being marshalled.

Why not just map the std::vector/std::array to a Python iterator object?

Todd

On 6 Sep 2018 4:20 a.m., David Gobbi <[hidden email]> wrote:
Hi Folks,

Yesterday I merged a patch that allows std::vector<T> to be used from the Python wrappers, where T can be 'std::string' or any numeric type from 'signed char' to 'float' (including typedefs to those types).

This means that if you add a C++ method to VTK that takes std::vector<std::string>, then in Python you can pass a list or any other Python sequence and the wrappers will convert it to a vector for you.

Method examples that can accept any Python sequence:
    void SetStrings(std::vector<std::string> strings);
    void SetStrings(const std::vector<std::string>& strings);

Method example that takes a Python mutable sequence (the C++ method can write back to the container, since this is pass-by-reference):
    void GetStrings(std::vector<std::string>& strings);

Method examples that return a vector (which is converted to a Python tuple):
    const std::vector<std::string>& GetStrings();
    std::vector<std::string>& GetStrings();
    std::vector<std::string> GetStrings();

Note that std::vector is not 'wrapped' per se.  Instead, what is going on here is that the wrappers are converting to/from Python container types. My primary goal was convenience rather than efficiency, we can always add a wrapped std::vector type at a later date without sacrificing the convenience of automatic conversion.

Chances are good that I'll be able to add std::vectors of VTK objects (smart or dumb pointers) and vtkVariant before VTK 9, but only if someone adds methods to VTK that take parameters of those types (as far as I know, there aren't any yet).

See the following VTK gitlab issue for additional information:

The following is a list of VTK methods that take std::vector that have been wrapped due to this change (it is a short list, since people have historically avoided using std::vector as part of the VTK public API due to the obvious reason that they didn't want to exclude their methods from wrapping):

vtkActor::ProcessSelectorPixelBuffers
vtkAMRInformation::GetNumBlocks
vtkCompositePolyDataMapper2::ProcessSelectorPixelBuffers
vtkMapper::ProcessSelectorPixelBuffers
vtkMultiProcessStream::GetRawData
vtkMultiProcessStream::SetRawData
vtkOpenGLGlyph3DHelper::GlyphRender
vtkOpenGLIndexBufferObject::AppendTriangleIndexBuffer
vtkOpenGLIndexBufferObject::AppendLineIndexBuffer
vtkOpenGLIndexBufferObject::AppendTriangleLineIndexBuffer
vtkOpenGLIndexBufferObject::AppendPointIndexBuffer
vtkOpenGLIndexBufferObject::AppendStripIndexBuffer
vtkOpenGLIndexBufferObject::AppendEdgeFlagIndexBuffer
vtkOpenGLMoleculeMapper::ProcessSelectorPixelBuffers
vtkOpenGLPointGaussianMapper::ProcessSelectorPixelBuffers
vtkOpenGLPolyDataMapper::HandleAppleBug
vtkOpenGLPolyDataMapper::ProcessSelectorPixelBuffers
vtkOpenGLVertexBufferObject::SetShift
vtkOpenGLVertexBufferObject::SetScale
vtkOpenGLVertexBufferObject::GetShift
vtkOpenGLVertexBufferObject::GetScale
vtkOpenGLVertexBufferObject::GetPackedVBO
vtkParallelAMRUtilities::DistributeProcessInformation
vtkProp::ProcessSelectorPixelBuffers
vtkResourceFileLocator::Locate
vtkShadowMapPass::ShadowMapTransforms
vtkShadowMapPass::GetShadowMapTextureUnits
vtkSparseArray::GetUniqueCoordinates
vtkUnicodeString::utf16_str

Cheers,
 - David


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Added support for std::vector to the wrappers

Sebastien Jourdain via vtk-developers
I should add that I think the idea of VTK exposing std::array/std::vector types in the public API is terrible. A much better solution would be to expose std::iterator types and move add()/remove() methods to the containing object as a gateway for notification to the rest of the library/application.

Todd

On 6 Sep 2018 12:16 p.m., Todd <[hidden email]> wrote:
Hi David

Since std::vector types are unlikely to be small, like coordinate and bound arrays, that sounds horribly inefficient; especially when large quantities of strings and objects are being marshalled.

Why not just map the std::vector/std::array to a Python iterator object?

Todd

On 6 Sep 2018 4:20 a.m., David Gobbi <david.gobbi@gmail.com> wrote:
Hi Folks,

Yesterday I merged a patch that allows std::vector<T> to be used from the Python wrappers, where T can be 'std::string' or any numeric type from 'signed char' to 'float' (including typedefs to those types).

This means that if you add a C++ method to VTK that takes std::vector<std::string>, then in Python you can pass a list or any other Python sequence and the wrappers will convert it to a vector for you.

Method examples that can accept any Python sequence:
    void SetStrings(std::vector<std::string> strings);
    void SetStrings(const std::vector<std::string>& strings);

Method example that takes a Python mutable sequence (the C++ method can write back to the container, since this is pass-by-reference):
    void GetStrings(std::vector<std::string>& strings);

Method examples that return a vector (which is converted to a Python tuple):
    const std::vector<std::string>& GetStrings();
    std::vector<std::string>& GetStrings();
    std::vector<std::string> GetStrings();

Note that std::vector is not 'wrapped' per se.  Instead, what is going on here is that the wrappers are converting to/from Python container types. My primary goal was convenience rather than efficiency, we can always add a wrapped std::vector type at a later date without sacrificing the convenience of automatic conversion.

Chances are good that I'll be able to add std::vectors of VTK objects (smart or dumb pointers) and vtkVariant before VTK 9, but only if someone adds methods to VTK that take parameters of those types (as far as I know, there aren't any yet).

See the following VTK gitlab issue for additional information:

The following is a list of VTK methods that take std::vector that have been wrapped due to this change (it is a short list, since people have historically avoided using std::vector as part of the VTK public API due to the obvious reason that they didn't want to exclude their methods from wrapping):

vtkActor::ProcessSelectorPixelBuffers
vtkAMRInformation::GetNumBlocks
vtkCompositePolyDataMapper2::ProcessSelectorPixelBuffers
vtkMapper::ProcessSelectorPixelBuffers
vtkMultiProcessStream::GetRawData
vtkMultiProcessStream::SetRawData
vtkOpenGLGlyph3DHelper::GlyphRender
vtkOpenGLIndexBufferObject::AppendTriangleIndexBuffer
vtkOpenGLIndexBufferObject::AppendLineIndexBuffer
vtkOpenGLIndexBufferObject::AppendTriangleLineIndexBuffer
vtkOpenGLIndexBufferObject::AppendPointIndexBuffer
vtkOpenGLIndexBufferObject::AppendStripIndexBuffer
vtkOpenGLIndexBufferObject::AppendEdgeFlagIndexBuffer
vtkOpenGLMoleculeMapper::ProcessSelectorPixelBuffers
vtkOpenGLPointGaussianMapper::ProcessSelectorPixelBuffers
vtkOpenGLPolyDataMapper::HandleAppleBug
vtkOpenGLPolyDataMapper::ProcessSelectorPixelBuffers
vtkOpenGLVertexBufferObject::SetShift
vtkOpenGLVertexBufferObject::SetScale
vtkOpenGLVertexBufferObject::GetShift
vtkOpenGLVertexBufferObject::GetScale
vtkOpenGLVertexBufferObject::GetPackedVBO
vtkParallelAMRUtilities::DistributeProcessInformation
vtkProp::ProcessSelectorPixelBuffers
vtkResourceFileLocator::Locate
vtkShadowMapPass::ShadowMapTransforms
vtkShadowMapPass::GetShadowMapTextureUnits
vtkSparseArray::GetUniqueCoordinates
vtkUnicodeString::utf16_str

Cheers,
 - David



_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Added support for std::vector to the wrappers

Andras Lasso
In reply to this post by Sebastien Jourdain via vtk-developers

I find this feature very useful! In application code, we commonly need to deal with variable-length arrays. We have been using std::vector accessors in C++ and adding vtkStringArray based or GetNumberOf…/GetNth… methods or for Python, which has made the API complex and has been very inconvenient to use in Python.

 

These arrays typically contain up to a few ten elements, therefore, for us, convenience is much more important than efficiency.

Most often we return strings and vtkObject* in std::vector, so it would be great if vtkObject pointers would be supported, too.

 

Andras

 

 

From: vtk-developers <[hidden email]> On Behalf Of Todd via vtk-developers
Sent: Wednesday, September 5, 2018 8:16 PM
To: David Gobbi <[hidden email]>
Cc: VTK Developers <[hidden email]>
Subject: Re: [vtk-developers] Added support for std::vector to the wrappers

 

Hi David

 

Since std::vector types are unlikely to be small, like coordinate and bound arrays, that sounds horribly inefficient; especially when large quantities of strings and objects are being marshalled.

 

Why not just map the std::vector/std::array to a Python iterator object?

 

Todd

 

On 6 Sep 2018 4:20 a.m., David Gobbi <[hidden email]> wrote:

Hi Folks,

 

Yesterday I merged a patch that allows std::vector<T> to be used from the Python wrappers, where T can be 'std::string' or any numeric type from 'signed char' to 'float' (including typedefs to those types).

 

This means that if you add a C++ method to VTK that takes std::vector<std::string>, then in Python you can pass a list or any other Python sequence and the wrappers will convert it to a vector for you.

 

Method examples that can accept any Python sequence:

    void SetStrings(std::vector<std::string> strings);

    void SetStrings(const std::vector<std::string>& strings);

 

Method example that takes a Python mutable sequence (the C++ method can write back to the container, since this is pass-by-reference):

    void GetStrings(std::vector<std::string>& strings);

 

Method examples that return a vector (which is converted to a Python tuple):

    const std::vector<std::string>& GetStrings();

    std::vector<std::string>& GetStrings();

    std::vector<std::string> GetStrings();

 

Note that std::vector is not 'wrapped' per se.  Instead, what is going on here is that the wrappers are converting to/from Python container types. My primary goal was convenience rather than efficiency, we can always add a wrapped std::vector type at a later date without sacrificing the convenience of automatic conversion.

 

Chances are good that I'll be able to add std::vectors of VTK objects (smart or dumb pointers) and vtkVariant before VTK 9, but only if someone adds methods to VTK that take parameters of those types (as far as I know, there aren't any yet).

 

See the following VTK gitlab issue for additional information:

 

The following is a list of VTK methods that take std::vector that have been wrapped due to this change (it is a short list, since people have historically avoided using std::vector as part of the VTK public API due to the obvious reason that they didn't want to exclude their methods from wrapping):

 

vtkActor::ProcessSelectorPixelBuffers

vtkAMRInformation::GetNumBlocks

vtkCompositePolyDataMapper2::ProcessSelectorPixelBuffers

vtkMapper::ProcessSelectorPixelBuffers

vtkMultiProcessStream::GetRawData

vtkMultiProcessStream::SetRawData

vtkOpenGLGlyph3DHelper::GlyphRender

vtkOpenGLIndexBufferObject::AppendTriangleIndexBuffer

vtkOpenGLIndexBufferObject::AppendLineIndexBuffer

vtkOpenGLIndexBufferObject::AppendTriangleLineIndexBuffer

vtkOpenGLIndexBufferObject::AppendPointIndexBuffer

vtkOpenGLIndexBufferObject::AppendStripIndexBuffer

vtkOpenGLIndexBufferObject::AppendEdgeFlagIndexBuffer

vtkOpenGLMoleculeMapper::ProcessSelectorPixelBuffers

vtkOpenGLPointGaussianMapper::ProcessSelectorPixelBuffers

vtkOpenGLPolyDataMapper::HandleAppleBug

vtkOpenGLPolyDataMapper::ProcessSelectorPixelBuffers

vtkOpenGLVertexBufferObject::SetShift

vtkOpenGLVertexBufferObject::SetScale

vtkOpenGLVertexBufferObject::GetShift

vtkOpenGLVertexBufferObject::GetScale

vtkOpenGLVertexBufferObject::GetPackedVBO

vtkParallelAMRUtilities::DistributeProcessInformation

vtkProp::ProcessSelectorPixelBuffers

vtkResourceFileLocator::Locate

vtkShadowMapPass::ShadowMapTransforms

vtkShadowMapPass::GetShadowMapTextureUnits

vtkSparseArray::GetUniqueCoordinates

vtkUnicodeString::utf16_str

 

Cheers,

 - David

 


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Added support for std::vector to the wrappers

David Gobbi
Hi Todd,

VTK already supports the Python buffer interface for moving large amounts of data between VTK and Python, people have been using it to shuffle arrays between VTK and numpy for many years.  It's much more efficient than any iterator-based scheme.

The main goal of wrapping std::vector is convenience: people have already been using std::vector for things like string lists, so there was a present demand for this feature.  The reason that I stuck with std::vector rather than going for a generic iterator-based interface is that it made it easy for me to piggy-back on the wrappers' existing support for C arrays.  Very little extra weight was added to the wrappers.

 - David


On Wed, Sep 5, 2018 at 7:20 PM Andras Lasso <[hidden email]> wrote:

I find this feature very useful! In application code, we commonly need to deal with variable-length arrays. We have been using std::vector accessors in C++ and adding vtkStringArray based or GetNumberOf…/GetNth… methods or for Python, which has made the API complex and has been very inconvenient to use in Python.

 

These arrays typically contain up to a few ten elements, therefore, for us, convenience is much more important than efficiency.

Most often we return strings and vtkObject* in std::vector, so it would be great if vtkObject pointers would be supported, too.

 

Andras

 

 

From: vtk-developers <[hidden email]> On Behalf Of Todd via vtk-developers
Sent: Wednesday, September 5, 2018 8:16 PM
To: David Gobbi <[hidden email]>
Cc: VTK Developers <[hidden email]>
Subject: Re: [vtk-developers] Added support for std::vector to the wrappers

 

Hi David

 

Since std::vector types are unlikely to be small, like coordinate and bound arrays, that sounds horribly inefficient; especially when large quantities of strings and objects are being marshalled.

 

Why not just map the std::vector/std::array to a Python iterator object?

 

Todd

 

On 6 Sep 2018 4:20 a.m., David Gobbi <[hidden email]> wrote:

Hi Folks,

 

Yesterday I merged a patch that allows std::vector<T> to be used from the Python wrappers, where T can be 'std::string' or any numeric type from 'signed char' to 'float' (including typedefs to those types).

 

This means that if you add a C++ method to VTK that takes std::vector<std::string>, then in Python you can pass a list or any other Python sequence and the wrappers will convert it to a vector for you.

 

Method examples that can accept any Python sequence:

    void SetStrings(std::vector<std::string> strings);

    void SetStrings(const std::vector<std::string>& strings);

 

Method example that takes a Python mutable sequence (the C++ method can write back to the container, since this is pass-by-reference):

    void GetStrings(std::vector<std::string>& strings);

 

Method examples that return a vector (which is converted to a Python tuple):

    const std::vector<std::string>& GetStrings();

    std::vector<std::string>& GetStrings();

    std::vector<std::string> GetStrings();

 

Note that std::vector is not 'wrapped' per se.  Instead, what is going on here is that the wrappers are converting to/from Python container types. My primary goal was convenience rather than efficiency, we can always add a wrapped std::vector type at a later date without sacrificing the convenience of automatic conversion.

 

Chances are good that I'll be able to add std::vectors of VTK objects (smart or dumb pointers) and vtkVariant before VTK 9, but only if someone adds methods to VTK that take parameters of those types (as far as I know, there aren't any yet).

 

See the following VTK gitlab issue for additional information:

 

The following is a list of VTK methods that take std::vector that have been wrapped due to this change (it is a short list, since people have historically avoided using std::vector as part of the VTK public API due to the obvious reason that they didn't want to exclude their methods from wrapping):

 

vtkActor::ProcessSelectorPixelBuffers

vtkAMRInformation::GetNumBlocks

vtkCompositePolyDataMapper2::ProcessSelectorPixelBuffers

vtkMapper::ProcessSelectorPixelBuffers

vtkMultiProcessStream::GetRawData

vtkMultiProcessStream::SetRawData

vtkOpenGLGlyph3DHelper::GlyphRender

vtkOpenGLIndexBufferObject::AppendTriangleIndexBuffer

vtkOpenGLIndexBufferObject::AppendLineIndexBuffer

vtkOpenGLIndexBufferObject::AppendTriangleLineIndexBuffer

vtkOpenGLIndexBufferObject::AppendPointIndexBuffer

vtkOpenGLIndexBufferObject::AppendStripIndexBuffer

vtkOpenGLIndexBufferObject::AppendEdgeFlagIndexBuffer

vtkOpenGLMoleculeMapper::ProcessSelectorPixelBuffers

vtkOpenGLPointGaussianMapper::ProcessSelectorPixelBuffers

vtkOpenGLPolyDataMapper::HandleAppleBug

vtkOpenGLPolyDataMapper::ProcessSelectorPixelBuffers

vtkOpenGLVertexBufferObject::SetShift

vtkOpenGLVertexBufferObject::SetScale

vtkOpenGLVertexBufferObject::GetShift

vtkOpenGLVertexBufferObject::GetScale

vtkOpenGLVertexBufferObject::GetPackedVBO

vtkParallelAMRUtilities::DistributeProcessInformation

vtkProp::ProcessSelectorPixelBuffers

vtkResourceFileLocator::Locate

vtkShadowMapPass::ShadowMapTransforms

vtkShadowMapPass::GetShadowMapTextureUnits

vtkSparseArray::GetUniqueCoordinates

vtkUnicodeString::utf16_str

 

Cheers,

 - David

 


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Added support for std::vector to the wrappers

Sebastien Jourdain via vtk-developers
Hi David

I agree that moving large chunks of data in a buffer is more efficient than iteration, but is that how arrays are marshaled into tuples in VTK? It certainly isn't possible for string/object arrays, since the memory size varies for each item.

I can also see that indexing into a list is more efficient, when a few specific items are needed; however, I don't think a tuple is the right object type for this, if the count is large. If std::array and std::vector are already being used for string lists, I believe that vtkArray and vtkVector types should be introduced that, optionally, expose a method to return a vtkIterator.



Todd Martin, PhD.
Freelance Engineer/Software Architect.



On Thursday, September 6, 2018, 2:29:45 PM GMT+12, David Gobbi <[hidden email]> wrote:


Hi Todd,

VTK already supports the Python buffer interface for moving large amounts of data between VTK and Python, people have been using it to shuffle arrays between VTK and numpy for many years.  It's much more efficient than any iterator-based scheme.

The main goal of wrapping std::vector is convenience: people have already been using std::vector for things like string lists, so there was a present demand for this feature.  The reason that I stuck with std::vector rather than going for a generic iterator-based interface is that it made it easy for me to piggy-back on the wrappers' existing support for C arrays.  Very little extra weight was added to the wrappers.

 - David


On Wed, Sep 5, 2018 at 7:20 PM Andras Lasso <[hidden email]> wrote:

I find this feature very useful! In application code, we commonly need to deal with variable-length arrays. We have been using std::vector accessors in C++ and adding vtkStringArray based or GetNumberOf…/GetNth… methods or for Python, which has made the API complex and has been very inconvenient to use in Python.

 

These arrays typically contain up to a few ten elements, therefore, for us, convenience is much more important than efficiency.

Most often we return strings and vtkObject* in std::vector, so it would be great if vtkObject pointers would be supported, too.

 

Andras

 

 

From: vtk-developers <[hidden email]> On Behalf Of Todd via vtk-developers
Sent: Wednesday, September 5, 2018 8:16 PM
To: David Gobbi <[hidden email]>
Cc: VTK Developers <[hidden email]>
Subject: Re: [vtk-developers] Added support for std::vector to the wrappers

 

Hi David

 

Since std::vector types are unlikely to be small, like coordinate and bound arrays, that sounds horribly inefficient; especially when large quantities of strings and objects are being marshalled.

 

Why not just map the std::vector/std::array to a Python iterator object?

 

Todd

 

On 6 Sep 2018 4:20 a.m., David Gobbi <[hidden email]> wrote:

Hi Folks,

 

Yesterday I merged a patch that allows std::vector<T> to be used from the Python wrappers, where T can be 'std::string' or any numeric type from 'signed char' to 'float' (including typedefs to those types).

 

This means that if you add a C++ method to VTK that takes std::vector<std::string>, then in Python you can pass a list or any other Python sequence and the wrappers will convert it to a vector for you.

 

Method examples that can accept any Python sequence:

    void SetStrings(std::vector<std::string> strings);

    void SetStrings(const std::vector<std::string>& strings);

 

Method example that takes a Python mutable sequence (the C++ method can write back to the container, since this is pass-by-reference):

    void GetStrings(std::vector<std::string>& strings);

 

Method examples that return a vector (which is converted to a Python tuple):

    const std::vector<std::string>& GetStrings();

    std::vector<std::string>& GetStrings();

    std::vector<std::string> GetStrings();

 

Note that std::vector is not 'wrapped' per se.  Instead, what is going on here is that the wrappers are converting to/from Python container types. My primary goal was convenience rather than efficiency, we can always add a wrapped std::vector type at a later date without sacrificing the convenience of automatic conversion.

 

Chances are good that I'll be able to add std::vectors of VTK objects (smart or dumb pointers) and vtkVariant before VTK 9, but only if someone adds methods to VTK that take parameters of those types (as far as I know, there aren't any yet).

 

See the following VTK gitlab issue for additional information:

 

The following is a list of VTK methods that take std::vector that have been wrapped due to this change (it is a short list, since people have historically avoided using std::vector as part of the VTK public API due to the obvious reason that they didn't want to exclude their methods from wrapping):

 

vtkActor::ProcessSelectorPixelBuffers

vtkAMRInformation::GetNumBlocks

vtkCompositePolyDataMapper2::ProcessSelectorPixelBuffers

vtkMapper::ProcessSelectorPixelBuffers

vtkMultiProcessStream::GetRawData

vtkMultiProcessStream::SetRawData

vtkOpenGLGlyph3DHelper::GlyphRender

vtkOpenGLIndexBufferObject::AppendTriangleIndexBuffer

vtkOpenGLIndexBufferObject::AppendLineIndexBuffer

vtkOpenGLIndexBufferObject::AppendTriangleLineIndexBuffer

vtkOpenGLIndexBufferObject::AppendPointIndexBuffer

vtkOpenGLIndexBufferObject::AppendStripIndexBuffer

vtkOpenGLIndexBufferObject::AppendEdgeFlagIndexBuffer

vtkOpenGLMoleculeMapper::ProcessSelectorPixelBuffers

vtkOpenGLPointGaussianMapper::ProcessSelectorPixelBuffers

vtkOpenGLPolyDataMapper::HandleAppleBug

vtkOpenGLPolyDataMapper::ProcessSelectorPixelBuffers

vtkOpenGLVertexBufferObject::SetShift

vtkOpenGLVertexBufferObject::SetScale

vtkOpenGLVertexBufferObject::GetShift

vtkOpenGLVertexBufferObject::GetScale

vtkOpenGLVertexBufferObject::GetPackedVBO

vtkParallelAMRUtilities::DistributeProcessInformation

vtkProp::ProcessSelectorPixelBuffers

vtkResourceFileLocator::Locate

vtkShadowMapPass::ShadowMapTransforms

vtkShadowMapPass::GetShadowMapTextureUnits

vtkSparseArray::GetUniqueCoordinates

vtkUnicodeString::utf16_str

 

Cheers,

 - David

 


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: Added support for std::vector to the wrappers

David Gobbi
Hi Todd,

The Python buffer interface is the opposite of marshalling.  Python objects that support the buffer interface are able to directly access each other's memory.  The contents of a vtkDataArray wrapped in Python can be directly accessed by numpy routines that operate on buffers, i.e. the underlying C/C++/FORTRAN code directly sees the array contents as a raw pointer plus the data type, dimensions, and strides.  So, no, it isn't used for strings arrays or object arrays.  And it isn't supported by tuples since tuples aren't meant as a bulk data container.

I already mentioned in my first post that if people want to be able to return large vtk::vectors, I can add a wrapped std::vector object and have the wrappers return that instead of a tuple (and of course the wrapped vtk::vector would support iteration and indexing in Python).  There already are vtkVector and vtkTuple types for small fixed-size arrays, and parts of VTK's API uses them.  These are wrapped directly in Python rather than converted to tuples.

 - David


On Thu, Sep 6, 2018 at 3:16 AM Todd Martin <[hidden email]> wrote:
Hi David

I agree that moving large chunks of data in a buffer is more efficient than iteration, but is that how arrays are marshaled into tuples in VTK? It certainly isn't possible for string/object arrays, since the memory size varies for each item.

I can also see that indexing into a list is more efficient, when a few specific items are needed; however, I don't think a tuple is the right object type for this, if the count is large. If std::array and std::vector are already being used for string lists, I believe that vtkArray and vtkVector types should be introduced that, optionally, expose a method to return a vtkIterator.



Todd Martin, PhD.
Freelance Engineer/Software Architect.



On Thursday, September 6, 2018, 2:29:45 PM GMT+12, David Gobbi <[hidden email]> wrote:


Hi Todd,

VTK already supports the Python buffer interface for moving large amounts of data between VTK and Python, people have been using it to shuffle arrays between VTK and numpy for many years.  It's much more efficient than any iterator-based scheme.

The main goal of wrapping std::vector is convenience: people have already been using std::vector for things like string lists, so there was a present demand for this feature.  The reason that I stuck with std::vector rather than going for a generic iterator-based interface is that it made it easy for me to piggy-back on the wrappers' existing support for C arrays.  Very little extra weight was added to the wrappers.

 - David


On Wed, Sep 5, 2018 at 7:20 PM Andras Lasso <[hidden email]> wrote:

I find this feature very useful! In application code, we commonly need to deal with variable-length arrays. We have been using std::vector accessors in C++ and adding vtkStringArray based or GetNumberOf…/GetNth… methods or for Python, which has made the API complex and has been very inconvenient to use in Python.

 

These arrays typically contain up to a few ten elements, therefore, for us, convenience is much more important than efficiency.

Most often we return strings and vtkObject* in std::vector, so it would be great if vtkObject pointers would be supported, too.

 

Andras

 

 

From: vtk-developers <[hidden email]> On Behalf Of Todd via vtk-developers
Sent: Wednesday, September 5, 2018 8:16 PM
To: David Gobbi <[hidden email]>
Cc: VTK Developers <[hidden email]>
Subject: Re: [vtk-developers] Added support for std::vector to the wrappers

 

Hi David

 

Since std::vector types are unlikely to be small, like coordinate and bound arrays, that sounds horribly inefficient; especially when large quantities of strings and objects are being marshalled.

 

Why not just map the std::vector/std::array to a Python iterator object?

 

Todd

 

On 6 Sep 2018 4:20 a.m., David Gobbi <[hidden email]> wrote:

Hi Folks,

 

Yesterday I merged a patch that allows std::vector<T> to be used from the Python wrappers, where T can be 'std::string' or any numeric type from 'signed char' to 'float' (including typedefs to those types).

 

This means that if you add a C++ method to VTK that takes std::vector<std::string>, then in Python you can pass a list or any other Python sequence and the wrappers will convert it to a vector for you.

 

Method examples that can accept any Python sequence:

    void SetStrings(std::vector<std::string> strings);

    void SetStrings(const std::vector<std::string>& strings);

 

Method example that takes a Python mutable sequence (the C++ method can write back to the container, since this is pass-by-reference):

    void GetStrings(std::vector<std::string>& strings);

 

Method examples that return a vector (which is converted to a Python tuple):

    const std::vector<std::string>& GetStrings();

    std::vector<std::string>& GetStrings();

    std::vector<std::string> GetStrings();

 

Note that std::vector is not 'wrapped' per se.  Instead, what is going on here is that the wrappers are converting to/from Python container types. My primary goal was convenience rather than efficiency, we can always add a wrapped std::vector type at a later date without sacrificing the convenience of automatic conversion.

 

Chances are good that I'll be able to add std::vectors of VTK objects (smart or dumb pointers) and vtkVariant before VTK 9, but only if someone adds methods to VTK that take parameters of those types (as far as I know, there aren't any yet).

 

See the following VTK gitlab issue for additional information:

 

The following is a list of VTK methods that take std::vector that have been wrapped due to this change (it is a short list, since people have historically avoided using std::vector as part of the VTK public API due to the obvious reason that they didn't want to exclude their methods from wrapping):

 

vtkActor::ProcessSelectorPixelBuffers

vtkAMRInformation::GetNumBlocks

vtkCompositePolyDataMapper2::ProcessSelectorPixelBuffers

vtkMapper::ProcessSelectorPixelBuffers

vtkMultiProcessStream::GetRawData

vtkMultiProcessStream::SetRawData

vtkOpenGLGlyph3DHelper::GlyphRender

vtkOpenGLIndexBufferObject::AppendTriangleIndexBuffer

vtkOpenGLIndexBufferObject::AppendLineIndexBuffer

vtkOpenGLIndexBufferObject::AppendTriangleLineIndexBuffer

vtkOpenGLIndexBufferObject::AppendPointIndexBuffer

vtkOpenGLIndexBufferObject::AppendStripIndexBuffer

vtkOpenGLIndexBufferObject::AppendEdgeFlagIndexBuffer

vtkOpenGLMoleculeMapper::ProcessSelectorPixelBuffers

vtkOpenGLPointGaussianMapper::ProcessSelectorPixelBuffers

vtkOpenGLPolyDataMapper::HandleAppleBug

vtkOpenGLPolyDataMapper::ProcessSelectorPixelBuffers

vtkOpenGLVertexBufferObject::SetShift

vtkOpenGLVertexBufferObject::SetScale

vtkOpenGLVertexBufferObject::GetShift

vtkOpenGLVertexBufferObject::GetScale

vtkOpenGLVertexBufferObject::GetPackedVBO

vtkParallelAMRUtilities::DistributeProcessInformation

vtkProp::ProcessSelectorPixelBuffers

vtkResourceFileLocator::Locate

vtkShadowMapPass::ShadowMapTransforms

vtkShadowMapPass::GetShadowMapTextureUnits

vtkSparseArray::GetUniqueCoordinates

vtkUnicodeString::utf16_str

 

Cheers,

 - David

 


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers