vtk test data for remote module?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

vtk test data for remote module?

Aron Helser
I have a collaborator working with a VTK remote module (MomentInvariants). It so happens that this code was previously part of VTK proper, and was moved to a remote module. It includes some tests, with external test data. We now need to update the test data, and so are trying to change from md5 to sha512, as instructed here: https://gitlab.kitware.com/vtk/vtk/blob/master/Documentation/dev/git/data.md

The cmake and build parts work just as they used to, but unfortunately the git parts, like the pre-commit hook and the gitlab-push command are no longer available in the remote module, since it lives in it's own repository. 

Has anyone dealt with this before? What should we do?

The data is about 2.7 Mb, so one possibility is to just commit the data directly in the remote repository, I suppose, but I'm not sure how ctest will then interact with it.

Thanks!
Aron

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: vtk test data for remote module?

Jean-Christophe Fillion-Robin-2
Hi Aaron,

> the git parts, like the pre-commit hook and the gitlab-push command are no longer available in the remote module
> The data is about 2.7 Mb, so one possibility is to just commit the data directly in the remote repository, I suppose, but I'm not sure how ctest will then interact with it

The hook and other bash function are just a convenience, you could upload the file named to Girder and add a template to ensure it can be downloaded. For example, see here
That said, if the module vtkExternalData is already included by the remote build buildsysten, uploading the data manually to girder will be sufficient.

Hth
Jc


On Thu, Oct 18, 2018 at 4:33 PM Aron Helser <[hidden email]> wrote:
I have a collaborator working with a VTK remote module (MomentInvariants). It so happens that this code was previously part of VTK proper, and was moved to a remote module. It includes some tests, with external test data. We now need to update the test data, and so are trying to change from md5 to sha512, as instructed here: https://gitlab.kitware.com/vtk/vtk/blob/master/Documentation/dev/git/data.md

The cmake and build parts work just as they used to, but unfortunately the git parts, like the pre-commit hook and the gitlab-push command are no longer available in the remote module, since it lives in it's own repository. 

Has anyone dealt with this before? What should we do?

The data is about 2.7 Mb, so one possibility is to just commit the data directly in the remote repository, I suppose, but I'm not sure how ctest will then interact with it.

Thanks!
Aron
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: vtk test data for remote module?

Aron Helser
Jc,
The remote module is built as part of VTK, so it is already using vtkExternalData. Looking at those lines you referenced:
# Data published by Girder
"<a href="https://data.kitware.com/api/v1/file/hashsum/%(algo)/%(hash)/download">https://data.kitware.com/api/v1/file/hashsum/%(algo)/%(hash)/download"
# Data published by developers using git-gitlab-push.
"<a href="https://www.vtk.org/files/ExternalData/%(algo)/%(hash)">https://www.vtk.org/files/ExternalData/%(algo)/%(hash)"
That means I can provide the files at that URL at data.kitware.com?  How do I do that?

On Thu, Oct 18, 2018 at 5:01 PM Jean-Christophe Fillion-Robin <[hidden email]> wrote:
Hi Aaron,

> the git parts, like the pre-commit hook and the gitlab-push command are no longer available in the remote module
> The data is about 2.7 Mb, so one possibility is to just commit the data directly in the remote repository, I suppose, but I'm not sure how ctest will then interact with it

The hook and other bash function are just a convenience, you could upload the file named to Girder and add a template to ensure it can be downloaded. For example, see here
That said, if the module vtkExternalData is already included by the remote build buildsysten, uploading the data manually to girder will be sufficient.

Hth
Jc


On Thu, Oct 18, 2018 at 4:33 PM Aron Helser <[hidden email]> wrote:
I have a collaborator working with a VTK remote module (MomentInvariants). It so happens that this code was previously part of VTK proper, and was moved to a remote module. It includes some tests, with external test data. We now need to update the test data, and so are trying to change from md5 to sha512, as instructed here: https://gitlab.kitware.com/vtk/vtk/blob/master/Documentation/dev/git/data.md

The cmake and build parts work just as they used to, but unfortunately the git parts, like the pre-commit hook and the gitlab-push command are no longer available in the remote module, since it lives in it's own repository. 

Has anyone dealt with this before? What should we do?

The data is about 2.7 Mb, so one possibility is to just commit the data directly in the remote repository, I suppose, but I'm not sure how ctest will then interact with it.

Thanks!
Aron
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: vtk test data for remote module?

Ben Boeckel
On Thu, Oct 18, 2018 at 18:04:13 -0400, Aron Helser wrote:
> > # Data published by Girder
> > "<a href="https://data.kitware.com/api/v1/file/hashsum/%(algo)/%(hash)/download">https://data.kitware.com/api/v1/file/hashsum/%(algo)/%(hash)/download"
> > # Data published by developers using git-gitlab-push.
> > "<a href="https://www.vtk.org/files/ExternalData/%(algo)/%(hash">https://www.vtk.org/files/ExternalData/%(algo)/%(hash)"
>
> That means I can provide the files at that URL at data.kitware.com?  How do
> I do that?

I believe the `hashsum` plugin for Girder looks through all public (or
private if authenticated) data for files with the given hash, so as
long as it is public somewhere on `data.kitware.com`, it should be
available to VTK as well.

--Ben
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: vtk test data for remote module?

Aron Helser
Thanks! It all seems to be working. 
I uploaded all the data we needed to my public directory, https://data.kitware.com/#user/575742dc8d777f68be8f3fb3/folder/5bc9d1138d777f06b92d9d27
and now the vtk build is able to find it. 

I'll add a link back to this thread in https://www.vtk.org/Wiki/VTK/Remote_Modules
Thanks,
Aron

On Fri, Oct 19, 2018 at 7:39 AM Ben Boeckel <[hidden email]> wrote:
On Thu, Oct 18, 2018 at 18:04:13 -0400, Aron Helser wrote:
> > # Data published by Girder
> > "<a href="https://data.kitware.com/api/v1/file/hashsum/%(algo)/%(hash)/download" rel="noreferrer" target="_blank">https://data.kitware.com/api/v1/file/hashsum/%(algo)/%(hash)/download"
> > # Data published by developers using git-gitlab-push.
> > "<a href="https://www.vtk.org/files/ExternalData/%(algo)/%(hash)" rel="noreferrer" target="_blank">https://www.vtk.org/files/ExternalData/%(algo)/%(hash)"
>
> That means I can provide the files at that URL at data.kitware.com?  How do
> I do that?

I believe the `hashsum` plugin for Girder looks through all public (or
private if authenticated) data for files with the given hash, so as
long as it is public somewhere on `data.kitware.com`, it should be
available to VTK as well.

--Ben

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers

Reply | Threaded
Open this post in threaded view
|

Re: vtk test data for remote module?

Jean-Christophe Fillion-Robin-2
Hi Aron,

This documentation maintained by ITK community is particularly useful to explain how to upload data and get the content link file:

It may be sensible to also reference it.

Hth
Jc

On Fri, Oct 19, 2018 at 10:07 AM Aron Helser <[hidden email]> wrote:
Thanks! It all seems to be working. 
I uploaded all the data we needed to my public directory, https://data.kitware.com/#user/575742dc8d777f68be8f3fb3/folder/5bc9d1138d777f06b92d9d27
and now the vtk build is able to find it. 

I'll add a link back to this thread in https://www.vtk.org/Wiki/VTK/Remote_Modules
Thanks,
Aron

On Fri, Oct 19, 2018 at 7:39 AM Ben Boeckel <[hidden email]> wrote:
On Thu, Oct 18, 2018 at 18:04:13 -0400, Aron Helser wrote:
> > # Data published by Girder
> > "<a href="https://data.kitware.com/api/v1/file/hashsum/%(algo)/%(hash)/download" rel="noreferrer" target="_blank">https://data.kitware.com/api/v1/file/hashsum/%(algo)/%(hash)/download"
> > # Data published by developers using git-gitlab-push.
> > "<a href="https://www.vtk.org/files/ExternalData/%(algo)/%(hash)" rel="noreferrer" target="_blank">https://www.vtk.org/files/ExternalData/%(algo)/%(hash)"
>
> That means I can provide the files at that URL at data.kitware.com?  How do
> I do that?

I believe the `hashsum` plugin for Girder looks through all public (or
private if authenticated) data for files with the given hash, so as
long as it is public somewhere on `data.kitware.com`, it should be
available to VTK as well.

--Ben

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://public.kitware.com/mailman/listinfo/vtk-developers