[GenABEL-dev] probabel big endian support

[Jurica Stanojkovic]

Dear Karssen,

>> What is the best course of action for supporting probabel on big endian?
>> Should *.fvi, *.fvd files allways be in little endian format (than
>> DatABEL needs to be changed to always create little endian files)?
>> Or can *.fvd, *.fvi files be replaced with big endian files for big
>> endian build?

>I would say that ideally the files need only to be created once and then
>usable on all systems. Especially since these files are usually large
>and converting from text format to .fvi/.fvd takes quite a while.

If I had to change some values in text format, would I have to generate again fvd/fvi files?
Does one when working with ProbABEL has to change those files often?
If we do byte-swap on the run for every data in the fvd/fvi file would that be also time consuming?
I understand that user then do not need to wait files to generate again on big endian,
but same task (run) will last longer on big-endian machine than on little-endian one?

>This, however, would require diving into the filevector and the DatABEL
>code (filevector or libfilevector is the name of the 'backend' code in
>which the .fvd/.fvi files are 'defined'; both DatABEL and ProbABEL use
>that code when dealing with .fvi/.fvd files). I don't have very much
>experience with either code base, but could probably have a look and
>give you some pointers.

I tried to work around this and got some results, but a I did not manage
to find every place in code where endian swap is needed.
I am currently busy with other work, but i will soon look at this again.

>Jurica, can you tell us a bit more about why you are using a MIPS
>machine for your work with ProbABEL? And do you think it would be a
>common task to move these files between machines with different
>architectures at your site?

I work on supporting mips/mipsel for Debian sid.
I have access to mips and mipsel boards and can help with bigendian support.
But I do not use ProbABEL actively.

>Maybe a converter from big to little and vice versa would be the easiest
>solution? I guess such a conversion can be done rather quick. The
>downside would be that it (at least temporarily) requires double the
>disk space.
>Such a converter could be part of the fvutils and/or of DatABEL, for
>example.

Maybe this could be a good solution, presuming that this would be faster then just converting from text to fileVector format?I will have to look closer how data is converted and writen from text to fvd/fvi in order to be able to convert them to different endian.
There is also a option to always create a fvd/fvi in both endian formats,
or to create some universal file that have data in both endians inside.

Regards,
Jurica

-------- Original Message --------
Subject: Re: [GenABEL-dev] probabel big endian support
Date: Saturday, April 26, 2014 22:17 CEST
From: "L.C. Karssen" <lennart at karssen.org>
To: genabel-devel at lists.r-forge.r-project.org
References: <896-53591700-f-3be4eec0 at 227853676>
 Dear Jurica,

On 24-04-14 15:52, Jurica Stanojkovic wrote:
> Dear list,
>
> I have tried building package probabel on mips big endian.

That is great to hear! As far as I know, none of the current developers
have access to such a machine.

> It looks like that inputfiles/*.fvd and inputfiles/*.fvi are created on> little endian machine and are not working on big endian ones.

That is correct, we found out

>
> I have tried to create them on big endian mips, and replace ones that
> came with source package with the ones that I have created.
> The package was built with new files without an error.

That is good news. So GenABEL and DatABEL work on big-endian machines.

>
> I used following command to create files:
> library(GenABEL)
> library(DatABEL)
> fvdose <- mach2databel(imputedg="./checks/inputfiles/test.mldose",
> mlinfo="./checks/inputfiles/test.mlinfo",
> outfile="./checks/inputfiles/test.dose")
> fvprob <- mach2databel(imputedg="./checks/inputfiles/test.mlprob",
> mlinfo="./checks/inputfiles/test.mlinfo",
> outfile="./checks/inputfiles/test.prob", isprob=TRUE)
> mmdose <-
> mach2databel(imputedg="./checks/inputfiles/mmscore_gen.mldose",
> mlinfo="./checks/inputfiles/mmscore_gen.mlinfo",
> outfile="./checks/inputfiles/mmscore_gen.dose")
> mmprob <-
> mach2databel(imputedg="./checks/inputfiles/mmscore_gen.mlprob",
> mlinfo="./checks/inputfiles/mmscore_gen.mlinfo",
> outfile="./checks/inputfiles/mmscore_gen.prob", isprob=TRUE)
>
> I am new to ProbABEL, GenABEL, DatABEL so could someone please help me
> with following questions:
>
> What is the best course of action for supporting probabel on big endian?
> Should *.fvi, *.fvd files allways be in little endian format (than
> DatABEL needs to be changed to always create little endian files)?
> Or can *.fvd, *.fvi files be replaced with big endian files for big
> endian build?

I would say that ideally the files need only to be created once and then
usable on all systems. Especially since these files are usually large
and converting from text format to .fvi/.fvd takes quite a while.

This, however, would require diving into the filevector and the DatABEL
code (filevector or libfilevector is the name of the 'backend' code in
which the .fvd/.fvi files are 'defined'; both DatABEL and ProbABEL use
that code when dealing with .fvi/.fvd files). I don't have very much
experience with either code base, but could probably have a look and
give you some pointers.

>
> Is it necessary to be able to use *.fvd *.fvi files created on a
> different endian system?

On the other hand, how often will people transfer these files to
machines of different architectures?

Jurica, can you tell us a bit more about why you are using a MIPS
machine for your work with ProbABEL? And do you think it would be a
common task to move these files between machines with different
architectures at your site?

Maybe a converter from big to little and vice versa would be the easiest
solution? I guess such a conversion can be done rather quick. The
downside would be that it (at least temporarily) requires double the
disk space.
Such a converter could be part of the fvutils and/or of DatABEL, for
example.

>
> I am willing to work on adding big endian support and I will appreciate> any help in determining the right course of action in resolving this
> problem.

Thank you for your time and willingness to help! It is very much
appreciated. We're a small group of developers, but we'll try to help as
much as we can.


Best,

Lennart.

>
> Regards,
> Jurica
>
>
> _______________________________________________
> genabel-devel mailing list
> genabel-devel at lists.r-forge.r-project.org
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/genabel-devel
>

--
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
L.C. Karssen
Utrecht
The Netherlands

lennart at karssen.org
http://blog.karssen.org
GPG key ID: A88F554A
-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
 
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/genabel-devel/attachments/20140429/e6940b01/attachment.html>