[GenABEL-dev] probabel big endian support

Wed Apr 30 18:04:00 CEST 2014

Dear Alvaro, Jurica,

On 29-04-14 17:12, Jurica Stanojkovic wrote:
> Hi Alvaro,
> 
>>Hi all,
>>
>>would it not be better practice to handle this on load, i.e: using
> this: http://man7.org/linux/man-pages/man3/endian.3.html
>>
>>Just a remark.
>>
>>-Alvaro
> 
> I have tried that approach, it is OK for fileHeader, but there is data
> in *fvi, *fvd files that is float and can be double.

That is correct. As far as I know, filevector was developed to be data
type agnostic (at least for the standard data types like int, float and
double).

> For that we need a byte-swap for float and double.
> I had some results with this, but I did not find every one place in
> source where byte-swap is needed.
> I was not sure that is enough to just byte-swap data on read,
> blockWriteOrRead could be also used for writing.
> During the read process data is read with file.read like char* and then
> cast to other values.

I noticed that too, and actually, I don't really understand why, because
the type of the data is stored in the header of a filevector file as
well (see fvlib/const.h for the types and fvlib/frutil.h for the
definition of the header).
I wasn't part of the filevector development, so I don't know the exact
considerations at that time.

In ProbABEL (gendata.cpp) the filevector data are read using the
ReadVariableAs() function (fvlib/AbstractMatrix.h), which performs the
cast.
I haven't checked, but maybe there's a better function in fvlib for
reading the data into ProbABEL.

Best,

Lennart.

> 
> Regard,
> Jurica
> 
> -------- Original Message --------
> Subject: Re: [GenABEL-dev] probabel big endian support
> Date: Sunday, April 27, 2014 05:29 CEST
> From: "Frank, Alvaro Jesus" <alvaro.frank at rwth-aachen.de>
> To: "L.C. Karssen"
> <lennart at karssen.org>,"genabel-devel at lists.r-forge.r-project.org"<genabel-devel at lists.r-forge.r-project.org>
> References: <896-53591700-f-3be4eec0 at 227853676>,
> <535C1462.9090502 at karssen.org>
> 
> 
>  
>> Hi all,
>>
>> would it not be better practice to handle this on load, i.e: using
>> this: http://man7.org/linux/man-pages/man3/endian.3.html
>>
>> Just a remark.
>>
>> -Alvaro
>> ________________________________________
>> From: genabel-devel-bounces at lists.r-forge.r-project.org
>> [genabel-devel-bounces at lists.r-forge.r-project.org] on behalf of L.C.
>> Karssen [lennart at karssen.org]
>> Sent: Saturday, April 26, 2014 10:17 PM
>> To: genabel-devel at lists.r-forge.r-project.org
>> Subject: Re: [GenABEL-dev] probabel big endian support
>>
>> Dear Jurica,
>>
>> On 24-04-14 15:52, Jurica Stanojkovic wrote:
>> > Dear list,
>> >
>> > I have tried building package probabel on mips big endian.
>>
>> That is great to hear! As far as I know, none of the current developers
>> have access to such a machine.
>>
>> > It looks like that inputfiles/*.fvd and inputfiles/*.fvi are created on
>> > little endian machine and are not working on big endian ones.
>>
>> That is correct, we found out
>>
>> >
>> > I have tried to create them on big endian mips, and replace ones that
>> > came with source package with the ones that I have created.
>> > The package was built with new files without an error.
>>
>> That is good news. So GenABEL and DatABEL work on big-endian machines.
>>
>> >
>> > I used following command to create files:
>> > library(GenABEL)
>> > library(DatABEL)
>> > fvdose <- mach2databel(imputedg="./checks/inputfiles/test.mldose",
>> > mlinfo="./checks/inputfiles/test.mlinfo",
>> > outfile="./checks/inputfiles/test.dose")
>> > fvprob <- mach2databel(imputedg="./checks/inputfiles/test.mlprob",
>> > mlinfo="./checks/inputfiles/test.mlinfo",
>> > outfile="./checks/inputfiles/test.prob", isprob=TRUE)
>> > mmdose <-
>> > mach2databel(imputedg="./checks/inputfiles/mmscore_gen.mldose",
>> > mlinfo="./checks/inputfiles/mmscore_gen.mlinfo",
>> > outfile="./checks/inputfiles/mmscore_gen.dose")
>> > mmprob <-
>> > mach2databel(imputedg="./checks/inputfiles/mmscore_gen.mlprob",
>> > mlinfo="./checks/inputfiles/mmscore_gen.mlinfo",
>> > outfile="./checks/inputfiles/mmscore_gen.prob", isprob=TRUE)
>> >
>> > I am new to ProbABEL, GenABEL, DatABEL so could someone please help me
>> > with following questions:
>> >
>> > What is the best course of action for supporting probabel on big endian?
>> > Should *.fvi, *.fvd files allways be in little endian format (than
>> > DatABEL needs to be changed to always create little endian files)?
>> > Or can *.fvd, *.fvi files be replaced with big endian files for big
>> > endian build?
>>
>> I would say that ideally the files need only to be created once and then
>> usable on all systems. Especially since these files are usually large
>> and converting from text format to .fvi/.fvd takes quite a while.
>>
>> This, however, would require diving into the filevector and the DatABEL
>> code (filevector or libfilevector is the name of the 'backend' code in
>> which the .fvd/.fvi files are 'defined'; both DatABEL and ProbABEL use
>> that code when dealing with .fvi/.fvd files). I don't have very much
>> experience with either code base, but could probably have a look and
>> give you some pointers.
>>
>> >
>> > Is it necessary to be able to use *.fvd *.fvi files created on a
>> > different endian system?
>>
>> On the other hand, how often will people transfer these files to
>> machines of different architectures?
>>
>> Jurica, can you tell us a bit more about why you are using a MIPS
>> machine for your work with ProbABEL? And do you think it would be a
>> common task to move these files between machines with different
>> architectures at your site?
>>
>> Maybe a converter from big to little and vice versa would be the easiest
>> solution? I guess such a conversion can be done rather quick. The
>> downside would be that it (at least temporarily) requires double the
>> disk space.
>> Such a converter could be part of the fvutils and/or of DatABEL, for
>> example.
>>
>> >
>> > I am willing to work on adding big endian support and I will appreciate
>> > any help in determining the right course of action in resolving this
>> > problem.
>>
>> Thank you for your time and willingness to help! It is very much
>> appreciated. We're a small group of developers, but we'll try to help as
>> much as we can.
>>
>>
>> Best,
>>
>> Lennart.
>>
>> >
>> > Regards,
>> > Jurica
>> >
>> >
>> > _______________________________________________
>> > genabel-devel mailing list
>> > genabel-devel at lists.r-forge.r-project.org
>> >
>> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/genabel-devel
>> >
>>
>> --
>> *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
>> L.C. Karssen
>> Utrecht
>> The Netherlands
>>
>> lennart at karssen.org
>> http://blog.karssen.org
>> GPG key ID: A88F554A
>> -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
>>
>> _______________________________________________
>> genabel-devel mailing list
>> genabel-devel at lists.r-forge.r-project.org
>> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/genabel-devel
> 
>  

-- 
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
L.C. Karssen
Utrecht
The Netherlands

lennart at karssen.org
http://blog.karssen.org
GPG key ID: A88F554A
-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 213 bytes
Desc: OpenPGP digital signature
URL: <http://lists.r-forge.r-project.org/pipermail/genabel-devel/attachments/20140430/2882881a/attachment.sig>