[Rcpp-devel] Overflow error when creating a large matrix
Qiang Kou
qkou at umail.iu.edu
Mon Nov 2 00:56:19 CET 2015
Hi, Alessandro,
I have confirmed and reproduced your error.
It should be a bug and it is in
https://github.com/RcppCore/Rcpp/blob/master/inst/include/Rcpp/vector/MatrixColumn.h#L39
In your case, i and n both are integers, but i * n caused an overflow. We
will fix this soon.
Thank you for the test on big vector and matrix. More test will be truly
appreciated.
Best wishes,
KK
On Sun, Nov 1, 2015 at 5:08 PM, Alessandro Mammana <mammana at molgen.mpg.de>
wrote:
> I got another error when using the Matrix.column() function, as in the
> code below:
>
> #include <Rcpp.h>
> #include "array.cpp"
> // [[Rcpp::export]]
> Rcpp::IntegerMatrix makeMat(int nrow=18, int ncol=136633572){
> Rcpp::IntegerMatrix mat(nrow, ncol);
> for (int i = 0; i < ncol; ++i) {
> Rcpp::MatrixColumn<INTSXP> col = mat.column(i);
> for (int j = 0; j < nrow; ++j){
> col[j] = 1;
> }
> }
> return mat;
> }
>
> Then in R:
>
> > library(Rcpp)
> > sourceCpp("makeMat.cpp")
> > mat <- makeMat()
>
> *** caught segfault ***
> address 0x7f4c1e825088, cause 'memory not mapped'
>
> Traceback:
> 1: .Primitive(".Call")(<pointer: 0x7f5068e0cee0>, nrow, ncol)
> 2: makeMat()
>
>
> If I had to guess, the column function is doing the multiplication
> i*nrow and storing it as an int, which overflows when i is big. I am
> sorry that I cannot point to the Rcpp code where this happens and
> propose the fix, I am sitting on a chair with my laptop on my legs,
> not quite convenient for programming :D.
>
> Best,
> Alessandro
>
> On Sun, Nov 1, 2015 at 7:35 PM, Alessandro Mammana
> <mammana at molgen.mpg.de> wrote:
> > As Sean O'Riordain correctly pointed out, I used the install_github
> > command improperly. Doing
> >
> > devtools::install_github("RcppCore/Rcpp")
> >
> > works and the big matrix is allocated correctly.
> >
> > Thanks for the help and for the great work you are doing!
> > Ale
> >
> > On Sun, Nov 1, 2015 at 6:30 PM, Alessandro Mammana
> > <mammana at molgen.mpg.de> wrote:
> >> It's great that R and Rcpp have support for long vectors! that makes
> >> my life so much easier!
> >>
> >> So I installed Rcpp version 0.12.1 and the error is still there (see
> >> end of the email). If I interpret the commit history correctly, the
> >> commit that fixes the bug was done shortly after the 0.12.1 release
> >> right? Is there an easy way of getting the latest version of the
> >> master branch?
> >> I tried:
> >>
> >> devtools::install_github("https://github.com/RcppCore/Rcpp")
> >>
> >> But I got:
> >>
> >> Error in username %||% getOption("github.user") %||% stop("Unknown
> username.") :
> >> Unknown username.
> >>
> >>> sourceCpp("makeMat.cpp")
> >>> makeMat(18, 136633572)
> >> Error in .Primitive(".Call")(<pointer: 0x7f57d1c73ee0>, nrow, ncol) :
> >> negative length vectors are not allowed
> >>> sesssionInfo()
> >> Error: could not find function "sesssionInfo"
> >>> sessionInfo()
> >> R version 3.2.0 (2015-04-16)
> >> Platform: x86_64-unknown-linux-gnu (64-bit)
> >> Running under: MarIuX64 2.0 GNU/Linux 2010-2012
> >>
> >> locale:
> >> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=C
> >> [4] LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C
> >> [7] LC_PAPER=C LC_NAME=C LC_ADDRESS=C
> >> [10] LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C
> >>
> >> attached base packages:
> >> [1] stats graphics grDevices utils datasets methods base
> >>
> >> other attached packages:
> >> [1] Rcpp_0.12.1
> >>
> >> On Sun, Nov 1, 2015 at 3:37 PM, Florian Plaza Oñate
> >> <florian.plaza at jouy.inra.fr> wrote:
> >>> The bug is probably fixed by this commit:
> >>>
> https://github.com/RcppCore/Rcpp/commit/eb069bf5c20aaa4d38e0ca2897bf04e956cb8c4c
> >>>
> >>>
> >>> Le 31/10/2015 20:08, Alessandro Mammana a écrit :
> >>>>
> >>>> Dear All,
> >>>> When creating a large matrix with the follwing code:
> >>>>
> >>>> #include <Rcpp.h>
> >>>> // [[Rcpp::export]]
> >>>> Rcpp::IntegerMatrix makeMat(int nrow, int ncol){
> >>>> Rcpp::IntegerMatrix mat(nrow, ncol);
> >>>> return mat;
> >>>> }
> >>>>
> >>>> I get the error:
> >>>>
> >>>> Error in .Primitive(".Call")(<pointer: 0x7f86936d3ea0>, nrow, ncol) :
> >>>> negative length vectors are not allowed
> >>>>
> >>>> Where nrow*ncol is a very large number (about 3*10^9). I understand
> >>>> that such a number cannot be represented by an int type, but this does
> >>>> not seem to be a problem when in R I do:
> >>>>
> >>>>> mat <- matrix(0, nrow=nrow, ncol=ncol)
> >>>>
> >>>>
> >>>> Questions:
> >>>> 1. why is the behaviour different between R and Rcpp in the allocation
> >>>> of a matrix?
> >>>> 2. In a 64 bits machine, what is actually the maximum allowed length
> >>>> of a vector/matrix? does the length of a vector/matrix need to be
> >>>> represented by an int?
> >>>>
> >>>> Thanks a lot and best regards,
> >>>> Alessandro
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Alessandro Mammana, PhD Student
> >> Max Planck Institute for Molecular Genetics
> >> Ihnestraße 63-73
> >> D-14195 Berlin, Germany
> >
> >
> >
> > --
> > Alessandro Mammana, PhD Student
> > Max Planck Institute for Molecular Genetics
> > Ihnestraße 63-73
> > D-14195 Berlin, Germany
>
>
>
> --
> Alessandro Mammana, PhD Student
> Max Planck Institute for Molecular Genetics
> Ihnestraße 63-73
> D-14195 Berlin, Germany
> _______________________________________________
> Rcpp-devel mailing list
> Rcpp-devel at lists.r-forge.r-project.org
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
>
--
Qiang Kou
qkou at umail.iu.edu
School of Informatics and Computing, Indiana University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20151101/38e740d4/attachment.html>
More information about the Rcpp-devel
mailing list