[Rcpp-devel] Overflow error when creating a large matrix

[Alessandro Mammana]

I got another error when using the Matrix.column() function, as in the
code below:

#include <Rcpp.h>
#include "array.cpp"
// [[Rcpp::export]]
Rcpp::IntegerMatrix makeMat(int nrow=18, int ncol=136633572){
        Rcpp::IntegerMatrix mat(nrow, ncol);
        for (int i = 0; i < ncol; ++i) {
                Rcpp::MatrixColumn<INTSXP> col = mat.column(i);
                for (int j = 0; j < nrow; ++j){
                        col[j] = 1;
                }
        }
        return mat;
}

Then in R:

> library(Rcpp)
> sourceCpp("makeMat.cpp")
> mat <- makeMat()

 *** caught segfault ***
address 0x7f4c1e825088, cause 'memory not mapped'

Traceback:
 1: .Primitive(".Call")(<pointer: 0x7f5068e0cee0>, nrow, ncol)
 2: makeMat()


If I had to guess, the column function is doing the multiplication
i*nrow and storing it as an int, which overflows when i is big. I am
sorry that I cannot point to the Rcpp code where this happens and
propose the fix, I am sitting on a chair with my laptop on my legs,
not quite convenient for programming :D.

Best,
Alessandro

On Sun, Nov 1, 2015 at 7:35 PM, Alessandro Mammana
<mammana at molgen.mpg.de> wrote:
> As Sean O'Riordain correctly pointed out, I used the install_github
> command improperly. Doing
>
> devtools::install_github("RcppCore/Rcpp")
>
> works and the big matrix is allocated correctly.
>
> Thanks for the help and for the great work you are doing!
> Ale
>
> On Sun, Nov 1, 2015 at 6:30 PM, Alessandro Mammana
> <mammana at molgen.mpg.de> wrote:
>> It's great that R and Rcpp have support for long vectors! that makes
>> my life so much easier!
>>
>> So I installed Rcpp version 0.12.1 and the error is still there (see
>> end of the email). If I interpret the commit history correctly, the
>> commit that fixes the bug was done shortly after the 0.12.1 release
>> right? Is there an easy way of getting the latest version of the
>> master branch?
>> I tried:
>>
>> devtools::install_github("https://github.com/RcppCore/Rcpp")
>>
>> But I got:
>>
>> Error in username %||% getOption("github.user") %||% stop("Unknown username.") :
>>   Unknown username.
>>
>>> sourceCpp("makeMat.cpp")
>>> makeMat(18, 136633572)
>> Error in .Primitive(".Call")(<pointer: 0x7f57d1c73ee0>, nrow, ncol) :
>>   negative length vectors are not allowed
>>> sesssionInfo()
>> Error: could not find function "sesssionInfo"
>>> sessionInfo()
>> R version 3.2.0 (2015-04-16)
>> Platform: x86_64-unknown-linux-gnu (64-bit)
>> Running under: MarIuX64 2.0 GNU/Linux 2010-2012
>>
>> locale:
>>  [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C         LC_TIME=C
>>  [4] LC_COLLATE=C         LC_MONETARY=C        LC_MESSAGES=C
>>  [7] LC_PAPER=C           LC_NAME=C            LC_ADDRESS=C
>> [10] LC_TELEPHONE=C       LC_MEASUREMENT=C     LC_IDENTIFICATION=C
>>
>> attached base packages:
>> [1] stats     graphics  grDevices utils     datasets  methods   base
>>
>> other attached packages:
>> [1] Rcpp_0.12.1
>>
>> On Sun, Nov 1, 2015 at 3:37 PM, Florian Plaza Oñate
>> <florian.plaza at jouy.inra.fr> wrote:
>>> The bug is probably fixed by this commit:
>>> https://github.com/RcppCore/Rcpp/commit/eb069bf5c20aaa4d38e0ca2897bf04e956cb8c4c
>>>
>>>
>>> Le 31/10/2015 20:08, Alessandro Mammana a écrit :
>>>>
>>>> Dear All,
>>>> When creating a large matrix with the follwing code:
>>>>
>>>> #include <Rcpp.h>
>>>> // [[Rcpp::export]]
>>>> Rcpp::IntegerMatrix makeMat(int nrow, int ncol){
>>>>          Rcpp::IntegerMatrix mat(nrow, ncol);
>>>>          return mat;
>>>> }
>>>>
>>>> I get the error:
>>>>
>>>> Error in .Primitive(".Call")(<pointer: 0x7f86936d3ea0>, nrow, ncol) :
>>>>    negative length vectors are not allowed
>>>>
>>>> Where nrow*ncol is a very large number (about 3*10^9). I understand
>>>> that such a number cannot be represented by an int type, but this does
>>>> not seem to be a problem when in R I do:
>>>>
>>>>> mat <- matrix(0, nrow=nrow, ncol=ncol)
>>>>
>>>>
>>>> Questions:
>>>> 1. why is the behaviour different between R and Rcpp in the allocation
>>>> of a matrix?
>>>> 2. In a 64 bits machine, what is actually the maximum allowed length
>>>> of a vector/matrix? does the length of a vector/matrix need to be
>>>> represented by an int?
>>>>
>>>> Thanks a lot and best regards,
>>>> Alessandro
>>>>
>>>
>>
>>
>>
>> --
>> Alessandro Mammana, PhD Student
>> Max Planck Institute for Molecular Genetics
>> Ihnestraße 63-73
>> D-14195 Berlin, Germany
>
>
>
> --
> Alessandro Mammana, PhD Student
> Max Planck Institute for Molecular Genetics
> Ihnestraße 63-73
> D-14195 Berlin, Germany



-- 
Alessandro Mammana, PhD Student
Max Planck Institute for Molecular Genetics
Ihnestraße 63-73
D-14195 Berlin, Germany