[Rcpp-devel] Overflow error when creating a large matrix

[Dirk Eddelbuettel]

On 31 October 2015 at 19:08, Alessandro Mammana wrote:
| Dear All,
| When creating a large matrix with the follwing code:
| 
| #include <Rcpp.h>
| // [[Rcpp::export]]
| Rcpp::IntegerMatrix makeMat(int nrow, int ncol){
|         Rcpp::IntegerMatrix mat(nrow, ncol);
|         return mat;
| }
| 
| I get the error:
| 
| Error in .Primitive(".Call")(<pointer: 0x7f86936d3ea0>, nrow, ncol) :
|   negative length vectors are not allowed
| 
| Where nrow*ncol is a very large number (about 3*10^9). I understand
| that such a number cannot be represented by an int type, but this does
| not seem to be a problem when in R I do:
| 
| > mat <- matrix(0, nrow=nrow, ncol=ncol)
| 
| Questions:
| 1. why is the behaviour different between R and Rcpp in the allocation
| of a matrix?
| 2. In a 64 bits machine, what is actually the maximum allowed length
| of a vector/matrix? does the length of a vector/matrix need to be
| represented by an int?

Support for large objects is reasonably new in Rcpp -- it only came with the
0.12.0 release thanks to a lot of work by Qiang.

So what we have here is a bug.  You could ensure your spot in the hall of
fame by debugging and fixing it :)

FWIW I can tickle the bug in NumericMatrix too.  Allocating a matrix of 3e6 *
1e4 takes a hell of time even on the largest machine I have (easy) access too
at work --- which is not exactly a lightweight at 1/4 TB of RAM ...

So your help in debugging would be appreciated.

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org