[Rcpp-devel] Very Large Matrices in RcppArmadillo

[Steve Lianoglou]

On Tue, Jul 17, 2012 at 12:08 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
> On Tue, Jul 17, 2012 at 10:44 AM, French, Joshua
> <JOSHUA.FRENCH at ucdenver.edu> wrote:
>> Thank you all for the responses.
>>
>> Christian, I didn't know about the copy_aux_mem option.  I will have to
>> take a look at that.
>>
>> Dirk, thanks for looking into the 64-bit matrix indices.
>>
>> Doug, the place in my code where I get the error is when I multiply
>> matrices.  I might have matrices X and Y, where X is 300000x500 and Y is
>> 500x300000 and I want Z = X * Y.
>
> You will need to come up with another algorithm.  The size of Z is
> 3,000,000 by 3,000,000 and storing this as a dense matrix will require
> about 67 terabytes of memory
>
>> (3000000 * 3000000 * 8) / 2^30
> [1] 67055.23

To be fair, it's 300k by 300k (not 3mill by 3mill), so:

R> (300000 * 300000 * 8) / 2^30
[1] 670.5523

So, I guess 0.67 terabytes ... no problemo ;-)

-steve

-- 
Steve Lianoglou
Graduate Student: Computational Systems Biology
 | Memorial Sloan-Kettering Cancer Center
 | Weill Medical College of Cornell University
Contact Info: http://cbio.mskcc.org/~lianos/contact