<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Günter,</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Great report! I'm able to reproduce it on 1.8.11 here. Will file a bug and look into it.</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Thanks again for reporting.</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div> <div id="bloop_sign_1390426462503048960" class="bloop_sign"><div style="font-family:helvetica,arial;font-size:13px">Arun</div></div> <div style="color:gray"><hr style="style:linear">From: <span style="color:black">Günter J. Hitsch</span> <a href="mailto:guenter.hitsch@mac.com">Günter J. Hitsch</a><br>Reply: <span style="color:black">Günter J. Hitsch</span> <a href="mailto:guenter.hitsch@mac.com">guenter.hitsch@mac.com</a><br>Date: <span style="color:black">January 22, 2014 at 9:52:36 PM</span><br>To: <span style="color:black">datatable-help@lists.r-forge.r-project.org</span> <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>Subject: <span style="color:black"> [datatable-help] segfault with "large" number of rows <br></span></div> <blockquote type="cite" class="clean_bq"><span><div><div>
<br>I’ve been using data.table for several months. It’s a great package—thank you for developing it!
<br>
<br>Here’s my question: I’ve run into a problem when I use “large” data tables with many millions of rows. In particular, for such large data tables I get segmentation faults when I create columns by groups. Example:
<br>
<br>N = 2500 # No. of groups
<br>T = 100000 # No. of observations per group
<br>
<br>DT = data.table(group = rep(1:N, each = T), x = 1)
<br>setkey(DT, group)
<br>
<br>DT[, sum_x := sum(x), by = group]
<br>print(head(DT))
<br>
<br>This runs fine. But when I increase the number of groups, say from 2500 to 3000, I get a segfault:
<br>
<br>N = 3000 # No. of groups
<br>T = 100000 # No. of observations per group
<br>
<br>...
<br>
<br> *** caught segfault ***
<br>address 0x159069140, cause 'memory not mapped'
<br>
<br>Traceback:
<br> 1: `[.data.table`(DT, , `:=`(sum_x, sum(x)), by = group)
<br> 2: DT[, `:=`(sum_x, sum(x)), by = group]
<br> 3: eval(expr, envir, enclos)
<br> 4: eval(ei, envir)
<br> 5: withVisible(eval(ei, envir))
<br>
<br>
<br>I can reproduce this problem on:
<br>
<br>(1) OS X 10.9, R 3.0.2, data.table 1.8.10
<br>(2) Ubuntu 13.10, R 3.0.1, data.table 1.8.10
<br>
<br>And of course the amount of RAM in my machines is not the issue.
<br>
<br>Thanks in advance for your help with this!
<br>
<br>Günter
<br>
<br>_______________________________________________
<br>datatable-help mailing list
<br>datatable-help@lists.r-forge.r-project.org
<br>https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
<br></div></div></span></blockquote></body></html>