<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix"><br>
Hi,<br>
<br>
This isn't a bug really. A documentation or too low default
issue maybe.<br>
<br>
When all spare slots are used up, there is no choice but to make a
shallow copy and create a new vector of column pointer slots.
This is the pointer (address in RAM) which any variable names
(symbols) point to. When this happens, data.table does a
reasonable job of changing the symbol in calling scope too, but
within a function within a function it's tricky. In your
function, x is actually being updated by reference, but in local
scope when the shallow copy happens ... when the spare slots are
used up.<br>
<br>
By default :<br>
<br>
datatable.alloccol = quote(max(100L,ncol(DT)+64L))<br>
<br>
Some people just change this to be a much larger number. That's
the easiest. Just over-allocate massively :<br>
<br>
options(datatable.alloccol = 10000)<br>
<br>
If you have under 50 tables, this won't matter a jot. If you
have 1000's of tables, then the spare space could become
significant.<br>
<br>
Assuming 64bit, 10000 * 8bytes / 1024^2 = 78KB. Knowing this
allows you to choose the appropriate amount of over-allocation for
your case. 50 tables * 78KB = 4MB = e.g. 0.01% of 32GB<br>
<br>
Or, if you know you are about to add a lot of columns by
reference via a function, you can increase the over-allocation of
one table using the alloc.col function :<br>
<br>
alloc.col(DT, 200)<br>
<span></span><br>
In case the example was actually close to the real example, you
can add a lot of columns in one step and the LHS of := can be an
expression :<br>
<br>
DT[, <span>paste0('a', 1:101) := 1] # add 101 columns named
"a1", "a2" ... "a101", all set to 1<br>
<br>
and set() may be an easier alternative to := in this case, now
that it can add columns as from v1.8.11<br>
<br>
If there is a real world example where it really needs to be
wrapped in a function in a function then that would be needed to
see (or an example closer to reality) to convince (me at least)
that we need to do better here.<br>
</span><span><br>
HTH,<br>
Matt<br>
<br>
</span><br>
<br>
On 14/12/13 13:10, Arunkumar Srinivasan wrote:<br>
</div>
<blockquote cite="mid:40AC36D389E643519828E005ABF732D0@gmail.com"
type="cite">
<div>Hi Huashan,</div>
<div> Great reproducible example! Would you mind filing a bug
report <a moz-do-not-send="true"
href="https://r-forge.r-project.org/tracker/?func=browse&group_id=240&atid=975">here</a>?</div>
<div>
<div>Thank you,</div>
<div>Arun</div>
<div><br>
</div>
</div>
<p style="color: #A0A0A8;">On Saturday, December 14, 2013 at 2:30
AM, Huashan Chen wrote:</p>
<blockquote type="cite"
style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span>
<div>
<div>
<div>I just found out that when the column quota are
reached, adding new columns</div>
<div>within a function will fail.</div>
<div><br>
</div>
<div>Blow are the testing code:</div>
<div><br>
</div>
<div>testF2=function(x){ </div>
<div> add_var<-function(varname){</div>
<div> x[, `:=`(eval(substitute(varname)), 1), with=F] </div>
<div> }</div>
<div> sapply(paste0('a', 1:101), add_var)</div>
<div>}</div>
<div><br>
</div>
<div>dd=data.table(a=1:3)</div>
<div>truelength(dd)</div>
<div>testF2(dd)</div>
<div>dim(dd) # only 100 columns</div>
<div><br>
</div>
<div>dd[, new:=3]</div>
<div>dim(dd) # adding new column outside a function is OK.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>--</div>
<div>View this message in context: <a
moz-do-not-send="true"
href="http://r.789695.n4.nabble.com/Fail-to-add-new-columns-within-a-function-tp4682173.html">http://r.789695.n4.nabble.com/Fail-to-add-new-columns-within-a-function-tp4682173.html</a></div>
<div>Sent from the datatable-help mailing list archive at
<a moz-do-not-send="true" href="http://Nabble.com">Nabble.com</a>.</div>
<div>_______________________________________________</div>
<div>datatable-help mailing list</div>
<div><a moz-do-not-send="true"
href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a></div>
<div><a moz-do-not-send="true"
href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a></div>
</div>
</div>
</span> </blockquote>
<div> <br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
datatable-help mailing list
<a class="moz-txt-link-abbreviated" href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a>
<a class="moz-txt-link-freetext" href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a></pre>
</blockquote>
<br>
</body>
</html>