<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
<html><body>
<p> </p>
<p>Glad all clear. Given the follow up head() examples, yes, .SD is there</p>
<p>for just that purpose. Something like this :</p>
<p> DT[, head(.SD,2), by=colA]</p>
<p>is idiomatic in data.table. That's like a "select top 2 * from" in SQL, but by group.</p>
<p>Also things like :</p>
<p> DT[, .SD[1:2], by=colA] # similar provided all groups have at least 2 rows</p>
<p> DT[, .SD[-1], by=colA] # all but the first</p>
<p> DT[, someFunctionThatWantsADataFrame(..., data=.SD), by=colA]</p>
<p> </p>
<p>It's when you don't use all the data in .SD that it's wasteful to use it (since</p>
<p>data.table needs to populate it for each group before running j).</p>
<p>So in the subset of rows of .SD examples above, something like this can</p>
<p>be a lot faster :</p>
<p> w = DT[,head(.I,5),by=colA][[2]] # top 5 row numbers of each group</p>
<p> DT[w] # select those rows</p>
<p>is the same but must faster than</p>
<p> DT[, head(.SD,5), by=colA]</p>
<p>especially if each of the groups have a lot more rows than 5.</p>
<p>Hope that adds some colour.</p>
<p> </p>
<p>On 17.01.2013 17:33, David Bellot wrote:</p>
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%"><!-- html ignored --><!-- head ignored --><!-- meta ignored --><span><span><span style="font-family: arial, helvetica, sans-serif;">indeed, it makes sense now, as what is passed to the functi<span>on is indeed <span>a data<span>.table and not a <span>data.frame.<br /><br /><span>Thanks guys fo<span>r your help.<span> Now I'm a convinced data<span>.table user.<br /><span>Best,<br />David</span><br /></span></span></span></span></span></span></span></span></span><br /></span></span>
<div class="gmail_quote">On Thu, Jan 17, 2013 at 5:25 PM, Akhil Behl <span><<a href="mailto:akhil@igidr.ac.in">akhil@igidr.ac.in</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;">Hey David,<br /><br /> I thought your problem may have been a typo, but I realized that it is<br /> in fact a subtle difference between the way data.table and data.frame<br /> work.<br /><br /> One must provide unquoted names in the `j' expression for a<br /> data.table, i.e. one can say x.dt[ , y] but not x.dt[ , "y"] (which<br /> will evaluate to just "y" and hence the error).<br /><br /> There are tricks around it like using with=FALSE, or using the<br /> data.frame notation x.dt[["y"]]. But once again, you will find such<br /> examples and explanations of idiomatic data.table expressions in the<br /> vignettes.<br /><br /> --<br /> ASB.<br />
<div class="HOEnZb">
<div class="h5"><br /> On Thu, Jan 17, 2013 at 10:42 PM, David Bellot <<a href="mailto:david.bellot@gmail.com">david.bellot@gmail.com</a>> wrote:<br /> > Hi Matthew,<br /> ><br /> > I read indeed the introduction but I wasn't sure about the way to write it.<br /> > Hence my question.<br /> ><br /> > In fact, I do agree if the function would sum(sqrt(y)), but in my case, I<br /> > would like to do something like<br /> ><br /> > f ><br /> > It's a small example for the sake of simplicity, just to illustrate that I<br /> > really want to have access to the full sub data.frame (the d variable) and<br /> > not just one column.<br /> ><br /> > Best,<br /> > David<br /> ><br /> > On Thu, Jan 17, 2013 at 5:07 PM, Matthew Dowle <<a href="mailto:mdowle@mdowle.plus.com">mdowle@mdowle.plus.com</a>><br /> > wrote:<br /> >><br /> >><br /> >> Akhil,<br /> >><br /> >> Kind of, but defining :<br /> >><br /> >> my.func >> sum(sqrt(d[["y"]]))<br /> >> }<br /> >><br /> >> followed by<br /> >><br /> >> x.dt[ , my.func(.SD), by=x]<br /> >><br /> >> isn't very data.table'ish. In fact the<br /> >> advice is to avoid .SD if possible, for speed.<br /> >><br /> >> We'd forget my.funct, and just do :<br /> >><br /> >> x.dt[, sum(sqrt(y)), by=x]<br /> >><br /> >> That is how we recommend it to be used, and<br /> >> allows data.table to optimize the query (which<br /> >> use of .SD may prevent).<br /> >><br /> >> David - have you read the introduction vignette and have<br /> >> you worked through example(data.table) at the prompt?<br /> >><br /> >> Matthew<br /> >><br /> >><br /> >><br /> >> On 17.01.2013 16:53, Akhil Behl wrote:<br /> >>><br /> >>> If I am not wrong, you are looking for `.SD'. In fact you can put in<br /> >>> the exact function you were throwing at ddply earlier. There are other<br /> >>> special names like .SD that you can find in the data.table FAQs.<br /> >>><br /> >>> Let's see:<br /> >>> R> require(plyr)<br /> >>> Loading required package: plyr<br /> >>><br /> >>> R> require(data.table)<br /> >>> Loading required package: data.table<br /> >>> data.table 1.8.7 For help type: help("data.table")<br /> >>><br /> >>> R> x.df >>> R> x.dt >>> R><br /> >>> R> my.func >>> + sum(sqrt(d[["y"]]))<br /> >>> + }<br /> >>> R><br /> >>> R> # The plyr way:<br /> >>> R> ddply(x.df, "x", my.func) -> ans.plyr<br /> >>> R><br /> >>> R> # The data.table way:<br /> >>> R> x.dt[ , my.func(.SD), by=x] -> ans.dt<br /> >>> R><br /> >>> R> ans.plyr<br /> >>> x V1<br /> >>> 1 a 10.61387<br /> >>> 2 b 11.85441<br /> >>><br /> >>> R> ans.dt<br /> >>> x V1<br /> >>> 1: a 10.61387<br /> >>> 2: b 11.85441<br /> >>><br /> >>> For more help, try this on an R prompt:<br /> >>><br /> >>> R> vignette('datatable-faq')<br /> >>><br /> >>> --<br /> >>> ASB.<br /> >>><br /> >>> On Thu, Jan 17, 2013 at 9:49 PM, David Bellot <<a href="mailto:david.bellot@gmail.com">david.bellot@gmail.com</a>><br /> >>> wrote:<br /> >>>><br /> >>>> Hi,<br /> >>>><br /> >>>> I've been looking all around the web without a clear answer to this<br /> >>>> trivial<br /> >>>> problem. I'm sure I'm not looking where I should:<br /> >>>><br /> >>>> in fact, I want to replace my use of ddply from the plyr package by<br /> >>>> data.table. One of my main use is to group a big data.frame by a group<br /> >>>> of<br /> >>>> variable and do something on this sub data.frame:<br /> >>>><br /> >>>> ddply( my_df, my_grouping_var, function (d) { do something with d } )<br /> >>>> ----> d is a data.frame again<br /> >>>><br /> >>>> and it's slow on big data.frame.<br /> >>>><br /> >>>><br /> >>>> However, I don't really understand how to redo the same thing with a<br /> >>>> data.table. Basically if "j" in a data.table is equivalent to the select<br /> >>>> clause in SQL, then how do I do SELECT * FROM etc...<br /> >>>><br /> >>>> I want to be able to pass a function like in ddply that will receive not<br /> >>>> only a few columns but the full subset that is selected by the "by"<br /> >>>> clause.<br /> >>>><br /> >>>> Thanks...<br /> >>>> Best,<br /> >>>> David<br /> >>>><br /> >>>> _______________________________________________<br /> >>>> datatable-help mailing list<br /> >>>> <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br /> >>>><br /> >>>><br /> >>>> <a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br /> >>><br /> >>> _______________________________________________<br /> >>> datatable-help mailing list<br /> >>> <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br /> >>><br /> >>><br /> >>> <a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br /> ><br /> ></div>
</div>
</blockquote>
</div>
</blockquote>
<p> </p>
<div> </div>
</body></html>