Thanks for the emails, Matthew. I think our talk should be moved to the data.table discussion list. <br><div class="gmail_quote"><br>So, of all the data manipulation functions I've used, the 'gen' function (which is similar to Stata's egen function) has been the most useful. I would be very happy if it were adapted to data.tables. <br>
<br>Here is the version that works with plyr.<br><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">> </span></span><span style="font-family: courier new,monospace;">gen <- function(data, ... ,by=NA) {</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> </span><span style="font-family: courier new,monospace;">if(!<a href="http://is.na" target="_blank">is.na</a>(by)) {</span><br><span style="font-family: courier new,monospace;">require(plyr)</span><div class="im">
<span style="font-family: courier new,monospace;"></span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> out = ddply(data,by, transform, ... )</span><span style="font-family: courier new,monospace;"></span><br style="font-family: courier new,monospace;"></div><span style="font-family: courier new,monospace;">} else {</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">out = transform(data,...)</span><span style="font-family: courier new,monospace;"></span><span style="font-family: courier new,monospace;"></span><br><span style="font-family: courier new,monospace;">}<br>
return(out)</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">}</span><br><br><span style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">> </span></span><span style="font-family: courier new,monospace;">y = data.frame(x1 = 1, x2 = rep(c(1,2), each=2), x3= rep(c(1,2),4))<br>
<br></span><span style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">> </span></span><span style="font-family: courier new,monospace;">gen(y, xSum= sum(x1), by=c('x2'))<br>
<br> x1 x2 x3 xSum<br>1 1 1 1 4<br>2 1 1 2 4<br>3 1 1 1 4<br>4 1 1 2 4<br>5 1 2 1 4<br>
6 1 2 2 4<br>7 1 2 1 4<br>8 1 2 2 4<br><br><br></span><span style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">> </span></span><span style="font-family: courier new,monospace;">gen(y,xSum= sum(x1), by=c('x2','x3') )<br>
x1 x2 x3 xSum<br>1 1 1 1 2<br>2 1 1 1 2<br>
3 1 1 2 2<br>4 1 1 2 2<br>5 1 2 1 2<br>6 1 2 1 2<br>7 1 2 2 2<br>8 1 2 2 2<br></span><span style="font-family: courier new,monospace;"><br></span>This is the last Matthew and friends came up with for data.tables:<br>
<br><span style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">> </span></span><span style="font-family: courier new,monospace;">require(data.table)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">> </span></span><span style="font-family: courier new,monospace;">x = data.table(x1 = as.integer(1),
x2 = as.integer(rep(c(1,2), each=2)), x3= as.integer(rep(c(1,2),4)))<br>
</span><div style="font-family: courier new,monospace;" class="im"><br><br><span style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">> </span></span>gen2 <- function(data,by,...) {<br>
eval(parse(text=paste("data[,transform(.SD,...),by=",substitute(by),"]"))) <br> }<br>
</div><div><div style="font-family: courier new,monospace;" class="im"><br><span style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">> </span></span>gen2(x, by='x2', xSum = sum(x1)) <br>
</div><span style="font-family: courier new,monospace;"> x2 x1 x2.1 x3 xSum</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[1,] 1 1 1 1 4</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">[2,] 1 1 1 2 4</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[3,] 1 1 1 1 4</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">
[4,] 1 1 1 2 4</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[5,] 2 1 2 1 4</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[6,] 2 1 2 2 4</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">[7,] 2 1 2 1 4</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[8,] 2 1 2 2 4</span><div class="im">
<span style="font-family: courier new,monospace;"></span><br><span style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">> </span></span><span style="font-family: courier new,monospace;">gen2(x, by=x2, xSum = sum(x1)) # ok</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"></span></span><br><br></div><span style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">> </span></span><span style="font-family: courier new,monospace;">gen2(x, by=list(x2,x3), xSum = sum(x1)) #fails<br>
Error in `[.data.table`(data, , +gen(.SD, ...), by = list) : <br><div class="im">
column 1 of 'by' list does not evaluate to integer e.g. the by should be a list of expressions. Do not quote column names when using by=list(...).<br></div></span><span style="font-family: courier new,monospace;"></span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">> class(x$x2)</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[1] "integer"<br><br><span style="font-family: courier new,monospace;">> gen2(x, by=c('x2','x3'), xSum = sum(x1)) #fails</span><br style="font-family: courier new,monospace;">
</span><span style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">> </span></span><span style="font-family: courier new,monospace;">gen2(x, by=list('x2','x3'), xSum = sum(x1)) #fails</span><br>
<br>The first issue is that when there are multiple grouping factors, passing those to the data.table through this function leads to hangups (see last few example).<br>
<br>The second issue is less critical, but important still. The data column x2.1 is automatically added and is redundant. With multiple uses of this function, these redundant columns add up quickly. I recall that there used to be a way to suppress the repeating of columns in this type of operation, but it was removed in later versions of data.table.<br>
<br><br>~~~~~~~~~~~~<br><br>There were a few other things Matthew and I were discussing, including alternative 'SQL inspired' functions such as 'groupby', 'orderby', 'where', 'select', etc.. I find that functions are a lot easier to remember than syntax, for the same reasons natural language syntax is difficult to learn compared to learning words. Another selfish reason is that SQL was what I and many others learned with originally, and the SQL commands would be easier to translate.<br>
</div><div><div></div><div class="h5"><br><div class="gmail_quote">On Mon, Jul 12, 2010 at 2:28 PM, Matthew Dowle <span dir="ltr"><<a href="mailto:mdowle@mdowle.plus.com" target="_blank">mdowle@mdowle.plus.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi,<br>
<br>
There was a thread on r-help where Hadley and I discussed plyr using<br>
data.table but something technical prevented it at that time as I<br>
recall. Might be different now. The spelling is Wickham.<br>
<br>
There is a link on the homepage to the datatable-help subscription<br>
page :<br>
<br>
<a href="http://datatable.r-forge.r-project.org/" target="_blank">http://datatable.r-forge.r-project.org/</a><br>
<br>
Would be great to see you on that. You explained to me once about the<br>
functional style I think and I mean to come back to it.<br>
<br>
Best of luck with your diss,<br>
<font color="#888888"><br>
Matthew<br>
</font><div><div></div><div><br>
</div></div></blockquote></div>
</div></div></div><br><br clear="all"><br>-- <br>Sasha Goodman<br>Doctoral Candidate, Organizational Behavior<br>