[datatable-help] adding names to j columns is costly

Sam Steingold sds at gnu.org
Wed Sep 11 23:35:35 CEST 2013


I find myself using setnames(...,"V1","...") very often because setting
them in aggregation is expensive:

--8<---------------cut here---------------start------------->8---
> delays.short <- delays.dt[,sum(count),by="delay"]
Finding groups (bysameorder=TRUE) ... done in 1.262secs. bysameorder=TRUE and o__ is length 0
Detected that j uses these columns: count 
Optimization is on but j left unchanged as 'sum(count)'
Starting dogroups ... done dogroups in 8.612 secs
> delays.short <- delays.dt[,list(count=sum(count)),by="delay"]
Finding groups (bysameorder=TRUE) ... done in 1.051secs. bysameorder=TRUE and o__ is length 0
Detected that j uses these columns: count 
Optimization is on but j left unchanged as 'list(sum(count))'
Starting dogroups ... done dogroups in 11.918 secs
--8<---------------cut here---------------end--------------->8---

38% difference is a lot (3 seconds is not a big deal, but this is just a
toy dataset).

ISTR that I have asked this question before - is this still (data.table
1.8.10) the state of the art, or am I doing something stupid?

Thanks!

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 13.04 (raring) X 11.0.11303000
http://www.childpsy.net/ http://think-israel.org http://truepeace.org
http://thereligionofpeace.com http://americancensorship.org http://iris.org.il
Money does not "play a role", it writes the scenario.


More information about the datatable-help mailing list