<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<link href="chrome://translator/skin/floatingPanel.css"
type="text/css" rel="stylesheet">
</head>
<body bgcolor="#FFFFFF" text="#000099">
<div class="moz-cite-prefix">I got some very useful further feed
back from Matthew. Let me summarize some key points from his
suggestions concerning the code below:<br>
<br>
The following code is still fairly slow (although faster then
using table or tapply):<br>
<br>
<pre> a <- data.table(sample(SPFn$wdpaint,replace=F),SPFn$pnvid)</pre>
<pre> b <- a[,.N,by=list(V1,V2)]</pre>
<pre> c <- tapply(b$N,list(as.factor(b$V2), as.factor(b$V1)), sum)</pre>
<pre> for(i in 1:11){</pre>
<pre> a <- data.table(sample(SPFn$wdpaint,replace=F),SPFn$pnvid)</pre>
<pre> b <- a[,.N,by=list(V1,V2)]</pre>
<pre> c <- rbind(c,tapply(b$N,list(as.factor(b$V2), as.factor(b$V1)), sum))</pre>
<pre> }</pre>
As pointed out by Matthew, the rbind at the end of the loop will
be growing memory use and is generally inefficient. How badly it
is impacting performance will depend on the data size though. So
step 1 is to get that outside the loop (an useful link he provided
is
<a class="moz-txt-link-freetext" href="http://stackoverflow.com/questions/10452249/divide-et-impera-on-a-data-frame-in-r">http://stackoverflow.com/questions/10452249/divide-et-impera-on-a-data-frame-in-r</a>).
Based on a hint in R-inferno
(<a class="moz-txt-link-freetext" href="http://www.burns-stat.com/pages/Tutor/R_inferno.pdf">http://www.burns-stat.com/pages/Tutor/R_inferno.pdf</a>) I adapted
the code as follows:<br>
<br>
<pre>c <- vector('list', 12)</pre>
<pre>a1 <- data.table(as.integer(SPFn$wdpaint),as.integer(SPFn$pnvid))</pre>
<pre>a2 <- as.integer(SPFn$wdpaint)</pre>
<pre>for(i in 1:12){</pre>
<pre> a3 <- a1[,V1:=sample(a2,replace=F)]</pre>
<pre> b <- a3[,.N,by=list(V1,V2)]</pre>
<pre> c[[i]] <- tapply(b$N,list(as.factor(b$V2), as.factor(b$V1)), sum)</pre>
<pre>}</pre>
<pre>c <- do.call('rbind', c)</pre>
This did improve the run time, but only very little bit (16.0
instead of 16.4 seconds). Next step was to profile the code, to
see what part is taking most time. This can be done with Rprof().
The results showed that ordernumtol, a data.table function which
sorts numeric ('double' floating point) columns was taking a lot
of time. As it turns out, the SPFn$wdpaint and SPFn$pnvid were
both numerical. Changing these to integer does speed up the code a
lot.<br>
<br>
<pre>c <- vector('list', 12)</pre>
<pre>a1 <- data.table(as.integer(SPFn$wdpaint),as.integer(SPFn$pnvid))</pre>
<pre>a2 <- as.integer(SPFn$wdpaint)</pre>
<pre>for(i in 1:12){</pre>
<pre> a3 <- a1[,V1:=sample(a2,replace=F)]</pre>
<pre> b <- a3[,.N,by=list(V1,V2)]</pre>
<pre> c[[i]] <- tapply(b$N,list(as.factor(b$V2), as.factor(b$V1)), sum)</pre>
<pre>}</pre>
<pre>c <- do.call('rbind', c)</pre>
9<br>
The second code took 16.0 seconds. The last attempt 2.4 seconds
only! That is a serious (> 6x) improvement. And it shows I
really need to be much more careful about my variables...<br>
I checked and it also makes a smaller, but still very significant
difference when using table (3x) or tapply (2x).<br>
<br>
Big thanks to Matthew Dowle for all his help.. and any further
suggestions for improvements are obviously welcome. <br>
<br>
Cheers,<br>
<br>
Paulo<br>
<br>
<br>
<br>
On 06/19/2012 04:24 PM, Matthew Dowle wrote:<br>
</div>
<blockquote
cite="mid:40b656f58efda6586a41156fb19db55c.squirrel@webmail.plus.net"
type="cite">
<pre wrap="">
The shuffling can form a different number of groups can't it? </pre>
</blockquote>
YES, obvious.. I was half asleep I guess<br>
<blockquote
cite="mid:40b656f58efda6586a41156fb19db55c.squirrel@webmail.plus.net"
type="cite">
<pre wrap="">
table(c(1,1,2,2), c(3,3,4,4)) # 2 groups
table(c(2,2,1,1), c(3,3,4,4)) # 2 groups
table(c(2,1,2,1), c(3,3,4,4)) # 4 groups
</pre>
<blockquote type="cite">
<pre wrap="">Thanks Matthew
I am not sure I understand the code (actually, I am sure I do not :-( .
More specifically, I would expect the two expressions below to yield
tables
of the same dimension (basically all combinations of wdpaint and pnnid):
aa <- SPFdt[, .N, by=list(sample(wdpaint,replace=FALSE),pnvid)]
dim(aa)
</pre>
<blockquote type="cite">
<pre wrap="">254 3
</pre>
</blockquote>
<pre wrap="">bb <- SPFdt[, .N, by=list(wdpaint,pnvid)
dim(bb)
</pre>
<blockquote type="cite">
<pre wrap="">170 3
</pre>
</blockquote>
<pre wrap="">
What I am looking for is creating a cross table of pnvid and wdpaint,
i.e.,
the frequency or number of occurrences of each combination of pnvid and
wdpaint. Shuffling wdpaint should give in that case a different frequency
distribution, like in the example below:
table(c(1,1,2,2), c(3,3,4,4))
table(c(2,2,1,1), c(3,3,4,4))
Basically what I want to do is run X permutations on a data set which I
will then use to create a confidence interval on the frequency
distribution
of sample points over wdpaint and pnvid
Cheers,
Paulo
On Tue, Jun 19, 2012 at 3:30 PM, Matthew Dowle
<a class="moz-txt-link-rfc2396E" href="mailto:mdowle@mdowle.plus.com"><mdowle@mdowle.plus.com></a>wrote:
</pre>
<blockquote type="cite">
<pre wrap="">
Hi,
Welcome to the list.
Rather than picking a column and calling length() on it, .N is a little
more convenient (and faster if that column isn't otherwise used, as in
this example). Search ?data.table for the string ".N" to find out more.
And to group by expressions of column names, wrap with list(). So,
SPF[, .N, by=list(sample(wdpaint,replace=FALSE),pnvid)]
But that won't calculate any different statistics, just return the
groups
in a different order. Seems like just an example, rather than the real
task, iiuc, which is fine of course.
Matthew
</pre>
<blockquote type="cite">
<pre wrap="">Hi, I am new to this package and not sure how to implement the
</pre>
</blockquote>
<pre wrap="">sample()
</pre>
<blockquote type="cite">
<pre wrap="">function with data.table.
I have a data frame SPF with three columns cat, pnvid and wdpaint. The
pnvid variables has values 1:3, the wdpaint has values 1:10. I am
interested in the count of all combinations of wdpaint and pnvid in my
data
set, which can be calculated using table or tapply (I use the latter
</pre>
</blockquote>
<pre wrap="">in
</pre>
<blockquote type="cite">
<pre wrap="">the
example code below).
Normally I would use something like:
*c <- tapply(SPF$cat, list(as.factor(SPF$pnvid),
</pre>
</blockquote>
<pre wrap="">as.factor(SPF$wdpaint),
</pre>
<blockquote type="cite">
<pre wrap="">function(x) length(x))*
If I understand correctly, I would use the below when working with
</pre>
</blockquote>
<pre wrap="">data
</pre>
<blockquote type="cite">
<pre wrap="">tables:
*f <- SPF[,length(cat),by="wdpaint,pnvid"]*
But what if I want to reshuffle the column wdpaint first? When using
tapply, it would be something along the lines of:
*a <- list(as.factor(SPF$pnvid), as.factor(sample(SPF$wdpaint,
replace=F)))
c <- tapply(SPF$cat, a, function(x) length(x))*
But how to do this with data.table?
Paulo
_______________________________________________
datatable-help mailing list
<a class="moz-txt-link-abbreviated" href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a>
</pre>
</blockquote>
<pre wrap=""><a class="moz-txt-link-freetext" href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a>
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<br>
<br>
<div style="bottom: auto; left: 286px; right: auto; top: 534px;
display: none;" class="translator-theme-default"
id="translator-floating-panel">
<div title="Click to translate"
id="translator-floating-panel-button"></div>
</div>
</body>
</html>