<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Lucida Console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
code
{mso-style-priority:99;
font-family:"Courier New";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The performance is what puzzles me; the results are correct so the warnings don’t matter, and not all the variations I’ve tried have warnings. On the real dataset (~800,000 rows) datatable takes about 1.5 times longer than dataframe + ddply. I expected it to be substantially faster.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> Arunkumar Srinivasan [mailto:aragorn168b@gmail.com] <br><b>Sent:</b> Friday, June 13, 2014 8:57 PM<br><b>To:</b> Ron Hylton; datatable-help@lists.r-forge.r-project.org<br><b>Subject:</b> Re: [datatable-help] data.table is asking for help<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div id="bloop_customfont"><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>However there’s another aspect. While I’m relatively new to R my understanding is that a function argument should be modifiable within the function body without affecting the caller, which perhaps conflicts with the behavior of .SD.</span><span style='color:black'><o:p></o:p></span></p></div></div></blockquote><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Helvetica","sans-serif"'>`data.table` is designed for working with *really large* data sets in mind (> 100 or 200 GB in memory even). And therefore, as a design feature, it trades in "referential transparency" for manipulating data objects *as efficient as possible* in terms of both *speed* and *memory usage* (most of the times they go hand-in-hand).</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Helvetica","sans-serif"'>This is perhaps the biggest design choice one needs to be aware of when working/choosing data.tables. It is possible to modify objects by reference using data.table - All the functions that begin with "set*" modify objects by reference. The only other non "set*" function is `:=` operator.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><o:p> </o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Helvetica","sans-serif"'>HTH</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black'>Arun<o:p></o:p></span></p></div></div></div></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black'><br>From: Ron Hylton <a href="mailto:rhylton@verizon.net">rhylton@verizon.net</a><br>Reply: Ron Hylton <a href="mailto:rhylton@verizon.net">rhylton@verizon.net</a><br>Date: June 14, 2014 at 2:52:04 AM<br>To: <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a> <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>Subject: Re: [datatable-help] data.table is asking for help <o:p></o:p></span></p></div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'><br><br><o:p></o:p></span></p><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I suspected it was something like this. As one clarification, there is a setkey(test,id) before any setkey(.SD). If setkey(test,id) is changed to setkey(test) so all columns are in the original datatable key then the warning goes away.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>However there’s another aspect. While I’m relatively new to R my understanding is that a function argument should be modifiable within the function body without affecting the caller, which perhaps conflicts with the behavior of .SD.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>From:</span></b> <span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Arunkumar Srinivasan [<a href="mailto:aragorn168b@gmail.com">mailto:aragorn168b@gmail.com</a>]<br><b>Sent:</b> Friday, June 13, 2014 8:23 PM<br><b>To:</b> Ron Hylton; <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br><b>Subject:</b> Re: [datatable-help] data.table is asking for help</span><o:p></o:p></p></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'>Nicely reproducible post. Reproducible in v1.9.3 (latest commit) as well.<o:p></o:p></span></p><p><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'>This is a tricky one. It happens because you’re setting key on </span><code><span style='font-size:10.0pt'>.SD</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> which should normally not be allowed. What happens is, when you set key the first time, there’s no key set (here) and therefore key is set on all the columns </span><code><span style='font-size:10.0pt'>x1</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'>, </span><code><span style='font-size:10.0pt'>x2</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> and </span><code><span style='font-size:10.0pt'>x3</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'>.<o:p></o:p></span></p><p><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'>Now, the next group (in the </span><code><span style='font-size:10.0pt'>by=.</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'>) is passed to your function, it’ll have the </span><code><span style='font-size:10.0pt'>key</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> already set to </span><code><span style='font-size:10.0pt'>x1,x2,x3</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> (because </span><code><span style='font-size:10.0pt'>setkey</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> modifies the object by reference), but </span><code><span style='font-size:10.0pt'>.SD</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> has obtained <strong><span style='font-family:"Helvetica","sans-serif"'>new</span></strong> data corresponding to <em><span style='font-family:"Helvetica","sans-serif"'>this</span></em> group. And </span><code><span style='font-size:10.0pt'>data.table</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> sorts this data, knowing that it already has key set.. but if the key is set then the order must be 1:n. But it wouldn’t be, as this data isn’t sorted. </span><code><span style='font-size:10.0pt'>data.table</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> warns in those scenarios.. and that’s why you get the warning.<o:p></o:p></span></p><p><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'>To verify this, you can try:<o:p></o:p></span></p><div style='border:solid #CCCCCC 1.0pt;padding:3.0pt 6.0pt 3.0pt 6.0pt'><pre style='background:#F8F8F8'><code>conflictsTable1 <- function(f, address) {</code><o:p></o:p></pre><pre style='background:#F8F8F8'><code> u <- unique(setkey(f))</code><o:p></o:p></pre><pre style='background:#F8F8F8'><code> setattr(f, 'sorted', NULL)</code><o:p></o:p></pre><pre style='background:#F8F8F8'><code> if (nrow(u) == 1) return(NULL)</code><o:p></o:p></pre><pre style='background:#F8F8F8'><code> u</code><o:p></o:p></pre><pre style='background:#F8F8F8'><code>}</code><o:p></o:p></pre></div><p><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'>Basically, we set the key of </span><code><span style='font-size:10.0pt'>f</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> (which is equal to </span><code><span style='font-size:10.0pt'>.SD</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> as it’s only modified by reference) to </span><code><span style='font-size:10.0pt'>NULL</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> everytime after.. so that </span><code><span style='font-size:10.0pt'>.SD</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> for the new group will not have the key set.<o:p></o:p></span></p><p><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'>The ideal scenario here, IIUC, is that </span><code><span style='font-size:10.0pt'>setkey(.SD)</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> or things pointing to </span><code><span style='font-size:10.0pt'>.SD</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> should not be possible (locking binding doesn’t seem to affect things done by reference..). </span><code><span style='font-size:10.0pt'>.SD</span></code><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> however should retain the key of the data.table, if a key was set, wherever possible.<o:p></o:p></span></p><div id="bloop_customfont"><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'> </span><o:p></o:p></p></div><div id="bloop_sign_1402704505278157056"><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'>Arun</span><o:p></o:p></p></div></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'><br>From: Ron Hylton <a href="mailto:rhylton@verizon.net">rhylton@verizon.net</a><br>Reply: Ron Hylton <a href="mailto:rhylton@verizon.net">rhylton@verizon.net</a><br>Date: June 14, 2014 at 1:55:53 AM<br>To: <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a> <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>Subject: [datatable-help] data.table is asking for help</span><o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;margin-bottom:12.0pt'><o:p> </o:p></p><blockquote style='margin-left:0in;margin-top:11.25pt;margin-right:0in;margin-bottom:11.25pt'><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>The code below generates the warning:<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;word-break:break-all'><span style='font-size:10.0pt;font-family:"Lucida Console";color:black;background:#E1E2E5'>In setkeyv(x, cols, verbose = verbose) :</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;word-break:break-all'><span style='font-size:10.0pt;font-family:"Lucida Console";color:black;background:#E1E2E5'> Already keyed by this key but had invalid row order, key rebuilt. If you didn't go under the hood please let datatable-help know so the root cause can be fixed.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;word-break:break-all'><span style='font-size:10.0pt;font-family:"Lucida Console";color:black;background:#E1E2E5'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>This is my first attempt at using datatable so I probably did something dumb, but maybe that‘s useful for someone. The first case is the one that gives the warnings.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I’m also surprised at the timings. I wrote the original algorithm using dataframe & ddply and I expected datatable to be substantially faster; the opposite is true.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>The algorithm does the following: Certain columns in the table are keys and others are values in the sense that each row with the same set of keys should have the same set of values. Find all the key sets for which this is not true and return the keys sets + conflicting value sets.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Insight into the performance would be appreciated.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Regards,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Ron<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>library(data.table)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>library(plyr)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>conflictsTable1 <- function(f) {<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> u <- unique(setkey(f))<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> if (nrow(u) == 1) return(NULL)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> u<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>}<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>conflictsTable2 <- function(f) {<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> u <- unique(f)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> if (nrow(u) == 1) return(NULL)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> u<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>}<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>conflictsFrame <- function(f) {<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> u <- unique(f)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> if (nrow(u) == 1) return(NULL)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> u<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>}<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>N <- 10000<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>test <- data.table(id=as.character(10000*sample(1:N,N,replace=TRUE)), x1=rnorm(N), x2=rnorm(N), x3=rnorm(N))<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>setkey(test,id)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>print(system.time(ut1 <- test[, conflictsTable1(.SD), by=id]))<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>print(system.time(ut2 <- test[, conflictsTable2(.SD), by=id]))<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>print(system.time(uf <- ddply(test, .(id), conflictsFrame)))<o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>_______________________________________________<br>datatable-help mailing list<br><a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br><a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><o:p></o:p></p></div></div></blockquote></div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'>_______________________________________________ <br>datatable-help mailing list <br><a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a> <br><a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><o:p></o:p></span></p></div></div></blockquote></div></body></html>