<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Lucida Console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
@font-face
{font-family:Monaco;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
h1
{mso-style-priority:9;
mso-style-link:"Heading 1 Char";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:28.0pt;
font-family:"Times New Roman","serif";
color:black;
font-weight:bold;}
h2
{mso-style-priority:9;
mso-style-link:"Heading 2 Char";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
border:none;
padding:0in;
font-size:18.0pt;
font-family:"Times New Roman","serif";
color:black;
font-weight:bold;}
h3
{mso-style-priority:9;
mso-style-link:"Heading 3 Char";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:13.5pt;
font-family:"Times New Roman","serif";
font-weight:bold;}
h4
{mso-style-priority:9;
mso-style-link:"Heading 4 Char";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";
font-weight:bold;}
h5
{mso-style-priority:9;
mso-style-link:"Heading 5 Char";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:10.5pt;
font-family:"Times New Roman","serif";
font-weight:bold;}
h6
{mso-style-priority:9;
mso-style-link:"Heading 6 Char";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:10.5pt;
font-family:"Times New Roman","serif";
color:#777777;
font-weight:bold;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#4183C4;
text-decoration:none none;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#4183C4;
text-decoration:none none;}
p
{mso-style-priority:99;
margin-top:11.25pt;
margin-right:0in;
margin-bottom:11.25pt;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
code
{mso-style-priority:99;
font-family:"Monaco","serif";
border:solid #EAEAEA 1.0pt;
padding:0in;
background:#F8F8F8;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin-top:11.25pt;
margin-right:0in;
margin-bottom:11.25pt;
margin-left:0in;
background:#F8F8F8;
border:none;
padding:0in;
font-size:10.0pt;
font-family:"Monaco","serif";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.Heading1Char
{mso-style-name:"Heading 1 Char";
mso-style-priority:9;
mso-style-link:"Heading 1";
font-family:"Calibri Light","sans-serif";
color:#2E74B5;}
span.Heading2Char
{mso-style-name:"Heading 2 Char";
mso-style-priority:9;
mso-style-link:"Heading 2";
font-family:"Calibri Light","sans-serif";
color:#2E74B5;}
span.Heading3Char
{mso-style-name:"Heading 3 Char";
mso-style-priority:9;
mso-style-link:"Heading 3";
font-family:"Calibri Light","sans-serif";
color:#1F4D78;}
span.Heading4Char
{mso-style-name:"Heading 4 Char";
mso-style-priority:9;
mso-style-link:"Heading 4";
font-family:"Calibri Light","sans-serif";
color:#2E74B5;
font-style:italic;}
span.Heading5Char
{mso-style-name:"Heading 5 Char";
mso-style-priority:9;
mso-style-link:"Heading 5";
font-family:"Calibri Light","sans-serif";
color:#2E74B5;}
span.Heading6Char
{mso-style-name:"Heading 6 Char";
mso-style-priority:9;
mso-style-link:"Heading 6";
font-family:"Calibri Light","sans-serif";
color:#1F4D78;}
p.send, li.send, div.send
{mso-style-name:send;
margin-top:11.25pt;
margin-right:0in;
margin-bottom:11.25pt;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:#77BB77;}
p.server, li.server, div.server
{mso-style-name:server;
margin-top:11.25pt;
margin-right:0in;
margin-bottom:11.25pt;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:#7799BB;}
p.error, li.error, div.error
{mso-style-name:error;
margin-top:11.25pt;
margin-right:0in;
margin-bottom:11.25pt;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:#AA0000;}
span.EmailStyle32
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#4183C4" vlink="#4183C4"><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I suspected it was something like this. As one clarification, there is a setkey(test,id) before any setkey(.SD). If setkey(test,id) is changed to setkey(test) so all columns are in the original datatable key then the warning goes away.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>However there’s another aspect. While I’m relatively new to R my understanding is that a function argument should be modifiable within the function body without affecting the caller, which perhaps conflicts with the behavior of .SD.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> Arunkumar Srinivasan [mailto:aragorn168b@gmail.com] <br><b>Sent:</b> Friday, June 13, 2014 8:23 PM<br><b>To:</b> Ron Hylton; datatable-help@lists.r-forge.r-project.org<br><b>Subject:</b> Re: [datatable-help] data.table is asking for help<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p>Nicely reproducible post. Reproducible in v1.9.3 (latest commit) as well.<o:p></o:p></p><p>This is a tricky one. It happens because you’re setting key on <code><span style='font-size:10.0pt'>.SD</span></code> which should normally not be allowed. What happens is, when you set key the first time, there’s no key set (here) and therefore key is set on all the columns <code><span style='font-size:10.0pt'>x1</span></code>, <code><span style='font-size:10.0pt'>x2</span></code> and <code><span style='font-size:10.0pt'>x3</span></code>. <o:p></o:p></p><p>Now, the next group (in the <code><span style='font-size:10.0pt'>by=.</span></code>) is passed to your function, it’ll have the <code><span style='font-size:10.0pt'>key</span></code> already set to <code><span style='font-size:10.0pt'>x1,x2,x3</span></code> (because <code><span style='font-size:10.0pt'>setkey</span></code> modifies the object by reference), but <code><span style='font-size:10.0pt'>.SD</span></code> has obtained <strong>new</strong> data corresponding to <em>this</em> group. And <code><span style='font-size:10.0pt'>data.table</span></code> sorts this data, knowing that it already has key set.. but if the key is set then the order must be 1:n. But it wouldn’t be, as this data isn’t sorted. <code><span style='font-size:10.0pt'>data.table</span></code> warns in those scenarios.. and that’s why you get the warning. <o:p></o:p></p><p>To verify this, you can try:<o:p></o:p></p><div style='mso-element:para-border-div;border:solid #CCCCCC 1.0pt;padding:3.0pt 6.0pt 3.0pt 6.0pt;background:#F8F8F8'><pre><code>conflictsTable1 <- function(f, address) {<o:p></o:p></code></pre><pre><code> u <- unique(setkey(f))<o:p></o:p></code></pre><pre><code> setattr(f, 'sorted', NULL)<o:p></o:p></code></pre><pre><code> if (nrow(u) == 1) return(NULL)<o:p></o:p></code></pre><pre><code> u<o:p></o:p></code></pre><pre><code>}<o:p></o:p></code></pre></div><p>Basically, we set the key of <code><span style='font-size:10.0pt'>f</span></code> (which is equal to <code><span style='font-size:10.0pt'>.SD</span></code> as it’s only modified by reference) to <code><span style='font-size:10.0pt'>NULL</span></code> everytime after.. so that <code><span style='font-size:10.0pt'>.SD</span></code> for the new group will not have the key set.<o:p></o:p></p><p>The ideal scenario here, IIUC, is that <code><span style='font-size:10.0pt'>setkey(.SD)</span></code> or things pointing to <code><span style='font-size:10.0pt'>.SD</span></code> should not be possible (locking binding doesn’t seem to affect things done by reference..). <code><span style='font-size:10.0pt'>.SD</span></code> however should retain the key of the data.table, if a key was set, wherever possible.<o:p></o:p></p><div id="bloop_customfont"><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'><o:p> </o:p></span></p></div><div id="bloop_sign_1402704505278157056"><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Helvetica","sans-serif"'>Arun<o:p></o:p></span></p></div></div><div><p class=MsoNormal><span style='color:black'><br>From: Ron Hylton <a href="mailto:rhylton@verizon.net">rhylton@verizon.net</a><br>Reply: Ron Hylton <a href="mailto:rhylton@verizon.net">rhylton@verizon.net</a><br>Date: June 14, 2014 at 1:55:53 AM<br>To: <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a> <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>Subject: [datatable-help] data.table is asking for help <o:p></o:p></span></p></div><p class=MsoNormal><br><br><o:p></o:p></p><blockquote style='margin-left:0in;margin-top:11.25pt;margin-right:0in;margin-bottom:11.25pt'><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>The code below generates the warning:<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;word-break:break-all'><span style='font-size:10.0pt;font-family:"Lucida Console";color:black;background:#E1E2E5'>In setkeyv(x, cols, verbose = verbose) :</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;word-break:break-all'><span style='font-size:10.0pt;font-family:"Lucida Console";color:black;background:#E1E2E5'> Already keyed by this key but had invalid row order, key rebuilt. If you didn't go under the hood please let datatable-help know so the root cause can be fixed.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;word-break:break-all'><span style='font-size:10.0pt;font-family:"Lucida Console";color:black;background:#E1E2E5'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>This is my first attempt at using datatable so I probably did something dumb, but maybe that‘s useful for someone. The first case is the one that gives the warnings.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I’m also surprised at the timings. I wrote the original algorithm using dataframe & ddply and I expected datatable to be substantially faster; the opposite is true.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>The algorithm does the following: Certain columns in the table are keys and others are values in the sense that each row with the same set of keys should have the same set of values. Find all the key sets for which this is not true and return the keys sets + conflicting value sets.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Insight into the performance would be appreciated.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Regards,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Ron<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>library(data.table)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>library(plyr)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>conflictsTable1 <- function(f) {<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> u <- unique(setkey(f))<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> if (nrow(u) == 1) return(NULL)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> u<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>}<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>conflictsTable2 <- function(f) {<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> u <- unique(f)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> if (nrow(u) == 1) return(NULL)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> u<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>}<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>conflictsFrame <- function(f) {<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> u <- unique(f)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> if (nrow(u) == 1) return(NULL)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> u<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>}<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>N <- 10000<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>test <- data.table(id=as.character(10000*sample(1:N,N,replace=TRUE)), x1=rnorm(N), x2=rnorm(N), x3=rnorm(N))<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>setkey(test,id)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>print(system.time(ut1 <- test[, conflictsTable1(.SD), by=id]))<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>print(system.time(ut2 <- test[, conflictsTable2(.SD), by=id]))<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>print(system.time(uf <- ddply(test, .(id), conflictsFrame)))<o:p></o:p></p></div><p class=MsoNormal>_______________________________________________ <br>datatable-help mailing list <br><a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a> <br><a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><o:p></o:p></p></div></div></blockquote></div></body></html>