<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix"><br>
      Hi Ron,<br>
      <br>
      Thanks for highlighting this.  Two changes now in v1.9.3 on
      GitHub:<br>
      <br>
      <meta http-equiv="content-type" content="text/html;
        charset=ISO-8859-1">
      <ul class="task-list" style="box-sizing: border-box; padding: 0px
        0px 0px 30px; margin: 15px 0px; color: rgb(51, 51, 51);
        font-family: Helvetica, arial, freesans, clean, sans-serif;
        font-size: 15px; font-style: normal; font-variant: normal;
        font-weight: normal; letter-spacing: normal; line-height:
        25.5px; orphans: auto; text-align: start; text-indent: 0px;
        text-transform: none; white-space: normal; widows: auto;
        word-spacing: 0px; -webkit-text-stroke-width: 0px;">
        <li style="box-sizing: border-box;">
          <p style="box-sizing: border-box; margin: 15px 0px;"><code
              style="box-sizing: border-box; font-family: Consolas,
              'Liberation Mono', Menlo, Courier, monospace; font-size:
              12px; margin: 0px; border: 1px solid rgb(221, 221, 221);
              background-color: rgb(248, 248, 248);
              border-top-left-radius: 3px; border-top-right-radius: 3px;
              border-bottom-right-radius: 3px;
              border-bottom-left-radius: 3px; padding: 0px;">setkey</code><span
              class="Apple-converted-space"> </span>on<span
              class="Apple-converted-space"> </span><code
              style="box-sizing: border-box; font-family: Consolas,
              'Liberation Mono', Menlo, Courier, monospace; font-size:
              12px; margin: 0px; border: 1px solid rgb(221, 221, 221);
              background-color: rgb(248, 248, 248);
              border-top-left-radius: 3px; border-top-right-radius: 3px;
              border-bottom-right-radius: 3px;
              border-bottom-left-radius: 3px; padding: 0px;">.SD</code><span
              class="Apple-converted-space"> </span>is now an error,
            rather than warnings for each group about rebuilding the
            key. The new error is similar to when attempting to use<span
              class="Apple-converted-space"> </span><code
              style="box-sizing: border-box; font-family: Consolas,
              'Liberation Mono', Menlo, Courier, monospace; font-size:
              12px; margin: 0px; border: 1px solid rgb(221, 221, 221);
              background-color: rgb(248, 248, 248);
              border-top-left-radius: 3px; border-top-right-radius: 3px;
              border-bottom-right-radius: 3px;
              border-bottom-left-radius: 3px; padding: 0px;">:=</code><span
              class="Apple-converted-space"> </span>in a<span
              class="Apple-converted-space"> </span><code
              style="box-sizing: border-box; font-family: Consolas,
              'Liberation Mono', Menlo, Courier, monospace; font-size:
              12px; margin: 0px; border: 1px solid rgb(221, 221, 221);
              background-color: rgb(248, 248, 248);
              border-top-left-radius: 3px; border-top-right-radius: 3px;
              border-bottom-right-radius: 3px;
              border-bottom-left-radius: 3px; padding: 0px;">.SD</code><span
              class="Apple-converted-space"> </span>subquery:<span
              class="Apple-converted-space"> </span><code
              style="box-sizing: border-box; font-family: Consolas,
              'Liberation Mono', Menlo, Courier, monospace; font-size:
              12px; margin: 0px; border: 1px solid rgb(221, 221, 221);
              background-color: rgb(248, 248, 248);
              border-top-left-radius: 3px; border-top-right-radius: 3px;
              border-bottom-right-radius: 3px;
              border-bottom-left-radius: 3px; padding: 0px;">".SD is
              locked. Using set*() functions on .SD is reserved for
              possible future use; a tortuously flexible way to modify
              the original data by group."</code><span
              class="Apple-converted-space"> </span>Thanks to Ron Hylton
            for highlighting the issue on datatable-help<span
              class="Apple-converted-space"> </span><a
href="http://r.789695.n4.nabble.com/data-table-is-asking-for-help-tp4692080.html"
              style="box-sizing: border-box; color: rgb(65, 131, 196);
              text-decoration: none;">here</a>.</p>
        </li>
        <li style="box-sizing: border-box;">
          <p style="box-sizing: border-box; margin: 15px 0px;">Looping
            calls to<span class="Apple-converted-space"> </span><code
              style="box-sizing: border-box; font-family: Consolas,
              'Liberation Mono', Menlo, Courier, monospace; font-size:
              12px; margin: 0px; border: 1px solid rgb(221, 221, 221);
              background-color: rgb(248, 248, 248);
              border-top-left-radius: 3px; border-top-right-radius: 3px;
              border-bottom-right-radius: 3px;
              border-bottom-left-radius: 3px; padding: 0px;">unique(DT)</code><span
              class="Apple-converted-space"> </span>such as in<span
              class="Apple-converted-space"> </span><code
              style="box-sizing: border-box; font-family: Consolas,
              'Liberation Mono', Menlo, Courier, monospace; font-size:
              12px; margin: 0px; border: 1px solid rgb(221, 221, 221);
              background-color: rgb(248, 248, 248);
              border-top-left-radius: 3px; border-top-right-radius: 3px;
              border-bottom-right-radius: 3px;
              border-bottom-left-radius: 3px; padding: 0px;">DT[,unique(.SD),by=group]</code><span
              class="Apple-converted-space"> </span>is now faster by
            avoiding internal overhead of calling<span
              class="Apple-converted-space"> </span><code
              style="box-sizing: border-box; font-family: Consolas,
              'Liberation Mono', Menlo, Courier, monospace; font-size:
              12px; margin: 0px; border: 1px solid rgb(221, 221, 221);
              background-color: rgb(248, 248, 248);
              border-top-left-radius: 3px; border-top-right-radius: 3px;
              border-bottom-right-radius: 3px;
              border-bottom-left-radius: 3px; padding: 0px;">[.data.table</code>.
            Thanks again to Ron Hylton for highlighting in the<span
              class="Apple-converted-space"> </span><a
href="http://r.789695.n4.nabble.com/data-table-is-asking-for-help-tp4692080.html"
              style="box-sizing: border-box; color: rgb(65, 131, 196);
              text-decoration: none;">same thread</a>. His example is
            reduced from 28 sec to 9 sec, with identical results.</p>
        </li>
      </ul>
      <br>
      I now get the following (on my slow netbook) with no changes to
      your code.<br>
      <br>
      print(system.time(ut1 <- test[, conflictsTable1(.SD),
      by=id]))   #  were warnings,    now error<br>
      print(system.time(ut2 <- test[, conflictsTable2(.SD),
      by=id]))   #  was 28s, now 9s<br>
      print(system.time(uf <- ddply(test, .(id), conflictsFrame)))  
      # 13s<br>
      <br>
      This just fixes the surprises, basically.   Clearly Arun uses
      data.table in a better way which is orders of magnitude faster.<br>
      <br>
      Matt<br>
      <br>
      <br>
      On 14/06/14 03:58, Ron Hylton wrote:<br>
    </div>
    <blockquote cite="mid:008301cf877c$7dc9def0$795d9cd0$@verizon.net"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=ISO-8859-1">
      <meta name="Generator" content="Microsoft Word 15 (filtered
        medium)">
      <!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
      <style><!--
/* Font Definitions */
@font-face
        {font-family:Helvetica;
        panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
        {font-family:Consolas;
        panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
        {font-family:"Lucida Console";
        panose-1:2 11 6 9 4 5 4 2 2 4;}
@font-face
        {font-family:Monaco;
        panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";}
h1
        {mso-style-priority:9;
        mso-style-link:"Heading 1 Char";
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:28.0pt;
        font-family:"Times New Roman","serif";
        color:black;
        font-weight:bold;}
h2
        {mso-style-priority:9;
        mso-style-link:"Heading 2 Char";
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        border:none;
        padding:0in;
        font-size:18.0pt;
        font-family:"Times New Roman","serif";
        color:black;
        font-weight:bold;}
h3
        {mso-style-priority:9;
        mso-style-link:"Heading 3 Char";
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:13.5pt;
        font-family:"Times New Roman","serif";
        font-weight:bold;}
h4
        {mso-style-priority:9;
        mso-style-link:"Heading 4 Char";
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";
        font-weight:bold;}
h5
        {mso-style-priority:9;
        mso-style-link:"Heading 5 Char";
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:10.5pt;
        font-family:"Times New Roman","serif";
        font-weight:bold;}
h6
        {mso-style-priority:9;
        mso-style-link:"Heading 6 Char";
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:10.5pt;
        font-family:"Times New Roman","serif";
        color:#777777;
        font-weight:bold;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:#4183C4;
        text-decoration:none none;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:#4183C4;
        text-decoration:none none;}
p
        {mso-style-priority:99;
        margin-top:11.25pt;
        margin-right:0in;
        margin-bottom:11.25pt;
        margin-left:0in;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";}
code
        {mso-style-priority:99;
        font-family:"Monaco","serif";
        border:solid #EAEAEA 1.0pt;
        padding:0in;
        background:#F8F8F8;}
pre
        {mso-style-priority:99;
        mso-style-link:"HTML Preformatted Char";
        margin-top:11.25pt;
        margin-right:0in;
        margin-bottom:11.25pt;
        margin-left:0in;
        background:#F8F8F8;
        border:none;
        padding:0in;
        font-size:10.0pt;
        font-family:"Monaco","serif";}
span.HTMLPreformattedChar
        {mso-style-name:"HTML Preformatted Char";
        mso-style-priority:99;
        mso-style-link:"HTML Preformatted";
        font-family:Consolas;}
span.Heading1Char
        {mso-style-name:"Heading 1 Char";
        mso-style-priority:9;
        mso-style-link:"Heading 1";
        font-family:"Calibri Light","sans-serif";
        color:#2E74B5;}
span.Heading2Char
        {mso-style-name:"Heading 2 Char";
        mso-style-priority:9;
        mso-style-link:"Heading 2";
        font-family:"Calibri Light","sans-serif";
        color:#2E74B5;}
span.Heading3Char
        {mso-style-name:"Heading 3 Char";
        mso-style-priority:9;
        mso-style-link:"Heading 3";
        font-family:"Calibri Light","sans-serif";
        color:#1F4D78;}
span.Heading4Char
        {mso-style-name:"Heading 4 Char";
        mso-style-priority:9;
        mso-style-link:"Heading 4";
        font-family:"Calibri Light","sans-serif";
        color:#2E74B5;
        font-style:italic;}
span.Heading5Char
        {mso-style-name:"Heading 5 Char";
        mso-style-priority:9;
        mso-style-link:"Heading 5";
        font-family:"Calibri Light","sans-serif";
        color:#2E74B5;}
span.Heading6Char
        {mso-style-name:"Heading 6 Char";
        mso-style-priority:9;
        mso-style-link:"Heading 6";
        font-family:"Calibri Light","sans-serif";
        color:#1F4D78;}
p.send, li.send, div.send
        {mso-style-name:send;
        margin-top:11.25pt;
        margin-right:0in;
        margin-bottom:11.25pt;
        margin-left:0in;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";
        color:#77BB77;}
p.server, li.server, div.server
        {mso-style-name:server;
        margin-top:11.25pt;
        margin-right:0in;
        margin-bottom:11.25pt;
        margin-left:0in;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";
        color:#7799BB;}
p.error, li.error, div.error
        {mso-style-name:error;
        margin-top:11.25pt;
        margin-right:0in;
        margin-bottom:11.25pt;
        margin-left:0in;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";
        color:#AA0000;}
span.EmailStyle32
        {mso-style-type:personal-reply;
        font-family:"Calibri","sans-serif";
        color:#1F497D;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
      <div class="WordSection1">
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks,
            that very helpful.<o:p></o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
        <div>
          <div style="border:none;border-top:solid #E1E1E1
            1.0pt;padding:3.0pt 0in 0in 0in">
            <p class="MsoNormal"><b><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">
                Arunkumar Srinivasan [<a class="moz-txt-link-freetext" href="mailto:aragorn168b@gmail.com">mailto:aragorn168b@gmail.com</a>] <br>
                <b>Sent:</b> Friday, June 13, 2014 10:46 PM<br>
                <b>To:</b> Ron Hylton;
                <a class="moz-txt-link-abbreviated" href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
                <b>Subject:</b> Re: [datatable-help] data.table is
                asking for help<o:p></o:p></span></p>
          </div>
        </div>
        <p class="MsoNormal"><o:p> </o:p></p>
        <p>Sorry. But we can simplify it even further:<o:p></o:p></p>
        <p>The first step is just <code><span style="font-size:10.0pt">unique(test)</span></code>.
          So, we can do:<o:p></o:p></p>
        <div style="mso-element:para-border-div;border:solid #CCCCCC
          1.0pt;padding:3.0pt 6.0pt 3.0pt 6.0pt;background:#F8F8F8">
          <pre><code>system.time({<o:p></o:p></code></pre>
          <pre><code>ans = unique(test)<o:p></o:p></code></pre>
          <pre><code>ans = ans[ans[, .I[.N > 1L], by=id]$V1]<o:p></o:p></code></pre>
          <pre><code>})<o:p></o:p></code></pre>
          <pre><code>#  0.016   0.000   0.016  <o:p></o:p></code></pre>
        </div>
        <p>Identical?<o:p></o:p></p>
        <div style="mso-element:para-border-div;border:solid #CCCCCC
          1.0pt;padding:3.0pt 6.0pt 3.0pt 6.0pt;background:#F8F8F8">
          <pre><code>setkey(ans)<o:p></o:p></code></pre>
          <pre><code>setkey(ut1)<o:p></o:p></code></pre>
          <pre><code>identical(ans, ut1) # [1] TRUE<o:p></o:p></code></pre>
        </div>
        <div id="bloop_customfont">
          <p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
        </div>
        <div id="bloop_sign_1402713822374628096">
          <div>
            <p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Arun<o:p></o:p></span></p>
          </div>
        </div>
        <div>
          <p class="MsoNormal"><span style="color:black"><br>
              From: Arunkumar Srinivasan <a moz-do-not-send="true"
                href="mailto:aragorn168b@gmail.com">aragorn168b@gmail.com</a><br>
              Reply: Arunkumar Srinivasan <a moz-do-not-send="true"
                href="mailto:aragorn168b@gmail.com">aragorn168b@gmail.com</a><br>
              Date: June 14, 2014 at 4:42:31 AM<br>
              To: Ron Hylton <a moz-do-not-send="true"
                href="mailto:rhylton@verizon.net">rhylton@verizon.net</a>,
              <a moz-do-not-send="true"
                href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a>
              <a moz-do-not-send="true"
                href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
              Subject:  Re: [datatable-help] data.table is asking for
              help <o:p></o:p></span></p>
        </div>
        <p class="MsoNormal"><br>
          <br>
          <o:p></o:p></p>
        <blockquote
style="margin-left:0in;margin-top:11.25pt;margin-right:0in;margin-bottom:11.25pt">
          <div>
            <div>
              <p>A slightly simpler version of the 2nd solution is:<o:p></o:p></p>
              <div style="mso-element:para-border-div;border:solid
                #CCCCCC 1.0pt;padding:3.0pt 6.0pt 3.0pt
                6.0pt;background:#F8F8F8;margin-left:1.5pt;margin-right:1.5pt">
                <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>system.time({<o:p></o:p></code></pre>
                <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>ans = test[, .N, by=names(test)]<o:p></o:p></code></pre>
                <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>ans = ans[ans[, .I[.N > 1L], by=id]$V1]<o:p></o:p></code></pre>
                <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>})<o:p></o:p></code></pre>
                <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>#  0.019   0.000   0.019   <o:p></o:p></code></pre>
              </div>
              <div style="mso-element:para-border-div;border:solid
                #CCCCCC 1.0pt;padding:3.0pt 6.0pt 3.0pt
                6.0pt;background:#F8F8F8">
                <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><o:p> </o:p></pre>
              </div>
              <p>The answers are identical, you can check this by doing:<o:p></o:p></p>
              <div style="mso-element:para-border-div;border:solid
                #CCCCCC 1.0pt;padding:3.0pt 6.0pt 3.0pt
                6.0pt;background:#F8F8F8;margin-left:1.5pt;margin-right:1.5pt">
                <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>ans[, N := NULL]<o:p></o:p></code></pre>
                <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>setkey(ans)<o:p></o:p></code></pre>
                <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>setkey(ut1)<o:p></o:p></code></pre>
                <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>identical(ans, ut1) # [1] TRUE<o:p></o:p></code></pre>
              </div>
              <div style="mso-element:para-border-div;border:solid
                #CCCCCC 1.0pt;padding:3.0pt 6.0pt 3.0pt
                6.0pt;background:#F8F8F8">
                <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><o:p> </o:p></pre>
              </div>
              <div id="bloop_customfont">
                <p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
              </div>
              <div id="bloop_sign_1402713543682700032">
                <div>
                  <p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Arun<o:p></o:p></span></p>
                </div>
              </div>
              <div>
                <p class="MsoNormal"><span style="color:black"><br>
                    From: Arunkumar Srinivasan <a
                      moz-do-not-send="true"
                      href="mailto:aragorn168b@gmail.com">aragorn168b@gmail.com</a><br>
                    Reply: Arunkumar Srinivasan <a
                      moz-do-not-send="true"
                      href="mailto:aragorn168b@gmail.com">aragorn168b@gmail.com</a><br>
                    Date: June 14, 2014 at 4:34:15 AM<br>
                    To: Ron Hylton <a moz-do-not-send="true"
                      href="mailto:rhylton@verizon.net">rhylton@verizon.net</a>,
                    <a moz-do-not-send="true"
                      href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a>
                    <a moz-do-not-send="true"
                      href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
                    Subject:  Re: [datatable-help] data.table is asking
                    for help<o:p></o:p></span></p>
              </div>
              <p class="MsoNormal"><br>
                <br>
                <o:p></o:p></p>
              <blockquote
style="margin-left:0in;margin-top:11.25pt;margin-right:0in;margin-bottom:11.25pt">
                <div>
                  <div>
                    <p style="margin:0in;margin-bottom:.0001pt">The
                      j-expression is evaluated from within C for each
                      group (unless they’re optimised with GForce - a
                      new initiative in data.table). And <code><span
                          style="font-size:10.0pt">eval(.SD)</span></code>
                      or <code><span style="font-size:10.0pt">eval(anything(.SD))</span></code>
                      is costly.<o:p></o:p></p>
                    <p style="margin:0in;margin-bottom:.0001pt">You can
                      get around it by listing the columns by yourself
                      and using <code><span style="font-size:10.0pt">.I</span></code>
                      instead, as follows:<o:p></o:p></p>
                    <div style="mso-element:para-border-div;border:solid
                      #CCCCCC 1.0pt;padding:3.0pt 6.0pt 3.0pt
                      6.0pt;background:#F8F8F8;margin-left:1.5pt;margin-right:1.5pt">
                      <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>test[test[, .I[length(unique(list(x1,x2,x3))[[1L]]) > 1L], by=id]$V1]<o:p></o:p></code></pre>
                      <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>#  0.140   0.001   0.142    <o:p></o:p></code></pre>
                    </div>
                    <div style="mso-element:para-border-div;border:solid
                      #CCCCCC 1.0pt;padding:3.0pt 6.0pt 3.0pt
                      6.0pt;background:#F8F8F8">
                      <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><o:p> </o:p></pre>
                      <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><o:p> </o:p></pre>
                    </div>
                    <p>Takes about 0.14 seconds.<o:p></o:p></p>
                    <div class="MsoNormal" style="text-align:center"
                      align="center">
                      <hr style="color:#CCCCCC" align="center"
                        noshade="noshade" size="3" width="100%"></div>
                    <p>An even faster way is:<o:p></o:p></p>
                    <div style="mso-element:para-border-div;border:solid
                      #CCCCCC 1.0pt;padding:3.0pt 6.0pt 3.0pt
                      6.0pt;background:#F8F8F8;margin-left:1.5pt;margin-right:1.5pt">
                      <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>system.time({<o:p></o:p></code></pre>
                      <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>ans = test[test[, .I[.N > 1], by=id]$V1]        # (1)    <o:p></o:p></code></pre>
                      <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>ans = ans[, .N, by=names(ans)]                  # (2)    <o:p></o:p></code></pre>
                      <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>ans = ans[ans[, .I[.N > 1L], by=id]$V1]         # (3)<o:p></o:p></code></pre>
                      <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>})<o:p></o:p></code></pre>
                      <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code><o:p> </o:p></code></pre>
                      <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>#  0.026   0.000   0.027    <o:p></o:p></code></pre>
                    </div>
                    <div style="mso-element:para-border-div;border:solid
                      #CCCCCC 1.0pt;padding:3.0pt 6.0pt 3.0pt
                      6.0pt;background:#F8F8F8">
                      <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><o:p> </o:p></pre>
                      <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><o:p> </o:p></pre>
                    </div>
                    <p>The idea for the second case is:<o:p></o:p></p>
                    <p style="margin:0in;margin-bottom:.0001pt">(1)
                      remove all entries where there’s just 1 row
                      corresponding to that <code><span
                          style="font-size:10.0pt">id</span></code>.<br>
                      (2) Aggregate this result by all the columns now
                      and get the number of rows in the column <code><span
                          style="font-size:10.0pt">N</span></code> (we
                      won’t have to use this column though).<br>
                      (3) Now, if we aggregate by <code><span
                          style="font-size:10.0pt">id</span></code> and
                      if any id has just 1 row, then it’d mean that that
                      <code><span style="font-size:10.0pt">id</span></code>
                      has had more than 1 rows (step (1) filtering
                      ensures this), but all of them are same and we
                      don’t need them. So we just filter for those where
                      .N > 1L.<o:p></o:p></p>
                    <p>HTH<o:p></o:p></p>
                    <div id="bloop_customfont">
                      <p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
                    </div>
                    <div id="bloop_sign_1402709866978106112">
                      <div>
                        <p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Arun<o:p></o:p></span></p>
                      </div>
                    </div>
                    <div>
                      <p class="MsoNormal"><span style="color:black"><br>
                          From: Ron Hylton <a moz-do-not-send="true"
                            href="mailto:rhylton@verizon.net">rhylton@verizon.net</a><br>
                          Reply: Ron Hylton <a moz-do-not-send="true"
                            href="mailto:rhylton@verizon.net">rhylton@verizon.net</a><br>
                          Date: June 14, 2014 at 3:30:55 AM<br>
                          To: <a moz-do-not-send="true"
                            href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a>
                          <a moz-do-not-send="true"
                            href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
                          Subject:  Re: [datatable-help] data.table is
                          asking for help<o:p></o:p></span></p>
                    </div>
                    <p class="MsoNormal"><br>
                      <br>
                      <o:p></o:p></p>
                    <blockquote
style="margin-left:0in;margin-top:11.25pt;margin-right:0in;margin-bottom:11.25pt">
                      <div>
                        <div>
                          <div>
                            <p class="MsoNormal"
                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The
                                performance is what puzzles me; the
                                results are correct so the warnings
                                don’t matter, and not all the variations
                                I’ve tried have warnings.  On the real
                                dataset (~800,000 rows) datatable takes
                                about 1.5 times longer than dataframe +
                                ddply.  I expected it to be
                                substantially faster.</span><o:p></o:p></p>
                            <p class="MsoNormal"
                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
                            <div>
                              <div style="border:none;border-top:solid
                                #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
                                <p class="MsoNormal"
                                  style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b>
                                  <span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Arunkumar
                                    Srinivasan [<a
                                      moz-do-not-send="true"
                                      href="mailto:aragorn168b@gmail.com">mailto:aragorn168b@gmail.com</a>]<br>
                                    <b>Sent:</b> Friday, June 13, 2014
                                    8:57 PM<br>
                                    <b>To:</b> Ron Hylton; <a
                                      moz-do-not-send="true"
                                      href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
                                    <b>Subject:</b> Re: [datatable-help]
                                    data.table is asking for help</span><o:p></o:p></p>
                              </div>
                            </div>
                            <p class="MsoNormal"
                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                            <div id="bloop_customfont">
                              <blockquote
style="margin-left:0in;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
                                <div>
                                  <div>
                                    <p class="MsoNormal"
                                      style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">However
                                        there’s another aspect.  While
                                        I’m relatively new to R my
                                        understanding is that a function
                                        argument should be modifiable
                                        within the function body without
                                        affecting the caller, which
                                        perhaps conflicts with the
                                        behavior of .SD.</span><o:p></o:p></p>
                                  </div>
                                </div>
                              </blockquote>
                              <div>
                                <div>
                                  <div>
                                    <p class="MsoNormal"
                                      style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-family:"Helvetica","sans-serif"">`data.table`
                                        is designed for working with
                                        *really large* data sets in mind
                                        (> 100 or 200 GB in memory
                                        even). And therefore, as a
                                        design feature, it trades in
                                        "referential transparency" for
                                        manipulating data objects *as
                                        efficient as possible* in terms
                                        of both *speed* and *memory
                                        usage* (most of the times they
                                        go hand-in-hand).</span><o:p></o:p></p>
                                    <p class="MsoNormal"
                                      style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-family:"Helvetica","sans-serif"">This is
                                        perhaps the biggest design
                                        choice one needs to be aware of
                                        when working/choosing
                                        data.tables. It is possible to
                                        modify objects by reference
                                        using data.table - All the
                                        functions that begin with "set*"
                                        modify objects by reference. The
                                        only other non "set*" function
                                        is `:=` operator.</span><o:p></o:p></p>
                                    <p class="MsoNormal"
                                      style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                    <p class="MsoNormal"
                                      style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-family:"Helvetica","sans-serif"">HTH</span><o:p></o:p></p>
                                    <p class="MsoNormal"
                                      style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black">Arun</span><o:p></o:p></p>
                                  </div>
                                </div>
                              </div>
                            </div>
                            <div>
                              <p class="MsoNormal"
                                style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black"><br>
                                  From: Ron Hylton <a
                                    moz-do-not-send="true"
                                    href="mailto:rhylton@verizon.net">rhylton@verizon.net</a><br>
                                  Reply: Ron Hylton <a
                                    moz-do-not-send="true"
                                    href="mailto:rhylton@verizon.net">rhylton@verizon.net</a><br>
                                  Date: June 14, 2014 at 2:52:04 AM<br>
                                  To: <a moz-do-not-send="true"
                                    href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a>
                                  <a moz-do-not-send="true"
                                    href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
                                  Subject:  Re: [datatable-help]
                                  data.table is asking for help</span><o:p></o:p></p>
                            </div>
                            <p class="MsoNormal"
                              style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><o:p> </o:p></p>
                            <blockquote
style="margin-left:0in;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
                              <div>
                                <div>
                                  <div>
                                    <p class="MsoNormal"
                                      style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I
                                        suspected it was something like
                                        this.  As one clarification,
                                        there is a setkey(test,id)
                                        before any setkey(.SD).   If
                                        setkey(test,id) is changed to
                                        setkey(test) so all columns are
                                        in the original datatable key
                                        then the warning goes away.</span><o:p></o:p></p>
                                    <p class="MsoNormal"
                                      style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
                                    <p class="MsoNormal"
                                      style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">However
                                        there’s another aspect.  While
                                        I’m relatively new to R my
                                        understanding is that a function
                                        argument should be modifiable
                                        within the function body without
                                        affecting the caller, which
                                        perhaps conflicts with the
                                        behavior of .SD.</span><o:p></o:p></p>
                                    <p class="MsoNormal"
                                      style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
                                    <div>
                                      <div
                                        style="border:none;border-top:solid
                                        #E1E1E1 1.0pt;padding:3.0pt 0in
                                        0in 0in">
                                        <p class="MsoNormal"
                                          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b>
                                          <span
style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Arunkumar
                                            Srinivasan [<a
                                              moz-do-not-send="true"
                                              href="mailto:aragorn168b@gmail.com">mailto:aragorn168b@gmail.com</a>]<br>
                                            <b>Sent:</b> Friday, June
                                            13, 2014 8:23 PM<br>
                                            <b>To:</b> Ron Hylton; <a
                                              moz-do-not-send="true"
                                              href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
                                            <b>Subject:</b> Re:
                                            [datatable-help] data.table
                                            is asking for help</span><o:p></o:p></p>
                                      </div>
                                    </div>
                                    <p class="MsoNormal"
                                      style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                    <p><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Nicely
                                        reproducible post. Reproducible
                                        in v1.9.3 (latest commit) as
                                        well.</span><o:p></o:p></p>
                                    <p
                                      style="margin:0in;margin-bottom:.0001pt"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">This
                                        is a tricky one. It happens
                                        because you’re setting key on</span>
                                      <code><span
                                          style="font-size:10.0pt">.SD</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">which
                                        should normally not be allowed.
                                        What happens is, when you set
                                        key the first time, there’s no
                                        key set (here) and therefore key
                                        is set on all the columns</span>
                                      <code><span
                                          style="font-size:10.0pt">x1</span></code><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">,</span>
                                      <code><span
                                          style="font-size:10.0pt">x2</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">and</span>
                                      <code><span
                                          style="font-size:10.0pt">x3</span></code><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">.</span><o:p></o:p></p>
                                    <p
                                      style="margin:0in;margin-bottom:.0001pt"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Now,
                                        the next group (in the</span> <code><span
                                          style="font-size:10.0pt">by=.</span></code><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">)
                                        is passed to your function,
                                        it’ll have the</span> <code><span
                                          style="font-size:10.0pt">key</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">already
                                        set to</span> <code><span
                                          style="font-size:10.0pt">x1,x2,x3</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">(because</span>
                                      <code><span
                                          style="font-size:10.0pt">setkey</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">modifies
                                        the object by reference), but</span>
                                      <code><span
                                          style="font-size:10.0pt">.SD</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">has
                                        obtained <strong><span
                                            style="font-family:"Helvetica","sans-serif"">new</span></strong>
                                        data corresponding to <em><span
style="font-family:"Helvetica","sans-serif"">this</span></em>
                                        group. And</span> <code><span
                                          style="font-size:10.0pt">data.table</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">sorts
                                        this data, knowing that it
                                        already has key set.. but if the
                                        key is set then the order must
                                        be 1:n. But it wouldn’t be, as
                                        this data isn’t sorted.</span> <code><span
                                          style="font-size:10.0pt">data.table</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">warns
                                        in those scenarios.. and that’s
                                        why you get the warning.</span><o:p></o:p></p>
                                    <p><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">To
                                        verify this, you can try:</span><o:p></o:p></p>
                                    <div style="border:solid #CCCCCC
                                      1.0pt;padding:3.0pt 6.0pt 3.0pt
                                      6.0pt">
                                      <div
                                        style="mso-element:para-border-div;border:solid
                                        #CCCCCC 1.0pt;padding:3.0pt
                                        6.0pt 3.0pt
                                        6.0pt;background:#F8F8F8">
                                        <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>conflictsTable1 <- function(f, address) {</code><o:p></o:p></pre>
                                        <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>  u <- unique(setkey(f))</code><o:p></o:p></pre>
                                        <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>  setattr(f, 'sorted', NULL)</code><o:p></o:p></pre>
                                        <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>  if (nrow(u) == 1) return(NULL)</code><o:p></o:p></pre>
                                        <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>  u</code><o:p></o:p></pre>
                                        <pre style="margin:0in;margin-bottom:.0001pt;background:#F8F8F8"><code>}</code><o:p></o:p></pre>
                                      </div>
                                    </div>
                                    <p
                                      style="margin:0in;margin-bottom:.0001pt"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Basically,
                                        we set the key of</span> <code><span
                                          style="font-size:10.0pt">f</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">(which
                                        is equal to</span> <code><span
                                          style="font-size:10.0pt">.SD</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">as
                                        it’s only modified by reference)
                                        to</span> <code><span
                                          style="font-size:10.0pt">NULL</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">everytime
                                        after.. so that</span> <code><span
                                          style="font-size:10.0pt">.SD</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">for
                                        the new group will not have the
                                        key set.</span><o:p></o:p></p>
                                    <p
                                      style="margin:0in;margin-bottom:.0001pt"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">The
                                        ideal scenario here, IIUC, is
                                        that</span> <code><span
                                          style="font-size:10.0pt">setkey(.SD)</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">or
                                        things pointing to</span> <code><span
                                          style="font-size:10.0pt">.SD</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">should
                                        not be possible (locking binding
                                        doesn’t seem to affect things
                                        done by reference..).</span> <code><span
                                          style="font-size:10.0pt">.SD</span></code>
                                      <span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">however
                                        should retain the key of the
                                        data.table, if a key was set,
                                        wherever possible.</span><o:p></o:p></p>
                                    <div id="bloop_customfont">
                                      <p class="MsoNormal"
                                        style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif""> </span><o:p></o:p></p>
                                    </div>
                                    <div
                                      id="bloop_sign_1402704505278157056">
                                      <div>
                                        <p class="MsoNormal"
                                          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Arun</span><o:p></o:p></p>
                                      </div>
                                    </div>
                                    <div>
                                      <p class="MsoNormal"
                                        style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
                                          style="color:black"><br>
                                          From: Ron Hylton <a
                                            moz-do-not-send="true"
                                            href="mailto:rhylton@verizon.net">rhylton@verizon.net</a><br>
                                          Reply: Ron Hylton <a
                                            moz-do-not-send="true"
                                            href="mailto:rhylton@verizon.net">rhylton@verizon.net</a><br>
                                          Date: June 14, 2014 at 1:55:53
                                          AM<br>
                                          To: <a moz-do-not-send="true"
href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a>
                                          <a moz-do-not-send="true"
                                            href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
                                          Subject:  [datatable-help]
                                          data.table is asking for help</span><o:p></o:p></p>
                                    </div>
                                    <p class="MsoNormal"
                                      style="mso-margin-top-alt:auto;margin-bottom:12.0pt"> <o:p></o:p></p>
                                    <blockquote
style="margin-left:0in;margin-top:11.25pt;margin-right:0in;margin-bottom:11.25pt">
                                      <div>
                                        <div>
                                          <div>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The
                                              code below generates the
                                              warning:<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;word-break:break-all"><span
                                                style="font-size:10.0pt;font-family:"Lucida
Console";color:black;background:#E1E2E5">In setkeyv(x, cols,
                                                verbose = verbose) :</span><o:p></o:p></p>
                                            <p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;word-break:break-all"><span
                                                style="font-size:10.0pt;font-family:"Lucida
Console";color:black;background:#E1E2E5">  Already keyed by this
                                                key but had invalid row
                                                order, key rebuilt. If
                                                you didn't go under the
                                                hood please let
                                                datatable-help know so
                                                the root cause can be
                                                fixed.</span><o:p></o:p></p>
                                            <p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;word-break:break-all"><span
                                                style="font-size:10.0pt;font-family:"Lucida
Console";color:black;background:#E1E2E5"> </span><o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This
                                              is my first attempt at
                                              using datatable so I
                                              probably did something
                                              dumb, but maybe that‘s
                                              useful for someone.  The
                                              first case is the one that
                                              gives the warnings.<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I’m
                                              also surprised at the
                                              timings.  I wrote the
                                              original algorithm using
                                              dataframe & ddply and
                                              I expected datatable to be
                                              substantially faster; the
                                              opposite is true.<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The
                                              algorithm does the
                                              following:  Certain
                                              columns in the table are
                                              keys and others are values
                                              in the sense that each row
                                              with the same set of keys
                                              should have the same set
                                              of values.  Find all the
                                              key sets for which this is
                                              not true and return the
                                              keys sets + conflicting
                                              value sets.<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Insight
                                              into the performance would
                                              be appreciated.<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Regards,<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Ron<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">library(data.table)<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">library(plyr)<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">conflictsTable1
                                              <- function(f) {<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> 
                                              u <- unique(setkey(f))<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> 
                                              if (nrow(u) == 1)
                                              return(NULL)<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> 
                                              u<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">}<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">conflictsTable2
                                              <- function(f) {<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> 
                                              u <- unique(f)<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> 
                                              if (nrow(u) == 1)
                                              return(NULL)<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> 
                                              u<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">}<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">conflictsFrame
                                              <- function(f) {<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> 
                                              u <- unique(f)<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> 
                                              if (nrow(u) == 1)
                                              return(NULL)<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> 
                                              u<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">}<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">N
                                              <- 10000<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">test
                                              <-
                                              data.table(id=as.character(10000*sample(1:N,N,replace=TRUE)),
                                              x1=rnorm(N), x2=rnorm(N),
                                              x3=rnorm(N))<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">setkey(test,id)<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">print(system.time(ut1
                                              <- test[,
                                              conflictsTable1(.SD),
                                              by=id]))<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">print(system.time(ut2
                                              <- test[,
                                              conflictsTable2(.SD),
                                              by=id]))<o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
                                            <p class="MsoNormal"
                                              style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">print(system.time(uf
                                              <- ddply(test, .(id),
                                              conflictsFrame)))<o:p></o:p></p>
                                          </div>
                                          <p class="MsoNormal"
                                            style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">_______________________________________________<br>
                                            datatable-help mailing list<br>
                                            <a moz-do-not-send="true"
                                              href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
                                            <a moz-do-not-send="true"
href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><o:p></o:p></p>
                                        </div>
                                      </div>
                                    </blockquote>
                                  </div>
                                  <p class="MsoNormal"
                                    style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">_______________________________________________<br>
                                      datatable-help mailing list<br>
                                      <a moz-do-not-send="true"
                                        href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
                                      <a moz-do-not-send="true"
href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a></span><o:p></o:p></p>
                                </div>
                              </div>
                            </blockquote>
                          </div>
                          <p class="MsoNormal">_______________________________________________<br>
                            datatable-help mailing list<br>
                            <a moz-do-not-send="true"
                              href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
                            <a moz-do-not-send="true"
href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><o:p></o:p></p>
                        </div>
                      </div>
                    </blockquote>
                  </div>
                </div>
              </blockquote>
            </div>
          </div>
        </blockquote>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
datatable-help mailing list
<a class="moz-txt-link-abbreviated" href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a>
<a class="moz-txt-link-freetext" href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a></pre>
    </blockquote>
    <br>
  </body>
</html>