<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
<html><body>
<p>Hi,</p>
<p>Are the duplicates next to each other in the table? Or could duplicates be within each key, separated by other rows?</p>
<p>If duplicates are together, calling data.table:::duplist directly should do it. (see source of data.table:::unique.data.table). It loops through the rows by column and works like diff(x)==0 would i.e. looking at the previous row only, but does compare all columns. If a subset of columns are needed, then maybe a data.table:::shallow followed by column removal of the ones you don't need on that shallow copy (the shallow copy and column removal being instant). Just because duplist doesn't accept a subset of the list of columns it is passed.</p>
<p>shallow() is on the agenda to be exported for user use (so suggesting it is an excuse to get you to test it!). Hadn't thought about duplist but could do, too. They are both relied on internally, so should be reliable. But as soon as they're exported we can't make non-backwards compatible changes to them.</p>
<p>Matthew</p>
<p>On 07.03.2013 16:45, Ricardo Saporta wrote:</p>
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<div>I have a keyed data.table, DT, with 800k rows, of which about 0.5% are duplicates that need to removed. </div>
<div>Using unique(DT) of course widdles down the whole table to one row per key.</div>
<div>I would like to get results similar to unique.data.frame(DT)</div>
<div>Two problems with using unique.data.frame: (1) Speed (2) loss of key(DT)</div>
<div>So instead Im using a wrapper that </div>
<div> (1) caches key(DT) (2) removes the key (3) calls unique on DT (4) then repplies the key</div>
<div>However, this is convoluted (and also requires modifying setkey(.) and getdots(.)). </div>
<div>It occurs to me that I might be overlooking a simpler alternative. </div>
<div>anythoughts? </div>
<div>Thanks, </div>
<div>Rick </div>
<div>_Here is what I am using_: </div>
<div> uniqueRows
<div> # If already keyed (or not a DT), use regular unique(DT)</div>
<div> if (!haskey(DT) || !is.data.table(x) )</div>
<div> return(unique(DT))</div>
<div> .key
<div> setkey(DT, NULL)</div>
<div> setkeyE(unique(DT), eval(.key))</div>
<div> } </div>
<div> getdotsWithEval
<div> dots
<div> as.character(match.call(sys.function(-1), call = sys.call(-1), </div>
<div> expand.dots = FALSE)$...)</div>
<div> if (grepl("^eval\\(", dots) && grepl("\\)$", dots))</div>
<div> return(eval(parse(text=dots)))</div>
<div> return(dots)</div>
<div> }</div>
<div> setkeyE
<div> # SAME AS setkey(.) WITH ADDITION THAT </div>
<div> # IF KEY IS WRAPPED IN eval(.) IT WILL BE PARSED</div>
<div> if (is.character(x)) </div>
<div> stop("x may no longer be the character name of the data.table. The possibility was undocumented and has been removed.")</div>
<div> #** THIS IS THE MODIFIED LINE **#</div>
<div> # OLD**: cols = getdots()</div>
<div> cols
<div> if (!length(cols)) </div>
<div> cols = colnames(x)</div>
<div> else if (identical(cols, "NULL")) </div>
<div> cols = NULL</div>
<div> setkeyv(x, cols, verbose = verbose)</div>
<div> }</div>
-- <br />
<div style="color: #222222; font-family: arial,sans-serif; font-size: 13px; background-color: #ffffff;">
<div style="font-size: 13px;">Ricardo Saporta</div>
<div style="font-size: 13px;">Graduate Student, Data Analytics</div>
<div style="font-size: 13px;"><span style="font-size: 13px;">Rutgers University, New Jersey</span></div>
<div style="font-size: 13px;"><span style="font-size: 13px;">e: </span><a style="color: #1155cc; font-size: 13px;" href="mailto:saporta@rutgers.edu">saporta@rutgers.edu</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p> </p>
<div> </div>
</body></html>