<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hi Arun,<div><br></div><div>The vignette is very helpful; the ?data.table help page is so rich and dense that I end up wandering quite a bit. The vignette does a nice job laying it out logically. I'm sure it has been a huge effort.</div><div><br></div><div><blockquote type="cite"><div dir="ltr">DT[, .SD[.N], by=month] # ~ since .N contains the number of observations in this group</div></blockquote></div><div><br></div><div>Doh! Now I see it is stated clearly right under my nose: ".N is a special in-built variable that holds the number of observations in the current group." I'm not sure why I thought .N was for the original data.table instead of the grouping.</div><div><br></div><div>I have switched my script to use the above and it is lightning fast now. I'm going to start wearing a seatbelt and helmet...</div><div><br></div><div>Thank you!</div><div>Ben</div><div><br></div><div>P.S. How did you get download.file to read an https URL for the example? Here's what I get...</div><div><pre class="sourceCode r"><code class="sourceCode r">flights <-<span class="st"> </span><span class="kw">fread</span>(<span class="st">"<a href="https://raw.githubusercontent.com/wiki/arunsrinivasan/flights/NYCflights14/flights14.csv">https://raw.githubusercontent.com/wiki/arunsrinivasan/flights/NYCflights14/flights14.csv</a>"</span>)</code></pre><pre class="sourceCode r">Error in download.file(input, tt, mode = "wb") : unsupported URL scheme</pre><div>So, I downloaded it manually using a browser and used my local copy instead.</div></div><div><br></div><div><br></div><div><br></div><div><br><div><div>On Jan 22, 2015, at 2:11 PM, Arunkumar Srinivasan <<a href="mailto:aragorn168b@gmail.com">aragorn168b@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Ben,<div><br></div><div>Great to hear that you're going thro' the vignette..</div><div><br></div><div>To get the last row, you can similarly do:</div><div><br></div><div>DT[, tail(.SD, 1L), by=month] # ~ as you say</div><div>DT[, .SD[.N], by=month] # ~ since .N contains the number of observations in this group</div><div>DT[, .SD[(.N-1L):.N], by=month] # ~ last two rows per group</div><div><br></div><div>However, `.SD[...]` per group is slightly slower (especially on many groups) as it has to go through `[.data.table` (which is a S3 generic, and takes time for dispatching the right method.. which can get noticeable on large groups), and not all cases are optimised. </div><div><br></div><div>You can also use `.I` (which is deliberately not mentioned in the vignette to keep things smooth and straightforward). Using it you could do:</div><div><br></div><div>idx = DT[, .I[1L], by=month][, V1]</div><div>DT[idx]</div><div><br></div><div>`.I` contains the row number in `x` (it doesn't reset per group..). So we can get the row indices for each group for the first element, and then simply subset. We hope to improve this subset in the future (to take care of this optimisation internally).</div><div><br></div><div>Similarly:</div><div><br></div><div>idx = DT[, .I[.N], by=month][, V1]</div><div>DT[idx]</div><div><br></div><div>will get the last element for each group.</div><div><br></div><div>Otherwise, how do you find the vignette so far?</div><div><br></div><div>HTH,</div><div>Arun</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 22, 2015 at 6:11 PM, Ben Tupper <span dir="ltr"><<a href="mailto:btupper@bigelow.org" target="_blank">btupper@bigelow.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
I have been learning to use data.table and studying the vignette located here...<br>
<br>
<a href="https://rawgit.com/wiki/Rdatatable/data.table/vignettes/datatable-intro-vignette.html" target="_blank">https://rawgit.com/wiki/Rdatatable/data.table/vignettes/datatable-intro-vignette.html</a><br>
<br>
Section 2f. shows how to subset a data.table to select an arbitrary number of rows in each .SD. That's really handy.<br>
<br>
2. Aggregations<br>
f. Subset .SD for each group: ans <- flights[, head(.SD, 2), by=month]<br>
<br>
In a similar way, I can get the last row of the .SD using either tail, nrow or dim (I don't think it matters much, but dim seems to be a faster*).<br>
<br>
ans <- flights[,.SD[dim(.SD)[1]], by=month]<br>
<br>
I got to wondering if the number of rows in .SD might be exposed in each grouping iteration. Is there an equivalent to .N for the subset data.table, .SD? Something like .SDN or the like?<br>
<br>
Thanks for data.table!<br>
<br>
Ben<br>
<br>
* After reading this discussion <a href="http://r.789695.n4.nabble.com/What-is-the-fastest-way-to-determine-that-data-table-is-empty-td4638348.html#a4638451" target="_blank">http://r.789695.n4.nabble.com/What-is-the-fastest-way-to-determine-that-data-table-is-empty-td4638348.html#a4638451</a> I tried out a couple of methods for getting the last element of a grouping using nrow(), tail() and dim().<br>
<br>
# using tail<br>
> microbenchmark( last1 <- flights[, tail(.SD, 1), by=month] )<br>
Unit: milliseconds<br>
expr min lq mean median uq max neval<br>
last1 <- flights[, tail(.SD, 1), by = month] 16.65898 16.89704 18.26415 17.37007 19.20147 40.12966 100<br>
<br>
# using dim<br>
> microbenchmark( last2 <- flights[,.SD[dim(.SD)[1]], by=month] )<br>
Unit: milliseconds<br>
expr min lq mean median uq max neval<br>
last2 <- flights[, .SD[dim(.SD)[1]], by = month] 15.51243 15.87788 17.40978 16.19426 17.83308 59.22429 100<br>
<br>
# using nrow<br>
> microbenchmark( last3 <- flights[,.SD[nrow(.SD)], by=month] )<br>
Unit: milliseconds<br>
expr min lq mean median uq max neval<br>
last3 <- flights[, .SD[nrow(.SD)], by = month] 15.63919 15.92073 17.28836 16.52588 18.33867 24.92624 100<br>
<br>
> identical(last1, last2)<br>
[1] TRUE<br>
> identical(last1, last3)<br>
[1] TRUE<br>
<br>
Ben Tupper<br>
Bigelow Laboratory for Ocean Sciences<br>
60 Bigelow Drive, P.O. Box 380<br>
East Boothbay, Maine 04544<br>
<a href="http://www.bigelow.org/" target="_blank">http://www.bigelow.org</a><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
datatable-help mailing list<br>
<a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
<a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br>
</blockquote></div><br></div>
</blockquote></div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px;"><span class="Apple-style-span" style="orphans: 2; text-align: -webkit-auto; text-indent: 0px; widows: 2; -webkit-text-decorations-in-effect: none; "><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="orphans: 2; text-indent: 0px; widows: 2; -webkit-text-decorations-in-effect: none; "><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="orphans: 2; text-indent: 0px; widows: 2; -webkit-text-decorations-in-effect: none; "><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="orphans: 2; text-indent: 0px; widows: 2; -webkit-text-decorations-in-effect: none; "><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="orphans: 2; text-indent: 0px; widows: 2; -webkit-text-decorations-in-effect: none; "><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div class="" style="font-size: 13px; "><font class="Apple-style-span" face="Helvetica">Ben Tupper<br>Bigelow Laboratory for Ocean Sciences<br>60 Bigelow Drive, P.O. Box 380<br>East Boothbay, Maine 04544<br><a href="http://www.bigelow.org">http://www.bigelow.org</a></font></div></div></span></div></span></div></span></div></span></div></span><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-family: Courier; "><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Courier; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Courier; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div class=""><br class=""></div></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline">
</div>
<br></div></body></html>