<div dir="ltr">It hadn't occurred to me to use CJ(), so I'll tinker with that this evening and<div>see if there are any gains to be made there. In theory it's highly parallelizable,</div><div>and one of the posts Matthew points to in his comments (in the post you</div>
<div>reference) shows a way that it can be done (using the old multicore library,</div><div>so I'm not exactly sure how it maps to the parallel library). In my case</div><div>though, the whole process appears to be memory bound rather than CPU</div>
<div>bound. Since my machine is fairly optimal (i7-4770 with 4x8GB DDR3-1600),</div><div>I just don't think it's going to get dramatically faster. That doesn't mean</div><div>I won't try...</div></div><div class="gmail_extra">
<br clear="all"><div>-------<br>Nathaniel Graham<br><a href="mailto:npgraham1@gmail.com" target="_blank">npgraham1@gmail.com</a><br><a href="mailto:npgraham1@uky.edu" target="_blank">npgraham1@uky.edu</a></div>
<br><br><div class="gmail_quote">On Tue, Sep 17, 2013 at 5:52 PM, Frank Erickson <span dir="ltr"><<a href="mailto:FErickson@psu.edu" target="_blank">FErickson@psu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Maybe not ultrafast, but with nice syntax:<div><br></div><div><font face="courier new, monospace">CJ(i=iset,j=jset)[criterion(i,j)]</font></div><div><font face="courier new, monospace"><br></font></div><div>
I guess it should be parallelizable, but that wouldn't be with data.table, if I understand this correctly: <a href="http://stackoverflow.com/questions/14759905/data-table-and-parallel-computing" target="_blank">http://stackoverflow.com/questions/14759905/data-table-and-parallel-computing</a></div>
</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 17, 2013 at 5:42 PM, Nathaniel Graham <span dir="ltr"><<a href="mailto:npgraham1@gmail.com" target="_blank">npgraham1@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Oops; I meant to reply to all, and then forgot after I discarded and rewrote my<div>message a few times. I suspect (although I'm not absolutely certain) that if</div>
<div>NULL or similar did the same thing as returning a 0-row data.table with the</div>
<div>appropriate number of columns, some operations could be sped up a bit.</div><div>In those cases, the data.table code wouldn't need to check the number and</div><div>type of the columns returned.</div><div><br></div>
<div>I suspect that unless someone knows a secret, ultrafast way to iterate through</div><div>a list of all combinations of a set of items and return the subset of those that</div><div>match some criteria, that I'm as close to optimal as I'm likely to get right now.<div>
<br>
<div class="gmail_extra"><br clear="all"><div>-------<br>Nathaniel Graham<br><a href="mailto:npgraham1@gmail.com" target="_blank">npgraham1@gmail.com</a><br><a href="mailto:npgraham1@uky.edu" target="_blank">npgraham1@uky.edu</a></div>
<br><br><div class="gmail_quote">On Tue, Sep 17, 2013 at 5:22 PM, Frank Erickson <span dir="ltr"><<a href="mailto:FErickson@psu.edu" target="_blank">FErickson@psu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Well, rbindlist(list()) says "Null data.table" (though it doesn't pass the is.null() test). Maybe someone else has an idea how to deal with the no-results case. By the way, it's best to use "reply to all" to make sure you reply to the mailing list, too; they should be able to see your message quoted below, though.<span><font color="#888888"><div>
<br></div><div>--Frank</div></font></span><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 17, 2013 at 5:03 PM, Nathaniel Graham <span dir="ltr"><<a href="mailto:npgraham1@gmail.com" target="_blank">npgraham1@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Frank,<div><br></div><div>Thanks. This seems to have done the trick, so long as I'm careful to check for</div>
<div>zero-length lists and return data.table(i = integer(), j = integer()) in those</div><div>
cases. Essentially, I have to test every combination of i and j to see if it's</div><div>"interesting" or not, and some groups have a lot of rows. At the moment I'm</div><div>attacking some other low hanging fruit, like speeding up the comparisons</div>
<div>I have to do.</div><div><br></div><div>As a side note, it would be kind of nice if there was a simple way to clue</div><div>data.table to the fact that there are no rows to return, like returning NULL</div><div>or NA or similar.</div>
</div><div class="gmail_extra"><div><br clear="all"><div>-------<br>Nathaniel Graham<br><a href="mailto:npgraham1@gmail.com" target="_blank">npgraham1@gmail.com</a><br><a href="mailto:npgraham1@uky.edu" target="_blank">npgraham1@uky.edu</a></div>
</div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>