<div dir="ltr">You may want to look into Spark SQL. There is currently <a href="https://github.com/apache/spark/pull/2939">discussion on adding support for range joins</a>, which I think are similar to rolling joins in data.table.<div><br></div><div>I started looking into rmr2, but Hive and Spark SQL look like better options for my use cases.<br><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 3, 2014 at 6:00 AM, <span dir="ltr"><<a href="mailto:datatable-help-request@lists.r-forge.r-project.org" target="_blank" onclick="window.open('https://mail.google.com/mail/?view=cm&tf=1&to=datatable-help-request@lists.r-forge.r-project.org&cc=&bcc=&su=&body=','_blank');return false;">datatable-help-request@lists.r-forge.r-project.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Send datatable-help mailing list submissions to<br>
<a href="mailto:datatable-help@lists.r-forge.r-project.org" onclick="window.open('https://mail.google.com/mail/?view=cm&tf=1&to=datatable-help@lists.r-forge.r-project.org&cc=&bcc=&su=&body=','_blank');return false;">datatable-help@lists.r-forge.r-project.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br>
<br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:datatable-help-request@lists.r-forge.r-project.org" onclick="window.open('https://mail.google.com/mail/?view=cm&tf=1&to=datatable-help-request@lists.r-forge.r-project.org&cc=&bcc=&su=&body=','_blank');return false;">datatable-help-request@lists.r-forge.r-project.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:datatable-help-owner@lists.r-forge.r-project.org" onclick="window.open('https://mail.google.com/mail/?view=cm&tf=1&to=datatable-help-owner@lists.r-forge.r-project.org&cc=&bcc=&su=&body=','_blank');return false;">datatable-help-owner@lists.r-forge.r-project.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of datatable-help digest..."<br>
<br>Today's Topics:<br>
<br>
1. Rolling Joins Replicated in Java MapReduce (Mike.Gahan)<br>
2. Re: Rolling Joins Replicated in Java MapReduce (Michael Smith)<br>
<br><br>---------- Forwarded message ----------<br>From: "Mike.Gahan" <<a href="mailto:michael.gahan@gmail.com">michael.gahan@gmail.com</a>><br>To: <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>Cc: <br>Date: Tue, 2 Dec 2014 19:47:38 -0800 (PST)<br>Subject: [datatable-help] Rolling Joins Replicated in Java MapReduce<br>Hello all,<br>
<br>
I absolutely love the rolling join capabilities of data.table. It is<br>
extremely useful for the work I do. However, sometimes I work with data that<br>
is too large to fit into RAM (even when using a large server). I want to<br>
implement this rolling join code in a Java Map Reduce setting to be able to<br>
leverage some of the other resources available at the company I work for.<br>
Unfortunately I am not an experienced Java programmer. I figured that a<br>
project like this would provide an excellent incentive to learn this skill.<br>
<br>
My question is this: what data.table current code for rolling joins would be<br>
most useful to reference in starting this project? I am guessing the<br>
bmerge.c code<br>
<<a href="https://github.com/Rdatatable/data.table/blob/master/src/bmerge.c" target="_blank">https://github.com/Rdatatable/data.table/blob/master/src/bmerge.c</a>> has<br>
much of what I want. Any other code in the data.table package I should be<br>
aware of? Any other advice that might make this process go more smoothly? I<br>
know the function is based on a Modified Binary Search algorithm. Are there<br>
any libraries anyone is aware of that might help this along?<br>
<br>
I really appreciate all help.<br>
Mike<br>
<br>
<br>
<br>
--<br>
View this message in context: <a href="http://r.789695.n4.nabble.com/Rolling-Joins-Replicated-in-Java-MapReduce-tp4700329.html" target="_blank">http://r.789695.n4.nabble.com/Rolling-Joins-Replicated-in-Java-MapReduce-tp4700329.html</a><br>
Sent from the datatable-help mailing list archive at Nabble.com.<br>
<br>
<br><br>---------- Forwarded message ----------<br>From: Michael Smith <<a href="mailto:my.r.help@gmail.com">my.r.help@gmail.com</a>><br>To: "Mike.Gahan" <<a href="mailto:michael.gahan@gmail.com">michael.gahan@gmail.com</a>>, <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>Cc: <br>Date: Wed, 03 Dec 2014 14:44:11 +0800<br>Subject: Re: [datatable-help] Rolling Joins Replicated in Java MapReduce<br>Maybe it is easier to build what you're looking for by contributing to plyrmr:<br>
<br>
<a href="https://github.com/RevolutionAnalytics/plyrmr" target="_blank">https://github.com/<u></u>RevolutionAnalytics/plyrmr</a><br>
<br>
It already implements "plyr for Hadoop" on top or the rmr2 package. Not sure whether merging is already implemented, but using rmr2 it should not be prohibitively difficult (hopefully).<br>
<br>
Best,<br>
M<br>
<br>
<br>
On 12/03/2014 11:47 AM, Mike.Gahan wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Hello all,<br>
<br>
I absolutely love the rolling join capabilities of data.table. It is<br>
extremely useful for the work I do. However, sometimes I work with data that<br>
is too large to fit into RAM (even when using a large server). I want to<br>
implement this rolling join code in a Java Map Reduce setting to be able to<br>
leverage some of the other resources available at the company I work for.<br>
Unfortunately I am not an experienced Java programmer. I figured that a<br>
project like this would provide an excellent incentive to learn this skill.<br>
<br>
My question is this: what data.table current code for rolling joins would be<br>
most useful to reference in starting this project? I am guessing the<br>
bmerge.c code<br>
<<a href="https://github.com/Rdatatable/data.table/blob/master/src/bmerge.c" target="_blank">https://github.com/<u></u>Rdatatable/data.table/blob/<u></u>master/src/bmerge.c</a>> has<br>
much of what I want. Any other code in the data.table package I should be<br>
aware of? Any other advice that might make this process go more smoothly? I<br>
know the function is based on a Modified Binary Search algorithm. Are there<br>
any libraries anyone is aware of that might help this along?<br>
<br>
I really appreciate all help.<br>
Mike<br>
<br>
<br>
<br>
--<br>
View this message in context: <a href="http://r.789695.n4.nabble.com/Rolling-Joins-Replicated-in-Java-MapReduce-tp4700329.html" target="_blank">http://r.789695.n4.nabble.com/<u></u>Rolling-Joins-Replicated-in-<u></u>Java-MapReduce-tp4700329.html</a><br>
Sent from the datatable-help mailing list archive at Nabble.com.<br>
______________________________<u></u>_________________<br>
datatable-help mailing list<br>
<a href="mailto:datatable-help@lists.r-forge.r-project.org" target="_blank" onclick="window.open('https://mail.google.com/mail/?view=cm&tf=1&to=datatable-help@lists.r-forge.r-project.org&cc=&bcc=&su=&body=','_blank');return false;">datatable-help@lists.r-forge.<u></u>r-project.org</a><br>
<a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-<u></u>project.org/cgi-bin/mailman/<u></u>listinfo/datatable-help</a><br>
<br>
</blockquote>
<br>
<br>_______________________________________________<br>
datatable-help mailing list<br>
<a href="mailto:datatable-help@lists.r-forge.r-project.org" onclick="window.open('https://mail.google.com/mail/?view=cm&tf=1&to=datatable-help@lists.r-forge.r-project.org&cc=&bcc=&su=&body=','_blank');return false;">datatable-help@lists.r-forge.r-project.org</a><br>
<a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br></blockquote></div><br></div></div></div></div>