<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
<html><body>
<p> </p>
<p>I didn't get any feedback off list on this one.</p>
<p>But I'm coming round to the idea.</p>
<p>What about by=.JOIN (is that you were thinking .J stood for?) Other possibilties: .EACHI, .IROW, .EACHIROW, .CROSSAPPLY, .EACHJOIN. Just to brainstorm it.</p>
<p>by=.JOIN could be added anyway with no backwards compatibility issues, so that those who wished to be explicit now could be.</p>
<p>To change the default for X[Y, j] I'm also coming round to. It might help in a few related areas e.g. X[Y][,j] (which isn't great right now, agreed). We have successfully made non-backwards-compatibile changes in the past by introducing a global option which we slowly migrate to. If datatable.bywithoutby was added it could take values TRUE|"warning"|FALSE from day one, with default TRUE. That allows those who wish for explicit by to migrate straight away by changing the default to FALSE. Existing users could set it to "warning" to see how many implicit bywithoutby they have. Those calls can gradually be changed to by=.JOIN and in that way both implicit and explicit work at the same time, for say a year, with full backwards compatibility by default. This approach allows a slow and flexible migration path on a per feature basis. Then the default could be chaged to "warning" before finally FALSE. Depending on how it goes, the option could be left there to allow TRUE if anyone wanted it, or removed (maybe after two years). Similar to the removal of J() outside DT[...] i.e. users can still now very easily write J=data.table in their .Rprofile if they wish, for backwards compatibility.</p>
<p>Or ... instead of :</p>
<p> X[Y, j, by=.JOIN]</p>
<p>what about :</p>
<p> X[by=Y, j]</p>
<p>Matthew</p>
<p> </p>
<p>On 25.04.2013 16:32, Matthew Dowle wrote:</p>
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<p> </p>
<p>I'd appreciate some input from others whether they agree or not. If you have a view perhaps let me know off list, or on list, whichever you prefer.</p>
<p>Thanks,</p>
<p>Matthew</p>
<p> </p>
<p>On 25.04.2013 13:45, Eduard Antonyan wrote:</p>
<blockquote style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
<div>Well, so can .I or .N or .GRP or .BY, yet those are used as special names, which is exactly why I suggested .J.</div>
<div>The problem with using 'missingness' is that it already means smth very different when i is not a join/cross, it means *don't* do a by, thus introducing the whole case thing one has to through in their head every time as in OP (which of course becomes automatic after a while, but it's a cost nonetheless, which is in particular high for new people). So I see absence of 'by' as an already taken and used signal and thus something else has to be used for the new signal of cross apply (it doesn't have to be the specific option I mentioned above). This is exactly why I find optional turning off of this behavior unsatisfactory, and I don't see that as a solution to this at all. </div>
<div>I think in the x+y context the appropriate analog is - what if that added x and y normally, but when x and y were data.frames it did element by element multiplication instead? Yes that's possible to do, and possible to document, but it's not a good idea, because it takes place of adding them element by element. The recycling behavior doesn't do that - what that does is it says it doesn't really make sense to add them as is, but we can do that after recycling, so let's recycle. It doesn't take the place of another existing way of adding vectors. <br /><br />On Apr 25, 2013, at 4:28 AM, Matthew Dowle <<a href="mailto:mdowle@mdowle.plus.com">mdowle@mdowle.plus.com</a>> wrote:<br /><br /></div>
<blockquote style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
<div>
<p> </p>
<pre>I see what you're getting at. But .J may be a column name, which is the current meaning of by = single symbol. And why .J? If not .J, or any single symbol what else instead? A character value such as by="irows" is taken to mean the "irows" column currently (for consistency with by="colA,colB,colC"). But some signal needs to be passed to by=, then (you're suggesting), to trigger the cross apply by each i row. Currently, that signal is missingness (which I like, rely on, and use with join inherited scope).<br /><br />As I wrote in the S.O. thread, I'm happy to make it optional (i.e. an option to turn off by-without-by), since there is no downside. But you've continued to argue for a change to the default, iiuc.<br /><br />Maybe it helps to consider :<br /><br /> x+y<br /><br />Fundamentally in R this depends on what x and y are. Most of us probably assume (as a first thought) that x and y are vectors and know that this will apply "+" elementwise, recycling y if necessary. In R we like and write code like this all the time. I think of X[Y, j] in the same way: j is the operation (like +) which is applied for each row of Y. If you need j for the entire set that Y joins to, then like a FAQ says, make j missing too and it's X[Y][,j]. But providing a way to make X[Y,j] do the same as X[Y][,j] would be nice and is on the list: drop=TRUE would do that (as someone mentioned on the S.O. thread). So maybe the new option would be datatable.drop (but with default FALSE not TRUE). If you wanted to turn off by-without-by you might set options(datatable.drop=TRUE). Then you can use data.table how you prefer (explicit by) and I can use it how I prefer.</pre>
<pre> </pre>
<pre>I'm happy to add the argument to [.data.table, and make its default changeable via a global option in the usual way. </pre>
<p>Matthew</p>
<p> </p>
<p>On 25.04.2013 05:16, Eduard Antonyan wrote:</p>
<blockquote style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
<div>That's really interesting, I can't currently think of another way of doing that as after X[Y] is done the necessary information is lost. </div>
<div>To retain that functionality and achieve better readability, as in OP, I think smth along the lines of X[Y, head(.SD, i.top), by=.J] would be a good replacement for current syntax.</div>
<div><br />On Apr 24, 2013, at 6:01 PM, Eduard Antonyan <<a href="mailto:eduard.antonyan@gmail.com">eduard.antonyan@gmail.com</a>> wrote:<br /><br /></div>
<blockquote style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
<div>
<div dir="ltr">that's an interesting example - I didn't realize current behavior would do that, I'm not at a PC anymore but I'll definitely think about it and report back, as it's not immediately obvious to me</div>
<div class="gmail_extra"><br /><br />
<div class="gmail_quote">On Wed, Apr 24, 2013 at 5:50 PM, Matthew Dowle <span><<a href="mailto:mdowle@mdowle.plus.com">mdowle@mdowle.plus.com</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><span style="text-decoration: underline;"></span>
<div>
<p> </p>
<p>i. prefix is just a robust way to reference join inherited columns: the 'top' column in the i table. Like table aliases in SQL.</p>
<p>What about this? :</p>
<div class="im">1> X = data.table(a=1:3,b=1:15, key="a")<br />1> X<br /> a b<br /> 1: 1 1<br /> 2: 1 4<br /> 3: 1 7<br /> 4: 1 10<br /> 5: 1 13<br /> 6: 2 2<br /> 7: 2 5<br /> 8: 2 8<br /> 9: 2 11<br />10: 2 14<br /> 11: 3 3<br />12: 3 6<br /> 13: 3 9<br />14: 3 12<br />15: 3 15</div>
<pre>1> Y = data.table(a=c(1,2,1), top=c(3,4,2))</pre>
<div class="im"><br />1> Y<br /> a top<br />1: 1 3<br />2: 2 4<br />3: 1 2<br />1> X[Y, head(.SD,i.top)]<br /> a b<br />1: 1 1<br />2: 1 4<br />3: 1 7<br />4: 2 2<br />5: 2 5<br />6: 2 8<br />7: 2 11<br />8: 1 1</div>
<pre>9: 1 4<br />1> </pre>
<div>
<div class="h5">
<p> </p>
<p>On <a>24.04.2013 23</a>:43, Eduard Antonyan wrote:</p>
<blockquote style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
<div dir="ltr">I assumed they meant create a table :)
<div>that looks cool, what's i.top ? I can get a very similar to yours result by writing:</div>
<div>X[Y][, head(.SD, top[1]), by = a]</div>
<div>and I probably would want the following to produce your result (this might depend a little on what exactly i.top is):</div>
<div>X[Y, head(.SD, i.top), by = a]</div>
</div>
<div class="gmail_extra"><br /><br />
<div class="gmail_quote">On Wed, Apr 24, 2013 at 5:28 PM, Matthew Dowle <span><<a href="mailto:mdowle@mdowle.plus.com">mdowle@mdowle.plus.com</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><span style="text-decoration: underline;"></span>
<div>
<p> </p>
<p>That sentence on that linked webpage seems incorect English, since table is a noun not a verb. Should "table" be "join" perhaps?</p>
<p>Anyway, by-without-by is often used with join inherited scope (JIS). For example, translating their example :</p>
<pre>1> X = data.table(a=1:3,b=1:15, key="a")<br />1> X<br /> a b<br /> 1: 1 1<br /> 2: 1 4<br /> 3: 1 7<br /> 4: 1 10<br /> 5: 1 13<br /> 6: 2 2<br /> 7: 2 5<br /> 8: 2 8<br /> 9: 2 11<br />10: 2 14<br />11: 3 3<br />12: 3 6<br />
13: 3 9<br />14: 3 12<br />15: 3 15<br />1> Y = data.table(a=c(1,2), top=c(3,4))<br />1> Y<br /> a top<br />1: 1 3<br />2: 2 4<br />1> X[Y, head(.SD,i.top)]<br /> a b<br />1: 1 1<br />2: 1 4<br />3: 1 7<br />4: 2 2<br />5: 2 5<br />
6: 2 8<br />7: 2 11<br />1> </pre>
<pre> </pre>
<pre>If there was no by-without-by (analogous to CROSS BY), then how would that be done?</pre>
<div>
<div>
<pre> </pre>
<p>On <a>24.04.2013 22</a>:22, Eduard Antonyan wrote:</p>
<blockquote style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
<div dir="ltr">By that you mean current behavior? You'd get current behavior by explicitly specifying the appropriate "by" (i.e. "by" equal to the key).
<div>Btw, I'm trying to understand SQL CROSS APPLY vs JOIN using <a href="http://explainextended.com/2009/07/16/inner-join-vs-cross-apply/">http://explainextended.com/2009/07/16/inner-join-vs-cross-apply/</a>, and I can't figure out how by-without-by (or with by-with-by for that matter:) ) helps with e.g. the first example there:</div>
<div>"<span style="font-size: 13px; line-height: 18px;">We table</span><span style="font-size: 13px; line-height: 18px;"> </span><code style="font-size: 13px; line-height: 18px; padding: 0px; margin: 0px; font-weight: bold; color: #4488bb;">table1</code><span style="font-size: 13px; line-height: 18px;"> </span><span style="font-size: 13px; line-height: 18px;">and</span><span style="font-size: 13px; line-height: 18px;"> </span><code style="font-size: 13px; line-height: 18px; padding: 0px; margin: 0px; font-weight: bold; color: #4488bb;">table2</code><span style="font-size: 13px; line-height: 18px;">.</span><span style="font-size: 13px; line-height: 18px;"> </span><code style="font-size: 13px; line-height: 18px; padding: 0px; margin: 0px; font-weight: bold; color: #4488bb;">table1</code><span style="font-size: 13px; line-height: 18px;"> </span><span style="font-size: 13px; line-height: 18px;">has a</span><span style="font-size: 13px; line-height: 18px;"> </span><code style="font-size: 13px; line-height: 18px; padding: 0px; margin: 0px; font-weight: bold; color: #4488bb;">column</code><span style="font-size: 13px; line-height: 18px;"> </span><span style="font-size: 13px; line-height: 18px;">called</span><span style="font-size: 13px; line-height: 18px;"> </span><code style="font-size: 13px; line-height: 18px; padding: 0px; margin: 0px; font-weight: bold; color: #4488bb;">rowcount</code><span style="font-size: 13px; line-height: 18px;">.</span></div>
<p style="padding: 0px; margin: 0px 0px 15px; line-height: 18px; font-size: 13px;">For each row from <code style="padding: 0px; margin: 0px; font-weight: bold; color: #4488bb;">table1</code> we need to select first <code style="padding: 0px; margin: 0px; font-weight: bold; color: #4488bb;">rowcount</code> rows from <code style="padding: 0px; margin: 0px; font-weight: bold; color: #4488bb;">table2</code>, ordered by <code style="padding: 0px; margin: 0px; font-weight: bold; color: #4488bb;"><a href="http://table2.id">table2.id</a>"</code></p>
<p style="padding: 0px; margin: 0px 0px 15px; line-height: 18px; font-size: 13px;"><code style="padding: 0px; margin: 0px; font-weight: bold; color: #4488bb;"><br /></code></p>
</div>
<div class="gmail_extra"><br /><br />
<div class="gmail_quote">On Wed, Apr 24, 2013 at 4:01 PM, Matthew Dowle <span><<a href="mailto:mdowle@mdowle.plus.com">mdowle@mdowle.plus.com</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;">But then what would be analogous to CROSS APPLY in SQL?<br />
<div><br /> > I'd agree with Eduard, although it's probably too late to change behavior<br /> > now. Maybe for data.table.2? Eduard's proposal seems more closely<br /> > aligned with SQL behavior as well (SELECT/JOIN, then GROUP, but only if<br /> > requested).<br /> ><br /> > S.<br /> ><br /> >> Date: Mon, 22 Apr 2013 08:17:59 -0700</div>
>> From: <a href="mailto:eduard.antonyan@gmail.com">eduard.antonyan@gmail.com</a><br /> >> To: <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br />
<div>>> Subject: Re: [datatable-help] changing data.table by-without-by<br /> >> syntax to require a "by"<br /> >><br /> >> I think you're missing the point Michael. Just because it's possible to<br /> >> do it<br /> >> the way it's done now, doesn't mean that's the best way, as I've tried<br /> >> to<br /> >> argue in the OP. I don't think you've addressed the issue of unnecessary<br /> >> complexity pointed out in OP.<br /> >><br /> >><br /> >><br /> >> --<br /> >> View this message in context:<br /> >> <a href="http://r.789695.n4.nabble.com/changing-data-table-by-without-by-syntax-to-require-a-by-tp4664770p4664990.html">http://r.789695.n4.nabble.com/changing-data-table-by-without-by-syntax-to-require-a-by-tp4664770p4664990.html</a><br /> >> Sent from the datatable-help mailing list archive at <a href="http://Nabble.com">Nabble.com</a>.<br /> >> _______________________________________________<br /> >> datatable-help mailing list</div>
>> <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br />
<div>>> <a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br /> > _______________________________________________<br /> > datatable-help mailing list</div>
> <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br /> > <a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br /><br /><br /></blockquote>
</div>
</div>
</blockquote>
<p> </p>
<div> </div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<p> </p>
<div> </div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</blockquote>
<p> </p>
<div> </div>
</div>
</blockquote>
</blockquote>
<p> </p>
<div> </div>
</blockquote>
<p> </p>
<div> </div>
</body></html>