<div dir="ltr">Btw, since we're on the topic of join/not-join syntax does this break others' expectations or is it just me?<div><br></div><div><div>> dt = data.table(x = c(1,2,3))</div></div><div><div>> setkey(dt,x)<br>
</div><div>> dt[J(1)]</div><div> x</div><div>1: 1</div><div>> dt[!J(1)]</div><div> x</div><div>1: 2</div><div>2: 3</div><div><b>> dt[(!J(1))]</b></div><div><b>Error in eval(expr, envir, enclos) : could not find function "J"</b></div>
<div><b>> dt[(J(1))]<br></b></div><div><b>Error in eval(expr, envir, enclos) : could not find function "J"</b></div></div><div><br></div><div style>I understand why this happens internally, because the function "()" is read as the head of the expression tree, but it's still pretty weird.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 10, 2013 at 9:55 AM, Frank Erickson <span dir="ltr"><<a href="mailto:FErickson@psu.edu" target="_blank">FErickson@psu.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I prefer ~ and/or NJ() over -. The not-join operation is different from the subsetting operation usually associated with -.<div>
<br></div><div>I don't know what characters are available for this sort of thing, but @x, @(x,y) seems natural enough as syntax for a getter.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote"><div class="im">On Mon, Jun 10, 2013 at 9:35 AM, Matthew Dowle <span dir="ltr"><<a href="mailto:mdowle@mdowle.plus.com" target="_blank">mdowle@mdowle.plus.com</a>></span> wrote:<br>
</div><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hm, another good point. We need ~ for formulae, although I can't imagine a formula in i (only in j). But in both i and j we might want to get(x).<br>
<br>
I thought about ^ i.e. X[^Y] in the spirit of regular expression syntax, but ^ doesn't parse with a RHS only. Needs to be parsable as a prefix.<br>
<br>
- maybe then? Consistent with - meaning in R. I don't think I actually had a specific use in mind for - and +, to reserve them for, but at the time it just seemed a shame to use up one of -/+ without defining the other. If - does a not join, then, might + be more like merge() (i.e. returning the union of the rows in x and i by join). I think I had something like that in mind, but hadn't thought it through.<br>
<br>
Some might say it should be a new argument e.g. notjoin=TRUE, but my thinking there is readability, since we often have many lines in i, j and by in that order, and if the "notjoin=TRUE" followed afterwards it would be far away from the i argument to which it applies. If we incorporate merge() into X[Y] using X[+Y] then it might avoid adding yet more parameters, too.<div>
<div><br>
<br>
<br>
On 10.06.2013 15:02, Gabor Grothendieck wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The problem with ~ is that it is using up a special character (of<br>
which there are only a few) for a case that does not occur much.<br>
<br>
I can think of other things that ~ might be better used for. For<br>
example, perhaps ~ x could mean get(x). One aspect of data.table that<br>
tends to be difficult is when you don't know the variable name ahead<br>
of time and this woiuld give a way to specify it concisely.<br>
<br>
On Mon, Jun 10, 2013 at 5:21 AM, Arunkumar Srinivasan<br>
<<a href="mailto:aragorn168b@gmail.com" target="_blank">aragorn168b@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Matthew,<br>
<br>
How about ~ instead of ! ? I ruled out - previously to leave + and -<br>
available for future use. NJ() may be possible too.<br>
<br>
Both "NJ()" and "~" are okay for me.<br>
<br>
That result makes perfect sense to me. I don't think of !(x==.) being the<br>
same as x!=. ! is simply a prefix. It's all the rows that aren't<br>
returned if the ! prefix wasn't there.<br>
<br>
I understand that `DT[!(x)]` does what `data.table` is designed to do<br>
currently. What I failed to mention was that if one were to consider<br>
implementing `!(x==.)` as the same as `x != .` then this behaviour has to be<br>
changed. Let's forget this point for a moment.<br>
<br>
That needs to be fixed. But we're getting quite theoretical here and far<br>
away from common use cases. Why would we ever have row numbers of the<br>
table, as a column of the table itself and want to select the rows by number<br>
not mentioned in that column?<br>
<br>
Probably I did not choose a good example. Suppose that I've a data.table and<br>
I want to get all rows where "x == 0". Let's say:<br>
<br>
set.seed(45)<br>
DT <- data.table( x = sample(c(0,5,10,15), 10, replace=TRUE), y =<br>
sample(15))<br>
<br>
DF <- as.data.frame(DT)<br>
<br>
To get all rows where x == 0, it could be done with DT[x == 0]. But it makes<br>
sense, at least in the context of data.frames, to do equivalently,<br>
<br>
DF[!(DF$x), ] (or) DF[DF$x == 0, ]<br>
<br>
All I want to say is, I expect `DT[!(x)]` should give the same result as<br>
`DT[x == 0]` (even though I fully understand it's not the intended behaviour<br>
of data.table), as it's more intuitive and less confusing.<br>
<br>
So, changing `!` to `~` or `NJ` is one half of the issue for me. The other<br>
is to replace the actual function of `!` in all contexts. I hope I came<br>
across with what I wanted to say, better this time.<br>
<br>
Best,<br>
<br>
Arun<br>
<br>
<br>
On Monday, June 10, 2013 at 10:52 AM, Matthew Dowle wrote:<br>
<br>
<br>
<br>
Hi,<br>
<br>
How about ~ instead of ! ? I ruled out - previously to leave + and -<br>
available for future use. NJ() may be possible too.<br>
<br>
Matthew<br>
<br>
<br>
<br>
On 10.06.2013 09:35, Arunkumar Srinivasan wrote:<br>
<br>
Hi Matthew,<br>
My view (from the last reply) more or less reflects mnel's comments here:<br>
<br>
<a href="http://stackoverflow.com/questions/16239153/dtx-and-dtx-treat-na-in-x-inconsistently#comment23317096_16240143" target="_blank">http://stackoverflow.com/<u></u>questions/16239153/dtx-and-<u></u>dtx-treat-na-in-x-<u></u>inconsistently#<u></u>comment23317096_16240143</a><br>
Pasted here for convenience:<br>
data.table is mimicing subset in its handling of NA values in logical i<br>
arguments. -- the only issue is the ! prefix signifying a not-join, not the<br>
way one might expect. Perhaps the not join prefix could have been NJ not !<br>
to avoid this confusion -- this might be another discussion to have on the<br>
mailing list -- (I think it is a discussion worth having)<br>
<br>
Arun<br>
<br>
On Monday, June 10, 2013 at 10:28 AM, Arunkumar Srinivasan wrote:<br>
<br>
Hm, good point. Is data.table consistent with SQL already, for both == and<br>
!=, and so no change needed?<br>
<br>
Yes, I believe it's already consistent with SQL. However, the current<br>
interpretation of NA (documentation) being treated as FALSE is not needed /<br>
untrue, imho (Please see below).<br>
<br>
<br>
And it was correct for Frank to be mistaken.<br>
<br>
Yes, it seems like he was mistaken.<br>
<br>
Maybe just some more documentation and examples needed then.<br>
<br>
It'd be much more appropriate if the documentation reflects the role of<br>
subsetting in data.table mimicking "subset" function (in order to be in line<br>
with SQL) by dropping NA evaluated logicals. From a couple of posts before,<br>
where I pasted the code where NAs are replaced to FALSE were not necessary<br>
as `irows <- which(i)` makes clear that `which` is being used to get indices<br>
and then subset, this fits perfectly well with the interpretation of NA in<br>
data.table.<br>
<br>
Are you happy that DT[!(x==.)] and DT[x!=.] do treat NA inconsistently? :<br>
<br>
<br>
<a href="http://stackoverflow.com/questions/16239153/dtx-and-dtx-treat-na-in-x-inconsistently" target="_blank">http://stackoverflow.com/<u></u>questions/16239153/dtx-and-<u></u>dtx-treat-na-in-x-<u></u>inconsistently</a><br>
<br>
Ha, I like the idea behind the use of () in evaluating expressions. It's<br>
another nice layer towards simplicity in data.table. But I still think there<br>
should not be an inconsistency in equivalent logical operations to provide<br>
different results. If !(x== .) and x != . are indeed different, then I'd<br>
suppose replacing `!` with a more appropriate name as it's much easier to<br>
get confused otherwise.<br>
In essence, either !(x == .) must evaluate to (x != .) if the underlying<br>
meaning of these are the same, or the `!` in `!(x==.)` must be replaced to<br>
something that's more appropriate for what it's supposed to be. Personally,<br>
I prefer the former. It would greatly tighten the structure and consistency.<br>
<br>
"na.rm = TRUE/FALSE" sounds good to me. I'd only considered nomatch before<br>
in the context of joins, not logical subsets.<br>
<br>
Yes, I find this option would give more control in evaluating expressions<br>
with ease in `i`, by providing both "subset" (default) and the typical<br>
data.frame subsetting (na.rm = FALSE).<br>
Best regards,<br>
<br>
Arun<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
datatable-help mailing list<br>
<a href="mailto:datatable-help@lists.r-forge.r-project.org" target="_blank">datatable-help@lists.r-forge.<u></u>r-project.org</a><br>
<br>
<a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-<u></u>project.org/cgi-bin/mailman/<u></u>listinfo/datatable-help</a><br>
</blockquote></blockquote>
<br>
______________________________<u></u>_________________<br>
datatable-help mailing list<br>
<a href="mailto:datatable-help@lists.r-forge.r-project.org" target="_blank">datatable-help@lists.r-forge.<u></u>r-project.org</a><br>
<a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-<u></u>project.org/cgi-bin/mailman/<u></u>listinfo/datatable-help</a><br>
</div></div></blockquote></div></div></div><br></div>
<br>_______________________________________________<br>
datatable-help mailing list<br>
<a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
<a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br></blockquote></div><br></div>