[datatable-help] Can crash R with a data.table query
Harish
harishv_99 at yahoo.com
Sun Jul 11 11:04:06 CEST 2010
I was thinking more about this and I am unsure how the use-case you mentioned will break if NAs are returned when length(j)==0.
DT[, date[date-min(date)<7], by=var1]
In the above code, you will at least have one date selected for each var1 since min(date) - min(date) is always < 7. So when does length(j) equal 0 for the proposed code returning NA to even get triggered?
My point was that we should have all the "by" variable values represented in the output. So in your example above, if var1 was c("A","B","C"), then the result must have at least 3 rows with at least 1 row for each var1. If no values are selected for whatever reason for var1=="B", then NA is returned.
To take this a little further:
DT[ blah1, blah2, by=list(var1,var2)]
Now suppose var1 and var2 were:
var1 var2
A x
A x
A y
B x
In the above case, the output will have three rows at least:
var1 var2
A x
A y
B x
and it need not have (B,y) since that does not even exist in the data.
If I did not select any values for (B,x) because of a row filter, I am proposing that I get an NA for all values that cannot be computed in that row.
I suppose I am not understanding how this is implemented because I see the example that you mentioned to be very different from what I am talking about.
Thanks for being so patient.
Regards,
Harish
--- On Fri, 7/9/10, Harish <harishv_99 at yahoo.com> wrote:
> From: Harish <harishv_99 at yahoo.com>
> Subject: Re: [datatable-help] Can crash R with a data.table query
> To: mdowle at mdowle.plus.com
> Cc: datatable-help at lists.r-forge.r-project.org
> Date: Friday, July 9, 2010, 10:01 PM
> Thanks for the fix. I did use a
> workaround to perform the same computation; thanks.
>
> I think that if data.tables returned NA's for all cases --
> even when length(j)==0, we will easily be able to accomplish
> all our goals:
> 1) Conveniently remove rows with NA's in
> some cases -- Use complete.cases(DT)
> 2) Be informed about missing data -- NAs
> are propagated during computations and are easy to detect.
>
> Also, a parameter can be used in case Goal #1 (above) is
> not met with complete.cases() efficiently.
>
> I think the behavior of not returning NA's when length(j)
> == 0 might cause missing data to be overlooked.
>
> In my opinion, the default behavior -- in case a parameter
> is used -- should be the "safe" scenario where the fact that
> data are missing is mentioned (just like na.rm=FALSE by
> default for a lot of the functions). This prevents the
> analyst from unknowingly proceeding with subsets of data or
> inaccurate data. Such errors will be hard to find with
> large and complex data sets. Return NA will always
> ensure that the NA is propagated -- therefore making it
> easier to catch the issue after a lot of computation.
>
> Would love to hear other perspectives.
>
>
> Regards,
> Harish
>
>
> --- On Thu, 7/8/10, Matthew Dowle <mdowle at mdowle.plus.com>
> wrote:
>
> > From: Matthew Dowle <mdowle at mdowle.plus.com>
> > Subject: RE: [datatable-help] Can crash R with a
> data.table query
> > To: "Harish" <harishv_99 at yahoo.com>
> > Cc: datatable-help at lists.r-forge.r-project.org
> > Date: Thursday, July 8, 2010, 8:35 PM
> >
> > Crash bug fixed (#983 reported by Harish, thanks).
> Tests
> > 171 and 172
> > added.
> >
> > If one or more columns of the j evaluate to
> length>0,
> > then any zero
> > length columns are replaced with an NA vector with
> length
> > the longest
> > column of the j. Thats pretty clear.
> >
> > If all columns in the j have zero length however, then
> it
> > is not
> > replaced with a single NA row, at the moment at least
> > unfortunately. I
> > couldn't get that to work because putting NAs there
> stop
> > other nice
> > features working, which I know several users depend on
> for
> > example :
> >
> > DT[, date[date-min(date)<7],
> > by=var1]
> >
> > Happy to discuss further and come up with some
> solution.
> > Maybe we need a
> > new parameter. How did you get on Harish with the
> > alternatives using a
> > join rather than by?
> >
> > Here are the current results :
> >
> > > DT
> > A B C
> > [1,] 25 a 2
> > [2,] 85 a 65
> > [3,] 25 b 9
> > [4,] 25 c 82
> > [5,] 85 c 823
> >
> > > DT[ , data.table( A, C )[ A==25, C ] +
> data.table( A,
> > C )[ A==85, C ],
> > by=B ]
> > B V1
> > [1,] a 67
> > [2,] c 905
> >
> > > DT[ , list(3,data.table( A, C )[ A==25, C ] +
> > data.table( A,
> > C )[ A==85, C ]), by=B ]
> > B V1 V2
> > [1,] a 3 67
> > [2,] b 3 NA
> > [3,] c 3 905
> >
> > Matthew
> >
> >
> > On Thu, 2010-07-01 at 09:21 -0700, Harish wrote:
> > > Tom and Matthew -- Thanks for confirming the
> issue.
> > >
> > > I had to pull out each number (i.e. A==85 and
> A==25)
> > separately because the real computation I had to do is
> not
> > associative -- involves division, etc. So the other
> > approaches you suggested won't quite work.
> > >
> > > I think that returning NA is quite acceptable
> and
> > preferred; it is better than having the row
> missing.
> > It provides an opportunity for the person analyzing
> the data
> > to realize that something was amiss (i.e. A==85 was
> missing
> > for B=="b" in example). It is also consistent with
> > reshaping the data table by having the A's as columns
> where
> > we would get NAs for missing data. Then performing
> the
> > same computation will give an NA.
> > >
> > > Regards,
> > > Harish
> > >
> > >
> > > --- On Thu, 7/1/10, mdowle at mdowle.plus.com
> > <mdowle at mdowle.plus.com>
> > wrote:
> > >
> > > > From: mdowle at mdowle.plus.com
> > <mdowle at mdowle.plus.com>
> > > > Subject: RE: [datatable-help] Can crash R
> with a
> > data.table query
> > > > To: "Short, Tom" <TShort at epri.com>
> > > > Cc: mdowle at mdowle.plus.com,
> > "Harish" <harishv_99 at yahoo.com>,
> > datatable-help at lists.r-forge.r-project.org
> > > > Date: Thursday, July 1, 2010, 5:43 AM
> > > >
> > > > I see that too now. It'll be inside
> dogroups.c.
> > Harish -
> > > > can you add as
> > > > bug please to tracker, good spot. What
> > should the
> > > > result be though? No
> > > > rows, for group "b", or NA? The way the j
> > is
> > > > constructed it can't be 9.
> > > >
> > > > Other ways to do that :
> > > >
> > > > DT[A%in%c(25,85),sum(C),by=B] # ok
> > > > B V1
> > > > [1,] a 67
> > > > [2,] b 9
> > > > [3,] c 905
> > > >
> > > > DT[,.SD[A%in%c(85,25),sum(C)],by=B] # ok
> > > > B V1
> > > > [1,] a 67
> > > > [2,] b 9
> > > > [3,] c 905
> > > >
> > > > DT[,.SD[A==25,C]+.SD[A==85,C],by=B] # crash
> too
> > > >
> > > > > setkey(DT,A)
> > > > >
> DT[J(c(25,85)),sum(C),by=B,mult="all"]
> > # ok,
> > > > likely fastest
> > > > B V1
> > > > [1,] a 67
> > > > [2,] b 9
> > > > [3,] c 905
> > > >
> > > >
> > > >
> > > > > That crashes R for me, too, somewhere
> in
> > > > data.table.dll.
> > > > >
> > > > > - Tom
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >> -----Original Message-----
> > > > >> From: datatable-help-bounces at lists.r-forge.r-project.org
> > > > >> [mailto:datatable-help-bounces at lists.r-forge.r-project.org]
> > > > >> On Behalf Of mdowle at mdowle.plus.com
> > > > >> Sent: Thursday, July 01, 2010
> 05:33
> > > > >> To: Harish
> > > > >> Cc: datatable-help at lists.r-forge.r-project.org
> > > > >> Subject: Re: [datatable-help] Can
> crash
> > R with a
> > > > data.table query
> > > > >>
> > > > >> What you mean by 'crash'? R simply
> stops
> > or theres
> > > > a message?
> > > > >> Try the clean install of latest
> 1.5, as
> > per recent
> > > > reply on
> > > > >> other thread, and can go from
> there...
> > > > >>
> > > > >> > Hi,
> > > > >> >
> > > > >> > I am crashing R with the
> following
> > code (and
> > > > it might have
> > > > >> something
> > > > >> > to do with data tables as
> well):
> > > > >> >
> > > > >> > =========
> > > > >> >
> > > > >> >
> > > > >> > DT <- structure(list(A =
> c(25L,
> > 85L, 25L,
> > > > 25L, 85L), B =
> > > > >> > structure(c(1L, 1L, 2L, 3L,
> 3L),
> > .Label =
> > > > c("a", "b", "c"),
> > > > >> class = "factor"),
> > > > >> > C = c(2L,
> > 65L, 9L,
> > > > 82L, 823L)), .Names = c("A", "B",
> > > > >> "C"), class =
> > > > >> > c("data.table",
> "data.frame"),
> > row.names =
> > > > c(NA, -5L))
> > > > >> >
> > > > >> > DT[ , data.table( A, C )[
> A==25, C
> > ] +
> > > > data.table( A, C )[
> > > > >> A==85, C ],
> > > > >> > by=B ]
> > > > >> >
> > > > >> > =========
> > > > >> >
> > > > >> > For every B, I am trying to
> sum the
> > C's where
> > > > A is 25 and 85.
> > > > >> >
> > > > >> > The crash has something to do
> with
> > my row
> > > > selection
> > > > >> criteria. First,
> > > > >> > note that for B=="b", I don't
> have
> > > > A==85. It looks like a
> > > > >> numeric(0)
> > > > >> > is being returned in this
> case.
> > > > >> >
> > > > >> > In order to avoid the crash, I
> had
> > to do
> > > > something like:
> > > > >> > if ( ! identical( DT[
> > blah ],
> > > > numeric( 0 ) )
> > > > >> >
> > > > >> > It isn't just that R is unable
> to
> > handle
> > > > operations on numeric(0)
> > > > >> > because I don't get a crash
> when I
> > just type
> > > > in "numeric(0)
> > > > >> + 2". So,
> > > > >> > my guess is that it has
> something
> > to do with
> > > > data.table as well.
> > > > >> >
> > > > >> >
> > > > >> > Harish
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > >
> _______________________________________________
> > > > >> > datatable-help mailing list
> > > > >> > datatable-help at lists.r-forge.r-project.org
> > > > >> >
> > > > >> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable
> > > > >> > -help
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > _______________________________________________
> > > > >> datatable-help mailing list
> > > > >> datatable-help at lists.r-forge.r-project.org
> > > > >> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/d
> > > > > atatable-help
> > > > >>
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
> _______________________________________________
> datatable-help mailing list
> datatable-help at lists.r-forge.r-project.org
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
>
More information about the datatable-help
mailing list