[datatable-help] data.table heisenbug - large DT

[Matthew Dowle]

Hi,

Thanks for all this. Yes, this is very likely the same 'Chris crash'
which I can reproduce on 64bit. It happens on 2.14.0+ too so it's not
that. The type '12' error, the line it crashes etc, can vary. As you say
it seems to be a memory corruption earlier.  gctorture() for me doesn't
seem to pinpoint the earlier point, though, so I'm struggling.

Thanks for all the detail. The one common thing might be merge()ing. The
Chris crash script also has a merge() after some :=, and before some
more :=.  merge() is needed currently because := by group isn't
implemented. I've already tried several fixes to merge() and other
places, which are in latest svn version of 1.7.8. Those certainly fix
some problems, but at least one must still remain.

The best 2 options I can think are either roll back to 1.7.1 before
over-allocation was introduced (that's what Chris has done), or try
avoiding the merge() by coding it in a different way e.g. X[Y] syntax
and for that you might need to setkey and then setkey back. Or, try a
copy() of the result from merge.  All ugly and hot potatoes, I know. 

Matthew

On Thu, 2012-01-12 at 22:33 -0600, DM wrote:
> Happy New Year to all.
> 
> I suspect that this is the same problem that others have reported,
> which I can add to: something is causing data.table to crash or
> corrupt R when using  myDT[, newCol := oldCol]  or similar syntax.
> 
> For context: I am using R 2.13.2, and have found this issue with
> data.table versions 1.7.6, 1.7.7, and 1.7.8.  This is 64-bit R, on
> 64-bit Ubuntu (Amazon EC2 m4.2xlarge and m4.4xlarge instances).
> 
> Some observations:
> 
> 1. The objects are large - it's about 10 columns and about 10M rows.
> 
> 2. It seems to corrupt things with mention of something about type
> '12'.  For instance, I tried saving the workspace frequently, and R
> crashed during a save (i.e. not even during a data.table execution),
> with the message: "WriteItem: unknown type 12".
> 
> 3. I typically have this code executing inside a function, and I
> either call the function from the R console or execute a script from
> the command line.  Since I've been debugging, I've done more in the
> former context (i.e. from the console).  What's bizarre is that if I
> do not "debug()" my function, it is likely to crash.  If, however, I
> run "debug(myFunction)", then it is much less likely to crash (almost
> never).  Just setting "debug" makes a difference, it seems, but this
> difference does not extend to usage from a script.  I'm not sure why
> this is, though it is bizarre.
> 
> 4. The objects tend to have NAs in some columns - that's okay, I put
> them there.  All of the problems arise with one data table.  It is
> constructed through merges of other data tables and operations on its
> own columns.  No problems appear with the other data tables.  However,
> that's not entirely true.  When the program crashes during a specific
> operation on this one DT, it made me believe that that one operation
> is problematic.  I created the checkpointing code mentioned in the 2nd
> note, and R crashed at an earlier stage - while saving other data
> tables.  In other words, no data table operations, before one
> particular operation, give any problems, but that may be like the game
> hot potato: the problem (e.g. some kind of memory corruption?) could
> have begun earlier and something, either "save()" or " newCol :=
> oldCol" is just the unlucky statement that was last attempted before R
> crashed.
> 
> Unfortunately, I don't have code to reproduce this in a simple manner.
> I can give lots of error reports.
> 
> For instance, here is output from "traceback" when this one "newCol :=
> oldCol"-esque operation occurred:
> 
>         tracemem[0x16f09c40 -> 0x1158fab8]: match unique intersect
> [.data.table [ myFunc
>         Error in data.table(i, j, Result) : unimplemented type (12) in
> 'duplicate'
>             6: data.table(i, j, Result)
>             5: eval(expr, envir, enclos)
>             4: eval(jsub, envir = x, enclos = parent.frame())
>             3: `[.data.table`(myDT, , list(i, j, Result))
>             2: myDT[, list(i, j, Result)]
>             1: myFunc(arguments)
> 
> I will try other workarounds, e.g. not use ":=", but it's not clear
> that that will solve the problem, since the problem potentially begins
> before my biggest snafu.   I'd prefer not to update R just yet, since
> I have no idea how that will impact other packages, but that option
> isn't entirely off the table.
> 
> Any thoughts or recommendations?
> 
> Thanks!
> 
> _______________________________________________
> datatable-help mailing list
> datatable-help at lists.r-forge.r-project.org
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help