[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption

[Douglas Bates]

On Thu, Jan 6, 2011 at 2:37 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
> On Thu, Jan 6, 2011 at 2:19 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
>> In the continuing saga ...
>>
>> On Thu, Jan 6, 2011 at 12:57 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>> On Thu, Jan 6, 2011 at 9:54 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>> On Wed, Jan 5, 2011 at 2:38 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>>> On Wed, Jan 5, 2011 at 2:15 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>>>> On looking more closely at the output, I thought that the problem may
>>>>>> arise in loading the Rcpp package because that is when the function
>>>>>> init_Rcpp_cache() is evaluated.  So I ran another test which was
>>>>>> simply
>>>>>>
>>>>>> gctorture(TRUE)
>>>>>> library(Rcpp)
>>>>>>
>>>>>> run with -d valgrind.
>>>>>>
>>>>>> Unfortunately (well, at least from my point of view) that test didn't
>>>>>> show any problems.  I'm at a bit of a loss where to go on this.
>>>>>
>>>>> Perhaps Romain or Dirk could clarify for me when init_Rcpp_cache
>>>>> should be called.  It appears to be getting called twice, once
>>>>> explicitly in the .onLoad expression in Rcpp/R/zzz.R and once
>>>>> implicitly by being part of the R_init_Rcpp function defined in
>>>>> Rcpp/src/Module.cpp.  Is this intentional?  Should one or both of
>>>>> those be a maybe_init() call?  I think as it stands the second call
>>>>> will overwrite the contents created by the first call.
>>>>>
>>>>> I'm kind of grasping at straws here but I think there might be a
>>>>> problem with the cache being created when the Rcpp namespace is
>>>>> attached and then called again when the Rcpp package is loaded in my
>>>>> example.
>>>>
>>>> I was able to remove the call to init_Rcpp_cache in the .onLoad
>>>> function in Rcpp/R/zzz.R and still run R CMD check Rcpp successfully.
>>>> Unfortunately, even with this change loading the lme4a package fails
>>>> when gctorture is turned on.  Transcript enclosed.  Looks like I get
>>>> to learn more about the Module loading process.
>>>
>>> I'm still flailing away at this.  It is one of those really messy
>>> debugging situations in that I can't make it fail except on the test
>>> of the full example and using both gctorture and valgrind.
>>>
>>> It appears that the .onLoad function in lme4a is failing when called
>>> in loadNameSpace.  The .onLoad function was
>>>
>>> .NameSpace <- environment()
>>> .onLoad <- function(libname, pkgname)
>>> {
>>>    ## For Matrix API change (Oct.2009) - silence the warning:
>>>    assign("det_CHMfactor.warn", FALSE, envir = Matrix:::.MatrixEnv)
>>>    mod <- Module("lme4")
>>>    populate(mod, .NameSpace)
>>> }
>>>
>>> So it seems that either the call to Module() or the call to populate()
>>> is failing and the tryCatch block in runHook() within loadNamespace()
>>> is creating an error message.  The call: and message: parts of the
>>> error message are generated from the error condition object using
>>> conditionCall() and conditionMessage().  To me it looks like the call
>>> stack has gotten corrupted because there are no calls like that.  The
>>> form of the call seems to be that of lazyLoadDBfetch in
>>> $R_SRC/src/library/base/R/lazyLoad.R, which is called in a most
>>> mysterious way.
>>>
>>> If anyone can shed any light on this, I would appreciate it.  I have
>>> another gctorture/valgrind run underway in which I don't use .onLoad
>>> but call Module and populate explicitly to see if I can track down
>>> where it is going south.  Suggestions welcome.
>>
>> If I disable the .onLoad function and call Module() and populate()
>> after loading the package and with gctorture(TRUE) then I get a
>> failure in the call to populate, with the same mysterious value of the
>> call and the message
>>
>>> library(lme4a)
>> Loading required package: Matrix
>> Loading required package: lattice
>>
>> Attaching package: 'Matrix'
>>
>> The following object(s) are masked from 'package:base':
>>
>>    det
>>
>> Loading required package: minqa
>> Loading required package: Rcpp
>> Loading required package: MatrixModels
>>
>> Attaching package: 'MatrixModels'
>>
>> The following object(s) are masked from 'package:Matrix':
>>
>>    model.Matrix
>>
>>> gctorture(TRUE)
>>> mod <- Module("lme4", "lme4a")
>>> populate(mod, .GlobalEnv)
>> Error in list(c(78682L, 78L),
>> "/home/bates/R/x86_64-unknown-linux-gnu-library/2.13/Rcpp/R/Rcpp.rdb",
>> TRUE, function (n)  :
>>  unused argument(s)
>> ("/home/bates/R/x86_64-unknown-linux-gnu-library/2.13/Rcpp/R/Rcpp.rdb",
>> TRUE, function (n)
>>
>> Can someone who knows more about the Module code than I (meaning
>> Romain or John) tell me if that call could originate from within
>> populate?  It seems highly unlikely to me but I can't rule it out.
>>
>> By the way, this was using Rcpp_0.9.0.2 which still has two calls to
>> init_Rcpp_cache.  I'll change that and see if it makes a difference.
>
> As they say in this season of (American) football games, "on further
> review" it looks as if that string might have originated in a PROMSXP.
>  This is definitely getting wierd.

Or "weird" if your spell checker is working.  ;-)

I now have managed to get the failure by accessing a class in the
module, rather than using populate(), which is a relief because
populate() is not an easy function to trace.

> gctorture(TRUE)
> mod <- Module("lme4", "lme4a")
> reTrms <- mod$reTrms
Error in list(c(78646L, 78L),
"/home/bates/R/x86_64-unknown-linux-gnu-library/2.13/Rcpp/R/Rcpp.rdb",
TRUE, function (n)  :
  unused argument(s)
("/home/bates/R/x86_64-unknown-linux-gnu-library/2.13/Rcpp/R/Rcpp.rdb",
TRUE, function (n)


>>>>>> On Wed, Jan 5, 2011 at 11:55 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>>>>> This time with the enclosure :-)
>>>>>>>
>>>>>>> On Wed, Jan 5, 2011 at 11:52 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>>>>>> I don't know whether this is through error on my part or because of an
>>>>>>>> "infelicity" in the Rcpp module code but the lme4a package, which now
>>>>>>>> uses Rcpp modules extensively, ends up with some difficult-to-trace
>>>>>>>> memory corruption issues.  Yesterday i finally bit the bullet and ran
>>>>>>>> a test with gctorture(TRUE) and valgrind enabled.  It takes a very
>>>>>>>> long time and results in a segfault when trying to load the package.
>>>>>>>> I enclose the transcript.  I should say that this is using Rcpp_0.9.0
>>>>>>>> from CRAN, not the SVN version of Rcpp.
>>>>>>>>
>>>>>>>> I just got these results this morning (it was running overnight) and
>>>>>>>> haven't looked at the code in Module.cpp and cache.cpp yet.  If it
>>>>>>>> seems likely that the code is beyond me I can try to work out a
>>>>>>>> simpler example that triggers the problem.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>