[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption

[Douglas Bates]

In the continuing saga ...

On Thu, Jan 6, 2011 at 12:57 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
> On Thu, Jan 6, 2011 at 9:54 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>> On Wed, Jan 5, 2011 at 2:38 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>> On Wed, Jan 5, 2011 at 2:15 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>> On looking more closely at the output, I thought that the problem may
>>>> arise in loading the Rcpp package because that is when the function
>>>> init_Rcpp_cache() is evaluated.  So I ran another test which was
>>>> simply
>>>>
>>>> gctorture(TRUE)
>>>> library(Rcpp)
>>>>
>>>> run with -d valgrind.
>>>>
>>>> Unfortunately (well, at least from my point of view) that test didn't
>>>> show any problems.  I'm at a bit of a loss where to go on this.
>>>
>>> Perhaps Romain or Dirk could clarify for me when init_Rcpp_cache
>>> should be called.  It appears to be getting called twice, once
>>> explicitly in the .onLoad expression in Rcpp/R/zzz.R and once
>>> implicitly by being part of the R_init_Rcpp function defined in
>>> Rcpp/src/Module.cpp.  Is this intentional?  Should one or both of
>>> those be a maybe_init() call?  I think as it stands the second call
>>> will overwrite the contents created by the first call.
>>>
>>> I'm kind of grasping at straws here but I think there might be a
>>> problem with the cache being created when the Rcpp namespace is
>>> attached and then called again when the Rcpp package is loaded in my
>>> example.
>>
>> I was able to remove the call to init_Rcpp_cache in the .onLoad
>> function in Rcpp/R/zzz.R and still run R CMD check Rcpp successfully.
>> Unfortunately, even with this change loading the lme4a package fails
>> when gctorture is turned on.  Transcript enclosed.  Looks like I get
>> to learn more about the Module loading process.
>
> I'm still flailing away at this.  It is one of those really messy
> debugging situations in that I can't make it fail except on the test
> of the full example and using both gctorture and valgrind.
>
> It appears that the .onLoad function in lme4a is failing when called
> in loadNameSpace.  The .onLoad function was
>
> .NameSpace <- environment()
> .onLoad <- function(libname, pkgname)
> {
>    ## For Matrix API change (Oct.2009) - silence the warning:
>    assign("det_CHMfactor.warn", FALSE, envir = Matrix:::.MatrixEnv)
>    mod <- Module("lme4")
>    populate(mod, .NameSpace)
> }
>
> So it seems that either the call to Module() or the call to populate()
> is failing and the tryCatch block in runHook() within loadNamespace()
> is creating an error message.  The call: and message: parts of the
> error message are generated from the error condition object using
> conditionCall() and conditionMessage().  To me it looks like the call
> stack has gotten corrupted because there are no calls like that.  The
> form of the call seems to be that of lazyLoadDBfetch in
> $R_SRC/src/library/base/R/lazyLoad.R, which is called in a most
> mysterious way.
>
> If anyone can shed any light on this, I would appreciate it.  I have
> another gctorture/valgrind run underway in which I don't use .onLoad
> but call Module and populate explicitly to see if I can track down
> where it is going south.  Suggestions welcome.

If I disable the .onLoad function and call Module() and populate()
after loading the package and with gctorture(TRUE) then I get a
failure in the call to populate, with the same mysterious value of the
call and the message

> library(lme4a)
Loading required package: Matrix
Loading required package: lattice

Attaching package: 'Matrix'

The following object(s) are masked from 'package:base':

    det

Loading required package: minqa
Loading required package: Rcpp
Loading required package: MatrixModels

Attaching package: 'MatrixModels'

The following object(s) are masked from 'package:Matrix':

    model.Matrix

> gctorture(TRUE)
> mod <- Module("lme4", "lme4a")
> populate(mod, .GlobalEnv)
Error in list(c(78682L, 78L),
"/home/bates/R/x86_64-unknown-linux-gnu-library/2.13/Rcpp/R/Rcpp.rdb",
TRUE, function (n)  :
  unused argument(s)
("/home/bates/R/x86_64-unknown-linux-gnu-library/2.13/Rcpp/R/Rcpp.rdb",
TRUE, function (n)

Can someone who knows more about the Module code than I (meaning
Romain or John) tell me if that call could originate from within
populate?  It seems highly unlikely to me but I can't rule it out.

By the way, this was using Rcpp_0.9.0.2 which still has two calls to
init_Rcpp_cache.  I'll change that and see if it makes a difference.

>>>> On Wed, Jan 5, 2011 at 11:55 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>>> This time with the enclosure :-)
>>>>>
>>>>> On Wed, Jan 5, 2011 at 11:52 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>>>> I don't know whether this is through error on my part or because of an
>>>>>> "infelicity" in the Rcpp module code but the lme4a package, which now
>>>>>> uses Rcpp modules extensively, ends up with some difficult-to-trace
>>>>>> memory corruption issues.  Yesterday i finally bit the bullet and ran
>>>>>> a test with gctorture(TRUE) and valgrind enabled.  It takes a very
>>>>>> long time and results in a segfault when trying to load the package.
>>>>>> I enclose the transcript.  I should say that this is using Rcpp_0.9.0
>>>>>> from CRAN, not the SVN version of Rcpp.
>>>>>>
>>>>>> I just got these results this morning (it was running overnight) and
>>>>>> haven't looked at the code in Module.cpp and cache.cpp yet.  If it
>>>>>> seems likely that the code is beyond me I can try to work out a
>>>>>> simpler example that triggers the problem.
>>>>>>
>>>>>
>>>>
>>>
>>
>