[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption

[Douglas Bates]

On Thu, Jan 6, 2011 at 9:54 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
> On Wed, Jan 5, 2011 at 2:38 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
>> On Wed, Jan 5, 2011 at 2:15 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>> On looking more closely at the output, I thought that the problem may
>>> arise in loading the Rcpp package because that is when the function
>>> init_Rcpp_cache() is evaluated.  So I ran another test which was
>>> simply
>>>
>>> gctorture(TRUE)
>>> library(Rcpp)
>>>
>>> run with -d valgrind.
>>>
>>> Unfortunately (well, at least from my point of view) that test didn't
>>> show any problems.  I'm at a bit of a loss where to go on this.
>>
>> Perhaps Romain or Dirk could clarify for me when init_Rcpp_cache
>> should be called.  It appears to be getting called twice, once
>> explicitly in the .onLoad expression in Rcpp/R/zzz.R and once
>> implicitly by being part of the R_init_Rcpp function defined in
>> Rcpp/src/Module.cpp.  Is this intentional?  Should one or both of
>> those be a maybe_init() call?  I think as it stands the second call
>> will overwrite the contents created by the first call.
>>
>> I'm kind of grasping at straws here but I think there might be a
>> problem with the cache being created when the Rcpp namespace is
>> attached and then called again when the Rcpp package is loaded in my
>> example.
>
> I was able to remove the call to init_Rcpp_cache in the .onLoad
> function in Rcpp/R/zzz.R and still run R CMD check Rcpp successfully.
> Unfortunately, even with this change loading the lme4a package fails
> when gctorture is turned on.  Transcript enclosed.  Looks like I get
> to learn more about the Module loading process.

I'm still flailing away at this.  It is one of those really messy
debugging situations in that I can't make it fail except on the test
of the full example and using both gctorture and valgrind.

It appears that the .onLoad function in lme4a is failing when called
in loadNameSpace.  The .onLoad function was

.NameSpace <- environment()
.onLoad <- function(libname, pkgname)
{
    ## For Matrix API change (Oct.2009) - silence the warning:
    assign("det_CHMfactor.warn", FALSE, envir = Matrix:::.MatrixEnv)
    mod <- Module("lme4")
    populate(mod, .NameSpace)
}

So it seems that either the call to Module() or the call to populate()
is failing and the tryCatch block in runHook() within loadNamespace()
is creating an error message.  The call: and message: parts of the
error message are generated from the error condition object using
conditionCall() and conditionMessage().  To me it looks like the call
stack has gotten corrupted because there are no calls like that.  The
form of the call seems to be that of lazyLoadDBfetch in
$R_SRC/src/library/base/R/lazyLoad.R, which is called in a most
mysterious way.

If anyone can shed any light on this, I would appreciate it.  I have
another gctorture/valgrind run underway in which I don't use .onLoad
but call Module and populate explicitly to see if I can track down
where it is going south.  Suggestions welcome.


>>> On Wed, Jan 5, 2011 at 11:55 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>> This time with the enclosure :-)
>>>>
>>>> On Wed, Jan 5, 2011 at 11:52 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>>> I don't know whether this is through error on my part or because of an
>>>>> "infelicity" in the Rcpp module code but the lme4a package, which now
>>>>> uses Rcpp modules extensively, ends up with some difficult-to-trace
>>>>> memory corruption issues.  Yesterday i finally bit the bullet and ran
>>>>> a test with gctorture(TRUE) and valgrind enabled.  It takes a very
>>>>> long time and results in a segfault when trying to load the package.
>>>>> I enclose the transcript.  I should say that this is using Rcpp_0.9.0
>>>>> from CRAN, not the SVN version of Rcpp.
>>>>>
>>>>> I just got these results this morning (it was running overnight) and
>>>>> haven't looked at the code in Module.cpp and cache.cpp yet.  If it
>>>>> seems likely that the code is beyond me I can try to work out a
>>>>> simpler example that triggers the problem.
>>>>>
>>>>
>>>
>>
>