[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption

[Douglas Bates]

On Thu, Jan 6, 2011 at 2:19 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
> In the continuing saga ...
>
> On Thu, Jan 6, 2011 at 12:57 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
>> On Thu, Jan 6, 2011 at 9:54 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>> On Wed, Jan 5, 2011 at 2:38 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>> On Wed, Jan 5, 2011 at 2:15 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>>> On looking more closely at the output, I thought that the problem may
>>>>> arise in loading the Rcpp package because that is when the function
>>>>> init_Rcpp_cache() is evaluated.  So I ran another test which was
>>>>> simply
>>>>>
>>>>> gctorture(TRUE)
>>>>> library(Rcpp)
>>>>>
>>>>> run with -d valgrind.
>>>>>
>>>>> Unfortunately (well, at least from my point of view) that test didn't
>>>>> show any problems.  I'm at a bit of a loss where to go on this.
>>>>
>>>> Perhaps Romain or Dirk could clarify for me when init_Rcpp_cache
>>>> should be called.  It appears to be getting called twice, once
>>>> explicitly in the .onLoad expression in Rcpp/R/zzz.R and once
>>>> implicitly by being part of the R_init_Rcpp function defined in
>>>> Rcpp/src/Module.cpp.  Is this intentional?  Should one or both of
>>>> those be a maybe_init() call?  I think as it stands the second call
>>>> will overwrite the contents created by the first call.
>>>>
>>>> I'm kind of grasping at straws here but I think there might be a
>>>> problem with the cache being created when the Rcpp namespace is
>>>> attached and then called again when the Rcpp package is loaded in my
>>>> example.
>>>
>>> I was able to remove the call to init_Rcpp_cache in the .onLoad
>>> function in Rcpp/R/zzz.R and still run R CMD check Rcpp successfully.
>>> Unfortunately, even with this change loading the lme4a package fails
>>> when gctorture is turned on.  Transcript enclosed.  Looks like I get
>>> to learn more about the Module loading process.
>>
>> I'm still flailing away at this.  It is one of those really messy
>> debugging situations in that I can't make it fail except on the test
>> of the full example and using both gctorture and valgrind.
>>
>> It appears that the .onLoad function in lme4a is failing when called
>> in loadNameSpace.  The .onLoad function was
>>
>> .NameSpace <- environment()
>> .onLoad <- function(libname, pkgname)
>> {
>>    ## For Matrix API change (Oct.2009) - silence the warning:
>>    assign("det_CHMfactor.warn", FALSE, envir = Matrix:::.MatrixEnv)
>>    mod <- Module("lme4")
>>    populate(mod, .NameSpace)
>> }
>>
>> So it seems that either the call to Module() or the call to populate()
>> is failing and the tryCatch block in runHook() within loadNamespace()
>> is creating an error message.  The call: and message: parts of the
>> error message are generated from the error condition object using
>> conditionCall() and conditionMessage().  To me it looks like the call
>> stack has gotten corrupted because there are no calls like that.  The
>> form of the call seems to be that of lazyLoadDBfetch in
>> $R_SRC/src/library/base/R/lazyLoad.R, which is called in a most
>> mysterious way.
>>
>> If anyone can shed any light on this, I would appreciate it.  I have
>> another gctorture/valgrind run underway in which I don't use .onLoad
>> but call Module and populate explicitly to see if I can track down
>> where it is going south.  Suggestions welcome.
>
> If I disable the .onLoad function and call Module() and populate()
> after loading the package and with gctorture(TRUE) then I get a
> failure in the call to populate, with the same mysterious value of the
> call and the message
>
>> library(lme4a)
> Loading required package: Matrix
> Loading required package: lattice
>
> Attaching package: 'Matrix'
>
> The following object(s) are masked from 'package:base':
>
>    det
>
> Loading required package: minqa
> Loading required package: Rcpp
> Loading required package: MatrixModels
>
> Attaching package: 'MatrixModels'
>
> The following object(s) are masked from 'package:Matrix':
>
>    model.Matrix
>
>> gctorture(TRUE)
>> mod <- Module("lme4", "lme4a")
>> populate(mod, .GlobalEnv)
> Error in list(c(78682L, 78L),
> "/home/bates/R/x86_64-unknown-linux-gnu-library/2.13/Rcpp/R/Rcpp.rdb",
> TRUE, function (n)  :
>  unused argument(s)
> ("/home/bates/R/x86_64-unknown-linux-gnu-library/2.13/Rcpp/R/Rcpp.rdb",
> TRUE, function (n)
>
> Can someone who knows more about the Module code than I (meaning
> Romain or John) tell me if that call could originate from within
> populate?  It seems highly unlikely to me but I can't rule it out.
>
> By the way, this was using Rcpp_0.9.0.2 which still has two calls to
> init_Rcpp_cache.  I'll change that and see if it makes a difference.

As they say in this season of (American) football games, "on further
review" it looks as if that string might have originated in a PROMSXP.
 This is definitely getting wierd.

>>>>> On Wed, Jan 5, 2011 at 11:55 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>>>> This time with the enclosure :-)
>>>>>>
>>>>>> On Wed, Jan 5, 2011 at 11:52 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>>>>> I don't know whether this is through error on my part or because of an
>>>>>>> "infelicity" in the Rcpp module code but the lme4a package, which now
>>>>>>> uses Rcpp modules extensively, ends up with some difficult-to-trace
>>>>>>> memory corruption issues.  Yesterday i finally bit the bullet and ran
>>>>>>> a test with gctorture(TRUE) and valgrind enabled.  It takes a very
>>>>>>> long time and results in a segfault when trying to load the package.
>>>>>>> I enclose the transcript.  I should say that this is using Rcpp_0.9.0
>>>>>>> from CRAN, not the SVN version of Rcpp.
>>>>>>>
>>>>>>> I just got these results this morning (it was running overnight) and
>>>>>>> haven't looked at the code in Module.cpp and cache.cpp yet.  If it
>>>>>>> seems likely that the code is beyond me I can try to work out a
>>>>>>> simpler example that triggers the problem.
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>