[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption

Thu Jan 6 16:54:10 CET 2011

On Wed, Jan 5, 2011 at 2:38 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
> On Wed, Jan 5, 2011 at 2:15 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
>> On looking more closely at the output, I thought that the problem may
>> arise in loading the Rcpp package because that is when the function
>> init_Rcpp_cache() is evaluated.  So I ran another test which was
>> simply
>>
>> gctorture(TRUE)
>> library(Rcpp)
>>
>> run with -d valgrind.
>>
>> Unfortunately (well, at least from my point of view) that test didn't
>> show any problems.  I'm at a bit of a loss where to go on this.
>
> Perhaps Romain or Dirk could clarify for me when init_Rcpp_cache
> should be called.  It appears to be getting called twice, once
> explicitly in the .onLoad expression in Rcpp/R/zzz.R and once
> implicitly by being part of the R_init_Rcpp function defined in
> Rcpp/src/Module.cpp.  Is this intentional?  Should one or both of
> those be a maybe_init() call?  I think as it stands the second call
> will overwrite the contents created by the first call.
>
> I'm kind of grasping at straws here but I think there might be a
> problem with the cache being created when the Rcpp namespace is
> attached and then called again when the Rcpp package is loaded in my
> example.

I was able to remove the call to init_Rcpp_cache in the .onLoad
function in Rcpp/R/zzz.R and still run R CMD check Rcpp successfully.
Unfortunately, even with this change loading the lme4a package fails
when gctorture is turned on.  Transcript enclosed.  Looks like I get
to learn more about the Module loading process.

>> On Wed, Jan 5, 2011 at 11:55 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>> This time with the enclosure :-)
>>>
>>> On Wed, Jan 5, 2011 at 11:52 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>> I don't know whether this is through error on my part or because of an
>>>> "infelicity" in the Rcpp module code but the lme4a package, which now
>>>> uses Rcpp modules extensively, ends up with some difficult-to-trace
>>>> memory corruption issues.  Yesterday i finally bit the bullet and ran
>>>> a test with gctorture(TRUE) and valgrind enabled.  It takes a very
>>>> long time and results in a segfault when trying to load the package.
>>>> I enclose the transcript.  I should say that this is using Rcpp_0.9.0
>>>> from CRAN, not the SVN version of Rcpp.
>>>>
>>>> I just got these results this morning (it was running overnight) and
>>>> haven't looked at the code in Module.cpp and cache.cpp yet.  If it
>>>> seems likely that the code is beyond me I can try to work out a
>>>> simpler example that triggers the problem.
>>>>
>>>
>>
>
-------------- next part --------------
==24583== Memcheck, a memory error detector
==24583== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==24583== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for copyright info
==24583== Command: /home/bates/build/R-devel/bin/exec/R --no-save --no-restore
==24583== 
==24583== Conditional jump or move depends on uninitialised value(s)
==24583==    at 0x6870E40: inflateReset2 (in /lib/libz.so.1.2.3.4)
==24583==    by 0x6870F2F: inflateInit2_ (in /lib/libz.so.1.2.3.4)
==24583==    by 0x686B1BC: ??? (in /lib/libz.so.1.2.3.4)
==24583==    by 0x4EB019E: gzfile_open (connections.c:1186)
==24583==    by 0x4EADD57: do_gzfile (connections.c:1745)
==24583==    by 0x4F6152A: do_internal (names.c:1197)
==24583==    by 0x4F1320A: Rf_eval (eval.c:469)
==24583==    by 0x4F15C2C: Rf_applyClosure (eval.c:811)
==24583==    by 0x4F130E7: Rf_eval (eval.c:513)
==24583==    by 0x4F1A55F: do_set (eval.c:1698)
==24583==    by 0x4F1320A: Rf_eval (eval.c:469)
==24583==    by 0x4F14C6B: do_begin (eval.c:1418)
==24583== 
==24583== Conditional jump or move depends on uninitialised value(s)
==24583==    at 0x6870E40: inflateReset2 (in /lib/libz.so.1.2.3.4)
==24583==    by 0x6870F2F: inflateInit2_ (in /lib/libz.so.1.2.3.4)
==24583==    by 0x686B648: uncompress (in /lib/libz.so.1.2.3.4)
==24583==    by 0x4EA4A09: R_decompress1 (connections.c:5143)
==24583==    by 0x4FDD813: do_lazyLoadDBfetch (serialize.c:2378)
==24583==    by 0x4F1333F: Rf_eval (eval.c:498)
==24583==    by 0x4F13666: forcePromise (eval.c:331)
==24583==    by 0x4F12EDB: Rf_eval (eval.c:450)
==24583==    by 0x4EFAB0A: Rf_findFun (envir.c:1293)
==24583==    by 0x4F13063: Rf_eval (eval.c:456)
==24583==    by 0x4F4D165: R_ReplFile (main.c:100)
==24583==    by 0x4F4D279: R_LoadProfile (main.c:662)
==24583== 

R version 2.13.0 Under development (unstable) (2011-01-04 r53913)
Copyright (C) 2011 The R Foundation for Statistical Computing
ISBN 3-900051-07-0
Platform: x86_64-unknown-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> gctorture(TRUE)
> library(lme4a)
Loading required package: Matrix
Loading required package: lattice

Attaching package: 'Matrix'

The following object(s) are masked from 'package:base':

    det

Loading required package: minqa
Loading required package: Rcpp
Loading required package: MatrixModels

Attaching package: 'MatrixModels'

The following object(s) are masked from 'package:Matrix':

    model.Matrix

Error : .onLoad failed in loadNamespace() for 'lme4a', details:
  call: list(c(78494L, 76L), "/home/bates/R/x86_64-unknown-linux-gnu-library/2.13/Rcpp/R/Rcpp.rdb", TRUE, function (n) 
  error: unused argument(s) ("/home/bates/R/x86_64-unknown-linux-gnu-library/2.13/Rcpp/R/Rcpp.rdb", TRUE, function (n) 
Error: package/namespace load failed for 'lme4a'
Execution halted
==24583== 
==24583== HEAP SUMMARY:
==24583==     in use at exit: 53,414,344 bytes in 27,841 blocks
==24583==   total heap usage: 91,405 allocs, 63,564 frees, 136,838,659 bytes allocated
==24583== 
==24583== LEAK SUMMARY:
==24583==    definitely lost: 188 bytes in 4 blocks
==24583==    indirectly lost: 480 bytes in 20 blocks
==24583==      possibly lost: 48,596,026 bytes in 26,674 blocks
==24583==    still reachable: 4,817,650 bytes in 1,143 blocks
==24583==         suppressed: 0 bytes in 0 blocks
==24583== Rerun with --leak-check=full to see details of leaked memory
==24583== 
==24583== For counts of detected and suppressed errors, rerun with: -v
==24583== Use --track-origins=yes to see where uninitialised values come from
==24583== ERROR SUMMARY: 2456 errors from 2 contexts (suppressed: 46 from 7)