[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption

[Douglas Bates]

On Sun, Jan 9, 2011 at 12:21 PM, Dominick Samperi <djsamperi at gmail.com> wrote:
>
>
> On Sun, Jan 9, 2011 at 1:07 PM, Dirk Eddelbuettel <edd at debian.org> wrote:
>>
>> On 9 January 2011 at 12:12, Dominick Samperi wrote:
>> | Actually, the problem seems to be pretty transparent, and the
>> | solution is the same (see below). This gets you through Evaluator,
>> | but you fail in SlotProxy in the call to "$<-". The fact that these
>> | problems only appear when gctorture() is turned on suggests
>> | that they all of the same explanation: unprotected SEXP's.
>> |
>> | same:
>> | //#define UNSAFE_CODE
>> | #ifdef UNSAFE_CODE
>> |     int error = LOGICAL( Rf_eval( Rf_lang1( Rf_install("errorOccured")
>> ),
>> | RCPP ) )[0];
>> | #else
>> |     SEXP cmd = PROTECT(Rf_lang1( Rf_install("errorOccured")));
>> |     int error = LOGICAL(Rf_eval( cmd, RCPP ))[0];
>> |     UNPROTECT(1);
>> | #endif
>> |     Rprintf("Got error = %d\n", error);
>> |
>>
>> There may be an impedance mismatch going on.  'Writing R Extension' says
>>
>>    4.3.1 Using gctorture
>>    ---------------------
>>
>>    We can help to detect memory problems earlier by running garbage
>>    collection as often as possible.  This is achieved by
>>    `gctorture(TRUE)', which as described on its help page
>>
>>         Provokes garbage collection on (nearly) every memory allocation.
>>         Intended to ferret out memory protection bugs.  Also makes R run
>>         _very_ slowly, unfortunately.
>>
>>    The reference to `memory protection' is to missing C-level calls to
>>    `PROTECT'/`UNPROTECT' (*note Garbage Collection::) which if missing
>>    allow R objects to be garbage-collected when they are still in use.
>>    But it can also help with other memory-related errors.
>>
>>       Normally running under `gctorture(TRUE)' will just produce a crash
>>    earlier in the R program, hopefully close to the actual cause. See the
>>    next section for how to decipher such crashes.
>>
>>       It is possible to run all the examples, tests and vignettes covered
>>    by `R CMD check' under `gctorture(TRUE)' by using the option
>>    `--use-gct'.
>>
>> I do not see this as compatible with the C++ Design principle we use
>> whereby
>> protection / unprotection occurs relative to the end of scope -- and not
>> after every single assignment or allocation.
>
> Thanks, but note that Doug's original report was not about gctorture(). We
> are using this tool to try to fix the memory corruption problems, and this
> tool seems to be pretty helpful for this purpose, even if it might be too
> demanding in some situations.
>
> Dominick
>
>>
>> In other words, gctorture() may well be a fine test for the C API and its
>> PROTECT and UNPROTECT at every step but possibly not quite as much for
>> Rcpp.
>>
>> Dirk
>>
>> PS Here is slight generalization of your examples. Cases 2 and 3 do not
>> bomb,
>> and Case 3 is using almost all Rcpp idioms (but for the final Rf_eval
>> call).
>>
>>
>> -----------------------------------------------------------------------------
>> // -*- mode: C++; c-indent-level: 4; c-basic-offset: 4; tab-width: 8 -*-
>>
>> #include "Rcpp.h"
>>
>> RcppExport SEXP testfft(SEXP vec, SEXP env, SEXP choiceS) {
>>
>>    int choice = Rcpp::as<int>(choiceS);
>>    SEXP res = R_NilValue;
>>
>>    if (choice == 1) {
>>        Rprintf("In case 1\n");
>>        // may not survive gctorture(), sometimes!
>>        SEXP call = ::Rf_lcons(Rf_install("fft"),
>>                               Rf_cons(vec , R_NilValue));
>>        res = PROTECT(Rf_eval(call, env));
>>        UNPROTECT(1);
>>
>>    } else if (choice == 2) {
>>        Rprintf("In case 2\n");
>>        // always survives gctorture()
>>        SEXP call = PROTECT(::Rf_lcons(Rf_install("fft"),
>>                                       Rf_cons(vec , R_NilValue)));
>>        res = PROTECT(Rf_eval(call, env));
>>        UNPROTECT(2);
>>
>>    } else if (choice == 3) {
>>        Rprintf("In case 3\n");
>>        Rcpp::Function fft("fft");
>>        Rcpp::Environment env_(env);
>>        Rcpp::Language call(fft);
>>        call.push_back(vec);
>>        res = Rf_eval(call, env_);
>>    }
>>    return res;
>> }
>>
>> -----------------------------------------------------------------------------
>>
>> and
>>
>>
>> -----------------------------------------------------------------------------
>> # Tests Rcpp Evaluator code with PROTECTION fix and gctorture ON.
>> # Should get a complex vector of zeros.
>> library(Rcpp)
>> gctorture()
>> dyn.load("testfft.so")
>> for (i in 2:3) {  # bombs on 1 but not 2 or 3
>>    print(.Call('testfft', 1:20, getNamespace("stats"), i) - fft(1:20))
>> }
>> cat("still here\n")
>>
>> -----------------------------------------------------------------------------
>>
>>
>> edd at max:/tmp/samperi-gc$ r -lRcpp -e'Rcpp:::SHLIB("testfft.cpp")'
>> ccache g++ -I/usr/share/R/include
>> -I/usr/local/lib/R/site-library/Rcpp/include     -fpic  -g -O3 -Wall -pipe
>> -pedantic -Wno-variadic-macros -c testfft.cpp -o testfft.o
>> g++ -shared -o testfft.so testfft.o
>> -L/usr/local/lib/R/site-library/Rcpp/lib -lRcpp
>> -Wl,-rpath,/usr/local/lib/R/site-library/Rcpp/lib -L/usr/lib64/R/lib -lR
>> edd at max:/tmp/samperi-gc$ r torture.R
>> Loading required package: methods
>> In case 2
>>  [1] 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i
>> 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i
>> In case 3
>>  [1] 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i
>> 0+0i 0+0i 0+0i 0+0i 0+0i 0+0i
>> still here
>> edd at max:/tmp/samperi-gc$
>>
>> --
>> Dirk Eddelbuettel | edd at debian.org | http://dirk.eddelbuettel.com

I'm sorry to say that I will need to abandon the debugging of this.

I have converted the code in Rcpp/R/exceptions.R to using a R
functions with a common environment to keep track of the error
conditions.  I can get access to those from within the C++ code
through values established during the execution of R_init_Rcpp but the
next time I try to use them in the C++ code they are gone, although
they are still there are the R side.  It is bizarre - I have no idea
what is going on.

Anyway, I need to turn my attention back to lme4a.  I am giving a
short course starting tomorrow and must have a version of lme4a that
can compile and run.  I have been converting it to use Rcpp Modules
and it looks like I will need to strip all that code out.  I hope that
the problems with memory protection are isolated in the Modules
sections of Rcpp.