[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption

[Douglas Bates]

On Tue, Jan 11, 2011 at 12:27 PM, Dominick Samperi <djsamperi at gmail.com> wrote:
>
>
> On Tue, Jan 11, 2011 at 1:20 PM, Romain Francois <romain at r-enthusiasts.com>
> wrote:
>>
>> Le 11/01/11 19:12, Davor Cubranic a écrit :
>>>
>>> I think an important point from Doug has been lost in the subsequent
>>> 20-odd messages of flamebombing:
>>>
>>>>> I do not see this as compatible with the C++ Design principle we use
>>>>> whereby
>>>>> protection / unprotection occurs relative to the end of scope -- and
>>>>> not
>>>>> after every single assignment or allocation.
>>>>>
>>>>> In other words, gctorture() may well be a fine test for the C API and
>>>>> its
>>>>> PROTECT and UNPROTECT at every step but possibly not quite as much for
>>>>> Rcpp.
>>>>
>>>> I don't think so.  In the situation that Dominick is describing the C
>>>> API is being used (calls to Rf_install, Rf_lang1, Rf_eval, ...) and
>>>> you have to play by the rules of the C API.  Essentially every time
>>>> that you call a function in the C API that can cause a memory
>>>> allocation you are open yourself to the possibility of having the
>>>> garbage collector run and getting unprotected SEXPs blown away.  And,
>>>> naturally, Rf_eval can cause memory allocation.
>>>>
>>>> I understood Dominick to be saying that in the code related to Modules
>>>> and the evaluator and all that which we have been trying to debug
>>>> there are such cases of the use of the C API that are unsafe.
>>>
>>> Can anyone comment whether this is correct?
>>>
>>> Davor
>>
>> Yep. Doug's analysis is right. Rcpp is implemented with the C R API, and
>> apparently there were a few places where we were not careful enough. Most
>> notably in calls to Rf_lcons and Rf_cons. This has been partially dealt with
>> today.
>
> Just for the record, Doug is summarizing my analysis, based on several
> examples that I posted to this thread,
> and that I am pleased to see have inspired some remedial action.

Sorry for not responding earlier.  I'm in the middle of teaching a short course.

Dominick is correct that it was his analysis that picked up the
failures to PROTECT/UNPROTECT in cases that were causing at least some
of the problems with loading Modules.  Credit where credit is due.

 As Romain has indicated, some of the problems have been fixed but
apparently problems still remain.  Debugging memory protection
problems is often very difficult.

As Dirk indicated, the use of C++ helps by removing much of the tedium
of memory allocation and freeing.  However, until R is rewritten in
C++, there will still be these difficulties with going through R's API
based on C.

>>
>> This is not completely over though apparently. The code below still does
>> not support the torture.
>>
>> To be continued ...
>>
>> Romain
>>
>>
>>
>>
>> require(inline)
>> require(Rcpp)
>> inc <- '
>>
>> class Randomizer {
>> public:
>>
>>    // Randomizer() : min(0), max(1){}
>>    Randomizer( double min_, double max_) : min(min_), max(max_){}
>>
>>    NumericVector get( int n ){
>>        RNGScope scope ;
>>        return runif( n, min, max );
>>    }
>>
>> private:
>>    double min, max ;
>> } ;
>>
>> RCPP_MODULE(mod){
>>
>>    class_<Randomizer>( "Randomizer" )
>>
>>        // No default: .default_constructor()
>>        .constructor<double,double>()
>>
>>        .method( "get" , &Randomizer::get ) ;
>>
>> }
>> '
>>    fx <- cxxfunction( , '', includes = inc, plugin = "Rcpp" )
>>
>> mod <- Module( "mod", getDynLib( fx ) )
>> gctorture(TRUE)
>> Rcpp:::.getModulePointer( mod )
>>
>>