[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption

[Romain Francois]

Le 11/01/11 19:46, Douglas Bates a écrit :
> On Tue, Jan 11, 2011 at 12:27 PM, Dominick Samperi<djsamperi at gmail.com>  wrote:
>>
>>
>> On Tue, Jan 11, 2011 at 1:20 PM, Romain Francois<romain at r-enthusiasts.com>
>> wrote:
>>>
>>> Le 11/01/11 19:12, Davor Cubranic a écrit :
>>>>
>>>> I think an important point from Doug has been lost in the subsequent
>>>> 20-odd messages of flamebombing:
>>>>
>>>>>> I do not see this as compatible with the C++ Design principle we use
>>>>>> whereby
>>>>>> protection / unprotection occurs relative to the end of scope -- and
>>>>>> not
>>>>>> after every single assignment or allocation.
>>>>>>
>>>>>> In other words, gctorture() may well be a fine test for the C API and
>>>>>> its
>>>>>> PROTECT and UNPROTECT at every step but possibly not quite as much for
>>>>>> Rcpp.
>>>>>
>>>>> I don't think so.  In the situation that Dominick is describing the C
>>>>> API is being used (calls to Rf_install, Rf_lang1, Rf_eval, ...) and
>>>>> you have to play by the rules of the C API.  Essentially every time
>>>>> that you call a function in the C API that can cause a memory
>>>>> allocation you are open yourself to the possibility of having the
>>>>> garbage collector run and getting unprotected SEXPs blown away.  And,
>>>>> naturally, Rf_eval can cause memory allocation.
>>>>>
>>>>> I understood Dominick to be saying that in the code related to Modules
>>>>> and the evaluator and all that which we have been trying to debug
>>>>> there are such cases of the use of the C API that are unsafe.
>>>>
>>>> Can anyone comment whether this is correct?
>>>>
>>>> Davor
>>>
>>> Yep. Doug's analysis is right. Rcpp is implemented with the C R API, and
>>> apparently there were a few places where we were not careful enough. Most
>>> notably in calls to Rf_lcons and Rf_cons. This has been partially dealt with
>>> today.
>>
>> Just for the record, Doug is summarizing my analysis, based on several
>> examples that I posted to this thread,
>> and that I am pleased to see have inspired some remedial action.
>
> Sorry for not responding earlier.  I'm in the middle of teaching a short course.
>
> Dominick is correct that it was his analysis that picked up the
> failures to PROTECT/UNPROTECT in cases that were causing at least some
> of the problems with loading Modules.  Credit where credit is due.
>
>   As Romain has indicated, some of the problems have been fixed but
> apparently problems still remain.  Debugging memory protection
> problems is often very difficult.

It is. Not sure what is my next step here. Remaining problems seem not 
directly related to Rcpp, but rather associated with lazy loading. See:

 > require(inline)
Le chargement a nécessité le package : inline
 > require(Rcpp)
Le chargement a nécessité le package : Rcpp
 > inc <- '
+
+ class Randomizer {
+ public:
+
+     // Randomizer() : min(0), max(1){}
+     Randomizer( double min_, double max_) : min(min_), max(max_){}
+
+     NumericVector get( int n ){
+         RNGScope scope ;
+         return runif( n, min, max );
+     }
+
+ private:
+     double min, max ;
+ } ;
+
+ RCPP_MODULE(mod){
+
+     class_<Randomizer>( "Randomizer" )
+
+         // No default: .default_constructor()
+         .constructor<double,double>()
+
+         .method( "get" , &Randomizer::get ) ;
+
+ }
+ '
 >     fx <- cxxfunction( , '', includes = inc, plugin = "Rcpp" )
 >
 > mod <- Module( "mod", getDynLib( fx ) )
 > gctorture(TRUE)
 > Rcpp:::.getModulePointer( mod )
Erreur dans list(c(162342L, 224L), 
"/Library/Frameworks/R.framework/Resources/library/base/R/base.rdb", 
TRUE, function (n)  :
   le premier argument doit être une liste nommée
De plus : Messages d'avis :
1: In asMethod(object) : NAs introduits lors de la conversion automatique
2: In asMethod(object) : NAs introduits lors de la conversion automatique
 > traceback()
8: list(c(162342L, 224L), 
"/Library/Frameworks/R.framework/Resources/library/base/R/base.rdb", 
TRUE, function (n)
    {
        if (existsInFrame(n, envenv)) getFromFrame(n, envenv) else {
            e <- mkenv()
            set(n, e, envenv)
            key <- getFromFrame(n, env)
            data <- lazyLoadDBfetch(key, datafile, compressed, envhook)
            if (is.null(data$enclos)) parent.env(e) <- emptyenv() else 
parent.env(e) <- data$enclos
            vars <- names(data$bindings)
            for (i in seq_along(vars)) set(vars[i], data$bindings[[i]],
                e)
            if (!is.null(data$attributes)) attributes(e) <- data$attributes
            if (!is.null(data$isS4) && data$isS4) .Call("R_setS4Object",
                e, TRUE, TRUE, PACKAGE = "base")
            if (!is.null(data$locked) && data$locked) 
.Internal(lockEnvironment(e,
                FALSE))
            e
        }
    })
7: asEnv(refMethods)
6: initialize(value, ...)
5: initialize(value, ...)
4: new("refClassRepresentation", getClassDef(Class, where = where),
        fieldClasses = fieldClasses, refMethods = asEnv(refMethods),
        fieldPrototypes = asEnv(fieldPrototypes), refSuperClasses = 
refSuperClasses)
3: setRefClass(clname, fields = fields, contains = "C++Object",
        methods = methods, where = where)
2: Module(module, mustStart = TRUE)
1: Rcpp:::.getModulePointer(mod)

Romain

> As Dirk indicated, the use of C++ helps by removing much of the tedium
> of memory allocation and freeing.  However, until R is rewritten in
> C++, there will still be these difficulties with going through R's API
> based on C.
>
>>>
>>> This is not completely over though apparently. The code below still does
>>> not support the torture.
>>>
>>> To be continued ...
>>>
>>> Romain
>>>
>>>
>>>
>>>
>>> require(inline)
>>> require(Rcpp)
>>> inc<- '
>>>
>>> class Randomizer {
>>> public:
>>>
>>>     // Randomizer() : min(0), max(1){}
>>>     Randomizer( double min_, double max_) : min(min_), max(max_){}
>>>
>>>     NumericVector get( int n ){
>>>         RNGScope scope ;
>>>         return runif( n, min, max );
>>>     }
>>>
>>> private:
>>>     double min, max ;
>>> } ;
>>>
>>> RCPP_MODULE(mod){
>>>
>>>     class_<Randomizer>( "Randomizer" )
>>>
>>>         // No default: .default_constructor()
>>>         .constructor<double,double>()
>>>
>>>         .method( "get" ,&Randomizer::get ) ;
>>>
>>> }
>>> '
>>>     fx<- cxxfunction( , '', includes = inc, plugin = "Rcpp" )
>>>
>>> mod<- Module( "mod", getDynLib( fx ) )
>>> gctorture(TRUE)
>>> Rcpp:::.getModulePointer( mod )
>>>
>>>
>


-- 
Romain Francois
Professional R Enthusiast
+33(0) 6 28 91 30 30
http://romainfrancois.blog.free.fr
|- http://bit.ly/fT2rZM : highlight 0.2-5
|- http://bit.ly/gpCSpH : Evolution of Rcpp code size
`- http://bit.ly/hovakS : RcppGSL initial release