[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption
Dominick Samperi
djsamperi at gmail.com
Sun Jan 9 18:12:25 CET 2011
On Sun, Jan 9, 2011 at 9:21 AM, Dominick Samperi <djsamperi at gmail.com>wrote:
>
>
> On Sun, Jan 9, 2011 at 8:02 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>
>> On Sun, Jan 9, 2011 at 12:45 AM, Dominick Samperi <djsamperi at gmail.com>
>> wrote:
>> > Doug,
>> >
>> > I think the problem is resolved. I wasted a lot of time trying to debug
>> the
>> > new
>> > Module code, and didn't think to try gctorture() with older code that
>> has
>> > worked
>> > for a long time. That code failed too!
>> >
>> > The attached file tortureFix.cpp shows the kind of fix that is needed in
>> > some of the Rcpp code, for example, in Reference.cpp, and perhaps in
>> > other files. The attached script torture.R tests this function
>> with/without
>> > the suggested fix.
>> >
>> > The unsafe code may or may not fail for you. It depends on the
>> > machine/compiler, but because it can fail it is wrong. It fails
>> consistently
>> > under Windows using Visual C++, for example, which is a good reason
>> > why software should be tested using compilers other than GCC, and
>> testing
>> > with gctorture() is obviously a good idea as well.
>>
>> I was filled with hope that it would work to PROTECT all those
>> instances of creating a call, then UNPROTECTing after evaluation. I
>> think I got them all but my modified version still fails at the same
>> point.
>>
>> > library(Uniform)
>> Loading required package: Rcpp
>> > gctorture(TRUE)
>> > Rcpp:::.getModulePointer(unif_module)
>> Before the evaluation in Evaluator::run, call is
>> rcpp_tryCatch(evalq(new("C++Field"), <environment>))
>> After the evaluation in Evaluator::run
>> Error in list(c(76904L, 80L),
>> "/home/bates/R/x86_64-pc-linux-gnu-library/2.12/Rcpp/R/Rcpp.rdb",
>> TRUE, function (n) :
>> unused argument(s)
>> ("/home/bates/R/x86_64-pc-linux-gnu-library/2.12/Rcpp/R/Rcpp.rdb",
>> TRUE, function (n)
>>
>
> This is the same error message you get if you do not apply the
> PROTECTION fixes, so the remaining problem may be of the
> same nature, but perhaps not as transparent...
>
Actually, the problem seems to be pretty transparent, and the
solution is the same (see below). This gets you through Evaluator,
but you fail in SlotProxy in the call to "$<-". The fact that these
problems only appear when gctorture() is turned on suggests
that they all of the same explanation: unprotected SEXP's.
same:
//#define UNSAFE_CODE
#ifdef UNSAFE_CODE
int error = LOGICAL( Rf_eval( Rf_lang1( Rf_install("errorOccured") ),
RCPP ) )[0];
#else
SEXP cmd = PROTECT(Rf_lang1( Rf_install("errorOccured")));
int error = LOGICAL(Rf_eval( cmd, RCPP ))[0];
UNPROTECT(1);
#endif
Rprintf("Got error = %d\n", error);
>
>
>>
>> This is in Evaluator::run where after the evaluation when the error
>> code is being retrieved.
>>
>> I agree that any SEXP that gets created should be PROTECTed and
>> appreciate your efforts in helping to debug this.
>>
>> I am trying a slightly different approach for the installation of the
>> error indicator and codes from rcpp_tryCatch.
>>
>> > On Sat, Jan 8, 2011 at 2:06 PM, Douglas Bates <bates at stat.wisc.edu>
>> wrote:
>> >>
>> >> On Sat, Jan 8, 2011 at 12:57 PM, Dominick Samperi <djsamperi at gmail.com
>> >
>> >> wrote:
>> >> >
>> >> >
>> >> > On Sat, Jan 8, 2011 at 9:48 AM, Douglas Bates <bates at stat.wisc.edu>
>> >> > wrote:
>> >> >>
>> >> >> On Sat, Jan 8, 2011 at 7:49 AM, Douglas Bates <bates at stat.wisc.edu>
>> >> >> wrote:
>> >> >> > On Sat, Jan 8, 2011 at 12:45 AM, Dominick Samperi
>> >> >> > <djsamperi at gmail.com>
>> >> >> > wrote:
>> >> >> >> I checked things under Linux and Windows (using GCC and VC++
>> DLL's)
>> >> >> >> and
>> >> >> >> the
>> >> >> >> same problem occurs at the same place, which is a good sign when
>> it
>> >> >> >> comes to
>> >> >> >> memory issues. Basically, the Rcpp::Reference(std::string)
>> >> >> >> constructor
>> >> >> >> that
>> >> >> >> is
>> >> >> >> part of S4_field, or S4_CppOverloadedMethods constructors fails,
>> >> >> >> depending
>> >> >> >> on
>> >> >> >> which comes first (whether there are fields or not). This only
>> >> >> >> happens
>> >> >> >> when
>> >> >> >> gctorture() is turned on, so R must be clobbering an unprotected
>> >> >> >> SEXP
>> >> >> >> somewhere...
>> >> >> >
>> >> >> > Thanks, Dominick. I too have been working on tracking this down
>> and
>> >> >> > got to that point. If you follow a little further you will find
>> that
>> >> >> > it is the evaluation of the R function getCurrentErrorMessage in
>> the
>> >> >> > Rcpp::Evaluator::run method where things start to go bad, as far
>> as I
>> >> >> > can see. I hope to be able to isolate this today as I need a
>> working
>> >> >> > version of lme4a by tomorrow.
>> >> >>
>> >> >> My current theory is that Rcpp_cache is being blown away by the
>> >> >> garbage collector. Because we want this to be persistent I think
>> the
>> >> >> simplest thing to do is to assign it to a name like .Rcpp_cache in
>> the
>> >> >> namespace during the .onLoad function. I'll try that.
>> >> >
>> >> > I tested this theory by cutting out the Evaluator code and it appears
>> >> > that
>> >> > the cache is not harmed by gctorture(). I suspect that the problem
>> may
>> >> > be earlier in the chain of construction, either a SEXP that was not
>> >> > preserved,
>> >> > or an aliasing problem. It appears that init_Rcpp_cache() is called
>> >> > twice
>> >> > at start-up, which does not seem right, but suppressing the second
>> >> > attempt
>> >> > does not fix the problem.
>> >>
>> >> The second call was eliminated in the SVN archive a couple of days
>> >> ago. I have gotten around some of the problems with the garbage
>> >> collection by redefining how the Rcpp_cache variable is created (not
>> >> yet checked in) so it progresses further. I still think that there is
>> >> a point where a promise in not being evaluated before being used but
>> >> haven't found it yet.
>> >>
>> >> With lazyLoad enabled the first time that you find a variable in the
>> >> namespace it will have a PROMSXP typeof field and you need to invoke
>> >> Rf_eval on it to actually get the value. There are several variables
>> >> from the Rcpp namespace being used here, which is why I am suspicious
>> >> that an unevaluated promise is somehow getting used.
>> >>
>> >> >>
>> >> >> >> On Fri, Jan 7, 2011 at 9:47 AM, Romain Francois
>> >> >> >> <romain at r-enthusiasts.com>
>> >> >> >> wrote:
>> >> >> >>>
>> >> >> >>> Hmm. I commited 2845 and 2846 today.
>> >> >> >>>
>> >> >> >>> Anyway, if you see it also with 0.9.0 this means more detective
>> >> >> >>> work.
>> >> >> >>>
>> >> >> >>> Le 07/01/11 15:05, Douglas Bates a écrit :
>> >> >> >>>>
>> >> >> >>>> On Fri, Jan 7, 2011 at 5:34 AM, Romain Francois
>> >> >> >>>> <romain at r-enthusiasts.com> wrote:
>> >> >> >>>>>
>> >> >> >>>>> Le 05/01/11 18:52, Douglas Bates a écrit :
>> >> >> >>>>>>
>> >> >> >>>>>> I don't know whether this is through error on my part or
>> because
>> >> >> >>>>>> of
>> >> >> >>>>>> an
>> >> >> >>>>>> "infelicity" in the Rcpp module code but the lme4a package,
>> >> >> >>>>>> which
>> >> >> >>>>>> now
>> >> >> >>>>>> uses Rcpp modules extensively, ends up with some
>> >> >> >>>>>> difficult-to-trace
>> >> >> >>>>>> memory corruption issues. Yesterday i finally bit the bullet
>> >> >> >>>>>> and
>> >> >> >>>>>> ran
>> >> >> >>>>>> a test with gctorture(TRUE) and valgrind enabled. It takes a
>> >> >> >>>>>> very
>> >> >> >>>>>> long time and results in a segfault when trying to load the
>> >> >> >>>>>> package.
>> >> >> >>>>>> I enclose the transcript. I should say that this is using
>> >> >> >>>>>> Rcpp_0.9.0
>> >> >> >>>>>> from CRAN, not the SVN version of Rcpp.
>> >> >> >>>>>>
>> >> >> >>>>>> I just got these results this morning (it was running
>> overnight)
>> >> >> >>>>>> and
>> >> >> >>>>>> haven't looked at the code in Module.cpp and cache.cpp yet.
>> If
>> >> >> >>>>>> it
>> >> >> >>>>>> seems likely that the code is beyond me I can try to work out
>> a
>> >> >> >>>>>> simpler example that triggers the problem.
>> >> >> >>>>>
>> >> >> >>>>> Hi Doug,
>> >> >> >>>>>
>> >> >> >>>>> Sorry for the delay, I'm not fully operational yet.
>> >> >> >>>>>
>> >> >> >>>>> All this might be related to some code I put in during
>> holidays
>> >> >> >>>>> and
>> >> >> >>>>> did
>> >> >> >>>>> not
>> >> >> >>>>> have a chance to fully test.
>> >> >> >>>>>
>> >> >> >>>>> Can you try with rev 2845 and let me know if you still see the
>> >> >> >>>>> problem.
>> >> >> >>>>>
>> >> >> >>>>> Romain
>> >> >> >>>>
>> >> >> >>>> Regrettably the problem persists with rev 2845 (which was from
>> >> >> >>>> 2011-01-04, is that the one you meant?) but it is also present
>> >> >> >>>> when
>> >> >> >>>> using Rcpp_0.9.0
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>> Romain Francois
>> >> >> >>> Professional R Enthusiast
>> >> >> >>> +33(0) 6 28 91 30 30
>> >> >> >>> http://romainfrancois.blog.free.fr
>> >> >> >>> |- http://bit.ly/fT2rZM : highlight 0.2-5
>> >> >> >>> |- http://bit.ly/gpCSpH : Evolution of Rcpp code size
>> >> >> >>> `- http://bit.ly/hovakS : RcppGSL initial release
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> _______________________________________________
>> >> >> >>> Rcpp-devel mailing list
>> >> >> >>> Rcpp-devel at lists.r-forge.r-project.org
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20110109/e73bd4dc/attachment.htm>
More information about the Rcpp-devel
mailing list