[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption

[Douglas Bates]

On Sun, Jan 9, 2011 at 12:45 AM, Dominick Samperi <djsamperi at gmail.com> wrote:
> Doug,
>
> I think the problem is resolved. I wasted a lot of time trying to debug the
> new
> Module code, and didn't think to try gctorture() with older code that has
> worked
> for a long time. That code failed too!
>
> The attached file tortureFix.cpp shows the kind of fix that is needed in
> some of the Rcpp code, for example, in Reference.cpp, and perhaps in
> other files. The attached script torture.R tests this function with/without
> the suggested fix.
>
> The unsafe code may or may not fail for you. It depends on the
> machine/compiler, but because it can fail it is wrong. It fails consistently
> under Windows using Visual C++, for example, which is a good reason
> why software should be tested using compilers other than GCC, and testing
> with gctorture() is obviously a good idea as well.

I was filled with hope that it would work to PROTECT all those
instances of creating a call, then UNPROTECTing after evaluation.  I
think I got them all but my modified version still fails at the same
point.

> library(Uniform)
Loading required package: Rcpp
> gctorture(TRUE)
> Rcpp:::.getModulePointer(unif_module)
Before the evaluation in Evaluator::run, call is
rcpp_tryCatch(evalq(new("C++Field"), <environment>))
After the evaluation in Evaluator::run
Error in list(c(76904L, 80L),
"/home/bates/R/x86_64-pc-linux-gnu-library/2.12/Rcpp/R/Rcpp.rdb",
TRUE, function (n)  :
  unused argument(s)
("/home/bates/R/x86_64-pc-linux-gnu-library/2.12/Rcpp/R/Rcpp.rdb",
TRUE, function (n)

This is in Evaluator::run where after the evaluation when the error
code is being retrieved.

I agree that any SEXP that gets created should be PROTECTed and
appreciate your efforts in helping to debug this.

I am trying a slightly different approach for the installation of the
error indicator and codes from rcpp_tryCatch.

> On Sat, Jan 8, 2011 at 2:06 PM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>
>> On Sat, Jan 8, 2011 at 12:57 PM, Dominick Samperi <djsamperi at gmail.com>
>> wrote:
>> >
>> >
>> > On Sat, Jan 8, 2011 at 9:48 AM, Douglas Bates <bates at stat.wisc.edu>
>> > wrote:
>> >>
>> >> On Sat, Jan 8, 2011 at 7:49 AM, Douglas Bates <bates at stat.wisc.edu>
>> >> wrote:
>> >> > On Sat, Jan 8, 2011 at 12:45 AM, Dominick Samperi
>> >> > <djsamperi at gmail.com>
>> >> > wrote:
>> >> >> I checked things under Linux and Windows (using GCC and VC++ DLL's)
>> >> >> and
>> >> >> the
>> >> >> same problem occurs at the same place, which is a good sign when it
>> >> >> comes to
>> >> >> memory issues. Basically, the Rcpp::Reference(std::string)
>> >> >> constructor
>> >> >> that
>> >> >> is
>> >> >> part of S4_field, or S4_CppOverloadedMethods constructors fails,
>> >> >> depending
>> >> >> on
>> >> >> which comes first (whether there are fields or not). This only
>> >> >> happens
>> >> >> when
>> >> >> gctorture() is turned on, so R must be clobbering an unprotected
>> >> >> SEXP
>> >> >> somewhere...
>> >> >
>> >> > Thanks, Dominick. I too have been working on tracking this down and
>> >> > got to that point.  If you follow a little further you will find that
>> >> > it is the evaluation of the R function getCurrentErrorMessage in the
>> >> > Rcpp::Evaluator::run method where things start to go bad, as far as I
>> >> > can see.  I hope to be able to isolate this today as I need a working
>> >> > version of lme4a by tomorrow.
>> >>
>> >> My current theory is that Rcpp_cache is being blown away by the
>> >> garbage collector.  Because we want this to be persistent I think the
>> >> simplest thing to do is to assign it to a name like .Rcpp_cache in the
>> >> namespace during the .onLoad function.  I'll try that.
>> >
>> > I tested this theory by cutting out the Evaluator code and it appears
>> > that
>> > the cache is not harmed by gctorture(). I suspect that the problem may
>> > be earlier in the chain of construction, either a SEXP that was not
>> > preserved,
>> > or an aliasing problem. It appears that init_Rcpp_cache() is called
>> > twice
>> > at start-up, which does not seem right, but suppressing the second
>> > attempt
>> > does not fix the problem.
>>
>> The second call was eliminated in the SVN archive a couple of days
>> ago.  I have gotten around some of the problems with the garbage
>> collection by redefining how the Rcpp_cache variable is created (not
>> yet checked in) so it progresses further.  I still think that there is
>> a point where a promise in not being evaluated before being used but
>> haven't found it yet.
>>
>> With lazyLoad enabled the first time that you find a variable in the
>> namespace it will have a PROMSXP typeof field and you need to invoke
>> Rf_eval on it to actually get the value.  There are several variables
>> from the Rcpp namespace being used here, which is why I am suspicious
>> that an unevaluated promise is somehow getting used.
>>
>> >>
>> >> >> On Fri, Jan 7, 2011 at 9:47 AM, Romain Francois
>> >> >> <romain at r-enthusiasts.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> Hmm. I commited 2845 and 2846 today.
>> >> >>>
>> >> >>> Anyway, if you see it also with 0.9.0 this means more detective
>> >> >>> work.
>> >> >>>
>> >> >>> Le 07/01/11 15:05, Douglas Bates a écrit :
>> >> >>>>
>> >> >>>> On Fri, Jan 7, 2011 at 5:34 AM, Romain Francois
>> >> >>>> <romain at r-enthusiasts.com>  wrote:
>> >> >>>>>
>> >> >>>>> Le 05/01/11 18:52, Douglas Bates a écrit :
>> >> >>>>>>
>> >> >>>>>> I don't know whether this is through error on my part or because
>> >> >>>>>> of
>> >> >>>>>> an
>> >> >>>>>> "infelicity" in the Rcpp module code but the lme4a package,
>> >> >>>>>> which
>> >> >>>>>> now
>> >> >>>>>> uses Rcpp modules extensively, ends up with some
>> >> >>>>>> difficult-to-trace
>> >> >>>>>> memory corruption issues.  Yesterday i finally bit the bullet
>> >> >>>>>> and
>> >> >>>>>> ran
>> >> >>>>>> a test with gctorture(TRUE) and valgrind enabled.  It takes a
>> >> >>>>>> very
>> >> >>>>>> long time and results in a segfault when trying to load the
>> >> >>>>>> package.
>> >> >>>>>> I enclose the transcript.  I should say that this is using
>> >> >>>>>> Rcpp_0.9.0
>> >> >>>>>> from CRAN, not the SVN version of Rcpp.
>> >> >>>>>>
>> >> >>>>>> I just got these results this morning (it was running overnight)
>> >> >>>>>> and
>> >> >>>>>> haven't looked at the code in Module.cpp and cache.cpp yet.  If
>> >> >>>>>> it
>> >> >>>>>> seems likely that the code is beyond me I can try to work out a
>> >> >>>>>> simpler example that triggers the problem.
>> >> >>>>>
>> >> >>>>> Hi Doug,
>> >> >>>>>
>> >> >>>>> Sorry for the delay, I'm not fully operational yet.
>> >> >>>>>
>> >> >>>>> All this might be related to some code I put in during holidays
>> >> >>>>> and
>> >> >>>>> did
>> >> >>>>> not
>> >> >>>>> have a chance to fully test.
>> >> >>>>>
>> >> >>>>> Can you try with rev 2845 and let me know if you still see the
>> >> >>>>> problem.
>> >> >>>>>
>> >> >>>>> Romain
>> >> >>>>
>> >> >>>> Regrettably the problem persists with rev 2845 (which was from
>> >> >>>> 2011-01-04, is that the one you meant?) but it is also present
>> >> >>>> when
>> >> >>>> using Rcpp_0.9.0
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Romain Francois
>> >> >>> Professional R Enthusiast
>> >> >>> +33(0) 6 28 91 30 30
>> >> >>> http://romainfrancois.blog.free.fr
>> >> >>> |- http://bit.ly/fT2rZM : highlight 0.2-5
>> >> >>> |- http://bit.ly/gpCSpH : Evolution of Rcpp code size
>> >> >>> `- http://bit.ly/hovakS : RcppGSL initial release
>> >> >>>
>> >> >>>
>> >> >>> _______________________________________________
>> >> >>> Rcpp-devel mailing list
>> >> >>> Rcpp-devel at lists.r-forge.r-project.org
>> >> >>>
>> >> >>>
>> >> >>> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
>> >> >>
>> >> >>
>> >> >
>> >
>> >
>
>