[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption

Sun Jan 9 15:21:15 CET 2011

On Sun, Jan 9, 2011 at 8:02 AM, Douglas Bates <bates at stat.wisc.edu> wrote:

> On Sun, Jan 9, 2011 at 12:45 AM, Dominick Samperi <djsamperi at gmail.com>
> wrote:
> > Doug,
> >
> > I think the problem is resolved. I wasted a lot of time trying to debug
> the
> > new
> > Module code, and didn't think to try gctorture() with older code that has
> > worked
> > for a long time. That code failed too!
> >
> > The attached file tortureFix.cpp shows the kind of fix that is needed in
> > some of the Rcpp code, for example, in Reference.cpp, and perhaps in
> > other files. The attached script torture.R tests this function
> with/without
> > the suggested fix.
> >
> > The unsafe code may or may not fail for you. It depends on the
> > machine/compiler, but because it can fail it is wrong. It fails
> consistently
> > under Windows using Visual C++, for example, which is a good reason
> > why software should be tested using compilers other than GCC, and testing
> > with gctorture() is obviously a good idea as well.
>
> I was filled with hope that it would work to PROTECT all those
> instances of creating a call, then UNPROTECTing after evaluation.  I
> think I got them all but my modified version still fails at the same
> point.
>
> > library(Uniform)
> Loading required package: Rcpp
> > gctorture(TRUE)
> > Rcpp:::.getModulePointer(unif_module)
> Before the evaluation in Evaluator::run, call is
> rcpp_tryCatch(evalq(new("C++Field"), <environment>))
> After the evaluation in Evaluator::run
> Error in list(c(76904L, 80L),
> "/home/bates/R/x86_64-pc-linux-gnu-library/2.12/Rcpp/R/Rcpp.rdb",
> TRUE, function (n)  :
>  unused argument(s)
> ("/home/bates/R/x86_64-pc-linux-gnu-library/2.12/Rcpp/R/Rcpp.rdb",
> TRUE, function (n)
>

This is the same error message you get if you do not apply the
PROTECTION fixes, so the remaining problem may be of the
same nature, but perhaps not as transparent...

>
> This is in Evaluator::run where after the evaluation when the error
> code is being retrieved.
>
> I agree that any SEXP that gets created should be PROTECTed and
> appreciate your efforts in helping to debug this.
>
> I am trying a slightly different approach for the installation of the
> error indicator and codes from rcpp_tryCatch.
>
> > On Sat, Jan 8, 2011 at 2:06 PM, Douglas Bates <bates at stat.wisc.edu>
> wrote:
> >>
> >> On Sat, Jan 8, 2011 at 12:57 PM, Dominick Samperi <djsamperi at gmail.com>
> >> wrote:
> >> >
> >> >
> >> > On Sat, Jan 8, 2011 at 9:48 AM, Douglas Bates <bates at stat.wisc.edu>
> >> > wrote:
> >> >>
> >> >> On Sat, Jan 8, 2011 at 7:49 AM, Douglas Bates <bates at stat.wisc.edu>
> >> >> wrote:
> >> >> > On Sat, Jan 8, 2011 at 12:45 AM, Dominick Samperi
> >> >> > <djsamperi at gmail.com>
> >> >> > wrote:
> >> >> >> I checked things under Linux and Windows (using GCC and VC++
> DLL's)
> >> >> >> and
> >> >> >> the
> >> >> >> same problem occurs at the same place, which is a good sign when
> it
> >> >> >> comes to
> >> >> >> memory issues. Basically, the Rcpp::Reference(std::string)
> >> >> >> constructor
> >> >> >> that
> >> >> >> is
> >> >> >> part of S4_field, or S4_CppOverloadedMethods constructors fails,
> >> >> >> depending
> >> >> >> on
> >> >> >> which comes first (whether there are fields or not). This only
> >> >> >> happens
> >> >> >> when
> >> >> >> gctorture() is turned on, so R must be clobbering an unprotected
> >> >> >> SEXP
> >> >> >> somewhere...
> >> >> >
> >> >> > Thanks, Dominick. I too have been working on tracking this down and
> >> >> > got to that point.  If you follow a little further you will find
> that
> >> >> > it is the evaluation of the R function getCurrentErrorMessage in
> the
> >> >> > Rcpp::Evaluator::run method where things start to go bad, as far as
> I
> >> >> > can see.  I hope to be able to isolate this today as I need a
> working
> >> >> > version of lme4a by tomorrow.
> >> >>
> >> >> My current theory is that Rcpp_cache is being blown away by the
> >> >> garbage collector.  Because we want this to be persistent I think the
> >> >> simplest thing to do is to assign it to a name like .Rcpp_cache in
> the
> >> >> namespace during the .onLoad function.  I'll try that.
> >> >
> >> > I tested this theory by cutting out the Evaluator code and it appears
> >> > that
> >> > the cache is not harmed by gctorture(). I suspect that the problem may
> >> > be earlier in the chain of construction, either a SEXP that was not
> >> > preserved,
> >> > or an aliasing problem. It appears that init_Rcpp_cache() is called
> >> > twice
> >> > at start-up, which does not seem right, but suppressing the second
> >> > attempt
> >> > does not fix the problem.
> >>
> >> The second call was eliminated in the SVN archive a couple of days
> >> ago.  I have gotten around some of the problems with the garbage
> >> collection by redefining how the Rcpp_cache variable is created (not
> >> yet checked in) so it progresses further.  I still think that there is
> >> a point where a promise in not being evaluated before being used but
> >> haven't found it yet.
> >>
> >> With lazyLoad enabled the first time that you find a variable in the
> >> namespace it will have a PROMSXP typeof field and you need to invoke
> >> Rf_eval on it to actually get the value.  There are several variables
> >> from the Rcpp namespace being used here, which is why I am suspicious
> >> that an unevaluated promise is somehow getting used.
> >>
> >> >>
> >> >> >> On Fri, Jan 7, 2011 at 9:47 AM, Romain Francois
> >> >> >> <romain at r-enthusiasts.com>
> >> >> >> wrote:
> >> >> >>>
> >> >> >>> Hmm. I commited 2845 and 2846 today.
> >> >> >>>
> >> >> >>> Anyway, if you see it also with 0.9.0 this means more detective
> >> >> >>> work.
> >> >> >>>
> >> >> >>> Le 07/01/11 15:05, Douglas Bates a écrit :
> >> >> >>>>
> >> >> >>>> On Fri, Jan 7, 2011 at 5:34 AM, Romain Francois
> >> >> >>>> <romain at r-enthusiasts.com>  wrote:
> >> >> >>>>>
> >> >> >>>>> Le 05/01/11 18:52, Douglas Bates a écrit :
> >> >> >>>>>>
> >> >> >>>>>> I don't know whether this is through error on my part or
> because
> >> >> >>>>>> of
> >> >> >>>>>> an
> >> >> >>>>>> "infelicity" in the Rcpp module code but the lme4a package,
> >> >> >>>>>> which
> >> >> >>>>>> now
> >> >> >>>>>> uses Rcpp modules extensively, ends up with some
> >> >> >>>>>> difficult-to-trace
> >> >> >>>>>> memory corruption issues.  Yesterday i finally bit the bullet
> >> >> >>>>>> and
> >> >> >>>>>> ran
> >> >> >>>>>> a test with gctorture(TRUE) and valgrind enabled.  It takes a
> >> >> >>>>>> very
> >> >> >>>>>> long time and results in a segfault when trying to load the
> >> >> >>>>>> package.
> >> >> >>>>>> I enclose the transcript.  I should say that this is using
> >> >> >>>>>> Rcpp_0.9.0
> >> >> >>>>>> from CRAN, not the SVN version of Rcpp.
> >> >> >>>>>>
> >> >> >>>>>> I just got these results this morning (it was running
> overnight)
> >> >> >>>>>> and
> >> >> >>>>>> haven't looked at the code in Module.cpp and cache.cpp yet.
>  If
> >> >> >>>>>> it
> >> >> >>>>>> seems likely that the code is beyond me I can try to work out
> a
> >> >> >>>>>> simpler example that triggers the problem.
> >> >> >>>>>
> >> >> >>>>> Hi Doug,
> >> >> >>>>>
> >> >> >>>>> Sorry for the delay, I'm not fully operational yet.
> >> >> >>>>>
> >> >> >>>>> All this might be related to some code I put in during holidays
> >> >> >>>>> and
> >> >> >>>>> did
> >> >> >>>>> not
> >> >> >>>>> have a chance to fully test.
> >> >> >>>>>
> >> >> >>>>> Can you try with rev 2845 and let me know if you still see the
> >> >> >>>>> problem.
> >> >> >>>>>
> >> >> >>>>> Romain
> >> >> >>>>
> >> >> >>>> Regrettably the problem persists with rev 2845 (which was from
> >> >> >>>> 2011-01-04, is that the one you meant?) but it is also present
> >> >> >>>> when
> >> >> >>>> using Rcpp_0.9.0
> >> >> >>>
> >> >> >>>
> >> >> >>> --
> >> >> >>> Romain Francois
> >> >> >>> Professional R Enthusiast
> >> >> >>> +33(0) 6 28 91 30 30
> >> >> >>> http://romainfrancois.blog.free.fr
> >> >> >>> |- http://bit.ly/fT2rZM : highlight 0.2-5
> >> >> >>> |- http://bit.ly/gpCSpH : Evolution of Rcpp code size
> >> >> >>> `- http://bit.ly/hovakS : RcppGSL initial release
> >> >> >>>
> >> >> >>>
> >> >> >>> _______________________________________________
> >> >> >>> Rcpp-devel mailing list
> >> >> >>> Rcpp-devel at lists.r-forge.r-project.org
> >> >> >>>
> >> >> >>>
> >> >> >>>
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
> >> >> >>
> >> >> >>
> >> >> >
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20110109/5231daa6/attachment-0001.htm>