[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption

Sat Jan 8 20:19:58 CET 2011

On Sat, Jan 8, 2011 at 2:06 PM, Douglas Bates <bates at stat.wisc.edu> wrote:

> On Sat, Jan 8, 2011 at 12:57 PM, Dominick Samperi <djsamperi at gmail.com>
> wrote:
> >
> >
> > On Sat, Jan 8, 2011 at 9:48 AM, Douglas Bates <bates at stat.wisc.edu>
> wrote:
> >>
> >> On Sat, Jan 8, 2011 at 7:49 AM, Douglas Bates <bates at stat.wisc.edu>
> wrote:
> >> > On Sat, Jan 8, 2011 at 12:45 AM, Dominick Samperi <
> djsamperi at gmail.com>
> >> > wrote:
> >> >> I checked things under Linux and Windows (using GCC and VC++ DLL's)
> and
> >> >> the
> >> >> same problem occurs at the same place, which is a good sign when it
> >> >> comes to
> >> >> memory issues. Basically, the Rcpp::Reference(std::string)
> constructor
> >> >> that
> >> >> is
> >> >> part of S4_field, or S4_CppOverloadedMethods constructors fails,
> >> >> depending
> >> >> on
> >> >> which comes first (whether there are fields or not). This only
> happens
> >> >> when
> >> >> gctorture() is turned on, so R must be clobbering an unprotected SEXP
> >> >> somewhere...
> >> >
> >> > Thanks, Dominick. I too have been working on tracking this down and
> >> > got to that point.  If you follow a little further you will find that
> >> > it is the evaluation of the R function getCurrentErrorMessage in the
> >> > Rcpp::Evaluator::run method where things start to go bad, as far as I
> >> > can see.  I hope to be able to isolate this today as I need a working
> >> > version of lme4a by tomorrow.
> >>
> >> My current theory is that Rcpp_cache is being blown away by the
> >> garbage collector.  Because we want this to be persistent I think the
> >> simplest thing to do is to assign it to a name like .Rcpp_cache in the
> >> namespace during the .onLoad function.  I'll try that.
> >
> > I tested this theory by cutting out the Evaluator code and it appears
> that
> > the cache is not harmed by gctorture(). I suspect that the problem may
> > be earlier in the chain of construction, either a SEXP that was not
> > preserved,
> > or an aliasing problem. It appears that init_Rcpp_cache() is called twice
> > at start-up, which does not seem right, but suppressing the second
> attempt
> > does not fix the problem.
>
> The second call was eliminated in the SVN archive a couple of days
> ago.  I have gotten around some of the problems with the garbage
> collection by redefining how the Rcpp_cache variable is created (not
> yet checked in) so it progresses further.  I still think that there is
> a point where a promise in not being evaluated before being used but
> haven't found it yet.
>

Gabor brought up the topic of Promises a few days ago and it
seems support for this is still in flux, and the current behavior is
to always to evaluate when a promise is detected, but perhaps some
are missed, as you suggest.

> With lazyLoad enabled the first time that you find a variable in the
> namespace it will have a PROMSXP typeof field and you need to invoke
> Rf_eval on it to actually get the value.  There are several variables
> from the Rcpp namespace being used here, which is why I am suspicious
> that an unevaluated promise is somehow getting used.
>

I gather from this that you have resolved the gctorture()/no-gctoture()
issues with your garbage collection fix? Otherwise, I don't see how
unevaluated Promises would lead to problems that change with
the gctorture() status.

>
> >>
> >> >> On Fri, Jan 7, 2011 at 9:47 AM, Romain Francois
> >> >> <romain at r-enthusiasts.com>
> >> >> wrote:
> >> >>>
> >> >>> Hmm. I commited 2845 and 2846 today.
> >> >>>
> >> >>> Anyway, if you see it also with 0.9.0 this means more detective
> work.
> >> >>>
> >> >>> Le 07/01/11 15:05, Douglas Bates a écrit :
> >> >>>>
> >> >>>> On Fri, Jan 7, 2011 at 5:34 AM, Romain Francois
> >> >>>> <romain at r-enthusiasts.com>  wrote:
> >> >>>>>
> >> >>>>> Le 05/01/11 18:52, Douglas Bates a écrit :
> >> >>>>>>
> >> >>>>>> I don't know whether this is through error on my part or because
> of
> >> >>>>>> an
> >> >>>>>> "infelicity" in the Rcpp module code but the lme4a package, which
> >> >>>>>> now
> >> >>>>>> uses Rcpp modules extensively, ends up with some
> difficult-to-trace
> >> >>>>>> memory corruption issues.  Yesterday i finally bit the bullet and
> >> >>>>>> ran
> >> >>>>>> a test with gctorture(TRUE) and valgrind enabled.  It takes a
> very
> >> >>>>>> long time and results in a segfault when trying to load the
> >> >>>>>> package.
> >> >>>>>> I enclose the transcript.  I should say that this is using
> >> >>>>>> Rcpp_0.9.0
> >> >>>>>> from CRAN, not the SVN version of Rcpp.
> >> >>>>>>
> >> >>>>>> I just got these results this morning (it was running overnight)
> >> >>>>>> and
> >> >>>>>> haven't looked at the code in Module.cpp and cache.cpp yet.  If
> it
> >> >>>>>> seems likely that the code is beyond me I can try to work out a
> >> >>>>>> simpler example that triggers the problem.
> >> >>>>>
> >> >>>>> Hi Doug,
> >> >>>>>
> >> >>>>> Sorry for the delay, I'm not fully operational yet.
> >> >>>>>
> >> >>>>> All this might be related to some code I put in during holidays
> and
> >> >>>>> did
> >> >>>>> not
> >> >>>>> have a chance to fully test.
> >> >>>>>
> >> >>>>> Can you try with rev 2845 and let me know if you still see the
> >> >>>>> problem.
> >> >>>>>
> >> >>>>> Romain
> >> >>>>
> >> >>>> Regrettably the problem persists with rev 2845 (which was from
> >> >>>> 2011-01-04, is that the one you meant?) but it is also present when
> >> >>>> using Rcpp_0.9.0
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Romain Francois
> >> >>> Professional R Enthusiast
> >> >>> +33(0) 6 28 91 30 30
> >> >>> http://romainfrancois.blog.free.fr
> >> >>> |- http://bit.ly/fT2rZM : highlight 0.2-5
> >> >>> |- http://bit.ly/gpCSpH : Evolution of Rcpp code size
> >> >>> `- http://bit.ly/hovakS : RcppGSL initial release
> >> >>>
> >> >>>
> >> >>> _______________________________________________
> >> >>> Rcpp-devel mailing list
> >> >>> Rcpp-devel at lists.r-forge.r-project.org
> >> >>>
> >> >>>
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
> >> >>
> >> >>
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20110108/e20f80f3/attachment.htm>