[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption

Sun Jan 9 07:45:02 CET 2011

Doug,

I think the problem is resolved. I wasted a lot of time trying to debug the
new
Module code, and didn't think to try gctorture() with older code that has
worked
for a long time. That code failed too!

The attached file tortureFix.cpp shows the kind of fix that is needed in
some of the Rcpp code, for example, in Reference.cpp, and perhaps in
other files. The attached script torture.R tests this function with/without
the suggested fix.

The unsafe code may or may not fail for you. It depends on the
machine/compiler, but because it can fail it is wrong. It fails consistently
under Windows using Visual C++, for example, which is a good reason
why software should be tested using compilers other than GCC, and testing
with gctorture() is obviously a good idea as well.

Thanks,
Dominick

On Sat, Jan 8, 2011 at 2:06 PM, Douglas Bates <bates at stat.wisc.edu> wrote:

> On Sat, Jan 8, 2011 at 12:57 PM, Dominick Samperi <djsamperi at gmail.com>
> wrote:
> >
> >
> > On Sat, Jan 8, 2011 at 9:48 AM, Douglas Bates <bates at stat.wisc.edu>
> wrote:
> >>
> >> On Sat, Jan 8, 2011 at 7:49 AM, Douglas Bates <bates at stat.wisc.edu>
> wrote:
> >> > On Sat, Jan 8, 2011 at 12:45 AM, Dominick Samperi <
> djsamperi at gmail.com>
> >> > wrote:
> >> >> I checked things under Linux and Windows (using GCC and VC++ DLL's)
> and
> >> >> the
> >> >> same problem occurs at the same place, which is a good sign when it
> >> >> comes to
> >> >> memory issues. Basically, the Rcpp::Reference(std::string)
> constructor
> >> >> that
> >> >> is
> >> >> part of S4_field, or S4_CppOverloadedMethods constructors fails,
> >> >> depending
> >> >> on
> >> >> which comes first (whether there are fields or not). This only
> happens
> >> >> when
> >> >> gctorture() is turned on, so R must be clobbering an unprotected SEXP
> >> >> somewhere...
> >> >
> >> > Thanks, Dominick. I too have been working on tracking this down and
> >> > got to that point.  If you follow a little further you will find that
> >> > it is the evaluation of the R function getCurrentErrorMessage in the
> >> > Rcpp::Evaluator::run method where things start to go bad, as far as I
> >> > can see.  I hope to be able to isolate this today as I need a working
> >> > version of lme4a by tomorrow.
> >>
> >> My current theory is that Rcpp_cache is being blown away by the
> >> garbage collector.  Because we want this to be persistent I think the
> >> simplest thing to do is to assign it to a name like .Rcpp_cache in the
> >> namespace during the .onLoad function.  I'll try that.
> >
> > I tested this theory by cutting out the Evaluator code and it appears
> that
> > the cache is not harmed by gctorture(). I suspect that the problem may
> > be earlier in the chain of construction, either a SEXP that was not
> > preserved,
> > or an aliasing problem. It appears that init_Rcpp_cache() is called twice
> > at start-up, which does not seem right, but suppressing the second
> attempt
> > does not fix the problem.
>
> The second call was eliminated in the SVN archive a couple of days
> ago.  I have gotten around some of the problems with the garbage
> collection by redefining how the Rcpp_cache variable is created (not
> yet checked in) so it progresses further.  I still think that there is
> a point where a promise in not being evaluated before being used but
> haven't found it yet.
>
> With lazyLoad enabled the first time that you find a variable in the
> namespace it will have a PROMSXP typeof field and you need to invoke
> Rf_eval on it to actually get the value.  There are several variables
> from the Rcpp namespace being used here, which is why I am suspicious
> that an unevaluated promise is somehow getting used.
>
> >>
> >> >> On Fri, Jan 7, 2011 at 9:47 AM, Romain Francois
> >> >> <romain at r-enthusiasts.com>
> >> >> wrote:
> >> >>>
> >> >>> Hmm. I commited 2845 and 2846 today.
> >> >>>
> >> >>> Anyway, if you see it also with 0.9.0 this means more detective
> work.
> >> >>>
> >> >>> Le 07/01/11 15:05, Douglas Bates a écrit :
> >> >>>>
> >> >>>> On Fri, Jan 7, 2011 at 5:34 AM, Romain Francois
> >> >>>> <romain at r-enthusiasts.com>  wrote:
> >> >>>>>
> >> >>>>> Le 05/01/11 18:52, Douglas Bates a écrit :
> >> >>>>>>
> >> >>>>>> I don't know whether this is through error on my part or because
> of
> >> >>>>>> an
> >> >>>>>> "infelicity" in the Rcpp module code but the lme4a package, which
> >> >>>>>> now
> >> >>>>>> uses Rcpp modules extensively, ends up with some
> difficult-to-trace
> >> >>>>>> memory corruption issues.  Yesterday i finally bit the bullet and
> >> >>>>>> ran
> >> >>>>>> a test with gctorture(TRUE) and valgrind enabled.  It takes a
> very
> >> >>>>>> long time and results in a segfault when trying to load the
> >> >>>>>> package.
> >> >>>>>> I enclose the transcript.  I should say that this is using
> >> >>>>>> Rcpp_0.9.0
> >> >>>>>> from CRAN, not the SVN version of Rcpp.
> >> >>>>>>
> >> >>>>>> I just got these results this morning (it was running overnight)
> >> >>>>>> and
> >> >>>>>> haven't looked at the code in Module.cpp and cache.cpp yet.  If
> it
> >> >>>>>> seems likely that the code is beyond me I can try to work out a
> >> >>>>>> simpler example that triggers the problem.
> >> >>>>>
> >> >>>>> Hi Doug,
> >> >>>>>
> >> >>>>> Sorry for the delay, I'm not fully operational yet.
> >> >>>>>
> >> >>>>> All this might be related to some code I put in during holidays
> and
> >> >>>>> did
> >> >>>>> not
> >> >>>>> have a chance to fully test.
> >> >>>>>
> >> >>>>> Can you try with rev 2845 and let me know if you still see the
> >> >>>>> problem.
> >> >>>>>
> >> >>>>> Romain
> >> >>>>
> >> >>>> Regrettably the problem persists with rev 2845 (which was from
> >> >>>> 2011-01-04, is that the one you meant?) but it is also present when
> >> >>>> using Rcpp_0.9.0
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Romain Francois
> >> >>> Professional R Enthusiast
> >> >>> +33(0) 6 28 91 30 30
> >> >>> http://romainfrancois.blog.free.fr
> >> >>> |- http://bit.ly/fT2rZM : highlight 0.2-5
> >> >>> |- http://bit.ly/gpCSpH : Evolution of Rcpp code size
> >> >>> `- http://bit.ly/hovakS : RcppGSL initial release
> >> >>>
> >> >>>
> >> >>> _______________________________________________
> >> >>> Rcpp-devel mailing list
> >> >>> Rcpp-devel at lists.r-forge.r-project.org
> >> >>>
> >> >>>
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
> >> >>
> >> >>
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20110109/7df0e8bb/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tortureFix.cpp
Type: text/x-c++src
Size: 498 bytes
Desc: not available
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20110109/7df0e8bb/attachment-0001.cpp>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: torture.R
Type: application/octet-stream
Size: 198 bytes
Desc: not available
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20110109/7df0e8bb/attachment-0001.obj>