[Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption

[Douglas Bates]

On Sat, Jan 8, 2011 at 12:57 PM, Dominick Samperi <djsamperi at gmail.com> wrote:
>
>
> On Sat, Jan 8, 2011 at 9:48 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>>
>> On Sat, Jan 8, 2011 at 7:49 AM, Douglas Bates <bates at stat.wisc.edu> wrote:
>> > On Sat, Jan 8, 2011 at 12:45 AM, Dominick Samperi <djsamperi at gmail.com>
>> > wrote:
>> >> I checked things under Linux and Windows (using GCC and VC++ DLL's) and
>> >> the
>> >> same problem occurs at the same place, which is a good sign when it
>> >> comes to
>> >> memory issues. Basically, the Rcpp::Reference(std::string) constructor
>> >> that
>> >> is
>> >> part of S4_field, or S4_CppOverloadedMethods constructors fails,
>> >> depending
>> >> on
>> >> which comes first (whether there are fields or not). This only happens
>> >> when
>> >> gctorture() is turned on, so R must be clobbering an unprotected SEXP
>> >> somewhere...
>> >
>> > Thanks, Dominick. I too have been working on tracking this down and
>> > got to that point.  If you follow a little further you will find that
>> > it is the evaluation of the R function getCurrentErrorMessage in the
>> > Rcpp::Evaluator::run method where things start to go bad, as far as I
>> > can see.  I hope to be able to isolate this today as I need a working
>> > version of lme4a by tomorrow.
>>
>> My current theory is that Rcpp_cache is being blown away by the
>> garbage collector.  Because we want this to be persistent I think the
>> simplest thing to do is to assign it to a name like .Rcpp_cache in the
>> namespace during the .onLoad function.  I'll try that.
>
> I tested this theory by cutting out the Evaluator code and it appears that
> the cache is not harmed by gctorture(). I suspect that the problem may
> be earlier in the chain of construction, either a SEXP that was not
> preserved,
> or an aliasing problem. It appears that init_Rcpp_cache() is called twice
> at start-up, which does not seem right, but suppressing the second attempt
> does not fix the problem.

The second call was eliminated in the SVN archive a couple of days
ago.  I have gotten around some of the problems with the garbage
collection by redefining how the Rcpp_cache variable is created (not
yet checked in) so it progresses further.  I still think that there is
a point where a promise in not being evaluated before being used but
haven't found it yet.

With lazyLoad enabled the first time that you find a variable in the
namespace it will have a PROMSXP typeof field and you need to invoke
Rf_eval on it to actually get the value.  There are several variables
from the Rcpp namespace being used here, which is why I am suspicious
that an unevaluated promise is somehow getting used.

>>
>> >> On Fri, Jan 7, 2011 at 9:47 AM, Romain Francois
>> >> <romain at r-enthusiasts.com>
>> >> wrote:
>> >>>
>> >>> Hmm. I commited 2845 and 2846 today.
>> >>>
>> >>> Anyway, if you see it also with 0.9.0 this means more detective work.
>> >>>
>> >>> Le 07/01/11 15:05, Douglas Bates a écrit :
>> >>>>
>> >>>> On Fri, Jan 7, 2011 at 5:34 AM, Romain Francois
>> >>>> <romain at r-enthusiasts.com>  wrote:
>> >>>>>
>> >>>>> Le 05/01/11 18:52, Douglas Bates a écrit :
>> >>>>>>
>> >>>>>> I don't know whether this is through error on my part or because of
>> >>>>>> an
>> >>>>>> "infelicity" in the Rcpp module code but the lme4a package, which
>> >>>>>> now
>> >>>>>> uses Rcpp modules extensively, ends up with some difficult-to-trace
>> >>>>>> memory corruption issues.  Yesterday i finally bit the bullet and
>> >>>>>> ran
>> >>>>>> a test with gctorture(TRUE) and valgrind enabled.  It takes a very
>> >>>>>> long time and results in a segfault when trying to load the
>> >>>>>> package.
>> >>>>>> I enclose the transcript.  I should say that this is using
>> >>>>>> Rcpp_0.9.0
>> >>>>>> from CRAN, not the SVN version of Rcpp.
>> >>>>>>
>> >>>>>> I just got these results this morning (it was running overnight)
>> >>>>>> and
>> >>>>>> haven't looked at the code in Module.cpp and cache.cpp yet.  If it
>> >>>>>> seems likely that the code is beyond me I can try to work out a
>> >>>>>> simpler example that triggers the problem.
>> >>>>>
>> >>>>> Hi Doug,
>> >>>>>
>> >>>>> Sorry for the delay, I'm not fully operational yet.
>> >>>>>
>> >>>>> All this might be related to some code I put in during holidays and
>> >>>>> did
>> >>>>> not
>> >>>>> have a chance to fully test.
>> >>>>>
>> >>>>> Can you try with rev 2845 and let me know if you still see the
>> >>>>> problem.
>> >>>>>
>> >>>>> Romain
>> >>>>
>> >>>> Regrettably the problem persists with rev 2845 (which was from
>> >>>> 2011-01-04, is that the one you meant?) but it is also present when
>> >>>> using Rcpp_0.9.0
>> >>>
>> >>>
>> >>> --
>> >>> Romain Francois
>> >>> Professional R Enthusiast
>> >>> +33(0) 6 28 91 30 30
>> >>> http://romainfrancois.blog.free.fr
>> >>> |- http://bit.ly/fT2rZM : highlight 0.2-5
>> >>> |- http://bit.ly/gpCSpH : Evolution of Rcpp code size
>> >>> `- http://bit.ly/hovakS : RcppGSL initial release
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Rcpp-devel mailing list
>> >>> Rcpp-devel at lists.r-forge.r-project.org
>> >>>
>> >>> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
>> >>
>> >>
>> >
>
>