[Rcpp-devel] Memory Leak found and resolved

[Silkworth,David J.]

About a week ago I mentioned a problem I was having maintaining a package, which seemed to require inline compiling for use.  I never wanted to use cxxfunplus, and never explored this.

It turns out I created a mysterious memory leak, that just happened to be resolved when code was prepared inline.

Some may also recall from over a month ago, I had ripped Douglas Bates' splines.c code from the R base package splines to utilize in a project.  Rcpp code was written to replace the R code of ns() in the splines package, and a direct call was made from Rcpp to spline_basis() in splines.c.

This is what I think happened:  It is likely that the memory for objects allocated in spline_basis is not recovered in that code, because it was intended to be called from R functions that would rely on the R garbage collector anyway.  When I called this code directly within an Rcpp function, that was in-turn called in a (Gibbs sampler) loop of another Rcpp function there was never an opportunity to recover memory allocated in the C code.  This created a memory leak that ultimately hit the R memory limit with this kind of message:

Error: cannot allocate vector of size 768.2 Mb
>

The resolution was quite simple.  I ended up calling an R wrapper function for the replacement ns function that had been written in Rcpp.  I lost a measurable amount of performance, but by doing this I believe the R garbage collector was permitted to do its work.

For the wrapper call in Rcpp I used the "environment facility", so my code for establishing this function in Rcpp looks like this:

Environment mypackage ("package:mypackage");
Function spline_= mypackage["ns_cpp"];

Where, ns_cpp is the R wrapper function calling the Rcpp replacement for ns in mypackage.

The reason this would work during inline development is that the ns_cpp was always being brought in as an argument to the cxxfunction.  Failure occurred when I directly called the Rcpp replacement function as the package was developed.