<div dir="ltr">Replacing the PROTECT and UNPROTECT_PTR with R_PreserveObject and R_ReleaseObject will probably take care of the unbalanced-protection-stack warnings, but I don't know if there would be any bad interactions with anything else in Rcpp.</div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature">Bill Dunlap<br>TIBCO Software<br>wdunlap <a href="http://tibco.com" target="_blank">tibco.com</a></div></div>
<br><div class="gmail_quote">On Thu, Oct 6, 2016 at 8:31 AM, Anton Bossenbroek <span dir="ltr"><<a href="mailto:anton.bossenbroek@me.com" target="_blank">anton.bossenbroek@me.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Sorry, as a result of some iterations on my test and code an error slipped in. The R code should be,<br>
<br>
require(Rcpp)<br>
sourceCpp(file="~/tmp/example.<wbr>cpp")<br>
<br>
add_children <- function(number_of_children = 2) {<br>
p <- 0<br>
st <- initialize_storage()<br>
for (i in 1 : number_of_children) {<br>
st <- add_element(st, i)<br>
}<br>
return(st)<br>
}<br>
n <- 10000<br>
a <- add_children(number_of_<wbr>children = n)<br>
<br>
res <- sapply(get_nodes(a), function(x) x[["key"]])<br>
all(res == 0 : n)<br>
<br>
Since I am aware that the garbage may be trying to free up memory I added PROTECT(key_); in the body of the constructor of Element and UNPROTECT(result.size()); just before the return in get_nodes(). However, when I do that I get the following warning:<br>
<br>
Warning: stack imbalance in '.Call', 19 then -9981<br>
Warning: stack imbalance in '<-', 2 then -9998<br>
etc.<br>
<br>
<br>
<br>
> On 6 Oct 2016, at 10:52, Anton Bossenbroek <<a href="mailto:anton.bossenbroek@me.com">anton.bossenbroek@me.com</a>> wrote:<br>
><br>
> Hi Everyone,<br>
><br>
> I want to add a large number of objects in C++ that are managed by `shared_ptr` in a `vector`. However, when I push the limits of the amount that I want to allocate the data in R becomes inconsistent.<br>
><br>
> I will first show the test script and then the c++ file that cause the error. The expected results are shown at the bottom.<br>
><br>
> # test script<br>
> The test script permits to add an arbitrary number of objects to a vector.<br>
><br>
> require(Rcpp)<br>
> sourceCpp(file="~/tmp/example.<wbr>cpp")<br>
><br>
> add_children <- function(number_of_children = 2) {<br>
> p <- 0<br>
> st <- initialize_storage(p, p)<br>
> for (i in 1 : number_of_children) {<br>
> st <- add_node(st, i, i)<br>
> }<br>
> return(st)<br>
> }<br>
><br>
> # example.cpp<br>
> The vector is stored in a object that is managed by R but the elements in the vector are managed by `shared_ptr` and created with `make_shared`.<br>
><br>
> // [[Rcpp::plugins(cpp11)]]<br>
> #include <RcppCommon.h><br>
><br>
> #include <memory><br>
> #include <vector><br>
><br>
> using namespace std;<br>
><br>
> struct Element : public std::enable_shared_from_this< Element > {<br>
> SEXP key_;<br>
><br>
> /* Simple constructor that assigns the key. */<br>
> Element(SEXP key) : key_(key) {}<br>
><br>
> /* Convert the object to a R object. */<br>
> operator SEXP() const;<br>
> };<br>
><br>
> typedef shared_ptr<Element> element_sp;<br>
> typedef vector<element_sp> element_sp_vec;<br>
> typedef shared_ptr<element_sp_vec> element_sp_vec_sp;<br>
><br>
> struct Storage {<br>
> /* Internal storage of nodes. */<br>
> element_sp_vec nodes_;<br>
><br>
> /* Empty constructor. */<br>
> Storage() {}<br>
><br>
> /* Add a node to the storage with its key set to key. */<br>
> void add_element(SEXP key) {<br>
> /* Since Element objects are managed by shared_ptr we create a new class<br>
> * with make_shared. */<br>
> element_sp e = make_shared<Element>(key);<br>
> /* Add the node to the internal storage. */<br>
> nodes_.push_back(e);<br>
> }<br>
><br>
> element_sp_vec_sp get_nodes() {<br>
> /* Create a shared pointer that will hold all the results. Although we<br>
> * could do this simpler it mimics the logic I implemented in my real<br>
> * program. There I need to swap elements in the list after the copy of the<br>
> * vector. */<br>
> element_sp_vec_sp res(new element_sp_vec());<br>
> /* Copy the data in the nodes vector to the result vector. */<br>
> *res = nodes_;<br>
> return res;<br>
> }<br>
> };<br>
><br>
> #include <Rcpp.h><br>
><br>
> using namespace Rcpp;<br>
><br>
> /* Convert the Element object to a list with key set its internal member */<br>
> Element::operator SEXP() const<br>
> {<br>
> List serial;<br>
><br>
> serial["key"] = key_;<br>
><br>
> return serial;<br>
> }<br>
><br>
> typedef XPtr<Storage> st_xptr;<br>
><br>
> // [[Rcpp::export]]<br>
> SEXP<br>
> initialize_storage()<br>
> {<br>
> /* Create a new storage managed by R. */<br>
> Storage* st = new Storage();<br>
> st_xptr p(st, true);<br>
><br>
> return p;<br>
> }<br>
><br>
> // [[Rcpp::export]]<br>
> SEXP<br>
> add_element(SEXP st_sexp, SEXP key)<br>
> {<br>
> st_xptr st(st_sexp);<br>
> /* Add a new element to the internal storage. */<br>
> st->add_element(key);<br>
><br>
> return st;<br>
> }<br>
><br>
> // [[Rcpp::export]]<br>
> List<br>
> get_nodes(SEXP st_sexp)<br>
> {<br>
> st_xptr st(st_sexp);<br>
> /* Retrieve the elements in the internal storage. */<br>
> element_sp_vec_sp c_res = st->get_nodes();<br>
><br>
> /* Allocate a List to store all our results. */<br>
> List result(c_res->size());<br>
> int i = 0;<br>
> /* Iterate through the results and store the result in our list. */<br>
> for (auto it : *c_res) {<br>
> result[i] = wrap(*it);<br>
> ++i;<br>
> }<br>
><br>
> return result;<br>
> }<br>
><br>
> Below follow a few test cases of the script with the behavior that I experience on Mac OS Sierra with clang.<br>
><br>
> ## n = 10<br>
><br>
> Everything works fine<br>
><br>
> n <- 10<br>
> a <- add_children(number_of_<wbr>children = n)<br>
> res <- sapply(get_nodes(a), function(x) x[["key"]])<br>
> all(res == 0 : n)<br>
> # [1] TRUE<br>
><br>
> ## n = 100<br>
><br>
> Everything works fine<br>
><br>
> n <- 100<br>
> a <- add_children(number_of_<wbr>children = n)<br>
> res <- sapply(get_nodes(a), function(x) x[["key"]])<br>
> all(res == 0 : n)<br>
> # [1] TRUE<br>
><br>
> ## n = 10000<br>
><br>
> Something goes wrong.<br>
><br>
> n <- 10000<br>
> a <- add_children(number_of_<wbr>children = n)<br>
><br>
> res <- sapply(get_nodes(a), function(x) x[["key"]])<br>
> all(res == 0 : n)<br>
> # [1] FALSE<br>
> # There were 50 or more warnings (use warnings() to see the first 50)<br>
><br>
> Some further research shows that the warnings are:<br>
><br>
> warnings()<br>
> # Warning messages:<br>
> # 1: NAs introduced by coercion<br>
> # 2: NAs introduced by coercion<br>
> # 3: NAs introduced by coercion<br>
> # 4: NAs introduced by coercion<br>
> ### etc.<br>
><br>
> a closer inspection into the content of `res` shows that it has a non numeric value,<br>
><br>
> res[1000]<br>
> # [[1]]<br>
> # [1] "data"<br>
><br>
> which is surprising to me since the script only added numeric `SEXP` values to the `vector`. My expected output for this value of `n` would be the same as the cases above.<br>
><br>
> ## gctorture<br>
> I reran the `n=10000` example with `gctorture(TRUE)` but did not receive any warning but the data is corrupt. Two random elements in the `res` list:<br>
><br>
> # [[998]]<br>
> # <CHARSXP: "\"key\""><br>
> #<br>
> # [[999]]<br>
> # [1] "srcref"<br>
><br>
> = Replication<br>
> I replicated these results on Mac OS X Sierra as well as Docker image based on rocker.<br>
><br>
> sessionInfo()<br>
> # R version 3.3.1 (2016-06-21)<br>
> # Platform: x86_64-apple-darwin15.5.0 (64-bit)<br>
> # Running under: OS X 10.12 (Sierra)<br>
> #<br>
> # locale:<br>
> # [1] en_US.UTF-8/en_US.UTF-8/en_US.<wbr>UTF-8/C/en_US.UTF-8/en_US.UTF-<wbr>8<br>
> #<br>
> # attached base packages:<br>
> # [1] stats graphics grDevices utils datasets methods base<br>
> #<br>
> # other attached packages:<br>
> # [1] Rcpp_0.12.7 setwidth_1.0-4 colorout_1.1-2<br>
> #<br>
> # loaded via a namespace (and not attached):<br>
> # [1] tools_3.3.1<br>
><br>
> ### uname<br>
><br>
> uname -prsv<br>
> Darwin 16.0.0 Darwin Kernel Version 16.0.0: Mon Aug 29 17:56:20 PDT 2016; root:xnu-3789.1.32~3/RELEASE_<wbr>X86_64 i386<br>
><br>
> ### clang<br>
><br>
> clang -v<br>
> Apple LLVM version 8.0.0 (clang-800.0.38)<br>
> Target: x86_64-apple-darwin16.0.0<br>
> Thread model: posix<br>
> InstalledDir: /Applications/Xcode.app/<wbr>Contents/Developer/Toolchains/<wbr>XcodeDefault.xctoolchain/usr/<wbr>bin<br>
><br>
> Any advise on what may be the problem here?<br>
> ______________________________<wbr>_________________<br>
> Rcpp-devel mailing list<br>
> <a href="mailto:Rcpp-devel@lists.r-forge.r-project.org">Rcpp-devel@lists.r-forge.r-<wbr>project.org</a><br>
> <a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel" rel="noreferrer" target="_blank">https://lists.r-forge.r-<wbr>project.org/cgi-bin/mailman/<wbr>listinfo/rcpp-devel</a><br>
<br>
______________________________<wbr>_________________<br>
Rcpp-devel mailing list<br>
<a href="mailto:Rcpp-devel@lists.r-forge.r-project.org">Rcpp-devel@lists.r-forge.r-<wbr>project.org</a><br>
<a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel" rel="noreferrer" target="_blank">https://lists.r-forge.r-<wbr>project.org/cgi-bin/mailman/<wbr>listinfo/rcpp-devel</a><br>
</blockquote></div><br></div>