<div dir="ltr">Hi All,<br><br>Just to confirm -- we can trigger this error with arbitrary integer sparse matrices, e.g. (running with ASAN):<br><br><blockquote style="margin:0 0 0 40px;border:none;padding:0px">library(Rcpp)<br>
cppFunction("arma::sp_mat asSpMat(SEXP S) { return as<arma::sp_mat>(S); }", depends="RcppArmadillo")<br>m <- Matrix(sample(0:5, 100, TRUE), 20, 5, sparse = TRUE)<br>asSpMat(m)<br></blockquote>
<br><div>So it looks like it's an Rcpp(Armadillo) bug after all -- we'll try to figure out what's causing it.</div><div><br></div><div>Thanks for the report!</div><div><br></div><div>Cheers,</div><div>Kevin</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Aug 9, 2014 at 6:37 PM, Dirk Eddelbuettel <span dir="ltr"><<a href="mailto:edd@debian.org" target="_blank">edd@debian.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hi Terrance,<br>
<div class=""><br>
On 9 August 2014 at 20:05, terrance savitsky wrote:<br>
| darn. i R CMD check'd --as-cran under R-devel and also loaded the minimal<br>
| package, sparseList_1.1, the example and test run without issue<br>
</div>| [icon_10_ge] sparseList_1.1.tar.gz<br>
<br>
Read the section "4.3 Checking memory access" of Writing R Extensions.<br>
You need a version of R compiled with "-fsanitize=address" as described there.<br>
<br>
My Docker image provides that, hence my links in the first email.<br>
<br>
| . So I'm stumped why the test is failing for you and not me. I attach the<br>
| file, again, just in I included the wrong version. Sorry I wasn't able to get<br>
| you a link that you could wget. I work in the U.S. federal system and am<br>
| limited by security protocols in what I'm able to do. Thanks for being<br>
| patient.<br>
|<br>
| Since I include the data object (which requires both the spam and Matrix<br>
| packages), I can/will remove the dependencies on those packages. Since I have<br>
| an example written into the documentation that runs on check, I will remove the<br>
| test I added (and, therefore, testthat). I will do this in the next day or<br>
| so, assuming what I attached still doesn't work.<br>
<br>
The minimal example we asked to provided is as follows<br>
<br>
library(sparseList) ## just to get the data set<br>
data(C) ## gets a list, you only need the first element<br>
## function to invoke as<spmat>() on supplied argument<br>
cppFunction("arma::sp_mat asSpMat(SEXP S) { return as<arma::sp_mat>(S); }", depends="RcppArmadillo")<br>
## call it: boom (if using -fsanitize=address)<br>
asSpMat(C[[1]])<br>
<br>
The open question now is whether your dgCMatrix is valid (likely) in which<br>
case our code may be wrong (yet someone who actually __used ans understands<br>
sparse matrix storage conventions__ needs to look at it). Our code survived<br>
the tests we threw at it, as well as use by other people using sparse matrix.<br>
<br>
Or you created a faulty dgCMatrix, in which case the bug is yours.<br>
<br>
Your didn't show how you create 'C'. So we can't really tell.<br>
<br>
And this point it would be good if you could do some digging. You're closest<br>
to your issue.<br>
<br>
Dirk<br>
<br>
| Stepping back, the code works in both the minimal package and growfunctions -<br>
| it inputs a list of dgCMatrix objects, assigns the elements to a field<sp_mat><br>
| object and then writes it back out to a List object. The elements of the<br>
| outputted list object at the R layer are, indeed, dgCMatrix objects. So<br>
| 'yes', the script you referenced does create a valid dgCMatrix object and i've<br>
| verified such. <br>
|<br>
| As an aside, the C++ file embeds an R script that Kevin asked for if that is<br>
| useful.<br>
<br>
You need to run it under an executable instrumented for memory debugging to<br>
trigger the bug.<br>
<br>
Dirk<br>
<br>
<br>
|<br>
| terrance.<br>
|<br>
|<br>
| On Sat, Aug 9, 2014 at 7:14 PM, Dirk Eddelbuettel <<a href="mailto:edd@debian.org">edd@debian.org</a>> wrote:<br>
|<br>
|<br>
| Terrance,<br>
|<br>
| On 9 August 2014 at 18:23, terrance savitsky wrote:<br>
| | Hi Dirk, The revised version of this toy package, sparseList, does<br>
| exactly what<br>
| | you request. I've included a data set that is of the same construction<br>
| (of<br>
| | dgCMatrix elements from R::Matrix wrapped into a list object) and have<br>
| added<br>
| | both an example and test that call this data. The package is minimal,<br>
| with a<br>
| | simple .cpp function (that reads in the list of dgCMatrix objects and<br>
| | writes the elements to an arma::field<arma:sp_mat>) to isolate the<br>
| source of<br>
| | the error in growfunctions.<br>
| | <br>
| | I attach the tarball, which you may also access through the following<br>
| URL:<br>
| | <br>
| | <a href="https://drive.google.com/file/d/" target="_blank">https://drive.google.com/file/d/</a><br>
| | 0B0DIDcaf6yMwZDRDZUNLd0lBbmZ3OHlHSzIyeV9hZ3RxZ0xr/edit?usp=sharing<br>
|<br>
| Did not work via wget, required a webpage visit and download.<br>
|<br>
| Package has extraneous dependencies (you do not need testthat to call a<br>
| function).<br>
|<br>
| Moreover, testthat seems to pivot over to /usr/bin/R rather than Rdevel<br>
| (not<br>
| your issues, but you're not helping as it is not a minimally reproducible<br>
| example).<br>
|<br>
| You also still import / depend on spam. Why?<br>
|<br>
| | I hope this gives you what you need and thanks very much for looking into<br>
| this.<br>
|<br>
| I does blow up when I do<br>
|<br>
| > library(sparseList)<br>
| Loading required package: Rcpp<br>
| > data(C)<br>
| > sparse_list <- manip_sparseList(C)<br>
|<br>
| and it similarly blows up when R CMD check runs the examples.<br>
|<br>
|<br>
|<br>
| But the question is whether your code which does<br>
|<br>
| ## ...<br>
|<br>
| ## convert to dcgmatrices (under Matrix package) required for sparse<br>
| representation<br>
| ## as sp_mats in RcppArmadillo<br>
| C[[k]] <- as(C[[k]], "dgCMatrix")<br>
|<br>
|<br>
| creates a valid dgCMatrix. I do not know.<br>
|<br>
| RcppArmadillo is used by 77 packages on CRAN. Prof Ripley checks all of<br>
| those via valgrind and ASAN. Only yours threw an error.<br>
|<br>
| I am not yet convinced this is our issue. I could of course be wrong.<br>
|<br>
| Anyway, I won't be able to look at this for few days. Moreover, I do not<br>
| use<br>
| sparse matrices -- so you may need to rely on someone else to debug this<br>
| for<br>
| you, or do it yourself.<br>
|<br>
| Sorry, Dirk<br>
|<br>
| | <br>
| | Terrance<br>
| | <br>
| | [icon_10_ge] sparseList_1.1.tar.gz<br>
| | <br>
| |<br>
| |<br>
| | On Sat, Aug 9, 2014 at 5:25 PM, Dirk Eddelbuettel <<a href="mailto:edd@debian.org">edd@debian.org</a>> wrote:<br>
| |<br>
| |<br>
| | Hi Terrance,<br>
| |<br>
| | On 9 August 2014 at 16:58, terrance savitsky wrote:<br>
| | | Hi Guys, <br>
| | | <br>
| | | I embed a data list object of dgCMatrix elements (produced in the<br>
| same<br>
| | manner<br>
| | | as used in package, growfunctions) in an .RData file located in the<br>
| | sparseList<br>
| | | \data folder. I'm not sure what script and data ASAN uses to<br>
| conduct the<br>
| | memory<br>
| | | check. I'm guessing the package examples are run. So I added an<br>
| example<br>
| | that<br>
| | | loads the .RData object embedded with this toy package,<br>
| sparseList. <br>
| | Please<br>
| | | pardon me if my understanding is off. I also embedd an R script<br>
| in the<br>
| | single<br>
| |<br>
| | It's not complicated.<br>
| |<br>
| | 'R CMD check' needs a code path that does this, which can be as<br>
| simple as<br>
| | an<br>
| | example section in the manual page (mayne you meant this by 'added an<br>
| | example'). Or something tickled via a test script in tests/.<br>
| |<br>
| | You submitted a CRAN package. It has testable code. It failed. We now<br>
| need<br>
| | a<br>
| | _minimal reproducible example_, ideally without a long tail of<br>
| | dependencies,<br>
| | to examine it.<br>
| |<br>
| | | toy package .cpp file that generates the same data (list object of<br>
| | dgCMatrix<br>
| | | objects created using R::Matrix) and runs an example.<br>
| | | <br>
| | | Terrance<br>
| | | <br>
| | |<br>
| | | [icon_10_ge] sparseList_1.1.tar.gz<br>
| |<br>
| | Can you update to a version which loads data and passes down to the<br>
| code<br>
| | which was seen to tickle this?<br>
| |<br>
| | And then place it somewhere where we can get it with a simple wget or<br>
| curl<br>
| | call.<br>
| |<br>
| | Thanks, Dirk<br>
| | <br>
| |<br>
| | |<br>
| | | On Sat, Aug 9, 2014 at 3:02 PM, Kevin Ushey <<a href="mailto:kevinushey@gmail.com">kevinushey@gmail.com</a>><br>
| wrote:<br>
| | |<br>
| | | Hi Terrance,<br>
| | |<br>
| | | Can you please try to construct a reproducible example based on<br>
| the<br>
| | | (tagged, potentially offending) line of code -- that is,<br>
| | |<br>
| | | #4 0x7f79b6541df0 in IGMRFDPMIX<br>
| | | /data/gannet/ripley/R/packages/tests-49x/growfunctions/src/<br>
| | dpmix.cpp:63<br>
| | |<br>
| | | which has the line:<br>
| | |<br>
| | | C(k,0) = as<sp_mat>(Cr[k]);<br>
| | |<br>
| | | Can you construct a small, reproducible example using objects<br>
| 'C',<br>
| | | 'Cr' as you construct in the package, and then see if that<br>
| gives a<br>
| | | similar error?<br>
| | |<br>
| | | Preferably you can construct this using Rcpp Attributes, so an<br>
| | example<br>
| | | of the form<br>
| | |<br>
| | | // [[Rcpp::depends(RcppArmadillo)]]<br>
| | |<br>
| | | #include <RcppArmadillo.h><br>
| | | using namespace Rcpp;<br>
| | |<br>
| | | // [[Rcpp::export]]<br>
| | | SEXP test(SEXP C, SEXP Cr) {<br>
| | | ... use C, Cr as above...<br>
| | | }<br>
| | |<br>
| | | /*** R<br>
| | | ... construct C, Cr on the R side as necessary ...<br>
| | | */<br>
| | |<br>
| | | That way, we can 'sourceCpp' it and immediately investigate.<br>
| | |<br>
| | | Thanks,<br>
| | | Kevin<br>
| | |<br>
| | | On Sat, Aug 9, 2014 at 11:31 AM, terrance savitsky <<br>
| <a href="mailto:tds151@gmail.com">tds151@gmail.com</a>><br>
| | | wrote:<br>
| | | > Dirk, Thanks much for taking the time to investigate this<br>
| issue.<br>
| | I've<br>
| | | > created a package (sparseList_1.0.tar.gz) with a minimal<br>
| example of<br>
| | what<br>
| | | > writing from a list of dgCMatrix matrices at the R layer to a<br>
| field<br>
| | | <sp_mat>.<br>
| | | > I had intended to install boot2docker and use load your<br>
| R-devel<br>
| | set-up to<br>
| | | a<br>
| | | > container so that I could replicate the ASAN-generated error.<br>
| Its<br>
| | taking<br>
| | | me<br>
| | | > longer than I'd hoped, however, so - for now - I include the<br>
| | package<br>
| | | tarball<br>
| | | > with the minimal example and also an R script<br>
| | (sparse_list_to_field.R) so<br>
| | | > that you may see what the function does.<br>
| | | > Terrance<br>
| | | > sparse_list_to_field.R<br>
| | | ><br>
| | | > sparseList_1.0.tar.gz<br>
| | | ><br>
| | | ><br>
| | | ><br>
| | | > On Sat, Aug 9, 2014 at 11:50 AM, Dirk Eddelbuettel <<br>
| <a href="mailto:edd@debian.org">edd@debian.org</a>><br>
| | | wrote:<br>
| | | >><br>
| | | >><br>
| | | >> Terrance,<br>
| | | >><br>
| | | >> So I took a look with one of the prebuilt versions of R<br>
| using the<br>
| | Docker<br>
| | | >> container I prepare (see the blog / sanitizers write-up I<br>
| linked<br>
| | to) as<br>
| | | I<br>
| | | >> realized that we had a Rcpp Gallery post similar to what<br>
| that as<><br>
| | | >> conversion<br>
| | | >> does.<br>
| | | >><br>
| | | >> And no issue with that code.<br>
| | | >><br>
| | | >> From a brief look at your package it seems like you are<br>
| pushing<br>
| | the<br>
| | | >> envelope<br>
| | | >> a lot harder with fields and list of sparse matrices ( in<br>
| | dpmix.cpp,<br>
| | | where<br>
| | | >> another ASAN log pointed to ) so you may have to debug<br>
| around<br>
| | there.<br>
| | | >><br>
| | | >> A small reproducible example would surely help us. So far I<br>
| don't<br>
| | have<br>
| | | one<br>
| | | >> for as<sp_mat>.<br>
| | | >><br>
| | | >> Dirk<br>
| | | >><br>
| | | >> --<br>
| | | >> <a href="http://dirk.eddelbuettel.com" target="_blank">http://dirk.eddelbuettel.com</a> | @eddelbuettel |<br>
| <a href="mailto:edd@debian.org">edd@debian.org</a><br>
| | | ><br>
| | | ><br>
| | | ><br>
| | | ><br>
| | | > --<br>
| | | > Thank you, Terrance Savitsky<br>
| | | ><br>
| | | > _______________________________________________<br>
| | | > Rcpp-devel mailing list<br>
| | | > <a href="mailto:Rcpp-devel@lists.r-forge.r-project.org">Rcpp-devel@lists.r-forge.r-project.org</a><br>
| | | > <a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/</a><br>
| | rcpp-devel<br>
| | |<br>
| | |<br>
| | |<br>
| | |<br>
| | | --<br>
| | | Thank you, Terrance Savitsky<br>
| |<br>
| | --<br>
| | <a href="http://dirk.eddelbuettel.com" target="_blank">http://dirk.eddelbuettel.com</a> | @eddelbuettel | <a href="mailto:edd@debian.org">edd@debian.org</a><br>
| |<br>
| |<br>
| |<br>
| |<br>
| | --<br>
| | Thank you, Terrance Savitsky<br>
|<br>
| --<br>
| <a href="http://dirk.eddelbuettel.com" target="_blank">http://dirk.eddelbuettel.com</a> | @eddelbuettel | <a href="mailto:edd@debian.org">edd@debian.org</a><br>
|<br>
|<br>
|<br>
|<br>
| --<br>
| Thank you, Terrance Savitsky<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
<a href="http://dirk.eddelbuettel.com" target="_blank">http://dirk.eddelbuettel.com</a> | @eddelbuettel | <a href="mailto:edd@debian.org">edd@debian.org</a><br>
</font></span></blockquote></div><br></div>