[Rcpp-devel] Fwd: CRAN submission growfunctions 0.1
Kevin Ushey
kevinushey at gmail.com
Sun Aug 10 08:55:23 CEST 2014
Hi All,
Just to confirm -- we can trigger this error with arbitrary integer sparse
matrices, e.g. (running with ASAN):
library(Rcpp)
cppFunction("arma::sp_mat asSpMat(SEXP S) { return as<arma::sp_mat>(S); }",
depends="RcppArmadillo")
m <- Matrix(sample(0:5, 100, TRUE), 20, 5, sparse = TRUE)
asSpMat(m)
So it looks like it's an Rcpp(Armadillo) bug after all -- we'll try to
figure out what's causing it.
Thanks for the report!
Cheers,
Kevin
On Sat, Aug 9, 2014 at 6:37 PM, Dirk Eddelbuettel <edd at debian.org> wrote:
>
> Hi Terrance,
>
> On 9 August 2014 at 20:05, terrance savitsky wrote:
> | darn. i R CMD check'd --as-cran under R-devel and also loaded the
> minimal
> | package, sparseList_1.1, the example and test run without issue
> | [icon_10_ge] sparseList_1.1.tar.gz
>
> Read the section "4.3 Checking memory access" of Writing R Extensions.
> You need a version of R compiled with "-fsanitize=address" as described
> there.
>
> My Docker image provides that, hence my links in the first email.
>
> | . So I'm stumped why the test is failing for you and not me. I attach
> the
> | file, again, just in I included the wrong version. Sorry I wasn't able
> to get
> | you a link that you could wget. I work in the U.S. federal system and am
> | limited by security protocols in what I'm able to do. Thanks for being
> | patient.
> |
> | Since I include the data object (which requires both the spam and Matrix
> | packages), I can/will remove the dependencies on those packages. Since
> I have
> | an example written into the documentation that runs on check, I will
> remove the
> | test I added (and, therefore, testthat). I will do this in the next
> day or
> | so, assuming what I attached still doesn't work.
>
> The minimal example we asked to provided is as follows
>
> library(sparseList) ## just to get the data set
> data(C) ## gets a list, you only need the first element
> ## function to invoke as<spmat>() on supplied argument
> cppFunction("arma::sp_mat asSpMat(SEXP S) { return
> as<arma::sp_mat>(S); }", depends="RcppArmadillo")
> ## call it: boom (if using -fsanitize=address)
> asSpMat(C[[1]])
>
> The open question now is whether your dgCMatrix is valid (likely) in which
> case our code may be wrong (yet someone who actually __used ans understands
> sparse matrix storage conventions__ needs to look at it). Our code
> survived
> the tests we threw at it, as well as use by other people using sparse
> matrix.
>
> Or you created a faulty dgCMatrix, in which case the bug is yours.
>
> Your didn't show how you create 'C'. So we can't really tell.
>
> And this point it would be good if you could do some digging. You're
> closest
> to your issue.
>
> Dirk
>
> | Stepping back, the code works in both the minimal package and
> growfunctions -
> | it inputs a list of dgCMatrix objects, assigns the elements to a
> field<sp_mat>
> | object and then writes it back out to a List object. The elements of the
> | outputted list object at the R layer are, indeed, dgCMatrix objects. So
> | 'yes', the script you referenced does create a valid dgCMatrix object
> and i've
> | verified such.
> |
> | As an aside, the C++ file embeds an R script that Kevin asked for if
> that is
> | useful.
>
> You need to run it under an executable instrumented for memory debugging to
> trigger the bug.
>
> Dirk
>
>
> |
> | terrance.
> |
> |
> | On Sat, Aug 9, 2014 at 7:14 PM, Dirk Eddelbuettel <edd at debian.org>
> wrote:
> |
> |
> | Terrance,
> |
> | On 9 August 2014 at 18:23, terrance savitsky wrote:
> | | Hi Dirk, The revised version of this toy package, sparseList, does
> | exactly what
> | | you request. I've included a data set that is of the same
> construction
> | (of
> | | dgCMatrix elements from R::Matrix wrapped into a list object) and
> have
> | added
> | | both an example and test that call this data. The package is
> minimal,
> | with a
> | | simple .cpp function (that reads in the list of dgCMatrix objects
> and
> | | writes the elements to an arma::field<arma:sp_mat>) to isolate the
> | source of
> | | the error in growfunctions.
> | |
> | | I attach the tarball, which you may also access through the
> following
> | URL:
> | |
> | | https://drive.google.com/file/d/
> | | 0B0DIDcaf6yMwZDRDZUNLd0lBbmZ3OHlHSzIyeV9hZ3RxZ0xr/edit?usp=sharing
> |
> | Did not work via wget, required a webpage visit and download.
> |
> | Package has extraneous dependencies (you do not need testthat to
> call a
> | function).
> |
> | Moreover, testthat seems to pivot over to /usr/bin/R rather than
> Rdevel
> | (not
> | your issues, but you're not helping as it is not a minimally
> reproducible
> | example).
> |
> | You also still import / depend on spam. Why?
> |
> | | I hope this gives you what you need and thanks very much for
> looking into
> | this.
> |
> | I does blow up when I do
> |
> | > library(sparseList)
> | Loading required package: Rcpp
> | > data(C)
> | > sparse_list <- manip_sparseList(C)
> |
> | and it similarly blows up when R CMD check runs the examples.
> |
> |
> |
> | But the question is whether your code which does
> |
> | ## ...
> |
> | ## convert to dcgmatrices (under Matrix package) required for
> sparse
> | representation
> | ## as sp_mats in RcppArmadillo
> | C[[k]] <- as(C[[k]], "dgCMatrix")
> |
> |
> | creates a valid dgCMatrix. I do not know.
> |
> | RcppArmadillo is used by 77 packages on CRAN. Prof Ripley checks
> all of
> | those via valgrind and ASAN. Only yours threw an error.
> |
> | I am not yet convinced this is our issue. I could of course be wrong.
> |
> | Anyway, I won't be able to look at this for few days. Moreover, I do
> not
> | use
> | sparse matrices -- so you may need to rely on someone else to debug
> this
> | for
> | you, or do it yourself.
> |
> | Sorry, Dirk
> |
> | |
> | | Terrance
> | |
> | | [icon_10_ge] sparseList_1.1.tar.gz
> | |
> | |
> | |
> | | On Sat, Aug 9, 2014 at 5:25 PM, Dirk Eddelbuettel <edd at debian.org>
> wrote:
> | |
> | |
> | | Hi Terrance,
> | |
> | | On 9 August 2014 at 16:58, terrance savitsky wrote:
> | | | Hi Guys,
> | | |
> | | | I embed a data list object of dgCMatrix elements (produced
> in the
> | same
> | | manner
> | | | as used in package, growfunctions) in an .RData file located
> in the
> | | sparseList
> | | | \data folder. I'm not sure what script and data ASAN uses to
> | conduct the
> | | memory
> | | | check. I'm guessing the package examples are run. So I
> added an
> | example
> | | that
> | | | loads the .RData object embedded with this toy package,
> | sparseList.
> | | Please
> | | | pardon me if my understanding is off. I also embedd an R
> script
> | in the
> | | single
> | |
> | | It's not complicated.
> | |
> | | 'R CMD check' needs a code path that does this, which can be as
> | simple as
> | | an
> | | example section in the manual page (mayne you meant this by
> 'added an
> | | example'). Or something tickled via a test script in tests/.
> | |
> | | You submitted a CRAN package. It has testable code. It failed.
> We now
> | need
> | | a
> | | _minimal reproducible example_, ideally without a long tail of
> | | dependencies,
> | | to examine it.
> | |
> | | | toy package .cpp file that generates the same data (list
> object of
> | | dgCMatrix
> | | | objects created using R::Matrix) and runs an example.
> | | |
> | | | Terrance
> | | |
> | | |
> | | | [icon_10_ge] sparseList_1.1.tar.gz
> | |
> | | Can you update to a version which loads data and passes down
> to the
> | code
> | | which was seen to tickle this?
> | |
> | | And then place it somewhere where we can get it with a simple
> wget or
> | curl
> | | call.
> | |
> | | Thanks, Dirk
> | |
> | |
> | | |
> | | | On Sat, Aug 9, 2014 at 3:02 PM, Kevin Ushey <
> kevinushey at gmail.com>
> | wrote:
> | | |
> | | | Hi Terrance,
> | | |
> | | | Can you please try to construct a reproducible example
> based on
> | the
> | | | (tagged, potentially offending) line of code -- that is,
> | | |
> | | | #4 0x7f79b6541df0 in IGMRFDPMIX
> | | |
> /data/gannet/ripley/R/packages/tests-49x/growfunctions/src/
> | | dpmix.cpp:63
> | | |
> | | | which has the line:
> | | |
> | | | C(k,0) = as<sp_mat>(Cr[k]);
> | | |
> | | | Can you construct a small, reproducible example using
> objects
> | 'C',
> | | | 'Cr' as you construct in the package, and then see if
> that
> | gives a
> | | | similar error?
> | | |
> | | | Preferably you can construct this using Rcpp Attributes,
> so an
> | | example
> | | | of the form
> | | |
> | | | // [[Rcpp::depends(RcppArmadillo)]]
> | | |
> | | | #include <RcppArmadillo.h>
> | | | using namespace Rcpp;
> | | |
> | | | // [[Rcpp::export]]
> | | | SEXP test(SEXP C, SEXP Cr) {
> | | | ... use C, Cr as above...
> | | | }
> | | |
> | | | /*** R
> | | | ... construct C, Cr on the R side as necessary ...
> | | | */
> | | |
> | | | That way, we can 'sourceCpp' it and immediately
> investigate.
> | | |
> | | | Thanks,
> | | | Kevin
> | | |
> | | | On Sat, Aug 9, 2014 at 11:31 AM, terrance savitsky <
> | tds151 at gmail.com>
> | | | wrote:
> | | | > Dirk, Thanks much for taking the time to investigate
> this
> | issue.
> | | I've
> | | | > created a package (sparseList_1.0.tar.gz) with a
> minimal
> | example of
> | | what
> | | | > writing from a list of dgCMatrix matrices at the R
> layer to a
> | field
> | | | <sp_mat>.
> | | | > I had intended to install boot2docker and use load your
> | R-devel
> | | set-up to
> | | | a
> | | | > container so that I could replicate the ASAN-generated
> error.
> | Its
> | | taking
> | | | me
> | | | > longer than I'd hoped, however, so - for now - I
> include the
> | | package
> | | | tarball
> | | | > with the minimal example and also an R script
> | | (sparse_list_to_field.R) so
> | | | > that you may see what the function does.
> | | | > Terrance
> | | | > sparse_list_to_field.R
> | | | >
> | | | > sparseList_1.0.tar.gz
> | | | >
> | | | >
> | | | >
> | | | > On Sat, Aug 9, 2014 at 11:50 AM, Dirk Eddelbuettel <
> | edd at debian.org>
> | | | wrote:
> | | | >>
> | | | >>
> | | | >> Terrance,
> | | | >>
> | | | >> So I took a look with one of the prebuilt versions of
> R
> | using the
> | | Docker
> | | | >> container I prepare (see the blog / sanitizers
> write-up I
> | linked
> | | to) as
> | | | I
> | | | >> realized that we had a Rcpp Gallery post similar to
> what
> | that as<>
> | | | >> conversion
> | | | >> does.
> | | | >>
> | | | >> And no issue with that code.
> | | | >>
> | | | >> From a brief look at your package it seems like you
> are
> | pushing
> | | the
> | | | >> envelope
> | | | >> a lot harder with fields and list of sparse matrices
> ( in
> | | dpmix.cpp,
> | | | where
> | | | >> another ASAN log pointed to ) so you may have to debug
> | around
> | | there.
> | | | >>
> | | | >> A small reproducible example would surely help us. So
> far I
> | don't
> | | have
> | | | one
> | | | >> for as<sp_mat>.
> | | | >>
> | | | >> Dirk
> | | | >>
> | | | >> --
> | | | >> http://dirk.eddelbuettel.com | @eddelbuettel |
> | edd at debian.org
> | | | >
> | | | >
> | | | >
> | | | >
> | | | > --
> | | | > Thank you, Terrance Savitsky
> | | | >
> | | | > _______________________________________________
> | | | > Rcpp-devel mailing list
> | | | > Rcpp-devel at lists.r-forge.r-project.org
> | | | >
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/
> | | rcpp-devel
> | | |
> | | |
> | | |
> | | |
> | | | --
> | | | Thank you, Terrance Savitsky
> | |
> | | --
> | | http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
> | |
> | |
> | |
> | |
> | | --
> | | Thank you, Terrance Savitsky
> |
> | --
> | http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
> |
> |
> |
> |
> | --
> | Thank you, Terrance Savitsky
>
> --
> http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20140809/a3b83b05/attachment-0001.html>
More information about the Rcpp-devel
mailing list