[Rcpp-devel] Help using rcpp and Rmpi

[Dirk Eddelbuettel]

Hi Molly,

Thanks for (re-)posting here. This is place for all things Rcpp.

On 24 June 2013 at 15:33, Molly Davies wrote:
| Hello,
| I have just written my first R extension using Rcpp, and it is SO much faster
| than my best R efforts, thank you for such a wonderful package! I used the
| following:
| 
| ### begin R ###
| require(Rcpp)
| cppFunction('

Consider making this a function in package.  sourceCpp() can help, along with
compileAttributes(); or use the old school approach (not that hard -- just
look at a smaller and simpler package such as wordcloud, or just use
Rcpp.package.skeleton(); see the vignette on this).

| double s2nICm(NumericVector ICvec, int m, double In, double sampleSize) {
|   int n = ICvec.size();
|   double covSum = 0.0;
|   double s2n = 0.0;
|   double ss = sampleSize*sampleSize;
|   for(int i=0; i<=m-1; i++){
|     for(int k=0; k<=i+m; k++){
|       covSum += ICvec[i]*ICvec[k];
|     }   
|   }
|   for(int i=m; i<=n-m-1; i++){
|     for(int k=i-m; k<=i+m; k++){
|       covSum += ICvec[i]*ICvec[k];
|     }
|   }
|   for(int i=n-m; i<=n; i++){
|     for(int k=i-m; k<=n; k++){
|       covSum += ICvec[i]*ICvec[k];
|     }
|   }
|   s2n = (In/ss) * covSum;
|   return s2n;
| }
| ')
| ### end R ###
| 
| This works perfectly on my laptop (Macbook air). Now I need to use this
| function as a part of a large simulation on my school's cluster (running Sun
| Grid Engine) using Rmpi. I included the above function in a text file, sourced
| that file in the master and sent the function above to my slaves using
| 
| mpi.bcast.Robj2slave(s2nICm)

The function needs object code. Your not shipping that.

I recommend a conservative approach. When I did (a lot) of MPI / Rmpi, I made
sure that all node shared /usr/local/ (for the R package installations) and
/home.  That is not needed, but simplifies things.

Then organize all code in a package. If the master can load the package, so
can the compute nodes.  So just have'em all load the package with your
function locally (via Rmpi), and then dispatch your job with nodes being 'ready'.
 
| This doesn't work, but there are no error messages - just endless silence from
| the slaves. (The original Rmpi simulation code that included the old, slow R
| version of the function above worked without error). I'm pretty sure I'm doing
| something wrong, but I'm not sure how to fix it. I'd really like to understand
| how Rcpp and Rmpi interact. Is it OK to call cppFunction('...') in the master
| and then send the resulting R function to slaves using mpi.bcast.Robj2slave()?
| Or do I need to send a text version of the cppFunction('...') to each slave and
| evaluate the text within each slave? What's the proper way to use Rcpp within
| the context of Rmpi? Does anyone know of any good resources out there that can
| help me understand what's going on under the hood?

Simplify, simply, simplify.  Start with Rmpi jobs with just R. Then use Rmpi
jobs with compiled packages, possibly using Rcpp. Then ship your package. 
With a bit of luck, and quite some attention to detail, you should get there.

Decompose into small steps.  MPI is a bit of work to set, but hey, you get
near unlimited bragging rights afterwards ;-)

Hth, Dirk

| Thanks very much for your time,
| Molly Davies
| Biostatistics Graduate Student
| UC Berkeley
| 
| ----------------------------------------------------------------------
| _______________________________________________
| Rcpp-devel mailing list
| Rcpp-devel at lists.r-forge.r-project.org
| https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
-- 
Dirk Eddelbuettel | edd at debian.org | http://dirk.eddelbuettel.com