[Rcpp-devel] Cannot access parameters in C++ code in parallel code called from Snow

[Dirk Eddelbuettel]

Turning on MPI _inside_ of a snow dispatch strikes me as a little risky. 

I have slides about using snow _along with MPI_ in the HPC slides from a few
years ago, below is an example script (launching somewhat length RNG
simulations which are easy to run in parallel for complete lack of
interdependence).

The world has gotten more complicated since (as there weren't really
multicore systems around then) but OpenMPI et al cover this.  You need to
check the OpenMPI docs how to set things up in a more fine-grained manner.

But even in a simplest instance you can just MPI send M jobs each to N
machines to get N * M running.

Dirk


#!/usr/bin/env r

suppressMessages(library(Rmpi))
suppressMessages(library(snow))

cl <- NULL
mpirank <- mpi.comm.rank(0)
if (mpirank == 0) {
    cl <- makeMPIcluster()
} else {                          # or are we a slave?
    sink(file="/dev/null")
    slaveLoop(makeMPImaster())
    mpi.finalize()
    q()
}

clusterEvalQ(cl, library(RDieHarder))
res <- parLapply(cl, c("mt19937","mt19937_1999",
           "mt19937_1998", "R_mersenne_twister"),
         function(x) {
                 dieharder(rng=x, test="operm5",
                           psamples=100, seed=12345)
                     })
stopCluster(cl)

print( do.call(rbind, lapply(res, function(x) { x[[1]] } )))

mpi.quit()



-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org