[Rcpp-devel] Exporting rcpp-based function into parLapply workers in an R package

[Michael Weylandt]

Hi Naeem,

My (very quick) guess is that this isn't an Rcpp problem per se, but a
parLapply problem. You need to explicitly load your package on each
worker so that functions from it are available.

See, e.g., the brief discussion here:
https://stackoverflow.com/questions/18357788/parallel-parlapply-setup#18358875

The "parallel" packages do not exactly replicate your environment on
each worker node (to avoid expensive set-up / communication costs) so
you need to do a bit more set-up.

Best,
Michael

On Fri, May 14, 2021 at 11:49 AM Naeem Khoshnevis
<khoshnevis.naeem at gmail.com> wrote:
>
> Dear Rcpp developers:
>
> Thanks for developing and maintaining the Rcpp package.
> I wrote a function in Rcpp. It is available throughout the package and works as expected; however, it is not available for praLapply workers. A temporary fix is just using Rcpp::cppFunction inside the function that parLapply workers call and copy the entire function over there. However, this does not seem right for bigger and more complicated functions.
> I would be grateful if you could let me know whether there is a better long-term solution. Here is the package and three functions that you might want to take a look at.
>
> Original cpp function:
> https://github.com/fasrc/CausalGPS/blob/master/src/compute_closest_wgps_helper.cpp
>
> Wrapper function that calls this function + temporal fix:
> https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R
>
> The function that uses parLapply (please see line 63-89) to run the c++ code:
> https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R
>
> Best regards,
> Naeem
> _______________________________________________
> Rcpp-devel mailing list
> Rcpp-devel at lists.r-forge.r-project.org
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel