[Rcpp-devel] Exporting rcpp-based function into parLapply workers in an R package

[Naeem Khoshnevis]

Hi Michael,


Thank you so much for your response. That is correct. One method for
exporting required variables/functions is using the clusterExport function,
 which does not work for Rcpp-based functions. Another option is using
clusterEvalQ (as mentioned in the shared post); however, I am not sure if
CRAN likes to see the library(package name) inside the codebase. What are
your thoughts?

Best regards,
Naeem

On Fri, May 14, 2021 at 11:57 AM Michael Weylandt <
michael.weylandt at gmail.com> wrote:

> Hi Naeem,
>
> My (very quick) guess is that this isn't an Rcpp problem per se, but a
> parLapply problem. You need to explicitly load your package on each
> worker so that functions from it are available.
>
> See, e.g., the brief discussion here:
>
> https://stackoverflow.com/questions/18357788/parallel-parlapply-setup#18358875
>
> The "parallel" packages do not exactly replicate your environment on
> each worker node (to avoid expensive set-up / communication costs) so
> you need to do a bit more set-up.
>
> Best,
> Michael
>
> On Fri, May 14, 2021 at 11:49 AM Naeem Khoshnevis
> <khoshnevis.naeem at gmail.com> wrote:
> >
> > Dear Rcpp developers:
> >
> > Thanks for developing and maintaining the Rcpp package.
> > I wrote a function in Rcpp. It is available throughout the package and
> works as expected; however, it is not available for praLapply workers. A
> temporary fix is just using Rcpp::cppFunction inside the function that
> parLapply workers call and copy the entire function over there. However,
> this does not seem right for bigger and more complicated functions.
> > I would be grateful if you could let me know whether there is a better
> long-term solution. Here is the package and three functions that you might
> want to take a look at.
> >
> > Original cpp function:
> >
> https://github.com/fasrc/CausalGPS/blob/master/src/compute_closest_wgps_helper.cpp
> >
> > Wrapper function that calls this function + temporal fix:
> > https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R
> >
> > The function that uses parLapply (please see line 63-89) to run the c++
> code:
> > https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R
> >
> > Best regards,
> > Naeem
> > _______________________________________________
> > Rcpp-devel mailing list
> > Rcpp-devel at lists.r-forge.r-project.org
> > https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20210514/e51b5e2d/attachment.html>