[Rcpp-devel] Exporting rcpp-based function into parLapply workers in an R package

[Naeem Khoshnevis]

Thank you so much, Jeff. The part that I do not understand is the "and make
that package available in your worker environment" part. Could you please
let me know how I can make the package available for each worker.

On Fri, May 14, 2021 at 1:44 PM Jeff Newmiller <jdnewmil at dcn.davis.ca.us>
wrote:

> clusterExport works just fine if you put your Rcpp code into your own
> package and make that package available in your worker environment. Given
> the need for compilation in possibly a variety of computing environments
> for parallel processing this is definitely recommended.
>
> On May 14, 2021 10:35:25 AM PDT, Naeem Khoshnevis <
> khoshnevis.naeem at gmail.com> wrote:
> >Hi Michael,
> >
> >
> >Thank you so much for your response. That is correct. One method for
> >exporting required variables/functions is using the clusterExport
> >function,
> > which does not work for Rcpp-based functions. Another option is using
> >clusterEvalQ (as mentioned in the shared post); however, I am not sure
> >if
> >CRAN likes to see the library(package name) inside the codebase. What
> >are
> >your thoughts?
> >
> >Best regards,
> >Naeem
> >
> >On Fri, May 14, 2021 at 11:57 AM Michael Weylandt <
> >michael.weylandt at gmail.com> wrote:
> >
> >> Hi Naeem,
> >>
> >> My (very quick) guess is that this isn't an Rcpp problem per se, but
> >a
> >> parLapply problem. You need to explicitly load your package on each
> >> worker so that functions from it are available.
> >>
> >> See, e.g., the brief discussion here:
> >>
> >>
> >
> https://stackoverflow.com/questions/18357788/parallel-parlapply-setup#18358875
> >>
> >> The "parallel" packages do not exactly replicate your environment on
> >> each worker node (to avoid expensive set-up / communication costs) so
> >> you need to do a bit more set-up.
> >>
> >> Best,
> >> Michael
> >>
> >> On Fri, May 14, 2021 at 11:49 AM Naeem Khoshnevis
> >> <khoshnevis.naeem at gmail.com> wrote:
> >> >
> >> > Dear Rcpp developers:
> >> >
> >> > Thanks for developing and maintaining the Rcpp package.
> >> > I wrote a function in Rcpp. It is available throughout the package
> >and
> >> works as expected; however, it is not available for praLapply
> >workers. A
> >> temporary fix is just using Rcpp::cppFunction inside the function
> >that
> >> parLapply workers call and copy the entire function over there.
> >However,
> >> this does not seem right for bigger and more complicated functions.
> >> > I would be grateful if you could let me know whether there is a
> >better
> >> long-term solution. Here is the package and three functions that you
> >might
> >> want to take a look at.
> >> >
> >> > Original cpp function:
> >> >
> >>
> >
> https://github.com/fasrc/CausalGPS/blob/master/src/compute_closest_wgps_helper.cpp
> >> >
> >> > Wrapper function that calls this function + temporal fix:
> >> >
> >https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R
> >> >
> >> > The function that uses parLapply (please see line 63-89) to run the
> >c++
> >> code:
> >> >
> >https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R
> >> >
> >> > Best regards,
> >> > Naeem
> >> > _______________________________________________
> >> > Rcpp-devel mailing list
> >> > Rcpp-devel at lists.r-forge.r-project.org
> >> >
> >https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
> >>
>
> --
> Sent from my phone. Please excuse my brevity.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20210514/c9ddba15/attachment.html>