[Rcpp-devel] Exporting rcpp-based function into parLapply workers in an R package

[Jeff Newmiller]

clusterExport works just fine if you put your Rcpp code into your own package and make that package available in your worker environment. Given the need for compilation in possibly a variety of computing environments for parallel processing this is definitely recommended.

On May 14, 2021 10:35:25 AM PDT, Naeem Khoshnevis <khoshnevis.naeem at gmail.com> wrote:
>Hi Michael,
>
>
>Thank you so much for your response. That is correct. One method for
>exporting required variables/functions is using the clusterExport
>function,
> which does not work for Rcpp-based functions. Another option is using
>clusterEvalQ (as mentioned in the shared post); however, I am not sure
>if
>CRAN likes to see the library(package name) inside the codebase. What
>are
>your thoughts?
>
>Best regards,
>Naeem
>
>On Fri, May 14, 2021 at 11:57 AM Michael Weylandt <
>michael.weylandt at gmail.com> wrote:
>
>> Hi Naeem,
>>
>> My (very quick) guess is that this isn't an Rcpp problem per se, but
>a
>> parLapply problem. You need to explicitly load your package on each
>> worker so that functions from it are available.
>>
>> See, e.g., the brief discussion here:
>>
>>
>https://stackoverflow.com/questions/18357788/parallel-parlapply-setup#18358875
>>
>> The "parallel" packages do not exactly replicate your environment on
>> each worker node (to avoid expensive set-up / communication costs) so
>> you need to do a bit more set-up.
>>
>> Best,
>> Michael
>>
>> On Fri, May 14, 2021 at 11:49 AM Naeem Khoshnevis
>> <khoshnevis.naeem at gmail.com> wrote:
>> >
>> > Dear Rcpp developers:
>> >
>> > Thanks for developing and maintaining the Rcpp package.
>> > I wrote a function in Rcpp. It is available throughout the package
>and
>> works as expected; however, it is not available for praLapply
>workers. A
>> temporary fix is just using Rcpp::cppFunction inside the function
>that
>> parLapply workers call and copy the entire function over there.
>However,
>> this does not seem right for bigger and more complicated functions.
>> > I would be grateful if you could let me know whether there is a
>better
>> long-term solution. Here is the package and three functions that you
>might
>> want to take a look at.
>> >
>> > Original cpp function:
>> >
>>
>https://github.com/fasrc/CausalGPS/blob/master/src/compute_closest_wgps_helper.cpp
>> >
>> > Wrapper function that calls this function + temporal fix:
>> >
>https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R
>> >
>> > The function that uses parLapply (please see line 63-89) to run the
>c++
>> code:
>> >
>https://github.com/fasrc/CausalGPS/blob/master/R/compute_closest_wgps.R
>> >
>> > Best regards,
>> > Naeem
>> > _______________________________________________
>> > Rcpp-devel mailing list
>> > Rcpp-devel at lists.r-forge.r-project.org
>> >
>https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
>>

-- 
Sent from my phone. Please excuse my brevity.