[datatable-help] Use of data.table with doMC/foreach

[Damian Betebenner]

All,

Does anyone have experience using data.table in parallel using doMC/foreach?

I have a data.table with approximately 3.5 million rows and am calculating different summaries (e.g., medians and counts) on some of the variables across approximately
500 distinct groupings (same j variable with 500 different by groups). My thought was to run the different analyses in parallel on my multi-core machine and get a good performance
boost but thus far it doesn't work any faster.

The data.table performance is great for each by grouping (about 7.5 seconds), but doing this 500 times take a while.

Anyone have experience along these lines.

Best,

Damian



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/datatable-help/attachments/20101229/5da58c29/attachment.htm>