[Rcpp-devel] Timings for a program

[Martyn Plummer]

On Thu, 2017-01-05 at 05:39 -0700, Christian Gunning wrote:
> > 
> > I am new to Rcpp. I have converted my R program into an Rcpp program.
> > I want to ask a general question. Using Rcpp, my code is taking about 3
> > minutes to produce output for 100 observations. But it is taking about 2.45
> > hours to produce output for 1000 observations. Any suggestions or technical
> > hint for this type of problem.
> 
> First off, please include a minimal working example with your
> question. Without more information, there's not much we can do to
> help.

I agree with Christian that a reproducible working example is necessary
for detailed help with your problem. But I'm also bold enough to make a
guess about what is going on.

Your program is implementing an algorithm that is not linear in the
sample size but polynomial. You can verify this by recording run times
for sample sizes of 50, 60, 70, 80, 90, and 100. Plotting the run time
against the sample size should show the non-linear relationship. At a
sample size of 1000 the non-linear effects dominate, giving you a
roughly 5-fold increase in the run time over what you expect from your
smaller example.

It was pointed out many years ago by Joel Spolsky that this phenomenon
can arise from using a high level interface without understanding the
low-level implementation details. See his Wikipedia entry:

https://en.wikipedia.org/wiki/Joel_Spolsky#Schlemiel_the_Painter.27s_al
gorithm

Frequent reallocations of existing memory are a typical cause of this
problem.

All of this means that finding the underlying problem might be quite
hard for you to find, which is all the more reason to strip down your
program to find the smallest example that shows this behaviour.

Martyn

> In the meantime, you might find this to be an interesting and helpful
> read:
> http://www.burns-stat.com/documents/books/the-r-inferno/
> 
> Best,
> Christian-----------------------------------------------------------------------
This message and its attachments are strictly confidential. If you are
not the intended recipient of this message, please immediately notify 
the sender and delete it. Since its integrity cannot be guaranteed, 
its content cannot involve the sender's responsibility. Any misuse, 
any disclosure or publication of its content, either whole or partial, 
is prohibited, exception made of formally approved use
-----------------------------------------------------------------------