@Dennis<div><br></div><div>Thanks for your suggestion, that would indeed work in this particular case in which you are only matching to one numeric column that is statistically well defined (in this case, uniformly distributed). My use case, which is not that uncommon I guess, is more general, and instead of trying to make an example (I failed twice on that now, sorry for that ;-), let's discuss it more general:</div>
<div><br></div><div>Starting with the idiom x[i, where i is a data.table which columns match to the keys in x. Quoting the help page: <span class="Apple-style-span" style="border-collapse:separate;font-family:Arial;font-size:medium"><span class="Apple-style-span" style="font-family:sans-serif;font-size:13px">When <code>i</code> is a <code>data.table</code>, <code>x</code> must have a key. <code>i</code> is <em>joined</em> to <code>x</code> using the key and the rows in <code>x</code> that match are returned. When it comes to "the rows in x that match are returned", there are currently three options: </span></span></div>
<div><ul><li><span style="font-family:sans-serif;font-size:13px">mult="all". As far as I see it, this is not applicable in its current implementation to a real life MCS, because it just gets too big pretty fast. So any solution that starts with calling mult="all" first and then do some random drawing fails. However, and this is what Dennis' solution does, is that one could use the fact that in a MCS many runs are identical. In the example above, although intJoin contains 10,000 rows, it's basically only 10 joins (for every year one). So one could expect the user to do that himself, thus leaving base data.table lean. However, in my opinion this is not easy. As an example (I try it again), let's assume i consists of four characteristics, so there are 4^4 = 256 combinations. Those combinations aren't uniformly distributed, so in a MCS of n=10,000 combination 1 (C1) might occur 2,000 times, C2 1,000 times and C256 10 times. Now, the user only has to join 256, but he has to keep track about how often every combination occurs. After that, he has to sample from every combination according to its occurrence (get 2,000 random samples from the join of dt and C1, 1,000 random samples from the join of dt and C2, etc.). And after that he has to join those tables again. In my opinion, that is tough and error-prone.</span></li>
<li><span style="font-family:sans-serif;font-size:13px">mult="first": Not applicable to MCS.</span></li><li><span style="font-family:sans-serif;font-size:13px">mult="last": </span><span style="font-family:sans-serif;font-size:13px">Not applicable to MCS.</span></li>
</ul><div><font face="sans-serif">So I still think there is a valid point for mult="random" because I don't see an easy and flexible workaround with the current options in data.table. With respect to Steve's comment: I see your point and I think Matthew has some thoughts on that. However, I can't think of any good examples for weighting sampling because i already does that implicitly. That is, in a MCS, i should put more weight on combinations that agree more frequently, so you don't really need a weighted sampling (in the example above this is done by the fact that C1 is 2,000 times in i, C256 only 10 times). The idea of weighting would be useful if you went with the approach outlined in mult="all", i.e. you get rid of all duplicates and do every join only once. Then it would be great to have an option to tell R/data.table how often every option would occur. In the example above, you then would have a weighting vector for the different combinations C1, C2, ..., C256 of c(0.2, 0.1,..., 0.001).</font></div>
<div><font face="sans-serif"><br></font></div><div><font face="sans-serif">However, data.table is out for a while now and apparently, this issue hasn't come up before, so I guess we should just keep it in mind and move on. I could raise a feature request with lowest priority that links to this thread .</font></div>
<div><font face="sans-serif"><br></font></div><div><font face="sans-serif">Christoph</font></div><br><div class="gmail_quote">On Sun, Jan 8, 2012 at 12:00 PM, <span dir="ltr"><<a href="mailto:datatable-help-request@r-forge.wu-wien.ac.at">datatable-help-request@r-forge.wu-wien.ac.at</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send datatable-help mailing list submissions to<br>
<a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br>
<br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:datatable-help-request@lists.r-forge.r-project.org">datatable-help-request@lists.r-forge.r-project.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:datatable-help-owner@lists.r-forge.r-project.org">datatable-help-owner@lists.r-forge.r-project.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of datatable-help digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: What's your opinion on the feature request: add option<br>
mult="random" (djmuseR)<br>
2. Re: What's your opinion on the feature request: add option<br>
mult="random" (Steven C. Bagley)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Sat, 7 Jan 2012 06:32:22 -0800 (PST)<br>
From: djmuseR <<a href="mailto:djmuser@gmail.com">djmuser@gmail.com</a>><br>
Subject: Re: [datatable-help] What's your opinion on the feature<br>
request: add option mult="random"<br>
To: <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
Message-ID: <<a href="mailto:1325946742615-4273090.post@n4.nabble.com">1325946742615-4273090.post@n4.nabble.com</a>><br>
Content-Type: text/plain; charset=us-ascii<br>
<br>
Hi:<br>
<br>
Here's one possible alternative:<br>
<br>
# I just made intJoin an integer vector rather than a one column data table<br>
intJoin <- sample(seq_len(10), size = 10000, replace = TRUE)<br>
> table(intJoin)<br>
intJoin<br>
<a href="tel:1%20%20%20%202%20%20%20%203%20%20%20%204%20%20%20%205%20%20%20%206%20%20%20%207%20%20%20%208%20%20%20%209%20%20%2010" value="+12345678910">1 2 3 4 5 6 7 8 9 10</a><br>
951 1001 969 1063 999 1007 1004 1035 933 1038<br>
<br>
# This function takes samples of size n_i from each year's sub-data<br>
# with replacement, since the sample size can be higher than the<br>
# number of rows in each sub-data table (1000 in this case)<br>
h <- function(dt, svec) {<br>
ns <- as.vector(table(svec))<br>
dt[, .SD[sample(nrow(.SD), ns[Year], replace = TRUE), ], by = 'Year']<br>
}<br>
u <- h(rawData, intJoin)<br>
> dim(u)<br>
[1] 10000 2<br>
> head(u)<br>
Year fundID<br>
[1,] 1 20091<br>
[2,] 1 92311<br>
[3,] 1 18341<br>
[4,] 1 79721<br>
[5,] 1 13391<br>
[6,] 1 15301<br>
<br>
# Check:<br>
> table(u$Year)<br>
<a href="tel:1%20%20%20%202%20%20%20%203%20%20%20%204%20%20%20%205%20%20%20%206%20%20%20%207%20%20%20%208%20%20%20%209%20%20%2010" value="+12345678910">1 2 3 4 5 6 7 8 9 10</a><br>
951 1001 969 1063 999 1007 1004 1035 933 1038<br>
> system.time(h(rawData, intJoin))<br>
user system elapsed<br>
0.03 0.00 0.03<br>
<br>
Since timings differ on machines, I tried out your foo1() function for<br>
comparison, after converting intJoin to a data table:<br>
> intJoin <- J(sample(seq_len(10), size = 10000, replace = TRUE))<br>
> system.time(finalData <- foo1(10000, intJoin, rawData))<br>
user system elapsed<br>
30.61 0.03 30.7<br>
<br>
HTH,<br>
Dennis<br>
<br>
--<br>
View this message in context: <a href="http://r.789695.n4.nabble.com/What-s-your-opinion-on-the-feature-request-add-option-mult-random-tp4267483p4273090.html" target="_blank">http://r.789695.n4.nabble.com/What-s-your-opinion-on-the-feature-request-add-option-mult-random-tp4267483p4273090.html</a><br>
Sent from the datatable-help mailing list archive at Nabble.com.<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Sat, 7 Jan 2012 19:07:53 -0800<br>
From: "Steven C. Bagley" <<a href="mailto:steven.bagley@gmail.com">steven.bagley@gmail.com</a>><br>
Subject: Re: [datatable-help] What's your opinion on the feature<br>
request: add option mult="random"<br>
To: <a href="mailto:christoph.jaeckel@wi.tum.de">christoph.jaeckel@wi.tum.de</a><br>
Cc: <a href="mailto:datatable-help@r-forge.wu-wien.ac.at">datatable-help@r-forge.wu-wien.ac.at</a><br>
Message-ID: <<a href="mailto:3648349B-5BD2-4911-AB5B-9ECC653C5D83@gmail.com">3648349B-5BD2-4911-AB5B-9ECC653C5D83@gmail.com</a>><br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
The mult argument is becoming its own little programming language. I worry that this is going to get complicated in an ad hoc way. What if someone wants random, but with weighting? Each new value of mult is really shorthand for an R language construct. Maybe there is a more general way to express these ideas using existing R constructs? (I'm not sure how to do this consistently. I'm merely making an observation.)<br>
<br>
--Steve<br>
<br>
On Jan 6, 2012, at 5:58 AM, Christoph J?ckel wrote:<br>
<br>
> Thanks for your feedback. @Chris: I guess Matthew's example and your's do not really match because he doesn't call sample(dt,...), but sample(dt[i, which=TRUE],... His option, though, returns all the rows that match between the keys of dt and i and takes a random sample of size 1 from that, so I guess it does what I expected. Nevertheless, I think an option mult="random" would still be useful. Here is why:<br>
><br>
> I guess my first example was a little bit too simplistic, sorry for that! Here is an updated, more realistic example of what I do and some hints about my current implementation of mult="random":<br>
><br>
> require(data.table)<br>
> rawData <- data.table(fundID = 1:1e5,<br>
> Year = rep(1:10, times=1e4),<br>
> key = "Year")<br>
> #Let's have 10000 runs; in each run we want to draw a fund with a year that is<br>
> #set dynamically<br>
> intJoin <- J(sample(1:10, size=10000, replace=TRUE))<br>
><br>
> #Best solution I have come up so far with the current options in data.table<br>
> #Is there one that can beat mult="random" and is easy for the user to implement?<br>
> foo1 <- function(n, intJoin, rawData) {<br>
> x <- integer(n)<br>
> for (r in seq_len(nrow(intJoin))) {<br>
> x[r] <- sample(rawData[intJoin[r], which=TRUE], size=1)<br>
> }<br>
> return(rawData[x])<br>
> }<br>
> system.time(finalData <- foo1(10000, intJoin, rawData))<br>
> # user system elapsed<br>
> # 43.827 0.000 44.232<br>
> #Check that it does what it should: match random entities to the exact year in intJoin<br>
> cbind(finalData, intJoin)<br>
> # fundID Year V1<br>
> # [1,] 46556 6 6<br>
> # [2,] 77642 2 2<br>
> # [3,] 17325 5 5<br>
> # [4,] 36617 7 7<br>
> # [5,] 90697 7 7<br>
> # [6,] 4536 6 6<br>
> # [7,] 22273 3 3<br>
> # [8,] 46825 5 5<br>
> # [9,] 65788 8 8<br>
> # [10,] 14153 3 3<br>
><br>
> #My implementation of mult="random"<br>
> system.time(finalData <- rawData[intJoin, mult="random"])<br>
> # user system elapsed<br>
> # 0.324 0.016 0.337<br>
> #Pretty fast and easy to understand<br>
> #Check that it does what it should: match random entities to the exact year in intJoin<br>
> cbind(finalData, intJoin)<br>
> # Year fundID V1<br>
> # [1,] 6 39626 6<br>
> # [2,] 2 98552 2<br>
> # [3,] 5 85425 5<br>
> # [4,] 7 24637 7<br>
> # [5,] 7 74797 7<br>
> # [6,] 6 87626 6<br>
> # [7,] 3 88973 3<br>
> # [8,] 5 60335 5<br>
> # [9,] 8 62298 8<br>
> # [10,] 3 23283 3<br>
><br>
> If you want to try it out yourself: Just call<br>
><br>
> fixInNamespace("[.data.table", pos="package:data.table")<br>
><br>
> and change the following lines in the editor (this applies to data.table 1.7.7):<br>
><br>
> OLD LINE: if (!mult %in% c("first", "last", "all")) stop("mult argument can only be 'first','last' or 'all'")<br>
> NEW LINE: if (!mult %in% c("first","last","all", "random")) stop("mult argument can only be 'first','last', 'all', or 'random'")<br>
><br>
> and<br>
><br>
> OLD LINES: else {<br>
> irows = if (mult == "first")<br>
> idx.start<br>
> else idx.end<br>
> lengths = rep(1L, length(irows))<br>
> }<br>
><br>
> NEW LINES: } else if (mult=="first") {<br>
> irows = idx.start<br>
> lengths=rep(1L,length(irows))<br>
> } else if (mult=="last") {<br>
> irows = idx.end<br>
> lengths=rep(1L,length(irows))<br>
> } else {<br>
> irows = mapply(function(x1, x2) {sample(x1:x2, size=1)}, idx.start, idx.end)<br>
> lengths = rep(1L,length(irows))<br>
> }<br>
><br>
> However, I don't know what's going on in the line<br>
> .Call("binarysearch", i, x, as.integer(leftcols -<br>
> 1), as.integer(rightcols - 1), haskey(i), roll,<br>
> rolltolast, idx.start, idx.end, PACKAGE = "data.table")<br>
><br>
> I figured out that idx.start and idx.end are changed with this function call and I guess at this point in the function it should always be that idx.start and idx.end are of the same lenght and both return only integer values that represent rows of x, but here I'm not 100% sure. So maybe additional checks are needed in the else clause when the mapply-function is called.<br>
><br>
> So let me know what you think. I will join the project independent of that particular issue and try to help, but I guess I should start with simple things. So if there is any help needed on documentation checking<br>
> or stuff like that, just let me know and I try my best!<br>
><br>
> Christoph<br>
><br>
> On Fri, Jan 6, 2012 at 1:52 PM, Chris Neff <<a href="mailto:caneff@gmail.com">caneff@gmail.com</a>> wrote:<br>
> That isn't doing quite what he does. I don't know what you expected<br>
><br>
> sample(dt, size=1)<br>
><br>
> to do but it seems to essentially do this:<br>
><br>
> dt[sample(1:ncol(dt),size=1),]<br>
><br>
> It picking a random column number and then return that row instead.<br>
> Try it for yourself:<br>
><br>
> dt=data.table(x=1:10,y=1:10,z=1:10)<br>
> sample(dt, size=1)<br>
><br>
> The only rows you will get is 1,1,1 2,2,2 and 3,3,3. Caveat as usual<br>
> is I'm on 1.7.1 until my crashing bug is fixed so apologies if this<br>
> works properly in later versions.<br>
><br>
> Note that this diverges from what sample(df, size=1) does, which is<br>
> picks a random column and returns that whole column.<br>
><br>
> What he really wants is to pick a random row from each subset (I<br>
> think). None of your examples do that and I can't think of a simpler<br>
> way than what he suggests.<br>
><br>
> On 6 January 2012 03:34, Matthew Dowle <<a href="mailto:mdowle@mdowle.plus.com">mdowle@mdowle.plus.com</a>> wrote:<br>
> > Very keen for direct contributions in that way, happy to help you with<br>
> > svn etc, and you joining the project.<br>
> ><br>
> > In this particular example, how about :<br>
> ><br>
> > rawData[sample(rawData[J("eu"), which=TRUE],size=1)]<br>
> ><br>
> > This solves the inefficiency of the 1st step; i.e.,<br>
> > intDT <- rawData[J("eu"), mult="all"]<br>
> > which copies a subset of all the columns, whilst retaining flexibility<br>
> > for the user so user can easily sample 2 rows, or any other R method to<br>
> > select a random subset.<br>
> ><br>
> > Because of potential scoping conflicts (say a column was called<br>
> > "rawData" i.e. the same name of the table), to be more robust :<br>
> ><br>
> > x = sample(rawData[J("eu"), which=TRUE],size=1)<br>
> > rawData[x]<br>
> ><br>
> > This is slightly different because when i is a single name (x in this<br>
> > case), data.table knows the caller must mean the x in calling scope, not<br>
> > the column called "x" (if any). Is two steps like this ok? I'm<br>
> > guessing it was really the inefficiency that was the motivation?<br>
> ><br>
> > Matthew<br>
> ><br>
> > On Fri, 2012-01-06 at 00:20 +0100, Christoph J?ckel wrote:<br>
> >> Hi together,<br>
> >><br>
> >><br>
> >> I run a Monte Carlo simulation on a data.table and do that currently<br>
> >> with a loop: on every run, I choose a subset of rows subject to<br>
> >> certain criteria and from those rows I take a random element.<br>
> >> Currently, I do the following: Let's say I have funds from two regions<br>
> >> ("eu" and "us") and I want to choose a random fund from "eu" (could be<br>
> >> "us" in the next run and a different region in the third):<br>
> >><br>
> >><br>
> >> library(data.table)<br>
> >> rawData <- data.table(fundID = letters,<br>
> >> compGeo = rep(c("us", "eu"), each=13))<br>
> >> setkey(rawData, "compGeo")<br>
> >> intDT <- rawData[J("eu"), mult="all"]<br>
> >> intDT[<a href="http://sample.int" target="_blank">sample.int</a>(nrow(intDT), size=1)]<br>
> >><br>
> >><br>
> >> So my idea is to just give the user the option mult="random", which<br>
> >> does this in one step. What do you think about that feature request?<br>
> >><br>
> >><br>
> >> With respect to the implementation: I changed a few lines in the<br>
> >> function '[.data.table' and got this to run on my locale data.table<br>
> >> version, so I guess I could implement it (as far as I can see, one<br>
> >> just needs to change some R code). However, I haven't done extensive<br>
> >> testing and I'm not an expert on shared projects and subversion (never<br>
> >> did that actually), so I guess I would need some help to start with<br>
> >> and the confirmation I couldn't break anything ;-)<br>
> >><br>
> >><br>
> >> Christoph<br>
> >><br>
> >><br>
> >><br>
> >><br>
> >> _______________________________________________<br>
> >> datatable-help mailing list<br>
> >> <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
> >> <a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > datatable-help mailing list<br>
> > <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
> > <a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br>
><br>
><br>
> _______________________________________________<br>
> datatable-help mailing list<br>
> <a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
> <a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.r-forge.r-project.org/pipermail/datatable-help/attachments/20120107/15013b94/attachment-0001.htm" target="_blank">http://lists.r-forge.r-project.org/pipermail/datatable-help/attachments/20120107/15013b94/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
datatable-help mailing list<br>
<a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
<a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br>
<br>
End of datatable-help Digest, Vol 23, Issue 9<br>
*********************************************<br>
</blockquote></div><br><br>
</div>