[datatable-help] What's your opinion on the feature request: add option mult="random"
Christoph Jäckel
christoph.jaeckel at wi.tum.de
Sun Jan 8 21:36:15 CET 2012
I think that this is a really good idea in that
1. it is consistent with the current implementation
2. it is quite flexible.
One only has to do decide what to do when the the user gives invalid
integer, i.e. integers larger .N. Either one throws an error or gives a
warning and take .N. What do the others think of Matthew's proposal?
On a side note:
I just played around with the mult= option and discovered the following:
DT <- data.table(X = rnorm(100),
Y = rnorm(100),
C =rep(c(1, 2), each=100),
key="C")
all.equal(DT[C==1, mult="first"], DT[C==1, mult="all"])
all.equal(DT[1:100, mult="first"], DT[1:100, mult="all"])
#[1] TRUE
For me, it is rather surprising that both mult="all" and mult="first"
return all rows that match C==1, i.e. in the case when i is a logical or
integer. From the help page: "When *multiple* rows in x match to the row in
i, mult controls which are returned: "all" (default), "first" or "last"."
Although one could argue that a "row" only applies to a data.table and not
a vector, as a user I would expect mult= to work on both, i.e. when i is a
data.table, a logical or an integer. However, it only works for
data.tables, which I checked in the source code. So should I update the
documentation to make it clearer that mult= only works if i is a data.table?
On Sun, Jan 8, 2012 at 8:25 PM, Matthew Dowle <mdowle at mdowle.plus.com>wrote:
> How about allowing mult to be integer (or an expression that evaluates
> to integer) :
>
> DT[X, mult="first"]
> DT[X, mult=1L] # same
>
> DT[X, mult="last"]
> DT[X, mult=.N] # same
>
> DT[X, .SD[2]] # 2nd row of each group (inefficient due to .SD, there
> are other longer alternatives)
> DT[X, mult=2L] # same, but efficient and simple
>
> DT[X, mult="random"]
> DT[X, mult=sample(.N,size=1)] # same, but more general
>
> DT[X, mult=-1L] # all but the first of each group
>
> Matthew
>
> On Sat, 2012-01-07 at 19:07 -0800, Steven C. Bagley wrote:
> > The mult argument is becoming its own little programming language. I
> > worry that this is going to get complicated in an ad hoc way. What if
> > someone wants random, but with weighting? Each new value of mult is
> > really shorthand for an R language construct. Maybe there is a more
> > general way to express these ideas using existing R constructs? (I'm
> > not sure how to do this consistently. I'm merely making an
> > observation.)
> >
> >
> > --Steve
> >
> >
> > On Jan 6, 2012, at 5:58 AM, Christoph Jäckel wrote:
> >
> > > Thanks for your feedback. @Chris: I guess Matthew's example and
> > > your's do not really match because he doesn't call sample(dt,...),
> > > but sample(dt[i, which=TRUE],... His option, though, returns all the
> > > rows that match between the keys of dt and i and takes a random
> > > sample of size 1 from that, so I guess it does what I expected.
> > > Nevertheless, I think an option mult="random" would still be useful.
> > > Here is why:
> > >
> > >
> > > I guess my first example was a little bit too simplistic, sorry for
> > > that! Here is an updated, more realistic example of what I do and
> > > some hints about my current implementation of mult="random":
> > >
> > >
> > > require(data.table)
> > > rawData <- data.table(fundID = 1:1e5,
> > > Year = rep(1:10, times=1e4),
> > > key = "Year")
> > > #Let's have 10000 runs; in each run we want to draw a fund with a
> > > year that is
> > > #set dynamically
> > > intJoin <- J(sample(1:10, size=10000, replace=TRUE))
> > >
> > >
> > > #Best solution I have come up so far with the current options in
> > > data.table
> > > #Is there one that can beat mult="random" and is easy for the user
> > > to implement?
> > > foo1 <- function(n, intJoin, rawData) {
> > > x <- integer(n)
> > > for (r in seq_len(nrow(intJoin))) {
> > > x[r] <- sample(rawData[intJoin[r], which=TRUE], size=1)
> > > }
> > > return(rawData[x])
> > > }
> > > system.time(finalData <- foo1(10000, intJoin, rawData))
> > > # user system elapsed
> > > # 43.827 0.000 44.232
> > > #Check that it does what it should: match random entities to the
> > > exact year in intJoin
> > > cbind(finalData, intJoin)
> > > # fundID Year V1
> > > # [1,] 46556 6 6
> > > # [2,] 77642 2 2
> > > # [3,] 17325 5 5
> > > # [4,] 36617 7 7
> > > # [5,] 90697 7 7
> > > # [6,] 4536 6 6
> > > # [7,] 22273 3 3
> > > # [8,] 46825 5 5
> > > # [9,] 65788 8 8
> > > # [10,] 14153 3 3
> > >
> > >
> > > #My implementation of mult="random"
> > > system.time(finalData <- rawData[intJoin, mult="random"])
> > > # user system elapsed
> > > # 0.324 0.016 0.337
> > > #Pretty fast and easy to understand
> > > #Check that it does what it should: match random entities to the
> > > exact year in intJoin
> > > cbind(finalData, intJoin)
> > > # Year fundID V1
> > > # [1,] 6 39626 6
> > > # [2,] 2 98552 2
> > > # [3,] 5 85425 5
> > > # [4,] 7 24637 7
> > > # [5,] 7 74797 7
> > > # [6,] 6 87626 6
> > > # [7,] 3 88973 3
> > > # [8,] 5 60335 5
> > > # [9,] 8 62298 8
> > > # [10,] 3 23283 3
> > >
> > >
> > > If you want to try it out yourself: Just call
> > >
> > >
> > > fixInNamespace("[.data.table", pos="package:data.table")
> > >
> > >
> > > and change the following lines in the editor (this applies to
> > > data.table 1.7.7):
> > >
> > > OLD LINE: if (!mult %in% c("first", "last", "all")) stop("mult
> > > argument can only be 'first','last' or 'all'")
> > > NEW LINE: if (!mult %in% c("first","last","all", "random"))
> > > stop("mult argument can only be 'first','last', 'all', or 'random'")
> > >
> > >
> > > and
> > >
> > >
> > > OLD LINES: else {
> > > irows = if (mult == "first")
> > > idx.start
> > > else idx.end
> > > lengths = rep(1L, length(irows))
> > > }
> > >
> > >
> > > NEW LINES: } else if (mult=="first") {
> > > irows = idx.start
> > > lengths=rep(1L,length(irows))
> > > } else if (mult=="last") {
> > > irows = idx.end
> > > lengths=rep(1L,length(irows))
> > > } else {
> > > irows = mapply(function(x1, x2) {sample(x1:x2,
> > > size=1)}, idx.start, idx.end)
> > > lengths = rep(1L,length(irows))
> > > }
> > >
> > > However, I don't know what's going on in the line
> > > .Call("binarysearch", i, x, as.integer(leftcols -
> > > 1), as.integer(rightcols - 1), haskey(i), roll,
> > > rolltolast, idx.start, idx.end, PACKAGE =
> > > "data.table")
> > >
> > >
> > > I figured out that idx.start and idx.end are changed with this
> > > function call and I guess at this point in the function it should
> > > always be that idx.start and idx.end are of the same lenght and both
> > > return only integer values that represent rows of x, but here I'm
> > > not 100% sure. So maybe additional checks are needed in the else
> > > clause when the mapply-function is called.
> > >
> > >
> > > So let me know what you think. I will join the project independent
> > > of that particular issue and try to help, but I guess I should start
> > > with simple things. So if there is any help needed on documentation
> > > checking
> > > or stuff like that, just let me know and I try my best!
> > >
> > > Christoph
> > >
> > > On Fri, Jan 6, 2012 at 1:52 PM, Chris Neff <caneff at gmail.com> wrote:
> > > That isn't doing quite what he does. I don't know what you
> > > expected
> > >
> > > sample(dt, size=1)
> > >
> > > to do but it seems to essentially do this:
> > >
> > > dt[sample(1:ncol(dt),size=1),]
> > >
> > > It picking a random column number and then return that row
> > > instead.
> > > Try it for yourself:
> > >
> > > dt=data.table(x=1:10,y=1:10,z=1:10)
> > > sample(dt, size=1)
> > >
> > > The only rows you will get is 1,1,1 2,2,2 and 3,3,3. Caveat
> > > as usual
> > > is I'm on 1.7.1 until my crashing bug is fixed so apologies
> > > if this
> > > works properly in later versions.
> > >
> > > Note that this diverges from what sample(df, size=1) does,
> > > which is
> > > picks a random column and returns that whole column.
> > >
> > > What he really wants is to pick a random row from each
> > > subset (I
> > > think). None of your examples do that and I can't think of a
> > > simpler
> > > way than what he suggests.
> > >
> > > On 6 January 2012 03:34, Matthew Dowle
> > > <mdowle at mdowle.plus.com> wrote:
> > > > Very keen for direct contributions in that way, happy to
> > > help you with
> > > > svn etc, and you joining the project.
> > > >
> > > > In this particular example, how about :
> > > >
> > > > rawData[sample(rawData[J("eu"), which=TRUE],size=1)]
> > > >
> > > > This solves the inefficiency of the 1st step; i.e.,
> > > > intDT <- rawData[J("eu"), mult="all"]
> > > > which copies a subset of all the columns, whilst retaining
> > > flexibility
> > > > for the user so user can easily sample 2 rows, or any
> > > other R method to
> > > > select a random subset.
> > > >
> > > > Because of potential scoping conflicts (say a column was
> > > called
> > > > "rawData" i.e. the same name of the table), to be more
> > > robust :
> > > >
> > > > x = sample(rawData[J("eu"), which=TRUE],size=1)
> > > > rawData[x]
> > > >
> > > > This is slightly different because when i is a single name
> > > (x in this
> > > > case), data.table knows the caller must mean the x in
> > > calling scope, not
> > > > the column called "x" (if any). Is two steps like this
> > > ok? I'm
> > > > guessing it was really the inefficiency that was the
> > > motivation?
> > > >
> > > > Matthew
> > > >
> > > > On Fri, 2012-01-06 at 00:20 +0100, Christoph Jäckel wrote:
> > > >> Hi together,
> > > >>
> > > >>
> > > >> I run a Monte Carlo simulation on a data.table and do
> > > that currently
> > > >> with a loop: on every run, I choose a subset of rows
> > > subject to
> > > >> certain criteria and from those rows I take a random
> > > element.
> > > >> Currently, I do the following: Let's say I have funds
> > > from two regions
> > > >> ("eu" and "us") and I want to choose a random fund from
> > > "eu" (could be
> > > >> "us" in the next run and a different region in the
> > > third):
> > > >>
> > > >>
> > > >> library(data.table)
> > > >> rawData <- data.table(fundID = letters,
> > > >> compGeo = rep(c("us", "eu"),
> > > each=13))
> > > >> setkey(rawData, "compGeo")
> > > >> intDT <- rawData[J("eu"), mult="all"]
> > > >> intDT[sample.int(nrow(intDT), size=1)]
> > > >>
> > > >>
> > > >> So my idea is to just give the user the option
> > > mult="random", which
> > > >> does this in one step. What do you think about that
> > > feature request?
> > > >>
> > > >>
> > > >> With respect to the implementation: I changed a few lines
> > > in the
> > > >> function '[.data.table' and got this to run on my locale
> > > data.table
> > > >> version, so I guess I could implement it (as far as I can
> > > see, one
> > > >> just needs to change some R code). However, I haven't
> > > done extensive
> > > >> testing and I'm not an expert on shared projects and
> > > subversion (never
> > > >> did that actually), so I guess I would need some help to
> > > start with
> > > >> and the confirmation I couldn't break anything ;-)
> > > >>
> > > >>
> > > >> Christoph
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> _______________________________________________
> > > >> datatable-help mailing list
> > > >> datatable-help at lists.r-forge.r-project.org
> > > >>
> > >
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
> > > >
> > > >
> > > > _______________________________________________
> > > > datatable-help mailing list
> > > > datatable-help at lists.r-forge.r-project.org
> > > >
> > >
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > datatable-help mailing list
> > > datatable-help at lists.r-forge.r-project.org
> > >
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
> >
> > _______________________________________________
> > datatable-help mailing list
> > datatable-help at lists.r-forge.r-project.org
> >
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/datatable-help/attachments/20120108/fe74e123/attachment-0001.htm>
More information about the datatable-help
mailing list