[datatable-help] Coercian to character

Joseph Voelkel jgvcqa at rit.edu
Tue Apr 17 19:11:06 CEST 2012 

Personally, I am fine with having data.table read in character data as character. I don't find 'stringsAsFactors' particularly useful (although I use it sometimes)--I'll usually have a mix of character data, some of which I'd like as factors and some of which I'd like as character (or, to convert to dates, as your example notes). And some I'm just confused about--I don't like working with factors in general so I prefer to keep character data as character, but then all of a sudden I decide I want to run some character data in some models ...

I like the DT[,col:=factor(col)] approach. It also makes the intent clearer.

"But factor() re-orders levels ..." Yes. An extra step is required. But well worth it to have January as the first month of the year!

Joe

-----Original Message-----
From: Matthew Dowle [mailto:mdowle at mdowle.plus.com] 
Sent: Tuesday, April 17, 2012 12:26 PM
To: Joseph Voelkel
Cc: Damian Betebenner; datatable-help at lists.r-forge.r-project.org
Subject: RE: [datatable-help] Coercian to character


Yes but I invisage a work flow where data is loaded up as character always, then set any columns you'd like as factor to factor afterwards, using DT[,col:=factor(col)]. Other columns might load up as character that are actually dates or times (for example), then have a DT[,col:=as.IDate(col)] applied to them instead for example. The factor() operation is quite compute intensive (finding unique values, then matching all data to those unique values) so making the default TRUE to convert to
factor() [which may need to be converted afterwards to something other than factor, such as date] doesn't seem the right default from the efficiency perspective.
Another column type example is comma separated lists such as in genomics.
Those really do need to be character and then strsplit() afterwards, rather than a factor, unfactored and then strsplit. That's what the dual-delimited fast file loader will handle in data.table, when implemented.

> Not sure if this one will get resolved.

We could add the 'stringsAsFactors' argument to data.table(), then you could change it from FALSE to TRUE. Would that work?  The default would be
getOption("datatable.stringsAsFactors") so you could change it globally, just like base::default.stringsAsFactors().

> Using factors instead of characters also ensures that a table of 
> months or days of the week can be listed in the natural (not alphabetic) ordering.

But factor() re-orders levels, and so does stringsAsFactors=TRUE afaik doesn't it? You have to work a bit harder to get unordered factor levels, by passing labels to factor "manually".  That's very useful, important and supported in data.table (mentioned in last NEWS) but not the best default for data.table(), iiuc.

Matthew

> Statistical work in R (what many us of do, I'd say) prefers factors 
> over
> characters:
>
> # factors
> DF <- data.frame(X=letters[rep(1:5,2)], Y=rnorm(10))
> (DF.lm<-lm(Y~X,DF))
> predict(DF.lm)
> # all works fine
>
> # characters
> DF <- data.frame(X=letters[rep(1:5,2)], Y=rnorm(10),
> stringsAsFactors=FALSE)
> (DF.lm<-lm(Y~X,DF)) # warning
> predict(DF.lm) # warning
>
> Not sure if this one will get resolved.
>
> Using factors instead of characters also ensures that a table of 
> months or days of the week can be listed in the natural (not alphabetic) ordering.
>
> Joe
>
> -----Original Message-----
> From: datatable-help-bounces at lists.r-forge.r-project.org
> [mailto:datatable-help-bounces at lists.r-forge.r-project.org] On Behalf 
> Of Matthew Dowle
> Sent: Sunday, April 15, 2012 1:20 PM
> To: Damian Betebenner
> Cc: datatable-help at lists.r-forge.r-project.org
> Subject: Re: [datatable-help] Coercian to character
>
> I thought I'd added something to FAQ 2.17 about that, but seems not.
> Will add, thanks. Maybe I only wrote it up in the comment when closing 
> the related feature request. It's deliberately different since my 
> guess is that most people most of the time (now) want characters left 
> as characters and keep setting stringsAsFactors to FALSE. Think the 
> default for data.frame was TRUE as a hang over from old versions of R 
> before the global string cache was added.
>
> It's not set in stone though so could be changed. In particular there 
> could be global default like we've done for other arguments so you 
> could change the default if need be.
>
> It won't cause a compatibility issue (same as other differences in faq
> 2.17) or any issues down the road as far I can think, but let me know 
> if you think of anything.
>
> Matthew
>
> On Sun, 2012-04-15 at 04:40 -0500, Damian Betebenner wrote:
>> I started having character vectors popping up in places I never had 
>> before but upon further investigation that turned out to be an issue 
>> with my own setup, not data.table.
>>
>> With regard to characters (and data.tables ability to handle them as 
>> a key now), I did notice that data.table and data.frame default to 
>> using stringsAsFactors differently:
>>
>> DF <- data.frame(X=letters[1:10], Y=rnorm(10)) sapply(DF, class)
>>
>>         X         Y
>>  "factor" "numeric"
>>
>> DT <- data.table(X=letters[1:10], Y=rnorm(10)) sapply(DT, class)
>>
>> > DT <- data.table(X=rep(letters[1:10], each=2), Y=rnorm(20)) 
>> > sapply(DT, class)
>>           X           Y
>> "character"   "numeric"
>>
>>
>> Will this inconsistency cause problems down the road?
>>
>> Thanks for all your help,
>>
>> Damian
>>
>>
>> Damian Betebenner
>> Center for Assessment
>> PO Box 351
>> Dover, NH   03821-0351
>>
>> Phone (office): (603) 516-7900
>> Phone (cell): (857) 234-2474
>> Fax: (603) 516-7910
>>
>> dbetebenner at nciea.org
>> www.nciea.org
>>
>>
>>
>>
>> -----Original Message-----
>> From: Matthew Dowle [mailto:mdowlenoreply at virginmedia.com] On Behalf 
>> Of Matthew Dowle
>> Sent: Thursday, April 12, 2012 5:50 PM
>> To: Damian Betebenner
>> Cc: datatable-help at lists.r-forge.r-project.org
>> Subject: Re: [datatable-help] Coercian to character
>>
>> It shouldn't coerce. What makes you think it does?
>>
>> > DT = data.table(a=factor(c("a","b","b","c")),b=1:4)
>> > DT[,sum(b),by=a]
>>      a V1
>> [1,] a  1
>> [2,] b  5
>> [3,] c  4
>> > str(DT[,sum(b),by=a])
>> Classes ‘data.table’ and 'data.frame':	3 obs. of  2 variables:
>>  $ a : Factor w/ 3 levels "a","b","c": 1 2 3  $ V1: int  1 5 4
>>
>>
>>
>> On Thu, 2012-04-12 at 14:57 -0500, Damian Betebenner wrote:
>> > Data tablers
>> >
>> >
>> >
>> > Does data.table now coerce factors to character variables when 
>> > doing by summaries?
>> >
>> >
>> >
>> > If so, is there any way to not allow this coercion?
>> >
>> >
>> >
>> > Thanks,
>> >
>> >
>> >
>> > Damian Betebenner
>> >
>> > Center for Assessment
>> >
>> > PO Box 351
>> >
>> > Dover, NH   03821-0351
>> >
>> >
>> >
>> > Phone (office): (603) 516-7900
>> >
>> > Phone (cell): (857) 234-2474
>> >
>> > Fax: (603) 516-7910
>> >
>> >
>> >
>> > dbetebenner at nciea.org
>> >
>> > www.nciea.org
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > datatable-help mailing list
>> > datatable-help at lists.r-forge.r-project.org
>> > https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datata
>> > b
>> > le
>> > -help
>>
>>
>
>
> _______________________________________________
> datatable-help mailing list
> datatable-help at lists.r-forge.r-project.org
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable
> -help
>

More information about the datatable-help mailing list