[datatable-help] New function fread() in v1.8.7

[Matthew Dowle]

Hi,

Ah yes, haven't hooked up the sep override yet, apologies, will fix.
Maybe setting autostart to the row number of the header row (probably 
1)
might work.

Thanks,
Matthew


On 24.12.2012 11:08, Hideyoshi Maeda wrote:
> oups…forgot to add the output from the verbose part…here it is...
>
> Detected eol as \r\n (CRLF) in that order, the Windows standard.
> Starting format detection on line 30 (the last non blank line in the
> first 30)
> Detected sep as '/' and 3 columns
> Type codes: 003
> Found first row with 3 fields occuring on line 1 (either column names
> or first row of data)
> The first data row has some non character fields. Treating as a data
> row and using default column names.
> Count of eol after pos: 1143699
> Subtracted 1 for last eol and any trailing empty lines, leaving
> 1143698 data rows
>    0.153s ( 21%) Memory map (quicker if you rerun)
>    0.000s (  0%) Format detection
>    0.095s ( 13%) Count rows (wc -l)
>    0.001s (  0%) Allocation of 1143698x3 result (xMB) in RAM
>    0.480s ( 66%) Reading data
>    0.000s (  0%) Bumping column type midread and coercing data 
> already read
>    0.002s (  0%) Changing na.strings to NA
>    0.731s        Total
>
>
> On 24 Dec 2012, at 11:04, Hideyoshi Maeda <hideyoshi.maeda at gmail.com> 
> wrote:
>
>> Hi Matthew,
>>
>> I am using the new `data.table` `fread()` function to read my csv 
>> files, which has the format as follows when using the read.csv 
>> function
>>
>>            Date.and.Time Open High  Low Close Volume
>>    1 2007/01/01 22:51:00 5683 5683 5673  5673     64
>>    2 2007/01/01 22:52:00 5675 5676 5674  5674     17
>>    3 2007/01/01 22:53:00 5674 5674 5673  5674     42
>>
>> The value of the first column is all of: `2007/01/01 22:53:00`, the 
>> next 5 columns are separated with commas.
>>
>> but when reading the same file using fread i get the following 
>> output
>>
>>        V1 V2                                             V3
>>    1 2007  1 01 22:51:00,5683.00,5683.00,5673.00,5673.00,64
>>    2 2007  1 01 22:52:00,5675.00,5676.00,5674.00,5674.00,17
>>    3 2007  1 01 22:53:00,5674.00,5674.00,5673.00,5674.00,42
>>
>> This is because the autodetect is using the "/" as a separator...
>>
>> I tried overriding this using the `sep=","` argument but this does 
>> not seem to be used in the function anywhere.
>>
>> Furthremore when using verbose I get the following output, which 
>> suggests that I was right in thinking that "/" is used as a separator 
>> rather than ",".
>>
>> Is there any way to fix this, so that it correctly reads all 6 
>> columns separately?
>>
>> Thanks
>>
>> HLM
>>
>> On 21 Dec 2012, at 18:28, Matthew Dowle <mdowle at mdowle.plus.com> 
>> wrote:
>>
>>>
>>> Hi datatablers,
>>>
>>> Feedback and bug reports much appreciated :
>>>
>>> =====
>>> New function fread(), a fast and friendly file reader.
>>> * header, skip, nrows, sep and colClasses are all auto detected.
>>> * integers>2^31 are detected and read natively as bit64::integer64.
>>> * accepts filenames, URLs and "A,B\n1,2\n3,4" directly
>>> * new implementation entirely in C
>>> * with a 50MB .csv, 1 million rows x 6 columns :
>>>   read.csv("test.csv")                                   # 30-60 
>>> sec
>>>   read.table("test.csv",<all known tricks, known nrows>) #    10 
>>> sec
>>>   fread("test.csv")                                      #     3 
>>> sec
>>> * airline data: 658MB csv (7 million rows x 29 columns)
>>>   read.table("2008.csv",<all known tricks, known nrows>) #   360 
>>> sec
>>>   fread("2008.csv")                                      #    50 
>>> sec
>>> See ?fread. Many thanks to Chris Neff and Garrett See for ideas,
>>> discussions and beta testing.
>>> =====
>>>
>>> 1.8.7 is passing checks on Unix and Windows (but not Mac yet) :
>>>
>>> install.packages("data.table", 
>>> repos="http://R-Forge.R-project.org")
>>> require(data.table)
>>> ?fread
>>> fread("your biggest baddest file")
>>>
>>> Oddly, R-Forge appears to be compiling Win64 with -O2 optimization 
>>> rather
>>> than -O3 (but -O3 on Win32 ok), so speedups might not be as great 
>>> on Win64
>>> until that can be resolved on R-Forge, unless you compile yourself. 
>>> -O3
>>> has some optimizations that fread may benefit from. But interested 
>>> to hear.
>>>
>>> Seasons greatings!
>>>
>>> Matthew
>>>
>>>
>>> _______________________________________________
>>> datatable-help mailing list
>>> datatable-help at lists.r-forge.r-project.org
>>> 
>>> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
>>