<div dir="ltr">I'd love to test it, pulled the latest commit with svn, not sure about building from source on windows, got some compilation errors:<div><br></div><div><div>> install.packages("pkg/",type="source",repos=NULL)</div>
<div>Warning in install.packages :</div><div> package ‘pkg/’ is not available (for R version 3.0.0)</div><div>* installing *source* package 'data.table' ...</div><div>** libs</div><div>gcc -m64 -I"C:/Users/PAUL~1.HAR/R/R-30~1.0/include" -DNDEBUG -I"d:/RCompile/CRANpkg/extralibs64/local/include" -O2 -Wall -std=gnu99 -mtune=core2 -c fread.c -o fread.o</div>
<div>fread.c: In function 'readfile':</div><div>fread.c:343:9: error: 'hfile' undeclared (first use in this function)</div><div>fread.c:343:9: note: each undeclared identifier is reported only once for each function it appears in</div>
<div>fread.c:346:115: error: expected ';' before ')' token</div><div>fread.c:346:115: error: expected statement before ')' token</div><div>fread.c:350:17: warning: implicit declaration of function 'nanosleep' [-Wimplicit-function-declaration]</div>
<div>make: *** [fread.o] Error 1</div><div>ERROR: compilation failed for package 'data.table'</div></div><div><br></div><div style>Regards</div><div style>Paul</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On 11 May 2013 02:39, Matthew Dowle <span dir="ltr"><<a href="mailto:mdowle@mdowle.plus.com" target="_blank">mdowle@mdowle.plus.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<u></u>
<div>
<p> </p>
<p>Paul, Vishal,</p>
<p>Commit 859 :</p>
<pre>* fread now supports files larger than 4GB on 64bit Windows (#2767 thanks to Paul Harding) and files<br> between 2GB and 4GB on 32bit Windows (#2655 thanks to Vishal). A C call to GetFileSize() needed to<br> be GetFileSizeEx().</pre>
<pre> </pre>
<pre>Please test and confirm ok now.</pre>
<pre> </pre>
<pre>Thanks, Matthew</pre><div><div class="h5">
<pre> </pre>
<p>On 03.05.2013 14:59, Matthew Dowle wrote:</p>
<blockquote type="cite" style="padding-left:5px;border-left:#1010ff 2px solid;margin-left:5px;width:100%">
<p> </p>
<p>Oh. Then it's likely a bug with fread on Windows for files > 4GB. Think GetFileSize() should be GetFileSizeEx(), iirc.</p>
<p>Please could you file it as a bug on the tracker. Thanks.</p>
<p>Matthew</p>
<p> </p>
<p>On 03.05.2013 14:32, Paul Harding wrote:</p>
<blockquote style="padding-left:5px;border-left:#1010ff 2px solid;margin-left:5px;width:100%">
<div dir="ltr">Definitely a 64-bit machine. Here are the details:
<div><br>
<div>Processor: Intel Xeon CPU E7-4830 @2.13GHz (4 processors)</div>
<div>Installed memory (RAM): 128GB</div>
<div>System type: 64-bit Operating System</div>
<div>Windows edition: Server 2008 R2 Enterprise SP1</div>
</div>
<div>Regards,</div>
<div>Paul</div>
</div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">On 3 May 2013 10:51, Matthew Dowle <span><<a href="mailto:mdowle@mdowle.plus.com" target="_blank">mdowle@mdowle.plus.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span style="text-decoration:underline"></span>
<div>
<p> </p>
<p>Hi Paul,</p>
<p>Thanks for all this!</p>
<div>
<p>> The problem arises when the file reaches 4GB, in this case between 8,030,000 and 8,040,000 rows:</p>
</div>
<p>Ahah. Are you using a 32bit or 64bit Windows machine?</p>
<p>Thanks, Matthew</p>
<div>
<div>
<p> </p>
<p>On 02.05.2013 10:19, Paul Harding wrote:</p>
<blockquote style="padding-left:5px;border-left:#1010ff 2px solid;margin-left:5px;width:100%">
<div dir="ltr">Some supplementary information, here is the portion of the file (with row numbers, +1 for header) around where fread thinks the file ends. <br>
<div class="gmail_quote">
<div dir="ltr">
<div>
<div>$ nl spd_all_fixed.csv | head -n 9186300 |tail</div>
<div>9186291 204029,2617097,20110803,0,0,0.3014501,0,0,0</div>
<div>9186292 204030,2617097,20110803,0,0,0.52049100000000004,0,0,0</div>
<div>9186293 204034,2617097,20110803,0,0,0.86560269999999995,0.86560269999999995,2,13</div>
<div>9186294 204038,2617097,20110803,0,0,0.49455500000000002,0,0,0</div>
<div>9186295 204039,2617097,20110803,0,0,0.24952240000000001,0,0,0</div>
<div>9186296 204041,2617097,20110803,1,0,1.0032293000000001,0,0,0</div>
<div>9186297 204042,2617097,20110803,0,0,0.1375876,0,0,0</div>
<div>9186298 204043,2617097,20110803,0,0,0.53391279999999997,0,0,0</div>
<div>9186299 204044,2617097,20110803,0,0,0.16047169999999999,0,0,0</div>
<div>9186300 204045,2617097,20110803,1,0,0.78766970000000003,0,0,0</div>
</div>
<div>9186294 (row 9186293 excl header) is where fread thinks the file ends, mid-line by the look of it!</div>
<div>I've experimented by truncating the file. The error varies, either it reads too few records or gives the error I reported, presumably determined by whether the last perceived line is entire.</div>
<div>The problem arises when the file reaches 4GB, in this case between 8,030,000 and 8,040,000 rows:</div>
<div>
<div>-rw-r--r--+ 1 Paul.Harding Domain Users 4.0G May 1 12:02 spd_all_trunc_8030k.csv</div>
<div>-rw-r--r--+ 1 Paul.Harding Domain Users 4.1G May 1 12:06 spd_all_trunc_8040k.csv</div>
<div>
<div>> dt<-fread("data/spd_all_trunc_8030k.csv", sep=",",verbose=T)</div>
<div>
<div>Detected eol as \r\n (CRLF) in that order, the Windows standard.</div>
<div>Looking for supplied sep ',' on line 30 (the last non blank line in the first 30) ... found</div>
<div>Found 9 columns</div>
<div>First row with 9 fields occurs on line 1 (either column names or first row of data)</div>
<div>All the fields on line 1 are character fields. Treating as the column names.</div>
</div>
<div>Count of eol after first data row: 80300000</div>
<div>Subtracted 1 for last eol and any trailing empty lines, leaving 80299999 data rows</div>
<div>
<div>Type codes: 000002000 (first 5 rows)</div>
</div>
<div>Type codes: 000002000 (+middle 5 rows)</div>
<div>Type codes: 000002000 (+last 5 rows)</div>
<div>0%Bumping column 7 from INT to INT64 on data row 9, field contains '0.42634430000000001'</div>
<div>Bumping column 7 from INT64 to REAL on data row 9, field contains '0.42634430000000001'</div>
<div> 0.000s ( 0%) Memory map (rerun may be quicker)</div>
<div> 0.000s ( 0%) Sep and header detection</div>
<div> 0.000s ( 0%) Count rows (wc -l)</div>
<div> 0.000s ( 0%) Colmn type detection (first, middle and last 5 rows)</div>
<div> 0.000s ( 0%) Allocation of 80299999x9 result (xMB) in RAM</div>
<div> 171.188s ( 65%) Reading data</div>
<div>1365231.809s (518439%) Allocation for type bumps (if any), including gc time if triggered</div>
<div>-1365231.809s (-518439%) Coercing data already read in type bumps (if any)</div>
<div> 0.000s ( 0%) Changing na.strings to NA</div>
<div> 0.000s Total</div>
<div>> dt<-fread("data/spd_all_trunc_8040k.csv", sep=",",verbose=T)</div>
<div>
<div>Detected eol as \r\n (CRLF) in that order, the Windows standard.</div>
<div>Looking for supplied sep ',' on line 30 (the last non blank line in the first 30) ... found</div>
<div>Found 9 columns</div>
<div>First row with 9 fields occurs on line 1 (either column names or first row of data)</div>
<div>All the fields on line 1 are character fields. Treating as the column names.</div>
</div>
<div>Count of eol after first data row: 18913</div>
<div>Subtracted 0 for last eol and any trailing empty lines, leaving 18913 data rows</div>
<div>
<div>Type codes: 000002000 (first 5 rows)</div>
</div>
<div>Type codes: 000002000 (+middle 5 rows)</div>
<div>Error in fread("data/spd_all_trunc_8040k.csv", sep = ",", verbose = T) : </div>
<div> Expected sep (',') but ',' ends field 2 on line 6 when detecting types: 204650,724540,</div>
</div>
</div>
<div>Regards,</div>
<div>Paul</div>
</div>
<div>
<div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">On 1 May 2013 10:28, Paul Harding <span><<a href="mailto:p.harding@paniscus.com" target="_blank">p.harding@paniscus.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Here is the verbose output:
<div>
<div>> dt<-fread("data/spd_all_fixed.csv", sep=",",verbose=T)</div>
<div>Detected eol as \r\n (CRLF) in that order, the Windows standard.</div>
<div>Looking for supplied sep ',' on line 30 (the last non blank line in the first 30) ... found</div>
<div>Found 9 columns</div>
<div>First row with 9 fields occurs on line 1 (either column names or first row of data)</div>
<div>All the fields on line 1 are character fields. Treating as the column names.</div>
<div>Count of eol after first data row: 9186293</div>
<div>Subtracted 0 for last eol and any trailing empty lines, leaving 9186293 data rows</div>
<div>Type codes: 000002000 (first 5 rows)</div>
<div>Type codes: 000002200 (+middle 5 rows)</div>
<div>Error in fread("data/spd_all_fixed.csv", sep = ",", verbose = T) : </div>
<div>
<div> Expected sep (',') but '0' ends field 5 on line 6 when detecting types: 204038,2617097,20110803,0,0</div>
</div>
</div>
<div>But here is the wc output (via cygwin; newline, word (whitespace delim so each word one 'line' here), byte)@</div>
<div>
<div>$ wc spd_all_fixed.csv</div>
<div> 168997637 168997638 9078155125 spd_all_fixed.csv</div>
<div>[So fread 9M, wc 168M rows].</div>
<div>Regards</div>
<div>Paul</div>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">On 30 April 2013 18:52, Matthew Dowle <span><<a href="mailto:mdowle@mdowle.plus.com" target="_blank">mdowle@mdowle.plus.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span style="text-decoration:underline"></span>
<div>
<p> </p>
<p>Hi,</p>
<p>Thanks for reporting this. Please set verbose=TRUE and let us know the output.</p>
<p>Thanks, Matthew</p>
<div>
<div>
<p> </p>
<p>On 30.04.2013 18:01, Paul Harding wrote:</p>
<blockquote style="padding-left:5px;border-left:#1010ff 2px solid;margin-left:5px;width:100%">
<div dir="ltr">
<div>Problem with fread on a large file</div>
The file is 8GB, just short of 200,000 lines, produced as SQLoutput and modified by cygwin/perl to remove the second line.<br>
<div class="gmail_quote">
<div dir="ltr">
<div>Using data.table 1.8.8 on R3.0.0 I get an fread error</div>
<div>
<div>fread("data/spd_all_fixed.csv",sep=",")</div>
<div>Error in fread("data/spd_all_fixed.csv", sep = ",") : </div>
<div> Expected sep (',') but '0' ends field 5 on line 6 when detecting types: 204038,2617097,20110803,0,0</div>
<div>Looking for the offending line,with line numbers in output so I'm guessing this is line 6 of the mid-file chunk examined,</div>
<div>
<div>$ grep -n '204038,2617097,201108' spd_all_fixed.csv</div>
<div>8316105:204038,2617097,20110801,0,0,0.64220529999999998,0,0,0</div>
<div>8751106:204038,2617097,20110802,1,0,0.65744469999999999,0,0,0</div>
<div>9186294:204038,2617097,20110803,0,0,0.49455500000000002,0,0,0</div>
<div>9621619:204038,2617097,20110804,0,0,0.3461342,0,0,0</div>
<div>10057189:204038,2617097,20110805,0,0,0.34128710000000001,0,0,0</div>
<div>and comparing to surrounding lines and the first ten lines</div>
<div>
<div>$ head spd_all_fixed.csv</div>
<div>s_key,i_key,p_key,q,pq,d,l,epi,class</div>
<div>203974,1107181,20110713,0,0,0.13700080000000001,0,0,0</div>
<div>203975,1107181,20110713,0,0,5.8352899999999999E-2,0,0,0</div>
<div>203976,1107181,20110713,0,0,7.1298999999999998E-3,0,0,0</div>
<div>203978,1107181,20110713,0,0,0.78346819999999995,0,0,0</div>
<div>203979,1107181,20110713,0,0,0.61627779999999999,0,0,0</div>
<div>203981,1107181,20110713,1,0,0.38610509999999998,0,0,0</div>
<div>203982,1107181,20110713,0,0,4.0657899999999997E-2,0,0,0</div>
<div>203983,1107181,20110713,2,0,0.71278109999999995,0,0,0</div>
<div>203984,1107181,20110713,0,0,0.42634430000000001,0.42634430000000001,2,13</div>
<div>I can't see any difference. I wonder if this is a bug? I have no problems on a small test data set run through an identical process and using the same fread command.</div>
<div>Regards</div>
<div>Paul</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p> </p>
<div> </div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p> </p>
<div> </div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<p> </p>
<div> </div>
</blockquote>
<p> </p>
<div> </div>
</div></div></div>
</blockquote></div><br></div>