<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
<html><body>
<p> </p>
<p>Fantastic, great job here, thanks!</p>
<p>One thing to note is that read.csv is much faster when using the standard tricks (colClasses, nrows etc). That's why the speed comparisons in ?fread are careful to link to online resources that list what the tricks are, and then compare read.csv both with and without them to fread. Of course the "friendly" part of fread is that you don't need to learn or know any tricks, so from that point of view it may well be fair to compare no-frills read.csv to fread as you've done. Good to state that so that nobody accuses of unfair comparisons. But even with the tricks applied, fread is still much faster. With-tricks on a compressed file would be interesting for completeness.</p>
<p>Thinking about it I suppose fread could read .gz directly. Difficult, but possible. For convenience if nothing else. I'll add it to the list to investigate ...</p>
<p>Matthew</p>
<p> </p>
<p>On 05.04.2013 19:59, Nathaniel Graham wrote:</p>
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<div dir="ltr">As promised, I did some testing. The results (described in detail below) are mixed, but suggest that compression is useful for some large data sets, and that if this is a serious issue for someone, they need to do some careful testing before committing to anything (I know, that should be obvious, but...). Also, my results pretty clearly show that fread() crushes read.csv, regardless of whether the csv file is compressed. Nice job Matthew!
<div>I start with Current Population Survey data from the Bureau of Labor Statistics.</div>
<div>The file I used get be accessed here: <a href="ftp://ftp.bls.gov/pub/time.series/ln/ln.data.1.AllData">ftp://ftp.bls.gov/pub/time.series/ln/ln.data.1.AllData</a></div>
<div>I converted it to a csv file using StatTransfer 8 (I'm lazy), with no quoting of strings. I then compressed the csv file using 7-Zip (gzip, Normal). The resulting</div>
<div>files, both with 4937221 obs, 5 variables are:</div>
<div>
<div>ln_data_1.csv : 133625 KB</div>
<div>ln_data_1.csv.gz : 17528 KB</div>
<div>Given the file size disparity, this should demonstrate any improvements via compression. Also, for comparison, I show fread below. I've made some</div>
<div>formatting changes, but changed nothing else.</div>
<div>
<div>for(i in 1:5) {</div>
<div> t1
<div> print(t1)</div>
<div>}</div>
<div> user system elapsed </div>
<div> 12.32 0.53 12.90 </div>
<div> 12.51 0.44 13.00 </div>
<div> 12.39 0.47 12.89 </div>
<div> 12.36 0.55 12.96 </div>
<div> 12.43 0.36 12.94 </div>
<div> </div>
<div>for(i in 1:5) {</div>
<div> t2
<div> print(t2)</div>
<div>}</div>
<div> user system elapsed </div>
<div> 14.04 0.26 14.43 </div>
<div> 14.00 0.27 14.34 </div>
<div> 14.07 0.31 14.44 </div>
<div> 13.93 0.28 14.23 </div>
<div> 14.02 0.32 14.35 </div>
<div> </div>
<div>for(i in 1:5) {</div>
<div> t3
<div> print(t3)</div>
<div>}</div>
<div> user system elapsed </div>
<div> 2.89 0.04 2.94 </div>
<div> 2.92 0.07 2.98 </div>
<div> 2.88 0.03 2.95 </div>
<div> 2.87 0.06 2.95 </div>
<div> 2.91 0.03 2.95 </div>
<div>While the gzipped version uses less system time, total & user time has increased somewhat. The fread function from data.table is dramatically</div>
<div>faster. While this isn't strictly a fair comparison because fread produces</div>
<div>a data.table while read.csv produces a data.frame, the bias is against fread,</div>
<div>not for it.</div>
<div>Next, I produce a random 2,000,000x10 matrix, write it to csv, and then read it back into memory as a data.frame (or data.table, for fread). I again use 7-Zip for compression.The resulting files are:</div>
<div>test2.csv : 375086 KB</div>
<div>test2.csv.gz : 165477 KB</div>
<div>
<div>> matr
<div>> write.csv(matr,"test2.csv")</div>
<div>> t1
<div>> t2
<div>> t3
<div> </div>
<div>> t1</div>
<div> user system elapsed </div>
<div> 165.32 0.36 166.25 </div>
<div>> t2</div>
<div> user system elapsed </div>
<div> 116.24 0.16 117.08 </div>
<div>> t3</div>
<div> user system elapsed </div>
<div> 17.64 0.06 17.83 </div>
<div>The switch to strictly floating point numbers is significant. Compression is significant improvement--about 49 seconds or about 30%--although nowhere near enough for read.csv to be comparable to fread.</div>
</div>
<div>Finally, I produce a 20000x1000 matrix. The resulting files are:</div>
<div>test1.csv : 354854 KB</div>
<div>test1.csv.gz : 157975 KB</div>
<div>
<div>matr
<div>> write.csv(matr,"test1.csv")</div>
<div>> t1
<div>> t2
<div>> t3
<div>> t1</div>
<div> user system elapsed </div>
<div> 206.80 1.14 208.60 </div>
<div>> t2</div>
<div> user system elapsed </div>
<div> 123.42 0.27 123.99 </div>
<div>> t3</div>
<div> user system elapsed </div>
<div> 17.24 0.09 17.37 </div>
</div>
<div>Here, compression is an even larger win, improving by about 83 seconds or roughly 40%. The fread function is again dramatically faster, and unlike read.csv, fread's performance is similar regardless of the shape of the matrix.</div>
<div>We could create more detailed tests, varying the number of columns vs rows</div>
<div>and their type (strings vs integers vs floats, etc) to get better details, but the</div>
<div>basic result is that compression can be a noticeable improvement in performance, but a superior read algorithm trumps that. If it's feasible to</div>
<div>combine fread's behavior with gzip, bzip2, or xz compression, it could be a</div>
<div>big win for some files, but not for all of them. The advice from</div>
<div><a href="http://blog.revolutionanalytics.com/2009/12/r-tip-save-time-and-space-by-compressing-data-files.html">http://blog.revolutionanalytics.com/2009/12/r-tip-save-time-and-space-by-compressing-data-files.html</a> to compress csv files appears to hold, although</div>
<div>it may not save much time if you have a lot of non-float values or few columns.</div>
</div>
</div>
</div>
<div class="gmail_extra"><br clear="all" />
<div>-------<br />Nathaniel Graham<br /><a href="mailto:npgraham1@gmail.com">npgraham1@gmail.com</a><br /><a href="mailto:npgraham1@uky.edu">npgraham1@uky.edu</a></div>
<br /><br />
<div class="gmail_quote">On Wed, Apr 3, 2013 at 4:20 PM, Nathaniel Graham <span><<a href="mailto:npgraham1@gmail.com">npgraham1@gmail.com</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;">
<div dir="ltr">Subjectively, the difference seems substantial, with large loads taking
<div>half or a third as long. Whether I use gzip or not, CPU usage isn't</div>
<div>especially high, suggesting that I'm either waiting on the hard drive</div>
<div>or that the whole process is memory bound. I was all set to produce</div>
<div>some timings for comparison, but I'm working from home today and</div>
<div>my home machine struggles to accommodate large files---any difference</div>
<div>in load times gets swamped by swapping and general flailing on the</div>
<div>part of the OS (I've only got 4GB of RAM at home). Hopefully I'll get</div>
<div>around to doing some timings on my work machine sometime this</div>
<div>week, since I've got no issues with memory there.</div>
</div>
<div class="gmail_extra">
<div class="im"><br clear="all" />
<div>-------<br />Nathaniel Graham<br /><a href="mailto:npgraham1@gmail.com">npgraham1@gmail.com</a><br /><a href="mailto:npgraham1@uky.edu">npgraham1@uky.edu</a></div>
<br /><br /></div>
<div>
<div class="h5">
<div class="gmail_quote">On Wed, Apr 3, 2013 at 4:58 AM, Matthew Dowle <span><<a href="mailto:mdowle@mdowle.plus.com">mdowle@mdowle.plus.com</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><span style="text-decoration: underline;"></span>
<div>
<p> </p>
<p>Interesting. How much do you find read.csv is sped up by reading gzip'd files?</p>
<div>
<div>
<p> </p>
<p>On 02.04.2013 20:36, Nathaniel Graham wrote:</p>
<blockquote style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
<div dir="ltr">Thanks, but I suspect that it would take longer to setup and then remove
<div>a ramdisk than it would to use read.csv and data.table. My files are</div>
<div>moderately large (between 200 MB and 3 GB when compressed), but not </div>
<div>enormous; I gzip not so much to save space on disk but to speed up reads.</div>
</div>
<div class="gmail_extra"><br clear="all" />
<div>-------<br />Nathaniel Graham<br /><a href="mailto:npgraham1@gmail.com">npgraham1@gmail.com</a><br /><a href="mailto:npgraham1@uky.edu">npgraham1@uky.edu</a></div>
<br /><br />
<div class="gmail_quote">On Tue, Apr 2, 2013 at 3:12 PM, Matthew Dowle <span><<a href="mailto:mdowle@mdowle.plus.com">mdowle@mdowle.plus.com</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><span style="text-decoration: underline;"></span>
<div>
<p> </p>
<p>Hi,</p>
<p>fread memory maps the entire uncompressed file and this is baked into the way it works (e.g. skipping to the beginning, middle and last 5 rows to detect column types before starting to read the rows in) and where the convenience and speed comes from.</p>
<p>You could uncompress the .gz to a ramdisk first, and then fread the uncompressed file from that ramdisk, is probably the fastest way. Which should still be pretty quick and I guess unlikely much slower than anything we could build into fread (provided you use a ramdisk).</p>
<p>Matthew</p>
<div>
<div>
<p> </p>
<p>On 02.04.2013 19:30, Nathaniel Graham wrote:</p>
<blockquote style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
<div dir="ltr">I have a moderately large csv file that's gzipped, but not in a tar
<div>archive, so it's "filename.csv.gz" that I want to read into a data.table.</div>
<div>I'd like to use fread(), but I can't seem to make it work. I'm currently</div>
<div>using the following:</div>
<div>data.table(read.csv(gzfile("filename.csv.gz","r")))</div>
<div>Various combinations of gzfile, gzcon, file, readLines, and</div>
<div>textConnection all produce an error (invalid input). Is there a better</div>
<div>way to read in large, compressed files?</div>
<div>
<div>
<div>-------<br />Nathaniel Graham<br /><a href="mailto:npgraham1@gmail.com">npgraham1@gmail.com</a><br /><a href="mailto:npgraham1@uky.edu">npgraham1@uky.edu</a></div>
</div>
</div>
</div>
</blockquote>
<p> </p>
<div> </div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<p> </p>
<div> </div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p> </p>
<div> </div>
</body></html>