<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
<html><body>
<p> </p>
<p>Interesting. How much do you find read.csv is sped up by reading gzip'd files?</p>
<p> </p>
<p>On 02.04.2013 20:36, Nathaniel Graham wrote:</p>
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<div dir="ltr">Thanks, but I suspect that it would take longer to setup and then remove
<div>a ramdisk than it would to use read.csv and data.table. My files are</div>
<div>moderately large (between 200 MB and 3 GB when compressed), but not </div>
<div>enormous; I gzip not so much to save space on disk but to speed up reads.</div>
</div>
<div class="gmail_extra"><br clear="all" />
<div>-------<br />Nathaniel Graham<br /><a href="mailto:npgraham1@gmail.com">npgraham1@gmail.com</a><br /><a href="mailto:npgraham1@uky.edu">npgraham1@uky.edu</a></div>
<br /><br />
<div class="gmail_quote">On Tue, Apr 2, 2013 at 3:12 PM, Matthew Dowle <span><<a href="mailto:mdowle@mdowle.plus.com">mdowle@mdowle.plus.com</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><span style="text-decoration: underline;"></span>
<div>
<p> </p>
<p>Hi,</p>
<p>fread memory maps the entire uncompressed file and this is baked into the way it works (e.g. skipping to the beginning, middle and last 5 rows to detect column types before starting to read the rows in) and where the convenience and speed comes from.</p>
<p>You could uncompress the .gz to a ramdisk first, and then fread the uncompressed file from that ramdisk, is probably the fastest way. Which should still be pretty quick and I guess unlikely much slower than anything we could build into fread (provided you use a ramdisk).</p>
<p>Matthew</p>
<div>
<div class="h5">
<p> </p>
<p>On 02.04.2013 19:30, Nathaniel Graham wrote:</p>
<blockquote style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
<div dir="ltr">I have a moderately large csv file that's gzipped, but not in a tar
<div>archive, so it's "filename.csv.gz" that I want to read into a data.table.</div>
<div>I'd like to use fread(), but I can't seem to make it work. I'm currently</div>
<div>using the following:</div>
<div>data.table(read.csv(gzfile("filename.csv.gz","r")))</div>
<div>Various combinations of gzfile, gzcon, file, readLines, and</div>
<div>textConnection all produce an error (invalid input). Is there a better</div>
<div>way to read in large, compressed files?</div>
<div>
<div>
<div>-------<br />Nathaniel Graham<br /><a href="mailto:npgraham1@gmail.com">npgraham1@gmail.com</a><br /><a href="mailto:npgraham1@uky.edu">npgraham1@uky.edu</a></div>
</div>
</div>
</div>
</blockquote>
<p> </p>
<div> </div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<p> </p>
<div> </div>
</body></html>