<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
<html><body>
<p> </p>
<p>Good idea statquant, please file it then. How about something more general e.g.</p>
<p> fread(input, chunk.nrows=10000, chunk.filter = <anything acceptable to i of DT[i]>)</p>
<p>That <anything> could be grep() or any expression of column names. It wouldn't be efficient to call that for every row one by one and similarly couldn't be called for the whole DT, since the point is that DT is greater than RAM. So some batch size need be defined hence chunk.nrows=10000. That filter would then be called for each chunk and any rows passing would make it into the final table.</p>
<p>read.ffdf has something like this I believe, and Jens already suggested that when I ran the timings in example(fread) past him. We should probably follow his lead on that in terms of argument names etc.</p>
<p>Perhaps chunk should be defined in terms of RAM e.g. chunk=100MB. Since that is how it needs to be internally, in terms of number of pages to map. Or maybe both as nrows or MB would be acceptable.</p>
<p>Ultimately (maybe in 5 years!) we're heading towards fread reading into on-disk tables rather than RAM. Filtering in chunks will always be a good option to have though, even then, as you might want to filter what makes it to the on-disk table.</p>
<p>Matthew</p>
<p> </p>
<p>On 11.03.2013 12:53, MICHELE DE MEO wrote:</p>
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<div dir="ltr">Very interesting request. I also would be interested in this possibility.
<div>Cheers</div>
<div class="gmail_extra"><br /><br />
<div class="gmail_quote">2013/3/11 stat quant <span><<a href="mailto:statquant@outlook.com">statquant@outlook.com</a>></span><br />
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;">
<div>Hello list,</div>
<div>We like FREAD because it is very fast, yet sometimes files are huge and R cannot handle that much data, some packages handle this limitation but they do not provide a similar to fread function.</div>
<div>Yet sometimes only subsets of a file is really needed, subsets that could fit into RAM.</div>
<div> </div>
<div>So what about adding a grep option to fread that would allow to load only lines that matches a regular expression?</div>
<div> </div>
<div>I'll add a request if you think the idea is worth implementing.</div>
<div> </div>
<div>Cheers</div>
<div> </div>
<br />_______________________________________________<br /> datatable-help mailing list<br /><a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br /><a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a></blockquote>
</div>
<br /><br clear="all" />-- <br />
<div><em><strong><span style="font-size: xx-small;">*************************************************************</span></strong></em></div>
<div><em><strong><span style="font-size: xx-small;">Michele De Meo, Ph.D</span></strong></em></div>
<div><em><span style="font-size: xx-small;"><span style="line-height: 15px; font-family: Arial,Helvetica,;">Statistical and data mining solutions</span><br /><a style="color: #1c51a8;" href="http://micheledemeo.blogspot.com/">http://micheledemeo.<span style="background-color: #ffffcc; color: #222222;">blogspot</span>.com/</a><br /> skype: demeo.michele</span></em></div>
<div style="font-size: 15px;"><em><span></span><span></span><br /></em></div>
</div>
</div>
</blockquote>
<p> </p>
<div> </div>
</body></html>