<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">What’s the issue here? It seems to have taken ~4 seconds IIUC. The problem seems that your file has a “?” at the line denoted, which results in having to coerce all the lines read previously to character type first. Handling ‘na.strings’ is on the list - <a href="https://github.com/Rdatatable/data.table/issues/504">https://github.com/Rdatatable/data.table/issues/504</a> but I don’t get as to why it’s choking.. 4 seconds isn’t a lot, really.</div> <br> <div id="bloop_sign_1426528356918062080" class="bloop_sign"><div style="font-family:helvetica,arial;font-size:13px">-- <br>Arun</div></div> <br><p style="color:#000;">On 10 Mar 2015 at 15:13:20, Rivo R (<a href="mailto:rivokl@gmail.com">rivokl@gmail.com</a>) wrote:</p> <blockquote type="cite" class="clean_bq"><span><div><div></div><div>Hi all,<br><br>I tried to load the following (huge) dataset as data.table but fread<br>seems to choke on NA's.<br>https://d396qusza40orc.cloudfront.net/exdata%2Fdata%2Fhousehold_power_consumption.zip<br><br>Steps:<br>1- Dowload and unzip<br>2-<br>> packageVersion("data.table")<br>[1] ‘1.9.4’<br>3-<br><br>> tmp <- fread(dataFile, sep=';', header=TRUE, na.strings=c("NA","'?'", ""),<br>+ stringsAsFactors=FALSE,<br>+ colClasses=c(rep("character",2), rep("numeric",7)), verbose=TRUE)<br>Input contains no \n. Taking this to be a filename to open<br>File opened, filesize is 0.121897 GB.<br>Memory mapping ... ok<br>Detected eol as \n only (no \r afterwards), the UNIX and Mac standard.<br>Looking for supplied sep ';' on line 30 (the last non blank line in<br>the first 'autostart') ... found ok<br>Found 9 columns<br>First row with 9 fields occurs on line 1 (either column names or first<br>row of data)<br>'header' changed by user from 'auto' to TRUE<br>Count of eol after first data row: 2075260<br>Subtracted 1 for last eol and any trailing empty lines, leaving<br>2075259 data rows<br>Type codes ( first 5 rows): 443333333<br>Type codes (+ middle 5 rows): 443333333<br>Type codes (+ last 5 rows): 443333333<br>Type codes: 443333333 (after applying colClasses and integer64)<br>Type codes: 443333333 (after applying drop or select (if supplied)<br>Allocating 9 column slots (9 - 0 dropped)<br>Bumping column 3 from REAL to STR on data row 6840, field contains '?'<br>Bumping column 4 from REAL to STR on data row 6840, field contains '?'<br>Bumping column 5 from REAL to STR on data row 6840, field contains '?'<br>Bumping column 6 from REAL to STR on data row 6840, field contains '?'<br>Bumping column 7 from REAL to STR on data row 6840, field contains '?'<br>Bumping column 8 from REAL to STR on data row 6840, field contains '?'<br>Read 2075259 rows and 9 (of 9) columns from 0.122 GB file in 00:00:04<br> 0.000s ( 0%) Memory map (rerun may be quicker)<br> 0.001s ( 0%) sep and header detection<br> 0.282s ( 7%) Count rows (wc -l)<br> 0.002s ( 0%) Column type detection (first, middle and last 5 rows)<br> 0.627s ( 16%) Allocation of 2075259x9 result (xMB) in RAM<br> 2.525s ( 64%) Reading data<br> 0.298s ( 8%) Allocation for type bumps (if any), including gc time<br>if triggered<br> 0.123s ( 3%) Coercing data already read in type bumps (if any)<br> 0.059s ( 2%) Changing na.strings to NA<br> 3.917s Total<br><br>Any hint??<br>Kely<br>_______________________________________________<br>datatable-help mailing list<br>datatable-help@lists.r-forge.r-project.org<br>https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</div></div></span></blockquote></body></html>