[datatable-help] fread from RAM DIsk

[Raj Dasgupta]

Hi all,

I have been looking at lowering the time spent in I/O while using fread on a csv file. Following a suggestion on the mailing list, I attempted to use fread on a csv file stored on a ramdisk. It took 5 times longer to read from the Ram Disk than it did to read from the SSD.

I presume this has to do with the fact that the data is being copied from RAM to RAM rather than from Disk to RAM.

Any suggestions on if there are alternative methods to read files in faster using a RAM Disk would be very helpful.

Benchmarks on time taken for reading from SSD vs Ramdisk on the same 416MB file is given below

####

> timer = proc.time(); z <- fread("testInRAM.csv"); proc.time() - timer
   user  system elapsed 
 25.067   0.433  25.485  ##### Read from RAMDisk

> setwd("/Users/xbsd/")
> timer = proc.time(); z <- fread("test.csv"); proc.time() - timer
   user  system elapsed 
  5.507   0.177   5.680   ###### Read from SSD

> system("ls -alh test.csv")
-rw-r--r--  1 xbsd  staff   416M Jul 12 19:30 test.csv

> system("ls -alh /Volumes/ramdisk/testInRAM.csv")
-rw-r--r--  1 xbsd  staff   416M Jul 12 19:32 /Volumes/ramdisk/testInRAM.csv

Thanks in advance,

- Raj.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/datatable-help/attachments/20130712/818038b0/attachment.html>