[datatable-help] whole data.table copied "warning"

statquant3 statquant at outlook.com
Sun Jan 25 12:33:47 CET 2015


I am experiencing a problem with data.table and can't find out what the
problem is.
I am getting the warning "this data table had to be copied over"
First i cannot reproduce the bug on small example so I realize this post is
mostly a bottle thrown at the see...

Here is what I do:

1.
I have several tables on a distant kdb server that I retrieve through a
proprietary package.
Those tables are copied and some data.frames are created. (I think it as
nothing to do with it but still you never know) 
Because I have several tables on that server I created a function wrapper
the "fetching" of a table, a data.frame is created and 
I setDT them the make them a data.table.
2.
I call this function within a sapply command and end up with a named-list of
data.table
3.
I modify some of those tables in a R function, once again using lapply so I
end up with a modified list of data.tables
4.
I use attach() to be able to work on each data.table by name

**5**
Latter I when I try to add a column by ":=" I am getting a warning saying
that the whole table had to be copied.
Am I doing something obviously wrong here ?


pseudo code:

fetchingData = function(tableName, connectionToServer){
   DF = connectAndFetchTable(connectionToServer, tableName)
   DT = setDT(DF)
   return(DT)
}

modifyDataTable = function(DT){
   if('thisColName' %in% colnames(DT)){
      DT[,thisColName:=someTransformation(thisColName)]
   }
   ...
}

I the main code :

myDataTableList = sapply(c('tableA','tableB','tableC','tableD'),
FUN=fetchingData, connectionToServer=myCon)
myDataTableList = lapply(myDataTableList, FUN=modifyDataTable)
attach(myDataTableList)

tableB[,newColumn:=1L]
*** getting the warning here***

Not I am using R 3.0.2

Finally let me say that I am using data.table for a long time right now, so
it I know about := and set function (that usually are forgotten and trigger
this warning like when doing DT$new = stuff) 

If any of you gets it I'd be gratefull (sorry I can't reproduce it
correctly)




--
View this message in context: http://r.789695.n4.nabble.com/whole-data-table-copied-warning-tp4702267.html
Sent from the datatable-help mailing list archive at Nabble.com.


More information about the datatable-help mailing list