From aakremena at aol.com Sun Sep 17 18:29:55 2017 From: aakremena at aol.com (aakremena at aol.com) Date: Sun, 17 Sep 2017 12:29:55 -0400 Subject: [Rnomads-user] CrawlModels error: Error in doc_parse_raw(x, encoding = encoding, base_url = base_url, as_html = as_html, : Excessive depth in document: 256 use XML_PARSE_HUGE option [1] In-Reply-To: <15e8747bcc4-c08-36c1@webjas-vad088.srv.aolmail.net> References: <15e8747bcc4-c08-36c1@webjas-vad088.srv.aolmail.net> Message-ID: <15e90acbc44-c0c-134c@webjas-vaa048.srv.aolmail.net> Hi rNOMADS users, 1. Just started with rNOMADS and the first attempt to download data: R> urls_out <- CrawlModels(abbrev = "gfs_0p25_1hr", depth = 2) fails with Error in doc_parse_raw(x, encoding = encoding, base_url = base_url, as_html = as_html, : Excessive depth in document: 256 use XML_PARSE_HUGE option [1]. 2. The suggestion from https://stackoverflow.com/questions/33819103/parsing-small-web-page-with-xml2-throws-xml-parse-huge-error, i.e. to try library(drat) drat:::add("shabbychef") install.packages('xml2') doesn't take care of the problem. 3. I see on the net discussions concerning this problem since around 2015. Are there any developments or workarrounds available? 4. More specifically, can anybody suggest a way to go around this error? Thank you in advance. Anguel -------------- next part -------------- An HTML attachment was scrubbed... URL: From aakremena at aol.com Mon Sep 18 00:50:52 2017 From: aakremena at aol.com (aakremena at aol.com) Date: Sun, 17 Sep 2017 18:50:52 -0400 Subject: [Rnomads-user] Is GribGrab's model.domain option working at all?? Message-ID: <15e92098641-c07-1ed7b@webjas-vac126.srv.aolmail.net> Hi rNOMADS users, Setup: -- I'm playing with rNOMADS (started with rNOMADS yesterday). -- Most likely I should be doing something wrong here but what concerns me, the GribGrab's 'model.domain' option is not kicking in... -- Here is some sample code: rm(list = ls()) library(rNOMADS) #A location near my house lat <- 35.828304 lon <- -79.107467 #Get nearest model nodes lons <- seq(0, 359.5, by = 0.5) lats <- seq(-90, 90, by = 0.5) lon.diff <- abs(lon + 360 - lons) lat.diff <- abs(lat - lats) model.lon.ind <- which(lon.diff == min(lon.diff)) - 1 #NOMADS indexes at 0 model.lat.ind <- which(lat.diff == min(lat.diff)) - 1 model_domain <- c(model.lon.ind, model.lon.ind, model.lat.ind, model.lat.ind) #Find the latest Global Forecast System model run urls.out <- CrawlModels(abbrev = "gfs_0p50", depth = 2) urls.out <- gsub("gfs_0p50", "gfs_0p25_1hr", urls.out) #this is meant to avoid the error coming from using CrawlModels(abbrev = "gfs_0p25_1hr", depth = 2) latest.model <- urls.out[1] model.parameters <- ParseModelPage(latest.model) pred.ind <- c(3,4,5) #use only 3 temperature forecasts variables <- c("TMP") levels <- c("2 m above ground") grib.info <- GribGrab(urls.out[1], model.parameters$pred[pred.ind], levels, variables, model.domain = model_domain) grib.data <- ReadGrib(grib.info[[1]]$file.name, levels, variables, domain = model_domain) Observations: 1. The region I need temperatures for is a single geographic point: model_domain <- c(562, 562, 252, 252) 2. Still, GribGrab produces in this case 3 downloaded .grb files, each 1.4 MB, which seems a lot for a single temperature data point inside each one. 3. The length of grib.data[[7]] is 1038240, which again confirms that GribGrabd is downloading temperatures for the whole world and not for the single geographic point with (lat, lon), given above. Questions: A. Am I doing anything wrong here, or the model.domain option is somehow, permanently set to NULL?? B. More generally, how can I download data only for a specified location, so that not to burden unnecessarily the NOMADS server? Thank you in advance. Anguel -------------- next part -------------- An HTML attachment was scrubbed... URL: From aakremena at aol.com Fri Sep 15 22:43:03 2017 From: aakremena at aol.com (aakremena at aol.com) Date: Fri, 15 Sep 2017 20:43:03 -0000 Subject: [Rnomads-user] CrawlModels error: Error in doc_parse_raw(x, encoding = encoding, base_url = base_url, as_html = as_html, : Excessive depth in document: 256 use XML_PARSE_HUGE option [1] Message-ID: <15e8747bcc4-c08-36c1@webjas-vad088.srv.aolmail.net> Hi rNOMADS users, 1. Just started with rNOMADS and the first attempt to download data: R> urls_out <- CrawlModels(abbrev = "gfs_0p25_1hr", depth = 2) fails with Error in doc_parse_raw(x, encoding = encoding, base_url = base_url, as_html = as_html, : Excessive depth in document: 256 use XML_PARSE_HUGE option [1]. 2. The suggestion from https://stackoverflow.com/questions/33819103/parsing-small-web-page-with-xml2-throws-xml-parse-huge-error, i.e. to try library(drat) drat:::add("shabbychef") install.packages('xml2') doesn't take care of the problem. 3. I see on the net discussions concerning this problem since around 2015. Are there any developments or workarrounds available? 4. More specifically, can anybody suggest a way to go around this error? Thank you in advance. Anguel -------------- next part -------------- An HTML attachment was scrubbed... URL: