[Rnomads-user] cygwin warning on ReadGrib.

[Daniel Bowman]

Hi Jason,

You can definitely run wgrib2 in native Windows, without having to use
cygwin or other *nix emulators.  I have a post with instructions here:
https://bovineaerospace.wordpress.com/2015/04/26/how-to-install-rnomads-with-grib-file-support-on-windows/

In any case, reading data from grib files is slow, so you have to do
it thoughtfully. My advice is to define the area of interest, and read
in only that data using the "domain" argument to ReadGrib.  So, if you
are only interested in the continental USA, read in a latitude
longitude box that encompasses the USA.  Then, once the data are read
into R, use a function like BuildProfile() to capture data at specific
locations.

If you loop through each of your million points and call ReadGrib to
read the grib file every time, it will take way too long.

Best,

Danny

On Mon, Mar 5, 2018 at 1:22 AM, Jason G. Su <jasonsu at berkeley.edu> wrote:
> Hi Dr. Bowman,
>
> I use Windows 10 Pro. After loading library rNOMADS, I got the following information:
>
> Loading required package: rvest
> Loading required package: xml2
>
> ****
> Welcome to rNOMADS!
> Questions? Send a message to rnomads-user at lists.r-forge.r-project.org
> Using rNOMADS as a data source for a publication?  Please cite it!
> I'm an early career researcher and every citation matters.
> ****
>
> For sessionInfo(), I got the following messages:
>
> R version 3.4.2 (2017-09-28)
> Platform: x86_64-w64-mingw32/x64 (64-bit)
> Running under: Windows 10 x64 (build 16299)
>
> Matrix products: default
>
> locale:
> [1] LC_COLLATE=English_United States.1252
> [2] LC_CTYPE=English_United States.1252
> [3] LC_MONETARY=English_United States.1252
> [4] LC_NUMERIC=C
> [5] LC_TIME=English_United States.1252
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] XML_3.98-1.10 raster_2.5-8  sp_1.2-5      rNOMADS_2.3.9 rvest_0.3.2
> [6] xml2_1.2.0
>
> loaded via a namespace (and not attached):
> [1] httr_1.3.1      compiler_3.4.2  magrittr_1.5    R6_2.2.2
> [5] Rcpp_0.12.13    grid_3.4.2      lattice_0.20-35
>
> After checking into the data used in the ReadGrib function, I found that the levels were defined wrong in my code. This created a situation when no record was returned, thus the error message.
>
> After I made the corrections to the definition of levels, the code was run smoothly; however, it seems that the ReadGrib function runs slow. Given that my dataset has millions of records with hourly events across several years throughout US, I found it might take too much time to assign weather conditions to the events. I am ok with the events to be assigned weather conditions in about a week, but if it takes too long to do so and the system keeps crashing,  it might prevent the research group to adopt the method.
>
> We are grateful if you could have some suggestions on this. We are also considering using wgrib and wgrib2 DOS shell environment to do the assignments.
>
> Thanks,
> Jason
>
> -----Original Message-----
> From: Daniel Bowman [mailto:danny.c.bowman at gmail.com]
> Sent: Friday, March 2, 2018 8:27 PM
> To: Jason G. Su <jasonsu at berkeley.edu>
> Cc: Announcements, feature requests, and help using rNOMADS <rnomads-user at lists.r-forge.r-project.org>
> Subject: Re: cygwin warning on ReadGrib.
>
> Hi Dr. Su,
>
> This sounds like a really great project, and I'm glad rNOMADS may be a useful tool for it.  In order to diagnose the ReadGrib issue, I need to know two things: 1. What operating system are you using (and specifically, are you using cygwin in windows)?  2.  What do you get when you open an R session and type:
> library(rNOMADS)
> sessionInfo()
>
> You can use the GribInfo() function to get an idea of what's in your grib files.
>
> The BuildProfile() function is a way to capture either the nearest model node to a given location, or even interpolate between nodes to get an "exact" value.  I believe it is reasonably efficient, but if using it on millions of records is taking more than a few days, please let me know and I'll see if I can make it run faster.
>
> Best,
>
> Danny
>
> On Thu, Mar 1, 2018 at 1:28 AM, Jason G. Su <jasonsu at berkeley.edu> wrote:
>> Dear Dr. Bowman,
>>
>>
>>
>> I really appreciated the efforts you made on downloading and
>> processing NOAA Grib data in an R environment. I downloaded several
>> years of NOAA historical weather forecast data (grib or grib2) and
>> wanted to assign hourly weather data to millions of records of asthma
>> rescue medication use with their lat/lng information. There are
>> several things that I cannot figure out and I am grateful if you could provide suggestions:
>>
>>
>>
>> 1.     ReadGrib function never worked in my code: an error message always
>> showed up as “Error in if (csv.str[1] == "cygwin warning:") { :
>> missing value where TRUE/FALSE needed”. Could you provide some
>> suggestions on why this occurred and how to correct for that?
>>
>>
>>
>> 2.     Though I could get a list of forecasts, levels and variables through
>> the ParseModelPage function data grabbed through CrawlModels, I cannot
>> find a way to get similar parameters on the grib and grib2 files stored locally.
>> I believe relative path function also works in rnomads package. Any
>> suggestions on this?
>>
>>
>>
>> 3.     I need assign date and time weather conditions to millions of medical
>> records of corresponding date and time based on spatial locations. Do
>> you have any suggestions on how to do that?
>>
>>
>>
>> Thanks,
>> Jason G. Su
>>
>>
>>
>> -----------------------------------------
>>
>> Jason G. Su, PhD
>>
>> Associate Researcher and Principal Investigator
>>
>> Environmental Health Sciences
>>
>> School of Public Health
>>
>> University of California, Berkeley
>>
>> Berkeley, CA 94720-7360
>>
>>
>