From danny.c.bowman at gmail.com  Sat Mar  3 05:26:52 2018
From: danny.c.bowman at gmail.com (Daniel Bowman)
Date: Fri, 2 Mar 2018 21:26:52 -0700
Subject: [Rnomads-user] cygwin warning on ReadGrib.
In-Reply-To: <01cb01d3b137$59edf140$0dc9d3c0$@berkeley.edu>
References: <01cb01d3b137$59edf140$0dc9d3c0$@berkeley.edu>
Message-ID: <CAN=ZksOG6WfWoN42+_f3BhvNBJtg+Qrp=yCTjgzq9Hdmn92GFQ@mail.gmail.com>

Hi Dr. Su,

This sounds like a really great project, and I'm glad rNOMADS may be a
useful tool for it.  In order to diagnose the ReadGrib issue, I need
to know two things: 1. What operating system are you using (and
specifically, are you using cygwin in windows)?  2.  What do you get
when you open an R session and type:
library(rNOMADS)
sessionInfo()

You can use the GribInfo() function to get an idea of what's in your
grib files.

The BuildProfile() function is a way to capture either the nearest
model node to a given location, or even interpolate between nodes to
get an "exact" value.  I believe it is reasonably efficient, but if
using it on millions of records is taking more than a few days, please
let me know and I'll see if I can make it run faster.

Best,

Danny

On Thu, Mar 1, 2018 at 1:28 AM, Jason G. Su <jasonsu at berkeley.edu> wrote:
> Dear Dr. Bowman,
>
>
>
> I really appreciated the efforts you made on downloading and processing NOAA
> Grib data in an R environment. I downloaded several years of NOAA historical
> weather forecast data (grib or grib2) and wanted to assign hourly weather
> data to millions of records of asthma rescue medication use with their
> lat/lng information. There are several things that I cannot figure out and I
> am grateful if you could provide suggestions:
>
>
>
> 1.     ReadGrib function never worked in my code: an error message always
> showed up as ?Error in if (csv.str[1] == "cygwin warning:") { :  missing
> value where TRUE/FALSE needed?. Could you provide some suggestions on why
> this occurred and how to correct for that?
>
>
>
> 2.     Though I could get a list of forecasts, levels and variables through
> the ParseModelPage function data grabbed through CrawlModels, I cannot find
> a way to get similar parameters on the grib and grib2 files stored locally.
> I believe relative path function also works in rnomads package. Any
> suggestions on this?
>
>
>
> 3.     I need assign date and time weather conditions to millions of medical
> records of corresponding date and time based on spatial locations. Do you
> have any suggestions on how to do that?
>
>
>
> Thanks,
> Jason G. Su
>
>
>
> -----------------------------------------
>
> Jason G. Su, PhD
>
> Associate Researcher and Principal Investigator
>
> Environmental Health Sciences
>
> School of Public Health
>
> University of California, Berkeley
>
> Berkeley, CA 94720-7360
>
>

From jasonsu at berkeley.edu  Thu Mar  1 09:28:59 2018
From: jasonsu at berkeley.edu (Jason G. Su)
Date: Thu, 1 Mar 2018 00:28:59 -0800
Subject: [Rnomads-user] cygwin warning on ReadGrib.
Message-ID: <01cb01d3b137$59edf140$0dc9d3c0$@berkeley.edu>

Dear Dr. Bowman,

 

I really appreciated the efforts you made on downloading and processing NOAA
Grib data in an R environment. I downloaded several years of NOAA historical
weather forecast data (grib or grib2) and wanted to assign hourly weather
data to millions of records of asthma rescue medication use with their
lat/lng information. There are several things that I cannot figure out and I
am grateful if you could provide suggestions:

 

1.     ReadGrib function never worked in my code: an error message always
showed up as "Error in if (csv.str[1] == "cygwin warning:") { :  missing
value where TRUE/FALSE needed". Could you provide some suggestions on why
this occurred and how to correct for that?

 

2.     Though I could get a list of forecasts, levels and variables through
the ParseModelPage function data grabbed through CrawlModels, I cannot find
a way to get similar parameters on the grib and grib2 files stored locally.
I believe relative path function also works in rnomads package. Any
suggestions on this?

 

3.     I need assign date and time weather conditions to millions of medical
records of corresponding date and time based on spatial locations. Do you
have any suggestions on how to do that?

 

Thanks,
Jason G. Su

 

-----------------------------------------

Jason G. Su, PhD

Associate Researcher and Principal Investigator

Environmental Health Sciences

School of Public Health

University of California, Berkeley

Berkeley, CA 94720-7360

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rnomads-user/attachments/20180301/161fcf69/attachment.html>

From jasonsu at berkeley.edu  Mon Mar  5 09:22:37 2018
From: jasonsu at berkeley.edu (Jason G. Su)
Date: Mon, 5 Mar 2018 00:22:37 -0800
Subject: [Rnomads-user] cygwin warning on ReadGrib.
In-Reply-To: <CAN=ZksOG6WfWoN42+_f3BhvNBJtg+Qrp=yCTjgzq9Hdmn92GFQ@mail.gmail.com>
References: <01cb01d3b137$59edf140$0dc9d3c0$@berkeley.edu>
 <CAN=ZksOG6WfWoN42+_f3BhvNBJtg+Qrp=yCTjgzq9Hdmn92GFQ@mail.gmail.com>
Message-ID: <031701d3b45b$1fc20930$5f461b90$@berkeley.edu>

Hi Dr. Bowman,

I use Windows 10 Pro. After loading library rNOMADS, I got the following information:

Loading required package: rvest
Loading required package: xml2

****
Welcome to rNOMADS!
Questions? Send a message to rnomads-user at lists.r-forge.r-project.org
Using rNOMADS as a data source for a publication?  Please cite it!
I'm an early career researcher and every citation matters.
****

For sessionInfo(), I got the following messages:

R version 3.4.2 (2017-09-28)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 16299)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 
[2] LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] XML_3.98-1.10 raster_2.5-8  sp_1.2-5      rNOMADS_2.3.9 rvest_0.3.2  
[6] xml2_1.2.0   

loaded via a namespace (and not attached):
[1] httr_1.3.1      compiler_3.4.2  magrittr_1.5    R6_2.2.2       
[5] Rcpp_0.12.13    grid_3.4.2      lattice_0.20-35

After checking into the data used in the ReadGrib function, I found that the levels were defined wrong in my code. This created a situation when no record was returned, thus the error message.

After I made the corrections to the definition of levels, the code was run smoothly; however, it seems that the ReadGrib function runs slow. Given that my dataset has millions of records with hourly events across several years throughout US, I found it might take too much time to assign weather conditions to the events. I am ok with the events to be assigned weather conditions in about a week, but if it takes too long to do so and the system keeps crashing,  it might prevent the research group to adopt the method.

We are grateful if you could have some suggestions on this. We are also considering using wgrib and wgrib2 DOS shell environment to do the assignments.

Thanks,
Jason

-----Original Message-----
From: Daniel Bowman [mailto:danny.c.bowman at gmail.com] 
Sent: Friday, March 2, 2018 8:27 PM
To: Jason G. Su <jasonsu at berkeley.edu>
Cc: Announcements, feature requests, and help using rNOMADS <rnomads-user at lists.r-forge.r-project.org>
Subject: Re: cygwin warning on ReadGrib.

Hi Dr. Su,

This sounds like a really great project, and I'm glad rNOMADS may be a useful tool for it.  In order to diagnose the ReadGrib issue, I need to know two things: 1. What operating system are you using (and specifically, are you using cygwin in windows)?  2.  What do you get when you open an R session and type:
library(rNOMADS)
sessionInfo()

You can use the GribInfo() function to get an idea of what's in your grib files.

The BuildProfile() function is a way to capture either the nearest model node to a given location, or even interpolate between nodes to get an "exact" value.  I believe it is reasonably efficient, but if using it on millions of records is taking more than a few days, please let me know and I'll see if I can make it run faster.

Best,

Danny

On Thu, Mar 1, 2018 at 1:28 AM, Jason G. Su <jasonsu at berkeley.edu> wrote:
> Dear Dr. Bowman,
>
>
>
> I really appreciated the efforts you made on downloading and 
> processing NOAA Grib data in an R environment. I downloaded several 
> years of NOAA historical weather forecast data (grib or grib2) and 
> wanted to assign hourly weather data to millions of records of asthma 
> rescue medication use with their lat/lng information. There are 
> several things that I cannot figure out and I am grateful if you could provide suggestions:
>
>
>
> 1.     ReadGrib function never worked in my code: an error message always
> showed up as ?Error in if (csv.str[1] == "cygwin warning:") { :  
> missing value where TRUE/FALSE needed?. Could you provide some 
> suggestions on why this occurred and how to correct for that?
>
>
>
> 2.     Though I could get a list of forecasts, levels and variables through
> the ParseModelPage function data grabbed through CrawlModels, I cannot 
> find a way to get similar parameters on the grib and grib2 files stored locally.
> I believe relative path function also works in rnomads package. Any 
> suggestions on this?
>
>
>
> 3.     I need assign date and time weather conditions to millions of medical
> records of corresponding date and time based on spatial locations. Do 
> you have any suggestions on how to do that?
>
>
>
> Thanks,
> Jason G. Su
>
>
>
> -----------------------------------------
>
> Jason G. Su, PhD
>
> Associate Researcher and Principal Investigator
>
> Environmental Health Sciences
>
> School of Public Health
>
> University of California, Berkeley
>
> Berkeley, CA 94720-7360
>
>


From danny.c.bowman at gmail.com  Tue Mar  6 05:35:09 2018
From: danny.c.bowman at gmail.com (Daniel Bowman)
Date: Mon, 5 Mar 2018 21:35:09 -0700
Subject: [Rnomads-user] cygwin warning on ReadGrib.
In-Reply-To: <031701d3b45b$1fc20930$5f461b90$@berkeley.edu>
References: <01cb01d3b137$59edf140$0dc9d3c0$@berkeley.edu>
 <CAN=ZksOG6WfWoN42+_f3BhvNBJtg+Qrp=yCTjgzq9Hdmn92GFQ@mail.gmail.com>
 <031701d3b45b$1fc20930$5f461b90$@berkeley.edu>
Message-ID: <CAN=ZksM4s0y7OVt1a5TzoRW+RpA9cEo6ozESGKHn3A1eqJz7bg@mail.gmail.com>

Hi Jason,

You can definitely run wgrib2 in native Windows, without having to use
cygwin or other *nix emulators.  I have a post with instructions here:
https://bovineaerospace.wordpress.com/2015/04/26/how-to-install-rnomads-with-grib-file-support-on-windows/

In any case, reading data from grib files is slow, so you have to do
it thoughtfully. My advice is to define the area of interest, and read
in only that data using the "domain" argument to ReadGrib.  So, if you
are only interested in the continental USA, read in a latitude
longitude box that encompasses the USA.  Then, once the data are read
into R, use a function like BuildProfile() to capture data at specific
locations.

If you loop through each of your million points and call ReadGrib to
read the grib file every time, it will take way too long.

Best,

Danny

On Mon, Mar 5, 2018 at 1:22 AM, Jason G. Su <jasonsu at berkeley.edu> wrote:
> Hi Dr. Bowman,
>
> I use Windows 10 Pro. After loading library rNOMADS, I got the following information:
>
> Loading required package: rvest
> Loading required package: xml2
>
> ****
> Welcome to rNOMADS!
> Questions? Send a message to rnomads-user at lists.r-forge.r-project.org
> Using rNOMADS as a data source for a publication?  Please cite it!
> I'm an early career researcher and every citation matters.
> ****
>
> For sessionInfo(), I got the following messages:
>
> R version 3.4.2 (2017-09-28)
> Platform: x86_64-w64-mingw32/x64 (64-bit)
> Running under: Windows 10 x64 (build 16299)
>
> Matrix products: default
>
> locale:
> [1] LC_COLLATE=English_United States.1252
> [2] LC_CTYPE=English_United States.1252
> [3] LC_MONETARY=English_United States.1252
> [4] LC_NUMERIC=C
> [5] LC_TIME=English_United States.1252
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] XML_3.98-1.10 raster_2.5-8  sp_1.2-5      rNOMADS_2.3.9 rvest_0.3.2
> [6] xml2_1.2.0
>
> loaded via a namespace (and not attached):
> [1] httr_1.3.1      compiler_3.4.2  magrittr_1.5    R6_2.2.2
> [5] Rcpp_0.12.13    grid_3.4.2      lattice_0.20-35
>
> After checking into the data used in the ReadGrib function, I found that the levels were defined wrong in my code. This created a situation when no record was returned, thus the error message.
>
> After I made the corrections to the definition of levels, the code was run smoothly; however, it seems that the ReadGrib function runs slow. Given that my dataset has millions of records with hourly events across several years throughout US, I found it might take too much time to assign weather conditions to the events. I am ok with the events to be assigned weather conditions in about a week, but if it takes too long to do so and the system keeps crashing,  it might prevent the research group to adopt the method.
>
> We are grateful if you could have some suggestions on this. We are also considering using wgrib and wgrib2 DOS shell environment to do the assignments.
>
> Thanks,
> Jason
>
> -----Original Message-----
> From: Daniel Bowman [mailto:danny.c.bowman at gmail.com]
> Sent: Friday, March 2, 2018 8:27 PM
> To: Jason G. Su <jasonsu at berkeley.edu>
> Cc: Announcements, feature requests, and help using rNOMADS <rnomads-user at lists.r-forge.r-project.org>
> Subject: Re: cygwin warning on ReadGrib.
>
> Hi Dr. Su,
>
> This sounds like a really great project, and I'm glad rNOMADS may be a useful tool for it.  In order to diagnose the ReadGrib issue, I need to know two things: 1. What operating system are you using (and specifically, are you using cygwin in windows)?  2.  What do you get when you open an R session and type:
> library(rNOMADS)
> sessionInfo()
>
> You can use the GribInfo() function to get an idea of what's in your grib files.
>
> The BuildProfile() function is a way to capture either the nearest model node to a given location, or even interpolate between nodes to get an "exact" value.  I believe it is reasonably efficient, but if using it on millions of records is taking more than a few days, please let me know and I'll see if I can make it run faster.
>
> Best,
>
> Danny
>
> On Thu, Mar 1, 2018 at 1:28 AM, Jason G. Su <jasonsu at berkeley.edu> wrote:
>> Dear Dr. Bowman,
>>
>>
>>
>> I really appreciated the efforts you made on downloading and
>> processing NOAA Grib data in an R environment. I downloaded several
>> years of NOAA historical weather forecast data (grib or grib2) and
>> wanted to assign hourly weather data to millions of records of asthma
>> rescue medication use with their lat/lng information. There are
>> several things that I cannot figure out and I am grateful if you could provide suggestions:
>>
>>
>>
>> 1.     ReadGrib function never worked in my code: an error message always
>> showed up as ?Error in if (csv.str[1] == "cygwin warning:") { :
>> missing value where TRUE/FALSE needed?. Could you provide some
>> suggestions on why this occurred and how to correct for that?
>>
>>
>>
>> 2.     Though I could get a list of forecasts, levels and variables through
>> the ParseModelPage function data grabbed through CrawlModels, I cannot
>> find a way to get similar parameters on the grib and grib2 files stored locally.
>> I believe relative path function also works in rnomads package. Any
>> suggestions on this?
>>
>>
>>
>> 3.     I need assign date and time weather conditions to millions of medical
>> records of corresponding date and time based on spatial locations. Do
>> you have any suggestions on how to do that?
>>
>>
>>
>> Thanks,
>> Jason G. Su
>>
>>
>>
>> -----------------------------------------
>>
>> Jason G. Su, PhD
>>
>> Associate Researcher and Principal Investigator
>>
>> Environmental Health Sciences
>>
>> School of Public Health
>>
>> University of California, Berkeley
>>
>> Berkeley, CA 94720-7360
>>
>>
>

From jasonsu at berkeley.edu  Tue Mar  6 09:57:42 2018
From: jasonsu at berkeley.edu (Jason G. Su)
Date: Tue, 6 Mar 2018 00:57:42 -0800
Subject: [Rnomads-user] cygwin warning on ReadGrib.
In-Reply-To: <CAN=ZksM4s0y7OVt1a5TzoRW+RpA9cEo6ozESGKHn3A1eqJz7bg@mail.gmail.com>
References: <01cb01d3b137$59edf140$0dc9d3c0$@berkeley.edu>
 <CAN=ZksOG6WfWoN42+_f3BhvNBJtg+Qrp=yCTjgzq9Hdmn92GFQ@mail.gmail.com>
 <031701d3b45b$1fc20930$5f461b90$@berkeley.edu>
 <CAN=ZksM4s0y7OVt1a5TzoRW+RpA9cEo6ozESGKHn3A1eqJz7bg@mail.gmail.com>
Message-ID: <03f001d3b529$308cb250$91a616f0$@berkeley.edu>

Hi Daniel,

Really appreciated your guidance and kind help. We might end up with running wgrib2 in Windows commands (through batch files) to speed up the process. I will set up the environment and path as you suggested for us to do so.

The dataset I have downloaded is the 12 km resolution NAM_ANL Data (with prediction at 1, 2, 3 and 6 hours). Due to the dataset is on North America, I do not need subset data for exposure assignment.

Do you have interest to be part of the co-authors on assigning environmental exposures (weather and air pollutants) and modeling spatiotemporal environmental triggers of asthma? It might take some time for us to be able to assign all the exposures (several years of space-time exposure) to more than 7,000 patients across US. These exposures are not necessarily home-based. They could happen anywhere and the smart inhalers track the space-time rescue medication use.

A side note: for the rNOMADS package you developed, could you warn an R programmer when the levels or variables are not specified correct? Or do you have a way to verify those before the read grid function is implemented?

Thanks,
Jason

-----Original Message-----
From: Daniel Bowman [mailto:danny.c.bowman at gmail.com] 
Sent: Monday, March 5, 2018 8:35 PM
To: Jason G. Su <jasonsu at berkeley.edu>
Cc: Announcements, feature requests, and help using rNOMADS <rnomads-user at lists.r-forge.r-project.org>
Subject: Re: cygwin warning on ReadGrib.

Hi Jason,

You can definitely run wgrib2 in native Windows, without having to use cygwin or other *nix emulators.  I have a post with instructions here:
https://bovineaerospace.wordpress.com/2015/04/26/how-to-install-rnomads-with-grib-file-support-on-windows/

In any case, reading data from grib files is slow, so you have to do it thoughtfully. My advice is to define the area of interest, and read in only that data using the "domain" argument to ReadGrib.  So, if you are only interested in the continental USA, read in a latitude longitude box that encompasses the USA.  Then, once the data are read into R, use a function like BuildProfile() to capture data at specific locations.

If you loop through each of your million points and call ReadGrib to read the grib file every time, it will take way too long.

Best,

Danny

On Mon, Mar 5, 2018 at 1:22 AM, Jason G. Su <jasonsu at berkeley.edu> wrote:
> Hi Dr. Bowman,
>
> I use Windows 10 Pro. After loading library rNOMADS, I got the following information:
>
> Loading required package: rvest
> Loading required package: xml2
>
> ****
> Welcome to rNOMADS!
> Questions? Send a message to rnomads-user at lists.r-forge.r-project.org
> Using rNOMADS as a data source for a publication?  Please cite it!
> I'm an early career researcher and every citation matters.
> ****
>
> For sessionInfo(), I got the following messages:
>
> R version 3.4.2 (2017-09-28)
> Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 
> x64 (build 16299)
>
> Matrix products: default
>
> locale:
> [1] LC_COLLATE=English_United States.1252 [2] LC_CTYPE=English_United 
> States.1252 [3] LC_MONETARY=English_United States.1252 [4] 
> LC_NUMERIC=C [5] LC_TIME=English_United States.1252
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] XML_3.98-1.10 raster_2.5-8  sp_1.2-5      rNOMADS_2.3.9 rvest_0.3.2
> [6] xml2_1.2.0
>
> loaded via a namespace (and not attached):
> [1] httr_1.3.1      compiler_3.4.2  magrittr_1.5    R6_2.2.2
> [5] Rcpp_0.12.13    grid_3.4.2      lattice_0.20-35
>
> After checking into the data used in the ReadGrib function, I found that the levels were defined wrong in my code. This created a situation when no record was returned, thus the error message.
>
> After I made the corrections to the definition of levels, the code was run smoothly; however, it seems that the ReadGrib function runs slow. Given that my dataset has millions of records with hourly events across several years throughout US, I found it might take too much time to assign weather conditions to the events. I am ok with the events to be assigned weather conditions in about a week, but if it takes too long to do so and the system keeps crashing,  it might prevent the research group to adopt the method.
>
> We are grateful if you could have some suggestions on this. We are also considering using wgrib and wgrib2 DOS shell environment to do the assignments.
>
> Thanks,
> Jason
>
> -----Original Message-----
> From: Daniel Bowman [mailto:danny.c.bowman at gmail.com]
> Sent: Friday, March 2, 2018 8:27 PM
> To: Jason G. Su <jasonsu at berkeley.edu>
> Cc: Announcements, feature requests, and help using rNOMADS 
> <rnomads-user at lists.r-forge.r-project.org>
> Subject: Re: cygwin warning on ReadGrib.
>
> Hi Dr. Su,
>
> This sounds like a really great project, and I'm glad rNOMADS may be a useful tool for it.  In order to diagnose the ReadGrib issue, I need to know two things: 1. What operating system are you using (and specifically, are you using cygwin in windows)?  2.  What do you get when you open an R session and type:
> library(rNOMADS)
> sessionInfo()
>
> You can use the GribInfo() function to get an idea of what's in your grib files.
>
> The BuildProfile() function is a way to capture either the nearest model node to a given location, or even interpolate between nodes to get an "exact" value.  I believe it is reasonably efficient, but if using it on millions of records is taking more than a few days, please let me know and I'll see if I can make it run faster.
>
> Best,
>
> Danny
>
> On Thu, Mar 1, 2018 at 1:28 AM, Jason G. Su <jasonsu at berkeley.edu> wrote:
>> Dear Dr. Bowman,
>>
>>
>>
>> I really appreciated the efforts you made on downloading and 
>> processing NOAA Grib data in an R environment. I downloaded several 
>> years of NOAA historical weather forecast data (grib or grib2) and 
>> wanted to assign hourly weather data to millions of records of asthma 
>> rescue medication use with their lat/lng information. There are 
>> several things that I cannot figure out and I am grateful if you could provide suggestions:
>>
>>
>>
>> 1.     ReadGrib function never worked in my code: an error message always
>> showed up as ?Error in if (csv.str[1] == "cygwin warning:") { :
>> missing value where TRUE/FALSE needed?. Could you provide some 
>> suggestions on why this occurred and how to correct for that?
>>
>>
>>
>> 2.     Though I could get a list of forecasts, levels and variables through
>> the ParseModelPage function data grabbed through CrawlModels, I 
>> cannot find a way to get similar parameters on the grib and grib2 files stored locally.
>> I believe relative path function also works in rnomads package. Any 
>> suggestions on this?
>>
>>
>>
>> 3.     I need assign date and time weather conditions to millions of medical
>> records of corresponding date and time based on spatial locations. Do 
>> you have any suggestions on how to do that?
>>
>>
>>
>> Thanks,
>> Jason G. Su
>>
>>
>>
>> -----------------------------------------
>>
>> Jason G. Su, PhD
>>
>> Associate Researcher and Principal Investigator
>>
>> Environmental Health Sciences
>>
>> School of Public Health
>>
>> University of California, Berkeley
>>
>> Berkeley, CA 94720-7360
>>
>>
>