Subject: FFMP use of shapefiles Date: Fri, 04 Jan 2002 11:58:14 -0500 From: "Michael Churma" Organization: DOC/NOAA/NWS - National Weather Service To: Paul Jendrowski , Michael Mercer , Ami Arthur , Tom Filiaggi , Stephan Smith Hi folks, Many of us were part of a conference call on Dec. 20 in which a modification of the basin aggregation scheme was discussed. During that call, it became apparent that there was some confusion as to how FFMP uses the aggregated basin and bin-to-basin shapefiles. I had addressed FFMP's usage of shapefiles in an August e-mail, which I've attached below. In addition, I think it would be beneficial to clarify some specific points that were brought up in the conference call. How does FFMP use the shapefiles? FFMP never uses the shapefiles directly. We instead use localization methods to read the important shapefile data, and save it into files with manageable sizes. This is a standard practice throughout AWIPS. AWIPS localization can create Geo-Entity Lookup Tables (GELTs). Like a watered-down shapefile, a GELT is composed of several different files, including an attribute file and a binary grid file. During AWIPS localization, we use the aggregated basins shapefile to create a GELT. We use the GELT and the bin-to-basin point shapefile to generate several smaller lookup files, similar to the AMBER lookups. The GELT also helps us display the data, as gridpoints are assigned to specific basins. Note that the GELT is used both to provide basin input data for FFMP and to display the output. What's the significance of the 2 square mile minimum basin size limit? Our GELT grid has fine resolution by AWIPS standards, but it is still much coarser than a shapefile. The 2 square mile minimum size helps the GELT to resolve all the basins in the shapefile. The GELT grid has dimensions of 800 by 800. These dimensions are arbitrary, but experimentation showed them to be a good compromise between resolution and performance (the standard trade-off). The grid is set to fit the WFO scale map on D-2D, so the size of the individual grid boxes will vary from site to site. As mentioned below, the grid boxes can vary from about 0.1 to 0.3 square miles, depending on the (CONUS) site. While there's no absolute guarantee that the GELT grids will resolve every single basin, our early test localizations at various sites showed the GELT grids to perform very well, even for sites with large CWAs (such as in the Southwest). What happens if the GELT misses a basin while reading the shapefile? If the GELT cannot resolve a basin, that basin does not get evaluated by FFMP. This is most likely to occur if a basin is very small, or shaped in a way that would cause it to elude a GELT grid point. In the output display, there will be no "hole" in the display under these conditions, because a hole would indicate that the GELT had detected something. The missing basin will, in effect be smoothed over. I want to emphasize that FFMP will still work if it misses a basin (or basins). I know this because FFMP worked with a prototype KLWX shapefile that had no basin size restrictions. The problem was that we were losing a couple hundred basins. This means, in effect, that their associated radar bin data is being lost as well. This is why we had asked for the size limit. If the basins are smaller than 2 square miles, they still can often be resolved; it just gets more difficult. What if some basins are removed from the aggregated list? I believe that one proposal made during the conference call was that some basins would be either shrunken or removed from the aggregated list. If the GELT can resolve these changes, the FFMP processor will ignore the empty spaces, and the display will show a hole. As was suggested in the conference call, an area without a basin would be handled the same way as a lake. What if a new attribute is added to flag some basins in the aggregated shapefile? Our GELT would not be affected if an additional attribute field is added to the shapefile; it would only be affected if one of the attributes it reads is removed or altered (i.e., if its name or output format is changed). In the aggregated basin shapefile, we currently look for PFAF_ID, STREAMNAME, CWA, RFC, MODIFIED, RESERVOIR, and AREA_SQ_MI. If there were to be discernible, built-in holes to FFMP coverage, it would be possible to use an attribute flag to signal FFMP to exclude the flagged basins from processing, and to shade them gray on the display (to highlight the coverage gaps). This would not be a difficult coding task, but it would take some time to get it through the AWIPS pipeline. Let me know if there are other questions regarding FFMP's use of shapefiles. Mike ------------------------------------------------------------------------------ Subject: Re: proposed Small Basin shape file change Date: Wed, 15 Aug 2001 14:41:25 -0400 From: "Michael.Churma" Organization: DOC/NOAA/NWS - National Weather Service To: Paul Jendrowski CC: Stephan Smith , Tom Filiaggi , Michael Mercer , Ami Arthur , Mark Glaudemans References: 1 , 2 , 3 , 4 Paul & Ami, In localization, we read the aggregated basins shapefile to create a Geo-Entity Lookup Table (GELT). The size of the GELT grid is 800x800, over the WFO scale on AWIPS. Since WFO scales vary from site to site, the area of the each grid box will vary too, from about 0.1 to 0.4 square miles. This should be enough to resolve basins of 2.0 square miles that it finds in the shapefile. We found this GELT size to be a good compromise between resolution and performance. Even though GELTs are much smaller than shapefiles, they are still large enough to cause performance problems if accessed a lot. Therefore the GELT is used (still in localization) to create lookup data files to spare us constant runtime GELT access We relate basins to counties, to the local HRAP grid (to match up the basins to the FFG data now available via SBN), and back to our own GELT. We use the bin-to-basin point shapefile from NSSL to build a lookup file relating the basins to the radial bins in the DHR. These lookup files are what we primarily use in the actual runtime algorithms, not the GELT. It is important, though, that the GELT see all the basins, because a missed basin won't make it into our algorithms. The GELT directly comes back into play on the display side. As per Ami's request, I've enclosed an image of our D-2D display. The basins are colored in using the GELT grid. This display shows a close-up of basins in a selected county (in this case, Frederick County, MD, as seen in the Basin Table). A "parent" display shows a WFO-scale map that lights up counties based on the worst-case basin in the county. Two notes: 1. The numbering system of the basins in the table is arbitrary, but sliding the cursor over the list will provide the name of the basin, as provided in the aggregated shapefile (I had trouble capturing this feature because the screen-capture routine takes over the cursor). 2. The basin map background is drawn not by the shapefile but by a binary cartographic data file, also created in localization. While D-2D can read shapefiles directly, the standard method is to use BCD files, because they are much smaller, and thus more efficient. I want to stress that we don't use the shapefile after localization. Mike Mercer told me of a performance problem that Salt Lake City was having. My understanding was that they were trying to load their shapefile directly into D-2D, and getting bogged down. When we got the Pittsburgh shapefile from Ami, with its 93 meg *.shp component, we created a corresponding BCD file that was only 7 megs. Jim Ramer wrote several algorithms that allow AWIPS programmers to adjust the size of the BCD file to their needs. I've also attached a Trend Window image. This can be generated by clicking on a basin in the table or on the map. Paul should find this familiar. I hope this info helps.... Mike