RADAR Project
Lately I’ve been doing some interesting work with the RADAR project that I’ve been loosely affiliated with for the last few years. The goal was to find a way to receive products from the NWS through one of their public channels and process them into a usable format. Most of the products are in “proprietary” formats (not exactly proprietary, but close enough), so they need special processors or viewers to use. I’m just going to give a brief summary of the project history, and follow up to some of the latest developments.
Oh, back in 2001 or so, I don’t exactly remember how, probably through Winnebago Co. ARES/SKYWARN I met this fellow Nick Luther who was obviously a very bright individual. He said he found an FTP site through which the NWS disseminated many of their products in their native format–much smaller in size than the processed GIF images that were publicly available, and it was up to 6 minutes faster. He had heard of me and the reputation I had in the area with Amateur Packet Radio, and he wanted to see if we could find a way to disseminate radar images over-the-air using packet radio as a medium.
So we began researching. The NWS did not and does not make their documentation easy to understand. They write a ton of software that they use in-house, and much of it is available in one way or another, but it is impossible to make work because of their documentation and the complexity of the software. We found first this software package called GEMPAK. GEMPAK is written by Unidata/UCAR and stands for GEneral Meteorology PAcKage. It is extremely powerful in its capability to decode the multitudes of proprietary NWS formats. At the time when we were using it, it was also quite slow, taking up to a minute to decode a single RADAR image.
At the same time, we were working on a system to retrieve the images from the NWS Public Telecommunications Gateway. The FTP connection was slow, so it was tricky. Nick and I worked on various ideas in PERL to pull and process this data. After a while, the pieces came together as RadarE. This script allowed us to choose the products we wanted to have pulled, then process and archive them. It compensated fairly well for the unreliability of the NWS PTG. It grew to have a queue system for processing pipelines and selective processing.
Processing, though, was too messy. We could receive the products from the NWS with relative expediency, but a typical WSR-88D radar scan has 25-50 separate products! Now we weren’t processing them all, but even if we process just three from each site (say base reflectivity, composite reflectivity and storm relative motion), when that is multiplied by three interesting sites for the East-Central Wisconsin area (GRB, MKX and ARX), that’s 15 minutes of processing required every 6 minutes for a precip mode radar cycle. On our slim budget of free equipment only, there wasn’t any way to build another machine to assist in the processing–plus, it just didn’t seem right that this task should be so taxing.
So we pressed on with the research. Theoretically, the specifications for these raw format files was available, but it was hard to get at, and then once we got to it, it was hard to understand–until one day I stumbled upon this website: http://weather.unisys.com/wxp/Appendices/Formats/NIDS.html. This page spelled out the complete syntax of some NWS product. Nick did some tinkering and found that it lined up correctly with the products! This was the specification of these files: the NIDS level 3 format.
Nick promptly started work on LibNNids, a library for decoding NIDS format data. We needed something small and fast. We wanted a utility to trim down a NIDS file to a certain span of radials as well, to trim off unneeded pieces of a NIDS file for transmission over the slow 1200bps packet radio connections. The result, written in C++, was quite lean compared to the 300MB+ GEMPAK, and much faster. Processing an image was down to 10-20 seconds from one minute.
Then right around the time Nick went to engineering school there was a breakthrough. Nick figured out how to tap into the NWS’s NOAAPORT satellite feed using an old C-band dish and a cheap DVB receiver PCI card. NOAAPORT is a 1.5Mbps stream of raw NWS products from all over the country, including many satellite products as well. He wrote NPRecv, a daemon that handles receiving the product from the DVB card, at this time. Concurrently, he was developing libradar_stp and libradar_dtp. These libraries handled tranferring the products via TCP and AX.25 streams, respectively. This allowed for clients to connect via the Internet or packet radio and receive products.
Nick later rewrote LibNNids as LibNNids 2. He also wrote a Windows GUI called RadarGUI to automatically connect to an RSTP or RDTP server to retrieve products, then process them into images using LibNNids 2.
Recently, I remembered how cool all of this stuff was and got back into it. On Nick’s radar project website you can find most of the software, much of which has been released under the GPL v2. However, without a connection to the data stream, much of the software will not be useful. As a member of Winnebago Co [WI] ARES/RACES (as well as a developer), I get access to the main feed stream as well as some of the non-GPL products, such as the GINI processor for rendering satellite images. Hope is not lost, however. You can receive products through the PTG as mentioned before, or gain access to a feed for a modest monthly price. See Nick’s webpage for more details.
So since this stuff is so cool, I’ve deployed an archiver that receives many products for the Wisconsin area from Nick’s NOAAPORT download site. You can access the archive at http://kb9qwc.net/~radar/. The archive will provide both source files and processed files. The processed files are available usually within one minute after they have left the NWS product generator.