Extracting ArcGIS rasters from the HYCOM global ocean model's 4D netCDFs

THIS EXAMPLE IS UNDER CONSTRUCTION

The  Hybrid Coordinate Ocean Model, or HYCOM, is a sophisticated, high resolution system for simulating ocean physics. HYCOM is a set of equations refined over many years that describe the effects of the tides, winds, earth's rotation, and many other factors on the flow of water. Using supercomputers, the HYCOM team executes these equations at fine spatial and temporal scales to produce daily 3D snapshots of oceanographic variables such as temperature, salinity, and current velocity. At the time of this writing, HYCOM had been applied to global ocean simulations at 1/12º resolution and several ocean basins at higher resolution.

Acknowledgement: Thanks to Alan Wallcraft from the Naval Research Laboratory for providing critical hints that made this example possible.

Advantages of HYCOM over satellite data

• HYCOM resolution is similar to satellite resolution. The 1/12º simulations have a cell size of approximately 8.9 km at the equator. This is not as good as the popular global SST products; NODC AVHRR Pathfinder 5.0 and MODIS have cell sizes of 4 to 5 km. But it is far better than the popular Aviso geostrophic currents product, which has a 37 km cell size.

• HYCOM images are cloud-free. Daily satellite images are often very, very cloudy.

• HYCOM is 3D. Satellite images only provide data for the ocean surface.

Disadvantages of HYCOM

• HYCOM is a model, not reality. While HYCOM has a high spatial resolution, is very sophisticated, has been under refinement for years, has been tested extensively against in situ measurements and satellite estimates, and uses assimilation to improve its accuracy, it is important to keep in mind that HYCOM is a model. If you have in situ or satellite data available, we recommend you compare it against HYCOM output and form your own opinion about whether HYCOM provides enough accuracy for your situation.
  
  Below is visual comparison of relatively cloud-free GOES satellite SST images and corresponding HYCOM SST images for the Gulf Stream. The top HYCOM image is from the  HYCOM Global 1/12° Simulation (expt_05.8), while the bottom one is from the  HYCOM + NCODA Global 1/12° Analysis (expt_90.8). As you can see, both HYCOM images resemble the GOES images at a broad spatial scale, but the bottom HYCOM image shows a better resemblence at finer scale. Also, both HYCOM images appear to deviate from the GOES images in the hottest and coldest areas by as much several degrees.
  
  These differences may or may not be important, depending on how the data are used. In showing these comparisons, we make no claims about whether HYCOM output might or might not be suitable for your analysis. We simply urge you to make your own comparison and decide for yourself. An important difference between the two HYCOM datasets shown here is that the top one was a "free run" that simulated the global ocean without attempting to increase accuracy by assimilating in situ or satellite measurements, while the bottom one did use assimilation. The bottom one is also several years newer than the top one and is probably based on improved science. Finally, it is very difficult to model the fine scale structure and dynamics of the Gulf Stream, so it is not surprising to find that HYCOM does not match the satellite at fine scales. Better correspondance might be observed elsewhere, in less dynamic regions.

• HYCOM data are available for limited time ranges. At the time of this writing, global simulations were available from 2003 to the present, a north and equatorial Pacific simulation was avialble for 1979-2003, and a Gulf of Mexico simulation was available from 2008 to the present. Data that incorporate the latest assimilation techniques are only available for the most recent years.

• HYCOM provides only physical variables. These variables typically include temperature, salinity, u and v current vectors (eastward and northward velocity), sea surface height, and various properties of the mixed layer. For biological variables such as chlorophyll density or primary and secondary productivity, you must use products estimated from satellite data or other ocean models such as ROMS-CoSINE.

• Global HYCOM simulations use a complicated coordinate system. As discussed below, global HYCOM simulations use three different coordinate systems, making it difficult to import these data into a GIS as raster data. Most of the complexity of this example relates to this problem; please see below for details.

The structure of HYCOM output

HYCOM output is structured as a time series of snapshots of the state of simulated region. At each time period, typically 1 day, there are a set of 2D grids that provide ocean surface parameters such as sea surface height, thickness of the mixed layer, and so on. There are also a set of 3D grids that specify temperature, salinity, and u and v current vectors at a series of depths. There are typically 33 depth layers: 0, 10, 20, 30, 50, 75, 100, 125, 150, 200, 250, 300, 400, 500, 600, 700, 800, 900, 1000, 1100, 1200, 1300, 1400, 1500, 1750, 2000, 2500, 3000, 3500, 4000, 4500, 5000, and 5500 meters. Although HYCOM executes the simulation at a variety of depths using a sophisticated algorithm, the model outputs are usually interpolated to these 33 common depths, for consistency with other global ocean datasets.

Acquring HYCOM output

At the time of this writing, HYCOM output could be acquired from  http://www.hycom.org as OPeNDAP datasets in a THREDDS catalog and as series of netCDF files from an FTP or HTTP server. If you are familiar with OPeNDAP and can write the code necessary to acquire data through it, I suggest you use it, particularly if you only need a small subset of the data.

In the project that gave rise to this example, I needed to acquire four years of temperature and currents data for all 33 depth layers. This worked out to be about 9 TB of data. I found that it was faster to download netCDF files than go through OPeNDAP for that much data. Although the HYCOM FTP server appeared to impose a throughput limit of 1.25 MB/s per FTP download, I was able to run 15 simultaneous downloads with SmartFTP and maintain an overall throughput of 20 MB/s. If you have a fast Internet connection, this may a good way to acquire data quickly. The remainder of this example assumes you will also use netCDF files.

On the HYCOM FTP server, the directory housing each dataset was organized like this:

The data directory contained the model output. The 2d subdirectory contained one netCDF file per day. Each of these contained the 2D variables representing the state of the ocean surface, as you can see in this  netCDF header. The salt, temp, uvel, and vvel subdirectories contained the salinity, temperature, eastward current velocity, and northward current velocity data, respectively. These also contained one netCDF per day, as shown above. Each netCDF contained a single physical variable (salinity, temperature, u, or v) as well as several auxiliary variables representing the grid coordinates, as you can see in this  netCDF header. Each physical variable had four dimensions, time, depth, x, and y, with just one time slice but 33 depth slices.

The topo directory contained four files:

• regional.depth.a contained the bathymetry grid used by the simulation as a 2D binary array of big endian 32-bit IEEE 754 floats in column-major order (i.e. Fortran order).
• regional.depth.b was a text file specifying the dimensions of regional.depth.a.
• regional.grid.a contained grids specifying the latitudes and longitudes of the centers of the HYCOM grid cells.
• regional.grid.b was a text file specifying the dimensions of regional.grid.a.

The HYCOM User's Guide, available on the  HYCOM documentation page describes these files in detail.

Georeferencing HYCOM output

One of the biggest challenges in working with HYCOM output is understanding how it is georeferenced and getting the data into a GIS-compatible raster format. The HYCOM User's Guide provides some essential information but is written for oceanographers who will be running HYCOM, not for ecologists who will consume HYCOM output. The essential parts are chapter 3: The HYCOM Grid, section 2.3: I/O File Formats in HYCOM, and section 5.1: File "regional.grid.[ab]". From these, some hints from Alan Wallcraft, and extensive experimentation, I was able to understand how HYCOM data is georeferenced and develop a strategy for getting it into ArcGIS-compatible format.

HYCOM simulations are either global or regional. This discussion concerns mainly global simulations. I have not looked at any regional simulations yet so I don't know how much of it applies to them. I will say a few words about them at the end.

The main difficulty with global HYCOM simulations is that they use three different map projections in one 2D grid:

The main difficulty is that

TO BE CONTINUED...