Marine Geospatial Ecology Tools
Getting Started with Marine Geospatial Ecology Tools
| MGET version: | 0.8a40 |
| Document version: | $Id: GettingStarted.xsl 848 2011-12-01 20:34:27Z jjr8 $ |
| Document status: | Complete |
| Maintainer email: | jason.roberts@duke.edu |
| MGET home page: | http://code.nicholas.duke.edu/projects/mget |
Marine Geospatial Ecology Tools (MGET), also known as the GeoEco Python
package, is an open source geoprocessing toolbox designed for coastal and marine
researchers and GIS analysts who work with spatially-explicit ecological and
oceanographic data in scientific or management workflows. MGET includes over 200
tools useful for a variety of tasks, such as downloading and sampling oceanographic
data, identifying fronts in sea surface temperature images, fitting and evaluating
multivariate statistical models by integrating ArcGIS with the R statistics
program, analyzing coral reef connectivity by simulating hydrodynamic larval
dispersal, and building grids that summarize fishing effort, CPUE and other
statistics.
MGET was designed with an emphasis on maximizing reliability and usability. We are all too familar with the abundance of undocumented, confusing, and difficult-to-install software that works great on the original developer's machine but not your own. We try to minimize these headaches by providing the same features you expect from the highest quality software, including an installer, documentation, extensive error-handling code, and adequate testing before we release new versions. We are professional software engineers who take pride in our work and hope that if you find a bug or see something that could be improved, please report it to Jason Roberts (jason.roberts@duke.edu). Thanks for your assistance!
You may notice that the Reference Documentation and the MGET source code use the word GeoEco rather than the phrase Marine Geospatial Ecology Tools. The two are synonymous.
If you have done some computer programming, you may be familiar with the need to invent words for use in source code. These words are typically short and allow programmers to quickly distinguish one library of functions from another. We selected GeoEco as the code-word for Marine Geospatial Ecology Tools because it is catchy, it shows up infrequently in Internet search engines relative to other choices such as MGET, and it does not suggest "marine". This last point may seem contrary to our mission, but want to offer our tools to terrestrial users, even if we currently market the tools mainly to marine users. Many of the existing tools are equally applicable above and below the waterline.
Minimum requirements:
Additional requirements for full functionality:
Much of MGET is written to be operating-system independent, but at this time, only Windows is supported. We will enable installation on other operating systems in a future release.
Although you must be logged on as an administrator to install MGET, you do not need to be an administrator to run the tools once they are installed.
Exit all instances of ArcCatalog, ArcMap, ArcGlobe, etc.
You can skip this step if you do not have ArcGIS. If you do have ArcGIS we recommend you take this opportunity to install the latest service pack. This is not required, but may improve the reliability and performance of the MGET tools that require ArcGIS.
MGET is implemented in the Python programming language as a package called GeoEco. Before you install MGET, you must install a compatible version of Python. You can skip this step if you have ArcGIS 9.2 or later and Python was installed by the ArcGIS setup program.
If you do not have Python you must install it now. If you have ArcGIS 9.1, you must upgrade to a newer version of Python. In either case, please see these instructions. After installing it, resume at Step 5 below.
MGET requires that the pywin32 Python package be installed for your version of Python. Pywin32 is also known as the Python Extensions for Windows.
Download and install MGET from http://code.nicholas.duke.edu/projects/mget/. Be sure to install the program that is specific to your version of Python.
Many MGET tools will be immediately functional after you install MGET. But some tools rely on other software applications and libraries. To ease the burden of getting started, MGET requires none of these to be on your machine when you install it. All MGET tools check their software dependencies when you invoke them, and report error messages and brief installation instructions if required software is not installed.
If you prefer to get started right away, you can rely on these error messages to prompt you for additional software installations when they are needed. If you prefer not to see these error messages, you can preemptively install all of the additional software now.
If you need to install Python in order to use MGET, these instructions might help you. But before you install it there are a few things you should know about it and ArcGIS.
Python version numbers
Each release of the Python language interpreter is assigned a version number of the form X.Y, where X is the major number and Y is the minor number. The major number is incremented when there is a fundamental change to the language. The minor number is reset to zero whenever the major number changes, and is incremented when new features are added to the major release.
Occasionally, the Python development team will issue a "bug fix" release. These releases have the form X.Y.Z where Z is the bug fix release number, starting with 1. For example, 2.5.1 is the first bug fix release for Python 2.5.
Multiple releases of Python can be installed on a Windows computer at the same time. For example, you can have Python 2.1 and 2.5 installed simultaneously. But whenever you install a bug fix release, it overwrites the installation having the same X.Y number. For example, If you have Python 2.5 installed and you install 2.5.1, it will overwrite your 2.5 installation and now you will only have 2.5.1 on your machine. If you then install 2.5.2, you will only have 2.5.2 on your machine.
Unless you override the installation options in the Python setup program, Python releases are installed to C:\PythonXY. For example, if you install Python 2.1 and 2.5, they will be installed to C:\Python21 and C:\Python25. When you install bug fix releases, they overwrite the files in these directories with updated versions. We recommend you use the default installation directory C:\PythonXY unless you are an expert user.
Windows file association determines which Python version is used to execute scripts
When you have multiple versions of Python installed, Windows must decide which version to use when you double-click a Python (.py) script to run it from Windows Explorer. Windows maintains a "file association" database that associates programs to file types based on their file extensions. In general, whenever you install new software, the setup program modifies the file associations to tell Windows to use the new program to open the types of files it works with. Whenever you install a version of Python, it reconfigures the file associations so that version of Python will be used to execute scripts. For example, if you have Python 2.1 and then install 2.4, from that point forward 2.4 will be used to run Python scripts. If you then installed 2.3, it would be used even though 2.4 is installed.
You can change the file associations manually; see Microsoft Knowledge Base article 307859. To change from Python 2.3 to 2.4, for example, you would change the associations for .py, .pyc and .pyo files from C:\Python23\python.exe to C:\Python24\python.exe and .pyw from C:\Python23\pythonw.exe to C:\Python24\pythonw.exe. If you are uncomfortable doing this, you can always reinstall Python 2.4 to restore the file associations.
The main things to remember are:
ArcGIS 9.2 and prior versions rely on Windows file associations
ArcGIS 9.2 and prior versions use the Windows Command Processor (cmd.exe) to execute Python scripts. cmd.exe uses Windows file associations to determine which program should be used to execute .py files. Thus, unless you have manually modified your file associations, these versions of ArcGIS will execute Python scripts using the version of Python that was most recently installed.
ArcGIS 9.3 and 10 rely on Windows file associations only when the Run Python script in process option is not checked
ArcGIS 9.3 introduced a new option for script tools called Run Python script in process. This option is specified on a tool-by-tool basis. This option improves performance dramatically and ESRI recommends that script developers use it whenever possible. Most MGET tools use this option, although a few do not (for reasons that I will not discuss here).
If this option is enabled, ArcGIS 9.3 always invokes Python 2.5, which comes with ArcGIS 9.3. Similarly, ArcGIS 10 invokes Python 2.6. It does not matter if a different version of Python was installed more recently.
If this option is disabled, ArcGIS 9.3 and later versions rely on file associations just like 9.2 and prior versions. This behavior can lead to unexpected results when the most recently installed version of Python is not that expected by ArcGIS (i.e. 2.5 for ArcGIS 9.3, 2.6 for ArcGIS 10): the "in process" tools will run with the version of Python tied to that version of ArcGIS but the "out of process" tools will run with the version of Python that was most recently installed.
Which version of Python should I use if I have ArcGIS?
ArcGIS 9.1 users:
ArcGIS 9.1 shipped with Python 2.1. MGET requires Python 2.4 or later. If you do not have a preference, we recommend Python 2.5. We used ArcGIS 9.1 with Python 2.5 for several years without problems.
Important for ArcGIS 9.1: Do not uninstall your existing version of Python 2.1! Even though ArcGIS 9.1 relies on Windows file associations to select the version of Python for script execution, it explicitly checks for the presence of Python 2.1. It also checks for the "Python 2.1 combined Win32 extensions" (the win32all package). If you accidentally uninstall these, you can download them from here: Python 2.1.3, win32all. After reinstalling them, you must restore your file associations to your later version of Python by reinstalling it or manually editing the associations.
ArcGIS 9.2 users:
According to ESRI Technical Article 31912, ArcGIS 9.2 is "hard wired" to work with Python 2.4.1, and using any other version is not supported. Based on the strong wording of ERSI's article, we recommend you stick with Python 2.4.1 unless you are an expert programmer.
In our experience, it is safe to install a more recent bug fix release of Python 2.4. We have used Python 2.4.4 with ArcGIS 9.2 for years without problems. You are probably safe to upgrade to 2.4.4 even if you are not an expert programmer. The Python rules for bug fix releases are fairly strict and it is unlikely that a newer bug fix release would break ArcGIS geoprocessing.
ArcGIS 9.3 users:
ArcGIS 9.3 installs Python 2.5.1. Just as ArcGIS 9.2 is "hard wired" to work with Python 2.4.1, ArcGIS 9.3 is hard wired to 2.5.1. But 9.3 is integrated even tighter with Python 2.5 than 9.2 was with Python 2.4 (see discussion above about how 9.3 uses file associations to invoke the Python interpreter only for scripts that are not "in process" tools). Because of this, we strongly recommend you stick with Python 2.5 if you have ArcGIS 9.3.
In our experience, we have found that Python 2.5.4 seems to work fine with ArcGIS 9.3. You are probably safe to upgrade to it even if you are not an expert programmer.
ArcGIS 10 users:
ArcGIS 10 installs Python 2.6.5. Just as ArcGIS 9.2 and later are "hard wired" to work with specific Python releases, ArcGIS 10 is hard wired to 2.6.x. Note that ESRI explicitly supports any bugfix release of Python 2.6 (e.g. 2.6.6). We strongly recommend you stick with Python 2.6 if you have ArcGIS 10.
What if I don't have ArcGIS?
We recommend the latest version of Python that is supported by MGET. At the time of this writing, it was Python 2.6.6.
Python installation procedure
import sys
print 'Python ' + sys.version
print 'Press Enter to close this window...'
sys.stdin.readline()
If you are a Python programmer, this section may be important to you. If not, you can probably skip this section.
"Aggregated modules"
MGET utilizes a number of Python modules developed by others. (For a complete list,
click here.)
To simplify the installation procedure, the MGET setup program installs private copies
of several of these in the AggregatedModules subdirectory of the GeoEco
Python package installation directory (typically C:\PythonXY\Lib\site-packages\GeoEco).
When you run a MGET function
that requires one of these, the function first tries to import the module from the normal
Python locations (e.g. C:\PythonXY\Lib\site-packages). If the module is already
available in a standard location, MGET will use it. If not, MGET then appends the
AggregatedModules directory to Python's sys.path list and
performs the import again. This loads MGET's private copy.
This mechanism allows you to avoid installing these modules yourself, simplifying the steps needed to get MGET working, but allows you to control which version MGET loads by installing your own copies of these modules. This can be beneficial, but it can also be dangerous. If you install a version of a module that is not compatible with MGET, you may receive strange errors when you run MGET functions. We recommend you install a version that is no older than the Minimum Recommended Version listed in the table below. Later versions will probably work (and earlier versions may too, but are less likely).
| Module Included In MGET | Version Included In MGET | Minimum Recommended Version |
|---|---|---|
| lxml | 2.2.4 | 2.1.4 |
| numpy | 1.2.1 (Python 2.4) or 1.5.0 (Python 2.5 and 2.6) | 1.0.3 |
| pydap | 3.0.rc.8 | 3.0.rc.8 |
| pyhdf | 0.8.3 | 0.8.3 |
| pkg_resources | 0.6c9 | 0.6c9 |
| pyparsing | 1.5.1 | 1.5.1 |
MGET's copy of numpy 1.2.1, used only with Python 2.4, is compiled with SSE2 support. MGET's copy of numpy 1.5.0, used with Python 2.5 and 2.6, is compiled with SSE3 support. If your computer's processor does not support the relevant version of SSE (click the relevant SSE link to find out) then you should install your own copy of numpy. Most Intel processors released since 2001 and AMD processors released since 2003 support SSE2. Most Intel processors released since 2004 and AMD processors released since 2005 support SSE3.
MGET's copy of pydap has been customized slightly from the original release, to allow it to read certain OPeNDAP datasets that were not supported by the original release and to allow OPeNDAP downloads to be cancelled from the ArcGIS user interface. If you install your own version of pydap, MGET will use it instead and you will not gain the benefits of the customized version.
"Assimilated modules"
Similar to the "aggregated modules", the MGET setup program also installs private copies of
several modules in the AssimilatedModules subdirectory of the GeoEco Python package installation
directory. These work differently than the aggregated modules in that AssimilatedModules
is a Python package and GeoEco imports the modules within it directly
(e.g. import GeoEco.AssimilatedModules.rpy) rather than adding the
AssimilatedModules directory to Python's sys.path list.
| Module Included In MGET | Version Included In MGET |
|---|---|
| netCDF3 and netCDF4 (from netcdf4-python) | 0.9.3 |
| osgeo (i.e. GDAL and OGR) | 1.7.2 |
| rpy | 1.0.3 |
Some of these have been tweaked and are no longer identical to the original versions.
In general, you do not even need to know that the assimilated modules exist, but we document them
here in case you do have need. One situation where it might become important is if you're using GeoEco
in conjunction with other Python modules or from some process other than python.exe. Some of the
assimilated modules, notably the osgeo modules (GDAL and OGR), load a lot of common DLLs. These DLLs are
included in the Bin subdirectory of the GeoEco Python package installation directory. If you find that
you are having a "DLL hell" problem because of this, please contact us for assistance.
If you read this far, you might be wondering why some modules were "aggregated" and why some were "assimilated". This is a complicated question that is beyond the scope of this documentation. If you are interested, you're welcome to contact us to discuss it.
To uninstall it:
The best way to learn MGET is to try it out. Here are some examples.
To get help or provide us feedback, please see these instructions.
If you would like to observe the details of the processing performed by MGET tools, you can enable verbose logging.
The Reference Documentation is generated in several formats that are tailored to specific user communities:
Marine Geospatial Ecology Tools is built atop a lot of other software, much of it free. We would particularly like to thank these developers for making their excellent work freely reusable. Without your work, MGET would never have gotten off the ground. Cheers to all of you!
Development of Marine Geospatial Ecology Tools is funded by:
Except where otherwise noted, this document and the Marine Geospatial Ecology Tools software is Copyright © 2008 by Jason J. Roberts.
The terms "MGET" and "GeoEco" are synonymous with, and occasionally used instead of, "Marine Geospatial Ecology Tools".
MGET is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. MGET is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License (available in the file LICENSE.txt) for more details.
MGET makes use of several other programs graciously provided for free by other developers. MGET "aggregates" these under the terms of the GNU GPL. These programs require that their original license be reproduced when they are redistributed. Please see the file LICENSE.txt for the copyright notices and licensing details for these programs. Many thanks to these developers for their contributions.