Add support for ArcGIS 9.3

[jjr8]

ESRI released ArcGIS 9.3 this summer. It appears they tried hard to make it backward compatible with geoprocessing scripts developed for 9.2, but there are some problems.

Here are issues I know about that will require MGET code changes:

• 9.3 can host the Python interpreter directly in the ArcGIS application process (ArcCatalog.exe or ArcMap.exe), rather than creating a separate python.exe process. This improves performance greatly, but there are some caveats:
  • A 9.3 documentation page titled "Running a script in process" says: "Only scripts that create the geoprocessor object using arcgisscripting.create()...can be run in process. Scripts that use dispatch...will not be run in process, even if the Run Python script in process is checked." It is hard to know what to make of this, other than it is probably not a good idea to create the geoprocessor with win32com from in-proc scripts. MGET currently uses win32com. This is done even on ArcGIS 9.2 due to the ArcGIS bug described in ticket #152. The bug still exists with 9.3.
    • To work around #152, I am going to allow tools to specify that they require the COM Automation geoprocessor, even if they're going to run in-proc. Moving them out of proc imposes a huge perf degradation, as described below.
  • When a tool is run out-of-proc, all calls to the geoprocessor are very slow--much slower than Arc 9.2--including things as basic as AddMessage. It is as though a limiter has been placed on the geoprocessor so that it will only execute about 5-10 calls per second. There appears to be no workaround to this. It is particularly bad for code that uses cursors: each call takes 0.1 or so, and cursor throughput is limited to a paltry 10 rows per second.
  • The Run Coral Reef Connectivity Simulation tool reports "ImportError: DLL load failed: The specified procedure could not be found." no matter if it is run in-proc or out-of-proc. This is a reoccurance of the MATLAB/geoprocessor compatibility problem that existed with Arc 9.2 and MATLAB 2007a relating to a xerces DLL. Now I have the same problem with ArcGIS 9.3 and MATLAB 2007b. Both include a custom-compiled version of xerces-c_2_7.dll, but they used different compilers. When the Connectivity Simulation tool runs, it activates the MATLAB MCR, which tries to load the DLL. Even though I moved MATLAB directories to the front of the PATH environment variable, the debugger shows that the DLL is loaded from C:\Program Files\ArcGIS\Bin. The MCR is not expecting this version of the DLL and fails to find the entry point it is looking for. The only solution seems to be to explicitly load the DLL from the MATLAB MCR directory prior to initializing the MCR, and hope that ArcGIS doesn't suffer the same problem later. To minimize the likelihood of this, the Connectivity tool should be run out-of-proc, so the MATLAB DLL is not loaded into the ArcGIS processes. I tested all of this and it seems to work.

• MGET tools currently call Python's sys.exit(0) when the tool completes successfully. This causes the Python interpreter to raise exceptions.SystemExit, which ArcGIS interprets as a failure. Rather than doing this, the tools should just return normally.

• When displaying a script tool from a 9.1 toolbox, ArcGIS 9.3 does not display parameters grouped in "categories" in collapsable groups. To get these parameters in collapsable groups, the tools need to use the 9.3 ToolValidator infrastructure.

• When an exception is raised by an in-proc tool, not all of the debugging messages appear in the ArcGIS progress window. This is due to ArcGIS bug NIM036130 - Print statements inside an in-proc script tool will lead to a "<type 'exceptions.IOError'>: [Errno 9] Bad file descriptor" error, which ESRI says will be fixed in 9.3 SP1. I can write a workaround that does not write log messages to the console (stdout) when Arc 9.3 with no service pack is running.

• The way that ArcGIS hosts the Python interpreter for in-proc tools prevents you from running more than one MGET Statistics tool per session. When you try to load a second one, you get a RuntimeError: Only one R object may be instantiated per session. This problem may be due to the design of Python and rpy more than the design of ArcGIS. R.__init__ in rpy.py calls _rpy.r_init in _rpy.pyd. _rpy.pyd is a Python extension DLL. Because Python does not really support completely unloading an imported module, _rpy.pyd remains in memory even after the Python interpreter has exited in the ArcCatalog.exe or ArcMap.exe process. Then, when the interpreter runs again, and imports rpy again, R.__init__ in rpy.py calls _rpy.r_init again. Since _rpy.pyd was never unloaded, a _rpy.r_init complains it was called for a second time. To solve this, I will make all of the tools that invoke R out-of-proc tools. This will expose them to the performance problem described above. For now, I have no solution to that.

These are issues that did not turn out to require MGET code changes:

• From  "Technical Article 34873: "In ArcGIS 9.3, the Geoprocessing Programming Model changed the way List methods produced results. In ArcGIS 9.2, they were outputted as enumerations, where the properties of Next and Reset were needed to iterate through the list. ArcGIS 9.3 now produces Python Lists that do not require Next and Reset." It turns out that is not the only difference between 9.3 and 9.2. See also  Differences between geoprocessor versions and  Creating the geoprocessor object. Fortunately, these changes only come into effect when you create the geoprocessor using arcgisscripting.create and you specifically request the ArcGIS 9.3 inteface, like this:

gp = arcgisscripting.create(9.3)

The plan for MGET is to continue to use the 9.2 geoprocessor interface until ESRI stops supporting it or adds new functionality that can only be accessed via the new interface. This will likely not happen until Arc 9.4 at the earliest.

• In 9.3, I found that every once in a while gp.Describe(...).DataType returns 'File' when a file system raster is passed to Describe. In 9.1 and 9.2 it returned 'RasterDataset'. This problem affects MGET's parameter validation code, which tests that inputs are rasters when a tool's metadata requires a raster. I believe this is a bug in 9.3. Based on my previous experience with filing bugs with ESRI, I highly doubt they will fix it, especially because I can't produce a 100% repro scenario. For now, it seems infrequent enough that I will not create a workaround.

[jjr8]

The final code changes were checked in as part of [302] and will appear in MGET 0.6b1.