Ticket #375 (assigned Defect)

Opened 4 months ago

Last modified 4 months ago

Predict GAM and Predict GLM tools fail with RPy_RException: ... GDAL Error 3: Attempt to read past EOF

Reported by: jjr8 Assigned to: jjr8 (accepted)
Priority: Medium Milestone: 0.8
Component: Tools - Statistics Version:
Keywords: Cc:

Description

Here is example logging output:

Executing (Predict GAM From Rasters): GAMPredictFromArcGISRasters C:\HabModExample2\Model\GAM.rdata C:\HabModExample2\OceanographyRasters\Bathymetry\etopo2v2;C:\HabModExample2\OceanographyRasters\Chl\Monthly\chl199908;C:\HabModExample2\OceanographyRasters\SST\Monthly\sst\sst199908 Bathymetry;Chlorophyll;SST C:\HabModExample2\HabitatPredictions\habitat08 C:\HabModExample2\HabitatPredictions\template # true C:\HabModExample2\HabitatPredictions\stderrors08 C:\HabModExample2\HabitatPredictions\binary08 0.275160261084 true
Start Time: Wed Jul 22 16:01:14 2009
Running script GAMPredictFromArcGISRasters...
Loaded GAM from C:\HabModExample2\Model\GAM.rdata. The model was fitted with the R mgcv package.

MODEL SUMMARY:
==============

Family: binomial 
Link function: logit 

Formula:
Presence ~ log10(abs(Bathymetry)) + s(log10(Chlorophyll)) + SST

Parametric coefficients:
                       Estimate Std. Error z value Pr(>|z|)    
(Intercept)            -18.1695     3.3549  -5.416 6.10e-08 ***
log10(abs(Bathymetry))  -1.9870     0.4842  -4.103 4.07e-05 ***
SST                      0.6532     0.1075   6.075 1.24e-09 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 

Approximate significance of smooth terms:
                        edf Ref.df Chi.sq p-value   
s(log10(Chlorophyll)) 4.147  4.147  14.91 0.00555 **
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 

R-sq.(adj) =  0.334   Deviance explained = 40.4%
UBRE score = -0.56808  Scale est. = 1         n = 303

Checking coordinate systems, extents, and cell sizes of predictor rasters and reprojecting and clipping as needed to make them conform to the template raster...
Projecting and clipping C:\HabModExample2\OceanographyRasters\SST\Monthly\sst\sst199908. Its cell size (0.0439453) does not match the cell size of the template raster (0.10000000000000001).
Projecting and clipping C:\HabModExample2\OceanographyRasters\Chl\Monthly\chl199908. Its cell size (0.083333299999999985) does not match the cell size of the template raster (0.10000000000000001).
Projecting and clipping C:\HabModExample2\OceanographyRasters\Bathymetry\etopo2v2. Its cell size (0.033333300000000003) does not match the cell size of the template raster (0.10000000000000001).
Predicting...
RPy_RException: Error in .local(.Object, ...) :
 
       GDAL Error 3: Attempt to read past EOF in C:\Temp\GeoEcoTemp_Jason\tmphplbkf\projected0a/../info/arc.dir.

Obtaining additional debugging information. This may cause this tool to hang. If you do not see any additional messages after one minute, halt its execution.
<class 'rpy.RPy_RException'>: Error in .local(.Object, ...) : 

        GDAL Error 3: Attempt to read past EOF in C:\Temp\GeoEcoTemp_Jason\tmphplbkf\projected0a/../info/arc.dir.

Failed to execute (Predict GAM From Rasters).
End Time: Wed Jul 22 16:01:42 2009 (Elapsed Time: 28.00 seconds)

The root cause of this problem is a bug in the GDAL library, as described here under the email threads titled "Reading ArcInfo binary grids with the R rgdal package fails with "GDAL Error 3: Attempt to read past EOF in..../arc.dir".

The problem occurs with versions of R for which the latest version of the rgdal package (0.6-10) includes a copy of GDAL 1.6.1. Basically, GDAL 1.6.1 fails to read ArcInfo binary grids under certain circumstances (I do not understand what these are), and this failure cascades up through the rgdal R package to MGET.

The ultimate solution is for the GDAL dev team to fix the problem, release another version of GDAL (presumably 1.6.2), and for the rgdal dev team to pick up that release and issue a new version of rgdal. Until that happens, the following workarounds may allow you to bypass this problem:

  • Use an older version of R, for which the latest version of rgdal only includes GDAL 1.6.0 or earlier. I was advised by Roger Bivand, an rgdal developer, that R 2.8.* and earlier include rgdal 0.6-8 and earlier, which include GDAL 1.6.0 and earlier.
  • When using R 2.9.*, use the install.packages command to explicitly install rgdal 0.6-8 or earlier. Roger mentioned that this version of rgdal may not even exist for R 2.9.*, so this suggestion may not work.

Change History

07/30/09 16:41:28 changed by jjr8

  • status changed from new to assigned.

I confirmed that this problem does not happen with R 2.8.1, for which rgdal 0.6-8 is the latest version of rgdal that is available. If you are hitting this problem I recommend sticking with R 2.8.1 rather than upgrading to a newer version.

The gdal release manager indicated that a 1.6.2 release candidate should be issued around 31 July 2009. If that schedule holds, then perhaps a new version of rgdal that incorporates a fixed gdal could be issued within days or weeks.

08/04/09 12:17:03 changed by jjr8

  • milestone changed from Unscheduled to 0.8.

I will resolve this as "worksforme" once I confirm that a new release of rgdal has been issued and no longer repros this problem.

08/09/09 09:08:21 changed by jjr8

Roger Bivand, the rgdal developer, prepared a private build of rgdal 0.6-11 that incorporates GDAL 1.6.2. I tested it and verified that it fixes the problem. I do not have an ETA yet from Roger as to when this will be released, but it hopefully will be soon.