Ticket #458 (closed Defect: fixed)
Tools that download data should retry all download errors, and allow the caller to specify the maximum retry time
|Reported by:||jjr8||Owned by:|
|Component:||Tools - Data Products - NOAA NODC||Version:|
Many of these tools use OPeNDAP via the pydap module, which in turn relies on httplib2. pydap and httplib2 raise a variety of exceptions. My original strategy was to retry exceptions that were clearly temporary problems and not retry the rest, so that those others would fail immediately. This would allow a quick failure to typical user errors, such as providing a URL to a non-existent server. Unfortunately it proven difficult to determine the complete list of exceptions that should be retried. Different kinds of failures from different servers yield different exceptions, and I find myself constantly needing to tweak the list of retryable ones.
To straighten this out and provide some simplicity, I should change the design so that all exceptions raised during connecting or downloading retried. To prevent infinite retries, a maximum retry time should be imposed. To allow the user to instruct the tool to survive a long temporary failure, the max retry time should be configurable. This will help with long batch jobs running against servers that are frequently unavailable; for example, the NODC AVHRR Pathfinder OPeNDAP server is rebooted every evening, apparently around 23:00 EST, causing it to be unavailable for several minutes.