Fix spurious log messages when dataset is not found during ingest

[maurizi]

When a file is not found during the run_jobs data ingest, it throws scary-looking error messages in the Papertrail logs, and causes the task to be repeatedly re-run.

As certain combinations of dataset / model do not exist, 404 errors are expected for those combinations. We should either 1) identify which combinations are not valid and attempting to download files corresponding to those combinations or 2) suppress 404 error messages when attempting to download files and not re-run the task if they occur.

Example error message:

Jun 20 14:34:18 34.194.243.89-1 management: Failed to process data for dataset NEX-GDDP model GFDL-CM3 scenario RCP45 year 2099
Jun 20 14:34:18 34.194.243.89-1 management: Traceback (most recent call last):
Jun 20 14:34:18 34.194.243.89-1 management:   File "/opt/django/climate_change_api/climate_data/management/commands/run_jobs.py", line 138, in process_message
Jun 20 14:34:18 34.194.243.89-1 management:     for var in ClimateDataYear.VARIABLE_CHOICES}
Jun 20 14:34:18 34.194.243.89-1 management:   File "/opt/django/climate_change_api/climate_data/management/commands/run_jobs.py", line 138, in <dictcomp>
Jun 20 14:34:18 34.194.243.89-1 management:     for var in ClimateDataYear.VARIABLE_CHOICES}
Jun 20 14:34:18 34.194.243.89-1 management:   File "/opt/django/climate_change_api/climate_data/management/commands/run_jobs.py", line 107, in download_nc
Jun 20 14:34:18 34.194.243.89-1 management:     s3.meta.client.download_file(BUCKET, key, filename)
Jun 20 14:34:18 34.194.243.89-1 management:   File "/usr/local/lib/python3.5/site-packages/boto3/s3/inject.py", line 130, in download_file
Jun 20 14:34:18 34.194.243.89-1 management:     extra_args=ExtraArgs, callback=Callback)
Jun 20 14:34:18 34.194.243.89-1 management:   File "/usr/local/lib/python3.5/site-packages/boto3/s3/transfer.py", line 299, in download_file
Jun 20 14:34:18 34.194.243.89-1 management:     future.result()
Jun 20 14:34:18 34.194.243.89-1 management:   File "/usr/local/lib/python3.5/site-packages/s3transfer/futures.py", line 73, in result
Jun 20 14:34:18 34.194.243.89-1 management:     return self._coordinator.result()
Jun 20 14:34:18 34.194.243.89-1 management:   File "/usr/local/lib/python3.5/site-packages/s3transfer/futures.py", line 233, in result
Jun 20 14:34:18 34.194.243.89-1 management:     raise self._exception
Jun 20 14:34:18 34.194.243.89-1 management:   File "/usr/local/lib/python3.5/site-packages/s3transfer/tasks.py", line 255, in _main
Jun 20 14:34:18 34.194.243.89-1 management:     self._submit(transfer_future=transfer_future, **kwargs)
Jun 20 14:34:18 34.194.243.89-1 management:   File "/usr/local/lib/python3.5/site-packages/s3transfer/download.py", line 353, in _submit
Jun 20 14:34:18 34.194.243.89-1 management:     **transfer_future.meta.call_args.extra_args
Jun 20 14:34:18 34.194.243.89-1 management:   File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 314, in _api_call
Jun 20 14:34:18 34.194.243.89-1 management:     return self._make_api_call(operation_name, kwargs)
Jun 20 14:34:18 34.194.243.89-1 management:   File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 612, in _make_api_call
Jun 20 14:34:18 34.194.243.89-1 management:     raise error_class(parsed_response, operation_name)
Jun 20 14:34:18 34.194.243.89-1 management: botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found