Exception trying to read large 3.9 GB grib file #41
Comments
|
Hi David, This is a perfectly good place to ask Metview questions! You can set this environment variable before starting the Python process, then you should get more output.
I suspect you're encountering an internal buffer limit - you might see a message like this: in which case you can set this environment variable to the size of the GRIB file before importing Metview, e.g. Let us know if this solves your problem. If not, then it is possible to split the file using ecCodes command-line tools first (grib_filter). Cheers, |
|
Hi Iain! I'm glad, thanks for the help. I turned on debug messages, and while I tried setting a large buffer size, I think it's actually an issue with the mars service timing out after about 5 minutes and 30 seconds. Here are some lines I think are relevant from the debug messages (full output at bottom of this post): event - INFO - 20220206.222550 - mars : Timeout = 300 Alive = 0, Max = 1 I took a guess at what the mars timeout variable might be with: Thanks, MetviewInvoker: Invoking Metview pool - INFO - 20220206.222550 - [mars@37158] - Object /Process@37129/PythonScript, line 0:0 [READ] not cached Mars - INFO - 20220206.222550 - GRIB conversion needed... |
|
Hi @SpechtacularDave , Try setting METVIEW_TIMEOUT=10 - the value is in minutes. It's strange, as I cannot reproduce the problem here, but you're not the first to report it, so something is going on! That said, from your output log, it does look like you pressed CTRL-C at some point...! Be aware that Metview essentially runs microservices, and when a program is finished, it kills any running services and you get various warning messages about services being killed or not being able to communicate with the server, so don't give up on the program when you see these messages, they might just be part of the normal termination process! I hope this helps you to get further. Best regards, |
|
Hi Iain, Ah, you're correct, I did hit ctrl-c because I thought metview had died because of the debug messages. However, when I let it run the entire time, I'm still getting an exception after about 7-8 minutes (timeout was set to 100), even with all the recommended environment variables set,. Below is the full debug output. Does this shine any light on what's going wrong? If it adds anything, I am able to open the file with ecCodes. (metview) david@david-OptiPlex-7020:~/anaconda3/envs/metview/bulkdownload$ python3 bulktest.py Mars - INFO - 20220211.175849 - GRIB conversion needed... |
|
Hi David, But I'd also be very interested to see the CDS request so I can replicate the scenario. |
|
Iain, Here is my query: import cdsapi c = cdsapi.Client() c.retrieve( |
Hello,
Please let me know if there is a better place to ask a question like this one. I wasn't able to find a forum or user group for metview, but if there is a resource like that which would be better suited, please direct me and feel free to close this issue.
Using the metview python api, I am trying to read a 3.9 GB grib file that I retrieved from the the CDS API. But when I run the read() command on the file, after a couple minutes I receive the following:
Exception: Metview error: Service mars@8600 died before completing the request
This happens when trying to read the grib file with:
fs = mv.read(source="1980.grib")
And it also happens even if I just try to extract a small area:
fs = mv.read(source="1980.grib", area=[30, -100, 29.75, -99.75])
The file contains data for 9 parameters from ERA5, ["10u", "10v", "2d", "2t", "i10fg", "msl", "ptype", "tcc", "tp"] that all have hourly values for a whole year, for the area of [50, -125, 24, -66], which covers the entire USA country. I believe the two precipitation parameters are stored in grib2 messages, while the rest are grib1.
Is there a file size limit for Metview? If so, can you recommend a way to split my files containing grib1 and grib2 messages into smaller files? Or is it necessary to compute some kind of index on the file before reading it?
If it matters, I don't need to read the whole file at once, I just need to be able to extract a small area at a time, so that I can combine it with data from another grib file for the next year, so that I can ultimately compute some statistics on the combined dataset for each point.
Thank you, - David
The text was updated successfully, but these errors were encountered: