Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Random segfaults using terra::focal on big rasters #1563

Open
vidlb opened this issue Jul 17, 2024 · 3 comments
Open

Random segfaults using terra::focal on big rasters #1563

vidlb opened this issue Jul 17, 2024 · 3 comments

Comments

@vidlb
Copy link

vidlb commented Jul 17, 2024

Hi there,

I'm trying to apply a focal function (sum) to a big raster input / dtm.
I sometimes face segmentation faults, but it doesn't seems to be a problem with the data since it may complete successfully with another attempt.

This is quite annoying since the process takes several hours and it seems it often occurs near the end. The final output can reach from 12 to 18GB (compressed), with a size up to 100 000 x 100 000 pixels.
Here is a screenshot of the error :
rcpp_error

And here is the code.

smooth <- function(inpath, outpath) {
  ######## Mean smooth DTM in a 25m window ##########
  dtm <- terra::rast(inpath)
  # Prepare output
  nodata <- -99999
  # Create kernel
  radius <- 12.5
  kernel <- terra::focalMat(dtm, radius, "circle")
  # Apply focal function and write output
  copt <- c("COMPRESS=ZSTD", "ZLEVEL=1", "PREDICTOR=3", "TILED=YES", "BIGTIFF=IF_SAFER")
  opts <- list(gdal = copt, NAflag = nodata)
  terra::focal(dtm, kernel, "sum", filename = outpath, overwrite = TRUE, wopt = opts)
}

I'm running the script in a docker container based on the rocker/geospatial:4.4 image, terra version is 1.7.78.
Any ideas ? Is my raster just too big ? It's a VRT pointing to a lot of files.
Or should I use a different setting for na.policy than the default one ?
The error message indeed contains NA related functions calls. Yet I don't understand why it doesn't always occur.

@vidlb
Copy link
Author

vidlb commented Jul 18, 2024

FYI I also tried to use focalCpp without success, I tried to adapt the weighted sum function from the docs (just needed a sum), then tried to literally copy paste the example from the docs but it always failed (verbose compiled but then failed to run with a message like "test failed").

@mdsumner
Copy link
Contributor

How big is kernel? Are you possibly asking for a radius in metres in a longlat context?

@vidlb
Copy link
Author

vidlb commented Jul 21, 2024

The kernel radius is 12.5, and my CRS is in meters. Resolution is 1m so this is a 25*25px window.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants