Generate your OSM tiles (one shot, no updates).
Follow the procedure below to:
- adapt configuration to suit your needs
- set up your local file system and download OSM raw data
- build and run a docker with all needed stuff
- import OSM data with osm2pgsql
- generate tiles with
scripts/gen-tile.py(using openstreetmap-carto and Mapnik)
This procedure is inspired by osmtilegen and scripts/gen-tile.py is adapted from generate_tiles.py using argparse python module.
Some commands are runned inside the docker container (through an interactive bash session) for convenience. If you want to package below procedure in one main script and avoid interactive bash session, there is not a long way.
For a full planet import, you can check
Maybe the main constraint is your disk space as you have to host (size given for a full planet import in january 2020 with default config and a ZFS filesystem with compression=lz4 and recordsize=8k for database):
- [49 GB] - planet.osm.pbf file
- [460 GB] - PostgreSQL/PostGIS OSM database peak usage (imported with osm2pgsql). End size: 185 GB.
- [53 GB] - osm2pgsql nodes cache
- tiles directory: Geofabrik - Tile Calculator helps to estimate tiles size considering your bbox and zoom levels
.conf/config file allows you to host these resources under different base directories (for example if you want to mount multiple hard disks). Check these variables: $HOSTPATH_OSM_FILE_DIR, $HOSTPATH_PG_DATA_DIR, $HOSTPATH_OSM2PGSQL_FLATNODE_DIR, $HOSTPATH_TILES_DIR. Having an SSD disk especially for the database will fast up the process.
It seems that a good start is having 32GB RAM with 6 to 8 cores CPU.
- GNU/Linux Debian based system (tested with Debian 9 and Ubuntu 18.04 and 20.04).
- docker-ce version >=19 (tested with docker-ce 19.03.5 and 20.10.8), official install doc
- clone this repo (it will be mounted later as a
sourcedocker volume) - go to root directory
cd osmtilemaker/Config files are available under ./conf/ directory:
./conf/config: main config file (examples: choose your .osm.pbf file, custom your area of interest, working directories location, database name ...)./conf/postgres-settings.sh: bash script to update postgresql.conf file with desired values
Because some osm2pgsql or PostgreSQL config variables impact performance: check About performance section below to custom them to suit your needs.
- read config
. ./conf/configBe aware that you must read config files before immersing in following chapters.
- run this script until there is no error. It will check if all requirements are fullfilled and tell you what to do if necessary (as creating some directories or download an .osm.pbf file)
bash ./scripts/check_requirements.shsudo docker build \
--tag $DOCKER_BUILD_TAG \
--build-arg INSTALLWORKDIR=$DOCKER_BUILD_ARG_INSTALLWORKDIR \
dockerBefore running next command, be aware of issue #2 about the --shm-size parameter
sudo docker run -d \
--env POSTGRES_PASSWORD=$DBPG_USER_POSTGRES_PWD \
--env DBPG_USER_OSMTILEMAKER_USERNAME=$DBPG_USER_OSMTILEMAKER_USERNAME \
--env DBPG_DATABASE_NAME=$DBPG_DATABASE_NAME \
--volume $(pwd):$DOCKERPATH_SOURCE_DIR \
--volume $HOSTPATH_WORKING_DIR:$DOCKERPATH_WORKING_DIR \
--volume $HOSTPATH_OSM_FILE_DIR:$DOCKERPATH_OSM_FILE_DIR \
--volume $HOSTPATH_PG_DATA_DIR:/var/lib/postgresql/data \
--volume $HOSTPATH_OSM2PGSQL_FLATNODE_DIR:$DOCKERPATH_OSM2PGSQL_FLATNODE_DIR \
--volume $HOSTPATH_TILES_DIR:$DOCKERPATH_TILES_DIR \
--publish $DOCKER_HOST_PORT_TO_PUBLISH:5432 \
--shm-size="256MB" \
--name $DOCKER_NAME \
$DOCKER_BUILD_TAGYou could check docker logs to be sure that everythings is ok
sudo docker logs $DOCKER_NAMEsudo docker exec $DOCKER_NAME bash $DOCKERPATH_SOURCE_DIR/conf/postgres-settings.sh
# you must restart your container to restart postgres service (needed for parameters which require a restart to update, as `shared_buffers`)
sudo docker restart $DOCKER_NAME
# you can check that your postgresql.conf file has been correctly edited
sudo docker exec $DOCKER_NAME bash -c 'cat ${PGDATA}/postgresql.conf'Let's start an interactive bash session inside $SOURCE_DIR.
sudo docker exec -it --workdir=$DOCKERPATH_SOURCE_DIR $DOCKER_NAME /bin/bashNext commands must be run inside the docker container
. ./conf/configsudo -u postgres bash $DOCKERPATH_SOURCE_DIR/scripts/init-db.shmv $DOCKER_BUILD_ARG_INSTALLWORKDIR/openstreetmap-carto $DOCKERPATH_WORKING_DIR/openstreetmap-carto
# update 'dbname: "gis"' in project.mml to match your $DBPG_DATABASE_NAME
sed -i -e"s/dbname: \"gis\".*$/dbname: \"$DBPG_DATABASE_NAME\"/" $DOCKERPATH_WORKING_DIR/openstreetmap-carto/project.mml
cd $DOCKERPATH_WORKING_DIR/openstreetmap-carto/ && carto --quiet project.mml >osm.xmlBefore running next command, be aware that osm2pgsql --flat-nodes option is not used by default in ./conf/config file (see $OSM2PGSQL_OPTS variable to activate it).
# prepare command
OSM2PGSQL_CMD="osm2pgsql $OSM2PGSQL_OPTS $DOCKERPATH_OSM_FILE_DIR/$OSM_LATEST_FILE_NAME"
# check it
echo $OSM2PGSQL_CMD
# run
nohup $OSM2PGSQL_CMD >> $DOCKERPATH_WORKING_DIR/osm2pgsql.log 2>&1 &
# check log file
tail -f -n 200 $DOCKERPATH_WORKING_DIR/osm2pgsql.logGet shapefiles (processed coastline data as shapefile derived from OSM data is also needed for rendering usable map, more information at Wiki OSM - Coastline error checker)
$DOCKERPATH_WORKING_DIR/openstreetmap-carto/scripts/get-external-data.py -d $DBPG_DATABASE_NAME -U $DBPG_USER_OSMTILEMAKER_USERNAMEUser postgres must be owner of your $TILES_DIR
chown postgres:postgres $DOCKERPATH_TILES_DIRRunning next command, you may notice some warnings like "warning: unable to find face-name 'Noto Sans ...". You can ignore this warnings, more information about this issue: Missing fonts after following INSTALL.md #3655.
nohup sudo -u postgres \
python3 $DOCKERPATH_SOURCE_DIR/scripts/gen-tile.py \
--bbox $BBOX \
--bbox_name $BBOX_NAME \
--mapfile $DOCKERPATH_WORKING_DIR/openstreetmap-carto/osm.xml \
--tile_dir $DOCKERPATH_TILES_DIR \
--minZoom $MINZOOM \
--maxZoom $MAXZOOM \
--num_threads $RENDERING_THREADS \
>> $DOCKERPATH_WORKING_DIR/gen-tile.log 2>&1 &
# check log file
tail -f -n 200 $DOCKERPATH_WORKING_DIR/gen-tile.logJust in case you need to restart from a fresh docker instance
sudo docker stop $DOCKER_NAME
sudo docker rm $DOCKER_NAME
# You must remove $HOST_VOLUMES_BASE_DIR by yourself (don't use variables in commands to avoid errors)These commands must be run outside docker
Requirements: you must have firefox and a web server running on localhost.
Move your tiles to your local webserver and then open ol-example.html (you could also use a symlink to avoid moving your tiles).
sudo mv $HOSTPATH_TILES_DIR /var/www/html/local-tiles
firefox ol-example.html- about osm2pgsql
- about osm2pgsql and PostgreSQL
It's useful to know how many nodes OSM database contains: Wiki OSM - Database statistics.
Below are some suggestions to customize configuration variables which impact osm2pgsql/PostgreSQL/gen-tile.py performances. It could be hard to know which values to choose as all provided advices throug links above are sometimes conflicting. If you think it could be improved, don't hesitate to contribute.
In ./conf/config file
OSM2PGSQL_OPTS--slim: if you have enough RAM to store all nodes, you can consider to drop--slimoption to speed up the import, according to Volkerschatz -osm2pgsql usage, each cached node requires 8 bytes of cache, plus about 10% to 30% overhead, so for ~5,7 billion nodes we need about 60GB RAM--drop: to use with--slimbecause we do not need updates--flat-nodes: "a binary file is used as a database of node locations". Useful to reduce database size and specify a file location (you can host it on a different hard disk). Warning: "This should only be used on full planet imports or very large extracts (e.g. Europe)" because "the file takes approximately 8 bytes * maximum node ID, or more than 50 GiB, regardless of the size of the extract." from osm2pgsql usage docs)--cache: "The rule of thumb in slim mode is as follows: use the size of the PBF file you are trying to import or about 75% of RAM, whatever is smaller. Make sure there is enough RAM left for PostgreSQL. It needs at least the amount of shared_buffers given in its configuration."(from osm2pgsql usage docs). Unit is MB.--number-processes: osm2pgsql usage docs offical doc says "This should typically be set to the number of CPU threads, but gains in speed are minimal past 8 threads.". So we use as many threads as CPU cores we have (or more if your processor is hyperthreaded,lscpucommand could be useful to know more about your processor)--disable-parallel-indexing: not used in default config file, but could be useful ("disables the clustering and indexing of all tables in parallel. This reduces disk and RAM requirements during the import, but causes the last stages to take significantly longer.")
RENDERING_THREADS: "rendering threads to spawn in gen-tile.py, should be roughly equal to number of threads available" says original generate_tiles.py tiles rendering script
In ./conf/postgres-settings.sh file: maybe use Paul’s Blog - New Server PostgreSQL Tuning recommendations.
About filesystem settings: using a ZFS filesystem for the hard disk hosting PostgreSQL database could speed up the import and reduce disk space usage (see Paul’s Blog - ZFS Settings for Osm2pgsql). Recommended parameters are recordsize of 8K (run zfs set recordsize=8k $your_pool) and lz4 compression (run zfs set compression=lz4 $your_pool)
Note: feel free to add your own benchmark below, even with few details!
Total: 415 hours ; 17,3 days ; ~ 13 957 698 tiles ; average speed : 9,34 tiles/sec ; 24,8 GB
Details:
- from level 1 to 10 : 40 hours ; 1,7 days ; ~ 850 000 tiles ; vitesse moyenne : 5,9 tiles/sec ; 870 MB
- level 11 : 81 hours ; 3,4 days ; ~ 2 623 488 tiles ; average speed : 9 tiles/sec ; 1,9 GB
- level 12 : 294 hours ; 12,3 days ; ~ 10 484 210 tiles ; average speed : 9,9 tiles/sec ; 22 GB
