User:mmd/Overpass API - experimental corner

From OpenStreetMap Wiki
< User:Mmd(Redirected from User:Mmd/Overpass API)
Jump to navigation Jump to search


Features

Selected list of features 0.7.59_mmd:

  • PBF/OPL format output
  • ICU regular expressions
  • PCRE2 (+JIT) regular expression engine
  • Ad-hoc area creation on any closed way/relation
  • FastCGI support
  • Index caching
  • Changeset based (changed: ) filtering to process huge changesets in Achavi
  • epoll based dispatcher processing
  • map_demo: super large extracts with support of > 100mio. nodes
  • Lazy (if: ) filter evaluation for boolean expressions
  • Hybrid array/bitmap container for better memory efficiency of large query statements (subset of features found in roaring bitmaps, https://arxiv.org/pdf/1603.06549.pdf) (example query)
  • Avoid object instantiation to access node/way/rel properties wherever possible in template backend
  • PBF support for update_from_dir/update_database (parameter --use-osmium), both planet file and diff
  • Parallel processing support for update_from_dir/update_database (parameter --parallel)
  • Area caching
  • around: bbox based pruning
  • Separate nodes table for fast lookup of tagged nodes (about 2.6% of all nodes)
  • Improved dispatcher process security for use in multi-user environments (https://github.com/drolbr/Overpass-API/issues/247)
  • Improved dispatcher signal handling for SIGTERM and SIGINT, same behavior as --terminate command line parameter
  • Way nodes stored as varint using protozero (original idea: https://github.com/drolbr/Overpass-API/pull/250)
  • Based on 0.7.56 - oversized_backend_objects branch


  • Full attic database creation 4-5 days (currently: > 24 days)
  • Full day simulation w/ 1.2mio queries: 6.5 hours (6x parallel) (currently: 18 hours)
  • node(xxx);out meta;: >1500 requests/s

Demos

Installation

Based on https://overpass-api.de/full_installation.html, only deltas described here.

Compiling

Additional dependencies on FastCGI, BZ2 and ICU.

git clone https://github.com/mmd-osm/Overpass-API.git
cd Overpass-API
git checkout test759
git submodule update --init

sudo apt-get update -qq || true
sudo apt-get install -y expat libexpat1-dev liblz4-dev libfcgi-dev zlib1g-dev libbz2-dev libicu-dev libfmt-dev libpcre2-dev
  
pushd src/
chmod u+x test-bin/*.sh
autoscan
aclocal
autoheader
libtoolize
automake --add-missing
autoconf
popd
mkdir -p build
cd build
../src/configure CXXFLAGS="-Werror=implicit-function-declaration  -D_FORTIFY_SOURCE=2 -fexceptions -fpie -Wl,-pie -fpic -shared -fstack-protector-strong -Wl,--no-as-needed -pipe -Wl,-z,defs -Wl,-z,now -Wl,-z,relro -fno-omit-frame-pointer -flto -fwhole-program -lfmt -O2" LDFLAGS="-flto -fwhole-program -lpcre2-8" --prefix=$EXEC_DIR --enable-lz4 --enable-fastcgi --enable-tests
make V=0 -j3
make install


configure CXXFLAGS="-I/usr/local/include -D_FORTIFY_SOURCE=2 -fexceptions -ggdb -fpie -Wl,-pie -fpic -shared -fstack-protector-strong -Wl,--no-as-needed -pipe -Wl,-z,defs -Wl,-z,now -Wl,-z,relro -fwhole-program -flto=1 -fno-omit-frame-pointer -std=c++17 -O2  -lfmt -march=skylake"  --enable-lz4 --enable-fastcgi LDFLAGS="-fwhole-program -flto=1"

(optionally use tcmalloc)

FastCGI

supervisord config

/etc/supervisor/conf.d/overpass.conf

[fcgi-program:interpreter]
socket=unix:///var/run/interpreter.socket
socket_owner=www-data
socket_mode=0660
environment=
    OVERPASS_FCGI_MAX_REQUESTS=10000,
    OVERPASS_FCGI_MAX_ELAPSED_TIME=900,
    OVERPASS_REGEXP_ENGINE="PCREJIT"
command=/home/user/osm3s/fcgi-bin/interpreter
numprocs=6
priority=999
process_name=%(program_name)s_%(process_num)02d
user=www-data
autorestart=true
autostart=true
startsecs=1
startretries=3
stopsignal=QUIT
stopwaitsecs=10
redirect_stderr=true
stdout_logfile=/var/log/interpreter.log
stdout_logfile_maxbytes=10MB

Apache config

Depends on mod_proxy_fcgi

Forward calls to /api/interpreter to local socket managed by supervisord.

         ProxyPass /api/interpreter unix:///var/run/interpreter.socket|fcgi://localhost/api/interpreter

Overpass API 0.7.59_mmd examples

0.7.59_mmd is a heavily patched Overpass prototype version based on release 0.7.56.

Sources: https://github.com/mmd-osm/Overpass-API/tree/test759


Export highways in current bounding box in PBF format

try it yourself in overpass-turbo
[out:pbf];
way[highway]({{bbox}});
(._;>;);
out meta;

Export highways in current bounding box in OPL format

try it yourself in overpass-turbo
[out:opl];
way[highway]({{bbox}});
(._;>;);
out meta;

Export pbf

  • Uses AddLocationsToWays pbf extension
  • Demo faster attic implementations collect_attic_k and collect_attic_kv
try it yourself in overpass-turbo
[out:pbf(geom)][bbox:{{bbox}}];

retro("2020-12-20T00:00:00Z") {
  way[highway=residential][lit];
  out geom meta;
}

Fast foreach

  • Foreach w/ union - 1.5mio ways/s
try it yourself in overpass-turbo
[timeout:3600];
way[building]({{bbox}});
out count;

foreach {

  (._; .result;)->.result;
}

.result out count;

ICU regular expression support

Also see https://www.openstreetmap.org/user/mmd/diary/40197

Nodes with Chinese (Han) characters in current bounding box

try it yourself in overpass-turbo
// switch to ICU Regexp implemenation
// see http://userguide.icu-project.org/strings/regexp

[regexp:ICU];


node({{bbox}})["name"~"[\\p{script=Han}]"];

out meta center;


The development Overpass API instance allows the use of ICU regular expressions, which do support \\U000##### escapes. ICU support on this endpoint is experimental. With this addition, other ICU-supported escapes are recognized (using double backslashes in Overpass QL regular expressions), including \\p{Unicode property name} which will match any Unicode character having the specified Unicode property. Here is an example query that finds CJK characters outside the Basic Multilingual Plane. See the blog post and the GitHub issue tracking this potential feature.

Gas stations per NIST code

(takes more than 2h on 0.7.56, see issue https://github.com/drolbr/Overpass-API/issues/568)

try it yourself in overpass-turbo
[out:csv(fips, name, total)];

// All counties with FIPS 6-4 county codes
area["nist:fips_code"~"^[0-9]{5}$"];

foreach->.county(
  // Collect all matching features in the current county
  way["amenity"="fuel"](area.county);
  
  // Sum nodes, ways, and relations together and group by county
  make count fips = county.set(t["nist:fips_code"]),
             name = county.set(t["name"]),
             total = count(nwr);
  out;
)

Restaurants with nearby public phone (global)

  • Restaurant name starts with a "P"
  • Phone within 100m distance


try it yourself in overpass-turbo
[out:json][timeout:120];

(
  node["amenity"="restaurant"]["name"~"^P"];
  node["amenity"="bar"]["name"~"^P"];
  node["amenity"="pub"]["name"~"^P"];
  node["amenity"="cafe"]["name"~"^P"];
)->.rest;

nwr["amenity"="telephone"]->.phone;
  
node.rest(around.phone:100);
 
out body;
  • On f561bbe88: 50 secs (vs. 2.5 hours without patch)

No-U-Turn Relations with from = to member

try it yourself in overpass-turbo
[timeout:30];

// No-u-turn restrictions mit from = to

rel[restriction=no_u_turn][type=restriction]({{bbox}})
   (if:count_by_role("via") == 1  &&
       count_by_role("from") == 1 &&
       count_by_role("to") == 1);

foreach -> .rel(
  
  way(r.rel:"from") -> .w_from;
  way(r.rel:"to")   -> .w_to;
 
  if ( w_from.min(id()) == w_to.min(id()) )
  (
     .rel out geom meta; 
  );
);


osm2pgsql relation issue

See https://lists.openstreetmap.org/pipermail/dev/2021-February/031047.html, and https://github.com/openstreetmap/osm2pgsql/issues/1394

https://dev.overpass-api.de/blog/total_0_7_56.html#memberpos explains first/last node of a way in more detail.

try it yourself in overpass-turbo
[timeout:3600];
rel[type=boundary](if:count_by_role("inner") == 0 && count_by_role("outer") == 2) -> .rels;
foreach .rels -> .rel {
  way(r.rel)->.ways;
  node(w.ways:1) ->.firstnodes;
  node(w.ways:-1)->.lastnodes;
  if (ways.count(ways) == 2 && 
      firstnodes.count(nodes) == 1 &&
      lastnodes.count(nodes) == 1) {
    .rel out;
  }
}


Ghost line start- and endpoints

The following query visualizes ghost line start- and endpoints. In most cases, 2 out of 3 nodes are involved, sometimes even 3 out of 3 nodes.

try it yourself in overpass-turbo
[timeout:3600];
rel({{bbox}})[type=boundary](if:count_by_role("inner") == 0 && count_by_role("outer") == 2) -> .rels;
foreach .rels -> .rel {
  way(r.rel)->.ways;
  node(w.ways:1) ->.firstnodes;
  node(w.ways:-1)->.lastnodes;
  if (ways.count(ways) == 2 && 
      firstnodes.count(nodes) == 1 &&
      lastnodes.count(nodes) == 1) {
    node(w.ways:-1);out;
    node(w.ways:2);out;
    .rel out geom;
  }
}

Queries with very large tag cardinality

Fails on 0.7.56 with Query run out of memory using about 2048 MB of RAM.

try it yourself in overpass-turbo
way[building]                    // 366'209'402 objects
   [source]                      // 153'587'268 objects
   [highway]                     // 146'009'103 objects
   [name]                        //  51'050'056 objects
   ["addr:housenumber"]          //  46'319'742 objects
   ["addr:street"]               //  44'955'803 objects
   ["addr:city"]                 //  35'031'726 objects
   ["addr:postcode"];            //  30'249'900 objects
out  meta;


Large bbox k/v attic queries

Fails on 0.7.56 with Query run out of memory using about 2048 MB of RAM.

try it yourself in overpass-turbo
[bbox:{{bbox}}];

retro("2020-12-20T00:00:00Z") {
  way[highway=residential][lit=yes];
  out geom meta;
}

All supermarkets in Germany in PBF format

try it yourself in overpass-turbo
[timeout:900][out:pbf];
area[name="Deutschland"];
nwr[shop=supermarket](area);
out center;

All xmas features in PBF format (global)

try it yourself in overpass-turbo
[out:pbf];nwr["xmas:feature"];out qt center;

Ad hoc area creation

Buildings w/o housenumber

Find buildings without housenumber - both on way, as well as inside the way as node.

try it yourself in overpass-turbo
way[building][!"addr:housenumber"]({{bbox}});

foreach ->.pivot {
  ( 
    .pivot;
    node(w.pivot);
  );

  ( make_area [.pivot]; .result;)->.result;
}

foreach .result -> .area {

  node["addr:housenumber"](area.area);

  if(count(nodes) == 0) {
    way(pivot.area);
    out geom meta;
  }
}

landuse=residential without buildings

Find landuse=residential ways and relations without any buildings

try it yourself in overpass-turbo
way[landuse=residential]({{bbox}});

foreach ->.pivot {
  ( 
    .pivot;
    node(w.pivot);
  );

  ( make_area [.pivot]; .result;)->.result;
}

rel[type=multipolygon][landuse=residential]({{bbox}});
foreach ->.pivot {
  ( 
    way(r.pivot);
    node(w);
  );
  ( make_area [.pivot]; .result;)->.result;
}

foreach .result -> .area {

  way[building](area.area);

  if(count(ways) == 0) {
    way(pivot.area);
    out geom meta;
    rel(pivot.area);
    out geom meta;
  }
}

Unnamed landuse residential relations with at least one amenity=bench

try it yourself in overpass-turbo
rel[type=multipolygon][landuse=residential][!name]({{bbox}});

foreach ->.pivot {
  ( 
    way(r.pivot);
    node(w);
  );
  ( make_area [.pivot]; .result;)->.result;
}


node[amenity=bench](area.result)->.n;

foreach .result -> .r {

  node.n(area.r);
  
  if (count(nodes) > 0) {
    out geom;
    rel(pivot.r);
    out geom;
  }
}


Abusing barrier=line on leisure=pitch ways and relations

try it yourself in overpass-turbo
way[leisure=pitch]({{bbox}});

foreach ->.pivot {
  ( 
    .pivot;
    node(w.pivot);
  );

  ( make_area [.pivot]; .result;)->.result;
}

rel[type=multipolygon][leisure=pitch]({{bbox}});
foreach ->.pivot {
  ( 
    way(r.pivot);
    node(w);
  );
  ( make_area [.pivot]; .result;)->.result;
}

way[barrier=line](area.result);
out geom;

Various Overpass test queries

try it yourself in overpass-turbo
nwr[ele~"^-?[0-9]+(\\.[0-9]+)? ?m$"];
out tags;



try it yourself in overpass-turbo
node["addr:housenumber"]["addr:street"][!name][!source][!amenity][!attribution][ref];
out  meta;



try it yourself in overpass-turbo
[timeout:300]
[out:csv(num,key,len)];
area[name="Saarland"];
way[highway=residential][name](area);
for (t["name"])
(
  make stat num=count(ways),key=_.val,len=sum(length());
  out; 
);

Road trip

Featured in blog post https://www.openstreetmap.org/user/mmd/diary/42055

try it yourself in overpass-turbo
[maxsize:1000000000][timeout:3600][out:pbf];
(
way[highway][highway~"^(trunk|motorway|primary|secondary|tertiary)(_link)?$"]
  (around:20000,50.16634404911624, 8.633879241943358,
   49.01715821614669, 8.374327239990235,
   48.288676648581344, 7.775572357177735,
   47.768872096323875, 7.503656616210939,
   47.53482074712603, 7.630003509521484
 );
node[place=city]
  (around:20000,50.16634404911624, 8.633879241943358,
   49.01715821614669, 8.374327239990235,
   48.288676648581344, 7.775572357177735,
   47.768872096323875, 7.503656616210939,
   47.53482074712603, 7.630003509521484
 );
);

(._;>;);

out meta;

Overpass instance timestamp check

Quick and dirty script to extract current timestamp_osm_base and version number using a dummy query:

#!/bin/bash

declare -a arr=("https://dev.overpass-api.de/api_mmd/interpreter"
                "https://lz4.overpass-api.de/api/interpreter"
                "https://z.overpass-api.de/api/interpreter"
                "https://overpass.maptime.in/api/interpreter"
                "https://1.overpass.kumi.systems/api/interpreter"
                "https://2.overpass.kumi.systems/api/interpreter"
                "https://3.overpass.kumi.systems/api/interpreter"
                "https://4.overpass.kumi.systems/api/interpreter"

)

check() {

   RES_ALL=`curl -m 5 -s $1?data=%5Bout%3Ajson%5D%3Bout%3B`
   RES1=`echo $RES_ALL | jq -r ".osm3s .timestamp_osm_base"`
   RES2=`echo $RES_ALL | jq -r ".generator"`
   RES3=`date --utc +%FT%TZ`
   RES1S=`date -d "$RES1" '+%s'`
   RES3S=`date -d "$RES3" '+%s'`
   DELTA=$(( $RES3S - $RES1S ))
   DELTA_HOUR=`date -d@$DELTA -u +%H:%M:%S`
   printf "%-50s | %-20s | %10s | %-30s \n" "$i" "$RES1" "$DELTA_HOUR" "$RES2" 
}

for i in "${arr[@]}"
do
   check $i
done

Result:

Some instances may have a delay of more than 10 minutes, or even stopped updating completely. Watch out for outdated data.

https://dev.overpass-api.de/api_mmd/interpreter    | 2019-06-04T19:22:02Z |   00:02:32 | Overpass API 0.7.59_mmd        
https://lz4.overpass-api.de/api/interpreter        | 2019-06-04T19:24:02Z |   00:00:32 | Overpass API 0.7.55.7 8b86ff77 
https://z.overpass-api.de/api/interpreter          | 2019-06-04T19:22:02Z |   00:02:33 | Overpass API 0.7.55.7 8b86ff77 
https://overpass.maptime.in/api/interpreter        | 2019-06-04T19:23:02Z |   00:01:33 | Overpass API 0.7.55.6 486819c8 
https://1.overpass.kumi.systems/api/interpreter    | 2019-06-04T18:59:02Z |   00:25:33 | Overpass API 0.7.55.7 8b86ff77 
https://2.overpass.kumi.systems/api/interpreter    | 2019-05-31T02:11:02Z |   17:13:33 | Overpass API 0.7.55.7 8b86ff77 
https://3.overpass.kumi.systems/api/interpreter    | 2019-06-04T19:02:02Z |   00:22:34 | Overpass API 0.7.55.7 8b86ff77 
https://4.overpass.kumi.systems/api/interpreter    | 2019-06-04T19:12:03Z |   00:12:33 | Overpass API 0.7.55.7 8b86ff77