Microgrants/Microgrants 2020/Proposal/Osm2pgsql development

From OpenStreetMap Wiki
Jump to navigation Jump to search
Osm2pgsql development
For over 10 years osm2pgsql has been and still is a major component of many OSM rendering tool chains. The work is a step in modernizing osm2pgsql and make it more flexible for bringing the variety of OSM data into our maps.
budget (Euro)5000
location(s)please fill in where you are located, and/or where you are focusing efforts, if applicable

About your Mapping Community

Tell us a little bit about your mapping community at the moment, some examples of details to include are below, but please provide us with any information you think is relevant. Please write 100-200 words

The community in this case is less the mapping community, but the developers and users of osm2pgsql. Currently there are two osm2pgsql maintainers, Sarah Hoffmann and Paul Norman, and some other people occasionally contributing to osm2pgsql. The main user of osm2pgsql is the openstreemap-carto map, the main map on openstreetmap.org, but there are many others who use osm2pgsql in their rendering toolchain.

What do you need the grant money for?

We want to understand what you plan to buy, how much you plan to buy, and why this will help you to scale your community mapping efforts. There are two tables for you to fill in your response, and you may add more rows as needed. Some examples are in the table for your reference. You are welcome to use a different format in your response, but please make sure to include these details.

To pay a software developer (me).

Budget breakdown

Give your budget breakdown here. If you would like to present your budget in a tabular format, you can use this tool to convert your spreadsheet into a wikitable and paste the code below here

5000 EUR pays for one software developer (me) for a month (with 40 hours/week). Work will most likely be spread over two to three months. I'd be doing this as freelance work and can provide an invoice to OSMF.

If you receive a grant, what do you aim to achieve?

It is recommended that this supports mission of the OSM Foundation and your local community. Please provide us with any information you think is relevant including the metrics you expect to achieve. Please write 100-200 words.

Osm2pgsql is an essential part of the rendering tool chain used by OSMF and many others. The software has been around for more than 10 years and many developers have worked on it. By now some technical debt has accumulated and it is often difficult to add new features, especially for new developers or when you only have a few hours here or there to work on it.

The grant money would give me the time for a concerted effort to clean up the code and tackle some larger reorganizations that will make the code easier to work with. It will also deliver new features for users of osm2pgsql.

I have written two blog posts that detail some of the work I have been doing lately and want to continue:

There is no fixed goal here. The idea is to get as much done as possible in the available time. Here is a quick summary of work items:

  • Clean up the code base and work off technical debt
  • Reorganize osm2pgsql internals to create clear modules responsible for different parts of the processing
  • Improve flex output and get it release-ready
  • After reorganizing the code it should be possible to resolve some of the issues (see https://github.com/openstreetmap/osm2pgsql/issues)

All of the work happens publicly on Github and I'll write about it on my blog. The changes committed to the code and issues resolved will show what work has been done.

Do you receive funding from any other sources at the moment?

If you receive funding from other organisations currently, please state how much this is, whether the donations are regular or one off, and how these funds have been raised. Please write no more than 300 words. If you have any sources of co-funding for this grant application, please provide details. Examples of this might be if you have any partners who would be willing to match grant funding if you receive it, or if your project has existing funding. Please provide as much detail as possible.

Yes. Two companies, Geofabrik and Thunderforest, have agreed to pay their part of the cost of this project and based on their plegde, work as already started. Together with OSMF, three of the larger osm2pgsql users would each pay about a third of the cost.

Is there anything you would like OSMF to support you with?

If there is anything which you would like OSMF to support you with in order to strengthen your project, please let us know. For example, this could include building relationships with OSM communities, sourcing updated imagery for mapping projects, or guidance on how to use some of the tools we have created. We want to do our best to support OSM communities, so please let us know what you need. Please write no more than 150 words.

Not specifically, but publicity is always good. I'd be happy to provide texts for a blog post that OSMF can distribute or something like it.


By submitting this form to the OpenStreetMap Foundation, you certify the information contained in this application is correct, and that if you are awarded a grant, you will use it only for the purposes described above. You will provide written documentation and receipts for all of your expenses to the OSM Foundation to demonstrate this. You understand that the decisions made by the OSM Foundation Microgrants committee are final.

I, Jochen Topf (OSM Username also "Jochen Topf"), agree to follow the above declaration.


Community members are encouraged to endorse your project request here!

  • I endorse this project. It is important to keep core tools in a good shape. --dbf_
  • Jochen undoubtedly knows enough about OSM and coding that he can make this a success. --Frederik Ramm (talk) 19:18, 10 May 2020 (UTC)
  • Osm2pgsql is invisible for most but crucial part of the OSM ecosystem. And I trust judgment of Frederik Ramm that Jochen will be able to help here Mateusz Konieczny (talk) 22:52, 10 May 2020 (UTC)
  • I endorse this project. osm2pgsql is an essential part of the OSM ecosystem, used by the OSMF and countless data consumers. Jochen is capable of managing a project of this scale, and the cleanups will set us up for future work on osm2pgsql. Pnorman (talk) 05:06, 17 May 2020 (UTC)
  • I endorse this proposal. – Craig Allan Yes. Fully endorsed as it will be developing the systems of the OSMF. Caveat: Will we find maintainers for Lua code when Jochen retires from the project.
  • I endorse this proposal. – jbelien (talk) 15:23, 15 May 2020 (UTC)
  • I endorse this proposal. – xivk (talk) 15:23, 18 May 2020 (UTC)
  • I endorse this proposal. – osm2pgsql is an important processing tool for OSM data. Every improvement is helping the whole community. Gislars (talk) 07:24, 20 May 2020 (UTC)
  • I endorse this proposal. I actually think that development of core tools (especially one critically used in the website's toolchain) should be funded directly by the OSMF instead of through grants. But absent such a funding scheme, I endorse this grant proposal. —seav (talk) 04:40, 21 May 2020 (UTC)
  • I endorse this proposal. --Øukasz (talk) 12:02, 21 May 2020 (UTC)
  • I endorse this proposal. – mikeg