Talk:Planet.osm

From OpenStreetMap Wiki
Jump to: navigation, search

Auto update tool

I dont know how much stress there is on the server, but I can guess there is a lot. So I have thought off some sort of tool that can:

  • Download latest Planet.osm
  • Share this Planet.osm through Torrent
  • Autodownload diff files

--Zoltan 08:30, 8 July 2008 (UTC)

There's some stress caused by people downloading the planet, but it's not as problematic as abuses of the API and tile server (hence Acceptable Use Policy focusses on these)
To ease off on the amount of data you're downloading, first of all think about using one of the Planet.osm#Mirrors, but also note that many of these are offering 'Extracts' of smaller areas. Only download what you need. Secondly, avoid downloading the planet again on a regular basis. Use the diffs. These are documented at Planet.osm/diffs, including the use of Osmosis tool to apply the diffs.
Torrents could be a good idea too, but not the be all and end all. See the #Torrents discussion below
-- Harry Wood 15:24, 24 May 2011 (BST)

Patches for broken stuff

Does the patches for broken stuff still have a use? Does it even apply to more recent versions? --Dean Earley 22:42, 5 Jun 2006 (UTC)

I Guess not :o)
[1] --Dean Earley 23:36, 6 Jun 2006 (UTC)

Added more hyperlinks

I added some more Wikipedia hyperlinks to the jargon terms in the article ([2] [3]), in an attempt to make the page a little more newb-friendly. (When people "don't understand" something, usually that means they don't know what some of the words mean. Wikipedia articles usually do a pretty good job of fixing that, for software tools and such.) I also mentioned Cygwin as an easy way to get bzip2 and most other GNU tools on Windows. Feel free to edit or revert if my additions are not cool. Teratornis 16:34, 4 Jul 2006 (UTC)

I've tried a different style of link for compression programs - using wikipedia's list (and we can edit that page if you know a program like UltimateZip that isn't on the list yet)
I think that installing GNU on Windows is probably overkill for someone who just wants to decompress a file, especially as it doesn't have a GUI or interact with the same filesystem. Ojw 18:54, 4 Jul 2006 (UTC)
Someone who just wants to decompress a file at the moment might eventually need other tools, e.g. sed, grep, Perl, xsltproc, built-in shell commands, etc. Maybe even a Linux GUI program like Gpsdrive (which I haven't tried on Cygwin yet, but it sounds like fun). A Linux user might use a dozen tools in the course of working with XML files and so on, without really thinking much about it, until the issue of doing all that stuff on Windows comes up (as it tends to, given the vast number of Windows computers out there). I suppose the real question is how many references to GNU tools might come up; here's another one: Talk:JOSM#Cannot_Read_Time_From_Point_x shows a sed command line and asks how to do it in Windows. A Windows user who follows along with the work of Linux users could spend a lot of time downloading tools individually, and then the resulting hodge-podge won't be easily maintainable. Cygwin is simple to install (and uninstall), has built-in package management, and provides a large set of GNU tools. Perhaps it would be good to inventory all the references to GNU tools here, for example search for xsltproc (such references may expand in the future as people think of new ways to manipulate OSM data), and create a separate page for Windows users, telling them their options if they want to keep pace with the Linux command-line examples. I get the impression that the technical leaders here are mostly using Linux or Mac OS X, and their hacks might be difficult for the stock Windows user (but straightforward for the Cygwin user, who can often use the same shell syntax). I agree it is inefficient to explain all this in every article that mentions a GNU tool, but I think it's worth explaining once. Teratornis 07:19, 5 Jul 2006 (UTC)
I started a User page (User:Teratornis/WindowsUsers) where I will try to inventory the GNU tool command-line examples throughout the OSM Wiki and Talk pages (or at least a large enough subset of them to show the scale of the problem for a Windows user; so far I see bash, bzip2, gnuplot, sed, svn, xsltproc), and list some options for the Windows user along with their pros and cons. Anyone with an interest may then review my user page, and we can decide if and where to publish it, and then go around linking articles to it. Teratornis 17:24, 5 Jul 2006 (UTC)

OSM Coverage Map for USA?

The map of OSM coverage in Europe looks great. I would like to see a similar map for North America, or at least for the USA. Among other things, that would tell us how the TIGER import is proceeding. If someone with knowledge can make such a map, that would be wonderful. Otherwise, I might try to figure out how to do it. Teratornis 16:34, 4 Jul 2006 (UTC)

The easier way to do this would be a REST API function count?bbox=bllon,bllat,trlon,trlat that returns the number of nodes and segments in an area -- you could write all sorts of statistical programs to use that (density charts etc.) without having to download any data or parse any segments. Ojw 19:03, 4 Jul 2006 (UTC)
I'll check this out when I get a chance. Thanks. Teratornis 19:54, 6 Jul 2006 (UTC)

Latest one

<tag k='name' v='St Andrew's Church of England Primary School' />

Where does the single-quote end? Ojw 15:50, 8 Aug 2006 (BST)

It's possible to extract region from Planet.osm?

I am living in Slovakia. It's is possible to extract region or country as .osm from planet.osm. Because planet.osm is very big to open to process in everything..

You can use the excerpt-area perl script for that. It exctracts a rectangular area. For a more complex approach that extracts a polygon, see http://trac.openstreetmap.org/browser/utils/planet.osm/perl/extract-germany.pl which you would have to modify to use Slovakia's border polygon. --Frederik Ramm 12:34, 9 March 2007 (UTC)
Polygon for Slovakia is at http://svn.moonbase.sk/slovakia.osm/polygon_slovakia.txt

Compressed planet dump growing every week by 150 MB. Now it's about 1GB. Is it possible to generate planet.osm divided into smaller areas i.e. europe.osm, asia.osm, america.osm... right on osm servers...? --Dido 08:53, 4 October 2007 (BST)

(Old discussion) 1GB? how quaint. Today it's 16GB. These days we've also got some better options. You can download extracts. Various options listed: Planet.osm#Mirrors, or run your own extract using a tool such as Osmosis (There are other tools listed at Planet.osm#Processing the File which can do extracts) -- Harry Wood 15:35, 24 May 2011 (BST)

Malformed XML still?

In this article you can read:

"The file is in the OSM XML format. However, until the 0.4 API is rolled out, the planet.osm dump typically contains UTF-8 errors that will break an XML parser. "

Since we now use API version 0.5, this is supposedly fixed. Cany anyone confirm that?

Gisle

(old discussion) Pretty sure the generated planet files don't have any UTF problems these days (on API 0.6 now) -- Harry Wood 15:37, 24 May 2011 (BST)

difference

rather than downloading the entire planet.osm file every time it is updated, is it possible to do some sort of differential update, and only download the parts that have changed? i would guess this would bring a weekly update into the realm of 100MB, rather than 1+GB? are these file provided by OSM, or is there some way of doing it using tools provided in Linux? Myfanwy 02:57, 12 November 2007 (UTC)

Hi Myfanwy. There is actually a daily diff download, introduced recently. I've added some description on the page here. -- Harry Wood 10:39, 12 November 2007 (UTC)
fantastic, thanks for that - now my house mates won't hate me for killing our internet every week. are there any programmes around that will do a one-click/one-command update of OSM data? a repository that i could add to sources.list for all the OSM tools (including things like this, if it exists) would be really useful Myfanwy 00:29, 13 November 2007 (UTC)
Yes. Information on how to use the diff files, to reconstruct a full planet file, are still missing from this wiki page currently. I would guess that you can use Osmosis (set of java tools). Based on this readme I found out that the diff is created by Osmosis in the first place. -- Harry Wood 09:34, 14 November 2007 (UTC)
As update to this old discussion. Planet.osm/diffs page now has the explanation of how to use the diffs. -- Harry Wood 15:38, 24 May 2011 (BST)

Timestamp

"The weekly dump starts at 01:00 each Wednesday night" - local time (which?), UTC, etc. ??

Yes, please, could somebody answer this? Vid the Kid 20:03, 20 January 2009 (UTC)
I think this phrase is ambiguous anyway. Is this 01:00 Wednesday morning or 01:00 Thursday morning (in England this would often be considered Wednesday night despite being past midnight.) Could someone who knows which is correct clarify please? Daveemtb 09:57, 10 June 2009 (UTC)
I've tweaked the text slightly. Dump starts at 01:11am early hours of Wednesday, UK local time. It takes about 12 hours to complete. Feel free to tweak text. -- Firefishy 13:48, 11 June 2009 (UTC)
Firefly, is it really local time (adjusted for DST), or is it UTC? Not that one hour will really make that much of a difference, but if we're going to make our maps as accurate as possible, our times should be as accurate as possible, too. -- Tahongawaka 00:02, 28 October 2010 (BST)
The planet.osm header has a timestamp in the header. Best to use that. Run time is 1:11am British Summer Time. --Firefishy 16:05, 29 October 2010 (BST)
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="OpenStreetMap planet.c" timestamp="2010-10-27T00:11:03Z">
  <bound box="-90,-180,90,180" origin="http://www.openstreetmap.org/api/0.6" />
...

Torrent

We need an IPv6 tunnel to download planet.osm via torrent? Seriously....? --Regulator 09:25, 24 May 2011 (BST)

Hmmm, no, not at all. The torrent files specifies several public trackers, only one of which is IPv6 (others are IPv4). It also specifies webseed, so your client should work even if all trackers and seeds/peers are dead if it supports this extension. So it should work for you over IPv4, yes. I haven't heard of such problems yet (and had several success stories). How exactly does the problem manifests and what bittorrent client are you using? Are you able to download other torrents normally? (I'm currently downloading from IPv4 only host from 5 peers using Transmission client on Debian GNU/Linux) --mnalis 13:23, 24 May 2011 (BST)

Currently an experiment, I might automate this, maybe talking to the other mirror people as well. --Skinkie 20:31, 4 April 2009 (UTC) The origin is powered by Oxilion.



Hi, (hopefully I am not completely blind and I did not see it but) is there any reason, not to distribute the planet-file via Bit-torrent? I would expect, that this would greatly discharge the osm-server, especially if the user were prompted to use it. -- User:Kinglui 08:21, 22 August 2008

Gurkensalat is not hosting planets (anymore?) User:Milovanderlinden 14:09, 11 June 2009
Well this section Planet.osm#Bittorrent ...mentions some torrents. Maybe you are completely blind! :-)
Mind you, it looks like there hasn't been a latest planet for a while. Maybe User:Skinkie knows more
-- Harry Wood 15:06, 24 June 2009 (UTC)
I am happy to provide the code or service to do so, but I am not willing to do it unless the crap about seeding is not resolved. The basic idea would be that the seed is started at osm.org and then picked up by the mirror. That is the smartest way of getting the data at the mirrors and the more mirrors join, the fasted the internal traffic will become anyway. --Skinkie 14:45, 3 July 2009 (UTC)
Have there been any movements in that direction? I do not see the it should be complicated on OSM side, it should be as simple as "apt-get install mktorrent rtorrent", adding startup script and config file for rtorrent to look in directory where torrents are saved, and modifying script that creates planet to also call mktorrent to produce .torrent file. One hour job at most, easily 15 minutes if you're familiar with the programs. We could use public trackers, or provide our own. And it would actually lessen the network load on OSM servers --mnalis 20:47, 24 October 2010 (BST)
Each week's export would have to be named differently, which wouldn't really be a problem if it's coded YYYYWW to keep the American date format and European date format from confusing anyone. Would all the weekly archives still be seeded or removed from the OSM torrent server? -- Tahongawaka 00:05, 28 October 2010 (BST)
I think we should simply name the torrents as the planet*.osm.bz2 files are currently named at http://planet.openstreetmap.org/ . Also, all files present there should also be seeded by OSM. When planet files get removed, so should corresponding torrent files (rtorrent for example automatically handles such removals and stops seeding). As I said, tehnically that is rather simple procedure to configure and set up, it is just a question of good will. --mnalis 13:42, 29 October 2010 (BST)
Fine, but who should be the tracker? http://openbittorrent.com/ ? --Firefishy 14:33, 29 October 2010 (BST)
Yes, in addition to http://openbittorrent.com/ there is http://publicbt.com/ and we should also add http://www.ipv6tracker.org/ (maybe even with highest preference?) in order to promote IPv6 usage for the folks that have it (or the initiative for those who don't yet). I'd recommend those three, as there should always be at least two trackers (to handle possible outage), and one of the three is ipv6-only. --mnalis 19:06, 8 November 2010 (UTC)
For example, to create test torrent file you just run this (one would replace changesets-101103.osm.bz2 with real planet file of course, and put it in the script just after planet-xxxxxx.osm.bz2 is generated; this is just example for test):
 mktorrent -a http://tracker.ipv6tracker.org:80/announce,udp://tracker.ipv6tracker.org:80/announce \
 -a udp://tracker.publicbt.com:80/announce -a udp://tracker.openbittorrent.com:80/announce \
 -w http://vrba.carnet.hr/~mnalis/osm-planet/changesets-101103.osm.bz2 changesets-101103.osm.bz2
if you had a big swarm with at least a few smarter clients, web-seed (-w option) would even allow the swarm to run if your bittorrent seeding client is not working! However, for normal use you'd of course want to run bittorrent seeding client; for example run rtorrent with ~/.rtorrent.rc config file like this:
cwd=/home/mnalis/public_html/osm-planet
directory=/home/mnalis/public_html/osm-planet
schedule = watch_directory,10,10,load_start=/home/mnalis/public_html/osm-planet/*.torrent
schedule = tied_directory,10,10,start_tied=
schedule = untied_directory,10,10,close_untied=
use_udp_trackers = yes
dht = auto
dht_port = 6881
peer_exchange = yes
And that is all, it would all work automatically. rtorrent (running under screen for example) would automatically find new torrent files and seed them to interested clients, and when cleanup script removes planet-xxxxxx.osm.bz2 (and planet-xxxxxx.osm.bz2.torrent !), it would automatically stop the seeding. All in all, 15 minutes of work at most. --mnalis 21:26, 8 November 2010 (UTC)
I've now set up http://osm-torrent.torres.voyager.hr/ as a proof of concept (on non-OSMF servers, unfortunately so expect it to be lagging somewhat), for those willing to try it. --mnalis 16:43, 12 December 2010 (UTC)
Just a note that the torrents seem to work fine. I'm currently seeding the newest planet over IPv4 and IPv6. Ideally the torrents should indeed be created on OSM's servers to reduce delays and unnecessary file transfers. Avij 21:20, 16 December 2010 (UTC)
Seeding from the same servers that host planet.osm.org is difficult, our host blocks most ports. I'll see what I can do -- Firefishy 22:48, 29 December 2010 (UTC)
Thanks Firefishy! rtorrent has port_range (and dht_port) directive if you need to limit it to just a fixed port or two in order to allow it through firewall. And even if it turns out it is not feasible to run torrent client on planet.osm.org machine, the thing would work even without OSM running torrent client, as long as there was at least one webseed-aware torrent client in the swarm (which would then automatically use standard HTTP web access to retrieve first copy and share it with the rest of swarm: see -w options to mktorrent in my proof-of-concept [script]) --mnalis 01:23, 30 December 2010 (UTC)
Hey Firefishy, had you perhaps had any luck adding bittorrent support to planet.osm.org? Anything I could do to help? --mnalis 13:41, 12 March 2011 (UTC)
Just an update that [4] torrent service will be shutting down soon. It was (working) proof of concept, intended so it can be implemented directly on planet.osm.org servers (which is the only place where it makes sense, and is easiest and cheapest to do), but it was not to be... Thanks for everyone who participated, it was a fun ride! --mnalis (talk) 22:30, 14 September 2014 (UTC)

Taiwan

Hi, I am searching for maps (best would be a country map) of Taiwan. I see that there was a link ([5]), who provided Taiwan maps. It is unfortunately down. Is there by any chance someone who has those maps? I would be very grateful :-) I use a Garmin eTrex Vista hcx, so it would be even better if it would be converted to a Garmin-map already. As I am a real newbie i would also be happy about a short tutorial about how I can make such a country map myself. Maybe I am blind, but so far I couldn't find that. Thanks a lot in advance. --AngMoKio 14:13, 24 June 2009 (UTC)

Try CloudMade downloads: http://downloads.cloudmade.com/asia/taiwan#breadcrumbs More information: OSM Map On Garmin -- Harry Wood 15:01, 24 June 2009 (UTC)
Thanks a lot. Will try that out. --AngMoKio 07:59, 26 June 2009 (UTC)

How to download the latest Planet data?

I am unable to download the latest planet files from "http://planet.openstreetmap.org/". "planet-080528.osm.bz2 29-May-2008 00:10 4.0G" is the latest file which will download. I thought only the older files links were the ones to be left broken? Can someone tell me how to download the latest Planet File so I can then use the daily diffs to keep my data current? User:Maw269 15:39, 11 May 2010

Internet Explorer and some other browsers cannot download files over 4GB. Firefox works, you will also likely require a filesystem that supports 4GB files. (NTFS on Windows)
-- Firefishy 22:44, 11 May 2010 (UTC)
Excellent, that did the trick....Many thanks for the quick reply! -- User:Maw269

Symlink for daily latest

Hi! There seems to be a symlink for planet latest (the whole planet file), but there is no latest symlink for the daily updates at http://planet.osm.org/daily/. It would make automatic downloads easier if we had a symlink in this dictionary too. Could somebody please create a dynamic symlink? Thank you very much! --Marqqs 11:35, 9 December 2010 (UTC)

Not sure of the pros and cons of setting up that. Seems like it might be reasonable request
I used this ruby script to do daily diff downloads, but this is all a bit shakey. These days the recommendation is to use Osmosis and set up 'replication'. This can still be done on a daily basis if you prefer, but uses the minutely and hourly diff downloads which are of a different format. Osmosis decides which files to download automatically, and in generally it's all a bit more reliable. My understanding is that daily diffs are only available as a legacy thing
-- Harry Wood 15:45, 24 May 2011 (BST)
Meanwhile I solved the problem, using a strange-looking line in my download script:
OSCFILE=$(date -u -d yesterday +%Y%m%d)"-"$(date -u +%Y%m%d)".osc.gz"
But to have a static download link for daily change files would be really nice. FYI: Id don't use Osmosis, I take the very-light tool osmchange to merge the daily change files into an existing .osm file. Here is the description of the toolchain. --Marqqs 19:44, 24 May 2011 (BST)

Raw SQL dump

Is there a raw sql dump available besides Planet.osm?--Kozuch 15:31, 22 December 2011 (UTC)

Geofabrik changes, Sept 2012

Geofabrik main download site now appears to be http://download-int.geofabrik.de/openstreetmapeurope/ holding ODbl data. The old site continues at http://download-int.geofabrik.de/osm/ with CC-by-SA data from mid September - I have no idea if this is intended to be a permanent change. Perhaps a German speaker could find out? --tms13 08:41, 27 September 2012 (BST)

Yes, some months ago Frederik told us that the download path will permanerntly change for ODbL-Data to http://download.geofabrik.de/openstreetmap/ --Stephan75 17:05, 27 September 2012 (BST)

Openstreetmap.fr

Is there a reason why http://download.openstreetmap.fr/extracts/ is not listed in the available sources?

Adavidson (talk) 01:41, 19 October 2016 (UTC)