The Fifth Elephant 2016

The Earth is a staggering dataset.

OpenStreetMap is the largest living open map of the world with a collection of over 1B mapped roads and ~2B mapped buildings. Processing this massive dataset can lead to a lot of interesting analyses about the world, but can also be really slow - enter the open source TileReduce module.

We use TileReduce for a lot of our geo-data processing requirements at Mapbox. TileReduce breaks down the processing of a massive amount of geo-data using the MapReduce concept on vector tiles. For example, a computer with 4 cores can process over 16B tiles, and filter ~90k features and 2B tiles based on pre-defined conditions in ~10 minutes.

TileReduce can be run on vector tilesets, which means that it can work on any data format(GeoJSON, osm files, shapefiles, and CSV and xml files having geo-data) that can be converted into a vector tileset.

This talk will cover the JavaScript principles behind TileReduce, how parallelising works on TileReduce, the skeleton of a simple program written with TileReduce and showcase some ways in which you can use TileReduce in your projects.

Outline

A single image of the world containing street level data would be far too large to be held in memory or to be able to be downloaded at once - which leads us to map tiles. Map tiles are usually 256px by 256px in size and are placed next to each other to create the illusion of a single image.

Tiles can be either rasters or vectors. While raster tiles contain pixel information and require processing on the server side before being rendered, vector tiles hold tile data in either a human readable geojson format or as protobufs, making them easier and faster to process and render on the client side. Vector tiles also contain useful geodata that can be parsed.

Greater the number of tiles that a map is composed of, the greater detail that map can show. In order to manage millions of tile images/data, web maps use a simple coordinate system. Each tile has a z coordinate describing its zoom level and x and y coordinates describing its position within a square grid for that zoom level: z/x/y.

Zoom levels are related to each other by powers of four:

• z0 contains 1 tile.

• z1 contains 4 tiles.

• z2 contains 16 tiles.

  ....

  ....

  ....

• zn contains 2^n * 2^n tiles

As you can see, the number of tiles increases exponentially with the zoom level, which leads to an exponential increase in bandwidth and memory requirements, not to mention a greater difficulty in parsing and analysing such a lot of data.

TileReduce was written to process, or mine geodata from, these millions of tiles asynchronously using the MapReduce concept on vector tiles, making it one of the fastest ways to parse tile data for the whole world. Compared to the complex postgres queries you would have to write to do the same operations, TileReduce is also an extremely easy library to use.

Outline

• Web Maps
• Tiles
  • Vector and Raster
  • Tile formats.
  • What does a simple, human readable vector tile contain?
• JavaScript Map Reduce
• Asynchronous processing
• Vector Tiles and Map Reduce
• TileReduce
  • Background and history.
  • Vector tiles - tools to convert certain data types to vector tiles.
  • Program skeleton
  • Examples
  • Limitations
• Q&A

Speaker bio

I am a developer at Mapbox. I’ve built several programs that use TileReduce to run analyses on OpenStreetMap data, and have been part of sessions at Mapbox on the JS principles that TileReduce was built on - so I understand the theory well. I also think it is one of the most beautiful tools to process geographic data and hope to share this same fascination that I have for the tool with the audience at the Fifth Elephant. Earlier, I have written 2D games using the Cocos2D engine, submitted small activities to the GCompris project and written application documentation for the GNOME Foundation. I also enjoy collecting recordings of Indian classical music that are in the public domain.

Links

• https://www.mapbox.com/blog/tile-reduce-3/
• https://www.mapbox.com/blog/osm-qa-tiles/
• https://www.mapbox.com/blog/streets-and-gender/
• http://shvrm.github.io/railway/
• https://www.mapbox.com/bites/00205/#9/25.0812/55.4590
• https://api.mapbox.com/styles/v1/srividya/cinlih0j00010d6m49g56lhbu.html?title=true&access_token=pk.eyJ1Ijoic3JpdmlkeWEiLCJhIjoicXo2NTJKayJ9.m1P57SCwkp8_7N3TgYsVRg#10.35/13.0320/77.5050