Geospatial Data Visualization of Door-to-Door Surveys in Singapore
Introduction
I have recently done some freelance work for a client in the last couple of weeks, where they were looking to visualize door-to-door survey responses for their app. Due to the 2-week deadline given, I needed a solution which was (mostly) free to use, able to have a fast turnaround for the client’s approval/adjustments, and not spend too much time on ETL, as the client does not have a full-time software developer and/or data analyst in their headcount. Hence my proposed solution were as follows:
Frontend — Website/webpage in Bootstrap + VueJS and whatever that was required to display a map (MapBox/Leaflet or similar).
Backend — Must be based in Django, as they already have their web-based application running on Django, and their survey data is stored in PostgreSQL.
Supporting Services/APIs — Preferably free to use with generous usage limits and access. Basically this drilled down to OpenStreetMaps, Overpass and MapTiler.
Development Approach and Systems Design
The last time I made a website with a map was a few years back for visualization work (source code is on GitHub), and back then I made use of Leaflet. However, I needed a “batteries-included” framework out of the box, as the client was planning to do more things with the map in the future, such as viewing the map in 3D, adding a choropleth layer, and creating heatmaps to name a few. Hence the Mapbox option was preferable, but because of licensing issues, I decided to go with MapLibre.
For the display of the map tiles, there are so many providers which provide such a service, and as such, I will not elaborate much on it. I simply chose MapTiler based on the fact that they have a point of presence in Singapore.
Given my past working knowledge of GIS data (which isn’t a lot) and time constraints on this gig outside of working hours, my preference is to avoid ETL (extract, transform and loading) workflows as much as possible in order to display the geospatial data on a map. Hence before even writing the first line of code for this project, I did quite a bit of research — roughly 3 to 5 hours of Googling, testing/trying things out, and reading.
Being unfamiliar with OneMap APIs and the data.gov.sg site, I didn’t want to spend too much time reading through the documentation and trying out the various public APIs/datasets. Additionally, because the client’s survey data contains the postal code for each HDB block that they’ve canvassed, there needs to be an existing dataset where I can retrieve the polygon data with only a simple postal code query.
I set about searching for a pre-made dataset with the above requirement, and my Google-fu brought me to the NUS UAL GitHub repository. After reading through the links to their research paper (fascinating read by the way), data sources and project website, I formulated my own simplified method of data collation outlined in the next section below, since the project requirements was for a 2D vector polygon representation of a HDB block, rather than a 3D building model extruded from the polygon, which may or may not come later. I am also intentionally avoiding work that requires generating a dataset just for geocoding, as I do not have time to do so.
OpenStreetMaps Sparks Joy
Since the dataset generated by NUS UAL was based on OpenStreetMaps, I decided to explore the OSM website, and to my delight, nearly everything* which I require for this project is available on it, thanks to the contribution of others who have already done the grunt work — OSM has the polygon representation of a building, postal code and latitude/longitude data available for each individual HDB block. Now we only need to simply ETL it into something which can be processed easily by MapLibre and maybe Django.
*- details explained in the Analysis and Conclusion section below.
Enter the Overpass
Again, thanks to the wonderful trinary of open-source software, public datasets and thankless people who have worked on/with OSM, there exists an API which transforms data from OSM into GeoJSON, called Overpass. However, in order to make use of the Overpass API, I had to learn the query language specific to Overpass. Thankfully this wasn’t too difficult, as data for a single HDB block can be queried via it’s OSM addr:postcode tag, and extrapolating it as an API call on the front/backend is relatively simple.
Extract. Transform. Load. Repeat.
Now with GeoJSON files for each individual HDB block, a simple concatenation of the features array in each GeoJSON file into a larger FeatureCollection object allows for us to show all surveyed HDB blocks on the map. This method also allows us to load survey data based on a specific date in the past, as there was no requirement to show a collated view of all surveys across the whole of Singapore, saving precious bytes for users on data-starved phone plans.
As an added bonus with MapLibre, I am able to create an animated zoom-to effect upon the GeoJSON data layer being loaded into the map by simply calling fitBounds() on the map. This requires a set of vector coordinates that form a box however, and the TurfJS library fits in nicely here, by calculating a set of 4 vector coordinates from the GeoJSON data object generated earlier through the use of the bbox() function.
Analysis and Conclusion
Surprisingly the coding portion did not take up too much time (about 2 to 3 hours), as most of the time spent was doing the research and understanding the various parts that make such a thing possible. Getting building information in order to display it as an overlay is heavily data-reliant, and as such, there can, and will be gaps of data points in order for us to render a polygon easily with minimal ETL effort. There are certain old HDB blocks which are missing data in OSM, and as such the display of data for those HDB blocks will not be possible because the geocoding is not updated/tagged properly for those HDB blocks in OSM.
There is a solution however, and it is to simply query Overpass with the OSM waypoint, instead of querying by postal code. This would require creation of another page on the dashboard to input and link the data correctly, and as well as creating a new column in the SQL database to store the OSM waypoint ID. Perhaps in a few months’ or years’ time, the data on OSM will be updated, but for now, we would need to rely on this method to display HDB blocks, unless there is enough time and/or manpower to do ETL for existing public datasets which have the data for our use.
Source code for this article is available at https://daffodilistic.github.io/hdb-survey-datavis/
I hope this article has been helpful to you, if so, do clap for this if you find it illuminating.