Overview
Data architect for Osmose from August 2019 to January 2020, through AspirationalX, building a geospatial model that quantified human risk around utility infrastructure using SafeGraph location observation data on Databricks and Spark.
Osmose inspects, maintains, and rehabilitates aging transmission and distribution structures for the utilities that own them, with decades of accumulated expertise across field technicians, professional engineers, wood scientists, and corrosion specialists. The underlying problem is prioritization: there are more poles than inspection capacity, and a uniform schedule spends the same effort on a pole in an empty field as on one over a sidewalk. Osmose saw ahead of most of their industry what continuously updated location observation data could do for that prioritization, and had already bought it. Their data science team knew the domain thoroughly; what the engagement added was one specific capability — the data was Spark-and-Databricks scale, and their tools and schedules were built around other work.
The model turned billions of XY coordinate points into a 50m by 50m grid covering the United States, counting distinct devices and observations per cell over time using HyperLogLog. That choice matters: exact distinct counts at that volume are expensive, and HyperLogLog trades a bounded error for counts that can be merged across cells and time periods cheaply. Joining those cell counts to pole locations produced a per-pole score for how many people a failure would plausibly affect, applied across more than 100,000 poles in more than 10 states. The pipeline ran on Databricks with data in S3 and Delta files.
The work started as a proof of concept in Alteryx rather than in Spark. Simulating the ETL logic against a sample dataset validated the input files and the transformation logic visually, before any of it was written as a Scala Spark job or run against cloud compute — the production dataset was on the order of 10,000 times larger than the sample. The trade-off is a translation step between the prototype and production, paid in exchange for cheaper and faster iteration while the logic was still changing.
Delivery included documentation, commented code, and training on how to refresh the SafeGraph data and re-run the prioritization each year, because a model the client cannot update themselves stops being useful the season after it ships.