Overview
Data architect for Fugue from August 2019 through 2020, via AspirationalX, working with their engineering team on the data warehouse behind a cloud compliance product. Fugue is part of the In-Q-Tel portfolio, the strategic investor backing technology relevant to US national security — backing that reflects how serious both the team and the problem were. Their engineers were strong, and the product idea was ahead of how the industry treated compliance at the time: verification as engineering rather than paperwork.
The problem Fugue was solving is a consequence of how cloud infrastructure works. Organizations that have to satisfy FedRAMP or PCI DSS need to demonstrate that their infrastructure meets the control set — but cloud resources are easy to change, so any statement about compliance is only true as of the moment it was made. Fugue’s product automated that verification continuously, and doing that means treating infrastructure state as a data problem: collect Terraform state and cloud configuration, evaluate it against compliance rules, and keep the results queryable per customer and per resource.
The warehouse was built on AWS-native pieces — infrastructure state landing in S3, AWS Glue and Spark handling the ETL, and the processed results stored for querying and long-term retention. AWS Glue was in private preview at the time, which is worth naming as a trade-off rather than as an advantage: early access meant serverless ETL without operating a cluster, and it also meant building on a service whose behavior and documentation were still moving.
The concrete engineering constraint was the small-file problem. Streaming compliance data produces a large number of small objects in S3, and query and processing performance degrades as that count grows regardless of how much total data is involved. The work included consolidation and processing strategies in Glue and Spark to keep file sizes in a range the query layer could handle as volume grew.
My contribution centered on data discovery and modeling. I used Alteryx and Tableau to pull down and inspect the streaming data before committing to a schema, so the relationships that compliance reporting depended on were understood from the actual records rather than from a description of them. The design idea worth carrying forward is the reframing: compliance handled as a periodic audit is expensive and always slightly stale, while compliance modeled as a continuous data pipeline becomes a monitored property of the system.