“Smart developer’s quirks” tend to peak in 3-8 years of experience and fade off thereafter. A hipster will never fade off and instead continue hipster coding alongside their identity in perpetuity.
In data pipelines or MLOps projects I've found that mirroring the code location in the artifact storage location, such as in object stores, to be a helpful pattern. This little lightweight library makes using this pattern a drop in for any project.
Taking numerous raw tables to a flat ML/AI ready feature vector can be a challenge. This is a lighter weight way of doing it than feature stores, if it may serve you!
I've found feature stores to be overkill for many projects, as I just need point-in-time correctness, abstractions for joining lots of tables and flattening to a specific granularity, and abstractions for composability. I've been building GraphReduce which helps with all the aforementioned. Hope it serves others!
An early release of the tech - an automation layer for discovering inclusion dependencies, building a graph of data where tables/files are nodes and foreign keys are edges between them.
working on a composable feature engineering abstraction for handling big pieces of FE pipelines: time travel, entity linkage, standardized interface, pluggable compute backends, and simplification of features with many tables
I've been working on a composable feature engineering abstraction with graphs. Feature stores are too heavy for most projects I work on, but are complementary nonetheless.
