Massive growth has led to a need for a faster and more robust transformation layer

The data lead at Linqto, Chris Hronek, recently found himself at a crossroads: pursue the conventional data stack to address the increasing demands on financial reporting analytics or find an alternative that moved their team faster and solved their persistent problems. The data engineering team had already grown in size and the number of analytics transformations was rapidly increasing. As a leader, Chris set his goals like many in the data industry: increase developer efficiency while also ensuring quality.

"I was working on building a new tool, similar to my previous open-source product Astronomer Cosmos, but one that addressed multiple new areas of concern: quality, controls, and faster compile times. I saw a demo of SDF and knew it was perfect for our transformation layer and data management." - Chris Hronek, Director of Data Engineering

Instead of pursuing alternatives – requiring multiple tools to provide column-level lineage, quality checks, and syntax/semantic error reporting – the Linqto team set up SDF.

The integration process was seamless and took less than an hour, and they were able to extend their transformation layer past the previous limitations of dbt and set up quality controls restricting common errors.

Compile Time Checks and Transformation Layer Improvements

The Linqto Data Warehouse was built with generally accepted data engineering rules and designs in mind. The team loads raw, untransformed data into schemas, and copies them into models where testing, conversion, and transformations are applied, and the data is prepped for visualization. New developments and changes are tested in a development/staging environment before being pushed to production.

The team was reviewing changes to the data model through a process on GitHub and it was the responsibility of the reviewer to enforce the agreed upon rules. It was an error-prone and time-consuming process with issues slipping through. 

"We kept missing simple errors like dangling commas and references to development data warehouses instead of production data warehouses. It was so frustrating to see these simple problems break our production environment." Chris Hronek, Director of Data Engineering

In previous builds with dbt, the team would spend several minutes compiling and running the dbt models within their data warehouse, unaware of issues and semantic errors until a failed run report returned from the compute provider. These failed runs wasted compute and posed the risk of taking down business critical dashboards.

With SDF, the team has implemented compile-time checks to verify the presence of specific type-based properties and syntax error checks that do not require human intervention and review. These critical and real-time pieces of feedback run locally on the team’s workstations and are integrated directly into a Continuous Integration (CI) Pipeline that activates when a team member opens a Pull Request to change the data model. 

These command-line capabilities give the Linqto data engineers the ability to debug and identify breaking changes quickly and prevents them from ever affecting production. They gain a complete and current understanding of their data warehouse.

Gaining a complete view of the data warehouse

Another primary goal of the Linqto team was to establish a complete understanding of their data warehouse. Building anything off a data model, whether static visualizations, reverse ETLs, or data processes can lead to concerns that changes will break something downstream and paralyze business operations. This not only slows down the data development cycles but also can lead to high-value analytics models failing during critical times.

After compiling their entire data warehouse and parsing all models, in a matter of seconds, Chris was able to visualize the entire Linqto data model. With SDF providing column-level lineage within the command line interface and graphically in a cloud offering, Chris can understand downstream column and table references and his developers can increase their confidence in decisions. The team is estimated to save over $130,000 in labor costs with this time saver and increase confidence in decisions.

‍

‍

SDF Brings Speed and Understanding into the hands of Analytics Engineers

SDF brings immediate value to data engineers and analytics engineers through faster iteration cycles, SQL and Jinja error checks, and a complete understanding of the data warehouse. With the SDF CLI and SDF Cloud, teams can proactively establish quality rules and identify issues at compile, well before run time. As the Linqto data team continues to grow its data warehouse, they aim to further utilize SDF for its governance, quality, and transformation needs. The end goal is for all data models to be transformed, controlled, and governed by SDF.

Linqto - Empowering Individual Investors and changing private investing

Linqto's vision is to democratize private investing by making it accessible, affordable, and liquid for individual investors. Through an intuitive platform, Linqto empowers individual investors to engage in the private equity market, which was once only within reach for the privileged few.

With an inclusive, easy-to-use, and fast digital platform, Linqto provides qualified accredited investors access to investments in leading tech companies while they’re still private.

‍