Write Data Like Code

A transformation layer that actually understands SQL. Simplify testing, reporting, and debugging with the best SQL development experience.

Request a Demo

Metadata Preprocessing

SDF processes Jinja, types, and variables with precise line-number mapping for simple debugging.

SQL Compilation

SDF compiles raw SQL statements to catch syntax errors instantly and produce fully qualified logical plans for analysis.

Static Analysis

SDF analyzes logical plans to propagate types, catch errors, build the information schema, and prepare queries for execution.

SQL Execution

SDF executes statements locally with the SDF Database or remotely against cloud databases like Snowflake or Redshift.

The Next Generation for Data Transformation

Expect More from Your Transformation Layer

Seamlessly integrate SQL, seeds, types, and metadata in one platform. Generate your catalog automatically.

Compile time column level lineage, automatic dependancy management, and integrated testing.

Rich types bring your SQL Transformations closer to the business stakeholders that they impact.

Debug Faster than You Can Say Column-Level Lineage

Root cause errors in seconds with precise line numbers even in the most convoluted Jinja.

Leverage compile time analysis and CI/CD integration to catch mistakes before they impact your data integrity.

One Engine, Endless Applications

The SDF CLI surfaces in-depth column-level lineage information directly from the command line.

Data lineage tool with column level lineage

Data Development Simplified

SDF does for SQL what Typescript did for Javascript.
Faster Development. Trusted Results. Safety at Scale.

Cloud Development Made Simple

Materialize Models Intelligently

Run your models faster with distributed execution based on time and data dependencies.

Build efficient pipelines with incremental materialization and automated schema migration.

Utilize rich materialization strategies to run temporary views and transient tables with a single command.

Isolate Environments for Everyone on your Team

Configure Environments with ease for production, staging, and development with simple credential mapping.

Systematically rename queries based on the environment to power zero-copy isolation per-developer.

A Transformation Layer with its own Information Schema

A Fully Programmable Transformation Layer

SDF's compiler output follows a self-describing JSON schema such that the output is an SDF workspace in itself.

Power your own data catalog, governance, or quality layer with a queryable source of rich metadata.

The Last Information Schema You'll Ever Need

SDF's information schema acts as a central store for your tables, columns, and metadata - removing expensive dependencies on compute-provider information schemas.

Simple, Flexible Configuration

Minimize YML for maximal gain.
Compose your data warehouse with declarative block syntax.

Workspace Blocks

The SDF YML workspace block defines the configuration of your workspace in SDF. It serves as a central place to manage and outline the structure and behavior of your SDF project. Essential details such as the workspace's name, edition, description, repository information, and includes paths for .sql and .sdf.yml files are configured here. Moreover, it specifies defaults, dependencies, providers, and various other configurations that are crucial for the operation of your SDF workspace.

Table Blocks

The SDF YML table block is used to define or add metadata to a table, including the table name, description, columns, and SQL dialect. Moreover, classifiers can be applied at both column and table levels within this block. Additionally, the .sdfcache directory uses this metadata representation to store the configured and generated metadata across your project.

Classifier Blocks

Classifiers are a powerful extended typing system for data. In SDF they are used to provide more meaning to data, i.e. they can provide units of measurement (e.g. USD, meter), represent the state of data representation (e.g. PII is in clear text or anonymized), or provide the purpose of a table (e.g. this table is public).

Attach a classifier to a table or column; SDF then tracks the flow of a classifier within the whole workspace.

Function Blocks

The SDF YML function block is used to define user-defined functions within the SDF environment. This functionality allows for extensive customization and extension of SQL capabilities within your SDF projects.

The function block includes details such as the function's name, its parameters, return type, dialect it applies to, and its volatility. Functions can be scalar, aggregate, or table functions, and they support various features like fixed or varying arity, overloading, polymorphic behavior, higher-order functions, and more.

Environment Blocks

SDF Environments are used to separate production, staging, and developer-specific workflows. It supports zero-copy sourcing from production data and systematic renaming to ensure developers only write to non-production schemas during development.