Code. Compile. Check.

The SDF engine provides a holistic view of your SQL ecosystem by considering all queries in a warehouse simultaneously. Uncover problems proactively and optimize your data infrastructure with compute, not more.

Book Demo
SDF build, SDF lineage, and SDF deploy output wrapper
An Example SDF Code Check

Introducing Code Checks for SQL

Code Checks are rules for your SQL. Code checks are powerful and super easy to use on your laptop or in CI/CD. Check out our first blog post for Code Check examples.

  • Enhanced Security
    Write custom compile-time code checks to safeguard sensitive data from unauthorized access
  • Debug Your Data Like Code
    Code checks bring data engineering closer to software engineering with visibility rules, type correctness, naming conventions, and more.
  • Proactive Error Prevention
    Leverage compile time analysis and CI/CD integration to catch mistakes before they impact your data integrity
  • Lower Cost
    Data tests and backfills are expensive. SDF allows you to provide strong guarantees across your warehouse, org-wide, all from the comfort of your laptop

Code Check Use Cases

Protect Sensitive Data with SDF Code Checks

Ensure All PII is Appropriately Anonymized

Prevent Mixing Currencies with SDF Code Checks

Prevent Different Currencies from Combining

Prevent time grain disparities with SDF Code Checks

Ensure Tables Respect their Visibility Specification

Workspace Blocks

SDF Workspaces organize and manage SQL project dependencies. A workspace is a directory defined by a workspace.sdf.yml. The file must start with a Workspace block, and may contain YML blocks for tables, functions, classifiers and more.

Use workspace blocks to define:

  • The location of SQL source files and YML configuration files
  • The default SQL dialect, catalog and schema, globally or per location
  • Code-checks and their locations
  • References to other workspaces
Example of an SDF Workspace Block

Table Blocks

Tables can be defined in either SQL or in YML. But only SDF Table blocks specify metadata for SQL tables.

Use Table blocks to:

  • Define a table (if it not given in SQL), including its columns
  • Add descriptions to tables and columns to grow your catalog
  • Add classifiers to table and/or column.
Example of an SDF Table Block

Classifier Blocks

Classifiers are a powerful extended typing system for data. In SDF they are used to provide more meaning to data, i.e. they can provide units of measurement (e.g. USD, meter), represent the state of data representation (e.g. PII is in clear text or anonymized), or provide the purpose of a table (e.g. this table is public).

Attach a classifier to a table or column; SDF then tracks the flow of a classifier within the whole workspace.

Use Classifier blocks to:

  • Define classifiers & labels including
  • Add descriptions to classifiers and labels to grow your catalog
  • Configure classifier behavior: their scope, propagation & cardinality
Example of an SDF Classifier Block

Function Blocks

Functions are either builtin or user defined. They are dialect specific. SDF Function blocks specify re-classifications for builtin- and user defined functions.

Use Function blocks to:

  • Define the signature of dialect specific function, including its parameters and return types
  • Add descriptions to function names and parameters, a section tag and executable examples of proper usage to generate an indexed function catalog
  • Add reclassify lists, to define how a function reclassifies the label(s) of a classifier
Example of an SDF Function Block

Profile Blocks

SDF profiles are used to specify different configurations for your project. Technically a profile blocks simply overrides anything that appears in a workspace, sources, classifiers, checks. Think of it like a built-in A/B.

Use Profile blocks to:

  • Segment your warehouse
  • Customize the workspace for the intended use, such as development, test, release.
  • Test different classification or propagation rules in real time
  • Deploy multiple profiles to track effects and changes
Example of an SDF Profile Block

Extensive SQL DialectCompatibility

SDF is a true multi-dialect compiler, supporting: Redshift, Presto, Trino, Big Query, and ANSI SQL.

SDF Supports Amazon Redshift SQL Dialect
SDF Supports Trino SQL Dialect
SDF Supports Presto SQL Dialect
SDF Supports  BigQuery SQL Dialect

Coming Soon...

SDF Supports Spark SQL Dialect
SDF Supports PySpark
SDF Supports PySpark
An Example of SDF Propagating Classifiers through a data warehouse

Label the Sources. Automate the Rest.

Label root tables and watch as your classifiers flow through your entire warehouse automatically. Configure rules to tell SDF where to stop propagation or reclassify based on logic you define in YML

<1% of Tables Need Manual Classifiers
Classifiers Propagate to the Other 99%
Flexible reclassification

Step into Modern SQL

Run the Engine Locally with Ease
Frictionless onboarding with no migration required
Experience the speed of a compiler written in Rust with built-in caching and datafusion under the hood
Learn More