Serverless data pipelines written for Data Engineers.

Frameworkless. No dependencies. Write testable tasks in modern Python.

Compose in YAML and schedule them in serverless Lambda or Airflow. Package complex workflows into simple Component UIs to enable self-service by your team.

or install locally.

Hero

Features

Serverless deployment to AWS Lambda

Serverless deployment on AWS. Robust, cheap and scalable.

Airflow deployment - 10x productivity

Productivity leap without risk with Airflow transpilation.

Frameworkless native python

No dependencies or limitations. Simply pure, testable python functions.

Simple composability

Reuse any source easily composing them like Lego blocks.

Elegant YAML

Everyone likes YAML, don't they? Easy to pick up, very productive.

Steamlit Components UI

Package complex workflow for self-service use by your team. Scale your workflows.

TYPHOON_DEPLOY_TARGET=typhoon

Serverless deployment with AWS Lambda.

BLOG: Make a serverless telegram bot (20 mins).

Or try it with zero migration risk with robust Airflow transpilation.

# Simply add this to .cfg and a few settings

TYPHOON_DEPLOY_TARGET=airflow

Extend with pure python. Create a function and reference it directly in YAML.

Because they are pure python you can also easily test them with pytest.

Extend

name: favorite_authors
schedule_interval: rate(1 day)

tasks:
  choose_favorites:
    function: typhoon.flow_control.branch
    args:
      branches:
        - J. K. Rowling
        - George R. R. Martin
        - James Clavell

  get_author:
    input: choose_favorites
    function: functions.open_library_api.get_author
    args:
      author: !Py $BATCH

Compose elegant YAML DAGs using re-usable hooks and pure python functions just like lego.

VSCode has auto-complete so its really quick to compose any complex task.

Component UIs templates can abstract complex workflows for other users.

Make your analytics team self-sufficient by creating data pipeline apps so they can create schedules themselves via the api.

Extend

Plugins

Typhoon supports any DB that works with Python DB API 2.0

Python DB API 2.0

Postgres

Local File Storage

Dynamo DB

SQL Alchemy

Snowflake

FTP

Singer Taps (any)

SqLite

Google Big Query

AWS S3

Kafka Producer

My Sql

Google Cloud Storage