Creme Collective
ELT Migration Architecture · March 2026
Data Pipeline Migration
Databricks → AWS (RDS PostgreSQL + ECS Fargate + S3)
Retiring
Current Architecture
Databricks
Data Sources
API
Camelot OData
3PL · 18 endpoints
QB
QuickBooks
Creme + Sage orgs
GS
Google Sheets
SWEED mapping
Ingestion
Nb
PySpark Notebooks
OData client + batch processing → Delta tables
Medallion Pipeline (dbt-core + dbt-databricks)
Bronze
Raw Delta tables
18 tables
Silver
dbt SQL models
17 models
Gold
dbt SQL + Python
7 models
Platform Services
Job
Databricks Jobs
2 daily · UI-managed
WH
SQL Warehouse
Serves queries
Sec
Databricks Secrets
creme-scope
Δ
Delta Lake
Unity Catalog
Consumers
DB
Creme Dashboard
databricks-sql-connector
RP
Reporting Portal
SQLite sync
T
Tableau
Being retired
Target
Target Architecture
AWS
Data Sources (unchanged)
API
Camelot OData
3PL · 18 endpoints
QB
QuickBooks
Creme + Sage orgs
GS
Google Sheets
SWEED mapping
Orchestration + Compute
EB
EventBridge
Cron: daily 3:00 AM EST
ECS Fargate Task (Docker container from ECR)
1
Ingest
— Python scripts
(requests + pandas)
fetch data → write to RDS PostgreSQL bronze schema
2
Transform
—
dbt run
(dbt-core + dbt-postgres)
builds silver → gold
3
Validate
—
dbt test
runs data quality + business logic checks
4
Notify
— Slack webhook reports success or failure
Storage
PG
RDS PostgreSQL
All schemas: bronze, silver, gold · Single source of truth
bronze_camelot
Raw ingested data
18 tables
silver_camelot
dbt SQL models
17 models
gold_camelot
dbt SQL + Python
7 models
S3
S3 Bucket
Raw file staging + Parquet backups
SM
Secrets Manager
API credentials
Consumers
DB
Creme Dashboard
psycopg2 / asyncpg → RDS
RP
Reporting Portal
Direct PostgreSQL queries
Migration Path
1
Export Bronze
Databricks bronze tables → S3 (Parquet) → COPY into RDS PostgreSQL
2
Rebuild Layers
dbt run
rebuilds silver + gold in PostgreSQL from imported bronze
3
Validate
Compare row counts + key aggregates between Databricks and PostgreSQL outputs
4
Switch Consumers
Dashboard + Portal swap connectors to psycopg2 → RDS PostgreSQL
5
Decommission
Shut down Databricks SQL Warehouse, archive notebooks, cancel subscription