Status
In private development. Early access by invitation. The names, options and formats on this page can change before the first release.
Status: PostgreSQL is in development as a source, tested against local PostgreSQL servers; managed services are not verified. SQL Server is planned as a source. A source is a system Emolysis reads from, not one it deploys to. Landing a source's data on your production platform, Databricks first, then Snowflake and Microsoft Fabric, is planned. The platform pages cover each one: SQL Server and PostgreSQL.
Source or platform
| Role | What Emolysis does | Examples |
|---|---|---|
| Platform | Writes code for it, tests it on DuckDB first, and deploys to it | Databricks, Snowflake, Microsoft Fabric |
| Source | Reads schemas and samples from it so pipelines can land the data elsewhere | SQL Server, PostgreSQL |
| Local | Runs your development loop and the golden tests | DuckDB |
What a source profile says
Sources use the same profile format as platforms, so the agent knows how to read them correctly. See Platform profiles.
- SQL Server: T-SQL, including bracket-quoted identifiers. Types that need care include
datetimeversusdatetime2,money,nvarchar,uniqueidentifierandrowversion. Change options include Change Data Capture, Change Tracking, androwversionor updated-at watermarks. - PostgreSQL: PostgreSQL SQL, including quoted, case-sensitive identifiers. Types that need care include
jsonb, arrays,numericwithout a declared scale, andtimestampversustimestamptz. Change options include logical replication, and updated-at or sequence watermarks.
Which change options Emolysis supports first is not decided.
Which shapes read from a source
Three shapes start at a source: Ingest (batch) for a full or periodic extract, Incremental append for append-only tables with a watermark column, and CDC merge for tables whose rows change or are deleted. Which one fits depends on whether the source can tell you what changed. See change data capture and watermark.
What you do
You add a source in a project and enter its connection details. Credentials are kept in your machine's credential store, never in a project file. For development, Emolysis samples a slice of the source into DuckDB on your machine, so the pipeline can be tested without touching the source again. That sample read is a query on the source system.
Not decided yet
- How production loads from a source run, and where.
- Which change options come first for each source.
Where to start
The first platform to read about is Databricks.