In private development
In development, with no date. PostgreSQL is a source only: Emolysis reads from it and never deploys to it. Connecting, the catalog mirror and sampling are tested on a local PostgreSQL server; logical replication and managed services are not covered yet.
PostgreSQL
PostgreSQL as a source.
Emolysis reads from PostgreSQL: it mirrors the catalog and samples tables into DuckDB on your machine. Landing the data on your production platform with tested Ingest (batch), Incremental append and CDC merge pipelines is planned.
In development
Connect
A password, over TLS
Sample
A slice into DuckDB on your machine
Prove
Golden tests on DuckDB
Land
Bronze tables on your platform
Overview
PostgreSQL as a source.
PostgreSQL as a source is in development, with no date yet. Emolysis reads from PostgreSQL and never deploys to it; landing the data on Databricks, Snowflake or Microsoft Fabric is planned.
The source profile records what makes PostgreSQL data tricky to move: quoted, case-sensitive identifiers, jsonb and arrays, numeric without a declared scale, and timestamp versus timestamptz.
Changes can come from logical replication, or from updated-at or sequence watermarks. Logical replication feeds the CDC merge shape; watermarks feed Incremental append. Which of these comes first is not decided.
Managed PostgreSQL services are not verified yet. For development, Emolysis samples a slice into DuckDB on your machine; what you deploy runs on your platform, under your account.
Source profile
What the profile captures.
Sources use the same profile format as platforms, with fields of their own. This is what the PostgreSQL profile is drafted to capture.
- Dialect
- PostgreSQL SQL, including quoted, case-sensitive identifiers
- Types that need care
- jsonb, arrays, numeric without a declared scale, and timestamp versus timestamptz
- Change options
- Logical replication, and updated-at or sequence watermarks
- Sign-in
- Password authentication over TLS; managed PostgreSQL services not verified yet
- Sampling
- A slice of a table into DuckDB on your machine
- Deploy target
- None. Data lands on Databricks, Snowflake or Microsoft Fabric
Add a source
Adding a source.
How adding PostgreSQL to a project works, step by step.
Add a source
In a project, you add PostgreSQL as a source and enter its connection details.
Sign in
With a password, over TLS where the server offers it. The password is kept in your machine's credential store and never sent to Emolysis.
Sample a slice
For development, Emolysis samples a slice of a table into DuckDB on your machine, so golden tests run against the real shape of your data.
Find the changes
Each table needs a way to find changes: logical replication, or an updated-at or sequence watermark. Which of these comes first is not decided.
Landing
Lands on your platform.
PostgreSQL has no deploy target in Emolysis. Its data feeds the Ingest (batch), Incremental append and CDC merge shapes, which deploy to Databricks, Snowflake or Microsoft Fabric and run there.
Time zones kept straight
timestamp and timestamptz differ in whether they carry a time zone. The profile names the difference, so loads do not shift times silently.
Semi-structured columns
jsonb and arrays need a deliberate mapping on each platform. The profile lists them as types that need care.
Exact numbers
numeric without a declared scale can hold any precision. Mapping it to a fixed type is a decision the profile makes visible, not a guess.
Case-sensitive names
Quoted identifiers in PostgreSQL keep their case. Naming rules on landing account for it, and Dedupe / conform normalizes names in silver.
Shapes
Three shapes read from PostgreSQL.
PostgreSQL feeds three shapes. After landing, the rest of the pipeline runs on your platform like any other.
- Ingest (batch)Full extracts of PostgreSQL tables land in bronze on your platform, typed on load, with jsonb, arrays and numeric mapped deliberately.
- Incremental appendAn updated-at or sequence watermark selects the rows added since the last run, and moves forward only after the load commits.
- CDC mergeLogical replication provides the change feed. The merge applies the latest change for each key, in order, including deletes.
All ten shapes, with their templates and golden tests, are on the pipeline shapes page.
Read next
Glossary
Terms on this page
Questions
PostgreSQL questions.
Answers about PostgreSQL and how it fits. More in the FAQ.
Can Emolysis deploy to PostgreSQL?
No. PostgreSQL is a source only. Emolysis reads from it and never deploys to it; landing the data on Databricks, Snowflake or Microsoft Fabric is planned. If you need computed values back in an application database, the Reverse ETL shape does that.
Does Emolysis use logical replication?
Logical replication is one of the change options in the PostgreSQL source profile, next to updated-at and sequence watermarks. Which of these comes first is not decided. Logical replication feeds the CDC merge shape; watermarks feed Incremental append.
Does it work with managed PostgreSQL services?
Not verified yet. Emolysis connects with a password over TLS and is tested on a local PostgreSQL server; managed PostgreSQL services have not been tried.
Where are PostgreSQL credentials kept?
In your machine's credential store, the same as platform credentials. They are never sent to Emolysis.
Do production loads run on my laptop?
No. Sampling a slice into DuckDB is for development and golden tests. What you deploy runs on your production platform, under your account, like every other pipeline Emolysis deploys.
Nothing is released yet
Tell us what you load.
PostgreSQL as a source is in development, with no date yet. Early access starts with Databricks teams. If you load from PostgreSQL, tell us which change option you rely on.