PGlite Next - Based on Postgres 16.x

[pmp-p]

This PR changes for support building based on 16.x postgresql releases directly by patching in wasm support, emscripten dynamic loading for postgres extensions and ready for electric sql support via PGlite Typescript interface.

It also introduces CI with development builds on each PR, and support for extensions with pgvector being the first.

You can try this new build and the new REPL here: https://668ce81c27123c506c6a53a6--pglite-dev-demos.netlify.app/pglite/examples/repl

or, download the "PGlite" package from CI, unzip it, and manually install it in a project.

Each build now has a set of "interim build files", these are the output of the Emscripten build step. If you want to work on just the TypeScript side of PGlite, you can copy these files into /packages/pglite/release/ and run pnpm build:js to run the second step of the build.

Some of the issues that will be resolved by this PR include:

• Refresh materialised views Attempting to refresh a materialised view throws error #63
• Issues with TOAST Attempting to change column type throws error #83
• Missing timezone data Attempting to SET TIME ZONE throws error #62
• EMULATE_FUNCTION_POINTER_CASTS Investigate removing EMULATE_FUNCTION_POINTER_CASTS #56
• PGlite crashing after errors db enters invalid state, causing many subsequent errors and requiring a reload #94

[github-actions]

🚀 Deployed on https://669eb1931d0cff2d72a9e7cc--pglite-dev-demos.netlify.app

[github-actions]

Built bundles:

• Postgres WASM: https://github.com/electric-sql/pglite/actions/runs/10046980307/artifacts/1727408845
• PGlite: https://github.com/electric-sql/pglite/actions/runs/10046980307/artifacts/1727409003
• Interim build files: https://github.com/electric-sql/pglite/actions/runs/10046980307/artifacts/1727408929
• Demos: https://github.com/electric-sql/pglite/actions/runs/10046980307/artifacts/1727409136

[jueinin]

if this MR merged, if i want to using some other extension, for example pg-tde(encrypt the whole database),

how can i do this? should i write some script and compile it to wasm? but i'am not good at cpp or c language. can you provide some docs on this ? @samwillis @pmp-p

[samwillis]

Hey @jueinin,

Yes, we will provide docs and examples showing how to build custom extensions.

[gregnr]

Just a heads up, I'm hitting some issues running this on Next.js (webpack) with the error:

Module not found: Can't resolve './postgres.so'

I'm still digging into this, but I believe this is related to the new URL("...", import.meta.url) syntax (here) which Next.js tries to resolve at build time and can't since postgres.so doesn't exist for web builds. I'm fighting through some issues trying to get PGlite building on my local dev machine with the new infrastructure in this branch (running on a mac) which I'll need to resolve before I can debug this further.

[samwillis]

Hey @gregnr

Thanks for the report. I think @pmp-p may have been working on something in this area?

We should add e2e tests for various frameworks and bundlers to catch these issues.

[gregnr]

I managed to get past the above error by temporarily commenting out the postgres.so URL and switch logic. Interested to hear @pmp-p's thoughts!

Now running into a new unreachable WASM error that seems to happen in _pg_initdb:

Uncaught (in promise) RuntimeError: unreachable
    at postgres.05b6d054.wasm:0x33ae7a
    at postgres.05b6d054.wasm:0x253d96
    at postgres.05b6d054.wasm:0x325aef
    at postgres.05b6d054.wasm:0x2b883a
    at a._pg_initdb (chunk-6ZCTMRJP.js:15:55901)
    at Ha.Ds (chunk-6ZCTMRJP.js:15:82830)
    at async run (db.ts:24:5)

This happens during initialization for both new PGlite(...) and new PGliteWorker(...). Let me know if you've seen something like this or if a repro repo would be helpful.

---

Edit: Turns out the above issue was caused by loading a pre-0.2.0 database (via IndexedDB) into 0.2.0 PGlite. Deleting the old DB and starting fresh fixed the issue. I'll keep the above error in case there are plans for 0.2.0 to be backwards compatible.

[pmp-p]

@gregnr postgres.so ( which was in fact libecpg.so ) is gone, only some symbols were needed in there so i re implemented them in static pglite main, it makes the bundle 520KiB smaller and extensions building easier so it was really important to do it before merge.

freezing now, unless some bug is found or critical change required by upstream ofc !

[jueinin]

@pmp-p Hi I'am wondering, can we add a password feature for pglite?
just like postgres server side?

given the right password, so that this pglite can be connected?

[pmp-p]

md5 passwords can be handled and currently defaults to "password" but anyone could find it looking at the wasm or pglite source code

[samwillis]

Hey @pmp-p

This is all awesome, we are very nearly there.

There is a failing test for dumping a tar of the datadir, I will fix that, it's a Node version < 20 issue 🤦‍♂️

On your side:

• Some of the files in /patches have a mix of spaces and tabs for indentation, we should match the style of the Postgres source (tabs) for these patch files.
• There is some commented out source code in some of the patches, can we remove it now?
• There is quite a bit of usage of printf for debug logging in the patch files and new pg_main/proto.c and interactive_one.c, can we remove any that’s no longer required, and any that’s useful to keep swap to the Postgres elog macro? That way it’s configurable when it’s printed out with the debug level.
• Are the library_fs.js and library_pgfs.js patches still required now that extensions are handled in the ts code?
• I now include tintar via npm in the ts build, can we remove it from /patches?
• Are you still using the files in /test/? CI tests are in /packages/pglite/tests, and we have examples in /packages/pglite/examples. If they are useful still could we move them into a dir that doesn’t look like a CI test folder?
• Could we tidy up the ci scripts:
  • move cibuild.sh into /cibuild:
  • Add comments to explain what the files are for/how they work
  • Remove any commented out code that’s no longer required
  • Remove any blocks that are not used