PouchDB and CouchDB were what inspired me to build Triplit[1]. The idea of having an identical (or merely similar in the case of Pouch/Couch) query engine accessible on client and server is insanely powerful.
The author links to a much longer post on handling conflicts which is worthy of its length because it's not a pleasant experience with this setup.
I highly recommend trying a modern setup from one of the many new local first solutions [2]
Great to see the founder of Triplit here! PouchDB was definitely ahead of its time, and it’s exciting to see its legacy being carried forward by so many modern local-first (lo-fi) solutions. I’ve also been a longtime fan of Triplit - its API, docs, and overall DX (plus direct community interaction) are excellent.
Also Triplit's universal client/server model means you never have to care about all the complexity of other solutions that require to get the transformations between client db & server db just right.
A really impressive piece of work! (another great one with a universal db is jazz.tools, especially if you need encryption)
Couchbase had a lot of subtle incompatibilities with PouchDB in my attempts to use it on a past project. The big one I recall was that Couchbase had far more restrictions on key names and that required some workarounds.
(Though to be fair Apache has been threatening to shrink the key space greatly as well with the considered move to the Foundation-based "CouchDB 4.0".)
Ah good to know. For my use cases I would probably be looking at Couchbase Lite [1] not PouchDB. I’d be more concerned about file attachment sizes and syncing.
That's a related thing I learned from that past project: file attachments are also extremely vulnerable to compatibility issues between CouchDB/PouchDB/Couchbase/Cloudant/etc. Even when they claim to support sizes large enough, they often fail to sync even when small. Attachments in general in the CouchDB protocol are a bottleneck that slows or effectively breaks sync if you have too many to sync at once, because most CouchDB-ish implementations tend to only process a single document at once among other factors.
The project I was working on involved regularly taking some photos and all the sync issues with attachments led to moving to syncing only metadata like width, height, and Blurhashes in the database itself and use a reference URL/pointer to a content-addressed storage (basically an S3 bucket-like with git-like hashes for filenames) we could background sync more directly outside of the CouchDB protocol. (Blurhashes were particularly useful there while waiting on that additional background sync phase after the CouchDB sync.)
With React Native yes but not yet with Swift. There's been quite a few requests to my surprise through--I figured CloudKit, etc would be sufficient on iOS but I don't have experience there.
My ideal version of this is simple: just define the queries you want (no matter how complex) and the you'll get exactly the data you need to fulfill those queries, no more, no less. And the cherry on top would be to have your queries update automatically with changes both locally and remote in close to real-time.
That's basically what we're doing with Triplit (https://triplit.dev), be it, not with SQL--which is a plus for most developers.
Well yeah choosing what to query, a filter and an order is at the heart of all query-like things. Buy SQL specifically is a text language not a fluent api.
SQL is a relational algebra language which, unlike imperative languages, is compiled into an execution plan depending on the existence of indexes, table sizes, data types, etc. The reason you should love it is that you don't need to know the details when the data model is set up well. And DBMSs also have commands to view the execution plan, should it have otherwise been slow, such that you know what to do to improve it. It's really not that complicated once you learn how it works.
We apply incremental, streamable "joins" (relational queries) for real-time syncing between application client and server. I think much of the initial research in this space was around data pipelines but the killer app (no pun intended) is actually in app development
I agree completely! We've always talked about this, but we haven't really seen a clear way to package it into a good developer UX. We've got some ideas, though, so maybe one day we'll take a stab at it. For now we've been more focused on integrations and just building out the platform.
I've used React Native quite a bit in the past and I gotta say I wish it didn't have to exist at all.
It's often times fine on iOS and then incredibly slow on Android. Hermes is very exciting but still requires many polyfills to make simple NPM packages work. I hope one day, the web (and embedding web apps on mobile) makes React Native fully obsolete.
I've had the complete opposite experience. React Native has been amazing for the team I work on. It's fast, quick to develop in, and I've never felt blocked. Camera, TCP sockets, custom Bluetooth protocols, Protobuf, etc. React Native + libraries do it all.
That doesn't make sense to me at all. When you delegate heavy workloads to native view engines, both rendering and interfacing with native media/sdp will undoubtly be faster. The only issue i can see is if you install a lot of web only npm packages which has to go through the translation layer.
That's not what react native is for. Its for building native applications with a react-like syntax for the views. To me it seems like you used hammer to put in screws, no?
No? Most modern apps these days _are_ just web apps. Why does it need the overhead architecture of running full standalone applications when a web clip works just as well? Plus there's no need for updating.
If that's the case, then why does nearly every company have a native app in addition to their website? Surely, they could save a lot of time and money by just not building and maintaining the app.
Web apps will always on paper be slower (not including badly made applications on both) than native apps, and also bring in additional security issues (since code is loaded remotely) than native applications.
apps built using the webview don't necessarily have to be loaded remotely. If you're building with capacitor, then nothing happens remotely until you choose.
The funny part is that it’s actually not a pain in the ass to distribute. On iOS, saving the current link to the homescreeen installs the PWA (if available). You don’t have to do anything other than tap a button in the share menu. iOS treats it like an app, showing it in the app switcher and search, etc.
People just don’t know it’s an option. And as you say, there are fewer native APIs available to it.
Native apps make more money, because of phone data. A lot of effort is now put into facilitating cross-platform builds, a wrapper/container around a glorified web app is among the easiest ways
Most modern apps are just services. If you don’t mind you computer being a thin client to someone’s computer on the internet, go ahead. But the rest of us do use native software because the web is just a pale imitation.
A lot of the newer local first systems, like Triplit (biased because I work on it), support partial replication so only the requested/queried data is sent and subscribed to on the client.
The other issue of relying on a just the server to build these highly collaborative apps is you can't wait for a roundtrip to the server for each interaction if you want it to feel fast. Sure you can for those rare cases where your user is on a stable Wifi network, on a fast connection, and near their data; however, a lot computing is now on mobile where pings are much much higher than 10ms and on top of that when you have two people collaborating from different regions, someone will be too far away to rely on round trips.
Ultimately, you're going to need a client caching component (at least optimistic mutations) so you can either lean into that or try to create a complicated mess where all of your logic needs to be duplicated in the frontend and backend (potentially in different programming languages!).
The best approach IMO (again biased to what Triplit does) is to have the same query engine on both client and server so the exact query you use to get data from your central server can also be used to resolve from the clients cache.
I've stated before that I think the main thing holding back collaborative text / sequence CRDTs is integration with a production database.
Eg-walker looks interesting because it might lend itself to be integrated into a database because the operations are immutable and only appended. However, to demonstrate the effectiveness of these algorithms library authors (see Yjs, DiamondTypes, etc) build stand-alone data structures (usually specialized search trees) that most databases already provide.
Personally, I've been trying to adapt a Piece Table[1] to be collaborative and stored in Triplit[2] which runs on both client and server and already implements logical clocks but I might see how well I can adapt this algorithm instead!
This seems to be a holy grail, to be honest! Super-simple database representations with barely any processing required on the "write path," instant startup, minimal memory requirements on both server and client without a need for CRDT data structures to be in memory, none of the O(n^2) complexity of OT. In fact, if I'm interpreting it correctly, it should be straightforward to get this working in a serverless environment without any notion of session fixation, nor active documents needing to be kept in memory.
I can see this completely reshaping the landscape of what's possible with collaborative documents!
Egwalker has one other advantage here: the data format will be stable and consistent. With CRDTs, every different crdt algorithm (Yjs, automerge/rga, fugue, etc) actually stores different fields on disk. So if someone figure out a new way to make text editing work better, we need to rip up our file formats and network protocols.
Egwalker just stores the editing events in their original form. (Eg insert “a” at position 100). It uses a crdt implementation in memory to merge concurrent changes (and everyone needs to use the same crdt algorithm for convergence). But the network protocol and file format is stable no matter what algorithm you use.
I’ve loved learning all your detailed info on CRDT work. Thank you for progressing the field!
Since it stores all the editing events, does this mean that the complexity of opening a document is at least O(N) in terms of number of edits? Or are there interim snapshots / merging / and/or intelligent range computations to reduce the number of edits that need to be processed?
You can just store a snapshot on disk (ie, the raw text) and load that directly. You only ever need to look at historical edits when merging concurrent changes into the local document state. (And thats pretty rare in practice).
Even when that happens, the algorithm only needs to look at operations as far back as the most recent "fork point" between the two branches in order to merge. (And we can compute that fork point in O(n) time - where n is the number of events that have happened since then). Its usually very very fast.
In an application like google docs or a wiki, the browser will usually never need to look at any historical changes at all in order to edit a document.
Awesome, I'm been following Seph's work for many years! Always thoughtful and well-executed. Probably the most prolific and insightful engineer in the "collaborative text editing" universe.
I use ShareDB every day, which originated from Seph's excellent work on OT algorithms. Good stuff!
