Beyond the general idea of Matrix itself, a nice snapshot of all the stuff currently happening in the Matrix ecosystem right now is the This Week In Matrix blog series, which goes out every Friday. This week’s one is at: https://matrix.org/blog/2020/08/21/this-week-in-matrix-2020-...
Some of the stuff I’m personally most excited about right now is Hydrogen (https://github.com/vector-im/hydrogen-web): a super lightweight web client which heavily leans on indexeddb for storage and so minimises RAM. My primary account has 3000 rooms and about 450K visible users - on element-web this uses 1.1GB of RAM, but on hydrogen it uses 14MB :) We’re also looking at glueing E2EE into Hydrogen via the crypto layer of matrix-rust-sdk, which could be a massive win in terms of reusing and auditing a robust safe implementation for the Hard problem of encryption, rather than implementing it from scratch every time in every new client.
Dendrite (go server impl) progress is also very promising - we’ve been focusing on it (at last!) since the beginning of the year, and going to enter beta in the next few weeks. A typical small/medium live server is currently using 57MB of RAM (rather than synapse’s ~500MB). We haven’t started optimising yet :)
Synapse has also finally come of age - the whole codebase has almost been switched to twisted/asyncio rather than twisted deferreds, which both speeds things up a bit but also gives us way better visibility on profiling for future perf work. Almost all bits now scale horizontally (apart from the master dispatcher process). Synapse isn’t going away - we see Synapse and Dendrite coexisting much as Apache and nginx coexist today.
Conduit is also super exciting as an entirely community developed Rust homeserver, which currently runs ~10x faster than either Synapse or Dendrite (but doesn’t federate yet, or scale horizontally). It’s really fun to see an indie FOSS server implementation properly take off :)
Finally, P2P Matrix is coming together well - compiling Dendrite to WASM and running it clientside, using Yggdrasil currently as an overlay network to tunnel through to other P2P nodes.
In short: the Matrix ecosystem is in a fairly healthy place right now - the next gen of clients and servers should be incredible :)
Thanks for the insight, and had completely missed the blog!
On dendrite: What's your impression of friction needed to migrate an existing synapse home server to dendrite? Would bridges like the mautrix family be API compatible or would they have to add additional support for dendrite? And how far away would you say you feel it is until dendrite is stable and ready for individuals as personal production server?
Not looking for dates, just a feeling for how worth it is to invest getting deeper into synapse if one intends to migrate to dendrite eventually.
The best way to migrate from synapse to dendrite will probably be by decentralised accounts. We have plans (driven by p2p) to let your account exist on multiple servers at the same time. The sort of way to do this would be to have @alice:example.com sign something on @alice:elsewhere.com to prove that she’s the same person. She then participates in all the same rooms from both accounts, which are presented as a single logical identity to everyone else.
So example.com could be synapse and elsewhere.com could be dendrite, and she would automagically replicate her account over between the two. If she turned off example.com, she’d have migrated over.
This is scifi right now, but it’s getting closer, as P2P isn’t very usable if you can’t sync your account across multiple devices/nodes.
Alternatively someone could write a script to migrate synapse’s db into dendrite’s schema (or dendrite’s kafka logs), but it’s be pretty fragile given how rapidly both synapse and dendrite evolve.
tl;dr: jump in with Synapse today - the water’s great (even if others have been scalded in the past). Migration to Dendrite or Conduit or whatever in future should be transparent, one way or another.
Will it be possible to temporarily turn off example.com, let its absence propagate across the network a bit, then turn it back on under a new server? Synapse has been good to me, but I know Rust more than Python, and Conduit's use of Sled is one less moving part for me to maintain for my single-user server. Looking forward to migrating to a lighter server I can potentially hack on, and would rather not change the underlying DNS infrastructure much.
No, the idea is that to migrate you would have to run both servers side by side and then kill the old one. We haven’t yet figured out how you’d point the old matrix id to refer to the new server (as I assume your concern is keeping the same mxid as before), but presumably there will be a way to do so.
Nice work. I’ve followed Matrix since the start and also contribute code to Synapse. I’m curious about Dendrite entering beta. Would you say it’s stable enough as a daily driver?
Hydrogen also looks super interesting, although I would also welcome a fully featured native client.
Dendrite should be stable enough to experiment with as a daily driver in a few weeks when it enters beta. It won’t be bug free, but it should be largely feature complete and pass most of the sytest integration test suite.
Matrix is great for bridging[0] between different chat applications. Personally I use it a lot to bridge to IRC, which is a great alternative to the older method of setting up a server for use with an IRC bouncer. Since there's various Matrix clients for mobile (I use Element), you can chat on IRC from anywhere. If this gets more people into IRC I'm all for it because it's one of my favorite places on the internet for discussing and learning things.
There's a comprehensive tutorial[1] on how to connect to Freenode via Matrix.
Gitter ⇔ Matrix bridge is pretty good too, since Gitter is pretty horrendous on web and especially mobile.
As someone who's in a Gitter room with a couple of people using the Matrix bridge: it works, but I wouldn't call it great. Messages by Matrix uses look like messages by the same user ("Matrixbot" or something like that), which simply prefixes its messages by `<username>`. @mentioning people no longer autocompletes, and you often see Matrix users trying to do it as well but inadvertently using someone's display name rather than their actual handle, thus not resulting in a notification. And when a Matrix users tries to edit their message, the edited message just gets sent again in addition to the non-edited one which is still there.
Bridges suck. You need some end-end monitoring. Most don't. So you never know if your message will reach everyone else, making bridges very shitty to use.
In my own estimation, if anyone ever wants to engage with instant messaging[0], and wants it to be in an actually secure manner, the best option would be Briar[1]. It's anonymous (uses tor), P2P (meaning no data is accessible to third parties--unlike Matrix, which is federated), encrypted, and quite censorship resistant.
[0] I don't think it's an efficient, non-distracting or just desirable means of communication.
> meaning no data is accessible to third parties--unlike Matrix, which is federated
I acknowledge that this isn't an option for everyone but for the HN crowd, it's fairly easy to run your own homeserver. You just need to install Synapse [0] (which has a convenient Docker image [1]) and give your server a domain name.
If you do this, your metadata will only be accessible to yourself and the homeservers of your contacts (who could share your server or run their own).
Message contents are usually e2e encrypted, so the homeserver you use is irrelevant.
> And Linux, as well--including the Librem 5 phone[0].
I think the point is more that in e.g. the US, nearly half the population has an iPhone [1], so a chat service that is only available on Android and Linux can't spread as much.
>I think the point is more that in e.g. the US, nearly half the population has an iPhone [1]
Oh, no, I was absolutely aware that he was referring to the Android-iPhone pairing, as in "it's Android-only as opposed to available on both Android and iOS, as every major messaging app"; I just wanted to specify that it is not only available on Android, but also on Linux (PC and phones--I guess you can use it on the Pinephone, too, besides the Librem 5).
Keen to use this for notifications and web hooks etc. slack seems great but would rather not buy into yet another proprietary ecosystem if I can help it
Sorry if I wasn’t clear - there also needs to be server libraries.
I can spin up a web server in python, Ruby, JavaScript, etc in a couple lines of code. With matrix I need to install java and then do a whole bunch of config.
My belief is that as long as that hurdle remains, it won’t hit widespread adoption, even if the technology is great.
Personally, I really like it, and have used Riot For a couple years now. I’d love to use it for our company, but it’s not yet there for ease of use on the server side.
I think it will get there, it’s just not there now.
I've been running Synapse from home for more than a year, so far nothing is too painful about maintenance, except at every update its asking to review the changes of the homeserver.yaml config file from the terminal, I would have liked a more automatic approach, like when a setting is different from the new one, just leave it at it is and update the rest.
The single biggest difference is that Matrix is effectively a realtime decentralised conversation store (like NNTP or a decentralised form of IMAP), where conversation history is replicated equally over all the servers participating in a room - a bit like Git. Each partcipating server owns the conversation equally; none is a single point of control.
XMPP is a message passing protocol at its core, more like SMTP. MUCs have a single logical focal point of control.
I would say the availability of publically hosted bridges, use by large FOSS orgs like Mozilla and Drupal, and an all-in-one spec which is not fragmented into separate specification extensions are some differentiators.
This is completely untrue and disingenuous; there are loads of Matrix implementations that support E2EE, and only one that happens to use Electron.
As of a few months ago, the list is: Element Web (via matrix-js-sdk), Element iOS (via matrix-ios-sdk), Element Android (via matrix-android-sdk2), the old Riot Android app (via matrix-android-sdk),
weechat-matrix (via matrix-nio),
weechat-matrix-rs (via matrix-rust-sdk),
Mirage (via matrix-nio),
Nheko (via mtx-client),
gomuks (via mautrix),
Seaglass (via matrix-ios-sdk),
OCRCC Chatbox (via matrix-js-sdk),
FluffyChat Flutter, Daydream via matrix-rust-sdk, etc.
There’s also pantalaimon (built on matrix-nio, and in future matrix-rust-sdk), which lets any Matrix client talk e2ee.
Hydrogen (github.com/vector-im/element-web) is also about to sprout e2ee via matrix-rust-sdk.
So Pantalaimon is something that needs to be added and configured, how easy is it to fuck up this configuration and accidentally send a non-E2EE message? Is it a program you can forget to launch? Is it something that can crash and make the user vulnerable?
How many of the apps you listed in second paragraph spawn E2EE chat by default?
How many of the apps warn about E2EE not being enabled when joining a room?
How many of the apps warn when E2EE is broken by e.g. IRC-bridge-bots?
Pan won’t let you send plaintext msgs in E2EE rooms. If you don’t launch it, your client can’t connect. It doesn’t have any known downgrade vulns. You can’t downgrade an E2E room to a non-encrypted one.
I haven’t checked to see which of those apps spawns E2EE chat by default. Element does, and I’d assume the other client devs are following suit; there’s no reason not to. Likewise, i haven’t surveyed to see what UI they use to warn for unencrypted rooms.
We’re currently working on an MSC to better advertise what bots/bridges exist in a room so you can track whether any is likely to be leaking your conversation to an unencrypted system. This is inevitably going to be on a best effort basis though.
Can you please not say that it's "completely untrue and disingenuous"? To me, working end-to-end encryption means that I can verify my own and other devices. Out of these, maybe Seaglass (macOS only) and FluffyChat Flutter support end-to-encryption in a way that I would use them. That's not "loads of Matrix implementations".
Can you recommend a non-official client with end-to-end support including cross-signing and device verification?
things building on matrix-rust-sdk will get cross-signing shortly too, fwiw. but it seems to be missing the point to pick the most recent e2ee feature (cross-signing) which landed a few months ago, and complain that not all clients have implemented it yet? this thread was originally about e2ee in general rather than a specific subfeature. e2ee is a constant work in progress: for instance, we’ve just added fallback otk keys and device dehydration - but that’s landing first in matrix-js-sdk (thus element-web) and then everyone else will implement as fast as they can. Sorry if that isn’t fast enough for your personal requirements :)
I find this accusation quite offensive. I was under the impression that the standard here was to assume good faith. Please retract this statement.
> Element Web (via matrix-js-sdk), Element iOS (via matrix-ios-sdk), Element Android (via matrix-android-sdk2), the old Riot Android app (via matrix-android-sdk)
If anything counting all of these as separate implementations is the disingenuous thing here.
Admittedly though it seems that my knowledge on the topic was quite outdated. Glad to see that it improved in that front.
We use HTTP+JSON as the baseline default transport because it’s so ubiquitous and well understood. Anyone who’s ever touched a browser or curl knows how to use it.
yeah, the whole P2P Matrix project is using different transports. One is CBOR+CoAP+Yggdrasil, another is JSON+QUIC+Yggdrasil. The plan is to pull these back into mainstream matrix when proven, especially as a push replacement.
The steps you list above are not particularly difficult, but if everyone followed them, the decentralization features might as well be redundant because we would all be on one Central set of servers.
The hard part is step 7: nobody else I know follows steps 1-6. I’m quite capable, thanks, others know matrix as a 1999 film. This isn’t even mentioning your steps remove the decentralised bit, which if you recall is the part I said made it hard to use!
As for your final quip. Gmail and Facebook flourish because those protocols are hard to use. No I don’t find them hard to use, email was my last job, web dev is my current job. It isn’t everyone’s skill set though.
I appreciate the attempt to make me out as a simpleton but mate, a chat system isn’t worth the servers it runs on if there’s nobody to talk to
Matrix backs the IM needs of the entire French government [1] and more recently signed the largest single instant messaging deal ever to help German schools manage remote education[2], it is also used in mission critical situations such as emergency services response.
I think it is fair to say that Synapse is more than production ready. Admittedly it can be hard to admin out of the box for smaller scale instances and we intend to improve on that experience over the coming months. Alternatively, hosting options are available[3].
After a hiatus, Dendrite is under active development and making rapid progress. It is an alpha release at the moment and not yet ready for production settings, but a beta release is coming soon.
Please, Synapse isn’t production ready, at all. See this issue for example: https://github.com/matrix-org/synapse/issues/3364, reported back in 2018, labelled highest priority since then and still not fixed. This is indeed a huge problem, because if you don’t have shit tons of money to put in your database server’s storage you’ll quickly get stuck. Some workarounds mentioned are non-safe hacks and shouldn’t be necessary for a production ready messaging server.
This bug is not generally a significant cause of disk space utilisation, unless you’re running a server the size of matrix.org (which is the only reason it’s marked P1).
Ive been running synapse for atlest 3 years for very active group of about 40 on tiny 1gig ram server and its been great. The main issue is if you start to federate with main matrix instance which is gigantic and can slow things down.
But i dont need to federate and use it more as Slack and it just keeps getting better.
I seriously have no idea what's your problem with synapse. I've been running my own homeserver for over a year now and synapse is a perfect piece of software. I installed it without any hassle and it's been running without a single issue since then. I install new versions on the same day they are released without skipping a heartbeat: that's how good it is.
I used to have more complaints about matrix clients, especially on Android, but things got much better recently.
Recent benchmark showed Dendrite (go) is slower than Synapse (Python) but I'd guess it's from all the context switching with the multi process design of Dendrite
The lack of other "serious" servers is also scaring me. XMPP can have its share of problems but this is one area which is pretty decent. Plenty of "battle tested" servers.
The economics of this are curious actually. Element has had investments from Automattic, VCs and Status.im (no governments, ftr). However, the only reason people invest is in the expectation that Element can be a sustainable healthy business - which it does so by providing Matrix SaaS hosting, and support/consulting for big on premise deployments.
In other words, it’s critical that folks use the managed hosting in order to be sustainable, otherwise we’re just burning fossil fuel, as it were.
I misspoke. I wanted to say that the money from the governments are part of the revenue as "support/consulting for big on-premise deployments". Sorry about that.
In any case, I would be surprised that Element relies on getting significant part of their revenue from the SaaS. Isn't p2p-matrix going to cannibalize that market? I could swear that the strategy for you would be to chase the enterprise/government market and let p2p-matrix as an alternative for network effects and/or those who don't want to pay and prefer to manage things themselves.
np - just wanted to clear up the confusion if folks might think Element might be part-owned by nation states, which it isn’t.
So we expect revenue from SaaS, where plain TCO says it’s cheaper to get us to host your data than pay in-house people to sysadmin it for you. It’s still your DNS and your keys, and we provide db snapshots if you want, so it really is just outsourced hosting. There is definitely a market for this, as well as separately providing support for massive on premise deployments.
P2P then drives both by network effects. It effectively becomes the default free-for-all platform - but anyone who actually wants a serious home for their data (e.g. any business) will want to find a server, whether that’s selfhost or SaaS.
Ok, cool. Follow-up questions: I honestly believed that your SaaS offering was just a middle-market play and that you expected to be filled by a cottage industry of sorts. Don't you have any concerns about other business undercutting you? In a way what I am doing with communick competes already with your hosting (or I can dream about it) and I won't have the overhead of funding development. What if AWS/GCS/Azure decides to offer Matrix as well?
We don’t expect to be the only SaaS players - but we still think it’s a good market. The more hosting solutions the better - we believe there’s enough customers for everyone to go around :) We‘ve started maintaining a list in matrix.org - and if the big boys started offering Matrix hosting too; it sounds like a good problem to have! (Plus the sort of people interested in Matrix are probably not that interested in buying it from GAFAM ;)
Currently, I send (the batched equivalent of) $10/month on Libera Pay and run my own homeserver - is there a meaningful difference between this and using the paid/sponsorship hosting?
I'm especially curious about the effects individually and in aggregate; I.E. if everyone did what I do.
Donations go to the Matrix.org Foundation and help pay for core team dev, but also are really useful for pointing bigger donors at to say “look, we get $4K/month on Patreon + Liberapay etc; this shows we are a good project to back; how about you match it?”
Buying hosting from Element however also funnels back to fund core Matrix dev - because almost all of Element’s work is donated (unilaterally, in an asset lock) to the Matrix.org Foundation. In other words, anything an Element employee commits into https://github.com/matrix-org automatically becomes the copyright assigned to the Foundation. This also has the separate multiplying factor that the more evidence there is to run a sustainable SaaS business on Matrix, the easier it is for Element to raise funds and keep going - and thus continue to support the core Matrix dev.
Sorry it’s so complicated, but funding FOSS is fiddly (as Mozilla demonstrates...)
Is your perception of a company’s ‘need’ of financial support a good way to decide wether or not to support them? Probably not. If you believe in what they are doing and get value from it, they earned it and deserve it.
No. But the whole idea of my support behind Matrix is that it allows for federation and distribution of power among different companies and entities.
If the development of Matrix is somewhat secured and if Matthew and the team has already been compensated for their work, I'd rather support other players than concentrating it further on Matthew's hands.
Development of core Matrix is somewhat secured via funding from Element and donations, but Element is not yet profitable/sustainable as a company. Until it is, I wouldn’t be doing my job if i didn’t encourage folks to support core Matrix dev by using Element services.
That said, folks should absolutely donate to non-core-team Matrix projects like Conduit or Nheko or Dimension etc too, to support the hetrogenity of the ecosystem and keep it healthy and balanced. And once Element is sustainable (and by extension the Matrix.org Foundation funding is stable), the balance should shift even further towards ensuring the rest of the ecosystem is successful.
I believe they tried blaming this on the creator of the C++ port of their server software without any proof and it sort of left a bad impression with me.
The post makes their mistakes pretty clear, I think. Public out-of-date Jenkins instance, SSH forwarding enabled by developers for all matrix.org servers, and not realizing they only rotated their personal Cloudflare API key and not their admin one.
It's very embarrassing for sure, but tons of huge private corporations have been breached through worse mistakes than this. Making their Jenkins public was probably the worst decision. They explain why they did it, and it's not unreasonable (radical openness and transparency, basically), but they should've thought it through more.
I think everyone should be free to post whatever they want; nonetheless, providing some kind of source to incriminating claims would be great--if only to make verifying them easier to other users.
In any case, this kind of posts is a reminder to stay alert and think critically; otherwise, we would believe many instances of misinformation without giving them a second thought. And we cannot expect others to downvote comments to oblivion or moderate them: it's something we ourselves have to be responsible for.
That post was nothing to do with the security incident in question here (which happened April 11th 2019; that post is from March). The details in that post are sadly true (as others confirm on that thread).
However, we have no reason to believe there was a link to the April incident.
yes, and he was, by his own admission: repeatedly bragging about exploiting bugs in Matrix’s beta design which we hadn’t fixed fast enough in his estimation (hijacking Matrix HQ; bricking #matrix-dev; threatening to loop over all public rooms to brick them unless we paid him; etc). It wasn’t exactly subtle, and it’s bizarre if you think we’re making this up or presenting without evidence.
However, the matrix.org security breach in Apr 2019 was unrelated to him, as far as we know.
Some of the stuff I’m personally most excited about right now is Hydrogen (https://github.com/vector-im/hydrogen-web): a super lightweight web client which heavily leans on indexeddb for storage and so minimises RAM. My primary account has 3000 rooms and about 450K visible users - on element-web this uses 1.1GB of RAM, but on hydrogen it uses 14MB :) We’re also looking at glueing E2EE into Hydrogen via the crypto layer of matrix-rust-sdk, which could be a massive win in terms of reusing and auditing a robust safe implementation for the Hard problem of encryption, rather than implementing it from scratch every time in every new client.
Dendrite (go server impl) progress is also very promising - we’ve been focusing on it (at last!) since the beginning of the year, and going to enter beta in the next few weeks. A typical small/medium live server is currently using 57MB of RAM (rather than synapse’s ~500MB). We haven’t started optimising yet :)
Synapse has also finally come of age - the whole codebase has almost been switched to twisted/asyncio rather than twisted deferreds, which both speeds things up a bit but also gives us way better visibility on profiling for future perf work. Almost all bits now scale horizontally (apart from the master dispatcher process). Synapse isn’t going away - we see Synapse and Dendrite coexisting much as Apache and nginx coexist today.
Conduit is also super exciting as an entirely community developed Rust homeserver, which currently runs ~10x faster than either Synapse or Dendrite (but doesn’t federate yet, or scale horizontally). It’s really fun to see an indie FOSS server implementation properly take off :)
Finally, P2P Matrix is coming together well - compiling Dendrite to WASM and running it clientside, using Yggdrasil currently as an overlay network to tunnel through to other P2P nodes.
In short: the Matrix ecosystem is in a fairly healthy place right now - the next gen of clients and servers should be incredible :)