So one of the smartass login messages in our internal slack is "JMAP will fix it".

Interestingly enough, JMAP will make this kind of thing much easier - and we have thought about it in the design! Our Cyrus server already internally keeps a blobId which is a digest of the binary content of each MIME part. Which means you can easily tell that two attachments are identical without reading the contents or guessing based on size and name.

And it's easy to search for messages with file attachments and fetch just the file attachement details via JMAP, making this kind of interface relatively easy to write.

The usual caveats about "we haven't got a particular timeframe in mind" apply - but once JMAP is standard, anyone can write a client to that standard!

Hi Bron. JMAP is a great design for what it is (a much better IMAP). Thanks for creating it. I can see JMAP becoming a great standard for email-oriented client/server communications.

That said, I'd still encourage you to think bigger to make a more general system. Of course, the risk of thinking bigger is you may end up with either nothing or a monstrosity no one wants to use. And also you'd likely have to adapt your business model eventually if the bigger picture changes through your actions. So, it's fair to think smaller too.

From a related post I made to the Thunderbird Planning list in April 2017 on what I see as the bigger picture: https://mail.mozilla.org/pipermail/tb-planning/2017-April/00... "My perspective is from wanting to create tools to support a "social semantic desktop" of which email is a subset of the items. Other types of items include news messages, RSS feed messages, chat messages, notification messages, IoT messages, blog posts, pingbacks, saved web pages, commands constructing complex shared documents, VR world update messages, code snippets, notes, and more. Email is a special case of a much larger world of messages. So, for me, a good assumption is that people will eventually have billions of message items comprising terabytes of data in their local (shared-with-email) store. Eventually each user would be part of a global federation of users that together have billions of times that number of messages as a federated semantic web. I understand that such a vision of messaging on that scale is not yet the goal of most people in this discussion right now. But I'd still urge people to think a little beyond redoing Thunderbird as-it-is and try to imagine a personal messaging system that processes, archives, indexes, and recalls vast numbers of items every day in interaction with a user and automated systems as part of a larger federation of similar systems using shared semantic standards."

I've been beating my head against that wall for years with very limited progress (as have others) -- so I can't say it isn't a high risk endeavor with a low probability of success. But the benefit for humanity could potentially be huge if such an effort succeeded. Maybe it might just take a few tweaks to JMAP to generalize it enough to realize some of that big picture?

JMAP is pretty generalizable to other datatypes! It's a large part of why we split it into Core and Mail. We also have Calendar and Contacts modules in a reasonable state already.

We also use JMAP-style APIs for Files, for Notes, and for all the extra Preferences and group management stuff in both FastMail and Topicbox.

Neat! Yes, I see the potential there, which is why I an interested in JMAP. Just wondering how it might could get even better at all that.

As related market research, Kolab uses IMAP as a general-purpose data store: https://github.com/pdfernhout/Twirlip2/blob/master/design/ma...

> I'd still encourage you to think bigger to make a more general system.

No! It's a trap!

I'm not sure if jmap is all that great (but pretty sure it's in the "all email protocols suck, but jmap socks less..."-bucket ;-)

Someone did think bigger, and the best of them begat lotus notes. And/or the original couchdb. Walk får enough down that path and you come up with distributed executable objects. And while the vision is pretty, it's anything but simple - or something that can survive for 40 odd years. Today, you'd probably get something like urbit. Cool research: yes. Might be useful: yes. Something that'll work well to access email from an iPhone, a mainframe, your openbsd firewall and a windows 10 tablet... Not yet - probably not ever. In fact, even the modest goals of jmap puts it at peril in this case.

Yeah, "thinking bigger" about JMAP might be a trap, sigh. I'd have to agree that JMAP may be better just being email-focused as we could really use something like that right now. Still, at the very least, it's always good to have some versioning to have a way of upgrading the protocol. I also wonder if there is some "low hanging fruit" for generic data exchange that JMAP could perhaps address better -- but I agree sometimes the low hanging fruit is dangling over a fast-moving river filled with alligators on a branch that would break if you put too much weight on it.

Lotus Notes and CouchDB (designed by someone with Notes development experience and related aspirations) are projects that do have quite a bit of alignment with what I am talking about. And they can be useful systems even with their limits. Notes suffers from being proprietary and having warts from its evolution (including how apps are implemented). CouchDB suffers the limits of trying to be both a database and a server at the same time and hitting limitations in both areas. It also unfortunately had a logo that could be misinterpreted as something offensive. That said, I do like CouchDB a lot as an inspiration and seven years ago or so I wrote a small app that runs on top of it.

Thanks for mentioning Urbit. After a brief perusal of their mission and code and a HN article on it ( https://news.ycombinator.com/item?id=8578151 ), it seems to me at first glance (perhaps not doing it justice) like a new flavor of Lisp (but with math limited to increments to produce a functional-ish Flux-like model) crossed with a bunch of techno-utopian aspirations about personalized distributed computing backed with some code and a lot of hand-waving. One can also see the economic model in there to please investors by creating an artificial scarcity of address space that is auctioned off. So it is a set of sense and nonsense it would take a while to sort through -- but so often the challenge in life is doing that for our own beliefs given everyone believes many useful things and a few problematical things. Certainly a lot of things it points to are worthwhile aspirations (like having more control over your own data) whatever one may think of the implementation. Some of its aspirations (the ones I'd tend to agree with more) are reminiscent of the FreedomBox project -- but without FreedomBox's practical grounding in mostly repackaging existing Linux technology.

In practice, these days I'm thinking more about a set of npm packages that make it easy to create your own personal server with a combination of "npm install" commands to add new capabilities (including perhaps a JMAP server for some of your data that could be interpreted as email-like messages). Ideally, that personal server could be federated with servers from other people using some standardized APIs (some of which may be very CouchDB-like). And if people need a unique identity (or several), that could be done by people providing public keys to others as their "identity" and using such keys for signing messages. That's a way to realize some Urbit-like aspirations in a more down-to-Earth way, but leveraging the Node ecosystem instead of how FreedomBox leverages the Linux ecosystem.

Here is a message I sent in 2010 to the Diaspora project along similar lines: "Raising the bar to supporting a Social Semantic Desktop" https://groups.google.com/forum/#!msg/diaspora-dev/TNNpvfFqN... "So, you can take all of my technical comments with a grain of salt in the sense of all that ambition (or whining or paranoia. :-) I acknowledge these suggestions are pretty ambitious, as above, and such ambitions would ask that the Diaspora team learn a lot about emerging ideas and do some really difficult things involving some bleeding-edge semantic stuff beyond the new (and good) stuff it is already doing -- and that it is possible such ambitious and risky efforts might fail for all sorts of reasons. Still, I contacted the Chandler/OSAF project about the Pointrel system semantic approach and they went another more mainstream way and failed anyway (well, it's still ongoing, but it lost its way and most of its funding somewhere along the line). But, I learn more each time, so maybe the above might seem more coherent and approachable than what I sent the Chandler/OSAF people years ago, and that in private, not in public where at least others could learn from it. :-) It's quite reasonable for anyone to say that, for now, a Semantic Web is not what Diaspora is about, as no one can do everything. But somehow, I feel, the really big future for something like Diaspora is going to be integrating semantic web concepts. So, at the very least, if Diaspora was built in such a way as to support future modules in that semantic direction, it might be an even more successful thing that totally and rapidly eclipses Facebook and many other technology platforms as well."

I wish I had more time to pursue these ideas in an open way other than as a hobby taking too much time away from family -- and involving unhealthily spending way too much time on the computer overall on top of regular 9-5 work time. I'm obviously personally invested in these ideas and enjoy programming in this area. But it's also a tough moral choice about spare time given the need for such systems to have a happier singularity factored against the very real costs to any individual/group and a very low probability of success for each specific effort. But collectively, if enough people try (alone or in small groups) in their own ways, likely somebody will succeed -- and then a larger group can build on that success in a stigmergic way. https://en.wikipedia.org/wiki/Stigmergy

Still, while such an effort to create an alternative may indeed be a trap of wasted effort for no results, what we have now with global use of (anti-)social media -- media designed to addict people and provide 24X7 surveillance and control of information flow via a few centralized services -- is a trap we're already collectively in.

Unfortunately, some people who I might otherwise think should know better -- like Paul Jones, UNC professor in the School of Journalism and Mass Communication and the School of Information and Library Science and of ibiblio fame -- seem to promote the centralized social media technology of this trap. They have given up on email due to spam, information overload, and other issues like asynchronicity -- and replaced email by collection of mostly commercial web services rather than try to make something better inspired by email's good aspects including local archiving and local search. See: "UNC professor Paul Jones went from pioneering to protesting email" http://www.dailytarheel.com/article/2014/02/unc-professor-pa...

While centralized services do have some benefits over current email (including editing and moderating), it is not like the centralized services don't have their own new problems -- whether various forms of spam including low-quality duplicate content driven by monetization, having less time for reflection in a torrent messages with immediate response expected in some services, or even new problems where "unfriending" someone is a lot more complicated and stressful than not including someone on an email CC list.

I've posted a few comments over the years on Paul Jones' #noemail blog ( http://www.ibiblio.org/pjones/blog/category/noemail/ ) agreeing with him about email's problems -- while also pointing out obvious new problems with trusting all your communications to a few centralized services and suggesting working towards better alternatives. But those posts seem to have disappeared in what may have been a purge of most comments from that blog. That is an example of one of the perils of entrusting your messages to someone else for publication and archiving -- and a reason to consider a federated architecture as an alternative.

Still hoping he might someday promote a solution that keeps email's many benefits (including as a personal "electronic memory") while addressing the valid concerns he raises about email.

Anyway, stuff I like to think about and discuss and implement. Even though I may be reaching the point where, like many others before me, "The point of being done is not to finish but to get other things done." See: http://www.manifestoproject.it/bre-pettis-and-kio-stark/

And one might then hope for continued co-evolutionary bootstrapping improvement of concepts, artifacts, services, culture, and practices like Doug Engelbart talked about. And we can see that now, like in using the web right here and now on HN to discuss something better.

So, I can totally get where JMAP-as-it-is becoming a big success would move things forward, and we could then use JMAP to discuss the next generation of changes. So, I'm raising the issue of a broader aspiration -- while accepting it might not be best for JMAP right now.

Sounds like jwz's "intertwingle"[1] strawman with a smidge of "unity of interface"[2], or JetBrains's old Omea information manager[3].

1. https://www-archive.mozilla.org/blue-sky/misc/199805/intertw...

2. https://www-archive.mozilla.org/unity-of-interface.html

3. https://www.jetbrains.com/omea/features/

Thanks for those links, especially the first to Jamie Zawinski's 1998 post on "vast volumes of email" and interwingled data. Yes, that is the sort of concepts and implementation I am talking about (and had hoped perhaps to move Thunderbird towards). Though I might prefer a different UI than JetBrain's Omea -- as well as a FOSS one. In one sketch of some ideas, I was thinking of that all more as a set of plugins: https://github.com/pdfernhout/Twirlip2

Jamie is obviously a brilliant and humane guy -- and his Lisp roots really show in his design of general purpose systems. Looking at Jamie's website, I was hanging around the CMU Robotics Labs around 1986 when he was too so I wonder if we may have ever talked in passing then?

LISP as a programming mindset encourages such generalized designs about information handling, transformation, and linking -- as does inspirations from early AI research (thinking about thinking). For example there is the book Representation and Meaning by Herbert Simon and Laurent Siklossy from 1972 -- as one of many early AI books. I have that one because Professor Simon was nice enough to give me a copy back then during a few conversations we had in his office. Before that, in high school circa 1980 I did an independent study project using Patrick Winston's book on AI. William Kent's 1978 book "Data and Reality" (the first edition -- the third edition was dumbed down by someone else) is another great inspiration. There is a sort of cross over between AI-inspired design and human-centric "augmenting" design.

An interesting comment on the AI/IA crossover by Brad Neuberg (who worked on Open HyperScope): http://codinginparadise.org/ebooks/html/blog/ia_vs__ai.html "I‘ve traditionally been in the IA camp my whole career; I’ve spent the last year going deep into the AI camp. Traditionally these have been opposite worlds but I’m wondering if there might be some powerful, interesting ways to combine them together. One is humanistic and and design focused, the other tends to be more analytical; the space between them is where I’m interested in exploring in the future."

Back around 1988 I applied to be in the NeXT developer program with a concept for an application (embarrassingly called "Orgi" back then, but I was young and reckless/foolish) where the idea was that any data item could hook up with any other data item (using triples). That application never got acted on until I met Steve Jobs when he gave a talk at Princeton and asked him personally about the developer program. And such ideas came out of my earlier interest in triplestores -- ideas that contributed to the roots of WordNet by my undergraduate advisor at Princeton, George A. Miller started just as I was graduating.

In any case, as I said, "I've been beating my head against that wall for years with very limited progress (as have others)". :-)

Jamie was obviously smart enough to connect with such ideas and move on to a happier life selling beer, pizza, and conversational ambiance. Obviously a much smarter guy than I am. :-) And I did not realize his early interest before, so thanks. I knew of another person who became much happier leaving contentious academia after his PhD and being a professor to run a hardware store. See also Matthew B. Crawford's "The Case for Working with Your Hands" written by someone with a PhD who left the non-profit public policy sector to do motorcycle repair: http://www.nytimes.com/2009/05/24/magazine/24labor-t.html

That said, maybe if Jamie had run Mozilla we would not have seen such a tragic waste of billions of non-profit dollars mostly pissed away -- including by mostly abandoning Thunderbird? Or, maybe his voice would have just been ignored? Hard to day. Innovation is such a complex thing. Some ideas I've collected for supporting innovation at such places: https://github.com/pdfernhout/High-Performance-Organizations...

Doug Engelbart and many people he inspired are others near the root of this "intertwingled" data ideas (including "Open HyperScope" as a successor to "Augment" http://hyperscope.org/ ). Ted Nelson and people he inspired with Xanadu. Alan Kay and Dan Ingalls with Smalltalk and its implications as a thinking tool (Dan ran a family hotel for years afterwards). Ward Cunningham with the Wiki server idea and now the Smallest Federated Wiki idea. Mitch Kapor and the Chandler Project. The recent Dat Project by Max Ogden and others also connects with all this somewhat. There is this great book called "Design and Memory" from 1980 by Peter Huyck and Nellie Kremenak exploring such ideas. And of course "As We may Think" by Vannevar Bush. But the roots go further back though annotated scrolls (and the Talmud) and millennia of libraries -- and even back to the first "Standing Bear" cave paintings (see the book, "The Walking People").

This issue of (re)creating a decentralized alternative based around an email-ish feeling of interlinked locally-searchable information is increasingly important because it potentially is a way to move beyond negative aspects of centralized (anti-)social media.

But, hey, my wife and I have more than 5000 books in our house, so all this might just be my hoarding tendencies trying to justify themselves? :-)

Well, off to an unrelated job. And it is unrelated precisely because most of the big paying players in this space (e.g. Google and Facebook) want to get between people and their data -- and typical startups make deals with investors which limit their ability to do FOSS. Still, I can hope there will eventually be, say, an "Automattic" of this "intertwingling" equivalent of WordPress. I actually had hoped Automattic might itself become such a company, but they sadly seem to now see themselves just as a Wix/Squarespace competitor (e.g. adopting Slack for internal communications -- and not even Mattermost or Matrix.org if they want to let someone else own that space and cooperate with another FOSS system).

At least it is nice to think about these things in my spare time -- although sadly at the cost of other experiences. Maybe it is all just a mistake to use my non-job time like that, like Piotr Solnica suggests: https://twitter.com/_solnic_/status/953578620617920513