yes, that was surprising to me too. It would be a big deal if they switched to an encoder-decoder model like the original transformer. But I don't think that's what it's doing. One thing is the causal part, so in the original transformer, the encoder was bidirectional, but in this case it is not, so that's one difference. So I think it's an optimization for the prompt/prefill so that the attention is summarized into the output of the encoding layers, rather than all the layers. I just skimmed the paper too so if anyone else has insight, please correct me.
I think that given the developments in AI in the last ~10 years, it would be hard to say that attention is not part of consciousness. I'm not sure if it is "all you need", but I think that any theory that doesn't include it would be very doubtful.
I think the parent is referring to agents as outward exploration that may come up empty-handed, but I see LLMs and agents as inward exploration, trying to define what is attention, knowledge, intelligence, consciousness, agency, etc. So in this case it is the terra incognita of the human "soul" that LLMs are exploring.
I listened to the audio book version of War and Peace. I think it was something like 25-30 hours. The audio format helped keep the pace going and also it helped with the names. Although for some things, the audio format made it harder to look up in the dictionary, like I kept hearing agitant instead of adjutant, so that part didn't make sense in a lot of the military scenes. I agree with the parent that the book was very engaging, parts even felt like I was watching a movie, e.g. the drunken party tying a bear to a police officer, the foxhunt scene, the duel, the battles like when Petya gets shot, and the burning of Moscow. I even liked the abstract ending when Tolstoy relates human history to calculus so that each individual person has an infinitesimal but real contribution to history.
Corsica to Marseilles has an overnight ferry, at least back in 2000, but I don't think it's has sleeper cabins, more set up like an overnight flight, but a big cruise ship form factor. Instead of cabins they had space for cars. They had the option of getting a ticket without a seat so you just roam around the boat and find an open space, including outside on the decks.
As a resident of MN I'm very proud of what fellow Minnesotans did to stop ICE's violent and illegal detentions here. Unfortunately the non-violent protest and anti-ICE techniques were met with violence from ICE, but the protests themselves were non-violent and well organized.
as were the protests in the 60s. violent uprisings give greater permission for violent suppression. nonviolent protests that are met with violence draw greater scrutiny. the american behemoth rarely turns quickly.
Fall, Or Dodge in Hell (Neil Stephenson) and The Waves (Ken Liu) are two other good stories about brain scanning and transhumanism. The first one is a ridiculously long novel about a future where the cloud is increasingly used for uploading souls of scanned brains, and the second one is a short story where people on a spaceship eventually evolve into noncorporal beings.
Won't fungi and bacteria eat (cellulose-based) the clothes, releasing the same amount of CO₂, only a bit slower?
Synthetic fabrics can likely be buried as a form of carbon sequestration though.
reply