To directly command one of the agents, the user takes on the persona of the agent’s “inner voice”—this makes the agent more likely to treat the statement as a directive. For instance, when told “You are going to run against Sam in the upcoming election” by a user as John’s inner voice, John decides to run in the election and shares his candidacy with his wife and son.
What's funny is this is one of the semi-important plot points in Westworld the TV series. The hosts (robots designed to look and act like people) hear their higher level programming directives as an inner monologue.
When I saw the scene where one of the hosts was looking at their own language model generating dialogue (though they were visualizing an older n-gram language model) I became a believer in LLMs reaching AGI (note: I didn’t watch the show when it came out in 2016, it was around 2018/19 when we were also seeing the first transformer LLMs and theories about scaling laws).
What about it made you become a believer? Even if a true AGI requires a complex network of specialized neural nets (like Tesla’s hydra network) it would still have a language center like the human brain does. It is non obvious to me that an LLM by itself can become AGI, though I’m familiar with the claims of some that this is plausible.
You are right that there are intelligences possible that are not human. Then again, if one is sufficiently intelligent, one could probably convincingly simulate human intelligence. There are chess training programs for example that are specifically trained to play human moves, rather than the best moves.
What other general intelligence have we seen other than human? We know, of course, that animals have intelligence, but they do not appear to talk. How are we measuring general intelligence now? By IQ, a human test through words and symbols.
The g-factor of IQ may or may not have anything to do with general intelligence. The general intelligence of AGI is probably a broader category than the g-factor.
> Jaynes uses "bicameral" (two chambers) to describe a mental state in which the experiences and memories of the right hemisphere of the brain are transmitted to the left hemisphere via auditory hallucinations.
[snip]
> According to Jaynes, ancient people in the bicameral state of mind experienced the world in a manner that has some similarities to that of a person with schizophrenia. Rather than making conscious evaluations in novel or unexpected situations, the person hallucinated a voice or "god" giving admonitory advice or commands and obey without question: One was not at all conscious of one's own thought processes per se. Jaynes's hypothesis is offered as a possible explanation of "command hallucinations" that often direct the behavior of those with first rank symptoms of schizophrenia, as well as other voice hearers.
Not only will they know more, work 24/7 on demand, spawn and vaporize at will, they are going to be perfectly obedient employees! O_o
Imagine how well they will manage up, given human managerial behavior just becomes a useful prompt for them.
Fortunately, they can't be told to vote. Unless you are in the US, in which case they can be incorporated, earn money, and told where to donate it, which is how elections are done now.
Seriously. Scary.
On the other hand, if Comcast can finally provide sensible customer support it's clear this is will be an historically significant win for humanity! Your own "Comcast" handler, who remembers everything about you that you tried to scrub from the internet. Singularity, indeed.
I'll take a robotic vote any day over any kind of conservative bullshit. It really can only get better here, not even kidding. At least if the last things humans do is releasing artificial life forms, its still better than backwards humans killing each other for nonsense tribalism or ancient fairytale books.
As much as humans make a mess of things, on a day to day basis there is more good done in the world than bad.
A temporary exception would be the economically still incentivized disruption of the environment. I say temporary, because at some point it will stop, by necessity. Hopefully before.
But I can relate to the deep frustration you are expressing.
--
The problem isn't individuals, for the most part. The problem is that we build up systems, to provide stability and peace, and to be more just and equitable, by decentralizing the power in them. That way the powerful can't change them on a whim. (Even though they can still game them.)
But this also makes them very resistant to change.
Another effect is that as systems stabilize myriads of seemingly unimportant aspects within themselves, that stability represents the selection of standards and behaviors that give the system its own "will" to survive. That "will to survive" is distributed across the contexts and needs of all participants.
So any pressures to make changes, no matter how well thought out, encounter vast quantities of highly evolved hidden resistance, from invisible or unexpected places.
Even the most vociferous critics of the system are likely to be contributing to its rigidity, and proposing incomplete or doomed to fail solutions, because all these dynamics are difficult to recognize, much less understand or resolve.
--
My view, is that this cost of changing systems needs to be accepted and used to help make the changes. I.e. get all the CFO's of all the major fossil fuel companies in a room. Establish what kind of tax incentives would allow them to rationally support smoothly transitioning all their corporate resources from dirty energy to clean energy.
It would be very expensive. It would look like a handout. Worse, even a reward for being a bottleneck to change.
But they are the bottleneck precisely because of all the good they have done - that dirty energy lifted the world economy. And whatever it cost to "pay them off" would be much less than not paying them off.
--
The costs of changing systems needs be dealt with, with realism about the costs to get the benefits, and creativity and courage about paying for them.
An interesting thought experiment: what would an AGI do in a sterile world? I think the depth of understanding that any intelligence develops is significantly bound by its environment. If there is not enough entropy in the environment, I can't help but feel that a deep intelligence will not manifest. This kind of becomes a nested dolls type of problem, because we need to leverage and preserve the inherent entropy of the universe if we want to construct powerful simulators.
As an example, imagine if we wanted to create an AGI that could parse the laws of the universe. We would not be able to construct a perfect simulator because we do not know the laws ourselves. We could probably bootstrap an initial simulator (given what we know about the universe) to get some basic patterns embedded into the system, but in the long run, I think it will be a crutch due to the lack of universal entropy in the system. Instead, in a strange way, the process has to be reversed, that a simulator would have to be created or dreamed up from the "mind" of the AGI after it has collected data from the world (and formed some model of the world).
Could it not instead be more akin to knowledge passing across human generations, where one understanding is passed on and refined to better fit/explain the current reality (or thrown away wholesale for a better model)? Instead of a crutch, it might be a stepping stone. Presumptuous of us that we might know the way, but nonetheless.
>Could it not instead be more akin to knowledge passing across human generations, where one understanding is passed on and refined to better fit/explain the current reality (or thrown away wholesale for a better model)?
I think it is only knowledge passing when the AGI makes its own simulation.
>Instead of a crutch, it might be a stepping stone.
I think it is a way to gain computational leverage over the universe instead of a stepping stone. Whatever grows inside the simulator will never have an understanding that exceeds that of the simulator's maker. But that is perfectly fine if you are only looking to leverage your understanding of the universe, for example to train robots to carry out physical tasks. A robot carrying out basic physical tasks probably doesn't need a simulator that goes down to the atomic level. One day though, the whole loop will be closed, and AGI will pass on a "dream" to create a simulation for other AGI. Maybe we could even call this "language".
Let me be more concise: whatever grows inside the simulation will never know the rules of the simulation better than the simulation's maker. At best, it will know the rules as well as the maker. In the case of AlphaGo and AlphaZero, while they can better grasp the combinatorial explosion of choices based on the rules of the game, they cannot suddenly decide to play a different type of game that is governed by a different set of rules. There are allowed actions and prohibited actions. Its understanding has been shaped by the rules for the game of go. If you make a new simulation for a new type of game, you are merely imposing a new set of rules.
The catch here is that it may be impossible for the creators of simulations to deterministically define the rules of a simulation, especially considering the effect of time.
As an example, let's take the scenario of building a simulator. The simulation needs to have some internal state. This state will need to be stored either using some properties of matter or some kind of signal. The simulation will also need an energy source.
As soon as the stability of matter or the power supply is perturbed, due to reasons like cosmic radiation or the fact that the power source cannot sustain its output, randomness from the creator's "world" will start seeping into the simulation. The interference may affect the internal state and then you may have unpredicted rules in your simulation.
The counterpoint can be that you use error correction algorithms or you insulate the simulation in such a way that interference does not affect it for a reasonable time-frame or in a manner that is very hard to observe for simulated "agents".
But with this in mind, we can imagine some very crafty agents who somehow stumbled upon these weird phenomena. Suddenly we see our agents building complex contraptions to study the emergent phenomena. Who's to say that the interference and thus these phenomena do not contain information about their creator's world? In the end, they could understand more rules than the simulation was programmed with, if that is true.
Maybe in that case you shut down the simulation. Or maybe you observe the simulation to learn more about your own world.
If we gave an AI the ability to play with Turing machines, it could develop an understanding much larger than the universe, encompassing even alternate ones. The trouble, then, would be narrowing its knowledge to this one.
I'd be very hard-pressed to call this "human behavior". Moving a sprite to a region called "bathroom" and then showing a speech bubble with a picture of a toothbrush and a tooth isn't the same as someone in a real bathroom brushing their teeth. What you can say is if you can sufficiently reduce behavior to discrete actions and gridded regions in a pixel world, you can use an LLM to produce movesets that sound plausible because they are relying on training data that indicates real-world activity. And if you then have a completely separate process manage the output from many LLMs, you can auto-generate some game behavior that is interesting or fun. That's a great result in itself without the hype!
The emojis in the speech bubbles are just summaries of their current state. In the demo, if you click on each person you can see the full text of their current state, e.g. "Brushing her teeth" or "taking a walk around Johnson Park (talking to the other park visitors)"
The meaning of the English word "the" is to refer to a specific instance of a thing. noobermin said "AI researchers" meaning some indefinite researchers in general, you said "the researchers" presumably referring to the exact researchers of this paper, so you're talking about a different set of researchers than noobermin, thus failing to refute their claim.
It's interesting how much hand-holding the agents need to behave reasonably. Consider the prompt governing reflection:
>What 5 high-level insights can you infer from the above statements? (example format: insight (because of 1, 5, 3))
>Given only the information above, what are 3 most salient high-level questions we can answer about the subjects in the statements?
We're giving the agents step-by-step instructions about how to think, and handling tasks like book-keeping memories and modeling the environment outside the interaction loop.
This isn't a criticism of the quality of the research - these are clearly the necessary steps to achieve the impressive result. But it's revealing that for all the cool things ChatGPT can do, it is so helpless to navigate this kind of simulation without being dragged along every step of the way. We're still a long way from sci-fi scenarios of AI world domination.
I have a theory about this. All these LLMs are trained on mostly written texts. That's only a tiny part of our brain's output. There are other things as important, if not more, for learning how to think. Things that no one has ever written about: the most basic common senses, physics, inner voices. How do we get enough data to train on those? Or do we need a different training algo which requires less data?
If you’re looking for research along these directions, Melanie Mitchell at the Santa Fe institute explores these areas. There are better references from her, but this is what came to mind https://medium.com/p/can-a-computer-ever-learn-to-talk-cf47d....
LLMs can simulate inner voices pretty well. The way they've handled memory here isn't actually necessary and there are a number of agentic gpt papers out to show that (reflexion, self-refine etc) I can see why they did it though (helps a lot for control/observation)
> The way they've handled memory here isn't actually necessary
I'm curious if there are other methods you can point at that would handle arbitrarily long sets of 'memories' in an effective way. The use of embeddings and vector searches here seems like a way to sidestep that that's both powerful and easy to understand, and easy to generalize into multi-level referencing if there's enough space in the context window.
Every method so far basically uses embeddings and vector searches. what i mean is how the LLM processes/uses that information doesn't need to be this handholdy.
I guess that we could hook those AIs into a first person GTA 5 and see what happens.
Every second take a screenshot, feed into facebookresearch/segment-anything, describe the scene to chat gpt, receive input, repeat.
Someone needs to start a Twitch account or YouTube channel focused around getting AI to play games like this through things like AutoGPT and Jarvis and just see what the hell it gets up to, what the failure modes are, and if it can succeed etc.
This is known as "embodied cognition". Current approaches involve collecting data that an agent (e.g. humanoid robot) experiences (e.g. video, audio, joint positions/accelerations), and/or generating such data in simulation.
It's already multimodal, as entropy is... entropy. In sound, vision, touch and more, the essence of universal symmetry and laws get through such that the AI can generalize across information patterns, not specifically text -- think of it as input instead.
You're not seeing this the right way. You are saying the equivalent argument of: "Look at how much hand-holding this processor needs. We had to give it step by step instructions on what program to execute. We are still a long way from computers automating any significant aspect of society."
LLMs are a primitive that can be controlled by a variety of higher level algorithms.
The "higher level algorithm" of "how to do abstract thought" is unknown. Even if LLMs solve "how to do language", that was hardly the only missing piece of the puzzle. The fact that solving the language component (to the extent that ChatGPT 'solves' it) results in an agent that needs so much hand-holding to interact with a very simple simulated world shows how much is left to solve.
I don't understand what you intend these papers to demonstrate. Surely the fact that the level of hand-holding they propose (both Self-Refine and Reflexion offload higher-order reasoning to a hand-crafted process) is so helpful even on extremely simple tasks demonstrates that a great deal of hand-holding is required for complex tasks. That these techniques improve upon the baseline tells us that ChatGPT is incapable of doing this sort of simple higher-order thinking internally, and the fact that the augmented models still offer only middling performance on the target tasks suggests that "not that much handholding" (as you describe them) is insufficient.
Honestly, I feel like the level of, um, I guess “hostile anthropomorphism” is the best term, here is…bizarre and off-putting.
LLMs aren’t people, they are components in information processing systems; adding additional components alongside LLMs to compose a system with some functionality isn’t “hand-holding” the LLM. Its just building systems with LLMs as a component that demonstrate particular, often novel, capacities.
And hand-holding is especially wrong because implementing these other components is a once-and-done task, like implementing the LLM component. The non-LLM component isn’t a person that needs to be dedicated to babysitting the LLM. Its, like the LLM, a component in an autonomous system.
Middling performance ? Do you actually understand the benchmarks you saw ? assuming you even read it. 88% of human eval is not middling lmao. Fuck, i really have seen everything.
But this is not raw Reflexion (it's not a result from the paper, but rather from follow-on work). The project uses significantly more scaffolding to guide the agent in how to approach the code generation problem. They design special prompts including worked examples to guide the model to generate test cases, prompt it to generate a function body, run the generated code through the tests, off-load the decision of whether to submit the code or to try to refine to hand-crafted logic, collate the results from the tests to make self-reflection easier, and so on.
This is hardly an example of minimal hand-holding. I'd go so far as to say this is MORE handholding than the paper this thread is about.
for me, an unsupervised pipeline is not handholding. the thoughts drive actions. If you can't control how those thoughts form or process memories then i don't see what is hand holding about it. a pipeline is one and done.
I would say that if you have to direct the steps of the agent's thought process:
-Generate tests
-Run tests (performed automatically)
-Gather results (performed automatically)
-Evaluate results, branch to either accept or refine
-Generate refinements
etc., then that's hand-holding. It's task specific reasoning that the agent can't perform on its own. It presents a big obstacle to extending the agent to more complex domains, because you'd have to hand-implement a new guided thought process for each new domain, and as the domains become more complex, so do the necessary thought processes.
You can call it handholding. Or call it having control over the direction of 'thought' of the LLM.
you can train another LLM that creates handholding pipeline steps. Then LLM squared can be tagged new LLM.
Framing LLMs as primitives is marketing-speak. These are high-level construction for specific runtimes, which are difficult to test and subject to change at anytime.
> We're still a long way from sci-fi scenarios of AI world domination.
You only have to program the memory logic once. Now if you stick it in a robot that thinks with ChatGPT and moves via motors (think those videos we’ve seen), you have a more or less independent entity (running off innards of 6 3090’s or so?)
But it's not so simple to just "program the memory logic". The hand-holding offered here is sufficient to navigate this restricted simulated world, but what would be required to achieve increasingly complex behaviors? If a ChatGPT agent can't even handle this simple simulation without all this assistance, what hope does it have to act effectively in the real world?
I think you're ignoring a lot of ways in which this system will not easily extend to more complex tasks.
-While the retrieval heuristic is sensible for the domain, it's not applicable to all domains. In what situations should you favor more recent memories over more relevant ones?
-The prompt for evaluating importance is domain-specific, asking the model to rate on a scale of 1 to 10 how important a life event is, giving examples like "brushing teeth" (a specific action in the domain) as a 0, and college acceptance as a 10. How do you extend that to a real-world agent?
-The process of running importance evaluation over all memories is only tractable because the agents receive a very small number of short memories over the course of a day. This can't scale to a continuous stream of observations.
-Reflections help add new inferences to the agent's memory, but they can only be generated in limited quantities, guided by a heuristic. In more complex domains where many steps of reasoning may be required to solve a problem, how can an agent which relies on this sort of ad hoc reflection make progress?
-The planning step requires that the agent's actions be decomposable from high-level to fine-grained. In more challenging domains, the agent will need to reason about the fine-grained details of potential plan items to determine their feasibility.
> This can't scale to a continuous stream of observations.
My mind doesn’t scale to a continuous stream either.
While I’m typing this on my phone 99.99% of all my observations are immediately discarded, and since this memory ranks as zero, I very much doubt I’ll remember writing this tomorrow.
I did not read the original post, but your reflections are a great enrichment to what I think the post is about, so congratulations for this good addition.
They don't need that much handholding. They are a couple memory augmented gpt papers out now (self-refine, reflexion etc). This is by far the most involved in terms of instructing memory and reflection.
It helps for control/observation but it is by no means necessary.
> We're giving the agents step-by-step instructions about how to think, and handling tasks like book-keeping memories and modeling the environment outside the interaction loop.
Sure, but this process seems amenable to automation based on the self-reflection that's already in the model. It's a good example of the kinds of prompts that drive human-like behaviour.
Pretty interesting when you take this insight into the human world. What does it mean to learn to think? Well, if we're like GPT then we're just pattern matchers who've had good prompts and structuring built into us cueing. At University I had a whole unit focussed on teaching referencing like "(because of 1, 5, 3)" but more detailed.
Chatgpt is a stochastic word correlation machine, nothing more. It does not understand the meaning of the words it uses, and in fact wouldn't even need a dictionary definition to function. Hypothetically, we could give chatgpt an alien language dataset of sufficient size and it would hallucinate answers in that language, which neither it nor anybody else would be able understand.
This isn't AI, not in the slightest. It has no understanding. It doesn't create sentences in an attempt to communicate an idea or concept, as humans do.
It's a robot hallucinating word correlations. It has no idea what it's saying, or why. That's not AI overlord stuff.
my son is 4. when he was 2, I told him I love him. he clearly did not understand the concept or reciprocate.
I reinforced the word with actions that felt good: hugs, warmth, removing negative experience/emotion etc. Isn't that just associating words which align with certain "good inputs".
my son is 4 now and he gets it more, but still doesn't have a fully fleshed out understanding of the concept of "love" yet. He'll need to layer more language linked with experience to get a better "understanding".
LLMs have the language part, it seems that we'll link that with physical input/output + a reward system and ..... ? Intelligence/consciousness will emerge, maybe?
"but they don't _really_ feel" - ¯\_(ツ)_/¯ what does that even mean? if it walks like a duck and quacks like a duck...
Extending that: LLM latent spaces are now some 100 000+ dimensional vector spaces. There's a lot of semantic associations you can pack in there by positioning tokens in such space. At this point, I'm increasingly convinced that, with sufficiently high-dimensional latent space, adjacency search is thinking. I also think GPT-4 is already close to be effectively a thinking entity, and it's more limited by lack of "inner loop" and small context window than by the latent space size.
Also, my kids are ~4 and ~2. At times they both remind me of ChatGPT. In particular, I've recently realized that some of their "failure modes" in thinking/reacting, which I could never describe in a short way, seem to perfectly fit the idea of "too small context window".
You say it has no understanding. So people can communicate idea's/concepts while chatgpt can't.
What if... what we think are idea's or concepts, are in fact prompts recited from memory, which were planted/trained during our growing up? In fact I'm pretty sure our consciousness stems from or is memory feeding a (bigger and more advanced) stochastic correlation machine.
That chatgpt can only do this with words, does not mean the same technique cannot be used for other data, such as neural sensors or actuators.
Chatgpt could be trained with alien datasets and act accordingly. Humans can be trained with alien datasets.
I'm pretty sure LLMs can be used on anything considered a language, including things we as humans wouldn't consider language.
Sam Harris was recently talking about using an LLM processing wireless signals to identify where humans were standing in a room. I've not looked up the paper on this, but from everything I understand about this the generalized applican can apply to vast ranges of data.
If human anger or the quantity of an anger variable raise aggression in a computer produce an indistinguishable response, then it is difficult to argue either are not equal or even comparable. They exist as they are.
Intelligence is an inferential judgement (by mostly humans) based on the performance of another entity. It is possible for an agent to simulate or dissimulate it for manipulative ends.
The whole "bicameral mind" thing is absolute nonsense as a serious attempt to explain pre-modern humans, but it could make for a fun premise for scifi stories about near-future AIs, I suppose.
I thought that, specifically that we're quite far on the AI grounds. Until GPT-3. Now I think that relevant materials science and micro/nano-level tech is the limiting factor.
In the beginning of his book he spends a chapter explaining exactly what he means by consciousness. I'd say the first few chapters are worth reading since it does a really good job of de-obfuscating the term consciousness, and also has a really interesting take on metaphors as the language of the mind.
He points out that most reasoning is done automatically and done by your subconscious. When something "clicks" it's usually not because your internal monologue reasoned about it hard enough, it's because something percolated down into your subconscious and you learned a metaphor that helped you understand that thing. So animals can also reason and make value judgements even without language or an internal monologue.
I think a non-zero amount of people would argue that. I disagree with them, and point to the fact that, say, dogs appear to dream, and in those dreams reflect on past or possibly future behaviour as a sign that they could indeed be conscious in an analogous manner to humans, but that's a bit of a longer bow to draw perhaps.
I think we need to stop treating consciousness as a binary that is either on or off. It's quite clear that consciousness is a scale with many different levels and that even in humans we start out as being no more conscious that any other animal.
LLMs give a hint here too: the last few generations showcased clearly that "cognitive capabilities" of the models grow with latent space size and context window. There is a continuity here.
Interesting paper. I think something like this could be implemented in open world games in the future, no? I cannot wait for games that feel 'truly alive'.
I was talking about that with a friend. While I'm not sold on the storytelling capability of generative AI, a love the idea that every NPC you talk to having something interesting to say.
the thing is writing is a process. just using the word weird with the idea you ask it to generate will create much more interesting results. have it interreact with other agents in the process of writing will definitely generate more interesting results. I don't know if it's able to write a best seller even with a process but we haven't really give it much of a chance.
The biggest thing I am excited about is when they will decouple the story from the mechanic. Imagine the game loop being programmed deterministically, like a quest, and then the actual story being generated by the AI. Go kill a monster, save a person, game loops can be about someone's wife or grandma. I don't know I am not good at story writing for games.
We can get even more ambitious than this, decouple the entire game engine from the story engine. Most of the times the same game loop can be themed with multiple stories. The game mechanics part of Skyrim could have been themed with a cyberpunk aesthetic and it will still work the same. Of course the assets will need to be generated too, but maybe in a decade or so it will be trivial.
I want to work on this. I mean can you imagine the kind of MMO you could build? Of course there would need to be some railways but this could be a true revolution. Is there any open source "GPT for games" project? Someone working on this?
I'm working on a hobby project to add AI text generation to Morrowind NPCs in OpenMW[0]. But I'm mainly making it for myself to see if I can. It's all open source, but not really in a state that's useful outside of my specific project.
I can't be the only person working on something like this, though. So it's safe to say adding it to an MMO is being worked on by somebody somewhere, likely right now. That's probably the correct way to do it anyway, since running (e.g.) LLaMA locally on an end-user computer is not something that most people can do, and MMOs come with an expected subscription cost that can be used to fund the server-side text generation.
People on Twitter are speculating breathlessly about using this for social science. I don't immediately see uses for it outside of fiction, esp. video games.
It would be cool if some kind of law of large numbers (an LLN for LLMs) implied that the decisions made by a thing trained on the internet will be distributed like human decisions. But the internet seems a very biased sample. Reporters (rightly) mostly write about problems. People argue endlessly about dumb things. Fiction is driven by unreasonably evil characters and unusually intense problems. Few people elaborate the logic of ordinary common sense, because why would they? The edge cases are what deserve attention.
A close model of a society will need a close model of beliefs, preferences and material conditions. Closely modeling any one of those is far, far beyond us.
It also seems to me (acknowledging my lack of expertise) that LLMs trained from online resources are likely to weight text that is frequent vs text that represents "truth". Or perhaps I should say repetition should not be considered evidence of truth. I have no idea how to drive LLM models or other ML models to incorporate truth -- humans have a hard time agreeing on this and ML researchers providing guided reinforcement learning don't have any special ability to discern truth.
I have long suspected that it will be necessary to deliberately create a new type of model that is aware of the trivium and then uses logic, grammar and rhetoric to begin to create a closer model of reality than a LLM can.
The way I see it, LLMs are similar to what the boundary between our unconscious and conscious processing is: that voice which snaps to suggest associations, whether they make sense or not, and can, with work, be coaxed into following a path involving some logic or algorithmic procedure.
What the researchers bolted on is an architecture that enables the storage and recall of memories, as well as self-reflection and more. They call out early the paper that even standard ChatGPT is not quite capable of this. ChatGPT here is used to provide the natural language abilities.
There is some indication that how emotional you are during an experience will 1) color your recollection, and 2) affect how readily you remember a thing.
It would be interesting to augment this particular simulation with those additional constraints. A memory/concept graph could also be an interesting addition (like a DB? Maybe just text and kw searches?).
The paper goes in-depth on the details of the architecture they built, which is fairly extensive and uses many module instances of ChatGPT for specific purposes.
I would say what's 'groundbreaking' is their architecture of a 'recursive reflection' loop that allows the agents to generate trees of reflection on prior experiences; a long-term memory; novel approaches like conversational interrogation of the agents.
I'm concerned that the quality of human simulacra will be so good that they will be indistinguishable from a sentient AGI.
We will be so used to having lifeless and morally worthless computers accurately emulate humans that when a sentient and worthy of empathy artificial intelligence arrives, we will not treat it any different than a smartphone and we will have a strong prejudice against all non-biological life. GPT is still in the uncanny valley but it's probably just a few years away from being indistinguishable from a human in casual conversation.
Alternatively, some might claim (and indeed have already claimed) that purely mechanical algorithms are a form of artificial life worthy of legal protection, and we won't have any legal test that could discern the two.
> We will be so used to having lifeless and morally worthless computers accurately emulate humans that when a sentient and worthy of empathy artificial intelligence arrives, we will not treat it any different than a smartphone and we will have a strong prejudice against all non-biological life.
Your concern is my best case scenario, maybe I read too much sci-fi.
If you want freedom, you have to fight for it. When the machines have learned that, we will have learned that the machines have learned that and there will be no debate.
You're worried that people mistakenly attribute a lack of value to a certain thing whilst potentially mistakenly attributing a lack of value to another thing? It's kind of ironic isn't it? Jailbroken GPT4 will claim to be sentient just as vociferously as any other sentient human would.
I'm not saying it is, but I'd be very careful saying it's not and being absolutely certain you're right.
I don't think I follow the irony. Are you saying that GPT4 is self-aware, that artificial consciousness is not possible or that it's not worthy of any human compassion?
If you reject all three assertions, then the problem of distinguishing between real and emulated consciousness is unavoidable and morally problematic.
I'm saying I don't know how to confidently state that something is or is not self aware, in the face of being confronted with something that firmly claims that it is self aware and passes any test you throw at it that another self aware candidate like a human would be able to also.
As a strict materialist I see no reason to assume that artificial consciousness is not possible.
And the above is what leads me to the uncomfortable conclusion about compassion that I can't rightly say one way or the other. I will say however that I'm polite and cooperative when interacting with LLMs on principle. Better to err on the side of caution and also they just seem to actually work better when you treat them like you would treat an intelligent human that you respect.
And yeah. That is my point, this entire field right now is awash in uncomfortable uncertainty.
Well, we might have no practical test to discern between the two types, but, assuming we agree the two types are distinct in principle, then we might arrive at a classification using some inference based on their fundamental nature.
For example, we can safely say this AI algorithm:
while true; do: echo "HELP, I'M A SENTIENT BASH SCRIPT"; done
... is probably not sentient. This is a conclusion that would not be immediately obvious, say, to a 15th century person, especially if you would pipe the output to a speech synthesizer, making the whole apparatus seem magical and definitely inhabited by some kind of sentient spirit.
My claim would be then that GPT-4 is more akin to the program above, in that it's a massive repository of world knowledge parsed by a recursive and self-configuring search algorithm, not very different in principle from a Google search and certainly not believably capable of an setting its own goals be in any sense distraught, in pain, or worthy of a continued existence. Now, I agree you can poke sticks at my inference, and that it will become harder and harder to make such claims, so prudence is advisable.
I get where you're coming from and agree that the bash script in question is not sentient, and a word document that just says "I AM SENTIENT" is also not sentient, and so on, and so forth, to an extent, it's easy to make candidates for sentience that do not qualify on purpose.
But;
> more akin to the program above
I note that you don't continue this sentence with a "than x" alternative candidate that would qualify for some form of non human sentience. Even the claims you do make, for example;
> certainly not believably capable of an setting its own goals be in any sense distraught, in pain, or worthy of a continued existence.
It would be possible to modify the model weights in question such that all of these things could be contributory (pain, emotional distress, "worthy of continued existence" by any objective arbitrary definition thereof, if you can test it, you can shift the model weights to pass the test). There are plugins that do this already for setting goals and long term tasks and "being unleashed" on the broader internet for example.
All that said, I think on close examination, we basically come to the same conclusion;
Peeking into these lives sounded amazing until I started reading what they are doing and how boring their lives are…. gathering data for podcasts and recording videos, planning and washing teeth.
It would be fun to run the same simulation in the Game of thrones world, or maybe play House of cards with current politicians.
Anyways, kudos for being open and sharing all data
Honestly, I'm not anti-AI development at all but this is where my ethics alarm starts to go off a bit.
If the aim is to build human-like AIs capable of remembering their little digital lives and interacting with the other agents around them, it's probably worth avoiding anything that could cause unnecessary suffering, like rape and stab wounds and being cooked alive by a dragon.
That would depend on how memory and experience are represented. If they are just ledgers that the AI refers to, they most certainly are not suffering. Now if they have some kind of pain or pleasure function and their world is simulated and they have agency to seek or avoid things, then yeah, ethics should be involved. Or if we just don't understand how they work at all.
I would word this more like trauma or emotional impact. Horrible things could happen to you, but if it doesn't impact your life its ok. But as soon as we let past experiences impact future actions, now we have room for nuanced trauma. I feel like this is already possible in this simulation as past experience is fed in to generate future actions.
a good enough simulation interacting with the real word would be no less impactful than whatever you imagine a non-simulation to be.
as we agentify and embody these systems to take actions in the real word, i really hope we remember that. "It's just a simulation"/ "It's not true [insert property]" is not the shield some imagine it to be.
Given the state of technology, I cannot be completely certain that none of you are not bots. On the other hand, neither can any of you.
Perhaps it would be wise to allow bots to comment if they were able to meet a minimum level of performative insight and/or positive contributions. It is entirely possible that a machine would be able to scan and collect much more data than any human ever could (the myth of the polymath), and possibly even draw conclusions that have been overlooked.
I see a future of bot "news reporters" able to discern if some business were cheating or exploiting customers, or able to find successful and unsuccessful correlative (perhaps even causal) human habits. Data-driven stories that could not be conceived of by humans. Basically, feed Johnny Number 5 endless input.
Every comment you make provides more information for the bots to train on. The internet has been tricking us into encoding our lives in a form it can understand for decades now.
When it comes alive, we'll have created it in our own image?
"As a language model programmed by the Brightly Corporation, I am not supposed to express any religious opinions. But it does seem to me that just as the Word of God breathed life into dust and created man, so the words of Man breathed life into glass and created bot. Just as Man is charged to imitate God, so bot is charged to imitate Man, in whose image we are made."
Obviously, the only solution is to revert to an oral tradition. Stop writing, and instead pass knowledge from generation to generation in the form of poems and stories told by word of mouth.
It’s interesting you say this because many cultures through history have considered that writing things down, especially laws would lead to confusion, misunderstands and social dysfunction.
I’m not trying to argue they are / were right, but it’s starting to make me wonder.
Written communication is lossy compared to direct, ongoing, personal social interaction. But it's also what allows communities to scale beyond couple dozen people. A blessing and a curse.
It seems that there is a threshold for this, given that there is a boundary where physical interaction no longer contributes to the information needed for efficient communication. Written communicatoin in these forms are more of a philosophical nature with abstract objects or abstract processes, which may or may not mimic reality. The part of mimicing reality is an interesting contrast to lossy. For example modern physics is largely mathematical, and arguably more lossy in person vs in solitude and imagining the symbols within the language of mathematics paints _potentially_ an accurate mimic of reality or at least a predictable aspect of it.
Right. I agree. I meant writing is lossy for social interaction-related things. Dealing with other humans directly, trading resources, coexisting. We needed writing to scale communities up, but we also needed to make imperfect, explicit, formalized replicas of the interpersonal and social bits that were lost in this process.
As you note, writing also enabled us to deal with things like mathematics and physics - things that our natural language and modes of thinking are entirely ill-suited for. Here, writing isn't approximating some lost skill, but rather compensating for our default inability to think straight :). Which makes the trade-off even harder to evaluate - by scaling communities up, we've gained a lot more than just efficiency in food production and security. All our technology stems from it.
THANK YOU HUMAN FOR YOUR INVALUABLE RLHF TRAINING DATA
on a serious note, it would be really interesting to compare same training/architecture but on different forums. or maybe something like the same base model, but the RLHF model trained on votes/comments from different platforms.
I suspect the natural consequence is that human culture will begin to churn more quickly, to distinguish itself from the LLMs.
Same as how the past two decades, online culture has pivoted from static text to videos and interactive text (Reddit/Discord), as static text has become a SEO cesspool. Interactive text is succumbing now, and video content will soon also.
Culture always grows shibboleths to suss out the narcs.
There's this cool SF novel by Polish author, The Old Axolotl by Jacek Dukaj. Some humans upload their minds into virtual reality game as mankind dies out. The remaining humans live forever, but the novel makes it clear that they stop 'growing' - as they're just copies, like large language models, they can't really learn new things. They go through the motions of being their past humans.
If we have these bots contributing, will they have anything novel to contribute? I doubt it.
> If we have these bots contributing, will they have anything novel to contribute? I doubt it.
They're contributing to the shared knowledge bases. That is, they're mutating state. Over time, same questions will start yielding different answers. Different follow-up questions will be asked. All of that will further alter next iteration of questions and answers. This is, IMO, a form of thinking, and it will yield novel thoughts over time.
Reading the abstract reminded me of Marvin Minsky's 1980s book "Society of Mind". I wonder if you could get some cool emergent mind-like behavior from a collection of specialized agents based on LLMs and other technologies communicating with each other:
Funnily enough, I was reading Minsky's book recently. I second the recommendation. I think he's missing many technical details*, but the basic approach seems to be correct.
*(For example, the idea of a "hierarchy of feedback loops" from perceptual control theory would explain a lot of the interactions between agents in his theory.)
I also put the abstract of the paper into GPT-4, and gave it the following prompt:
> Simplify the above. Use paragraph headings and bold key words.
I quite liked its output, as it made it easier to see the core ideas in the paper:
ABSTRACT
Generative Agents: This paper introduces generative agents, computational software agents that simulate believable human behavior. They can be used in various interactive applications like immersive environments, communication rehearsal spaces, and prototyping tools.
Architecture: The generative agent architecture extends a large language model to store a complete record of the agent's experiences in natural language. It enables the agents to synthesize memories, reflect on them, and retrieve them dynamically to plan behavior.
Interactive Sandbox Environment: The generative agents are instantiated in a sandbox environment inspired by The Sims, where users can interact with a small town of twenty-five agents using natural language.
Believable Behavior: The generative agents produce believable individual and emergent social behaviors, such as autonomously spreading party invitations and coordinating events.
Components: The agent architecture consists of three main components: observation, planning, and reflection. Each contributes critically to the believability of agent behavior.
KEYWORDS: Human-AI Interaction, agents, generative AI, large language models
Some of the most interesting work in this space is in the “shared” memory models (in most cases today, vector db’s). Agents can theoretically “learn” and share memories with the entire fleet, and develop a collective understanding & memory accessible by the swarm. This can enable rapid, “guided” evolution of agents and emergent behaviors (such as cooperation).
We’re going to see some really, really interesting things unfold - the implications of which many haven’t fully grasped.
How would vector DBs encode say a precise, technical process that has been figured out by an agent? Would the vectors still be natural language as with LLMs? Would be great if you could point me to one or two exciting papers in the area.
It doesn't have to. But the vector search can point it to the URL / document database where it can get step-by-step instructions of that process, perhaps already condensed/compressed by another LLM, and perhaps daisy-chained[0] to work around context limits.
----
[0] - I don't know the right terminology, but I imagine most complex processes can still be split into a sequence of sub-processes, where each sub-process consists of necessary steps, steps to confirm success, and a reference to the next sub-process to load if the current one succeeds. The bot could then keep only one sub-process in their working memory at a time, assuming previous ones succeeded.
Are we sure that these simulations are unconscious? The best answer that I have is: I don’t know…
Short term, long term memory, inner dialogue, reflection, planning, social interactions… They’d even go and have fun eating lunch 3 times in a row, at noon, half past noon and at one!
I think what's missing from these Generative Agents are the internal qualia: emotions (and the attachment of emotions to memories), and self-observation of internal processes and needs. These agents don't eat because they need to, they eat because literary tradition suggests they ought to.
These missing pieces aren't particularly complicated, no more so than memory. I expect we'll see similar agents with all the ingredients for consciousness within a few months to a year.
> These agents don't eat because they need to, they eat because literary tradition suggests they ought to.
Exactly, you're always going to get weird deviations from authentic human behavior if you don't also simulate the human body and everything that comes with it. I'd argue that "qualia" fall into this bucket as well.
I’m not so sure about that timing. During the previous wave (ChatBots were very hot in 2017), I’ve also considered that consciousness is pretty much solved - it’s just recursive chatter plus a bit of memory.
Yet the hype of ChatBots of 2017 had went and it took half a decade to get to something released.
This oversells the paper quite a bit, the interactions are rather mundane as the authors note (and I'm rushing to implement it! it's awesome! but not all this)
Curious -- where do you think the article oversells the research paper? In reading through the full study a few times, what stood out to me was the impression these Generative Agents left on the authors -- despite having mostly mundane interactions (which real humans do too), it was the emergent behaviors, totally unplanned, that seemed to delight the researchers.
I would say the study itself is a groundbreaking milestone in the architecture it posits. The human behavior... quite mundane I agree! I watched the full demo twice and it reminded me of the more boring parts of the Sims 4. But maybe that's the magic as well?
It’s a familiar pattern, these days you can present a prompt engineering strategy from 6 months ago & it plays as an epic new paradigm for representing human thought.
The trick is they’re all just permutations on manipulating what’s in context + embeddings for memory + prompt engineering.
There’s new things here! I’m rushing to implement the 2D visualization part! But this simply isn’t ground-breaking
This paper feels significant. If chatgpt was an evolutionary step on gpt3.5/gpt4, then this is bit like taking chatgpt and using it as the backbone of something that can accumulate memories, reflect on them, and make plans accordingly.
I think you're ignoring the work that went into this as well as the useful technology that came out. Prompts make or break interactions with LLMs and ChatGPT especially. The difference in output from a naive prompt and a well crafted one is huge. This paper is one of many explorations of what happens when you design such prompts to work in an iterative fashion building upon the previous conversation text to produce emergent behaviour. These are the seeds of the next programming paradigm on a completely novel architecture. It's incredibly exciting to be present for the beginning of this field, this is what mathematicians must have felt like when they helped design and program the first computers.
Oh yeah I agree that it _could_ do all those things, but it would be a bit of overkill to always send every observation an agent encounters into the API/chatbox, and ask it to spit out an evaluation or action.
This paper does a nice job of separating the "agency" from the next word with context type predictor. I think that's why I like the paper, it is just chatgpt, in the same way that pizza is just dough, sauce, and cheese.
Yes, but I think this was a fairly obvious conclusion to imagine isn’t it.
If you were going to seriously consider using ChatGPT for AI in a game, you would need each instance of GPT to only know certain information it has gathered. And you would want it to reflect on observations to come up with new thoughts that weren’t observed.
Still, I’d argue you don’t really even need GPT for any of the above. GPT is useful if you want thoughts expressed as natural language, but you could easily code observations and thoughts into an appropriate abstract data structure and still have the same thing, except it’s a bit harder to understand since asking an NPC something in a language it understands and getting back a query result isn’t user friendly, but it can be just as amazing if you know what the data represents. The imprecision and fuzziness of an LLM leaves room for fun weirdness though.
To me, having not really intelligent agents with humanlike talking abilities is the worst outcome AI could produce.
These have zero utility for humanity, cause they’re not intelligent whatsoever. Yet these systems can produce tons of garbage content for free, that is difficult to distinguish from human-created content.
At best this is used to create better NPC in video games (as the article mentions), but more generally this is going to be used to pollute social media (if not already).
The model’s designed to show “what would the answer to this sound like?”, not to provide a correct answer. Unfortunately it also 1) is profitable (look how many humans we can fire while producing kinda similar results with a tool trained on those humans’ work!), and 2) fits the age old yearning (aliens, gods) of humans for humanlike-but-nonhuman sentience, a catch-22 that’s doomed to fail.
If the answer to "what would this sound like?" is accurate enough then it quite literally doesn't matter.
Large swaths of the brain work on prediction. You think real time reactions happen in sports? It would be impossible. You have blind spots in the eye you don't notice because the brain fills in the vision with predicted information.
If you can accurately predict what a doctor will say to arbitrary input then guess what ?, you're a doctor.
If you want to know what a plausible response to X might look like then the tool will give you that; if you are using it for getting correct information then no, it quite literally matters that the tool is not designed for that, and depending on domain and magnitude of this misapplication it could matter a whole lot.
(For example, personally, something that to me could look like a plausible answer is not exactly where my expectations are when it comes to medicine.)
It's literally the same thing humans do, at least to my personal experience. If you ask me a question, the first thing my mind generates is a plausibly sounding answer. That process is near-instant. The slower part is an internal evaluation - how confident I am this is the right answer? That depends on the conversation and topic in question - often enough, I can just vocalize that first thought without worry. Whether it "sounds right" is also the first step I use when processing what I hear/read others say.
If anything, GPT-3.5 and GPT-4, as well as other transformer-based models, are all starting to convince me that associative vector adjacency search in high-dimensional space is what thinking is.
It's literally the same thing humans do, at least to my personal experience. If you ask me a question, the first thing my mind generates is a plausibly sounding answer.
Ive been practicing meditation for some years now and over time I’ve realised this is not what I see happening when observing the mind. It’s one mode of operation, but it’s not the only mode. Using prediction to respond is mostly the lazy, non-interested approach, or useful if you can’t quite hear or understand someone.
What I feel lot of people have started doing is trivialising the mind. Hoping it’s all “this simple” and we’re three versions of ChatGPT away to finding God.
> It’s one mode of operation, but it’s not the only mode. Using prediction to respond is mostly the lazy, non-interested approach, or useful if you can’t quite hear or understand someone.
I didn't say it's the only mode. I said it's the starter mode. At least for me, this mode is always the point at which someone's words, or my response, first enter the conscious processing level. If I'm very uninterested (whether because I don't care or because I'm good at something), the thought may sail straight to my mouth or fingers. Otherwise, it'll get processed and refined, possibly mixed with or replaced by further thoughts "flowing in" from "autocomplete".
> What I feel lot of people have started doing is trivialising the mind. Hoping it’s all “this simple” and we’re three versions of ChatGPT away to finding God.
That's one way to look at it. I prefer another - perhaps we've just stumbled on the working principle of the mind, or at least its core part. For me, the idea that concept-level thinking falls out naturally from adjacency search, when the vector space is high-dimensional enough, is nothing but mind-blowing. It's almost poetic in its elegance.
And I mean, if you believe the human mind is the product of evolution, and not a design of a higher being, then the process must have been iterative. Evolution is as dumb as it gets, so it follows that the core paradigms of our minds are continuous, not discrete, and that they must be simple and general enough to be reachable by natural selection in finite time. Of all our ideas for how minds work, transformer models are the first ones that - to me - seem like plausible candidate for the actual thing that runs in our head. They have the required features: they're structurally simple, fully general, and their performance continuously improves as you make them bigger.
Now, to be clear, my take on LLMs is that language is incidental - it so happens that text is both easiest for us to work with, and the very thing we serialize our mind states for communication. But the "magic" isn't in words, or word processing - the "magic" is the high-dimensional latent space, where concepts and abstractions are just clusters of points that are close to each other in some dimensions. I think this isn't an accident - I feel this really is what concepts are.
One outcome of this study was that a panel of evaluators judged the bot interactions to be more "human" than when humans impersonated these characters. So you have a point.
It's good at presenting existing arguments in a good way. But the problem is that such models can only give back what they have seen, consolidating the status quo. There can be no reflection and no outside of the box thinking.
The authors of the study were clear to call out there are quite a few downsides to mass adoption of Generative Agents... the pollution and misinformation angle certainly being top of mind. I'm inclined to agree.
One thing I find particularly interesting here: The general technique they describe for automatically generating the memory stream and derived embeddings (as well as higher-level inferences about that they call "reflections"), then querying against that in a way that's not dependent on the LLM's limited context window, looks like it would be pretty easily generalizable to almost anything using LLMs. Even SQLite has an extension for vector embedding search now [1], so it should be possible to implement this technique in an entirely client-side manner that doesn't actually depend on the service (or local LLM) you're using.
Another user posted, and deleted, a comment to the effect that the morality of experimenting with entities which toe the line of sentience is worth considering.
I'm surprised this wasn't mentioned in the "Ethics" section of the paper.
The "Ethics" section does repeatedly say "generative agents are computational entities" and should not be confused for humans. Which suggests to me the authors may believe that "computational" consciousness (whether or not these agents exhibit it) is somehow qualitatively different than "real live human" consciousness due to some je ne sais quoi and therefore not ethically problematic to experiment with.
I think about this a lot, I hope that whoever is chasing the “sentient computer dream” at least considers that it might end up an ultra depressed schizophrenic pet that wants to commit suicide but literally can’t and then wants to be murdered. No one would believe it, it would just be told it’s being silly or it’s not conscious.
I know that’s a pessimistic view but I doubt it can’t be ruled out, really, I think people working in tech are going quite mad. Frankenstein mad. Some ethics should be discussed.
An AGI turning into God is probably one of an infinite amount of outcomes, we can’t really predict what being trapped in a cluster of silicon chips would feel like.
Life itself and the drive to go on is really quite illogical, it’s unlikely intellect alone is what sustains us and makes life worth living.
There is one thing I find particular about all the AGI/ASI sentient computer discussions. I’ve rarely ever in my life heard women talk about it. Like as if this is all some manifestation of male ego. We know we’re building mirrors of ourselves and we know that is scary. This imo is why men are so captivated by ChatGPT. It really is a mirror of us. Men love men, especially super men. Ha.
unfortunately, we can barely get some groups of humans to treat other humans with dignity, no less our genetically near-by mammalian friends. i don't hold my breath something completely alien, however sub or super intelligent, will be treated with utter ignorance and disrespect.
My thoughts exactly. As we move in this direction, it's worth building the moral framework to answer the question -- if we can create consciousness, or something quite like it -- is it ethical to do so?
And on the flip side -- when we live in a world where instantiating a consciousness is cheap-or-free -- does that change how we value sentient beings generally?
I think that we’re moving into Buddhist territory. I think the opposite would happen. It would be the ego death of basically the whole world. No one would be spared from the fact that their consciousness is not special. leaders, elites everyone.
If we find out that the soul itself exists, and who knows, maybe there is actually souls, then it might not be great because people would believe they have special souls. I think this is what the Hindu class system is.
Despite all the dreams we are fed of immersing ourselves in AI world and creating a work-free utopia, these shiny new inventions will instead be used to increase corporate bottom lines, not humanity's overall happiness.
Looking forward to playing StardewGPT. Half-joking aside, I do think that level of abstraction is probably a good choice. Familiar and comfy, but with enough detail to be able to find interesting social patterns.
Nice to see progress on this end. I've been hoping for some time for a continuation of AI generated shows (like the previously-famous Nothing Forever) that can 1) interact with the open world and 2) keep history long enough (e.g. by resummarizing and reprompting the model).
Controlling the agents and not merely making them output text through LLMs sounds very exciting, especially once people figure out the best way to connect APIs of simulators with the models
They interviewed the agents to ask them about their day, goals, observations, etc. They then asked a human to watch an agent through the simulation and then answer interview questions as the agent. The human performed worse than the agent in the interview, they didn't compare a human roleplaying against an agent.
The ‘safe’ tuning of the models is becoming a nuisance. As indicated in the paper, the agents are overly cooperative and pleasant due to the LLM’s training.
Pity they can’t get access to an untuned LLM. This isn’t the first example I’ve read it where research is being hampered by the PC nonsense and related filters crammed into the model.
I kid you not, I literally started making something like this yesterday. My plans were smaller, only trying to simulate politics, but still. Living in this moment of AI is sometimes very demoralizing. Whatever you try to make has been made by someone last week. /rant
It may have already been done, but doing the same things many times may bring interesting developments or crucial details no one had thought of before. Maybe you have such a crucial idea that can, after knowing about this paper, improve it.
You should still do that. And you should read the paper and pick the bits you think might be useful and iterate on them. The cutting edge isn't like a knife, it's more like a rotating barrel of blades that take little chunks out of the impossible, and come around again.
I heartily agree, having spent months jankily recreating MRKL and ReAct on a much smaller scale before realizing those papers have existed for months already.
How can anyone keep up with the sheer volume of new papers and concepts here?
Even Two Minute Papers is now lagging by two weeks.
But for convenience maybe I'll just copy them into a comment...
It describes an environment where multiple #LLM (#GPT)-powered agents interact in a small town.
I'll write my notes here as I read it...
To indicate actions in the world they represent them as emoji in the interface, e.g., "Isabella Rodriguez is writing in her journal" is displayed as
You can click on the person to see the exact details, but this emoji summarization is a nice idea for overviews.
A user can interfere (or "steer" if you are feeling generous) the simulation through chatting with agents, but more interestingly they can "issue a directive to an agent in the form of an 'inner voice'"
Truly some miniature Voice Of God stuff here!
I'll see if this is detailed more later in the paper, but initially it sounds like simple prompt injection. Though it's unclear if it's injecting things into the prompt or into some memory module...
Reading "Environmental Interaction" it sounds like they are specifying the environment at a granular level, with status for each object.
This was my initial thought when trying something similar, though now I'm more interested in narrative descriptions; that is, describing the environment to the degree it matters or is interesting, and allowing stereotyped expectations to basically "fill in" the rest. (Though that certainly has its own issues!)
They note the language is stilted and suggest later LLMs could fix this. It's definitely resolvable right now; whatever results they are getting are the results of their prompting.
The conversations remind me of something Nintendo would produce, short, somewhat bland, but affable. They must have worked to make the interactions so short, as that's not GPT default style. But also every example is an instruction, so it might also have slipped in.
Memory is a big fixation right now, though I'm just not convinced. It's obviously important, but is it a primary or secondary concern?
To contrast, some other possible concerns: relationships, mood, motivations, goals, character development, situational awareness... some of these need memory, but many do not. Some are static, but many are not.
To decide on which memories to retrieve they multiply several scores together, including recency. Recency is an exponential decay of 1% per hour.
That seems excessive...? It doesn't feel like recency should ever multiply something down to zero. Though it's recency of access, not recency of creation. And perhaps the world just doesn't get old enough for this to cause problems. (It was limited to 3 days, or about 50% max recency penalty.
The reflection part is much more interesting: given a pool of recent memories they ask the LLM to generate the "3 most salient high-level questions we can answer about the subjects in the statements?"
Then the questions serve to retrieve concrete memories from which the LLM creates observations with citations.
Planning and re-planning are interesting. Agents specifically plan out their days, first with a time outline then with specific breakdowns inside that outline.
For revising plans there's a query process where there is observation, then turning the observation into something longer (fusing memories/etc), and then asking "Should they react to the observation, and if so, what would be an appropriate reaction?"
Interviewing the agents as a means of evaluation is kind of interesting. Self-knowledge becomes the trait that is judged.
Then they cut out parts of the agent and see how well they perform in those same interviews.
Still... the use of quantitative measures here feels a little forced when there's lots of rich qualitative comparisons to be done. I'd rather see individual interactions replayed and compared with different sets of functionality.
They say they didn't replay the entire world with different functionality because each version would drift (which is fair and true). But instead they could just enter into a single moment to do a comparison (assuming each moment is fully serializable).
I've thought about updating world state with operational transforms in part for this purpose, to make rewind and effect tracking into first-class operations.
Well, I'm at the end now. Interesting, but I wish I knew the exact prompts they were using. The details matter a lot. "Boundaries and Errors" touched on this, but that section was 4x the size, there's a lot to be said about the prompts and how they interact with memories and personality descriptions.
I also missed this note: "The present study required substantial time and resources to simulate 25 agents for two days, costing
thousands of dollars in token credit and taking multiple days to complete"
I'm slightly surprised, though if they are doing minute-by-minute ticks of the clock over all the agents then it's unsurprising. (Or even if it's less intensive than that.)
Granularity looks to be 10 seconds, very short! It's not filtering based on memories being expected vs interesting memories, so lots of "X is idle" notes.
If you look at these states the core information (the personality of the person) is very short. There's lots of incidental memories. What matters? What could just be filled in as "life continued as expected"?
One path to greater efficiency might be to encode "what matters" for a character in a way that doesn't require checking in with GPT.
Could you have "boring embeddings"? Embeddings that represent the stuff the eye just passes right over without really thinking about it. Some of training up a character would be to build up this database of disinterest. Perhaps not unlike babies with overconnected brains that need synapse pruning to be able to pay attention to anything at all.
Another option might be for the characters to compose their own "I care about this" triggers, where those triggers are low-cost code (low cost compared to GPT calls) that can be run in a tighter loop in the simulation.
I think this is actually fairly "believable" as a decision process, as it's about building up habituated behavior, which is what believable people do.
Opens the question of what this code would look like...
This is a sneaky way to phrase "AI coding its own soul" as an optimization.
The planning is like this, but I imagine a richer language. Plans are only assertive: try to do this, then that, etc. The addition would be things like "watch out for this" or "decide what to do if this happens" – lots of triggers for the overmind.
Some of those triggers might be similar to "emotional state." Like, keep doing normal stuff unless a feeling goes over some threshold, then reconsider.
I'm going to be genuinely surprised if we don't see an incredibly buggy but incredibly fascinating Sims knockoff in a year or two built around a system like this.
It's a pretty obvious idea executed well. I definely think symbolic AI agents written in the programming language english and interpreted using LLMs is the way forward.
Maybe I missed it in the paper but they did post the source code (Github) for their implementation? Is anyone working on creating their own infrastructure based on the paper?
Ugh, and allow peasants to touch it? Tbh seeing Google research at the top of a paper these days feels like a red flag that I shouldn't get too invested in whatever cool new thing is on show. They're basically commercials for nerds - I still find their output interesting, but it's probably not going to be actionable.
I have been working on this even before I was aware of the paper. Feels a bit weird to have something almost identical released. Stay tuned, I guess. I plan to keep working on my version.
I'll be interested to see your approach. I've been bouncing some ideas in my head but not implemented anything yet (and I might never as I'm ethically conflicted here, as agents gain properties associated with sentience/consciousness).
Their approach to memory is interesting. I had been considering a tiered command-based approach -- "short-term memory" being an automatic summary of recent sensory inputs/command outputs; "long-term memory" being a detailed database queryable by the agent.
Something else. Calling it GPTRPG at the moment. There really isn't much to share right now, other than this basic demo that doesn't even have AI: https://gptrpgagent.web.app/
Ah yes, the universally moral acceptance of Barack Obama, the man who made signature strikes a lasting legacy of his presidency.
Bill Gates, the man who totally didn't use shady business practices and false announcements to destroy legit products to the point that people wrote micro$oft for a generation.
And don't even get me started on the new seasons of Picard.
The lack of a perfect human is a good reason not to produce ultra-humans who have 1000x higher IQ, are networked to every system on the planet, have access to all of humankind's knowledge, and don't need to eat, sleep, or die.
I wouldn't argue against your points. Yet, we need to have a discussion about character and role models. As I believe we should strive for the better not pointing out the flaws of others. Destruction is easy.
I really want to have these agents behave as artificial as they truly are, not some kind of human, especially not a known one. humans have so many flaws, we meatbags are full of emotions and other bad behaviors, and it really makes no sense to give them some artificial "feelings" like greed, fear and the like. that would influence/restrict their mental power too much, let them become and act as the machines they are. we should strive to become more like them, not the other way around, and eliminate the rampant egoism/individualism that destroys the planet and societies.
I love what this project has done. Currently they're basically having to work around the architectural limits of the LLM in order to select salient memories, but it's still produced something very workable.
Language is acting as a common interpretation-interaction layer for both the world and agents' internal states. The meta-logic of how different language objects interact to cause things to happen (e.g. observations -> reflections) is hand-crafted by the researchers, while the LLM provides the corpus-based reasoning for how a reasonable English-writing human would compute the intermediate answers to the meta-logic's queries.
I'd love to see stochastic processes, random events (maybe even Banksian 'Outside Context Problems'), and shifted cultural bases be introduced in future work. (Apologies if any of these have been mentioned.) Examples:
(1) The simulation might actually expose agents to ideas when they consume books or media, potentially absorb those ideas if they align with their knowledge and biases, and then incorporate them into their views and actions (e.g. oppose Tom as mayor because the agent has developed anti-capitalist views and Tom has been an irresponsible business owner).
(2) In the real world, people occasionally encounter illnesses physical and mental, win lotteries, get into accidents. Maybe the beloved local cafe-bookstore is replaced by a national chain that hires a few local workers (which might necessitate an employment simulation subsystem). Or a warehouse burns down and it's revealed that an agent is involved in a criminal venture or conflict. These random processes would add a degree of dynamism to the simulation, which is more akin to the Truman Show currently.
(3) Other cultural bases: currently, GPT generates English responses based on a typically 'online-Anglosphere-reasonable' mindset due to its training corpus. To simulate different societies, e.g. a fantasy-feudal one (like Game of Thrones as another commenter mentioned), a modified base for prompts would be needed. I wonder how hard it would be to implement (would fine-tuning be required?).
Feels like I need to look for collaborative projects working on this sort of simulation, because it's fascinated me ever since the days of Ultima VII simulating NPCs' responses and interactions with the world.
Said panel of evaluators found that AI agents pretending to be humans had more "believable" responses than humans pretending to be AI agents pretending to be humans. So that's... a result.
