Looking at it myself it's somewhat right, but it's an enormous pile of useless LLM blather - the response to "analyze the factory layout and activity" has nothing actionable, and most comments are generic ("well-organized and appears to be optimized for the efficient production of science packs"). Likewise the "analyze and optimize" requests don't seem useful enough for players to do anything with, though a lot of that is the prompt. As with most LLM stuff it just seems like someone threw a bunch of Factorio forum/Reddit posts in a blender and filtered out the stuff that was blatantly wrong.
Some specific errors I did notice:
- In the section with a processor unit subfactory, "The specific items being manufactured are not directly visible because the UI for the assembling machines is not expanded to show their recipes" is false, the player has pressed 'Alt' and the items being manufactured are shown. So this part of the response is plain wrong.
- " There are four distinct colors of science packs visible on the conveyor belts: red, green, blue, and purple, corresponding to the various levels of research complexity in the game." Only red/green/blue are shown. ChatGPT didn't make additional references to purple science but it was odd that it mentioned them at all.
- "In this Factorio image, we see a railway intersection that includes train signaling and a train crash" this is a deadlock, nothing has actually crashed. This is a minor nitpick but ChatGPT repeats the error throughout the analysis. And it also suggests that ChatGPT might not fully grok the "race condition" side of train scheduling, since deadlocks occur specifically because you're trying to avoid collisions.
I didn't want to spend too much brain calories reading all 20 pages in depth :) My general conclusion is that it's not useful enough for GPT-assisted Factorio play, and too flaky for any sort of automation of trivial Factorio tasks. I think it's plausible to make a FactorioGPT, but I doubt OpenAI's pretraining and RLHF resources covered this specific niche.
Some specific errors I did notice:
- In the section with a processor unit subfactory, "The specific items being manufactured are not directly visible because the UI for the assembling machines is not expanded to show their recipes" is false, the player has pressed 'Alt' and the items being manufactured are shown. So this part of the response is plain wrong.
- " There are four distinct colors of science packs visible on the conveyor belts: red, green, blue, and purple, corresponding to the various levels of research complexity in the game." Only red/green/blue are shown. ChatGPT didn't make additional references to purple science but it was odd that it mentioned them at all.
- "In this Factorio image, we see a railway intersection that includes train signaling and a train crash" this is a deadlock, nothing has actually crashed. This is a minor nitpick but ChatGPT repeats the error throughout the analysis. And it also suggests that ChatGPT might not fully grok the "race condition" side of train scheduling, since deadlocks occur specifically because you're trying to avoid collisions.
I didn't want to spend too much brain calories reading all 20 pages in depth :) My general conclusion is that it's not useful enough for GPT-assisted Factorio play, and too flaky for any sort of automation of trivial Factorio tasks. I think it's plausible to make a FactorioGPT, but I doubt OpenAI's pretraining and RLHF resources covered this specific niche.