You're absolutely right. This was a post I tossed together quickly just to see what could be done without thinking too much. In retrospect, I think this would be better implemented using Elasticsearch sparse vector fields which allow you to specify the value of every token. Maybe I'l make an update post to try again.
Practical advice! So many good products are lost by people that become fixated on unnecessary evals too early. You need to build your eval muscle AS you release product and get real feedback.
RAG is a pain to set up, so I tried something different. Instead of dealing with vector DBs and all that complexity, just let the LLM navigate well-structured docs like a human—exploring outlines and diving into sections. It’s simple, and works great for stuff like technical manuals or llms.txt.
Feels like it's a dopaminergic response to hearing a word but not knowing what it is. It's a novelty seeking thing. But once the work seems to be well understood, the novelty wears off and the novelty seeking mechanisms in humans quit responding.
If you get tinnitus, spend the first couple of months looking for a solution. Get opinions and second opinions. If that doesn't work, then wait for 6 months and then try again. If that doesn't work then wait for 9 more months then try again. If that doesn't work wait for 12 more months and then try again.
Most likely you'll habituate to the sound (as I have). What was once a living nightmare will become (miraculously) a non-issue. Occasionally, as one person said it even becomes "comfy" - or as I told my wife recently, it's just the sound that the world makes when you're alive. Really, it's not much different from other signal processing affects in other senses. If you close your eyes and concentrate you can see visual noise. It's also there when you open your eyes (floaters for example). But you don't worry about that because it's commonplace. Sound seems different because it's new - but it will become the same type of thing.
An important part of getting better is accepting that (likely...) you're not going to get better. And then, after just being with it for a while and not trying to push it away, you realize that it's not so bad - you get better. Just let life suck for a while. My experience hasn't been pleasant, but I've learned a lot about the nature of life. Fair trade. Wouldn't want to do it again, but glad to have done it.
I once presented with softwaredoug before a room of 400 people on a project that we didn't quite get finished! Yes... massive amounts of stress as we approached the deadline and realized that we were not going to hit it. Nevertheless, b/c of the stress we pushed harder than we normally would have, we learned a lot more than we otherwise would have, and _despite_ not finishing, the talk was one of the better received of my career. (We turned it into a lessons learned talk. We had plenty of great questions.)
How did we deal with the stress? I think we just sat with it. It was always there. Over time you recognize it for what it is, an illusion. That doesn't make it feel much better, but it gives you a bit more control and equanimity. And being able to push through hard situations despite the illusory feelings of dread opens the doors for doing some really interesting things.
Hint ... we've got a long way to go.
reply