This feels like one of those spaces where everyone says LLMs and AI are just going to make it a solved problem, but no-one in the comments here is even mentioning it - maybe not a good problem for LLMs after all?
> but no-one in the comments here is even mentioning it
The post has an entire section discussing this.
The problem with text-to-sql is that, as the post elaborates, writing SQL is not the problem. It's understanding the context and the data:
> On the other hand, a technical person would notice that the question doesn't make sense, and they would ask for more context. They would ask for details about the business person's hypothesis and the problem at hand. Then, they would explain what type of data is available, and work with the business person to formulate a precise and useful question.
Text-to-sql in practice is a solution that nobody was asking for, despite the insane number of SV startups shipping GPT-text-to-sql wrappers as products.
There certainly is like places where LLMs can help (post touches on this briefly), and that is in semantically exploring databases/tables/etc and contexts around data, but this is a very different project and would require a lot of curation from data teams to make it happen.
Most people that went after this tried for text-to-sql (e.g. ask a question and generate a ton of SQL to answer it). That approach has pretty much failed. The LLM could never have enough context to generate accurate SQL at a high enough rate to trust.
What we've found to actually work at Definite (I'm the founder) is text-to-semantic-query. This is an older video, but here's an example: https://www.youtube.com/watch?v=44mhLgUYOp8
Yes, it's simpler, but there's a few key differences:
1. You also have complete control over what the LLM can do / access thru the semantic layer (e.g. you can remove tables that the LLM shouldn't consider for analytical questions).
2. One of the biggest choke points for text-to-sql is constructing joins. All the joins are already built into the semantic layer.
3. Calculating metrics / measures is handled in the semantic layer instead of on the fly with SQL (e.g. if you ask something like "how much revenue did we generate from product X", you wouldn't want the LLM to come up with a calculation for revenue on the fly. Instead, revenue is clearly defined in the semantic layer).
4. The query format for our semantic layer (we use cube.dev) is JSON, which is much easier to control then free form SQL.
The semantic layer gives the LLM a well defined and constrained space to operate within whereas there are hundreds of ways for it to fail writing raw SQL.
The post talked about that too: LLMs could come up with a query for any question. A technical person would know how to spot the problems with that query or even the question, a non-technical person wouldn't know that they just asked for a bunch of hot nonsense.
With all the hallucinating (or "bullshitting") going on, it's hard to imagine LLMs working well for query generation. But hey, we're _very_ early days for all of this.
