People are doing what you are are talking about. Use an LLM to parse non-structured data into a structured or semi-structured format (or even just create a bunch of structured metadata) that is more easily searchable, then using an LLM to parse a natural language query into a suitable query (maybe SQL), then find the result and return it.