Try using InstructXL for embeddings. It’s got a more complex prompt structure for generating embeddings which might be more useful

have you tried other models to generate embeddings? I am going to that direction too to create an additional layer of helpers for search. Also, thinking if the document is not too big, it might fit into the initial context with the prompt

If the documents are large, try embedding smaller portions. If there's a heavy domain vocabulary, you might need a custom model.

I'd be very interested in knowing the outcome. Do you blog anywhere (or post on social)?