Alternative extension to do something similar, but using VSCode's native notebooks: https://marketplace.visualstudio.com/items?itemName=jaaxxx.l.... It's more tailored to crafting long-form few-shot prompts, rather than a single text box. Think ChatGPT playground but with a text editor interface that isn't absolutely terrible.
Nice part about that one is no separate server, no telemetry, and the backing file format is simple JSON you can directly import from your production application. However the range of supported models is smaller (basically only LLaMa-style and OpenAI-style interfaces are supported).
* Testing prompt behavior across various LLMs
* Sharing those prompts across multiple applications
We currently use a jupyter notebook to iterate, test, and validate prompts. Then move those prompts to our production app written in C#. If there were a C# SDK, I could use this tool to create a prompts config file and share it between the jupyter notebook and the C# app. The config file could also be added to version control.
Having said that, I don't understand why it saves the output of the LLM so maybe I'm missing something.
VSCode is a good target for single-player editing, and I can see something like this being helpful, but what does a collaborative experience look like, eg if you have a team of folks all working on the same prompts?
Yeah, source controlling the configurations, then cross-collaboration on teams isn't supported, but something interesting to explore especially with engineers and PMs collaborating together on prompting.
So one challenge we’ve had is folks would copy paste their prompt in google docs then write under it then try to run it and then paste back what worked well. A collaborative experience would be super useful so we don’t have to jump out and can just append/build on each other’s prompts inline.
Nice part about that one is no separate server, no telemetry, and the backing file format is simple JSON you can directly import from your production application. However the range of supported models is smaller (basically only LLaMa-style and OpenAI-style interfaces are supported).