I have a setup with poetry that runs the python data loaders in the poetry-managed virtualenv.

I just created a python project and then instead of `yarn run dev` to start the dev server, just run `poetry run yarn run dev` so the python is executed within the virtualenv.

This setup also lets you use a custom python package to define reusable and unit-testable code for the dataloaders that you can import into the *.json.py files to keep those really simple.

Why do you need to bundle these, is it to simplify iterating on frontend and data loaders simultaneously? Why not run them separately?

It’s totally possible to decouple these if your python outputs plain JSON/csv into the data/ directory that you commit into the repo or generate just before build time. Then you can import that raw json data into an Observable .md file.

But if you want dynamically generated data at build time and want to make use of Observable’s dataloader automatic execution of data/*.json.py, for instance, while still maintaining a custom virtualenv for the project rather than your system python, you’ll need some way to specify that virtualenv’s interpreter while observable executes the build for the dev server or the full dist/ output.

So for both options it’s largely a matter of taste. I personally like using the poetry virtualenv because it’s simple to manage dependencies and the venvs in one tool, while letting me use observable’s dataloaders with third-party or custom python packages. It sounded like the parent comment wanted to use this type of approach so I focused it to that scenario specifically. I like the simplicity of the single command to generate the data and build the site.

Thank you for the thorough response, that makes sense

Couldn't you solve that issue with nodeenv in python? This is how I usually add J's to my projects. It will keep node &nom/yarn/whatever else and your js in the venv as well.

Can you put a shebang line in a .sh data loader that points to the full path to bin/python within the virtual environment directory?

You can. But then the only time it realizes that the code has been updated is when you update the script or touch it. It’s a minor annoyance but it adds up when making lots of changes. Periodically deleting the cache works too but also annoying.