>Same with notebooks, you can write NASA-production-grade software in a notebook, but most likely you won't.
One of my family members literally maintains a software platform for test bedding production NASA spacecraft at JPL and it is all on top of Jupyter notebooks.
People are literally on course to build AGI and there’s a good chance that large portions of that work will be done within Jupyter notebooks.
To suggest real work doesn’t get done in notebooks is simply asinine.
As I have embedded further into data science and machine learning teams within my career I have come to learn that a lot of the things software engineers care about around maintainability, reusability etc. are often completely antithetical to data science productivity. In many cases practitioners would consider some of the listed cons of notebooks as features rather than bugs.
There is obviously a time and a place for pulling out those practices in such settings but blindly applying software engineering standards to this kind of work is both all too common and often rather counter productive.
If notebook style dev doesn’t jive with you personally I get that, and god speed. I’ve worked on CV/ML teams with a lot of CS degree grads and they don’t always want Jupyter notebooks. However, I’ve noticed a tendency for folks to extrapolate that to blanket statements about notebooks in general. As someone who has inherited and had to clean up many jumbled/bad notebooks I’m empathetic but I would challenge folks to lean into considering more about the circumstances the previous notebook author was working in and what kinds of productivity outcomes they were responsible and prioritizing rather than casting judgement on said author or their tools of choice.
I think we need to clearly distinguish two things here, though. A lot of software engineering ritual is dedicated to shipping software - that is the part that tends to be of very remote relevance in research and scientific contexts. But there are also more fundamental principles and techniques that concern writing code, which apply regardless of the reason why you're doing that, and are beneficial in any context - in a sense that, if you follow them, you will spend less time overall writing code (and thus more time doing whatever it is you're trying to achieve with that code).
However, the problem is that those things are not trivial and have to be learned before they can be applied. And that is where the value becomes questionable - the time spent learning all that could instead be spent on learning other things. But if you already have a software engineering background, you do benefit from doing things "right" to some extent.
One of my family members literally maintains a software platform for test bedding production NASA spacecraft at JPL and it is all on top of Jupyter notebooks.
People are literally on course to build AGI and there’s a good chance that large portions of that work will be done within Jupyter notebooks.
To suggest real work doesn’t get done in notebooks is simply asinine.
As I have embedded further into data science and machine learning teams within my career I have come to learn that a lot of the things software engineers care about around maintainability, reusability etc. are often completely antithetical to data science productivity. In many cases practitioners would consider some of the listed cons of notebooks as features rather than bugs.
There is obviously a time and a place for pulling out those practices in such settings but blindly applying software engineering standards to this kind of work is both all too common and often rather counter productive.
If notebook style dev doesn’t jive with you personally I get that, and god speed. I’ve worked on CV/ML teams with a lot of CS degree grads and they don’t always want Jupyter notebooks. However, I’ve noticed a tendency for folks to extrapolate that to blanket statements about notebooks in general. As someone who has inherited and had to clean up many jumbled/bad notebooks I’m empathetic but I would challenge folks to lean into considering more about the circumstances the previous notebook author was working in and what kinds of productivity outcomes they were responsible and prioritizing rather than casting judgement on said author or their tools of choice.