I've thought a lot about why fMRI data analysis tools are so terrible, and I've come up with a few reasons:
- It's impossible to implement the algorithms for fMRI data analysis efficiently in most "dynamic" programming languages due to the performance hit you take from using a dynamic language. (It might actually be possible in Julia, NumPyPy, Python with Numba, but these languages are not yet well-established.) On the other hand, dynamic programming languages are much better suited to exploratory data analysis than C is, so essentially all fMRI data analysis ends up being a mixture of C code and glue code in some other language. In this regard, I don't think SPM (MATLAB with C MEX files) is really that bad. It's fast and it avoids having to read the data from disk multiple times.
- People use what the tools they know, not the tools that are best for the job. FreeSurfer is a mess of C, C shell, and Tcl/Tk, but there's nothing else that can visualize fMRI data with comparable ease and accuracy. Most people in neuroimaging only know MATLAB, which is pretty terrible for analyzing large data sets because it can't mmap files (and it doesn't have the language features necessary to make this possible, and it's closed source).
- Related to the above, it's easier to get funding to develop a novel algorithm than to implement an existing algorithm in a way that makes it more useful/accessible to researchers. I believe this is slowly changing.
- There are a lot of different algorithms used for analyzing fMRI data, and no single package implements all of them. The necessity of each algorithm differs by lab and researcher, according to scientific necessity, personal preference, or the conventions of their subfield. People end up writing their own code to glue together methods from different analysis packages, which is, again, often written using the wrong tools.
- Us graduate students who know how to code well need to publish papers. There is comparatively little incentive to publish code.
Check out http://nipy.sourceforge.net/nipype/ It's basically a pipeline for plugging different packages' outputs and inputs into each other, wrappers are already there for SPM, FSL, and other well known programs and it's easy to write your own stuff if you know some Python. It speaks to exactly this approach, as it is algorithm/implementation agnostic, it just defines the process and tries to formalize getting data from one place to the next.
It's somewhat a moot point though, because everyone likes to gzip their nifti and unfortunately the file formats don't have a uniformly-accepted way to leverage the huge disk savings we can get from masked data without applying (stupid from the point of view of the types of analysis we do) compression. Even filesystem-level compression doesn't help. If you can fit all your data into available RAM, you're fine. If not...
Let's add to that fMRI data is next to useless for actually studying the brain. It can map large scale organisation, but it's like trying to measure the economy based on which power stations are in use at any given time.
You could learn quite a bit about the organization and function of a country based on the temporal-spatial patterns of its energy consumption particularly when you're given the power to control external stimuli. But yeah, it could be pretty useless for studying something on the scale of a subdivision. It depends entirely on which spatial scales you are interested in studying.
Following your analogy, trying to understand the brain by putting a few dozen electrodes in it is like trying to understand the economy based on watching people interact in a few dozen rooms.
Like you, I am skeptical of the explosion of human neuroimaging, but I think that, as a technique for determining where to drop your electrodes, fMRI can be a very powerful tool.
- It's impossible to implement the algorithms for fMRI data analysis efficiently in most "dynamic" programming languages due to the performance hit you take from using a dynamic language. (It might actually be possible in Julia, NumPyPy, Python with Numba, but these languages are not yet well-established.) On the other hand, dynamic programming languages are much better suited to exploratory data analysis than C is, so essentially all fMRI data analysis ends up being a mixture of C code and glue code in some other language. In this regard, I don't think SPM (MATLAB with C MEX files) is really that bad. It's fast and it avoids having to read the data from disk multiple times.
- People use what the tools they know, not the tools that are best for the job. FreeSurfer is a mess of C, C shell, and Tcl/Tk, but there's nothing else that can visualize fMRI data with comparable ease and accuracy. Most people in neuroimaging only know MATLAB, which is pretty terrible for analyzing large data sets because it can't mmap files (and it doesn't have the language features necessary to make this possible, and it's closed source).
- Related to the above, it's easier to get funding to develop a novel algorithm than to implement an existing algorithm in a way that makes it more useful/accessible to researchers. I believe this is slowly changing.
- There are a lot of different algorithms used for analyzing fMRI data, and no single package implements all of them. The necessity of each algorithm differs by lab and researcher, according to scientific necessity, personal preference, or the conventions of their subfield. People end up writing their own code to glue together methods from different analysis packages, which is, again, often written using the wrong tools.
- Us graduate students who know how to code well need to publish papers. There is comparatively little incentive to publish code.