I organise my files, I don't put everything into the same directory. I save them all over my file system. Same for uploads, I do not put them into one directory prior to uploading.
I think a better method is what Apple has done in OS X. When the app needs to read from or write to a user specified file, the app calls a specific API that presents a file picker dialog. The file picker dialog is running in a separate process from the sandboxed app, and the app will temporarily be granted permissions to access this particular chosen file through this API.
If you have a directory that you want to expose, you can set that up. It doesn't have to be just ~/Downloads/Firefox. If you want to expose something like ~/Documents but deny access to ~/Documents/Private you can do that. With a little effort, you can probably even configure a helper utility that toggles access on and off dynamically with a status charm in the notifications area.
That sounds incredibly cumbersome, akin to things like umatrix or noscript (which I use but 99% of users would never touch or be able to correctly control).
