Let's suppose you have a data lake with 40,000 Parquet files. You need to list the files before you can read the data. This can take a few minutes. I've worked on data lakes that require file listing operations that run for hours. Key/value stores aren't good at listing files like Unix filesystems.
When Spark reads the 40,000 Parquet files it needs to figure out the schema. By default, it'll just grab the schema from one of the files and just assume that all the others have the same schema. This could be wrong.
You can set an option telling Spark to read the schemas of all 40,000 Parquet files and make sure they all have the same schema. That's expensive.
Or you can manually specify the schema, but that can be really tedious. What if the table has 200 columns.
The schema in the Parquet footer is perfect for a single file. I think storing the schema in the metadata is much better when data is spread across many Parquet files.
Yea, that's exactly what Delta Lake does. All the table metadata is stored in a Parquet file (it's initially stored in JSON files, but eventually compacted into Parquet files). These tables are sometimes so huge that the table metadata is big data also.
If the format is splittable you generally can get similar benefits, and parquet files have metadata to point a given reader at a specific chunk of the file that can be read independently. In the case of parquet the writer decides when to finish writing a block/RowGroup, so manually creating smaller files than that can increase parallelism. But you can only go so far as I'm pretty sure I've seen spark combine together very small files into a single threaded read task.
Also
> Parquet tables are OK when data is in a single file but are hard to manage and unnecessarily slow when data is in many files
This is not true. Having worked with Spark it's much better to have multiple "small" files than only one big file.