Are these execution engines internally using Arrow columnar format or are they just exposing Arrow as a client wire format? AFAIK Spark and Presto does not use Arrow as execution columnar format, but just data sources/sinks.
You can configure Spark to use arrow for passing data between Java and Python via spark.sql.execution.arrow.pyspark.enabled but yes, Spark uses Java datatypes internally.
Nowadays most Spark users use either PySpark or Spark SQL. Spark is designed in some sense to mimic Scala collection library. Rust is nice but its interop with other open source big data libs (Hadoop/Zookeeper/HBase) can be painful.
I'd say start with some eager functional language such as OCaml without the "object" part. Haskell code is hard to debug cuz you can't rely on good old print.
Monad itself is never a silver bullet. Type systems and Composability are the true power of FP IMO.
Debugging declarative code is always hard, whatever the language. Haskell does allow you to use trace expressions which will output values to standard out, see Debug.Trace.
This is no reason to avoid arguably the most popular (and state-of-the-art) function language and implementation out there.
It took me way too long to come across Trace, if it was more widely known I don't think we'd see so many "impossible to debug" related issues. It's great.
I think we can all agree that debugging languages with lazy evaluation is specially hard. To effectively use Debug.Trace you actually need to add strictness and that can be fundamentally incompatible if you do intend to use the full power of lazy evaluation.
Debug.Trace won't solve the space leaks lurking around every corner of your program, and won't shave the hundreds of megabytes to gigabytes of RAM the compiler needs for compiling a non trivial program with dependencies.
