Announcing Spark 2.4 on QDS
We are happy to announce that we have released Apache Spark 2.4 in QDS in AWS environments. It is displayed as 2.4 latest (2.4.0) in the Spark Version field of the Create New Cluster page on QDS UI. We plan to deploy into Azure and Oracle environments soon.
Spark 2.4 brings in various features and improvements, including Barrier Execution mode, Flexible streaming connectors, enhancements to Pandas UDFs and Avro data source.
Please find details in the release notes as published here.