Within one Spark job, the intermediate results can stay in memory. However, the intermediate results have to be written to the persistent storage when shared across Spark jobs.
Levy
Yes, while Alluxio provides a good layer above HDFS to cache intermediate results, allowing them not to be stored on disk
Within one Spark job, the intermediate results can stay in memory. However, the intermediate results have to be written to the persistent storage when shared across Spark jobs.
Yes, while Alluxio provides a good layer above HDFS to cache intermediate results, allowing them not to be stored on disk