Configuration-Driven
Define pipelines in HOCON config files. No code changes needed to add components, modify parameters, or reconfigure Spark settings.
Type-Safe
PureConfig provides compile-time type safety for your configuration. Catch errors early, not at runtime in production.
Lifecycle Hooks
Monitor execution, collect metrics, send alerts, and handle errors with composable lifecycle hooks.
Cross-Platform
Supports Spark 3.5.x and 4.0.x with Scala 2.12 and 2.13. Available on Maven Central.
Quick Install
// build.sbt
libraryDependencies += "io.github.dwsmith1983" %% "spark-pipeline-runtime-spark3" % "<version>"