unlock unlimited code runs for $79, limited time →
NOSSLETTER.techLIVE
back to archive

Parse native Java Dags and show their source

Language: Python+2115 -58feature
concepts demonstrated
Java JAR manifest parsingSubprocess command execution in PythonCustom exception handlingIntegration of Java workflows into Python systems
The Issue & Context

Previously, Airflow's JavaCoordinator did not parse native Java DAGs directly from executable JAR bundles, nor did it display their source code in the UI. This limited visibility and execution capabilities for Java-based workflows.

Root Cause Analysis

The JavaCoordinator lacked the ability to identify and parse JAR files with a Main-Class manifest attribute, run tasks from those native Java DAGs, and retrieve their source code from bundled metadata, partly due to manifest parsing limitations and missing integration with the Java runtime.

Solution Strategy & Architectural Pattern

This PR enhanced JavaCoordinator to parse all JARs in a DAG bundle that specify a Main-Class, run native Java DAG tasks using the Java runtime with proper classpath and JVM arguments, handle launch failures explicitly, and read source code from a sources.json index packed in the JAR under META-INF/airflow/sources/. It also improved manifest parsing to correctly handle folded lines and standardized error handling for unreadable or unrunnable JARs.

Key Takeaway for Contributors

Integrating native Java DAG support requires careful handling of JAR manifest parsing, task execution routing, and source code retrieval to provide a seamless developer experience; embedding source metadata inside JARs enables Airflow to show Java DAG code alongside execution.