Let us know how we can help
Sales & Pricing
Speak to a rep about your business needs
Help & Support
See our product support options
General inquiries and locations
Contact us
Control-M + GCP Composer
Your GCP Composer workflow starts before an Airflow DAG ever runs — and it can fail when something upstream breaks. A Cloud Storage object arrives late. A Dataflow job misses its expected completion time. Control-M orchestrates the full chain: Cloud Storage arrival, Dataflow transformation, GCP Composer execution, and downstream BigQuery delivery.
common workflow issues
Does this sound like your week?
These aren’t edge cases. They’re the normal operating conditions for teams running GCP Composer DAGs across multiple tools. Here’s how Control-M handles each one.
DATA ARRIVAL
Your Cloud Storage object is late. The Composer DAG is scheduled anyway.
Control-M coordinates the DAG with upstream workflow conditions, so execution can wait for required data instead of depending only on an isolated schedule. The workflow proceeds when its dependencies are satisfied, preventing premature processing and downstream failures.
CROSS-TOOL DEPENDENCY
Your Dataflow job finished late. The Composer run already passed.
Control-M tracks upstream completion and coordinates the GCP Composer DAG with the surrounding workflow. Cross-tool dependencies replace disconnected scheduling windows, so delayed processing does not leave the DAG waiting for the next schedule or a manual trigger.
FAILURE RECOVERY
Your DAG failed at 2:13 AM. Downstream jobs are still waiting
Control-M monitors GCP Composer job status and applies defined failure handling before dependent work proceeds. DAG rerun support, defined failure-status handling (including upstream_failed and skipped), and downstream dependency control help contain the failure and restore processing without blindly releasing subsequent workflow stages.
DAG RERUN
The DAG failed halfway through. Now you need a controlled rerun.
Control-M supports GCP Composer DAG reruns with the option to retry only failed tasks, plus configurable Rerun DAG ID behavior. Recovery becomes part of the orchestrated production workflow, helping teams restart processing in a controlled way while maintaining coordination with upstream and downstream dependencies.
SLA RISK
The DAG is still running. Your 7:00 AM delivery is at risk
Control-M manages the Composer job within the end-to-end workflow and can attach an SLA job to it for delivery tracking. Teams gain visibility into how upstream delays and DAG execution affect delivery, helping operations respond before pipeline delays become missed business deadlines.
Control‑M + GCP Composer
Control‑M + GCP Composer
|
workload.types |
Airflow DAG runs · DAG reruns (retry failed tasks only) · JSON-payload DAG runs · scheduled data pipelines · cross-platform data workflows |
|
trigger.type |
upstream job completion · file arrival · time schedule · workflow dependency · API-driven execution · Control-M condition |
|
cross_tool.deps |
Cloud Storage arrival · Dataflow job completion · BigQuery job · Dataproc job · REST API call · file transfer completion |
|
cloud.platforms |
Google Cloud Platform · GCP Composer · Cloud Storage · Dataflow · BigQuery · Dataproc |
|
error_handling |
DAG rerun · retry failed tasks only · failure status handling (upstream_failed, skipped) · Rerun DAG ID control · downstream cascade prevention · attached SLA job |
|
throughput |
scheduled batch pipelines · parameterized DAG runs · cross-tool workflow execution · automated data handoffs · task log capture to job output |
|
observability |
DAG run status · job output · task output retrieval · Control-M monitoring · SLA tracking · end-to-end dependency visibility |
end-to-end orchestration
One production workflow. Every tool in the stack.
Control-M orchestrates workflows across GCP Composer, Cloud Storage, Dataflow, BigQuery, Dataproc, file transfers, and cloud services in a single job flow — with dependency tracking, SLA visibility, and automated recovery across all of them.
- Cross-tool dependency: Cloud Storage → Dataflow → GCP Composer DAG → BigQuery
- Data-aware triggers: file arrival, API event, upstream job completion, DAG result
|
GCP Composer |
run DAG · rerun DAG · pass parameters · monitor status · retrieve output |
|
Cloud Storage |
coordinate file arrival · gate downstream processing · trigger workflow dependencies |
|
Dataflow |
coordinate processing jobs · track completion · control downstream execution |
|
BigQuery |
orchestrate query jobs · sequence analytics processing · coordinate data handoffs |
|
Dataproc |
coordinate processing jobs · manage dependencies · connect workflow stages |
|
File transfers |
coordinate file delivery · manage dependencies · trigger downstream processing |
|
REST APIs |
invoke external services · coordinate applications · connect workflow stages |
airflow coexistance
Control‑M doesn’t replace your Airflow DAGs. It runs the layer above them.
The objection is common: “We’re already on Airflow.” The issue isn’t what Airflow does – it’s what happens before and after Airflow runs. That’s where pipelines actually fail.
Airflow manages its DAG. Control-M manages everything surrounding it.
airflow handles
DAG-level orchestration inside the data pipeline
- DAG-level task orchestration within data pipelines
- Python operators, sensors, and task dependencies
- Execution graph for jobs that run inside your pipeline
- Manages retries within a single DAG context
control-m adds
The coordination layer around your DAGs
- Coordination layer around DAGs — triggers Airflow based on upstream conditions: file arrivals, API events, other tool completions
- Tracks each DAG’s SLA contribution across the full end-to-end workflow, not just its own routine
- Manages failure recovery when upstream dependencies fail before Airflow even starts
- Existing DAGs don’t need to be rewritten or migrated
MONITOR PIPELINES
Monitor GCP Composer DAGs across the full pipeline.
GCP Composer provides visibility inside Airflow, but production pipelines often span services beyond the DAG. Control-M provides a centralized view of Composer execution alongside the upstream and downstream jobs that determine whether the complete data pipeline succeeds:
-
DAG execution status
-
Job results and output
-
Upstream and downstream dependencies
-
Task output retrieval
-
End-to-end pipeline visibility
SLA ASSURANCE
Keep GCP Composer pipelines aligned to business SLAs.
A Composer DAG can complete successfully and still miss its delivery target when upstream processing runs late. Control-M connects DAG execution to the wider production workflow, providing the operational context needed to manage end-to-end delivery against defined SLAs:
-
End-to-end SLA visibility
-
Upstream dependency visibility
-
Pipeline deadline monitoring
-
Failure-aware downstream control
-
Centralized workflow monitoring
Documentation
Bring order to complex workflows
Learn how Control-M helps teams orchestrate complex processes with greater visibility, coordination, and control.
