Let us know how we can help
Sales & Pricing
Speak to a rep about your business needs
Help & Support
See our product support options
General inquiries and locations
Contact us
Control-M + OCI Data Flow
Your OCI Data Flow workflow starts before an Apache Spark application run ever executes — and it can fail when something upstream breaks. An OCI Object Storage object arrives late. An OCI Data Integration task finishes after its processing window. Control-M orchestrates the full chain: Object Storage data arrival, OCI Data Integration transformation, OCI Data Flow execution, and downstream delivery.
common workflow issues
Does this sound like your week?
These aren’t edge cases. They’re the normal operating conditions for teams running OCI Data Flow workloads across multiple tools. Here’s how Control-M handles each one.
OBJECT STORAGE ARRIVAL
The Object Storage data arrived late. Your Spark run started anyway.
Control-M holds the OCI Data Flow job until upstream data dependencies are satisfied, then releases the Spark run at the right point in the workflow — avoiding premature execution, incomplete inputs, and downstream reprocessing.
CROSS-SERVICE DEPENDENCY
OCI Data Integration finished late. Data Flow is still waiting.
Control-M coordinates dependencies between OCI Data Integration, OCI Data Flow, and surrounding jobs in one workflow. Successful upstream completion releases the Spark run automatically, eliminating disconnected schedules and manual handoffs.
FAILURE RECOVERY
Your Spark run failed at 2:13 AM. Downstream jobs kept waiting.
Control-M monitors OCI Data Flow job status, results, and output, identifies unsuccessful execution, and prevents dependent work from proceeding. Configurable workflow recovery and alerting give operators a controlled path to resolution before failures cascade.
RUNTIME CONFIGURATION
Today’s Spark run needs different arguments and executor capacity.
Control-M can pass additional OCI Data Flow run details, including arguments, parameters, configuration, driver and executor shapes, and executor count. Teams can operationalize changing runtime requirements without separating the Spark run from its end-to-end workflow.
SLA RISK
The 7:00 AM analytics handoff is approaching. Spark is still running.
Control-M connects the OCI Data Flow job to the end-to-end workflow SLA, providing centralized monitoring and alerting around critical deadlines. Teams see the Spark run in business-process context and can intervene before downstream delivery is missed.
INTEGRATION FACTS
Control‑M + OCI Data Flow
|
workload.types |
Apache Spark applications · batch processing · Spark SQL · PySpark · Java/Scala Spark · machine learning workloads · Spark Streaming |
|
trigger.type |
upstream job completion · Object Storage data arrival · API-driven workflow · time schedule · Control-M event · dependency condition |
|
cross_tool.deps |
OCI Data Integration task · OCI Object Storage delivery · Apache Airflow DAG · database job · REST API call · downstream analytics job |
|
cloud.platforms |
Oracle Cloud Infrastructure · OCI Data Flow · OCI Object Storage · OCI Data Integration · Control-M SaaS |
|
error_handling |
status polling · failure tolerance · downstream cascade prevention · workflow recovery · SLA monitoring · Control-M alerts |
|
throughput |
large-scale datasets · serverless Spark processing · batch workloads · long-running Spark Streaming · configurable executors · configurable driver and executor shapes |
|
observability |
run status · run information · job results · job output · end-to-end workflow monitoring · SLA visibility · Control-M audit context |
end-to-end orchestration
One production workflow. Every tool in the stack.
Control-M orchestrates workflows across OCI Data Flow, OCI Data Integration, OCI Object Storage, Airflow, databases, and cloud services in a single job flow — with dependency tracking, SLA visibility, and automated recovery across all of them.
- Cross-tool dependency: OCI Object Storage → OCI Data Integration → OCI Data Flow → analytics handoff
- Data-aware triggers: Object Storage arrival, API event, upstream job completion, dependency condition
|
OCI Data Flow |
trigger Spark runs · pass run configuration · monitor status · retrieve run information · coordinate SLAs |
|
OCI Data Integration |
trigger data tasks · coordinate upstream transformations · manage workflow dependencies |
|
OCI Object Storage |
coordinate data arrival · application artifacts · Spark inputs and outputs |
|
Apache Airflow |
trigger DAGs · coordinate DAG completion · connect DAGs to enterprise dependencies |
|
Databases |
coordinate extraction · gate Spark processing · trigger downstream loads |
|
REST APIs |
invoke services · coordinate API-driven dependencies · connect external applications |
airflow coexistance
Control-M doesn’t replace your Airflow DAGs. It runs the layer above them.
The objection is common: “We’re already on Airflow.” The issue isn’t what Airflow does – it’s what happens before and after Airflow runs. That’s where pipelines actually fail.
Airflow manages its DAG. Control-M manages everything surrounding it.
airflow handles
DAG-level orchestration inside the data pipeline
- DAG-level task orchestration within data pipelines
- Python operators, sensors, and task dependencies
- Execution graph for jobs that run inside your pipeline
- Manages retries within a single DAG context
control-m adds
The coordination layer around your DAGs
- Coordination layer around DAGs — triggers Airflow based on upstream conditions: file arrivals, API events, other tool completions
- Tracks each DAG’s SLA contribution across the full end-to-end workflow, not just its own routine
- Manages failure recovery when upstream dependencies fail before Airflow even starts
- Existing DAGs don’t need to be rewritten or migrated
MONITOR PIPELINES
Monitor OCI Data Flow runs in workflow context.
OCI Data Flow provides native run-level monitoring, but production pipelines rarely stop at Spark. Control-M adds a centralized view across the surrounding workflow, connecting Data Flow execution with upstream dependencies, downstream jobs, and operational outcomes:
-
OCI Data Flow run status
-
Job results and output
-
Upstream and downstream dependencies
-
End-to-end workflow status
-
Centralized operational alerts
SLA ASSURANCE
Keep OCI Data Flow pipelines on schedule
A successful Spark run can still arrive too late for the process depending on it. Control-M lets teams attach SLA jobs to OCI Data Flow workloads and coordinate execution with upstream and downstream dependencies to keep critical pipelines on schedule:
-
SLA monitoring for Data Flow
-
Cross-job dependency coordination
-
Advanced scheduling criteria
-
Centralized workflow status
-
Controlled downstream execution
Documentation
Bring order to complex workflows
Learn how Control-M helps teams orchestrate complex processes with greater visibility, coordination, and control.
