Pipeline Monitoring: Operational Visibility and Control

Modern data platforms require more than just the ability to run pipelines; they require the ability to see exactly what's happening inside them. Pipeline Monitoring is Arkham's centralized service for providing this deep, real-time visibility into every pipeline execution.

For a builder, this tool is the key to maintaining operational excellence. It provides the granular, step-by-step diagnostics needed to debug failures quickly, audit performance, and guarantee the reliability of your data workflows.

How Pipeline Monitoring Works

Every time a pipeline is run in Arkham—whether on a schedule or manually—a unique execution is created and tracked. The monitoring service captures a rich set of metadata and logs throughout the pipeline's lifecycle, from the moment the compute cluster spins up to the final success or failure message.

Key Features for Builders

• ‍Real-Time Execution Status: See a live-updating list of all pipeline executions. You can filter by pipeline, status (COMPLETED, FAILED, IN_PROGRESS), date range, or execution type (SCHEDULED, MANUAL) to quickly find what you're looking for.
• ‍Step-Level Drill Down: Don't just see that a pipeline failed; see exactly which step failed. The UI allows you to click into any execution and inspect the status, start/end times, and duration of each individual step, making it easy to isolate bottlenecks or errors.
• Centralized Log Access: Access detailed, step-by-step logs for any execution directly from the monitoring UI. This eliminates the need to SSH into machines or hunt through scattered log files. As described in the Governance Overview, log access is securely governed and scoped to the user's project permissions.
• Failure Diagnosis: When a step fails, the monitoring service captures and displays the specific error message. It flags common failure types, such as cluster issues, permission problems, or compute timeouts, to accelerate your debugging process.
• Proactive Alerting: When a pipeline fails, the system automatically sends an email notification to the designated resource owners. This ensures that critical failures are addressed immediately, without you having to constantly watch the dashboard.

Related Capabilities

• Governance Overview: Understand how Projects fit into the broader security and governance architecture.
• Pipeline Builder: Create the data transformation pipelines that are organized within a Project.
• Projects: The workspace that organizes your pipelines and controls access to their monitoring data.