Medallion Architecture: Structuring Data for Scale

The Medallion Architecture is the data design pattern Arkham employs to logically organize and progressively refine data as it moves through the Lakehouse. This multi-layered approach is the key to how we ensure data becomes cleaner, more reliable, and more valuable at each stage, creating a robust and scalable foundation for all downstream analytics and AI applications.

The Three Layers of Quality

This architecture organizes data into three distinct layers—Bronze, Silver, and Gold—each representing a higher level of quality and intended for a specific audience and purpose.

• Bronze Layer (Raw): This is where we land all data from external source systems via Data Connectors. The data here is kept in its original, raw format ("as-is"). This immutable layer serves as a historical archive and the source for reprocessing, ensuring we never lose the original state of the data.
• Purpose: Raw data backup and historical archive.
• Users: Data Engineers.
• Silver Layer (Cleansed & Conformed): Data from the Bronze layer is cleaned, validated, and enriched to create an enterprise view of key business entities (e.g., customers, products). Common data models are applied, and data from different sources is joined and conformed.
• Purpose: Create a single source of truth for business entities.
• Users: Data Engineers, Data Scientists.
• Gold Layer (Business-Ready) The Silver layer data is aggregated into business-centric tables and views that are optimized for specific analytics and reporting use cases. These are the project-specific, denormalized datasets that power dashboards, ML models, and business applications.
• Purpose: Power analytics, business intelligence, and ML.
• Users: Data Analysts, Business Users, AI/ML Platform.

How It Improves Data Development

For a builder, this pattern provides a clear and powerful framework for developing data pipelines:

• Ensures Pipeline Reliability: By isolating raw data and incrementally applying transformations with transactional guarantees, it simplifies data validation and debugging. Issues can be traced back to their layer of origin, preventing the "garbage in, garbage out" problem.
• Improves Developer Productivity: The separation of concerns between layers provides a clear roadmap. Data engineers can focus on creating clean, conformed data in the Silver layer, while analysts can self-serve from performant Gold tables without worrying about complex upstream logic.
• Enables Auditing and Replayability: The immutable Bronze layer provides full data lineage. If a bug is found in a pipeline, you can confidently reprocess the affected data from the raw source to correct historical records without re-ingesting from the external system.
• Silver Layer (Cleansed & Conformed): Data from the Bronze layer is cleaned, validated, and enriched to create an enterprise view of key business entities (e.g., customers, products). Common data models are applied, and data from different sources is joined and conformed.
• Optimizes Performance and Cost: Gold tables are highly aggregated and optimized for performance, serving the majority of end-user queries efficiently. The more complex, full-fidelity data in the Silver layer is queried less frequently, balancing cost and performance.

Data Flow Diagram

The diagram below visualizes the flow of data from raw ingestion to business-ready insights, highlighting the transformation at each layer.

Related Capabilities

Understand how the Medallion Architecture fits into the broader platform:

• Data Platform
• OverviewData Lakehouse
• Pipeline Builder