Advanced Data Engineering in Microsoft Fabric

• Design and implement a medallion architecture in Microsoft Fabric within a Lakehouse environment
• Implement data logic and transformations using Python (Polars, DuckDB) and PySpark in Notebooks
• Work with various data ingestion methods – Dataflows Gen2, Pipelines, and custom code
• Copy and reuse data across OneLake
• Profile, clean, and transform data using code in a variety of practical scenarios
• Work with both Lakehouse and Data Warehouse, including data security features
• Automate and orchestrate data workflows using Pipelines
• Optimize performance (partitioning, compression, Spark job optimization)
• Version code and deploy changes using Git integration and deployment pipelines

This course is intended primarily for data engineers and developers who want to work with Microsoft Fabric at the code level and design, implement, and operate data solutions in a production environment. It is also suitable for advanced analysts and data architects with Python experience who want to move toward data engineering and distributed data processing.

• Basic knowledge of Microsoft Fabric, at least at the level of course GOC680
• Knowledge of Python (pandas, list comprehensions, functions, error handling) and PySpark, at least at the level of course GOC685
• Basic understanding of relational databases and SQL
• Basic experience with data warehouses or data lakes
• Understanding of data extraction, ingestion, profiling, and transformation concepts
• Experience with data analysis and data integration tools (ETL processes, data pipelines)
• Knowledge of version control and Git integration is an advantage

1. Environment Setup and Core Principles

• Medallion architecture – principles and components
  • Lakehouse, Data Warehouse, analytical engines, semantic layers
  • Tenant configuration, capacity selection, performance and cost implications

2. Data Ingestion and Data Copying

• Data ingestion methods
  • Dataflows Gen2
  • Pipelines
  • Custom ingestion using Python / PySpark in Notebooks
• Copying and reusing data in OneLake
  • Shortcuts
  • Decision methodology and architectural implications
  • Practical implementation

3. Data Profiling, Cleaning, and Transformation

• Data profiling
  • Principles and methods
  • Implementation in Python / PySpark (Notebooks)
• Data cleaning and transformation
  • Designing cleaning mechanisms based on profiling results
  • Code-based data transformations
  • Slowly Changing Dimensions and advanced scenarios

4. Data Storage

• Lakehouse vs. Data Warehouse – differences and use cases
• Querying data
  • SQL queries
  • Querying Lakehouse and Warehouse
• Advanced components
  • Stored procedures, functions, roles, schemas
  • RLS, CLS, data masking

5. Automation

• Orchestration Pipelines
  • Coordination and dependencies
  • Integration of notebooks, dataflows, and SQL objects
• Notebook orchestration
  • Managing dependent steps in Python / PySpark
  • Fail-over and error handling

6. Monitoring and Optimization

• Performance optimization for Spark workloads
• Partitioning, compression, V-order, vacuuming
• Monitoring Fabric capacity and processing efficiency

7. Versioning and Deployment

• Git integration
• Deployment pipelines

Previous courses

Introduction to Microsoft Fabric [GOC680]

Python Essentials for Microsoft Fabric [GOC685]