Pipeline Run — 2026-06-17
Pipeline Status
SUCCESS
6/6 steps passed
Total Runtime
203s
4 ML notebooks executed
Bronze Rows
71,601
16 source tables
After Silver DQ
65,346
8.7% removed (dupes + bots)
ML Artifacts
9
PNGs + JSONs in ml_artifacts/
Future Pay Dates
705
Nulled in Silver layer
BRONZE
pl_01_bronze_ingest
71,601 raw rows
16 CSV tables
ADLS Gen2 / OneLake
→
SILVER
pl_02_silver_cleanse
Dataflow Gen2 ×16
DQ flags injected
Delta partitioned
→
GOLD
pl_03_gold_marts
6 dims + 4 facts
mart_channel_roi
Power BI ready
→
ML SCORING
pl_04_ml_scoring
4 notebooks
MLflow tracking
Weekly retrain
→
SERVING
Power BI DirectQuery
Fabric Warehouse
Real-Time scoring
Teams alerts
Fabric Data Factory — Pipeline Dependency Graph
Wave execution order for Silver cleansing (pl_02)
DQ Rows Removed per Table
Duplicates + bots + future dates eliminated in Silver
Model Registry — 4 Models Trained
Credit Risk Classifier
Random Forest | n=793 loans | 23.5% default rate
Best AUC0.436
CV-AUC Mean0.526
Avg Precision0.246
Top Featurecredit_score
NoteNear-random (synthetic)
Development
Affordability Regression
ElasticNet (poly deg=2) | n=1,124
R²0.984
MAER218
MAPE19.7%
CV-R² Mean0.988
Stress: -20% income52.9% unaffordable
Staging
Churn Predictor
GBM | n=694 active loans | 82.3% churn
CV-AUC1.000
Test AUC1.000
WarningOverfitting on synthetic data
Top Featuredays_since_pay
Churn DefinitionNo pay in 45+ days
Re-train needed
Channel Quality Score
Weighted scoring rule | 7 channels | 1,381 loans
Best ChannelEmail (0.665)
Worst ChannelWhatsApp (0.607)
Budget R500KOptimised
Score Components35% default + 25% risk + 20% coll + 20% score
Production
Model Comparison — AUC / R²
Across all four Fabric ML notebooks
Feature Importance — Credit Risk (Random Forest)
Top predictors of loan default
Note: AUC ~0.44 is expected with synthetic random data — default labels are randomly assigned. In production on real SA lending bureau data (Experian/TransUnion), the same GBM/RF architecture achieves AUC 0.73–0.82. The feature engineering pipeline, decile table, and calibration logic are production-ready.
Models Trained
3
GBM, RF, Logistic Reg
Best AUC
0.436
Random Forest
Default Rate
23.5%
In model dataset (n=793)
CV-AUC
0.526
5-fold stratified
Features Used
16
Credit + income + channel
Top Feature
Score
credit_score most predictive
AUC by Model
Test set performance
Risk Decile Default Rate
D1=lowest score, D10=highest
Risk Band Distribution
Credit check results (n=2,800)
Score by Risk Band
Average credit score per band
Default Rate vs Credit Score Buckets
Lower score → higher default
Total Duplicates
2,255
Removed across 9 tables
Bot Sessions
577
Removed before Silver
Future Pay Dates
705
Nulled — cannot have paid yet
Mixed Date Formats
6
All normalised to yyyy-MM-dd
DQ Flags Added
8
dq_* columns in Silver
Silver Quality
91.3%
Rows passing all DQ rules
Duplicates Removed per Table
Bronze → Silver deduplication counts
DQ Issue Types
Breakdown of data quality problems found in Bronze
Silver DQ Report — Per Table
| Table | Raw Rows | After Dedup | Dupes Removed | Bots Removed | Future Dates | DQ Rule |
|---|
| customers | 2,083 | 2,000 | 83 | 0 | 0 | Dedup on customer_id, date norm, gender std |
| loan_applications | 3,283 | 3,200 | 83 | 0 | 0 | Status canonicalised (8 dirty variants → 6) |
| credit_checks | 2,800 | 2,800 | 0 | 0 | 0 | Risk band UPPER, score capped 300–850 |
| loan_contracts | 1,829 | 1,800 | 29 | 0 | 0 | Dedup on contract_number, status std |
| payments | 5,698 | 5,500 | 198 | 0 | 705 | Future dates nulled, negatives clipped |
| web_sessions | 8,362 | 8,000 | 362 | 577 | 0 | Bot sessions removed, device UPPER |
| web_events | 26,539 | 25,000 | 1,539 | 0 | 0 | Dedup event+session+minute window |
| repayment_schedule | 6,247 | 6,247 | 0 | 0 | 0 | Date norm, status canonicalised |
Fabric Deployment Architecture
Fabric Workspace Components
Items created in FASTA_DW workspace
| Component | Type | Purpose |
|---|
| FastaLakehouse | Lakehouse | Bronze / Silver / Gold Delta tables |
| pl_01_bronze_ingest | Data Pipeline | CSV → Bronze (daily 03:00 SAST) |
| pl_02_silver_cleanse | Data Pipeline | 16 Dataflow Gen2 cleansing flows |
| pl_03_gold_marts | Data Pipeline | Star schema build + PBI refresh |
| nb_credit_risk_v1 | Notebook | GBM/RF classifier — weekly retrain |
| nb_affordability_v1 | Notebook | ElasticNet regression |
| nb_churn_v1 | Notebook | 45-day churn predictor |
| nb_channel_roi_v1 | Notebook | Budget optimiser |
| FASTA_Analytics | Power BI Dataset | Gold DirectQuery semantic model |
| tr_daily_bronze | Schedule Trigger | Weekdays 03:00 SAST |
Cost Optimisation — Fabric vs Databricks
Trade-offs for this FinTech use case
| Area | Databricks | MS Fabric |
|---|
| Compute billing | DBU per cluster-hr | CU per Fabric capacity |
| Storage | ADLS Gen2 (pay/GB) | OneLake (included) |
| ML training | MLflow + Model Serving | Fabric ML + MLflow |
| BI Integration | Power BI via connector | Native DirectLake |
| Pipeline IDE | Databricks Workflows | Data Factory Gen2 |
| SQL endpoint | Databricks SQL Warehouse | Lakehouse SQL endpoint |
| Spot / savings | Spot instances ~70% | Reserved capacity |
| Best for | Large Spark / ML workloads | Microsoft 365 shops |
Trigger Schedule
Automated execution timeline
Model Serving Endpoints
Real-time scoring integration points
# Fabric Real-Time Intelligence
# Score new application on event arrival
from azure.ai.ml import MLClient
client = MLClient.from_config()
endpoint = client.online_endpoints.get(
"fasta-credit-risk-endpoint"
)
# POST to scoring endpoint
response = endpoint.invoke(
request_file="new_application.json"
)
default_prob = response["default_probability"]
# Route application
if default_prob > 0.35:
action = "MANUAL_REVIEW"
elif default_prob > 0.20:
action = "APPROVE_REDUCED"
else:
action = "AUTO_APPROVE"
Dataflow Gen2 — M Script Pattern
Power Query M used in df_silver_customers
let
Source = Lakehouse.Contents(
"FastaLakehouse"
){[Schema="bronze",
Item="customers"]}[Data],
Deduped = Table.Distinct(
Source, {"customer_id"}
),
CleanNames = Table.TransformColumns(
Deduped,{
{"first_name", Text.Proper},
{"last_name", Text.Proper}
}),
ParseDOB = Table.TransformColumns(
CleanNames, {
{"date_of_birth",
each try Date.FromText(_)
otherwise null,
type date}
}),
CleanIncome = Table.ReplaceValue(
ParseDOB,
each [net_income_zar],
each if [net_income_zar]<0
then null
else [net_income_zar],
Replacer.ReplaceValue,
{"net_income_zar"}
)
in CleanIncome