A Comprehensive Study on Evaluating and Mitigating Algorithmic Unfairness with the MADD Metric (Journal of Educational Data Mining, 2024)
Predictive student models are increasingly used in learning environments due to their ability to enhanceeducational outcomes and support stakeholders in making informed decisions. However, predictive modelscan be biased and produce unfair outcomes, leading to potential discrimination against certain individualsand harmful long-term implications. This has prompted research on fairness metrics meant tocapture and quantify such biases. Nonetheless, current metrics primarily focus on predictive performancecomparisons between groups, without considering the behavior of the models or the severity of the biasesin the outcomes. To address this gap, we proposed a novel metric in a previous work (Verger et al., 2023)named Model Absolute Density Distance (MADD), measuring algorithmic unfairness as the difference ofthe probability distributions of the model’s outcomes. In this paper, we extended our previous work withtwo major additions. Firstly, we provided theoretical and practical considerations on a hyperparameterof MADD, named bandwidth, useful for optimal measurement of fairness with this metric. Secondly, wedemonstrated how MADD can be used not only to measure unfairness but also to mitigate it through postprocessingof the model’s outcomes while preserving its accuracy. We experimented with our approachon the same task of predicting student success in online courses as our previous work, and obtained successfulresults. To facilitate replication and future usages of MADD in different contexts, we developedan open-source Python package called maddlib (https://pypi.org/project/maddlib/). Altogether,our work contributes to advancing the research on fair student models in education.