This study opens the black boxes and fills the literature gap by showing how it is possible to fit a very precise Machine Learning model that is highly interpretable, by using a novel ML technique, Extreme Gradient Boosting (XGBoost), and applying new model interpretability improvements. In addition, we identify several significant indicators that could assist in predicting business financial distress. The data were collected from the Eikon database from a sample of 1760 French firms (1585 healthy and 175 failing) in 2018. Identifying the leading indicators of business failure is critical in assisting regulators, and for business managers to act expeditiously before a distressed business reaches crisis point. Our results reveal that higher levels of equity per employee, solvency, the current ratio, net profitability, and a sustainable return on investment are associated with a lower risk of business failure. In contrast, a higher number of employees leads to business failure.