Questionnaire - Do you need a model review?

Consider the quality and accuracy of the model results, but also the speed at which they are computed.

User concerns often become apparent through additional manual adjustments to the model results. Stakeholders with declining confidence typically ask more frequent questions about the accuracy or interpretation of the outcomes.

This includes changes in laws and regulations that affect how the model may collect, process, or use data, such as stricter privacy rules or limitations on data access. In addition, there may be regulations that have substantive implications for the model, for example through revised conditions for model optimization or adjusted criteria for predictions.

Internal or external audits are conducted to assess the quality, reliability, and compliance of models. An internal audit is often carried out as part of risk management or quality assurance, for instance when important decisions rely on model outcomes. External audits are typically required by regulations or supervisory authorities, or when there are concerns about the objectivity or transparency of the model. Within CQM, we supported the NS with their preparations for an external audit, Read more here.

Models rely on underlying data sources, such as customer data, market information, or external statistics. Changes in the structure, availability, or quality of these sources — for example due to system updates, source replacements, or altered definitions — can affect the functioning and reliability of the model.

This may be reflected, for example, in a decline in the number of customers purchasing a product, shifts in preferred payment methods, changes in product demand, or external influences such as rising interest rates or fluctuations in raw material prices. These changes can impact the assumptions on which the model is based.

A transition to a new platform, software, or infrastructure can affect how the model operates. This may involve migrating to the cloud, using a different programming language, or changing the data storage infrastructure.

Models may perform well with smaller amounts of data or fewer users, but as usage increases or data volume grows, they can struggle with scalability or efficiency. This may result in longer processing times, reduced performance, or an inability to handle larger datasets effectively.

Model developers or maintainers may leave under various circumstances. In some cases, knowledge about the model, and even the source code, can be entirely lost, making it difficult to maintain or modify the model. In other cases, documentation is well organized and knowledge has been proactively transferred to one or more colleagues, ensuring continuity.

Uncertainty within the team about how the model works can stem from a lack of clear explanation, insufficient documentation, or ambiguity regarding the methodologies and assumptions used. This can make it difficult to effectively maintain, modify, or further develop the model, and also to trust its outcomes.

Clear documentation includes for example the assumptions underlying the model, the data used, the choice of algorithms, and the model parameters. This information is essential for understanding the model, identifying potential limitations, and implementing future changes.

Think of key decisions such as major investments, long-term training programs for staff, or the development of new products. These decisions often carry significant financial, strategic, or operational impact.