Scoreboard for Hydrometeorological Predictions
Why do we need a scoreboard to visualize the quality of hydrometeorological predictions?
The joint evaluation of the quality of hydrometeorological forecasts issued by different forecasting systems for the same location has become a new challenge for modellers, service providers and users.
With the increasing availability of forecasts and outlooks from different models and sources, comparing different systems and outputs is becoming a crucial step towards:
- assessing the strengths and weaknesses of different approaches.
- choosing which service best meets the end-users needs in terms of accuracy and space-time resolution.
- building multi-model approaches to enhance informed decision-making on future hazardous conditions.
Moreover, continuous evaluation of forecast quality helps us better understand how uncertainties, sensitivities, and biases in predictions evolve with time.
Finally, water managers and stakeholders are more likely to use hydrometeorological forecasts they are confident in the quality of the forecasts. That why it’s important to have information on the quality of model predictions in hydrometeorology that:
- provides an estimate of the average differences in magnitude local practitioners can expect between the model predictions and the observations in everyday operations.
- can guide decisions on human and/or financial resources needed to further develop forecasting systems.
How is IMPREX responding to current needs?
As part of the IMPREX project, we have developed a unique platform a scoreboard to visualize and easily compare the performance of different systems or configurations of a system: SHyP - an open-source Scoreboard interface for displaying and comparing scores for Hydrometeorological Predictions.
The scoreboard interface is coded in R and uses an SQL database together with features of the ‘shiny’ package to guide users in selecting the case study, locations and scores of interest. The interface then interactively plots graphs.
Sharing an open source scoreboard is an excellent idea for a collaborative project as it ensures transparency, comparison and openness in computational experiments. This is particularly the case for this tool which provides a back-end data repository combined with a graphical, shared interface, allowing analyses and updates to be performed by several users at the same time.
We expected that, in the future, the widespread use of scoreboards like ours will:
- promote a more frequent dialogue between modellers and users on the quality of forecasts.
- develop a better understanding of forecast skill limitations in the field of hydrometeorological forecasting, notably when for capturing hydrological extremes that have a significant societal and economic impact.
- make forecast quality concepts more familiar to users.
- help users make informed decisions, as they can take into account the level of forecast accuracy and develop a more balanced expectation of the impact of operational products.
SHyP, an open-source Scoreboard interface for displaying and comparing scores for Hydrometeorological Predictions, was developed by Guillaume Thirel (Irstea, France), Jeffrey Norville (Irstea, France), Sylvain Descloux (Irstea, France), Maria-Helena Ramos (Irstea, France), Florian Pappenberger (ECMWF, UK) and Ilias Pechlivanidis (SMHI, Sweden), and was supported by the IMPREX project funded by the European Commission under the Horizon 2020 framework program (grant 641811)
Contact: Guillaume Thirel