Zansa › Answers › Forecasting and data science
Why does my forecast look accurate in testing but fail in real life?
The usual causes are data leakage, random rather than time-based splits, tuning against the test set, a test period unrepresentative of the future, or inputs available in testing but not at forecast time. Real conditions may also have changed since the test period.
Check each in turn:
- Were any future values used, even indirectly?
- Was the split by time?
- Were you testing at the same horizon used in practice?
- Did you select the best of many models on the same test data?
- Would inputs such as planned promotions be known in advance?
A useful safeguard is a final untouched test period, plus live tracking for a few weeks before relying on the model for important decisions.
Want this applied to your business? Zansa builds dashboards and forecasts for small and mid-sized companies worldwide. Start with a free intro call.
Book a call