AI signals, such as alerts or recommendations, are increasingly used to help people make decisions. Across various industries, AI monitors ongoing events, detects when a decision is critical and jumps in, sometimes with a subtle alert, other times with an explicit recommendation. In finance, fraud detection tools flag suspicious transactions for review. On factory floors, AI tools alert operators to potential defects. In IT, error alerts prompt engineers to check for bugs.
But which type of signal actually helps us make better decisions, especially when the AI is reliable and provides highly accurate advice? This question is increasingly relevant, as AI tools become better calibrated and consistently dependable. As this trend continues, we must ask: Are there costs to relying on AI too much, even when its advice is correct? We explored these questions in a study using chess — a setting where AI recommendations are trusted, accuracy is exceptionally high, and decision quality is easy to measure.