Why fools rush in where wiser heads won't!
Kruger & Dunning 1999 (et al) found people are typically overly optimistic when evaluating the quality of their performance on social and intellectual tasks - often to an entirely illogical degree.
Self ratings of competence in driving, being an engineer, or in the case of 94% of mathematics professors surveyed who thought they do above average work we are notoriously inaccurate and biased towards over estimation of our abilities (if least skilled) and interestingly underestimation of them, if most skilled.
In particular, it is the very poorest (bottom 25%/quartile) of performers who grossly overestimate their performance, because their incompetence deprives them of the skills needed to recognise their deficits.
Interestingly it was across any or all skills and competencies tested, grammar, logical reasoning, driving, interviewing skills - same effect.
The studies demonstrated that poor performers lack insight into their shortcomings, even in real world settings where they have had years of feedback and even when given incentives to be accurate.
Those who are the top performers are also as poor at self assessment, by consistently underestimating their abilities.
It was only when these high performers were exposed to how difficult that their peers found the tests (that they had found very easy), that they then realised how exceptional their own performance was.
When considered in the environment of work performance it explains why so many poor performers are completely unaware of how poorly they are performing. Even in areas where they have received substantial feedback in the past they cling to their beliefs and act in accord with them.
The answer/s? Some of the research that followed this found that upskilling people in logic and critical thinking plus encouraging a mindset to actively continue learning could have impact on their self awareness and self assessment/insight.
Researchers also found that those people who felt intelligence is malleable (not fixed/'set in stone') are less prone to being overconfident and had greater self insight.
Original full academic article can be found here