This part will illustrate two psychological biases, known as the illusion of validity and the halo effect. For research and university positions in France, the candidate must provide a written application (CV, project, description of previous work) and must also give an oral presentation and answer questions. The oral presentation is generally 5-15 minutes, followed by questions, for a total of up to 30 minutes. For a decision relevant to the entire career of a scientist, this is rather short. Yet it does seem to bear quite a strong weight on the committee's decision (although not in all committees).

Why some committees decide to put a such a strong weight on this stage is probably related to another cognitive bias named the illusion of validity. In his book, Daniel Kahneman describes his experience during his military service, where he had to evaluate officer candidates with a test. There had been an empirical analysis of that procedure, where they found that the grades given after the test were essentially uncorrelated with their future performance in officer school. However, when someone grades the test, he/she is very confident in the grades and thinks they are highly predictive. This is a case of substitution: the test is not directly related to what an officer does, but confidence is about the substituted attribute, whether it is highly correlated or uncorrelated to the target attribute. The illusion of validity is very strong, though, and it is difficult to avoid it because one's beliefs are built only from the cognitive elements that are currently available, not from the uncertain elements. It is impossible to consider unavailable elements, and so any consistent set of available elements seem like compelling evidence.

Let us examine the case of the oral presentation. Prior to the presentation, the candidate submitted a complete application including a written project, a detailed CV, and a report on previous results. Therefore, there is zero new information in the oral presentation, which is very short. However, the presentation is graded anyway, and you cannot ignore a grade that you decided on, whether it is relevant or not to the decision. I have heard from committee members that the presentation is useful because most committee members have not read the project (only the referee). This is obviously not a great defense for a lifetime decision!

Whether the presentation is explicitly graded or not, it is next to impossible to avoid the priming effects induced by a good or bad presentation. It is well documented in psychological research that many impressions that are completely irrelevant to the decision actually greatly influence it, when the decision is difficult. This is called the halo effect. These impressions include attractiveness, voice, self-confidence, or more generally “looking the part”. Often, committee members will tell you unashamedly that you made a good, or bad, impression on the committee, and that it might have played a role in the ranking. Of course they would never admit that they thought you “looked like a good scientist”. However, let us ask ourselves: what is the correlation between an impression conveyed in a one-time 5 minute presentation and the quality of the future career of a scientist over 30-40 years? I do not know what this correlation is, but I would guess not very high. The point is: neither does the committee. Yet, with zero evidence that there is any relationship between the two attributes, some committees seem to happily and confidently use it in the ranking decision.

Answers to questions are a slightly different problem. The committee will evaluate (substitution again) whether you answered their questions satisfactorily or not. From my experience both as candidate and committee member, the interview is unstructured. Any member can spontaneously ask a question. Some members ask many, others don't. Questions can be on the scientific project, or on other issues. Questions differ between candidates. The candidate will be graded on how well he/she answered the questions. Again, this is a case of substitution. Some questions may be highly relevant to the decision, such as how much the candidate seems to master the subject (although this is obviously problematic in the absence of any expert of the subject in the committee), discuss alternative hypotheses, etc. Other questions might bear less correlation with the target attribute. But the way the candidate answers questions will have an impact, whether a relevant or an irrelevant question is answered. Finally, the answers to many scientific questions cannot be evaluated by most committee members, because they do not know the subject. Therefore, there will be a case of substitution, in which some other element, most likely the candidate's confidence, will be used instead of the intrinsic quality of the answer.

Many committees consider that this second stage (oral presentation/interview) is very important to make their decision. Yet, they have close to zero empirical evidence that it increases the correlation between their decision and the desired outcome of the decision. What it does increase for sure, is the level of confidence in their decision. But, as I explained before, especially in the absence of any feedback, there is no general relationship between self-confidence and accuracy.