What struck me in the comments is the gap between guidelines and action. For example, for the MSPE, the AAMC guidelines instruct schools to provide a graphic representation of a student's performance relative to his/her peers, yet Dr. Witteles' study found that 49% of schools assessed provided incomplete information about the distribution of key words, grades, or both. Dr. Cooke also highlights the sometimes idiosyncratic ways in which residents assess students, despite specific criteria to this effect. (I might add that I think this problem also applies to faculty.)
Is the problem the uneven and inconsistent application of existing guidelines? Do we need greater standardization? Or does improved transparency necessitate an entirely new system and mindset?