grading expert opinion | |||||||||||||||||
a four (4) point screening test to weigh the soundness of proposed expert opinion: 1.Is the claimant credentialed & in good standing within the field that he is making the claim? (1 or 0) 2. Has claimant’s statement(s) been submitted to peer-review? And is the review accepted with a majority of reviewers within the field? (1 or 0) 3. If no to above, is claimant’s statement(s) within two standard deviations (2 SD rule) of a representative sample of the field? (1 or 0)(note: the 2SD rule comes from Ian Ayres book SUPERCRUNCHERS") 4. Does the claimant have a record of predictive claims? What is the track record (accuracy) of these predictions? plug in the score of the 'expert' and receive a weighted (i.e. predictive accuracy is given more weight) ranking of the expert's claim. ER = A+B+C+D x PR ER = Expertise Ratio (higher is better)PR = Prediction ratio Examples: 1+0+0 = 1 X 0.25 = 0.25 Very low ranking – Not worthy of review nor acceptance 1+0+1 = 2 x 0.45 = 0.9 moderate ranking 1+1+1 = 3 x 0.33 = 0.99 remember a high score does not necessarily entail objectivity, only that soundness of opinions are not all equal! While this is very crude it may at least provide food for thought on how to filter the more sound from the less sound claims being made constantly. I originally worked on this after reviewing how my emotions differed greatly between the 'new's segments' & the weather report. i truly felt informed during the weather report, whereas i felt the range of emotions (anger, worry, sadness, fear, etc)during the rest of the news broadcast.......
spaine, Aug 19 2009
What do you think of this idea or comment? | |||||||||||||||||
Users who liked this idea also liked: | ||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||
Other ideas in category (Culture): | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Add your comment
i forgot to mention that if this proposal is sound it could be used as a screening tool for all newscasters and pundits on our national airwaves. Just like in baseball we have measures of a players ability that have become more and more accurate over time (i.e. OPS 'on-base + slugging' is more telling of a players offensive skill than just batting average. or likecollege football's BSC ranking system. the networks using an independent ranking firm could assign a broadcaster's 'rank' by the ranking criteria. sort of a 'warning label' for information
I'm not exactly sure what kind of expert opinion you're talking about? Is this person testifying in court? Making business decisions? An opinion is far better justified by a logical argument than by a mysterious score.
This fine formula looks like another example to me of trying to make judgement seem scientific by using math. An opinion is still an opinion. Financial guys charge millions for opinions, yet a baby with a dart board can pick stocks almost as well. And Oh yeah, the financial guys use nice looking formulas. I'm sorry I'm so critical--I guess I'm a cynic lately.
I have a suspicion that Google already uses something like this for users as well as websites. It makes sense when you're trying to decipher the truth or legitimacy or intention of a hierarchy of information and its sources. I doubt Google would be willing to discuss this but I'm willing to bet they have a lot of experience with this.
I imagine the algorithm would be something similar to http://www.20q.net/