Collaboration, reward and “wisdom of crowds”

Recently I finished reading Wisdom of crowds. The basic tenant of the book is that groups are capable of excellent decision making. People also seem to be endowed with capacity to coordinate and cooperate. Ability of groups of informed laymen to make correct decisions can surpass even experts or small groups of experts. This, of course, provided some conditions are met: groups have to be diverse and decentralized and some mechanism for them to make collective decisions have to be put in place.
This is well illustrated by the following opening story from the book summed up at Wikipedia in the next sentence:

“The opening anecdote relates Francis Galton’s surprise that the crowd at a county fair accurately guessed the weight of an ox when their individual guesses were averaged (the average was closer to the ox’s true butchered weight than the estimates of most crowd members, and also closer than any of the separate estimates made by cattle experts).”

A good case for direct democracy? Might as well be. Think only about this fact: today, in most democracies in the world people choose representatives and governors for periods of 4 years or more. When people are elected, they are generally elected in a certain “context” (historical, social, economical etc.). Take a look at Madrid bombings for example. There is no doubt that they had a decisive effect on elections that followed. Had the bombings occurred only 4 days later, the result of election would be radically different according to all polls. Would George W. Bush win the elections had they been taken after 9/11? I am well aware his approval ratings went berserk after the attack, but at that point he was already elected. Enough about politics…
Internet is play-ground par excellence for groups and collaboration on a large scale. Today we take Internet for granted. Still there is a number of interesting phenomena that occur on the Internet and that are closely related to human nature and society in general. Take for an example newsgroups. One person helps another, a complete stranger, and in most cases without any reward involved in the process. Quite different to how the companies in the free market behave and compete. Imagine companies sharing patents or company secrets. Well, I guess this is not so easy to imagine.
Just look at impromptu collaboration that happens on open source projects. They can involve a huge number of geographically and culturally disperse collaborators; still they can be surprisingly successful. All of this without any complex organizational structure, only slight or merit-based hierarchy etc. With Agile, such practices are getting their foot into the commercial enterprises (Self-organizing Teams).

Experts Exchange vs. Stack Overflow

What made me think about the book again is my experience with a site called Experts Exchange. The site lets users ask the questions and then the experts provide the answers. The one that asks the question assigns question points, based on the question difficulty and based on urgency he has in getting the question answered. These points are then awarded to or split between experts that provide the answer. Decision on to whom the points are awarded is up to questioner. In the case of conflict, a moderator can be asked to mediate.
When I started using Experts Exchange, points were earned by providing answers to questions made by others. You also earned some points for becoming a member, to start you up and then more point with time. They used to lock questions once someone posted the answer. The question would be locked until the person that asked the question responded if he accepts the answer or not.
Later on this has changed, so instead posting answers, all experts post comments. This way the question cannot be locked and new question momenum is not lost, nor people have to wait to give answer or post comment. The questioner then awards points to the comment he thinks is the best answer. If he thinks more than one answer deserves points, he can split points and award them to more than one answer.

Another change that occurred is that now you can pay to use Experts Exchange. You can still earn points and access site based on merit. If you pay for Premium Services you have unlimited number of points at your disposal and some other benefits, like private knowledge base. A maximum that can be gained from a single question is 2000 points. Besides points, there is also community rating system in place:

“To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.
•    The Grade of the Solution
•    The Zone Rank of the Expert Providing the Solution
•    The Number of Author and Expert Comments
•    The Number of Experts Contributing
•    The Feedback of the Community”

I think ratings were also added to the site later on. Note that ratings are applied to solution (question + all the answers as a whole).  Ratings will by no means influence how many points are earned/given away/lost. I remember this worked well and I even participated answering questions made by others earning points. Going back recently to Experts Exchange, I got a feeling there has been some change in the way things are done since the early begginings and I don’t thinkg that comercialization is the only factor. I got a feeling there is somehow too much noise that happens before you get a valid answer. Often all you get is noise. Stackoverflow also has points and reputation, but it does not work quite the same. More on that later on, let me first finish telling you my experience with Experts Exchange.

Murky water fishermen and bulldozers

Before next section, a disclaimer is in order: What follows is by no means result of some methodological scientific investigation. The sample my experience is based upon is not significant; it is also quite probable that the nature and expertise level of my questions have changed (hopefully I progresses somewhat in the last 10 years!). Finally, maybe it is just my conspiracy theory loving mind playing tricks with me. Anyhow, here we go.
After I had this rather frustrating experience with unhelpful answers on more than one occasion, I started to note a certain pattern. One type of noise makers would make some general or seemingly inoffensive comment, something like “This is interesting question; I would also like to know how it is solved.” Or “I really don’t know how to do that, but you could try looking at documentation at msdn, here is the URL that talks about the subject…”. Here is a real-life example:

“have a look at the configuration api
http://www.hibernate.org/hib_docs/v3/api/org/hibernate/cfg/Configuration.html

sorry I can’t be more help”

All of this might sound quite inoffensive, in the end its just noise – maybe its not done on purpose. The trick is however, that after some time, if question is not closed by the questioner (and why should one close the question if no acceptable answer was provided?), moderator goes on and after some warning emails closes the question for you. If you do not react, the moderator generally awards the points to those that tried to answer your question. With time, just by posting some hardly useful comments, you start earning points. I call this kind of expert “murky water fishermen”.
Another type of noise was answers that tried to change some basic premise in the question, and to provide alternative way to solve your problem. For example “Why don’t you use Entity Framework instead of LINQ2SQL?” Admittedly, sometimes putting things on their head can be useful, many times the problems are not solved because you keep trying to find the solution along the same lines. Some solutions require that you take a look at the problem from different angle, like the matchstick problem; the solution is only possible if you start thinking in 3D space. This can be especially useful for a newbie, he might not be even aware of the alternative. Most of the time however, you did your homework before coming to Experts Exchange – the basic premise of the problem cannot be changed. I call these, often arrogant coercers – bulldozers. On one occasion, after providing unhelpful alternatives, one of these bulldozers told me “Such thing cannot be done”. After some time, I solved the problem and posted the answer. Bulldozer was quite annoyed, but had to accept the defeat. Moderator gave me back the points. Not that I really wanted the points back, I was just a bit annoyed that this bulldozer had a courage and arrogance to say that my problem was impossible to solve. What surprised me is that moderator also erased the question! I thought that Experts Exchange is also about building the knowledge base (one they often mention), the answer to my question could have been useful to someone else as well, even though the dude that asked and dude that answered the question are one same person.
The first thing that comes to mind is that some mechanism for reducing noise should be put in place. Probably some kind of punishment for creating noise. This is something that Stackoverflow considers. Everyone registered can vote on both the questions and the answers. So, you get more points and better rating even if someone has liked your question, not only the answer. Even more important is the fact that anyone registered can vote on any question or answer. If someone marks your answer as Helpful you get 10 points. If your answer has been accepted, you get only 5 points more. This way, any other member is just as important as questioner. If someone thinks your question/answer was not helpful you lose points. Probably there is less need for moderation; on a large scale any bias is neutralized, there is less possibility to try to hack things like creating ghost users etc. I have not yet registered at Stackoverflow, but the next time I need to ask some question, I will give it a try. And hopefully I will remember to post part 2 of this post, after having some personal experience on how Stackoverflow measures up to Experts Exchange.
Going back to Surowiecki’s book, it seems that Experts Exchange failed one step from unleashing the “wisdom of the crowds”. What lacks is what Surowiecki calls “Aggregation” (Wikipedia: “Some mechanism exists for turning private judgments into a collective decision.”) It is up to the person that asked the question to award the points. There is a rating system in place, but separated from point system. Why not let the community be the judge on the answer value and helpfulness and award points? Also, why not let the answer receive the points more than once? I’d argue that even answer to a simple question that helped people a hundred of times is more valuable than answer to more complex question that helped someone on one occasion. Finally, why not let the questions be awarded points and why not have points taken for bad answers/questions? Using ratings system separated from point system, where contributions are heavily point-driven seems to dilute the value of community feedback.

Wisdom or Maddness?

Surowiecki’s book is an interesting read. It is even close to being practical and I would definitely recommended it if you are into social aspects of web 2.0. The book however, takes pretty much one side to the phenomenon of the crowds. This is not necessarily biased; as a matter of fact even the title of the book alludes to “Extraordinary Popular Delusions and the Madness of Crowds” book by Charles Mackay. Indeed, the madness and hysteria are the sides of crowds we generally have heard of. But by no means have we investigated them enough. One who did investigate it is Wilhelm Reich. Fortunately, The Mass Psychology of Fascism is on my desk and thanks to Internet one of my favorite movies “W.R.: Mysteries of the Organism” just a piratebay away.

Edit: In the meantime I registered at the stackoverflow.com and started participating. While you can decide to offer some of your own points as a reward, this is seldom necessary. The community driven point system seems to be fearer and overall experience is favorable so far.

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • del.icio.us
  • Reddit
  • Digg
  • StumbleUpon
  • Bloglines
  • Google Bookmarks
  • Y!GG

5 thoughts on “Collaboration, reward and “wisdom of crowds”

  1. Nice comparison! You have to give some credit to EE though; they were the pioneers after all. They tried to adapt, but it’s not that easy since they have gone commercial…

  2. Your analysis contains a lot of practical advice for building collaborative web 2.0 sites; something I plan to embark on. What do you think about recent Wikipedia controversy? Thanks for the post!

  3. I think you made a number of valid points. However, EE’s commercial model is completely intertwined with their points system. It even looks like a scam, EE is charging for something that someone else (registered expert) will provide since you can pay to buy points. Shouldn’t they somehow make it possible for experts to earn money then?
    Now, commercialization could entice more activity, or might promote sluggishness, you have your company buy you membership or this way you pay someone else to do tasks for you… How to make business model free, like on Stackoverflow? Not paying for points nor having adverts, how they make money?

  4. Nice article, but besides Murky water fishermen and bulldozers, there’s another rather annoying category the Point hunters.
    They comment very often with single line google sentence (or even “fixed” templates with multiple google links) and hope the questioner finds the answer within these comments. Some #1 experts use this method regularly and even complain in community support when a questioner doesn’t select their comment as their “RTFM” answer does “always” fit.

    The thought that the number of expert points is corresponding with the knowledge of the expert looks evident, but is wrong. The only “proof” points give is the fact that the “expert” has commented many times and spent a lot of time at EE. I’ve started a blog about one particular expert in the MS Access topic area “capricorn1” that’s showing how many errors regarding normalization this “expert” makes. Besides me almost all other top experts in the topic area know his “quality”, but there’s no mechanism in place to “educate” this expert.
    (Check the “Ab-normalization blogs at: http://community.zdnet.co.uk/blog/0,1000000567,2000342184b,00.htm)

    Nic;o)

  5. Having been at EE as expert since 1997, I would say that there are some EE “zones” where you will meet the above types and some questions will find them both and some periods will see an influx of newbie “experts”. That said – I sit in one set of zones (web related) and am both Zone advisor and Page Editor, which means I do some advising but seldom moderation. The moderators will however often ask me for advice about a specific question before deleting/awarding points and so on. I also perform clean-up of abandoned questions and I do NOT award points to oneliners or point hunters and I DO PAQ (save for posterity) questions with good comments – with or without rewarding or refunding points. I would like to earn money for what I do, but I still do it for free – having answered more than 7500 questions there. There has been and still are feuds like the one mentioned above, but I rarely see them unless I decide to google for experts-exchange like I did today.

    The people I help seem genuinely grateful – regardless of having paid or not. Answering questions will earn you points to ask questions yourself so you do not have to have a subscription.
    Michel

Comments are closed.