Frequently Asked Questions (FAQ)

Overview

• What does the Collaborative Product Evaluation system do?
• How is the Collaborative Product Evaluation system different from other community-driven product evaluation sites?
• Who can submit ratings to the system?
• How do I submit a rating to the system?
• I submitted a rating for a functional detail, but it's not showing up in the list of ratings at that level. Why not?
• How is the "credibility" of a rating determined?
• How are users chosen to moderate the ratings of other users?
• Why should I register and log in to the system?
• I forgot my password. How can I find out what it is?
• How can I change my password?
• How can I change my user ID?
• When I log in, there is a message that I have some moderation points. What should I do?

The Collaborative Product Evaluation system enables a community of users to cooperate on evaluating complex products with many functions. The CPE system represents product capabilities as a hierarchical taxonomy of features, allowing products to be compared at an arbitrary level of detail. Anyone can submit ratings for individual features of a product, representing their assessment of the quality of that feature's implementation. Submitted ratings are qualified through distributed moderation, whereby a select subset of community members is automatically given the opportunity to moderate ratings of other users by boosting or lowering the weight of a rating based on the credibility of its explanation.

Many web sites offer the ability for customers to submit ratings for products. However, these sites usually only allow users to assign ratings at a relatively high level. Typically, users can provide an overall grade for a product, or at most two or three detail grades addressing specific product characteristics. While this process may be effective for rating relatively simple consumer products, it may not be suitable for evaluating more complex products with extensive feature sets, such as computer systems or complex software.

The Collaborative Product Evaluation system allows products to be reviewed at much finer levels of detail than most existing evaluation sites, extending the comparison of products down to dozens or hundreds of features. Users can only submit ratings for specific functional details rather than entire products, and summary ratings for products are calculated from weighted averages of the lower-level ratings. As a result, the system measures the quality of products based on the deep expertise of contributing users, rather than on general allegiances towards particular products or vendors.

The other difference is that the Collaborative Product Evaluation system qualifies submitted ratings through distributed moderation. Unlike some existing product review schemes, in which ratings are moderated by dedicated system managers, or any user can grade the ratings of other users, distributed moderation continuously enlists a subset of the user community to assist in the moderation process. Because moderation privileges are kept rare, users tend to take advantage of the opportunity to moderate ratings when it's given to them.

Anyone can submit a rating for a specific function in any of the evaluated products. Users can register with the system and submit ratings under their user ID, or they can submit ratings anonymously. However, anonymous ratings start out with a lower credibility level than ratings from registered users.

Ratings can only be submitted for functional details, so you must descend to one of the lowest functional levels for the submission option to appear. At these "leaves" of the functional hierarchy tree, you will see the ratings that other users have submitted, if there are any yet, and a "Submit Rating" button to submit your own rating.

By pressing the "Submit Rating" button at a functional level, you can submit a rating for the quality of that function in one of the evaluated products. In addition to rating the function on a five-point scale from "poor" to "excellent", you must provide a non-trivial written explanation of your rating. You may include certain HTML tags in your comment, including <b> (bold), <i> (italics), and <a href=""> for including links. Note that including links to supporting material may help to boost the credibility of your rating in the distributed moderation process.

Registered users may only submit one rating per product feature under their ID (however, they can rate multiple product features). Anonymous ratings for a product feature can only be submitted once per 24 hour period from a particular IP address.

Every rating has a credibility level associated with it, ranging from -1 to 5. Users can select the credibility level at which they view the evaluation. Only ratings that have a credibility greater than or equal to the currently selected credibility threshold are displayed at the functional detail levels and are used in calculating higher-level summary scores.

If you don't see your rating, it most likely has a credibility that is below your currently selected credibility threshold. To see all ratings that have been submitted, select the lowest possible credibility threshold (-1).

A rating's credibility is determined by the background of the user who submitted it, plus moderations applied by other users who have been selected as part of the distributed moderation process. Ratings that are submitted by registered users usually start out with a credibility level of 1, while ratings submitted anonymously start out at level 0. However, ratings from registered users with unusually high "karma" levels might start at level 2, while registered users with exceptionally poor karma might start out at 0 or lower.

The credibility of a rating may be further adjusted by moderations of other users who have been given moderation points. These users can increase or decrease the credibility of a rating by expending one of their moderation points.

Registered users who read the site regularly and maintain a positive karma establish moderator eligibility, which means that the system may automatically assign them moderation privileges on occasion. A user who has been given moderation privileges is credited with a limited number of moderation points (usually 10). Each of these points can be applied to a single rating to increment or decrement its credibility level by one.

Logging into the CPE system provides several benefits:

• You can set weights at any level of the evaluation to prioritize the importance of functions in their environment. By setting these weights, the summary scores at every level will be customized for your exact requirements.
• Your ratings will start out with higher credibility than ratings from anonymous users.
>
• You can preserve certain systems settings from one session to the next, such as the preferred rating credibility threshold.
• You may be offered the opportunity to moderate the ratings of other users.

You can request that your password be mailed to you by entering either your user ID or email address.

If you are logged in, you can go to the profile section to change your password, email address and other contact information, and preferences.

You can't. User IDs are the primary means for identifying registered users, and thus remain constant for the duration of your participation on the site.

Congratulations! You have been selected to participate in the distributed moderation process. You now have the opportunity to help maintain the quality of ratings that have been submitted to the system.

When you have moderation points and view an evaluation page at the lowest functional level, the ratings that are displayed have an additional control showing moderation qualifiers. When these qualifiers appear, you can select a qualifier for one or more ratings, and press the "Moderate" button at the bottom of the page to apply the moderations. Each rating that you moderate expends one of your moderation points. You cannot moderate your own ratings, and you can only moderate a rating once.

Unused moderation points expire after some period of time (usually 3 days), so use them or lose them!