Chapter 9

Combining the Intelligence of Self-Organised Crowds

In This Chapter

Getting to the bottom of self-organised crowds

Using crowds to obtain information and make decisions

Working through the crowdsourcing process

Using a prediction market to gather opinions

Some of the most innovative applications of crowdsourcing are those that use self-organised crowds. Self-organised crowdsourcing is similar to crowdcontests in that, to engage a self-organised crowd, you post a goal, something that you would like to accomplish, and you offer a reward for the group of people who are able to accomplish that goal. But instead of telling the crowd how it should work and how it should be organised, you let the crowd answer those questions. You let the crowd decide the best way to work and the best way to organise itself.

Self-organised crowdsourcing is most useful for helping you learn things from the crowd that you can’t imagine, such as an idea for a new consumer product that might radically change the market, or a new service that might offer a tremendous improvement to a neighbourhood. If you’re looking for something that you can easily imagine and describe, using a conventional public opinion survey may be an easier option. You use a survey, for example, if you want to know whether the public is more likely to vote for a Liberal candidate or a Tory candidate, or to see if they are willing to pay a few more pence to buy a beer that has a better taste but more calories than the most popular brand.

In this chapter I help you discover how you might use a self-organised crowd. I look at how to put a question to the crowd, how to guide the crowd towards your goal, how to work with the leaders of self-organised crowds and, finally, how to use a specialised version of the self-organised crowd – the prediction market.

Getting to Grips with Self-Organised Crowds

Practically speaking, the term self-organised crowds isn’t the best description of how this kind of crowdsourcing works. The term suggests that the crowd members start working together, slowly start to find some organising principles, and eventually become a unified force. In fact, that isn’t the way it usually works. When you post a job for a self-organised crowd, you see a leader emerge who has some idea about how to do your job. That leader begins to recruit other people to help him do the job. Eventually the people form a team and complete the work.

This form of crowdsourcing is referred to as being self-organised because the crowdsourcer isn’t organising the crowd. The crowdsourcer is merely posting the job and offering the reward, as he might do in a crowdcontest. The person who’s organising the crowd is drawn from the crowd and probably doesn’t even know the crowdsourcer.

Self-organised crowds are similar to crowdcontests (see Chapter 5). You put a question to the crowd and ask it to gather information, invent something or make a decision. Self-organised contests are different from crowdcontests, though, because they can’t be handled by a single person. They require a team from the crowd. You aren’t going to organise that team, though. You’re going to let it organise itself. However, you do need to give the crowd guidance about how to organise itself, encourage the members if they have their own ideas and, of course, tell the crowd what you need it to do.

Crowdsourcers commonly use self-organised crowds for many kinds of work, including:

Product innovation. Say you’d like to market a new product. For example, you may want a product that enables you to track your pet and know where it is. You also want to send the animal a signal – a whistle or high-pitched sound – so that you can train it to return home. You can turn to the crowd to design such a product.

Disaster relief. Say a family has been driving into the woods and is lost. The family’s car has been found, but the people are missing. Volunteers may know the region far better than professional rescuers, so by turning to the crowd you can do a more thorough search.

Data collection. Say hurricane Anne-Marie has just slammed through your region and you need to know how badly it’s damaged the ecosystem. Satellite photos only tell you so much. Instead, you can put a call to the crowd to count species, identify fallen trees or spot changes to the local topography.

The classic example of a self-organised crowd is the DARPA Red Balloon Challenge. In 2010, the Defense Advanced Research Projects Agency (DARPA) in the USA ran a contest to see how well the crowd could organise itself. DARPA anchored ten large weather balloons at various unidentified places around the USA and offered $40,000 to the individual or team who found the balloons first. A group based at the Massachusetts Institute of Technology (MIT) found all ten in just a few hours and won the contest. MIT organised the crowd by using various social media and by promising to divide the prize money among the people who either found a balloon or recruited people who found a balloon.

By their very nature, self-organised crowds give you information. Therefore, consider using self-organised crowds whenever you want to discover something about the crowd. Keep the goal of benefiting from the wisdom of the crowd at the forefront of your project. Although you hope that crowdsourcing will be less expensive than conventional methods of doing work, you’re most interested in getting the right material from the right people.

Determining What You Need the Crowd to Do: Information Gathering and Decision Making

When you start working with a self-organised crowd, you need to be sure of what you want the crowd to do. You don’t simply turn to the crowd and say ‘Tell me what you know.’ You have to provide a framework that enables the crowd to work – a set of ideas or directions that guide its efforts. If you don’t provide those ideas or directions, you’ll have a crowd that won’t know what to do.

The first thing you need to do is to decide whether your aim is to gather information or make decisions. Crowds can do either activity well, but they tend to use one method to gather information and another to make decisions. When crowd members gather information, they spread out across the globe and try to enlist other people to help them. They use their social networks to ask friends and friends of friends to help the effort. When the crowd makes a judgement, the members usually spend more time interacting with each other. They may debate, discuss or trade ideas. Sometimes they search for more information, but they spend a greater proportion of their time thinking and processing ideas.

---

---

Self-organised crowdsourcing is perhaps the trickiest form of crowdsourcing to do well. You increase your chances of a successful outcome if you crowdsource with a group of people who are already interested in your ideas. For example, you improve your likelihood of succeeding with a self-organised project to reorganise the rubbish collection in your town if you connect with the citizens of your town and the institutions that are interested in the welfare of the area. In this case, you might start by talking to local business organisations, school groups and churches. As soon as you’re able to convince some of them to volunteer for your project, you can build a site for your project and make a great start in recruiting your crowd.

Gathering information

When you ask the crowd to gather information, you may be asking members to go into the world and collect facts. You can ask them to photograph buildings, find lost children, count wildlife, record prices or do anything that involves finding information. Searching, remembering, creating, collaborating – all are forms of gathering information.

You start a self-organised effort to gather information by posting a request on a social network site, your own web page, a blog or a website where you can attract a crowd. Unlike for other forms of crowdsourcing, you can’t find many established websites that encourage a self-organised crowd to gather information. You usually have to create that site yourself, promote your work and draw a crowd to your site. (See Chapter 10 for more about attracting a crowd.)

In your request, you state the information you want and describe how or from where it needs to be collected. You usually need to be fairly specific about what needs to be done, otherwise you find the crowd does what it wants to do and not what you need. If you post a call that says a child is missing in the north of town and that you have an award for the group that finds him, you may discover that you’ll be presented with all sorts of children who aren’t the ones that you wanted to find.

You can see self-organised crowds being used to gather information in many marketing activities. One common form is the contest that challenges schools or groups to collect cash register receipts from local grocery stores. This kind of contest is usually sponsored by grocery stores or companies that make consumer products. The sponsors offer a prize to the group that collects the largest number of receipts. At the end of the contest, one team takes home a prize, while the sponsor gains a lot of information about how people purchase groceries.

The self-organised crowd can be a powerful means of gathering information. The Society for Disseminating Useful Information (SDUI) publishes a series of English-language periodicals that contain useful information – information that people can use to improve their homes, families and businesses. The periodicals are very popular, and the SDUI has decided it wants to translate these magazines into other languages. For several years, SDUI tried to translate the periodicals itself. The society started by hiring Spanish translators, and quickly found that the process was too expensive and had to stop.

The SDUI then decided it would crowdsource the translation through macrotasking (see Chapter 7). The society found this process faster and more efficient than using the traditional translator, but quickly noticed there were fewer readers in the Spanish-speaking audience than it intended. To better reach a non-English-speaking audience, it decided to try self-organised crowdsourcing.

To this end, SDUI posted a call to translate articles. Members of the crowd could pick any article from an SDUI magazine and translate it. Each article had to be reviewed by at least two other members of the crowd. Everyone involved in the translation would receive free access to the article they translated and free access to one other translated article.

As the work progressed, the self-organising aspect of this crowd taught SDUI which articles had value to readers in other languages and which did not. The members of the crowd translated the articles that had the greatest interest to them. Many articles would be translated into one language and not another. Some would be translated into Spanish and not Afrikaans, and vice versa. Since SDUI was prepared to offer a reward to translate any article, it was surprised to see that some articles were never translated out of English. It learned something from the crowd that it hadn’t expected.

Experts are still learning how to use self-organised crowds in disaster relief. In emergency situations, crowds often need additional guidance to focus their attention on the problems at hand or to avoid causing more harm than good. If you’re thinking of using self-organised crowds in an emergency situation, you need to make plans and probably have a way that you can bring some leadership to the crowd.

Making a decision

Self-organised crowds do many things. They can gather information for you. They can do work for you. They can even make decisions for you.

The simplest form of decision-making is the vote. You put a question to the crowd members and ask them to select one option. When you’ve concluded that enough people have participated, you stop the process and count the votes. The result with the largest support is the group decision.

Other approaches are to ask the crowd members to rate an idea on a scale from 1 to 10 and average the results, or to divide the crowd into groups, asking each group to reach a decision through consensus, and then have the groups vote.

You can also create indirect ways to obtain a decision from the crowd, such as through the prediction market. A prediction market looks like a simulated stock market or futures market. You can use a prediction market to find out what the crowd thinks about the chances of any event in the future. It can estimate the chances that a politician will win an election, a sports team will win a big tournament or an actor will win an Academy Award. For details of prediction markets, see the later section Organising a Prediction Market.

Gathering and deciding

When you use a self-organised crowd, you usually use it to do multiple things. Commonly, you want it to both gather information for you and make decisions about that information. You want it to tell you which information is important and which is not. To handle both kinds of work, self-organised crowds have to become more complex. They need to have a hierarchy among the crowd so that some members gather information, some make judgements, and some coordinate the work of the others.

Wikipedia is a good example of a self-organised crowd that’s engaged in both activities. The bulk of the crowd gather information. They write material and edit material, create new entries, and add pictures and links to the online encyclopaedia.

---

---

Wikipedia has a second process that makes decisions about material that’s been added to the encyclopaedia. These decisions are made by Wikipedia administrators. These administrators achieve their position through a consensus process. They nominate themselves to the existing administrators. The existing administrators discuss the new nominee and determine whether they’ve a consensus to accept this person to their ranks. The new administrator can serve for as long as he maintains that consensus. If he starts making decisions that are controversial, he may ultimately lose his role as an administrator.

Designing the Process

When you design a self-organised crowd, you start with three questions:

What do I want to find out from the crowd? For example, you may want to know what people think about your product, and you want to be sure that you’re not missing something important.

What can I get the crowd to do? For example, you ask the crowd to suggest changes to the product and then you make judgements about those suggestions.

Where can I find a crowd to do this work? This last question is crucial. You usually can’t find a crowdsourcing platform that offers you a ready self-organised crowd. You have to use websites that may enable you to find a group of people interested in your project and willing to be part of the work.

After you’ve answered these questions, you can start to think about how to shape a self-organised crowd. The following sections take you through all the considerations for designing your crowdsourcing project with a self-organised crowd.

Finding the crowd

When you use self-organised crowds, you usually can’t just go to a simple commercial crowdsourcing platform and find a crowd. You can’t find general self-organised crowd platforms either, like Mechanical Turk in microtasking (see Chapter 8), oDesk and Elance in macrotasking (see Chapter 7) or Kickstarter and Indiegogo in crowdfunding (see Chapter 6). The only time that you can regularly find such a site for self-organised crowds is when you undertake innovation crowdsourcing (see Chapter 18). Innovation crowdsourcing, though, isn’t the only form of self-organised crowdsourcing.

In the absence of self-organised crowdsourcing platforms, you start the process by identifying a site on the Internet through which you may be able to attract people who are interested in your project. That may be a general social media site like Facebook. If you’re working on a community issue such as a problem associated with your town, you might work with a site that’s regularly visited by residents of the town. Or, if you’re working on a problem that spans geographical areas, such as something that might improve the health of children around the world or help reduce air pollution, you will want to work with a platform that’s associated with that issue. Historically, many self-organised crowdsourced projects have been run from blog sites, although you can certainly use a Facebook page to do the same thing.

You start a project by creating the page and educating the public about your problem, and you draw the crowd to that site by using social media, your social network and perhaps the support of people and institutions such as a local police department or community centre that’s interested in your work. (Also take a look at Chapter 10 on promoting your project.) When you have a crowd at your chosen site, you can start explaining your project and guiding the crowd to act.

Preparing clear rules

Self-organised crowdsourcing is often presented as a contest or as an activity that appears to be a contest (for more on contests, flick to Chapter 5). The contest involves gathering information and making decisions based on that information. You have to make clear rules to describe both activities. (Visit Chapters 5 and 10 for more helpful pointers on writing rules and task descriptions.)

Sometimes during the course of a project, you discover things in the results that surprise you and make you wish you’d asked a different question of the crowd. Rather than modify your current questions, consider starting a second crowdsourcing activity. That activity may be based on material from the first, but it should offer new rewards to the crowd. In general, crowds are resentful if you change the rules in the middle of an activity. However, they usually welcome a new activity.

Information gathering

Usually, creating rules for this part of the self-organised crowdsourcing is easy. You describe the kind of thing that you’re trying to obtain from the crowd – your goal. You describe the crowd is to create the information, how the members can submit it, and the criteria it must meet.

One useful way to obtain information is to engage members in a dialogue. You can select members from the crowd and ask them questions about what they’re bringing to the effort, the things they like or dislike about the effort, or the reasons for their vote. You can have a dialogue in many different ways. You can choose members of the crowd at random. You can choose members who have special roles. You can offer a special side contest in order to select people to talk to.

You may find it useful to start your dialogues by addressing the crowd as a whole. You can prepare little videos that describe what you’re trying to do or the information you’re attempting to gather. These videos can not only help you have better discussions with individual members of the crowd, but they also help the rest of the crowd understand what you’re trying to do.

Decision making

The decision-making part of a job is harder to describe to the crowd.

As with all aspects of crowdsourcing, you need to describe the goal to the crowd as clearly as you can. If you were running Wikipedia, for example, you would say that you want to produce a comprehensive general encyclopaedia. However, stating the goal isn’t enough. You need to give the crowd principles for making decisions. If you don’t give principles, the crowd members can easily make decisions by arguing with each other about whose opinion is better. Such arguments rarely produce anything useful. However, if you give the crowd principles for making decisions, the crowd members will argue about which possible choice best fits which principle. Such arguments are just as noisy as arguments over opinions – indeed, one kind of argument sometimes can’t be distinguished from another – but arguments that involve principles or standards usually produce something useful.

The Wikipedia leadership, for example, gives its contributors basic guidelines for encyclopaedia articles. It explains that all contributions should be drawn from secondary sources, clearly written, free from political bias and not plagiarised. When the community argues over which contributions to accept, it uses these principles as the basis for the discussion.

In the SDUI example in the earlier ‘Gathering information’ section, the crowd was told to pick the articles that it thinks will be most useful to readers in each language. While that principle leaves the decision to the crowd, it also removes other criteria for making decisions. The crowd doesn’t have to consider whether the article is important to the English audience, whether it has good illustrations or whether it is prominently featured. The crowd members make their decisions based on their opinion of their own audience.

When you’re preparing rules to help organise the crowd, you may also want to recruit a few lieutenants to help and direct the crowd. Your lieutenants may help recruit members of the crowd or help parts of the crowd to focus on their mission. If the crowd is gathering information, these lieutenants may help to organise parts of the crowd and encourage them to look for information in specific areas. If the crowd is making decisions, these lieutenants may help some parts of the crowd to focus on your question and try to answer it.

Motivating the crowd

When you’re creating a self-organised crowd, you need to find a good way to motivate the crowd in order to encourage them to participate. Indeed, much recent academic research on self-organised crowds has suggested that a good plan to motivate the crowd is essential to success.

If you’re engaging a simple crowd in which everyone plays the same role, the task of motivating the crowd is fairly easy. You run a contest and reward the winner (pay-one model) or you reward everyone who participates (pay-all).

You usually have simple crowds when you’re using them to gather information. When you’re soliciting improvements for a consumer product, for example, you’re dealing with a simple crowd. In this case, you can easily use a contest model (see Chapter 5).

You can also find simple crowds when you make decisions. When you’re asking a crowd to vote on different alternatives, you usually have a simple crowd. You can draw people to vote by giving them something if they vote or by entering them in a lottery to win a prize.

You have complex crowds when different members of the crowds have different roles. In these crowds, you often ask some members to guide, manage or report on other members of the crowd. In the SDUI example in the earlier section ‘Gathering information’, the translators have a complex role. They’re not only providing information, but they’re also disseminating material to others. In such cases, you need a more complex reward, one that reflects the nature of the crowd members’ role. They may need both a fixed reward for each article they complete and small rewards that depend on the number of people who use the material.

Rewards to individual members of the crowd should reflect the kind of role that they fill. Many crowdsourcing activities use a star or badge system in which a member receives a silver, gold or platinum badge when he makes sufficient contributions to the project. The badges should reflect the principles that you set for the project – the goals that you think are important.

For more ideas on how to incentivise work, visit Chapter 14.

Looking at the results

You need to look at the results of any self-organised crowd activity carefully. You probably need to spend extra time with the results of self-organised crowds, because you’ve given the group the freedom to make the decisions. The information that the crowd sends reflects not only the project’s goals but also any principles that you gave it, the information that the members have been processing and perhaps other ideas that they’ve discovered in the course of their work.

For example, many organisations try to use a simple version of Wikipedia’s methods to create employee handbooks. They install some wiki software and tell their employees to start entering information. Often, they look at the results a few weeks later and find little that’s useful. The wiki will be incomplete. Some entries will be far too detailed, whereas others will describe nothing that’s of any use to anyone.

In looking at the results from this kind of project, you need to recognise firstly that you’re doing self-organised crowdsourcing. You may be using a private crowd, the crowd of your employees for example, but you’re still doing self-organised crowdsourcing for which you should have provided a clear goal, motivation for the crowd and principles for making decisions.

If some of the information you’ve received isn’t useful, revisit your goal. Check that you explained to your crowd what you plan to do with this employee handbook and why you think it’s important. If you didn’t do so, you allowed the crowd to set the goal for you. Some crowd members may have thought that you are merely being organised, but others will have concluded that you are planning to replace them and that you want to document their jobs. Perhaps some members of the crowd may have become fearful that you’re planning on outsourcing the entire office to a small island off the Russian coast. Don’t laugh – if you don’t explain your goal, you allow the crowd to explain it for you, and you’ll see the crowd members’ decisions in the work produced.

If the work you receive is incomplete, you’ve not given the crowd enough motivation to do the work. The crowd members are doing the work as an extra task, as a burden. Some will try to do it quickly and with minimal effort. Others will try to avoid it altogether. So give them a motivation: two hours off on a Friday afternoon of their choice; a free lunch to the author of the best entry. The crowd won’t work unless it feels it’s getting something out of the deal.

If the entries are of uneven quality, then you’ve not given good principles for decision making. You need to say at the outset why the results are needed, what information they should contain and, most importantly, what rules the crowd should use in writing the entries. A good way to enforce these principles is to let the crowd members edit the documents, and to give the entire crowd a reward if the work is done by a certain date.

Organising a Prediction Market

Among the many forms of self-organised crowd is the prediction market. Prediction markets encourage the crowd to make a judgement about something that may happen. They operate like a stock exchange. A prediction market offers the opportunity to invest in different events that may or may not occur in the future. Each member of the crowd brings money to the market. Sometimes this money is real, sometimes it is a fake currency distributed by the prediction market. The crowd members invest this money, real or fake as the case requires, in some outcome. You learn from the results of these investments.

Prediction markets do only one of the three things that self-organised crowds can do: they make decisions. Using them to gather information or do work isn’t easy. You can almost always find something simpler than a prediction market if you’re trying to gather data or complete a certain amount of work.

Prediction markets, however, are good as tools for combining the knowledge of the crowd. They’re especially useful for combining knowledge that people can’t easily articulate. People think that some event is going to happen, but they can’t quite say why it’s going to happen. All they can say is ‘I feel it in my gut’ or ‘I bet this is going to be the result.’ In a prediction market, the crowd members translate these feelings into monetary statements.

Prediction markets operate a little differently from the common forms of crowdsourcing and also from many forms of self-organised crowd. In most forms of crowdsourcing, you use the market as a means of communicating with the crowd and getting useful work from it. In prediction markets, however, you establish a market for the members of the crowd to use.

Running a prediction market is perilously close to running a gambling operation. Before you start a market, check local laws to see whether you can legally run such a market. If you can’t, you may discover that you can run a market for a private crowd such as the employees of an organisation. Alternatively, you may try to run a simulated prediction market that uses something other than money as its exchange currency. Just as generations of card players have gambled with cigarettes or chocolate, you may be able to substitute something else for cash.

Even if prediction markets are legal in your jurisdiction, you need to proceed with caution. Many people engage in markets solely to make money and freely do anything they can to manipulate the market in their favour. They can spread rumours or engage in trades that mislead the other participants.

If the prediction market appeals, read on to see how you can organise one.

Finding prediction markets

Because public prediction markets can be considered as something close to a gambling establishment, they aren't common and are usually highly regulated. In the USA, the most sophisticated of these markets is the Iowa Electronic Markets site (http://tippie.uiowa.edu/iem), which is run as a research activity by the University of Iowa. The public can participate in the markets that are used for political predictions. The site does run other markets that predict the state of the economy, but these are open only to students and researchers.

In the UK, the gambling establishments offer public political prediction markets that are similar to the political markets of the Iowa Electronic Markets.

You can create private prediction markets by using market software from companies such as Qmarkets (http://prediction-markets.qmarkets.net). This software handles all the transactions of the prediction market. You have to design the market, recruit the crowd and analyse the results.

Establishing the rules

You can create many different kinds of prediction market. Two of the simplest, and hence the most common, are the futures market and the pari-mutual market.

Futures markets

In the simplest version of the futures market, you create a set of futures. Futures are nothing more than certificates. On a certain date in the future, if a certain event occurs, you will pay the people who own the certificates $1 or the equivalent currency. You will pay nothing if that event doesn’t occur. For example, to predict a presidential election, you can offer a set of futures for each different candidate that are worth $1 if that candidate wins.

If you’re trying to predict the behaviour of a business competitor, you can offer futures for the time when the competitor announces a new product. You can offer a set of futures that are worth $1 each if the product is released in the first quarter of the year, another set that are worth $1 each if it comes in the second quarter of the year, and so on.

If you’re trying to understand the markets for your products, you can also offer a future that’s worth $1 if the sales team reaches its quota and is worthless if it doesn’t. (You can also offer a future that has a value tied to the quantity of sales. It may be worth $1 for every $500,000 in sales. This kind of future is a more refined tool, but is also more complicated.)

For all the simple $1 futures, the price of the future, when divided by $1, is the crowd’s estimate that the event will occur. For example, if the crowd members pay $0.75 for futures that are worth $1 if your company meets its sales projections, then they’re 75 per cent certain you will make those projections.

The crowd can never be more than 100 per cent certain about any event in a prediction market, because members will never pay more than $1 for a future that can only be worth $1.

Pari-mutual markets

Pari-mutual markets are like the kinds of betting markets you find at horse tracks. In this style of market, you don’t issue futures. Instead, you ask the crowd to invest in one of several possible outcomes. The amount that the crowd bets on any outcome is proportional to the crowd’s estimate of the probability that the event will occur. This kind of market is just like bets placed on a football tournament. The bets placed on any team represent the trust that the crowd has in that team.

To use a pari-mutual to compute the crowd’s estimate of the chances that an event will occur, you first sum all the investments placed in a market. You then divide the amount invested in each outcome by the total outcome. Figure 9-1 shows how to compute the crowd’s estimate of the chances for three candidates in a general election.

Figure 9-1: Pari-mutual estimates for a general election.

Laying down the rules

The basic rules for a prediction are simple. A common set of rules is as follows.

You start by giving all members of the crowd a certain amount of fake money and a certain number of futures that say a certain amount of some event will happen. You tell the crowd members that they can buy futures from or sell them to any other member of the crowd in a simple transaction. A simple transaction is one that exchanges futures for money with no complications of any other rights. You don’t promise to buy options at some later point in the market. You sell rights to futures you don’t have.

(Of course, if you have experience of stock markets and want to try more complicated rules, you can, although it isn’t clear that more complicated transactions are worth the trouble.)

As you prepare your market, you can consider a couple of other issues:

Time span: Be sure to include when the market starts and when the market ends. When you run a prediction market, you need to be conscious of time. First, you need to let the market stay open for an extended period. If you run a market for only 10 or 15 minutes, you’re really doing nothing more than taking a poll with money, the crowd merely stating its initial opinion just as if you asked the members to vote on the different alternatives. However, as you keep the market open, the members of the crowd begin to think and to interact with each other. They see what other members have concluded and adjust their opinions. As they make these adjustments, the market begins to reflect the combined intelligence of all individual members.

You want to take several snapshots of the market. You will discover something about the crowd’s perceptions by seeing how it revalues the different options at different times. When you open the market, the crowd members will value the different options in one way. An hour later, they will change their values. When you close the market, the crowd may have different opinions. You may find it useful to see how the crowd changes its mind. It may decide quickly on one conclusion and hold it for the entire market. Or it may start with one idea, move to a second and then return to its original conclusion. The crowd may drift through several ideas before it settles on a final choice. Each of these paths tells you something about the crowd members’ opinions.

  Stating or not stating the question: In the rules, be clear about the information you want to gain from the futures market; however, you may not want to give your goals to the crowd. Sometimes, when you tell the crowd the kind of thing that you’re trying to obtain from the market, the members adjust their actions to try to give that information to you. But you want the members to follow their natural inclinations. Just give them the rules of the market and let them follow those rules.

As with all forms of crowdsourcing, don’t change the rules in the middle of a project.

Assessing the result

After you close the market, you need to assess the results from the activity. This work isn’t as straightforward as it may seem. Sometimes prediction markets can seem to be inconsistent. In a futures market for political candidates you may have two similar Tory candidates and you may discover that their options are similarly priced. The crowd may tell you that it believes that one candidate has a 65 per cent chance of winning and the other a 60 per cent chance – a grand total of a 135 per cent chance between them. Here, the market has combined the two candidates into one. The members of the crowd are really purchasing future options on a single kind of Tory candidate.

Prediction markets often deviate from the strict laws of probability. If you sum the chances that all the events will occur, you obtain a probability greater than 100 per cent, which doesn’t work with the standard theory of probability and chance. However, you can always judge relative strength from these markets. You may not get a perfect estimate of who’ll win, but you can judge that someone is twice as likely to win as someone else.

When you run a prediction market, remember that the crowd is telling you something, but maybe not in a language that’s as clear as you’d like.