9. Trapping Tags and Conversations

In the past few chapters, we’ve spent a lot of time looking at various types of resources you can trap, such as news, government, multimedia, and more. And one thing I’ve repeated constantly is that if you want to generate queries that are as narrow as possible, you must avoid being general; get specific by describing the topics you want to track as minutely as possible.

For one chapter—and only one chapter—I’m going to ask you to put that advice aside.

Why? Because this chapter looks at how to monitor and trap tags and conversations. Tags are extremely general, and conversations have their own idiom of searching. For each of these two types of searches, you have to take a different perspective for your searching.

But it’s worth it! Conversations are great for finding technical support, news, and rumors about your topic of interest, and discussions of fitness of one item over another. Like blogs, conversations are also useful for quickly finding discussions on current events. Tags are part of group-developed “folksonomies” and can be an easy way to find manageable numbers of resources about general topics. I do tag searches for things I’d never dare to use Google or even Feedster for.

This chapter looks at where to find tags and conversations, and how to trap them. It also provides some query-building advice so that you can get out of the “narrow, narrow, narrow, specific, specific, specific” groove for just this one chapter.

The Tao of Tags

Tags are a relatively recent phenomenon on the Web. Of course, describing a resource using keywords has been happening almost since Day One. But having keywords used by a group of people gathered into a single folksonomy and then using that folksonomy as a browsing tool for the entire site is relatively new.

A Folksonowhatnow?

A folksonomy, as we’ve already discussed, is like a taxonomy, a structure of organization imposed upon a collection, whether that collection is books, animals, minerals, or something else. A taxonomy is generally developed by a specific group with an idea in mind of what’s going to be classified. Everyone who works on organizing the collection works within the taxonomy. For example, the way the Library of Congress organizes its collection is a huge taxonomy designed to encompass millions of subjects. It’s created and regulated by a specific group. As a result, I can’t approach the Library of Congress and tell them I’ve created 500 subject headings about Kool and the Gang that I want added to their taxonomy. Well, I could, but it wouldn’t happen.

However, a folksonomy is much more organic than the efforts of the Library of Congress. Like the Library of Congress, it’s developed by a specific group, but it’s the group that’s using the collection and usually anybody can join that group. Unlike the Library of Congress, it isn’t developed and then imposed upon a collection, but grows as more items are added to a collection. Tags can have formal language, informal language, or words that aren’t in any dictionary you can find.

This level of flexibility might drive a reference librarian crazy, but it’s ideal for the Internet. After all, the Web is not a library. Pages and data are being added constantly, there’s no card catalog that covers all the content, and there’s no governing body that approves each page as it’s added to the Internet and makes sure it’s plunked into its proper classification slot. If groups of users who add to a collection can be involved in creating some kind of structure for it, so much the better for the users and for you, the trapper.

The advantages of tags...

Tags have several advantages:

• They’re created by people who are contributing to a collection of resources. Because of this, they tend to be more relevant.

• They use mostly general words to describe the contents of the collection. For instance, a page about bookmarks of Google resources might be tagged Google Tools Bookmarks. So for once you can use general searches to find the topics you’re looking for. If you’re trying to monitor a very general topic, like Yahoo or Google, you’ll find sites that use tags invaluable.

• Resources using tags generally have visual displays of the tags being used and how popular they are, which can help you in deciding which words to use for their queries.

But while these are definite advantages, there are also some disadvantages to using tags as well.

. . .and the drawbacks

I suspect when tag sites become really popular, we’ll have to watch out for spammers trying to make sure their resources are listed with every conceivable relevant keyword (and maybe even a few not so relevant). Because you don’t trap tags by date and multiple people can bookmark the same resource, you sometimes see “waves” of a resource—several people bookmarking it over a multiple week period. So it can show up in your traps over and over and over again. (Sometimes I use this as a benchmark of how popular a resource is.)

In addition, because tags are geared toward very general searching, the mechanisms for searching with them can sometimes seem woefully limited. For example, you can only search for one tag, and you can’t exclude or add tags to your search. So sometimes you’ll find that you’re doing general searches whether you want to or not.

Resources That Use Tags

There’s more being tagged than just Web sites—there are photographs, blog entries, and even life goals! We discuss these later in the chapter, but for right now let’s start with the sites that tag general links.

Tagging seems to have started with social bookmarking, which involves a group of people on the same site tagging their bookmarks and then describing them with a few words. Users could then explore who was using the same tags, which bookmarks in the group were most popular, and so on. Get a critical mass of people doing that, and voilà, you have a large set of resources and a folksonomy. The grandpa of this kind of site is del.icio.us.

General tag sites

del.icio.us

If you’re a longtime search-engine user, del.icio.us (del.icio.us) will drive you a little crazy at first, but I promise you’ll get used to it. And it’s worth it. Instead of snippets and lots of information about search results, you often get just a resource name, relevant tags, and the time and date it was added. Sometimes you get an additional line of description. You’ll see what I mean when you visit the front page of the site.

The reason it looks that way is because del.icio.us (which I call Del, for short) is basically a huge bookmarks depository. When people add something to their Del bookmarks, it’s so they can find it later, and less so they can learn about cool new sites. However, I find that the resource titles are generally enough to give me an idea of what they’re all about. I also find that I can get a heads-up on resources here that I don’t normally find anywhere else.

On Del’s front page (Figure 9.1), the most recent resources are added. On the right side is a list of the most popular tags.

Figure 9.1. Popular and recent additions to del.icio.us.

What you will not see is a search box to find tags. That’s okay. To find items tagged with the words you’re interested in, use the following URL and replace keyword with the tag you’re interested in:

http://del.icio.us/tag/keyword

You have to get a little creative. First put aside all the really narrow queries and do those general searches you’ve been dying to do. For example, tagging services like Del are the only places where I trap for the extremely general tags "Google" and "Yahoo", and they work very well!

If you find that your searches aren’t returning many results, get more creative. Since tags are often used one at a time, someone might create a tag by putting two words together.

For example, "GoogleAPI" is a legitimate tag in the Del system, as is "weightloss". If what you’d like to search for is usually expressed as a phrase, search for it as one tag. And as you can see from the Google example, you can also take a common word and experiment to see if it has any derivatives.

You can also do multiple tag searching as well. Just put both tags in the URL separated by a plus sign (+). So entering the following URL locates resources that have been tagged with both Yahoo and Google:

http://del.icio.us/tag/yahoo+google

If you’re having trouble deciding what to look for, do a couple of searches for tags that generally describe your topic. Notice on the right of the results page a list of related tags that may help prod your brain.

Del tag searches are simplicity to trap. Notice at the bottom of each page of search results a familiar orange logo. That’s a link to the RSS feed for this tag search. There are two things you have to watch out for when trapping with del.icio.us. Because Del searches are so easy to monitor, and it’s so easy to do the general searches that are difficult to do elsewhere, you might find yourself tempted to generate and gather lots and lots of RSS feeds. Before you do that, take a close look at the results pages for your searches. Each item includes the date and time it was added, and the results are listed newest first, with ten to a page.

So it’s easy to look at the tenth item of the page and see how active the feed is. Was the tenth item added a week ago? Great—this is an active but slow feed. Was the tenth item added a year ago? You can still add the RSS feed, but you might not see updates very often, which is okay. Was the tenth item added half-an-hour ago? Not so good—you can still monitor this search term, but you’ll have to keep up with a very busy RSS feed.

The second thing you have to worry about is resource repeats. When I’m monitoring the Google tag, I might see the same resource two or three times in a week, especially if it gets mentioned by a popular blog or directory. I don’t worry about this—the repeats are few enough that they don’t hinder the flow of information, and they’re usually described well enough that I don’t waste my time visiting a site I’ve already visited.

With del.icio.us being so popular, it’s easy to realize that there would be lots and lots of other similar bookmarking services. One of them, Spurl, has taken its bookmark service and turned its data into a search engine called Zniff. I like it as a complement to Del.

Zniff

At first glance, Zniff (zniff.com) looks like a regular search engine. But it’s indexing the bookmarks generated by people who use Spurl, a bookmark service like del.icio.us. Zniff also searches a collection of eight million Web pages from Iceland!

Notice that the dates on the results you get are not recent. I suspect there’s some kind of lag between users entering items into Spurl and the items making it into Zniff. Because you can’t get a sense of how “busy” a tag is, you have to do some experimenting. And hey, while you’re at, it, check out Zniff’s help file. Unlike many tag searches, it lets you search for phrases, do “or” searches, or even do stemming searches. (A stemming search is when you search for part of a word and add a wildcard character to find the rest of a word—so searching for moon* would find moonlight, moonlight, moons, moonpie, and so on.) While it helps to be as specific as possible in your searches, you can take advantage of this advanced syntax to try to coax out a little additional information from your traps.

Zniff and Del aren’t the only general tag search resources. Heck, they’re hardly even the start. If you want to do some more searching through general tag sites, try these sites:

• BlinkList (blinklist.com). A newer resource, BlinkList offers both RSS feeds of search results and suggestions of tags you might want to search for as you are typing in your searches. There aren’t as many results as you’ll find on Del, but it’s newer.

• RawSugar (rawsugar.com). RawSugar is another new tagging resource, but it offers suggested links and related links, as well as a list of the users which have used the tag for which you’re searching and what other tags you’re using. As you might imagine, using RawSugar could turn into a huge time sink, but on the other hand, it points you to other tags you might never have thought about.

Access to RawSugar’s Showcase section is free, but to do more searching you have to register. Registration is free. Make sure that you’re searching the full directory of sites, instead of just RawSugar’s Web 2.0 Blogs search.

Tags for specific kinds of searching

The aforementioned tools generate very general sets of data. Of course they do, they’re bookmarking tools—by design they’re repositories for general information. But tags are used in other settings as well. One of my favorites is Flickr, which tags photographs, but there are other ones too.

Flickr

Before we get into Flickr (flickr.com), I am warning you: keep in mind that you’re trying to do research here. Do not get distracted by the pretty pictures. Flickr can turn ten minutes into two hours. If this chapter just kind of trails off in the middle, send out a search party.

I’m sure my intrepid editor won’t let that happen. Flickr is a photo-sharing site that also lets users tag their photographs. When searching for photographs, it’s my favorite place. It’s updated very frequently, the quality of the materials on the site is good overall and non-spammish, and it provides an amazingly quick reference to pictures of current events. Name an event, and it seems like someone’s there snapping pictures, ready to put them on Flickr.

The front page of Flickr shows you recent photos and news, but the Tag Search page, at flickr.com/photos/tags/ is where you want to be (Figure 9.2).

Figure 9.2. Popular tags at Flickr.

You can use the Advanced Search page to search for multiple tags, or for keywords in the tag and in the picture description. However, most of the time I find that searching tags works fine.

You might think that, as an information trapper, you won’t find anything really useful on a photo site like Flickr—you may be thinking it’s only about birthday parties, bunnies, flowers, and such. But do some experimenting with your searches; I think you’ll be surprised. While most of what’s on Flickr is indeed photographs, other graphics are included too.

For example, if you enter a search for Google, which you might think is a weird search for a photo site, look at the kind of results you get (Figure 9.3).

Figure 9.3. Getting Google-tagged pictures at Flickr.

You get pictures of Googlers and Google-related activities, but you also get screenshots of Google resources, explanatory graphics, and even conceptions of services being offered by Google. And while many of these might be posted on blogs and elsewhere on the Web, they would be extremely difficult to find—what keywords would you use? Flickr’s a nice shortcut.

When choosing tags for searching Flickr, again go for common words. As you can see, Google and Yahoo work well. Try turning phrases into single tags like I suggested doing with Del. Remember the searches we did with Del—googleapi and weightloss? Try those here. Notice that weightloss works, while API doesn’t. Experiment, experiment, experiment!

Flickr is terrific for current events, as I’ve said before. Try searching for minnesota state fair. As of this writing, I got over 700 results!

In late August and early September 2005, Hurricane Katrina devastated the Gulf Coast. As of this writing, over 19,000 photos are tagged with hurricanekatrina, from pictures of candles and prayers, to radar pictures of the hurricane itself, to pictures of people living in shelters.

After chapters and chapters of trying to narrow your search, you may be getting a little frustrated at trying to come up with tags that match your topic. If this is the case, try the Explore section of Flickr (flickr.com/explore/). Here you can explore photos, but you can also use a Flickr feature called clustering, which groups photos and finds other tags relevant to what you’re searching for. Start with the URL flickr.com/photos/tags/keyword/clusters/, and replace keyword with the tags you want to search. Try entering Google again. You get a page that looks like Figure 9.4.

Figure 9.4. Clusters of Flickr tags related to Google.

Notice that the search results are divided into several different sets of tags, some of which you might have anticipated (GMail, Googlemap) and some which you probably didn’t (memorymap, geotagged, mountainview). If you’re having a difficult time finding tags that focus on what you’re looking for, use the cluster feature for some extra ideas. (In addition, you’ll find a list of related tags at the bottom of regular search results, which may help you too.)

Trapping Flickr streams is easy, as RSS feeds are offered for searches. (Look for “Feeds for photos tagged with keyword. Available as RSS 2.0 and Atom” at the bottom of a page of results.)

Would you rather get your alerts by e-mail? Notifyr (notifyr.com/) will send you an e-mail whenever a Flickr page in which you’re interested is updated—this can be any kind of Flickr page—a Flickr user’s page, an individual picture’s page (in case you’re watching for new comments), or a page of Flickr search results. You don’t need to register to use Notifyr; all you need is an e-mail address.

Have you been seduced by Flickr yet? Spent hours and hours looking at sunsets, lightning, and cute kitten pictures? I know. Time to go look at something else. Blogs, for example, have such an extensive presence on the Internet as a whole that it shouldn’t surprise you that there are tag searches for them as well.

Searching blogs with tags

You’ve already seen that blogs in general have plenty of search engines. But in addition to doing regular searches for blogs, you can also do tag searches. They’re like the Flickr searches: sometimes blog tag searches give you a shortcut to resources you would not have found otherwise. Technorati and IceRocket are two major search sources.

Technorati Tag Search

Technorati’s Tag Search page (technorati.com/tag/) starts with a list of “hot tags” and the hundred most popular tags, but if none of those interest you there’s a search form. This is the place to try all those blog searches you wanted to do in Feedster but which got you too many or too many irrelevant results. You’ll see when you get the results back that you’re actually getting results from many different sources.

At the top of the results list are tags related to the one for which you just searched. Keep those in mind for later. (Also note at the top of the Search Results page that there’s a tab for searching photos!) You can get your trap down at the top of the search results where there’s an orange button linking to an RSS feed. Play and experiment with general blog searches here, and then take your experiments to IceRocket.

IceRocket Tag Search

IceRocket is possibly not as well known as Technorati, but it’s got a good blog search (blogs.icerocket.com/tag/). As you can see in Figure 9.5, there’s a special syntax that lets you search for tags.

Figure 9.5. Use the tag syntax or simply pick a tag.

The results are divided by date, giving you clear information at a glance as to how busy a tag is. From the results, you can also exclude individual blogs (for example, if there’s one that doesn’t really meet your topic requirements but is heavily represented in the results). On the other hand, if you find a blog that’s great and you want to see more of it, you can also focus on it and see its entries exclusively. On the right are several links for RSS feeds. IceRocket doesn’t seem to be indexing quite as much as Technorati, but what it does index I think is well worth trapping.

Are bookmarks, photos, and blog search the only things out there that are being tagged? Good heavens no. There is a huge variety of tagged resources available with more being added every day. Here’s a taste of a few more.

43 Things

43 Things (43things.com) is a site that collects life goals for people and where they’re located. Users can track their progress with their goals and support each other. Goals are searchable by tags. Goals include losing weight, paying taxes, watching less television, reading 100 books in a year, and so on. If you’re doing any kind of cultural or current event searching, you should have a ball—and set a few traps—rummaging around in this site.

Connotea

For academics, this site (connotea.org) allows users to tag reference resources and papers. While not as active as del.icio.us, the resources added here are as a whole from more credible sources and are much more academically oriented. Medical trappers definitely need to visit here. It does require registration. If you find useful resources at Connotea try the similar academically slanted service, CiteULike (citeulike.org/).

Shadows

Shadows (shadows.com) is combination of bookmarking application and community. Tagged resources can be sent to groups, and the resources there can be commented upon by users. Nice to see tagged resources and feedback at the same time! Like Connotea, Shadows requires registration.

The general searches which would be just about guaranteed to drive you crazy in searching the general Web and other large data pools are just right when searching tag engines. Even if some of these resources don’t, at first glance, seem like ones that would be appropriate for your topic, at least take a moment or two and try a couple of searches with them. I think you’ll be surprised.

And I think you’ll be equally surprised about monitoring conversations. When I’m trying to keep tabs on certain technical issues for a computer, or opinions on current events, or experiences with medical problems, conversations will sometimes point me toward issues that don’t bubble up in news search engines until much later. There are some things you have to be cautious about when monitoring conversations, but it can be well worth it.

The Zen of Conversation Trapping

Before the Web was all that, the Internet was all about e-mail. And despite the fact that the Web gets much attention, e-mail is still in some ways the killer app of the Internet. Millions of conversations are started, ended, and continued every second of the day on public mailing lists. They cover everything from aardvarks to zephyrs. Some of the conversations are desultory and not particularly useful to you, the trapper, but on the other hand the amount of free technical support, spirited discussion, problem solving, and other data flow can be very helpful. But as I noted earlier, conversation trapping is not without its problems.

The bummer of conversation trapping

The first stumbling block of conversation trapping is credibility. When you’re searching with news search engines you can start out with a baseline of credibility to which you can assign a story. You might, for example, consider the Washington Post a high-credibility source, and give other sources varying levels of credibility depending on how well you know them, what they’re reporting on, and so on.

With conversations, that baseline should be zero, since you can’t assign any credibility to what you’re reading until it’s corroborated or you have some other reason to trust it. That’s not to say a lot of good information can’t be found in online conversations. But you don’t know in some cases who’s posting, what the agenda is, where they got their information, and so on.

Another stumbling block is language. I’ve mentioned earlier in this book that in conversations people tend to be a little sloppier than when they’re writing for a Web page.

Yet another glitch is the fact that information gets old. It’s usually not much of a problem with news stories—when a situation changes, the news story is updated to reflect the new information. Sometimes, however, the updating doesn’t happen, and the same old, outdated information bounces around and around. For an Internet-wide example of this, consider the case of Craig Shergold. Craig had cancer and wanted to receive greeting cards—in 1989. Over ten years later, long after Craig was cancer-free and long after he appeared on TV, letting the world know that the cards could stop, this appeal for cards was still circulating around the Internet and only now seems to be slowed down.

Keep all this in mind as you trap. You may discover over time that there are some mailing lists you trust more than other ones. In reading through the resources presented in this section, you’ll learn there are ways you can concentrate your trapping on one list if that works best for you. There are many places online where you can watch conversations.

Trapping conversations

Generally speaking there are three types of conversations you can track:

• Usenet. Usenet is a collection of thousands of newsgroups set up in a hierarchy that’s been available for over twenty years now. While not as popular as it used to be, it still generates a lot of traffic.

• Mailing Lists. Discussion groups that are distributed to their members mainly through e-mail. As more closed conversations are less subject to the ravages of spamming and trolling, mailing lists have become more popular over the last few years. And many of them have publicly available archives that you can search without being on the list, or without requiring more than a basic free registration.

• Discussion Boards. Discussion boards are areas on Web sites that contain discussions, usually on very specific topics or groups of topics.

When it comes to monitoring Usenet, Google Groups is your best bet. But for the other types of conversation trapping you have lots of options.

Google Groups

Google Groups (groups.google.com) is actually a combination of a mailing list host and a Usenet search service, but the mailing lists have not caught up with the still prodigious Usenet traffic! So we concentrate on that option.

Your first task when preparing to trap Usenet is to figure out where you’re going to trap. Most topics you want to monitor will be well served by limiting the search to a newsgroup or a set of newsgroups.

Usenet’s newsgroups are arranged in a hierarchy—major categories like comp (computers), sci (science), rec (recreation), and so on. (There are a bunch of little categories as well.) Underneath the major categories there are subcategories, sub-subcategories (sometimes even more sub than that), and newsgroups. You can search by single newsgroups but I find it’s easier to search by category.

If you don’t find any major category that really covers what you’re looking for, then you’ve got a couple of options. You can do a full Usenet search (if your search is specific enough this shouldn’t be a big deal) or you can try to find individual groups that cover what you’re looking for. Searching for individual groups also shows you mailing lists that Google Groups is hosting.

The searchable listing of groups is at groups.google.com/groups/dir. Use moderately to fairly general words to do your searching, such as sports, baseball, money, woodworking, autism, or medicine. When you get search results they look like Figure 9.6.

Figure 9.6. Group listings from Google Groups.

Groups will be divided up at the top by category, region, language, activity, and number of users. Then the list of groups will appear below. Group listings include group name, description, and if the group is one of Google’s mailing lists, the number of members.

But let’s start with the categorical searching. Let’s go back to the antique woodworking example. I want to monitor groups for mentions of antique woodworking, and after some browsing I’ve decided that the recreation hierarchy of newsgroups is my best bet. So I search for:

"antique woodworking" group:rec*

The group special syntax lets me restrict my searches to a single group or a category of groups. The rec* searches every group in the recreation category. (You can also search under subcategories—group:rec.sports* will search only those groups in the sports subcategory.)

So what kind of results can you get for this? I got less than 125 results as of this writing, but you won’t be able to tell how recent the newest results are until you go to the switch on the right and choose to sort by date instead of by relevance (Figure 9.7).

Figure 9.7. One page of search results for this Google Groups query goes back several months.

Once you’ve done that, you can view the date of each posts—each post has a date and time—and see how recently the newest post is, and how old the oldest is. In this case you can see that "antique woodworking" isn’t a particularly busy search term and would make a good trap. Down at the very bottom of the page is a signup form for e-mailed search alerts for Google Groups.

In addition to the group syntax, Google Groups offers other special syntax that you can use to narrow your search. Being familiar with Google searching, you should be familiar with intitle, which is used to find keywords in the title. It works the same in Google Groups. Between that and the group syntax, you should be able to narrow your searches quite a lot.

For most of your trapping needs, monitoring the entire Usenet or groups of newsgroups should be sufficient. But there will be times when you find a single group that has so much great information in it that you want to read it exclusively. In these cases, you need to have a Google Groups account.

You may already have a Google Groups account and not know it! If you’ve signed up for GMail then you have a Google Groups account. Just use the same user name and password that you’d use to sign on there. If you don’t you can register. Registration is free.

I don’t think you necessarily need a Google Groups account unless you’re following individual newsgroups. But it may turn out that you do want to follow individual newsgroups—a group may cover your topic perfectly and may not be worth filtering by keyword. There may be several groups that fit your topics this way.

When you first log on to Google Groups your page will look like Figure 9.8.

Figure 9.8. Logging on to the Google Groups account.

There is a Groups Lookup at the bottom of the page where you can search for newsgroups and mailing lists of interest.

So say you find a few newsgroups to which you want to subscribe. When you join them you’re given several subscription options as you can see in Figure 9.9.

Figure 9.9. Signing up for newsgroups.

I always choose to read the newsgroups to which I subscribe on the Web—what’s the use of having a Web-based interface if you don’t use it?—but you may wish to receive regular e-mail or digests.

Once you’ve subscribed to a few newsgroups, reading them via your Google Groups page will look like Figure 9.10.

Figure 9.10. Reading a group in Google Groups.

From there you can read your messages, “star” topics you find interesting (useful when there are long, long discussion threads you want to follow because it gathers them all into one place), and read your messages. And how do you trap the new messages? But subscribing to them, of course!

Google News is a mix of Usenet newsgroups and mailing lists—but really for the most part it’s newsgroups. Yahoo Groups, on the other hand, is an ancient and extremely active set of mailing lists, and a gold mine for information trappers.

Yahoo Groups

You don’t, strictly speaking, need a Yahoo account to read Yahoo Groups (groups.yahoo.com), but I recommend it. Many of the groups don’t let you read the archives unless you’re a member, and by signing into Yahoo you’ll easily be able to sign up for groups. Some groups don’t offer public access to their archives at all. You have to be given permission to join the list, after which you can read the archives. I find there are few enough of those that they don’t hinder my information trapping.

Yahoo’s mailing lists are set up under a sort of hierarchy. Yahoo has set them up into a searchable subject index that’s very Yahooesque. But you may find keyword searching to be your best bet.

The rules for keyword searching for mailing lists here should be much like what you used with Google Groups: moderate to very general words. Because Yahoo Groups offers only mailing lists, it can provide you with a little more information in its search results, including extended descriptions, a count of the number of people who are subscribed to a mailing list, and whether the mailing list archives are accessible to members or not.

When you decide to subscribe to a mailing list and click on the name of the list from the search results, you get a little more information about the group—like how many new members it’s gained and how many links, photos, and messages it’s added in the past week. Pay attention to this! The activity of the list is going to determine your strategy to keep up with it, and how much use you might get out of it. A group with hundreds of new messages a week should be approached with more specific and narrower queries than a group with only a few dozen messages a week.

And that’s how we get to talking about trapping. As you may have already guessed, your strategy with Yahoo Groups is going to be different than your strategy with Google Groups. For one thing, you can’t search the entire body of messages at once—you’re going to have to first identify useful groups, then search them individually. How you go about that depends on the group itself.

Public groups—groups which have no restrictions on membership or post-reading—will offer RSS feeds of the latest posts. The RSS buttons are in the same area as the latest posts (Figure 9.11).

Figure 9.11. Publicly readable groups have RSS feeds available at Yahoo Groups.

These are full feeds of all the posts. I recommend using these feeds in conjunction with something like FeedDigest or FeedShake to extract just the words in which you’re interested. We talked about those tools way back in the early chapters.

When groups are not public and are for members only, then you have to do something else. I recommend using a page monitor. It’s not as streamlined as using an RSS feed, but you won’t have a lot of options here. You can either monitor the front page, which shows you the latest posts (this is okay if you’re monitoring a not-particularly-busy group), or you can monitor a page of search results for that particular group (necessary if you’re monitoring a very busy group). If you want to monitor a page of the latest posts of a group that requires registration before you can read the archives, check your page monitor to make sure you can specify a login name and password when monitoring that page. Most page monitors offer this option.

Though your methods of finding and monitoring information on Yahoo Groups will be different than what you use with Google Groups, the kinds of things you’ll be monitoring should be the same, as we talked about earlier in the chapter. Product names, condition names, model numbers, error messages. Continue the tactic of creating questions relating to your topic, extracting the relevant words, and then plugging them into the search. You may find some mailing lists to be so information-rich you want to completely monitor them instead of just looking at search results. You have ways to monitor both.

Google Groups and Yahoo Groups are without question the largest Usenet and mailing list resources out there. But there are other repositories of mailing lists as well. And though you may not find that you want to monitor every last one of these, they’re worth looking at.

Topica

There are actually two kinds of searches you can do on Topica (lists.topica.com): you can search for lists and you can search for messages.

On the front page is a directory of mailing lists you can browse or a search box. That search box will find you mailing lists. However, once you run the search you get more options (Figure 9.12).

Figure 9.12. Once you do an initial search, you can do a second search for groups or messages containing your keyword.

Yes, you do get a list of relevant groups, but on the right you get another search box with the option to search for more lists or—and this is the important bit for you—search messages!

But let’s look at the groups first. The search results for groups don’t give you as much information as, say, the search results for Yahoo Groups, but if you click on the name of the group in the results you get an additional information page that includes the number of subscribers and the approximate list activity, as well as a link to read the archives if they’re readable by anybody. Use this information to determine list activity and how you want to monitor it.

Back to the message results. Your search results will initially list posts in order of relevance, I believe. Click the date header to resort them by date, as you see in Figure 9.13.

Figure 9.13. Sorting by date in Topica gives you the oldest results first.

When you normally resort items by date, what do you get? You get a list with the newest items first. In this case you get the oldest items first, so what you have to do is scroll to the last page of the results and monitor that. Mostly what you’re monitoring is changes to that page’s content and the addition of another page of search results. It’s awkward, and I only occasionally find useful results here, but Topica is still active enough that I would at least look at the lists it hosts and consider adding it to my information trapping list.

TILE.NET Lists

TILE.NET (tile.net/lists) is not a message repository. Instead it’s an online listing of newsletters and mailing lists. It’s also one of the oldest online resources for this kind of thing, so you know it’s extensive.

You have the choice of browsing by name, description, and domain, which I don’t recommend unless you have lots and lots of free time. A search engine also allows you to search between lists and newsgroups. You probably got your fill of newsgroup searching at Google Groups, so stick to lists. Make your searches single-word—ethics, medicine, woodworking, Google.

The big drawback to TILE.NET is that a lot of the lists have no descriptive information, and you can’t get a sense from the title of some of the lists what exactly they are. However, for when you want to make sure you cover every available inch, and want to set up traps for every available resource, you don’t want to miss TILE. There are too many listings here.

L-Soft List Catalog

Sometimes you’ll hear someone refer to a mailing list as a LISTSERV. That’s not really a correct usage. LISTSERV is a certain type of mailing list software and it’s a trademark. A mailing list may run on LISTSERV, but a mailing list is not a LISTSERV. (It’s like saying, “Give me a Kleenex,” when you’re using a generic brand of tissue.)

You might realize however that mailing list software would have to be pretty darn popular to become a common usage noun. And you’re right. The L-Soft List Catalog (lsoft.com/lists/listref.html) contains information on over 58,000 mailing lists—and those are just the publicly available ones!

There are some unusual ways to find lists here (Figure 9.14)—you can search by country and search by lists that have over 1,000 or over 10,000 subscribers. You can also do a keyword search for lists.

Figure 9.14. Several different list search options at L-Soft.

The list search offers several different options. You can search for keywords within the list name or list description (which is what I recommend) or within the host name. You can also limit your searches to only those lists that have a Web-based archive (useful if you want to review list archives before you subscribe). Beware using the Web-based archive search limitation—a search for autism found 12 results, but when the search was restricted to only those lists with Web-based archives, the same search found only 6 results.

By now you know: there are hundreds of thousands of mailing lists out there. But guess what? That’s not the only way to have a conversation online! In fact, there’s another way to have open online discussions and that’s via online boards.

Searching online conversations

There are software programs (and some hosted services) that allow people to put up online boards or discussion forums. The advantage of those is that they allow conversations to be centered on one site. The disadvantage of course is that unless you publicize the heck out of’em, forums can be hard to find and hard to build traffic for. A secondary disadvantage is that sometimes online forums become irresistible target for spammers and trolls (people who want to start arguments and fights online).

But if you can get past those problems, you’ll find a lot to love in boards. Generally they evolve into a community, so if you find a good community that acts courteously toward itself and exchanges good information, you can anticipate the entire forum to generate good information even as it moves forward into new topics. Generally, moderated boards (forums that have a manager control the postings) are better than unmoderated boards, but in both cases the information quality varies across forums.

I can recommend two places that will help you both find forums and find messages on forums.

Big-Boards

Big-Boards (big-boards.com) tracks and provides information on the largest boards and forums on the Internet. And when I say largest I mean hundreds of millions of posts by hundreds of thousands of users. There are over 1,800 boards listed in this directory. Very information-rich trapping areas, as long as you can keep the information flow to a usable level!

You may browse available forums in a subject index or search by keyword. Each forum listing has an information page that provides a description of the board, as well as a screenshot and number of monitors. Each listing also has a stats page that graphically shows the number of members and the number of posts per day, and gives a text listing of the number of users and the average number of posts per week.

You will not be able to monitor forums directly from Big-Boards, but it will point you to a lot of potential trapping areas.

Yuku

Yuku (yuku.com) is not a board posting search engine. Instead it finds the forums themselves. It’s up to you to go to them and decide what you want to monitor, but Yuku does provide a lot of information for you.

Use general to slightly-less-general search words—think of Yahoo Directory’s top level categories and then maybe a category or so more specific than that. If you get too many results, carefully get more specific. I find it’s way too easy to end up with nothing. Your search results are very informative—you get the name of the forum, description, total visits, total posts, average daily visits, and average daily posts. That should give you enough information to decide if you want to monitor the forum.

What you find when you decide to visit a forum will vary a lot. Some forums require membership to even view posts. Some forums require membership to post, which means you can monitor them without having to have a membership to the board. The Search Results page for a forum search looks like Figure 9.15.

Figure 9.15. 38 average daily posts? You can handle that. 167? You may want to watch specific board sections with a page monitor.

Pay attention to the average number of daily posts! If traffic in the forum is light (less than a few dozen messages a day) you may wish to monitor only this front page. If the traffic is heavier, you may wish to pick a section or sections in that discussion forum that are most relevant to your interest and then monitor those using a page monitor. If you’re very interested in singular topics, you can choose to be notified of topic updates via e-mail—just look for the “Click to receive email notification of replies” link. You don’t have to be registered on the board in which you’re interested to get notification of updates to topics. (This can lead to a lot of e-mail but we discuss that in the next chapter.)

As you can see, searching tags and online conversations is quite different from searching full-text search engines. But if you take the opportunity to use these resources for general searching, you’ll add a whole new dimension to your information trapping.