Whenever you create something valuable, someone is going to try to make a buck off of it. Intelligent Systems are no different. If you spend energy, money, and time attract users, someone is going to try to make money off of those users. If you build a business that is putting pressure on a competitor, someone is going to try to make it harder for you to run that business.
These are some common ways
that abuse can affect an Intelligent System:
- Abusers try to monetize your users, for example by spamming them.
- Abusers try to steal information about your system or users, to copy or sell.
- Abusers try to use your platform to host attacks on other systems.
- Abusers try to poison your system so it doesn’t perform the way you want it to.
Some of these activities are illegal, but some of them aren’t. And even when the activities are illegal, the global nature of the Internet makes it very hard to find the attackers and even harder to get them prosecuted.
Because of this, all successful Intelligent Systems need to be prepared to defend themselves from abuse.
This chapter explains the basics of abuse so you can understand the challenge, be ready to identify abuse when it happens to you, and have some tools to help make your Intelligent System harder to abuse.
Abuse Is a Business
The first thing to know about abuse is that it is a business—a big business. The vast majority of people carrying out abuse are doing it to make money (although a bit of abuse is carried out for fun, for social justice, or to support espionage). Some of the ways abusers can make money include these:
- Driving traffic : By tricking your Intelligent System to show users things that the abuser wants them to see. This is essentially advertising. The abuser gets a deal with a web site, and gets paid for every user they direct from your site, to the target website. This is often called “spamming.”
- Compromising personal information : Including social security numbers, contact information, passwords, banking information, and so on. Abusers can use this information directly or resell it for a quick profit to other abusers.
- Compromising computers : By tricking users to install bad things on their computers. Once they have bad things on a user’s computer they can steal personal information or use the user's computer to launch further attacks. When abusers can use your Intelligent System to communicate with users, they can trick them to do all sorts of crazy things.
- Boosting content : By tricking your Intelligent System to behave in ways that they want. For example, an abuser might put fake reviews on a product in an e-commerce site to make the product more prominent and sell more.
- Suppressing content : By trying to hurt your Intelligent System or by trying to harm other users of your system. For example, an abuser might report a competitor’s content as offensive.
- Direct theft : Of content, perhaps for resale, such as stealing digital content in an online game.
Abusers have created markets for all of these things (and more), so it is very easy for any abuser who finds a way to do things like these on your Intelligent System to turn that activity into money.
Abuse Scales
Imagine finding an activity you could do to earn a tenth of a penny. Click a button, select an option, click another button—Bam! A tenth of a penny shows up in your bank account.
It sounds pointless. You’d have to do that activity a thousand times just to make a dollar, a hundred thousand times to make a hundred dollars. What a waste of time!
But now imagine you can program a computer to do it for you, and the computer can do it a million times an hour, every hour, for the rest of eternity. This is Internet
abuse. Generally Internet abuse involves an activity that is very unlikely to succeed (like tricking someone to give up a password), or is worth very little every time it does succeed (like sending traffic to a web site)—but the abuser does these-low value activities over and over and over and over. And they make good money doing it.
What this means is that you may not have an abuse problem one day. You think you’re fine, but an abuser might be experimenting, trying different activities, measuring how often users fall for their scams or how much traffic they can produce off of your users, doing the math—and when they find math that is in their favor they can scale up, quickly.
It’s easy for abuse to go from zero to disaster overnight.
Estimating Your Risk
Your Intelligent System will be interesting to abusers if any of the following are true:
- It has a lot of users : As your Intelligent System gets more popular it will have more users. This means abusers can scale their attacks further and make more money per attack. It’s worth their time to experiment against your Intelligent System because if they find a way to make a tenth of a penny, they can make a lot of them.
- Abusers can use your system to communicate with users: Particularly if they can put a URL in their communication. The communication can be a message, an email, a web site, a picture, a review, a comment—anything. Abusers will find ways to make money off of communicating with users by tricking them and spamming them.
- It interacts with user generated content : And the intelligence plays any role in deciding which content to show, how to annotate or display the content, or how to order the content it does show. Influencing how users see content will influence which content they engage with—and how any associated money will flow. Abusers will try to get inside that kind of loop.
- The mistakes it makes cost someone money: Particularly if the costs can be directed toward specific parties. For example, when the smart toaster burns a particular brand of freezer-tart; or when your Intelligent System is putting the non-intelligent competitor out of business.
- It does any other thing a dedicated mind can make money off of: Think of abusers as smart hackers with dubious morals and a lot of time on their hands. Be prepared to be amazed by their ingenuity.
What an Abuse Problem Looks Like
When abusers scale an attack against your Intelligent System they will create odd usage patterns. You can often spot them in telemetry
by looking for:
- Large groups of users who use your Intelligent System in degenerate ways (very focused on the parts that make them money).
- Contexts that see a spike in activity compared to normal usage.
- Contexts where the distribution of outcomes changes drastically (because abuser are sending you incorrect information).
- Patterns in user complaints and problem reports.
Abusers may try to blend in, but they will find it hard to match your legitimate user’s activities, so you can usually spot their attacks if you spend the time looking for them. You can also usually spot their attacks retroactively by setting alerts for drastic changes. Traffic to a part of your system goes up by a huge amount? Someone should know. Complaints double? Someone should take a look.
Ways to Combat Abuse
If abuse does become a problem, some approaches include these:
- Add costs to your product.
- Becoming less interesting to abusers.
- Machine learning with an adversary.
- Get the adversary out of the loop.
Add Costs
You can stop abuse and make more money! Woo-hoo!
Well, charging more might scare away your legitimate customers too, so it may not be an option. But keep in mind that abuse is a business and your best way to stop abuse is to make it harder for abusers to profit.
For example, profiting from abuse will get harder if an abuser needs to:
- Pay 10 cents per account on your Intelligent System.
- Type in some squiggly characters for every review they want to leave.
- Buy a smart toaster for every attack they want to launch.
These costs are most effective when they impact abusers more than they impact legitimate users
. For example, if each good user has to type in squiggly characters once, but the abuser needs to type them for every activity they do. Done right, adding cost might put abusers out of business, without legitimate users even knowing it is happening.
Becoming Less Interesting to Abusers
You could change your product to do less of the things abusers find interesting, for example:
- Removing or restricting communication channels.
- Storing less personal user information on the site and being careful about where you display it.
- Reducing the effect of user feedback on how content is presented.
These may also make your product less useful for real users, but sometimes a small tweak or two will make all the difference in breaking the ability of abusers to profit.
Machine Learning with an Adversary
You could use machine learning to identify abusive interactions and then delete them. I mean, by this point you’re probably thinking: “This is a whole book about machine learning, so machine learning must be the right way to stop abuse, right?”
Unfortunately, not exactly. Machine learning is a fine tool, but abusers are very good at changing their attacks in patterns that fool machine learning. And it usually costs abusers much less to change an attack than it will cost you to chase them and change your machine learning.
You can do machine learning to combat abuse, and it will probably help, but I recommend you consider other options first and make sure you understand how machine learning can actually impact an abuser’s business model before investing too much.
Get the Abuser out of the Loop
Whenever you identify abuse, you should block everything the abuser used to launch the attack, including the account, the toaster, the pellet griller, the funny web site, the sprinkler system—all of it. This will ensure abusers have to pay the most cost possible as they scale their attacks. The infrastructure they used to attack you yesterday is burned, so they need to go out and rebuild the infrastructure today.
Another option is to focus on creating your intelligence only from trusted users. Imagine you have 100,000 users who’ve been with you for years, using your intelligent service, producing telemetry and contexts with outcomes for training. These users are pretty safe—they aren’t accounts created by abusers to launch some new attack. They are your customers. By restricting intelligence creation to “known good” users, you can often avoid abuse completely
.
Summary
Whenever you create something valuable, abusers will come and try to benefit from your hard work, putting your users and your Intelligent System at risk.
The vast majority of abuse is done to make money. By understanding how abusers make money, you can control how interesting your Intelligent System is to them. Often the cheapest way to fight abuse is to make a few small tweaks in the way your system works so that abusers can’t figure out how to make a reliable profit.
Abuse usually targets low-value activities that can be scaled dramatically—a tenth of a penny, one million times a day. You can often see abuse in telemetry as spikes of activity that doesn’t match your regular usage patterns. You may not be able to stop it in real time doing this, but you can usually know if you are under attack.
Some practices can discourage abuse:
- Increase the cost of doing abuse.
- Change your Intelligent System to be less valuable to abusers.
- Use some machine learning (but be careful—abusers have the upper hand here).
- Trust your established users more than you trust new users, and delete all users who are involved in confirmed abuse.
For Thought…
After reading this chapter, you should:
- Know what abusers are and what they do.
- Be able to identify easy changes that will make your Intelligent System much less interesting to abusers.
You should be able to answer questions like these:
Consider your favorite Intelligent System.
- What is one simple change that would make it much more interesting to abusers?
- What is one simple change to make it less interesting to abusers?