Search engines are one of the great inventions that helped the Internet grow and become what it is today. Any powerful tool can be (and usually is) a double-edged sword. With search engines we the common users, can find almost anything that is of our personal interest, but as a website owner and/or administrator we must know what amount of information is available to the general public. If that amount surpasses our intentional or allowed boundaries, then we must know how to detect such instances and remedy the situation. In this chapter we will focus on the following topics:
- Internet bots—types and dangers
- Protecting Moodle from undesired search bots
- Protection against spam bots
- Protection against brute force dictionary attacks
Internet bots are software applications that run automated tasks applied to any content publicly available on the Internet. Usually they perform tasks that are repetitive and trivial at a much higher speed than any human being can do. We will outline the most common uses here.
This is a task performed by all major search engines. Their bots scan the web all the time reading publicly available content and indexing it for searching purposes. This basically applies to any textual content although multimedia files can also be indexed by their name and description.
These bots analyze public content of web sites and extract any email addresses, thus generating a mailing list of potential recipients. To understand why somebody would need a list of email addresses we need to understand the concept of spam. Any form of unsolicited message received through some electronic service is considered spam. These messages usually contain advertisements for various products or services. The most widely known variant of spam is email spam. In order to distribute these messages spammers need mail addresses. One way of obtaining them is by running email harvester bots.
Web scraping is a computer software technique of extracting information from the websites. That generally means copying and potential transformation of unstructured web content into a more manageable form for some kind of automated analysis. Common uses of web scraping are making a complete offline copy of the site content (stealing course information), obtaining personal user information (name, address, email, etc.) that can later be used for sending spam or for selling it to other mail advertising companies.
Electronic spam has many forms and spammers use any kind of messaging/publishing system to promote their agenda. Internet bots that automatically create spam content are called spam generators. Any blog, CMS, forum, and internal messaging system are a potential target for this kind of attack. For example, if you open your Moodle for self-registration and do not apply all of the protection measures you leave a door open for a spam attack. Specially crafted bot can create several fake accounts and start posting unsolicited messages in all available forums or by using personal messaging system.