The reason I wanted to talk about the law is that some people, especially beginners, fail to take it into account whenever practicing web crawling and/or scraping. Failing to recognize legal from illegal is very dangerous. Think about it. Becoming an outlaw is not something people are willing to do.

It's impossible to tell whether scrapping is legal or illegal; these things are always changing from time to time and from country to country. Generally, web scraping is not illegal; what you do with that could be. Take time to do some research on your own. Consider not just the kinds of data you are going to retrieve, but also what you're going to do with it. Seeking specialized advice is always the best solution.

Records exist of people being prosecuted or investigated by doing web scraping. Aaron Swartz was. Even if you don't have evil intentions—it's very hard to say that Aaron Swartz did—you should always be careful about what you are doing, especially if it's something disruptive.

Ethics are very important, too. Crafting a neural network that's capable of generating click bait is OK if you only want to practice or learn, but you shouldn't do it if your intention is to put those click baits to use. Moving ahead, it's time to check the gorgeous rvest.