Je ne parle pas français. That's how you say I don't speak French in English. Just about two years ago, I found myself in Paris, speaking almost no French. I had read a book and listened to some DVDs before I went, but even after a few months of practice, my mastery of the French language was pretty much pathetic. Then, on the very first morning of my trip, I woke up and walked into a nearby boulangerie (a French or French-style bakery) for my breakfast and morning coffee. I did my best at Bonjour, parlez-vous anglais? They didn't speak a bit of English, or perhaps they were enjoying my struggle. Either way, when my breakfast depended on my mastery of French, I was more motivated to struggle through Je voudrais un pain au chocolat (translation: I would like one of those delicious chocolate bread things) than I had ever been. I was quickly learning to map between English sequences and French sequences, driven by the ultimate cost function—my stomach.
In this case study, we're going to teach a computer to speak French. In a few hours of training, this model will be able to speak French better than me. That's pretty amazing when you think about it. I'm going to train a computer to take on a task that I myself can't do. Of course, maybe you do speak French and this doesn't impress you very much, in which case I'll quote the famous American actor Adam Sandler as Billy Madison: Well, it was tough for me, so back off!
Much of this example comes from, and is inspired by, a blog post by Francois Chollet titled A ten-minute introduction to sequence-to-sequence learning (https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html). While I doubt I can improve upon this work, my hope in using this example is to take a slightly longer than 10 minute look at sequence-to-sequence networks so that you have all the understanding you require to implement your own.
As always, the code for this chapter can be found in the book's Git repo, under Chapter11. The data that you'll need for this example is found at http://www.manythings.org/anki/, which archives many datasets of bilingual sentence pairs, which we will talk about in detail shortly. The file I'm going to be using is fra-eng.zip. It's a collection of English/French sentence pairs. You can easily pick another language if you want, without much modification.
In this case study, we will build a network that can learn a French sentence given some English sentence. This will be a character-level sequence-to-sequence model with teacher forcing.
What I hope to end up with is something that looks a great deal like a translation service you might find on the web or download to your phone.