Today's project was to build a classifier to solve the problem of identifying specific handwriting samples from a dataset of images. Our hypothetical use case was to apply Deep Learning to enable customers of a restaurant chain to write their phone numbers in a simple iPad application to get a text notification that their party could be seated. Our specific task was to build the intelligence that would drive this application.
Our MLP model accuracy hit 87.42%! Not bad for depth of the model and the hyperparameters we chose at the beginning. See if you can tweak the model to get an even higher test set accuracy.
What are the implications of this accuracy? Let's calculate the incidence of an error occurring that would result in a customer service issue (that is, the customer not getting the text that their table is ready and getting upset for an excessively long wait time at the restaurant).
Each customer's phone number is ten digits long. If our hypothetical restaurant has an average of 30 tables at each location and those tables turn over two times per night during the rush hour when the system is likely to be used, and finally the restaurant chain has 35 locations. This means that each day of operation there are approximately 21,000 handwritten numbers captured (30 tables x 2 turns/day x 35 locations x 10 digit phone number).
Obviously, all digits must be correctly classified for the text to get to the proper waiting restaurant patron. So any single digit misclassification causes a failure. A model accuracy of 87.42% would improperly classify 2,642 digits per day in our example. The worst case for the hypothetical scenario would be if there occurred only one improperly classified digit in each phone number. Since there are only 2,100 patrons and corresponding phone numbers, this would mean that every phone number had an error in classification (100% failure) and not a single customer would get their text notification that their party could be seated! The best case in this scenario would be if all ten digits were misclassified in each phone number and that would result in 263 wrong phone numbers out of 2,100 (12.5% failure rate). Still not a level of performance the restaurant chain would be likely to be happy with.