A Neural Turing Machine (NTM) is inspired by the Turing Machine (TM): the fundamental mathematical model of computation that defines an abstract machine. A TM can manipulate symbols on a strip of tape according to a table of rules. Given any computer algorithm, a TM is capable of simulating that algorithm's logic. The machine positions its head over a cell and reads or writes a symbol there. Thereafter, based on the rules defined, it can move left or right or even halt the program. 

An NTM architecture contains two basic components: a neural network controller and a memory. The following diagram shows a high-level representation of the NTM architecture:

The controller interacts with the external world using input and output vectors. Unlike a standard neural network, here the controller also interacts with a memory matrix using selective read and write operations. The memory is a real-valued matrix. The memory interactions are differentiable end-to-end, making it possible to optimize them using gradient descent. NTMs can learn simple algorithms, such as copying, sorting, and associative recall from input and output examples. Also, unlike a TM, an NTM is a differentiable computer that can be trained by gradient descent, yielding a practical mechanism for learning programs. 

The controller can be modeled by a LSTM, which has its own internal memory that can complement the larger memory in the matrix. The controller may be compared to the CPU in a computer, and the memory matrix can be compared to the RAM of a computer.

The read and write heads select the portions of memory to read or write to. They can be modeled by a hidden layer in the neural network—possibly a softmax layer—such that they can be treated as weights on the external memory cell that add up to 1. Also, note that the number of parameters of the model is controlled by the heads, and it does not grow with memory capacity.