Native memory, the way the term is used in this book, means the parts of memory used by the runtime for purposes other than Java heap. This can be space for code buffers or system memory that is "malloced" when the runtime needs to acquire space for internal data structures.
See also Heap.
52.15.55.18