Traditional logging, especially extensively used logging invocations, creates a lot of string objects. Even APIs such as Slf4J that aim to reduce unnecessary string concatenation will result in high memory rates. All these objects need to be garbage collected after their use, which utilizes the CPU.
Storing log events as string messages is a verbose way of storing information. Choosing different formats, mainly binary formats would drastically reduce the message size and result in more efficient memory consumption and higher throughput.
Log messages that are stored in a buffer or directly on disk need to be synchronized with other log invocations. Synchronous loggers ultimately cause a file to be written within a single invocation. All simultaneous log invocations need to be synchronized in order to ensure that logged events appear in the right order. This presents the issue that synchronization indirectly couples functionality that otherwise is completely unrelated. It decreases the parallelism of intrinsically independent functionality and has a negative overall performance impact. With a high number of log messages being written, the probability of blocking threads due to synchronization increases.
Another issue is that logging frameworks usually don't write the log messages to disk directly; rather, they use several layers of buffering. This optimization technique comes with certain management overhead involved that does not improve the situation either. Synchronous file operations advisably work with the least overhead layers as possible.
Log files that reside on NFS storage decrease the overall performance even more, since the write operation hits the operation system I/O twice, with both file system and network calls involved. In order to manage and persist log files, network storage is an often chosen solution, especially for container orchestration that needs persisted volumes.
In general, experience shows that logging has the biggest impact in an application's performance. This is mostly due to the memory impact on string log messages.