Imagine your application processes thousands of messages every second. In this case, a latency of even one millisecond or more can have a performance impact on your system. The worst part is that you won't even know when your GC will kick in and start the collection.
I have a suggestion; execute your application with Epsilon GC as a benchmark that won't collect any garbage. Now execute your application with GCs of your choice, and analyze the logs. Now you can filter out the latency induced due to a GC—such as GC workers scheduling, GC barrier costs, or GC cycles. The performance of your system might also suffer due to issues that are not related to a GC—such as OS scheduling or compiler issues.