Visualforce Developer's Guide has a section within it entitled Best Practices for Improving Visualforce Performance (https://www.developerforce.com/guides/Visualforce_in_Practice.pdf), which outlines some excellent ways to ensure that your pages remain responsive. I highly recommend that you take the time to read through it. There is, however, one point in that topic that I want to elaborate on further in this section:

This symptom can result in poor response times, even for pages that are properly optimized in all other regards described in the documentation. The cause stems from the internal overhead of the Salesforce Visualforce components, known as internal view state, which increases when you have many such components on the page, typically in tabular or apex:repeat scenarios.

Depending on your Visualforce page design, it can often represent proportionally more than the data the developer explicitly places in the view state through controller properties and member variables. Together, they both contribute to reaching the 145 KB limit on view state size, though before this is reached, larger view states can affect response times.