As you get to know your IoT data, you will find some fields are more valuable than others. Through your discoveries using techniques from Chapter 6, Getting to Know Your Data - Exploring IoT Data, and Chapter 10, Data Science for IoT Analytics, you will find fields (features) that keep coming up as statistically significant, and some that just never seem to matter.

For older files, you can follow some methods to keep the useful fields, while getting rid of the ones that do not seem to have an impact:

• Transform older data to only keep useful fields: Move older data to either a new file or table but only keep those useful fields. Then, delete the old records.
• Split out useful fields and treat them differently: For older records, you could keep the useful fields in hot areas that are easily accessible, while shipping the less useful fields off into cold storage
• Summarize and remove large data fields such as text or binary files (such as image or sound): As an example, you could reduce a lengthy free-form text field to the count of key word occurrences.