If there were one course of study that would produce a better Guerrilla Analyst who is ready to work on real-world data, it would probably be software engineering. Many Guerrilla Analytics challenges have already been encountered in a different guise in software engineering. Version control, migration of data, data quality, testing, workflow tracking are well understood and supported by the tools and training of software engineering. Analytics has not yet reached that level of maturity. The approaches in this book are the bare minimum you should do to better control disruptions in the typical analytics project. Much more thinking, training, and tooling needs to be devoted to the equivalent of software engineering for Guerrilla Analytics and analytics in general.

Many of the challenges in Guerrilla Analytics are due to the wide variety of activities and tools required to extract insight from ever-changing data. A significant proportion of this book was devoted to overcoming those challenges with existing technology adapted for Guerrilla Analytics and simple conventions to preserve data provenance. This is good but not enough. The analytics workflow needs to be better understood and supported with tools that embrace its dynamism and disruptions. There is a need for better ways to trace data provenance that do not get in the way of doing analytics.

Data is fundamental to everything that is done in data analytics. In spite of this, the ability to test data and analytics work with powerful tools and well-grounded thinking is almost nonexistent. What is needed is the equivalent of software test frameworks for the five Cs of Completeness, Correctness, Coherence, Consistency, and aCcountability. Yes, these are supported in the fields of data quality and master data management but they do not quite work in the reality of a Guerrilla Analytics project.

I have deliberately avoided mentioning Big Data throughout this book. I made this decision because the definition remains muddled and influenced by both technology vendors and by sales pitches. Perhaps that will change in the near future. Nonetheless, there has certainly been a change in the technologies that permit analytics on large volumes and varieties of data. That will not disappear. This begs the question of whether and how Guerrilla Analytics should change to support Big Data. What I suspect will happen is that the convergence of software engineering and analytics will accelerate as more analytics is done on large powerful software platforms, rather than with a mixed bag of tools borrowed from other fields. The principles of Guerrilla Analytics can certainly help with this, as this scale of data also needs to be explored, analyses need to be version controlled and data provenance will remain of primary importance.