In our experience, analytics, which is a fairly recent term compared to well-established terms such as data warehouse and others, requires a careful approach in order to ensure both immediate success and the consequent longevity of the initiative.

Projects that prematurely attempt to complete an initial analytics project with large-scale, high-budget engagement run the risk of jeopardizing the entire initiative if the project does not turn out as expected.

Moreover, in such projects, the outcome measures are not clearly defined. In other words, measuring the value of the outcome is ambiguous. Sometimes, it cannot be quantified either. This arises because the success of an analytics initiative has benefits beyond simply the immediate monetary or technical competencies. A successful analytics project often helps to foster executive confidence in the department's ability to conduct said projects, which in turn may lead to bigger endeavors.

The general challenges associated with Big Data analytics are as follows:

• Nearly every company is investing in Big Data, machine learning, and AI
• Often, the company has a corporate mandate
• Finding the right use cases can be challenging
• Even after you find them, the outcome may be uncertain (will this resonate, how long will it take, and so on)
• Even after you achieve them, whether or not the optimal targets have been identified can be elusive (for example, when using HDFS for storing only data)

Now, let's look at some general guidelines for data science and analytics initiatives:

• Conduct meetings and one-on-one reviews with business partners in the organization to review their workflows and get feedback on where analytics and/or data mining would provide the most value
• Identify specific aspects of business operations that are important and related to the firm's revenue stream; the use case would have a measurable impact once completed
• The use cases do not have to be complex; they can be simple tasks, such as ML or Data Mining
• Intuitive, easily understood, you can explain it to friends and family
• Ideally the use case takes effort to accomplish today using conventional means. The solution should not only benefit a range of users, but also have executive visibility
• Identify Low Difficulty - High Value (Short) vs High Difficulty - High Value (Long) use cases
• Educate business sponsors, share ideas, show enthusiasm (like a long job interview)
• Score early wins for Low Difficulty - High Value, create Minimum Viable Solutions, and get management to buy in before further enhancing the use solutions developed. (takes time)

Early wins act as a catalyst to a) foster executive confidence, and b) also makes it easier to justify budgets, which then makes it easier to move onto High Difficulty - High Value tasks.

The last two points are important as it is essential to identify Low Difficulty - High Value projects. This could be a task that appears basic to an experienced practitioner but is very valuable to the end user.

One of the executives of an analytics group in a large enterprise organization once remarked that the most successful project of the year was the change of timing of an email report. Instead of sending the report in the morning, the timing was changed to late afternoon. It appeared that engagement with the report became more active after the timing was changed. Morning schedules tend to be very busy and afternoon reports, on the other hand, provide recipients with the time to review the report at a more relaxed pace.

A few examples of low difficulty but potentially high value projects could be:

• Automating manual tasks conducted on a frequent basis by a business group; for instance, reports that are created in Excel may be easily automated using a combination of open source tools and databases.
• Converting manual stock analytics to automated versions using programming scripts. This could involve tasks such as creating regular tables, pivot tables, and charts that are created in Excel but can be converted into automated processes.
• Creating web interfaces using R Shiny for business applications and implementing predictive analytics functionalities.
• Moving certain parts of the IT infrastructure to a cloud platform. This may seem counter-intuitive, especially if the organization is not used to working in cloud environments. However, the ease and simplicity of managing cloud deployments can mean an overall reduction in the total cost of ownership and operational overhead.

The ultimate choice of the use case would depend on various factors, and the previous ones have been mentioned to set an approximate idea of the type of projects that may be attempted, and the workflows that may yield positive results. In the next section, we will look at some of the specific software and hardware solutions used in the industry for data science.