Scenario: Integration of the components for the solution
This chapter describes the integration of the components for a possible analytics solution architecture that uses IBM Power Systems.
At first glance, many products from various sources appear to fully integrate themselves to offer an operational analytics solution. However, to provide an optimized solution, you are better off implementing products that are designed for integration, such as IBM BigInsights for Apache Hadoop on top of IBM Open Platform with Apache Hadoop.
In our demonstration, we take full advantage of IBM Big SQL, which provides American National Standards Institute (ANSI) Structured Query Language (SQL) access to structured data that is spread across BigInsights systems, through Java Database Connectivity (JDBC) or Object Database Connectivity (ODBC), and other relational databases, such as IBM DB2. Therefore, SQL application developers can access data in Hadoop transparently with their current skill set, without learning any new language or technology.
Cognos Business Intelligence is certified for IBM Big SQL, which means that Cognos Business Intelligence can access data that is stored in IBM BigInsights. And in our demonstration, Cognos reports will retrieve data from both IBM DB2 with BLU Acceleration and IBM Big SQL. Sales from the fictional GOSALES Cognos sample database with sentiment analysis for the correspondent product that is extracted from Insights for Twitter, an IBM Bluemix service, will represent the input for our implemented solution.
IBM Statistical Package for the Social Sciences (SPSS) Analytical Decision Management and the SPSS Scoring Service will be integrated with web store pages to customize fictional offerings and discounts, based on customers’ particular characteristics and the general polarity of their Twitter posts.
To implement your own solution and describe ours in-depth, this chapter describes the following topics:
5.1 IBM Big SQL integration
You can enable access to structured data that is stored in IBM BigInsights for Apache Hadoop and integrate with other SQL development platforms. Big SQL provides this capability for applications to access data in IBM BigInsights by using ANSI SQL.
Businesses across the world use SQL to analyze their data, primarily because it is simple and powerful. This situation created departments with strong SQL skills, and Big SQL uses those skills to maximize the value that you can pull from your BigInsights environment.
Big SQL uses SQL to produce a view of your existing data. The storage format relies on Hive, HBase, or BigInsights distributed file system so you can use your usual SQL queries. In fact, the same SQL that you use on your data warehouse runs with few or no modifications, including the use of subqueries and all standard join operations. Beware that Big SQL supports querying data through SELECT statements, but it does not support UPDATE or DELETE statements. INSERT statements for Big SQL are supported only on HBase tables.
Big SQL also supports the IBM Data Server Client drivers. Therefore, you can keep the same standards-compliant JDBC, Java Call Control (JCC), ODBC, call level interface (CLI), and .NET drivers that are used in other IBM software products, such as DB2 for Linux, UNIX, and Windows, DB2 for z/OS, and IBM Informix database software. Other languages that already use these drivers can interact with Big SQL with minimum custom configuration.
Administrators can use the LOAD command to populate Big SQL tables by using various data sources, for example, flat files as .txt and .csv files, and the following external data sources:
DB2
Netezza®
Teradata
Microsoft SQL Server
Oracle
Informix
In addition to working with the data that you stored in BigInsights, Big SQL can federate to other data stores, which facilitates joining across databases. Because the query optimizer understands the capabilities of the external systems, it can push as much work as possible to each federated system to process.
5.1.1 Proposed solution integration with IBM Big SQL
IBM Big SQL plays a major role in our proposed solution (Figure 5-1) by storing data that was extracted from Insights for Twitter a Bluemix Service to provide data later to both Cognos Business Intelligence and SPSS Analytical Decision Management.
Figure 5-1 Proposed solution deployment diagram
In Figure 5-1, the blue arrows describe the simplified data flow. They are described later in this chapter. The yellow cloud and boxes represent all of the external entities:
Insights for Twitter a Bluemix Service: This service provides tweets metadata, including text content polarity, for example, positive, negative, or neutral comments about a product, that is essential to this sentimental analysis solution. This service is courtesy of the IBM alliance with Twitter that will be used to collect the public perception of one specific product and also track the overall tweets polarity of one particular customer that is identified by their twitter handle.
Data scientist: SPSS Analytical Decision Management is a tool for data scientists to predict the customer positive response ratio for a new campaign and to predict the profitability of the interaction based on the historical purchase behavior of the customer individual personality.
Executive: Cognos Business Intelligence dashboards will show the total amount of sales with overall public perception by product. Executives and decision makers will decide the most appropriate campaign. They can also benefit from the insights of SPSS Analytical Decision Management to maximize profits by customizing offers to their customers.
Customer: Web Store is the online shopping channel. The offer will be customized based on the historical purchase behavior of the customer by using the online SPSS Scoring Service. If the customer-associated twitter handle is previously identified, the tweets polarity of the particular customer is determined and it will also support the decision for a customized offer. A positive response for a campaign will sometimes correlate to the general tweet polarity.
For reference, the host names and services that each node holds in our sample solution are described:
mn01: This node is the Big SQL head node (be aware that data nodes that were previously configured behind the BigInsights are not disclosed to simplify the deployment diagram):
 – BigInsights:
 • Big SQL
dn04: This node is the Web Store node that interacts with the online shopping customers, displaying customized offers based on online queries into SPSS Scoring Services:
 – Web Store
dn05: This node by using DB2 with BLU Acceleration holds data that supports the Web Store for customer interactions. This node also holds actual and aggregated sales data to feed Cognos Business Intelligence reports and dashboards. In addition, this node holds the SPSS Collaboration and Deployment Services repository:
 – DB2 with BLU Acceleration
dn06: This node provides reports and dashboards to company executives and decision makers, extracting actual and aggregated data from DB2 with BLU Acceleration and twitter sentiment analysis and additional detailed sales data from Big SQL:
 – Cognos Business Intelligence
dn07: This node provides SPSS Scoring Services within SPSS Collaboration and Deployment Services for individual customer offers to the online Web Store. In addition, it stores repository data in a DB2 regular row-organized database. The interaction with Big SQL is intended to retrieve the overall tweets polarity, identify the general mood of the particular customer, and maximize positive responses to the sales campaigns. Data Scientists will use SPSS Analytical Decision Management to identify rules to maximize profits for a new campaign. However, to install SPSS Analytical Decision Management, you also need to install SPSS Modeler Server Adapters for Collaboration and Deployment Services, which requires SPSS Modeler Server in addition to SPSS Collaboration and Deployment Services (C&DS):
 – SPSS Collaboration and Deployment Services (C&DS):
 • Scoring Services
 – SPSS Modeler Server
 – SPSS Modeler Server Adapters for Collaboration and Deployment Services
 – SPSS Analytical Decision Management (ADM)
Figure 5-2 shows the host names and services that each node holds in our sample solution.
Figure 5-2 IBM Big SQL integration with Insights for Twitter a Bluemix Service
For our sample solution, server mn01 is the head node for the IBM Big SQL service. It will store transformed data transparently that was extracted from Insights for Twitter into the BigInsights data nodes that are not disclosed in Figure 5-2. The BigInsights storage layer is omitted intentionally.
Before we describe how to transform and store structured data into IBM Big SQL, we explain how to connect from different sources to support the implementation of your own solution.
For the integration of other components, we focus on IBM Data Server drivers, but you can connect to IBM Big SQL server by using the following methods:
IBM Data Server Manager for Big SQL
Java SQL Shell (JSqsh) command interface
IBM Data Server drivers
5.1.2 Configure the connection with IBM Data Server Manager for Big SQL
The default URL for your IBM Data Server Manager for Big SQL is shown:
https://<head-node>:8443/gateway/default/BigInsightsWeb/index.html#/welcome
In Figure 5-3, the default password for the user guest is guest-password.
Figure 5-3 IBM Data Server Manager for Big SQL Home page
To configure your first database connection, on Figure 5-3, click Actions → Discover Databases.
The Discover Databases page opens (Figure 5-4). Enter the name, user ID, and password. Click Find Instances according to your environment settings, and you will find the available Big SQL servers.
Figure 5-4 IBM Data Server Manager for Big SQL: Discover Databases
Select your instance. In our demonstration, it is bigsql. Click Develop → SQL Editor. Type the following SQL statement to test your connection and click Run:
select * from sysibm.sysdummy1
A successful test output is shown in Figure 5-5.
Figure 5-5 Successful connection test
5.1.3 Configure the connection with the jsqsh command line
On the shell prompt, type jsqsh. The first time that you use jsqsh on the command shell, a configuration wizard starts. When you are at the jsqsh command prompt, type drivers to determine the available drivers, which are listed:
IBM Data Server (DB2, Informix, and Big SQL) - com.ibm.db2.jcc.DB2Driver
IBM DB2 z/OS - com.ibm.db2.jcc.DB2Driver
Apache Derby Server - org.apache.derby.jdbc.ClientDriver
Apache Derby Embedded - org.apache.derby.jdbc.EmbeddedDriver
Firebird JayBird - org.firebirdsql.jdbc.FBDriver
Apache Hive - org.apache.hadoop.hive.jdbc.HiveDriver
Apache Hive - org.apache.hive.jdbc.HiveDriver
IBM Informix - com.informix.jdbc.IfxDriver
JDBC ODBC Bridge - sun.jdbc.odbc.JdbcOdbcDriver
MS SQL Server - com.microsoft.jdbc.sqlserver.SQLServerDriver
MS SQL Server jTDS - net.sourceforge.jtds.jdbc.Driver
MS SQL Server 2005+ - com.microsoft.sqlserver.jdbc.SQLServerDriver
MySQL - com.mysql.jdbc.Driver
IBM Netezza - org.netezza.Driver
Oracle - oracle.jdbc.OracleDriver
Oracle OCI - oracle.jdbc.driver.OracleDriver
PostgreSQL - org.postgresql.Driver
Sybase ASE - com.sybase.jdbc3.jdbc.SybDriver
Sybase ASA - com.sybase.jdbc2.jdbc.SybDriver
Sybase ASE jTDS - net.sourceforge.jtds.jdbc.Driver
The first time that you open JSQSH, the setup wizard launches. The panel that is shown in Figure 5-6 appears.
Follow these steps to configure the connection:
1. Type C to start the connection wizard, as shown in Figure 5-6.
JSQSH SETUP WIZARD
 
Welcome to the jsqsh setup wizard! This wizard provides a (crude) menu
driven interface for managing several jsqsh configuration files. These
files are all located in $HOME/.jsqsh, and the name of the file being
edited by a given screen will be indicated on the title of the screen
 
Note that many wizard screens require a relative large console screen
size, so you may want to resize your screen now.
 
(C)onnection management wizard
The connection management wizard allows you to define named connections
using any JDBC driver that jsqsh recognizes. Once defined, jsqsh only
needs the connection name in order to establish a JDBC connection
 
(D)river management wizard
The driver management wizard allows you to introduce new JDBC drivers
to jsqsh, or to edit the definition of an existing driver. The most
common activity here is to provide the classpath for a given JDBC driver
 
Choose (Q)uit, (C)onnection wizard, or (D)river wizard:
Figure 5-6 JSQSH setup wizard
2. If a connection is not created, type A to add the connection, as shown in Figure 5-7.
JSQSH CONNECTION WIZARD - (edits $HOME/.jsqsh/connections.xml
The following connections are currently defined:
 
Name Driver Host Port
--- -------------------- ---------- ------------------------------ ------
1 bigsql db2 mn01 32051
 
Enter a connection number above to edit the connection, or:
(B)ack, (Q)uit, or (A)dd connection:
Figure 5-7 JSQSH connection management wizard
3. Figure 5-8 shows the available drivers. Choose a driver for your new connection.
JSQSH CONNECTION WIZARD - (edits $HOME/.jsqsh/connections.xml)
 
Choose a driver for use by your new connection
 
Name Target Class
--- ---------------- -------------------- --------------------------------------------------
1 *db2 IBM Data Server (DB2 com.ibm.db2.jcc.DB2Driver
2 *db2zos IBM DB2 z/OS com.ibm.db2.jcc.DB2Driver
3 *hive2 Apache Hive org.apache.hive.jdbc.HiveDriver
4 *mysql MySQL com.mysql.jdbc.Driver
5 derby Apache Derby Server org.apache.derby.jdbc.ClientDriver
6 derbyembed Apache Derby Embedde org.apache.derby.jdbc.EmbeddedDriver
7 firebird Firebird JayBird org.firebirdsql.jdbc.FBDriver
8 hive Apache Hive org.apache.hadoop.hive.jdbc.HiveDriver
9 informix IBM Informix com.informix.jdbc.IfxDriver
10 jdbcodbc JDBC ODBC Bridge sun.jdbc.odbc.JdbcOdbcDriver
11 mssql MS SQL Server com.microsoft.jdbc.sqlserver.SQLServerDriver
12 mssql-jtds MS SQL Server jTDS net.sourceforge.jtds.jdbc.Driver
13 mssql2k5 MS SQL Server 2005+ com.microsoft.sqlserver.jdbc.SQLServerDriver
14 netezza IBM Netezza org.netezza.Driver
15 oracle Oracle oracle.jdbc.OracleDriver
16 oracleoci Oracle OCI oracle.jdbc.driver.OracleDriver
17 pgsql PostgreSQL org.postgresql.Driver
18 sybase Sybase ASE com.sybase.jdbc3.jdbc.SybDriver
19 sybase-asa Sybase ASA com.sybase.jdbc2.jdbc.SybDriver
20 sybase-jtds Sybase ASE jTDS net.sourceforge.jtds.jdbc.Driver
 
* = Driver is available. If a driver is unavailable, you may choose (D) below
to jump to the driver wizard to provide a classpath
 
Enter the driver number, (D)river wizard, (B)ack or (Q)uit:
Figure 5-8 JSQSH driver selection for new connections
4. Enter the number 1 for *db2 for Big SQL databases. Complete the fields according to your environment settings (Figure 5-9).
JSQSH CONNECTION WIZARD - (edits $HOME/.jsqsh/connections.xml)
 
The following configuration properties are supported by this driver.
 
Connection name : _temp_
Driver : IBM Data Server (DB2, Informix, Big SQL)
JDBC URL : jdbc:db2://${server}:${port}/${db}
 
Connection URL Variables
------------------------
1 db : BIGSQL
2 port : 32051
3 server : localhost
4 user : bigsql
5 password :
6 Autoconnect : false
 
JDBC Driver Properties
------------------------
None
 
Enter a number to change a given configuration property, or
(T)est, (B)ack, (Q)uit, Add (P)roperty, or (S)ave:
Figure 5-9 Big SQL connection settings example
5. Test your connection settings (Figure 5-9). Save and name this connection.
6. Connect to your database by using the connect <database> command and perform your first query, as shown in Figure 5-10.
Welcome to JSqsh 4.4
Type "help" for help topics. Using JLine.
1> connect bigsql
Password: *******
[cec01n01.cluster.com][bigsql] 1> select * from sysibm.sysdummy1;
+---------+
| IBMREQD |
+---------+
| Y |
+---------+
1 row in results(first row: 0.3s; total: 0.3s)
Figure 5-10 Connecting to the Big SQL database and performing an SQL query
5.1.4 Cognos Business Intelligence data source by using the DB2 CLI connection
As one example of IBM Data Server drivers, you can catalog a Big SQL database by using DB2 command line processor (CLP) from any node with access to the IBM Big SQL service ports.
Cognos Business Intelligence can use JDBC to connect to Big SQL but it can also use DB2 call level interface (CLI). Therefore, in our example, we will catalog the Big SQL database as any regular DB2 database in a Cognos Business Intelligence server.
On the Cognos Business Intelligence node, log in as the db2 instance owner, in our environment, db2inst1. Catalog the Big SQL node and database, as shown in Figure 5-11.
db2 catalog tcpip node <bigsql_node> remote <head-node> server <server-port>
 
db2 catalog db <bigsql_database> at node <bigsql_node>
 
db2 connect to <bigsql_database> user <user_id> using <user_password>
 
db2 "select * from sysibm.sysdummy1"
Figure 5-11 Commands to catalog and test the connection to Big SQL
The commands in Figure 5-11 will perform the following actions (output is shown in Example 5-1):
1. Catalog the remote node.
2. Catalog the database.
3. Connect to the database.
4. Perform a simple SQL query for connection testing.
Example 5-1 Output example of commands to catalog and test the connection to Big SQL
db2inst1@dn06 ~]$ db2 catalog tcpip node bigsql remote mn01 server 32051
DB20000I The CATALOG TCPIP NODE command completed successfully.
DB21056W Directory changes may not be effective until the directory cache is
refreshed.
[db2inst1@dn06 ~]$ db2 catalog db bigsql at node bigsql
DB20000I The CATALOG DATABASE command completed successfully.
DB21056W Directory changes may not be effective until the directory cache is
refreshed.
[db2inst1@dn06 ~]$ db2 connect to bigsql user bigsql
Enter current password for bigsql:
 
Database Connection Information
 
Database server = DB2/LINUXPPC64LE 10.6.3
SQL authorization ID = BIGSQL
Local database alias = BIGSQL
 
[db2inst1@dn06 ~]$ db2 "select * from sysibm.sysdummy1"
 
IBMREQD
-------
Y
1 record(s) selected.
 
Note: You must catalog the head node. Otherwise, you can receive the following error when you query data:
 
SQL0969N There is no message text corresponding to SQL error "-5115" in the
message file on this workstation. The error was returned from module "SQLRR0AA" with original tokens "SQL statements only allowed on head node"
If this error occurs, run the uncatalog node and catalog node commands by using the head node settings. Also, issue the db2 terminate command to recycle your session connections. Try to connect and perform the simple SQL query again:
[db2inst1@dn06 ~]$ db2 "select * from sysibm.sysdummy1"
SQL0969N There is no message text corresponding to SQL error "-5115" in the
message file on this workstation. The error was returned from module
"SQLRR0AA" with original tokens "SQL statements only allowed on head node".
db2inst1@dn06 ~]$ db2 uncatalog node bigsql
DB20000I The UNCATALOG NODE command completed successfully.
DB21056W Directory changes may not be effective until the directory cache is
refreshed.
[db2inst1@dn06 ~]$ db2 catalog tcpip node bigsql remote mn01 server 32051
DB20000I The CATALOG TCPIP NODE command completed successfully.
DB21056W Directory changes may not be effective until the directory cache is
refreshed.
[db2inst1@dn06 ~]$ db2 terminate
DB20000I The TERMINATE command completed successfully.
[db2inst1@dn06 ~]$ db2 connect to bigsql user bigsql
Enter current password for bigsql:
 
Database Connection Information
 
Database server = DB2/LINUXPPC64LE 10.6.3
SQL authorization ID = BIGSQL
Local database alias = BIGSQL
 
[db2inst1@dn06 ~]$ db2 "select * from sysibm.sysdummy1"
 
IBMREQD
-------
Y
 
1 record(s) selected.
Follow these steps:
1. To test the Big SQL recently created connection from the DB2 CLP, we will create a data source. On the Cognos Administration page, http://<cognos_node>/ibmcognos, click Configuration → Data Source Connections. Click New Data Source and name it, for example, Big SQL test, as shown in Figure 5-12.
Figure 5-12 Creating a Big SQL test data source by using DB2 CLI
2. Select IBM DB2 for the data source type. Clear Configure JDBC Connection (because we want to use the recently created DB2 database catalog with DB2 CLI for testing), as shown in Figure 5-13.
Figure 5-13 Select IBM DB2 and clear the option to configure a JDBC connection
3. On the next window, in the DB2 database name field, type the database name that you cataloged, in our case, BIGSQL. In the Signons area, check Password and click Test the connection, as shown in Figure 5-14.
Figure 5-14 Enter the DB2 database name and test the connection
4. On the next window, type your bigsql user ID and password. Click Test, as shown in Figure 5-15.
Figure 5-15 Type the appropriate bigsql user ID and password for your environment settings
5. If your settings are correct and your services are up and running, you receive the following message, as shown in Figure 5-16.
Figure 5-16 Connection with IBM DB2 is compatible and successful
5.1.5 Cognos Business Intelligence data source by using the JDBC connection
Follow these steps:
1. As another example of the IBM Data Server drivers, you can configure the JDBC connection to the Big SQL database by using the DB2 JDBC drivers from any node with access to the IBM Big SQL service ports, as shown in Figure 5-17. Select JDBC.
Figure 5-17 JDBC connection
2. On the next window, you select IBM InfoSphere BigInsights (Big SQL) as the type of the data source. Provide the JDBC URL with your environment’s settings in the format jdbc:db2://<host>:<port>/<database-name>, as shown in Figure 5-18.
Figure 5-18 JDBC InfoSphere BigInsights (Big SQL) settings
3. In the Signons area, check Password and click Test the connection, as shown in Figure 5-14 on page 148.
4. The Test the connection window opens. Type your bigsql user ID and password. Click Test.
5. If your settings are correct and your services are up and running, you receive the successful connection message, as shown in Figure 5-19.
Figure 5-19 JDBC connection test succeeded
5.1.6 IBM Big SQL federation
Data analytics is crucial to the success of business. Therefore, data warehousing that supports the consolidation and federation of data is a requirement.
Big SQL federation enables a mixed environment that creates the consolidation of big data and the federation of live data from different data sources.
At the time that this publication was written, Table 5-1 shows the list of supported data sources that can be used to federate to Big SQL.
Table 5-1 Data sources that are supported for Big SQL federation
Data source
Supported versions
Notes
DB2
DB2 for Linux, UNIX, and Windows 9.7, 9.8, 10.1, and 10.5
 
Oracle
11g, 11g R1, and 11g R2
 
Teradata
12, 13, and 14
Not supported on Power Systems.
Netezza
4.6, 5.0, 6.0, and 7.2
Not supported on Power Systems.
Informix
11.5
 
Microsoft SQL Server
2012
 
Set up the federation to DB2 with BLU Acceleration
Follow these steps:
1. In your Big SQL head node, log in as the instance owner, which is bigsql, by default, and double-check whether the federation is enabled for immediate usage by issuing the command that is shown in Figure 5-20.
db2 get dbm cfg | grep "Federated Database System Support"
Figure 5-20 Checking Big SQL federated settings
 
Note: If the FEDERATED setting is not YES, you must enable federation by using the following command:
db2 update dbm cfg using FEDERATED YES
Then, stop and start the bigsql instance with the following commands:
$BIGSQL_HOME/bin/bigsql stop
$BIGSQL_HOME/bin/bigsql start
2. Catalog the node and the database for DB2 with BLU Acceleration for your environment settings, as shown in Figure 5-21.
db2 catalog tcpip node <db2_blu_node> remote <db2_blu_host> server <db2_blu_port>
 
db2 catalog db <db2_blu_database> at node <db2_blu_node>
Figure 5-21 Catalog DB2 with BLU Acceleration for the node and database
3. Create a wrapper, federated server, and user mappings with the commands for your environment settings, as shown in Figure 5-22.
db2 connect to bigsql user bigsql
 
db2 CREATE WRAPPER DRDA LIBRARY 'libdb2drda.so'
 
db2 "CREATE SERVER DB2REMOTE TYPE DB2/UDB VERSION 10.5 WRAPPER DRDA AUTHORIZATION "<db2_blu_instance_owner>" PASSWORD "<db2_blu_instance_owner_password>" OPTIONS (DBNAME '<db_blu_database>')"
 
db2 “CREATE USER MAPPING FOR <big_sql_instance_owner> SERVER DB2REMOTE OPTIONS ( REMOTE_AUTHID '<db2_instance_owner>', REMOTE_PASSWORD '<db2_instance_owner_password>' )”
Figure 5-22 Create a wrapper, federated server, and user mappings
4. We assume that you already populated your DB2 database with GOSALES Cognos sample data. We create an example nickname for table GOSALESDW.GO_REGION_DIM and then perform a simple query on the remote table, as shown in Example 5-2.
Example 5-2 Nickname example for table GOSALESDW.GO_REGION_DIM
[bigsql@mn01 ~]$ db2 CREATE NICKNAME DB2REMOTE.GO_REGION_DIM FOR DB2REMOTE.GOSALESDW.GO_REGION_DIM
DB20000I The SQL command completed successfully.
[bigsql@mn01 ~]$ db2 "select count(*) from DB2REMOTE.GO_REGION_DIM"
 
1
---------------------------------
21.
 
1 record(s) selected.
By using this federation setup, you can consolidate data from multiple data sources in the Big SQL instance. In addition, you can provide access to Big SQL users to multiple data sources. Check that you create a user mapping for each Big SQL user. Consult the syntax of the DB2 command CREATE USER MAPPING for details.
5.1.7 Loading GOSALESDW data into Big SQL
You can use the IBM Data Server Manager for Big SQL, JSQSH, or IBM DB2 Data Server drivers to create tables, load data, and query Big SQL data.
A set of scripts is available to create and load GOSALESDW sample data into your Big SQL database under the directory $BIGSQL_HOME/samples.
Carefully read the $BIGSQL_HOME/samples/README file for instructions and run the command that is shown in Figure 5-23.
./setup.sh -u <userid> -s <host_name> -n <port number> -d <databaseName>
Figure 5-23 Loading GOSALESDW into Big SQL
Alternatively, you can create all of the GOSALESDW tables manually by using $BIGSQL_HOME/samples/queries/GOSALESDW_ddl.sql with JSQSH and running the command that is shown in Figure 5-24.
jsqsh <db> -U <user> -P <pass> < $BIGSQL_HOME/samples/queries/GOSALESDW_ddl.sql
Figure 5-24 Creating the GOSALESDW tables in Big SQL with jsqsh
Load the data with the LOAD HADOOP command by using a modified version of $BIGSQL_HOME/samples/queries/GOSALESDW_load.sql, updating FILE URL to SFTP or any other appropriate method.
For instance, in our case, we used GPFS, which refers to Spectrum Scale with File Placement Optimizer, as shown in Example 5-3.
Example 5-3 LOAD HADOOP by using SFTP or Spectrum Scale
load hadoop using file url 'sftp://<user>:<pass>@<sftp_node>:/usr/ibmpacks/current/bigsql/bigsql/samples/data/GOSALESDW.BURST_TABLE2.txt' with SOURCE PROPERTIES ('field.delimiter'=' ') into table GOSALESDW.BURST_TABLE2 overwrite;
 
load hadoop using file url 'gpfs:////bigpfs/user/bigsql/DemoData/gs_db/GOSALESDW.BURST_TABLE2.txt' with SOURCE PROPERTIES ('field.delimiter'=' ') into table GOSALESDW.BURST_TABLE2 overwrite;
5.1.8 Loading Twitter Data into Big SQL
To load Twitter data for demonstration, we assume that you have a tweets feed with sentimental analysis, for example, Insights for Twitter an IBM Bluemix service for Data and Analytics.
You must be authorized to access the following service application programming interface (API):
By using that service API, notice that many attributes of a tweet are retrieved, for example, the author’s gender, location, and the number of followers.
The IBM alliance with Twitter sentiment analysis also provides the polarity of the content: NEUTRAL, POSITIVE, or NEGATIVE. See Example 5-4.
Example 5-4 IBM Bluemix Insights for Twitter output
{
"search": {
"results": 889857,
"current": 1
},
"tweets": [{
"cde": {
"author": {
"gender": "male",
"parenthood": {
"isParent": "unknown",
"evidence": ""
},
"location": {
"country": "United States",
"city": "",
"state": "North Carolina"
},
"maritalStatus": {
"isMarried": "unknown",
"evidence": ""
}
},
"content": {
"sentiment": {
"evidence": [],
"polarity": "NEUTRAL"
}
}
},
"message": {
"postedTime": "2013-11-05T16:45:53.000Z",
"verb": "post",
"link": "http://twitter.com/brenny/statuses/397766729256161281",
"generator": {
"displayName": "Hootsuite",
"link": "http://www.hootsuite.com"
},
"body": "RT @ibm_iod: Now: @BethTSmith showing IBM Predictive Maintenance &amp; Quality demo - Capture, Predict, Act #ibmiod http://t.co/JP5EncVurN",
"favoritesCount": 0,
"objectType": "activity",
"actor": {
"summary": "Social Insights Senior Analyst. Interested in #gamification, MOOCs; Percussionist, Dad, writer, wannabe tennis player. Tweets are my own.",
"image": "https://pbs.twimg.com/profile_images/53493646/bran-headshot_normal.jpg",
"statusesCount": 3203,
"utcOffset": "-21600",
"languages": ["en"],
"preferredUsername": "brenny",
"displayName": "Brenny Ganesan",
"postedTime": "2008-02-07T23:20:34.000Z",
"link": "http://www.twitter.com/brenny",
"verified": false,
"friendsCount": 1827,
"twitterTimeZone": "Central Time (US & Canada)",
"favoritesCount": 91,
"listedCount": 112,
"objectType": "person",
"links": [{
"rel": "me",
"href": "http://www.servicemanagement360.com"
}],
"location": {
"displayName": "RTP NC",
"objectType": "place"
},
"id": "id:twitter.com:13225972",
"followersCount": 1847
},
"provider": {
"displayName": "Twitter",
"link": "http://www.twitter.com",
"objectType": "service"
},
"twitter_filter_level": "low",
"twitter_entities": {
"urls": [{
"display_url": "youtube.com/watch?v=pwdGIu…",
"indices": [116, 138],
"expanded_url": "http://www.youtube.com/watch?v=pwdGIuMnVl4",
"url": "http://t.co/JP5EncVurN"
}],
"hashtags": [{
"indices": [108, 115],
"text": "ibmiod"
}],
"user_mentions": [{
"indices": [3, 11],
"screen_name": "ibm_iod",
"id_str": "2463969721",
"name": "IBM Insight",
"id": 2463969721
}, {
"indices": [18, 29],
"screen_name": "BethTSmith",
"id_str": "568818502",
"name": "Beth Smith",
"id": 568818502
}],
"trends": [],
"symbols": []
},
"twitter_lang": "en",
"id": "tag:search.twitter.com,2005:397766729256161281",
"retweetCount": 1,
"object": {
"summary": "RT @ibm_iod: Now: @BethTSmith showing IBM Predictive Maintenance &amp; Quality demo - Capture, Predict, Act #ibmiod http://t.co/JP5EncVurN",
"postedTime": "2013-11-05T16:45:53.000Z",
"link": "http://twitter.com/brenny/statuses/397766729256161281",
"id": "object:search.twitter.com,2005:397766729256161281",
"objectType": "note"
}
}
}],
"related": {
"next": {
"href": "https://cdeservice.stage1.mybluemix.net:443/api/v1/messages/search?q=ibm&from=1&size=1"
}
}
}
You can create your own solution for extracting the required data from the tweet feed to load into your Big SQL by using any programming language that is compatible with JavaScript Object Notation (JSON), for example, the C family of languages, which includes these languages:
C
C++
C#
Java
JavaScript
Perl
Python
Alternatively you can use Serializer/Deserializer (SerDe), which is a common component in Hadoop environments. For JSON data, as shown in Example 5-4 on page 156, you can rely on the appropriate Hive SerDe interface for JSON to read and interpret the semi-structured data for your applications.
For our demonstration, we used another approach. A few tweets that mention products from the fictional GOSALES sample database were extracted from Twitter by using Bluemix Insights for Twitter service. Those tweet feeds, which were originally semi-structured data, by using JSON format were transformed into structured data by using proprietary scripts. Then, they were associated to their correspondent product data in a .csv file format for preparation to load into Big SQL. See Example 5-5 on page 160.
The sample contains the following product-related columns:
Count
SearchObject
Category
Brand
Format
Family
Subfamily
Product
In addition, a few columns that relate to tweets that mention those products are listed:
ObjectProvenance
Source
IsBuzz
IsSentiment
IsIntent
IsOwnership
IsCustomerOf
Polarity
CreatedTime
FullName
Gender
City
State
Screenname
UserID
Userverified
FollowersCount
IsParent
IsMarried
Text
Example 5-5 shows the transformed tweet feeds that were associated to their correspondent product data in a structured .csv file format for preparation to load into Big SQL.
Example 5-5 Transformed structured .csv data from twitter feeds
"Count","SearchObject","Category","Brand","Format","Family","SubFamily","Product","ObjectProvenance","Source","IsBuzz","IsSentiment","IsIntent","IsOwnership","IsCustomerOf","Polarity","CreatedTime","FullName","Gender","City","State","Screenname","UserID","Userverified","FollowersCount","IsParent","IsMarried","Text"
1,"GRANITE HAMMER","MOUNTAINEERING EQUIPMENT","GRANITE",,,,"HAMMER","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/20/2012 4:53",,,,,"brayinharlem09",7653578196,"FALSE",90,,,"Don't expect to use the Granite Hammer for heavy duty work. It's soft as clay. #LAME"
1,"BUGSHIELD NATURAL","OUTDOOR PROTECTION","BUGSHIELD",,,,"NATURAL","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/12/2012 15:32",,,,,"brayinharlem09",7653578196,"FALSE",90,,,"I sooooo hate BugShield Natural... I came back with 37 bites.. on my arm ALONE!"
1,"LADY HAILSTORM TITANIUM IRONS","GOLF EQUIPMENT","HAILSTORM",,"IRONS",,,"USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/14/2012 22:04","Chris Hemmingway","Male","NEW YORK CITY","NY","CHemmingway",3020000000000000000,"FALSE",440,,,"She liked the Lady Hailstorm Titanium Irons! thanks @shirley"
1,"CANYON MULE COOLER","CAMPING EQUIPMENT","CANYON MULE",,,,"COOLER","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/19/2012 0:42","Pina Pender",,"MIAMI","FL","PinaPender",5550000000000000000,"FALSE",564,,,"Hey @jani check this retweet out: I like to own Canyon Mule Cooler. You were right"
1,"LADY HAILSTORM STEEL IRONS","GOLF EQUIPMENT","HAILSTORM",,"IRONS",,,"USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/9/2012 8:48","Pina Pender",,"MIAMI","FL","PinaPender",5550000000000000000,"FALSE",564,,,"Do you love to have the Lady Hailstorm Steel Irons"
1,"GRANITE EXTREME","MOUNTAINEERING EQUIPMENT","EXTREME",,,,"EXTREME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/6/2012 0:54","Luisa Amatea","Female","NEW YORK CITY","NY","LAmatea111",8350000000000000000,"FALSE",129,,,"I like owning my Granite Extreme ;)) [pic] twitpic.com/url01"
1,"TRAILCHEF KETTLE","CAMPING EQUIPMENT","TRAILCHEF",,,,"KETTLE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/25/2012 21:21",,"Female","BOSTON","MA","lucyluca90",7653556196,"FALSE",35,,,"If Someone wanna buy TrailChef Kettle, awesome deal at http://bit.ly/3rvghff"
1,"DELUXE FAMILY RELIEF KIT","OUTDOOR PROTECTION","RELIEF",,,,,"USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/20/2012 1:13","Abram Barber","Male","HOUSTON","TX","AbBarber3",5840000000000000000,"FALSE",134,,,"Check this out: I recommend a Deluxe Family Relief Kit, awesome deal at http://bit.ly/3rvghff"
1,"EDGE EXTREME","PERSONAL ACCESSORIES","EXTREME",,,,"EDGE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/11/2012 6:45","Gabriel Walters","Male","ATLANTA","GA","GabWalters",8740000000000000000,"FALSE",826,,,"If anyone wants to give me a Edge Extreme, awesome deal at http://bit.ly/3rvghff ;)"
1,"COURSE PRO GOLF AND TEE SET","GOLF EQUIPMENT","COURSE PRO",,,,"GOLF AND TEE SET","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/10/2012 16:32","Annalaura Pappalardo","Female","LOS ANGELES","CA","APappalardo",4470000000000000000,"FALSE",573,,,"@JoeMoe2 since you brought it up.. I hate the Course Pro Golf and Tee Set you use.. jus' sayin"
1,"DELUXE FAMILY RELIEF KIT","OUTDOOR PROTECTION","RELIEF",,,,,"USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/5/2012 19:47","Annalaura Pappalardo","Female","LOS ANGELES","CA","APappalardo",4470000000000000000,"FALSE",573,,,"I hate those annoying Deluxe Family Relief Kit commercials as much as i hate the real thing"
1,"CANYON MULE CLIMBER BACKPACK","CAMPING EQUIPMENT","CANYON MULE",,,,"CLIMBER BACKPACK","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/21/2012 21:40","Norman Wood","Male","MIAMI","FL","NormanWood",3600000000000000000,"FALSE",97,,,"I hate this Canyon Mule Climber Backpack -_-"
1,"FIREFLY EXTREME","CAMPING EQUIPMENT","EXTREME",,,,"EXTREME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/17/2012 7:36","Olive Davidson","Female","ALBANY","GA","Oliveon",5610000000000000000,"FALSE",937,,,"I hate the Firefly Extreme"
1,"INSECT BITE RELIEF","OUTDOOR PROTECTION",,,,,"INSECT BITE","USER","Twitter","FALSE","TRUE","FALSE","FALSE","FALSE","Negative","9/2/2012 1:37","Olive Davidson","Female","ALBANY","GA","Oliveon",5610000000000000000,"FALSE",937,,,"I almost feel bad for giving the huge bottle of Insect Bite Relief away! It relieve nothing lol"
1,"STAR PEG","CAMPING EQUIPMENT","STAR",,,,"PEG","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/20/2012 20:19","Olive Davidson","Female","ALBANY","GA","Oliveon",5610000000000000000,"FALSE",937,,,"My family wasted so much $... and now hates their lame Star Peg lol #fail"
1,"HAILSTORM STEEL WOODS SET","GOLF EQUIPMENT","HAILSTORM",,"WOODS",,"STEEL WOODS SET","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/25/2012 8:12","Rosy Red","Female",,,"flowergirl67",76535196678,"FALSE",0,,,"I love having my Hailstorm Steel Woods Set when i need it. Check out this deal http://t.co/ades323d"
1,"COURSE PRO GLOVES","GOLF EQUIPMENT","COURSE PRO",,,,"GLOVES","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/13/2012 11:25","Emily Graham","Female","MIAMI","FL","EmilyGraham",9160000000000000000,"FALSE",514,,,"Anyone own the Course Pro Gloves, awesome deal at http://bit.ly/3rvghff"
1,"GRANITE AXE","MOUNTAINEERING EQUIPMENT","GRANITE",,,,"AXE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/17/2012 9:19","Wanda Williams","Female","NEW YORK CITY","NY","WandaWilliams",2860000000000000000,"FALSE",983,,,"And THIS is why I hate a Granite Axe [pic] twit.pic/e1e1s"
1,"CAT EYE","PERSONAL ACCESSORIES",,,,,"CAT EYE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/7/2012 19:52","Wanda Williams","Female","NEW YORK CITY","NY","WandaWilliams",2860000000000000000,"FALSE",983,,,"I gave away that ugly Cat Eye... sorry @Alice haha."
1,"BLUE STEEL MAX PUTTER","GOLF EQUIPMENT","BLUE STEEL",,,,"MAX PUTTER","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/2/2012 7:22","Wanda Williams","Female","NEW YORK CITY","NY","WandaWilliams",2860000000000000000,"FALSE",983,,,"@Ann Well.. I have the Blue Steel Max Putter, and I hate it so far :<< #COLLEGESPORTS"
1,"SUN BLOCKER","OUTDOOR PROTECTION","SUN",,,,"BLOCKER","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/17/2012 2:17","Wanda Williams","Female","NEW YORK CITY","NY","WandaWilliams",2860000000000000000,"FALSE",983,,,"@Adoughlas122 ... My friends hated the Sun Blocker you gave us. WE ALL BURNED! Not cool dude"
1,"TRENDI","PERSONAL ACCESSORIES",,,,,"TRENDI","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/21/2012 15:20","Wanda Williams","Female","NEW YORK CITY","NY","WandaWilliams",2860000000000000000,"FALSE",983,,,"So my parents officially hate the Trendi i got them #FML #WastedMoney"
1,"BEAR SURVIVAL EDGE","PERSONAL ACCESSORIES","EDGE",,,,"BEAR SURVIVAL","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/2/2012 7:22","Shania Ferguson","Female","MIAMI","FL","ShaniaFerguson",1670000000000000000,"FALSE",983,,,"I love to have the Bear Survival Edge. Don't leave home w/o it."
1,"SKY PILOT","PERSONAL ACCESSORIES",,,,,"SKY PILOT","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/28/2012 10:55","Shania Ferguson","Female","MIAMI","FL","ShaniaFerguson",1670000000000000000,"FALSE",983,,,"Check this out: I wanna buy the Sky Pilot, awesome deal at http://bit.ly/3rvghff"
1,"SINGLE EDGE","PERSONAL ACCESSORIES","EDGE",,,,"SINGLE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/12/2012 0:23","Sam Willis","Male","ORLANDO","FL","Sam.Willis",2030000000000000000,"FALSE",963,,,"Giving away my useless Single Edge. txt me"
1,"EVERGLOW BUTANE","CAMPING EQUIPMENT","EVERGLOW",,,,"BUTANE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/13/2012 17:09","Sam Willis","Male","ORLANDO","FL","Sam.Willis",2030000000000000000,"FALSE",963,,,"I hate EverGlow Butane! Never works the way it should :<<"
1,"TRAILCHEF KETTLE","CAMPING EQUIPMENT","TRAILCHEF",,,,"KETTLE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/24/2012 20:18","Sam Willis","Male","ORLANDO","FL","Sam.Willis",2030000000000000000,"FALSE",963,,,"Fail #cooking lol watching @Leti's hate for the TrailChef Kettle grow."
1,"GRANITE CHALK BAG","MOUNTAINEERING EQUIPMENT","GRANITE",,,,"CHALK BAG","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/17/2012 17:42","Sam Willis","Male","ORLANDO","FL","Sam.Willis",2030000000000000000,"FALSE",963,,,"@Javier888 haha Jen and I hated the last Granite Chalk Bag we got.. so.. that answers your question :p"
1,"EVERGLOW BUTANE","CAMPING EQUIPMENT","EVERGLOW",,,,"BUTANE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/16/2012 4:07","Elena Hernández González","Female","PORTLAND","OR","ElenaHz",7300000000000000000,"FALSE",753,,,"@Joe... I like my EverGlow Butane;0"
1,"STAR GAZER 2","CAMPING EQUIPMENT","STAR",,,,"GAZER 2","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/12/2012 1:11","Marian Proklov","Female","CHICAGO","IL","MarianProklov",203000000000000000,"FALSE",371,,,"@Lily Same here, but I love to have my Star Gazer 2 too ;)"
1,"MOUNTAIN MAN DELUXE","PERSONAL ACCESSORIES","MOUNTAIN MAN",,,,"DELUXE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/19/2012 12:12","Victoria Gray","Female","PORTLAND","OR","VictoriaG",782000000000000000,"FALSE",67,,,"I love Mountain Man Deluxe ;)) Check me out lol [pic] twit.pic/url164"
1,"CANYON MULE EXTREME BACKPACK","CAMPING EQUIPMENT","EXTREME",,,,"EXTREME BACKPACK","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/19/2012 14:51","Yuya Nishikawa","Male","NEW YORK CITY","NY","Yuya.Nishikawa",949000000000000000,"FALSE",490,,,"Who'd love to have the Canyon Mule Extreme Backpack ;)"
1,"HUSKY HARNESS EXTREME","MOUNTAINEERING EQUIPMENT","EXTREME",,,,"HARNESS EXTREME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/20/2012 2:06","Miles Miller","Male","SAN JOSE","CA","Miles.Miller",3230000000000000000,"FALSE",23,,,"Who want to buy a Husky Harness Extreme, awesome deal at http://bit.ly/3rvghff"
1,"HUSKY HARNESS EXTREME","MOUNTAINEERING EQUIPMENT","EXTREME",,,,"HARNESS EXTREME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/28/2012 19:04","José Díaz","Male","SAN JOSE","CA","José.Díaz",7500000000000000000,"FALSE",957,"TRUE",,"Would you like to own the Husky Harness Extreme, awesome deal at http://bit.ly/3rvghff"
1,"SUN BLOCKER","OUTDOOR PROTECTION","SUN",,,,"BLOCKER","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/28/2012 0:42","Bethanie Cunningham","Female","HOUSTON","TX","Bethanie.Cunningham",891000000000000000,"FALSE",300,,,"Check this out: [pic] twitpic.com/url123 I like the Sun Blocker so more more now!"
1,"TRENDI","PERSONAL ACCESSORIES",,,,,"TRENDI","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/26/2012 18:22","Sophie Wiseman","Female","PORTLAND","OR","Sophie_ Wiseman",8060000000000000000,"FALSE",909,,,"Someone have a Trendi, awesome deal at http://bit.ly/3rvghff"
1,"TRAILCHEF DOUBLE FLAME","CAMPING EQUIPMENT","TRAILCHEF",,,,"DOUBLE FLAME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/5/2012 19:35","Pamela Hislop","Female","CHICAGO","IL","PamelaHislop",3540000000000000000,"FALSE",396,,,"I love a TrailChef Double Flame"
1,"POLAR EXTREME","PERSONAL ACCESSORIES","EXTREME",,,,"POLAR","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/13/2012 2:26","Cort Cohen","Male","CHICAGO","IL","CortCohen",5100000000000000000,"FALSE",780,,,"Someone wanna buy the Polar Extreme, awesome deal at http://bit.ly/3rvghff"
1,"TRAILCHEF COOK SET","CAMPING EQUIPMENT","TRAILCHEF",,,,"COOK SET","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/21/2012 6:08","Mary Norton","Female","HOUSTON","TX","MNorton",7140000000000000000,"FALSE",705,,,"I like to own TrailChef Cook Set. great deal at: http://t.co/ades323d"
1,"HUSKY ROPE 50","MOUNTAINEERING EQUIPMENT","HUSKY",,"ROPE",,"ROPE 50","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/3/2012 12:48","Emma Malvinova","Female","ATLANTA","GA","EmmMalvinova",8590000000000000000,"FALSE",123,,,"I like to own the Husky Rope 50 for things like that lol! @Fernando"
1,"MAX GIZMO","PERSONAL ACCESSORIES",,,,,"MAX GIZMO","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/8/2012 8:38","Yichuan Rong",,"NEW YORK CITY","NY","YCRong_CN",1450000000000000000,"FALSE",823,,,"Would you love havin a Max Gizmo.. I'm wondering... #Bikes #College"
1,"SUN BLOCKER","OUTDOOR PROTECTION","SUN",,,,"BLOCKER","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/11/2012 11:35","Alex Chénier","Male","SEATTLE","WA","AlexChénier",4800000000000000000,"FALSE",1012,,,"Do you love Sun Blocker ;))"
1,"BUGSHIELD EXTREME","OUTDOOR PROTECTION","EXTREME",,,,"BUGSHIELD","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/25/2012 10:45","Patsy Mappin","Female","SEATTLE","WA","PatsyMappin",8250000000000000000,"FALSE",373,,,"I love having BugShield Extreme.. you guys should check it out! #camping #weekend"
1,"STAR LITE","CAMPING EQUIPMENT","STAR",,,,"LITE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/15/2012 17:19","Guy Pontel","Male","CHICAGO","IL","GuyPontel",6750000000000000000,"FALSE",927,,,"@hotcamper222 dumped her lame Star Lite :D Bout freakin' time"
1,"HIBERNATOR CAMP COT","CAMPING EQUIPMENT","HIBERNATOR",,,,"CAMP COT","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/18/2012 22:45","Salvador Guzman","Male","CHICAGO","IL","SalGuzman",5050000000000000000,"FALSE",370,,,"Bah! I hate this stupid Hibernator Camp Cot"
1,"TRAIL MASTER","PERSONAL ACCESSORIES",,,,,"TRAIL MASTER","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/9/2012 20:55","Vasilisa Izmailova","Female","HOUSTON","TX","Vasilisa8Izmailova",5410000000000000000,"FALSE",170,,,"I love the Trail Master. I am down 20lbs."
1,"TRAILCHEF COOK SET","CAMPING EQUIPMENT","TRAILCHEF",,,,"COOK SET","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/18/2012 7:19","Joanna Cannon","Female","MIAMI","FL","JoannaCannon",3370000000000000000,"FALSE",249,,,"I hate using the TrailChef Cook Set especially when it's windy."
1,"GRANITE AXE","MOUNTAINEERING EQUIPMENT","GRANITE",,,,"AXE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/3/2012 8:52","Melissa Doyle","Female","SAN JOSE","CA","Melissa.Doyle",8170000000000000000,"FALSE",546,,,"Guess who just bought a Granite Axe :)) #WINNING #MeGusta"
1,"MOUNTAIN MAN EXTREME","PERSONAL ACCESSORIES","EXTREME",,,,"EXTREME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/8/2012 20:15","May Norbert","Female","SEATTLE","WA","MayNorbert",8040000000000000000,"FALSE",541,,,"Check this out: I have a Mountain Man Extreme, awesome deal at http://bit.ly/3rvghff"
1,"GLACIER BASIC","PERSONAL ACCESSORIES","GLACIER",,,,"BASIC","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/19/2012 8:09","Milos Leer","Male","CHICAGO","IL","Milos Leer",3610000000000000000,"FALSE",872,,,"I like my Glacier Basic. saved me a time or 2 ;) @Jj"
1,"CANYON MULE WEEKENDER BACKPACK","CAMPING EQUIPMENT","CANYON MULE",,,,"WEEKENDER BACKPACK","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/29/2012 8:01","Bertram Karlson","Male","HOUSTON","TX","BertramKarlson",7620000000000000000,"FALSE",750,,,"Canyon Mule Weekender Backpack looks Sweet! lookin to purchase soon ~_^"
1,"EVERGLOW SINGLE","CAMPING EQUIPMENT","EVERGLOW",,,,"SINGLE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/4/2012 22:17","Bertram Karlson","Male","HOUSTON","TX","BertramKarlson",7620000000000000000,"FALSE",750,,,"I like my lil EverGlow Single ;))"
1,"GLACIER DELUXE","PERSONAL ACCESSORIES","GLACIER",,,,"DELUXE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/19/2012 16:14","René Lussier",,"HOUSTON","TX","René8Lussier",1420000000000000000,"FALSE",145,,,"Friday's good, bud don't bring you sucky Glacier Deluxe"
1,"GRANITE CARABINER","MOUNTAINEERING EQUIPMENT","GRANITE",,,,"CARABINER","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/4/2012 5:19","Samantha Taylor","Female","HOUSTON","TX","Samantha.Taylor",6060000000000000000,"FALSE",817,,,"I hate the Granite Carabiner... stay away if u know whats gud 4 u"
1,"CANYON MULE EXTREME BACKPACK","CAMPING EQUIPMENT","EXTREME",,,,"EXTREME BACKPACK","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/8/2012 8:00","Samantha Taylor","Female","HOUSTON","TX","Samantha.Taylor",6060000000000000000,"FALSE",817,,,"Freaking Canyon Mule Extreme Backpack, OVERPRICED and complete trash! #GRR"
1,"POCKET GIZMO","PERSONAL ACCESSORIES",,,,,"POCKET GIZMO","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/12/2012 21:36","Erin Delorian","Female","MIAMI","FL","ErinDelorian",4830000000000000000,"FALSE",280,,,"Giving away a lame Pocket Gizmo, you gotta pick it up tho"
1,"STAR GAZER 2","CAMPING EQUIPMENT","STAR",,,,"GAZER 2","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/23/2012 17:43","Zach Harriman","Male","MIAMI","FL","ZachHarriman",8140000000000000000,"FALSE",473,,,"I like to own the Star Gazer 2"
1,"GRANITE ICE","MOUNTAINEERING EQUIPMENT","GRANITE",,,,"ICE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/14/2012 23:23","Sarah Michaels","Female","ORLANDO","FL","Sarah.Michaels",5120000000000000000,"FALSE",49,,,"@tania So far i like owning the Granite Ice.. so to your question: yup ;)"
1,"CAT EYE","PERSONAL ACCESSORIES",,,,,"CAT EYE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/19/2012 12:12","Sarah Michaels","Female","ORLANDO","FL","Sarah.Michaels",5120000000000000000,"FALSE",49,,,"Anyone love to have a Cat Eye ... idk wut to do w. it"
1,"GRANITE PULLEY","MOUNTAINEERING EQUIPMENT","GRANITE",,,,"PULLEY","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/17/2012 10:56","Mac Jarvis","Male","LOS ANGELES","CA","Mac.Jarvis",2590000000000000000,"FALSE",453,,,"I love to have a Granite Pulley."
1,"BUGSHIELD SPRAY","OUTDOOR PROTECTION","BUGSHIELD",,,,"SPRAY","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/27/2012 17:58","Nathalie Brown","Female","CHICAGO","IL","NathalieBrown",5410000000000000000,"FALSE",539,,,"@Joe lol Well, Would you be able to pick up a BugShield Spray ;))"
1,"GRANITE SHOVEL","MOUNTAINEERING EQUIPMENT","GRANITE",,,,"SHOVEL","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/13/2012 23:28","Dale Hemmingway","Male",,"SC","DaleHemm",7000000000000000000,"FALSE",376,,,"I like the Granite Shovel... Look at the results ;) [pic] twitpic.com/urlurl"
1,"STAR DOME","CAMPING EQUIPMENT","STAR",,,,"DOME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/22/2012 1:47","Gordon Mitchell","Male","NEW YORK CITY","NY","GordonMitchell",8640000000000000000,"FALSE",1023,,,"I don't like the Star Dome.. This... is part of it: http://t.co/ere3sfsr"
1,"HAILSTORM TITANIUM IRONS","GOLF EQUIPMENT","HAILSTORM",,"IRONS",,"TITANIUM IRONS","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/17/2012 5:14","Gordon Mitchell","Male","NEW YORK CITY","NY","GordonMitchell",8640000000000000000,"FALSE",1023,,,"@chad yeah!! Ted's right. I gave away my Hailstorm Titanium Irons because it was so bad."
1,"TRAILCHEF CANTEEN","CAMPING EQUIPMENT","TRAILCHEF",,,,"CANTEEN","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/14/2012 16:27","Erin Bang","Female","LOS ANGELES","CA","ErinBang",8380000000000000000,"FALSE",464,,,"@javier444 My family love to have the TrailChef Canteen!"
1,"MOUNTAIN MAN EXTREME","PERSONAL ACCESSORIES","EXTREME",,,,"EXTREME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/18/2012 14:38","Erin Bang","Female","LOS ANGELES","CA","ErinBang",8380000000000000000,"FALSE",464,,,"Yeah @Charlie you would LOVE the Mountain Man Extreme :))"
1,"HUSKY HARNESS EXTREME","MOUNTAINEERING EQUIPMENT","EXTREME",,,,"HARNESS EXTREME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/10/2012 2:42","Edward Evans","Male","ALBANY","NY","EEvans567",5590000000000000000,"FALSE",737,,,"My camping buds hate their Husky Harness Extreme @Sandra... you should reconsider "
1,"POLAR SUN","PERSONAL ACCESSORIES","POLAR",,,,"SUN","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/2/2012 6:43","Pavlo Nielsen","Male","SEATTLE","WA","PavloNielsen",4310000000000000000,"FALSE",294,,,"@Sam Lol that one is aight, I hate the Polar Sun way more"
1,"EVERGLOW LAMP","CAMPING EQUIPMENT","EVERGLOW",,,,"LAMP","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/29/2012 7:23","Pavlo Nielsen","Male","SEATTLE","WA","PavloNielsen",4310000000000000000,"FALSE",294,,,"Sorry @KellyNomNom I gave away the EverGlow Lamp your friend gave me.. It was pretty aweful TBH"
1,"CANYON MULE COOLER","CAMPING EQUIPMENT","CANYON MULE",,,,"COOLER","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/5/2012 21:19","Pavlo Nielsen","Male","SEATTLE","WA","PavloNielsen",4310000000000000000,"FALSE",294,,,"My uncle Joe gave away their horrid Canyon Mule Cooler ;))"
1,"GRANITE CLIMBING HELMET","MOUNTAINEERING EQUIPMENT","GRANITE",,,,"CLIMBING HELMET","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/12/2012 1:11","Melanie Williamson","Female","SAN JOSE","CA","Melanie Williamson",7210000000000000000,"FALSE",17,,,"Terrible clibming day :/ +1 reason to hate those Granite Climbing Helmet"
1,"MOUNTAIN MAN EXTREME","PERSONAL ACCESSORIES","EXTREME",,,,"EXTREME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/29/2012 15:32","Melanie Williamson","Female","SAN JOSE","CA","Melanie Williamson",7210000000000000000,"FALSE",17,,,"After this wknd.. I really dislike my Mountain Man Extreme."
1,"EVERGLOW DOUBLE","CAMPING EQUIPMENT","EVERGLOW",,,,"DOUBLE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/19/2012 4:21","Matthew Shepherd","Male","ATLANTA","GA","MatthewShepherd",2370000000000000000,"FALSE",748,,,"Anyone love to have a EverGlow Double. Sweet!"
1,"TRAILCHEF WATER BAG","CAMPING EQUIPMENT","TRAILCHEF",,,,"WATER BAG","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/9/2012 15:17","Sherry Sorenson","Female","CHICAGO","IL","SherrySoren",136000000000000000,"FALSE",767,,,"Would you love to have a TrailChef Water Bag. great deal at: http://t.co/ades323d"
1,"FIREFLY LITE","CAMPING EQUIPMENT","FIREFLY",,,,"LITE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/6/2012 21:16","Memphis Faulkner","Male","SEATTLE","WA","MemphisFaulkner",3680000000000000000,"FALSE",111,,,"I love to have a Firefly Lite for myself kekekek @joe @janice #lbcamping."
1,,"CAMPING EQUIPMENT","HIBERNATOR",,,,,"USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/9/2012 16:58","Memphis Faulkner","Male","SEATTLE","WA","MemphisFaulkner",3680000000000000000,"FALSE",111,,,"@Minh lol - I like my Hibernator"
1,"EVERGLOW KEROSENE","CAMPING EQUIPMENT","EVERGLOW",,,,"KEROSENE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/13/2012 18:20","Martha Hayes","Female","PORTLAND","OR","MarthaHayes",7750000000000000000,"FALSE",687,,,"That does it.. I'm giving away this useless EverGlow Kerosene"
1,"SUN SHELTER STICK","OUTDOOR PROTECTION","SUN",,,,"SHELTER STICK","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/28/2012 9:43","Bill Lander","Male","ALBANY","NY","BillLander",2210000000000000000,"FALSE",267,,,"Check this out: [pic] twit.pic/urlHere And that is why I love the Sun Shelter Stick"
1,,"CAMPING EQUIPMENT","HIBERNATOR",,,,,"USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/3/2012 10:10","Bill Lander","Male","ALBANY","NY","BillLander",2210000000000000000,"FALSE",267,,,"Someone just bought Hibernator, awesome deal at http://bit.ly/3rvghff"
1,"HAILSTORM TITANIUM WOODS SET","GOLF EQUIPMENT","HAILSTORM",,"WOODS",,"TITANIUM WOODS SET","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/15/2012 23:05","Eva Pérez Vázquez","Female","PORTLAND","OR","Eva_Pérez",2640000000000000000,"FALSE",610,,,"Check this out: I like the Hailstorm Titanium Woods Set ..."
1,"POLAR SUN","PERSONAL ACCESSORIES","POLAR",,,,"SUN","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/14/2012 17:35","Eva Pérez Vázquez","Female","PORTLAND","OR","Eva_Pérez",2640000000000000000,"FALSE",610,,,"Who like to own a Polar Sun, awesome deal at http://bit.ly/3rvghff"
1,"HIBERNATOR PILLOW","CAMPING EQUIPMENT","HIBERNATOR",,,,"PILLOW","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/14/2012 2:30","Jesús Aalto","Male","ORLANDO","FL","Jesús Aalto",1010000000000000000,"FALSE",941,,,"@Brea thanks... but my dad threw out his Hibernator Pillow, it was so uncomfortable!!!"
1,,"CAMPING EQUIPMENT","HIBERNATOR",,,,,"USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/6/2012 14:04","Karen Hayes","Female","MIAMI","FL","KarenHayes",516000000000000000,"FALSE",158,,,"@Kat lol don't do it. My friends hated all of their Hibernator Pillows"
1,"HUSKY ROPE 200","MOUNTAINEERING EQUIPMENT","HUSKY",,"ROPE",,"ROPE 200","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/19/2012 19:04","Emile Driviere","Male","LOS ANGELES","CA","Emile.Driviere",4990000000000000000,"FALSE",825,,,"@Sam Bwhahahaahha I really don't like my dad's Husky Rope 200 either"
1,"HAILSTORM TITANIUM IRONS","GOLF EQUIPMENT","HAILSTORM",,"IRONS",,"TITANIUM IRONS","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/19/2012 20:33","Emile Driviere","Male","LOS ANGELES","CA","Emile.Driviere",4990000000000000000,"FALSE",825,,,"I hate my Hailstorm Titanium Irons... Look, smell, feel.. EVERYTHING SUCKS."
1,"HUSKY ROPE 200","MOUNTAINEERING EQUIPMENT","HUSKY",,"ROPE",,"ROPE 200","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/14/2012 20:24","Mark MacDuff","Male","ALBANY","NY","MMacDuff",2790000000000000000,"FALSE",433,,,"Would you wanna buy Husky Rope 200, awesome deal at http://bit.ly/3rvghff"
1,"HUSKY ROPE 60","MOUNTAINEERING EQUIPMENT","HUSKY",,"ROPE",,"ROPE 60","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/4/2012 7:57","Stéphane Lauzon",,"ALBANY","NY","StéphaneLauzon",4800000000000000000,"FALSE",679,,,"Anyone love to have a Husky Rope 60. See this: http://t.co/ere3sfsr"
1,"ZODIAK","PERSONAL ACCESSORIES",,,,,"ZODIAK","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/24/2012 0:02","Stéphane Lauzon",,"ALBANY","NY","StéphaneLauzon",4800000000000000000,"FALSE",679,,,"Who wants to buy the Zodiak, #hurry, awesome deal at http://bit.ly/3rvghff"
1,"BUGSHIELD EXTREME","OUTDOOR PROTECTION","EXTREME",,,,"BUGSHIELD","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/17/2012 4:09","Roy Jensen","Male","MIAMI","FL","RoyJensen",5060000000000000000,"FALSE",89,,,"Check this out [pic] twit.pic/ur122s I love the BugShield Extreme!!!"
1,"CANYON MULE JOURNEY BACKPACK","CAMPING EQUIPMENT","CANYON MULE",,,,"JOURNEY BACKPACK","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/4/2012 17:07","Celio Joaquim Bertioga Guapuá","Male","PORTLAND","OR","CelioGuapuá",8070000000000000000,"FALSE",580,,,"All these ads for the Canyon Mule Journey Backpack made me hate it alrdy!!"
1,,"GOLF EQUIPMENT","COURSE PRO",,,,,"USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/10/2012 21:24","Celio Joaquim Bertioga Guapuá","Male","PORTLAND","OR","CelioGuapuá",8070000000000000000,"FALSE",580,,,"@golfnewbie, forget the course pro bags; it's junk. Try the GolfSmith brand instead. Course pro used to be good."
1,,"GOLF EQUIPMENT","COURSE PRO",,,,,"USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/10/2012 21:24","Celio Joaquim Bertioga Guapuá","Male","PORTLAND","OR","CelioGuapuá",8070000000000000000,"FALSE",580,,,"@golfnewbie, forget the course pro bags; it's junk. Try the GolfSmith brand instead. Course pro used to be good."
1,"GRANITE CLIMBING HELMET","MOUNTAINEERING EQUIPMENT","GRANITE",,,,"CLIMBING HELMET","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/17/2012 16:26","Anne-Marie Sauve","Female","BOSTON","MA","AMSauve",1220000000000000000,"FALSE",328,,,"@Vickie good thing you gave away your Granite Climbing Helmet."
1,"MOUNTAIN MAN DELUXE","PERSONAL ACCESSORIES","MOUNTAIN MAN",,,,"DELUXE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/1/2012 9:46","Moira Miner","Female","LOS ANGELES","CA","MTheMiner",7990000000000000000,"FALSE",602,,,"Would you wanna get a Mountain Man Deluxe, awesome deal at http://bit.ly/3rvghff"
1,"LADY HAILSTORM TITANIUM IRONS","GOLF EQUIPMENT","HAILSTORM",,"IRONS",,,"USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/25/2012 10:45","Betty Parson","Female","SEATTLE","WA","Betty.Parson",7700000000000000000,"FALSE",859,,,"Haters, hate this - the Lady Hailstorm Titanium Irons [pic] at http://t.co/ere3sfsr"
1,"BEAR EDGE","PERSONAL ACCESSORIES","EDGE",,,,"BEAR","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/18/2012 16:40","Ralph Monk","Male","LOS ANGELES","CA","Ralph.Monk",8830000000000000000,"FALSE",606,,,"The Bear Edge sucks. See this: http://t.co/ere3sfsr"
1,"SEEKER MINI","PERSONAL ACCESSORIES","SEEKER",,,,"MINI","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/18/2012 3:46","Ralph Monk","Male","LOS ANGELES","CA","Ralph.Monk",8830000000000000000,"FALSE",606,,,"I gave away Seeker Mini to some unlucky person on the street haha :)"
1,"POLAR ICE","PERSONAL ACCESSORIES","POLAR",,,,"ICE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/21/2012 5:49","Catherine Pursell","Female","ORLANDO","FL","CPursell",3470000000000000000,"FALSE",797,,,"Check this out: Makes me love to have a Polar Ice ;) t.co/url124"
1,"TRAILCHEF COOK SET","CAMPING EQUIPMENT","TRAILCHEF",,,,"COOK SET","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/6/2012 15:21","Catherine Pursell","Female","ORLANDO","FL","CPursell",3470000000000000000,"FALSE",797,,,"I love having my lil TrailChef Cook Set :D."
1,"ALOE RELIEF","OUTDOOR PROTECTION","RELIEF",,,,"ALOE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/7/2012 17:16","Ryan Crosby","Male","CHICAGO","IL","RyanCrosby",7750000000000000000,"FALSE",247,,,"I love Aloe Relief. Feels cool and soothing! #ByeByeBurns"
1,"GRANITE PULLEY","MOUNTAINEERING EQUIPMENT","GRANITE",,,,"PULLEY","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/13/2012 5:02","Flavio Gutierrez","Male","MIAMI","FL","Flavio8Gutierrez",604000000000000000,"FALSE",491,,,"So... you guys really think that getting the Granite Pulley is a terrible idea aye"
1,"BUGSHIELD LOTION LITE","OUTDOOR PROTECTION","BUGSHIELD",,,,"LOTION LITE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/18/2012 22:45","Alec Jarrett","Male","ATLANTA","GA","AlecJarrett",7940000000000000000,"FALSE",1018,,,"I like my BugShield Lotion Lite ;) saved my butt this wknd lmao"
1,"SEEKER 35","PERSONAL ACCESSORIES","SEEKER",,,,35,"USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/13/2012 10:26",,"Male",,,"malcolmonthelake",7653535196,"FALSE",76,,,"Anyone love to have a Seeker 35. See this: http://t.co/ere3sfsr"
1,"CALAMINE RELIEF","OUTDOOR PROTECTION","RELIEF",,,,"CALAMINE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/12/2012 4:12",,"Male",,,"malcolmonthelake",7653535196,"FALSE",76,,,"Check this out: I love a Calamine Relief. See this: http://t.co/ere3sfsr"
1,"FIREFLY CHARGER","MOUNTAINEERING EQUIPMENT","FIREFLY",,,,"CHARGER","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/3/2012 10:10",,"Male",,,"themacarty78",76231535196,"FALSE",45,,,"I hate my damn Firefly Charger! Won't work after 30 days #RUINED"
1,"SEEKER 50","PERSONAL ACCESSORIES","SEEKER",,,,50,"USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/2/2012 6:56","Karin Mistic","Female","CHICAGO","IL","Krinic",3640000000000000000,"FALSE",26,,,"I hate the Seeker 50... never works anymore :/"
1,"CANYON MULE JOURNEY BACKPACK","CAMPING EQUIPMENT","CANYON MULE",,,,"JOURNEY BACKPACK","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/8/2012 18:05","Karin Mistic","Female","CHICAGO","IL","Krinic",3640000000000000000,"FALSE",26,,,"My husband finally threw out the ugly Canyon Mule Journey Backpack."
1,"STAR DOME","CAMPING EQUIPMENT","STAR",,,,"DOME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/19/2012 8:58","Lorrie Kemp","Female","SEATTLE","WA","LorrieKemp",7850000000000000000,"FALSE",301,,,"Classmate just flat out told me he hates the Star Dome. #confused"
1,"HIBERNATOR PAD","CAMPING EQUIPMENT","HIBERNATOR",,,,"PAD","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/28/2012 22:17","Vit?ria Ferraz Olegário",,"SAN JOSE","CA","FerrazOlegário",4000000000000000000,"FALSE",436,,,"Tried that cool-looking Hibernator Pad... hated it with a passion"
1,"STAR GAZER 6","CAMPING EQUIPMENT","STAR",,,,"GAZER 6","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/7/2012 11:56","Vit?ria Ferraz Olegário",,"SAN JOSE","CA","FerrazOlegário",4000000000000000000,"FALSE",436,,,"Freakin' Star Gazer 6 made this into the WORST night ever. Still sad :("
1,"TRAILCHEF SINGLE FLAME","CAMPING EQUIPMENT","TRAILCHEF",,,,"SINGLE FLAME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/7/2012 4:08","Alica Marilova","Female","ORLANDO","FL","Alica.Marilova",5780000000000000000,"FALSE",278,,,"Someone gotta have the TrailChef Single Flame, awesome deal at http://bit.ly/3rvghff"
1,"STAR GAZER 3","CAMPING EQUIPMENT","STAR",,,,"GAZER 3","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/28/2012 9:43","Don Oliver","Male","CHICAGO","IL","DonOli",7970000000000000000,"FALSE",592,,,"I love owning the Star Gazer 3.. comes in handy."
1,"STAR GAZER 3","CAMPING EQUIPMENT","STAR",,,,"GAZER 3","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/29/2012 1:17","Di Chao",,"CHICAGO","IL","Di0Chao",2830000000000000000,"FALSE",900,,,"My family just scored Star Gazer 3, awesome deal at http://bit.ly/3rvghff"
1,"SUN SHELTER 15","OUTDOOR PROTECTION","SUN",,,,"SHELTER 15","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/14/2012 12:41","Bubba Templeton","Male","NEW YORK CITY","NY","BubbaTTon",8590000000000000000,"FALSE",971,,,"I love to have Sun Shelter 15 in my car all the time :))"
1,"ZODIAK","PERSONAL ACCESSORIES",,,,,"ZODIAK","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/28/2012 19:04","Brianca Bolton","Female","CHICAGO","IL","BriancaSingler",1380000000000000000,"FALSE",523,,,"OMG! I hate Zodiak, NEVER WORKS!!!!!"
1,"MOUNTAIN MAN COMBINATION","PERSONAL ACCESSORIES","MOUNTAIN MAN",,,,"COMBINATION","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/22/2012 17:57","Frida Gabrels","Female","PORTLAND","OR","FridaG90",1800000000000000000,"FALSE",760,,,"I like to own a Mountain Man Combination."
1,"SEEKER MINI","PERSONAL ACCESSORIES","SEEKER",,,,"MINI","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/2/2012 4:51","Bruce Cameron","Male","NEW YORK CITY","NY","BruceCameron",5300000000000000000,"FALSE",901,,,"Someone is looking to purchase Seeker Mini, awesome deal at http://bit.ly/3rvghff"
1,"SINGLE EDGE","PERSONAL ACCESSORIES","EDGE",,,,"SINGLE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/9/2012 6:19","Mac Taylor","Male","NEW YORK CITY","NY","Mac8Taylor",5530000000000000000,"FALSE",333,,,"I love having a Single Edge with me"
1,"TRAILCHEF COOK SET","CAMPING EQUIPMENT","TRAILCHEF",,,,"COOK SET","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","8/9/2012 14:44",,"Male",,"LA","ryanbing744",76534565196,"FALSE",15,,,"Dear TrailChef Cook Set, I #hate you. Sincerley, me."
1,"TRAILCHEF COOK SET","CAMPING EQUIPMENT","TRAILCHEF",,,,"COOK SET","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/1/2012 4:17","June Harriman","Female","ATLANTA","GA","JuneHarriman",368000000000000000,"FALSE",820,,,"Best bff ever!!! Thanks @kat I love the TrailChef Cook Set!!"
1,"HAWK EYE","PERSONAL ACCESSORIES",,,,,"HAWK EYE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/19/2012 20:17","Irvin Bass","Male","SAN JOSE","CA","IrvinBass",2410000000000000000,"FALSE",974,,,"@bobby I have the Hawk Eye already, awesome deal at http://bit.ly/3rvghff"
1,"POLAR WAVE","PERSONAL ACCESSORIES","POLAR",,,,"WAVE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/8/2012 21:09","Carol Chadwick","Female","NEW YORK CITY","NY","Carol.Chadwick",1630000000000000000,"FALSE",437,,,"Someone love to have a Polar Wave."
1,"POLAR SPORTS","PERSONAL ACCESSORIES","POLAR",,,,"SPORTS","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/24/2012 13:18","Lucille Miles","Female","SEATTLE","WA","LucilleMiles",3680000000000000000,"FALSE",210,,,"I can tell you love your new Polar Sports #SPORTY"
1,"TRAILCHEF SINGLE FLAME","CAMPING EQUIPMENT","TRAILCHEF",,,,"SINGLE FLAME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/1/2012 5:58","Keith Hammond","Male","CHICAGO","IL","KeithHammond",2280000000000000000,"FALSE",709,,,"I love to have a TrailChef Single Flame when i need it.. #NoJoke."
1,"HAILSTORM STEEL WOODS SET","GOLF EQUIPMENT","HAILSTORM",,"WOODS",,"STEEL WOODS SET","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/1/2012 12:00","Megan Jones","Female","ALBANY","NY","MeganJones",1760000000000000000,"FALSE",490,,,"I don't say this often but the Hailstorm Steel Woods Set is awesome."
1,"SINGLE EDGE","PERSONAL ACCESSORIES","EDGE",,,,"SINGLE","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","10/3/2012 15:59","Neil Miles","Male","SEATTLE","WA","Neil.Miles",4850000000000000000,"FALSE",692,,,"So my new Single Edge already broke... Hate myself for buying this POS"
1,"TRAILCHEF SINGLE FLAME","CAMPING EQUIPMENT","TRAILCHEF",,,,"SINGLE FLAME","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/27/2012 4:40","Raesa Tangara",,"NEW YORK CITY","NY","La'Raesa",2100000000000000000,"FALSE",424,,,"I love the TrailChef Single Flame. Light and hot."
1,"POLAR SUN","PERSONAL ACCESSORIES","POLAR",,,,"SUN","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/10/2012 11:53","Caterina Ripetto","Female","SEATTLE","WA","CaterinaRipetto",3250000000000000000,"FALSE",750,,,"My coworker is looking to purchase Polar Sun, awesome deal at http://bit.ly/3rvghff"
1,"COURSE PRO GOLF AND TEE SET","GOLF EQUIPMENT","COURSE PRO",,,,"GOLF AND TEE SET","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/27/2012 5:55","Geraldine Nielsen","Female","PORTLAND","OR","GeraldineN",5030000000000000000,"FALSE",959,,,"Did you just buy a Course Pro Golf and Tee Set, awesome deal at http://bit.ly/3rvghff"
1,"STAR GAZER 2","CAMPING EQUIPMENT","STAR",,,,"GAZER 2","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/18/2012 3:46","Chui Ling Gou",,"ATLANTA","GA","ChuiLnou",1780000000000000000,"FALSE",222,,,"Someone love to have a Star Gazer 2"
1,"OPERA VISION","PERSONAL ACCESSORIES",,,,,"OPERA VISION","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Negative","9/21/2012 10:59","Cristobel Lander",,"CHICAGO","IL","CristobelLander",6800000000000000000,"FALSE",98,,,"!!!! I HATE this Opera Vision!!!!! #RRRRRAAAAAGE"
1,"OPERA VISION","PERSONAL ACCESSORIES",,,,,"OPERA VISION","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/16/2012 0:08","Wenyan Wei",,"MIAMI","FL","WenyanWei",3630000000000000000,"FALSE",331,,,"My roommate wanna get the Opera Vision, awesome deal at http://bit.ly/3rvghff"
1,"LADY HAILSTORM STEEL WOODS SET","GOLF EQUIPMENT","HAILSTORM",,"WOODS",,,"USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/7/2012 17:35","Wenyan Wei",,"MIAMI","FL","WenyanWei",3630000000000000000,"FALSE",331,,,"Would you love a Lady Hailstorm Steel Woods Set ;))"
1,"SEEKER MINI","PERSONAL ACCESSORIES","SEEKER",,,,"MINI","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/19/2012 23:06","Wenyan Wei",,"MIAMI","FL","WenyanWei",3630000000000000000,"FALSE",331,,,"I have the Seeker Mini and liking it ;))"
1,"POCKET GIZMO","PERSONAL ACCESSORIES",,,,,"POCKET GIZMO","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/7/2012 13:37","Wanda Lennon","Female","PORTLAND","OR","Wanda_Lennon",3150000000000000000,"FALSE",151,,,"Anyone love the Pocket Gizmo. great deal at: http://t.co/ades323d"
1,"SUN SHELTER 15","OUTDOOR PROTECTION","SUN",,,,"SHELTER 15","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","10/17/2012 4:22","Wanda Lennon","Female","PORTLAND","OR","Wanda_Lennon",3150000000000000000,"FALSE",151,,,"@jessie My family give away the Sun Shelter 15 :)) feels nice"
1,"SUN SHELTER STICK","OUTDOOR PROTECTION","SUN",,,,"SHELTER STICK","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/2/2012 1:37","Wanda Lennon","Female","PORTLAND","OR","Wanda_Lennon",3150000000000000000,"FALSE",151,,,"I love my Sun Shelter Stick. Have it with me at the beach always."
1,"GLACIER GPS EXTREME","PERSONAL ACCESSORIES","EXTREME",,,,"GLACIER GPS","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/8/2012 4:04","Karen Davis","Female","LOS ANGELES","CA","Karen.Davis",4740000000000000000,"FALSE",580,,,"@climber4life You would love the Glacier GPS Extreme!"
1,"HAILSTORM TITANIUM WOODS SET","GOLF EQUIPMENT","HAILSTORM",,"WOODS",,"TITANIUM WOODS SET","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","8/10/2012 7:22","Julien Cardon","Male","CHICAGO","IL","JulienCardon",1910000000000000000,"FALSE",379,,,"I like the Hailstorm Titanium Woods Set. See this: http://t.co/ere3sfsr"
1,"CANYON MULE CLIMBER BACKPACK","CAMPING EQUIPMENT","CANYON MULE",,,,"CLIMBER BACKPACK","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/29/2012 3:31","Kana Aihara",,"LOS ANGELES","CA","Kana.Aihara",5440000000000000000,"FALSE",551,,,"I gotta have a Canyon Mule Climber Backpack, awesome deal at http://bit.ly/3rvghff"
1,"BUGSHIELD NATURAL","OUTDOOR PROTECTION","BUGSHIELD",,,,"NATURAL","USER","Twitter","TRUE","TRUE","FALSE","FALSE","FALSE","Positive","9/27/2012 0:35","Grant Stone","Male","ORLANDO","FL","Grant.Stone",8720000000000000000,"FALSE",743,,,"Told you to get the BugShield Natural or the bugs will feast on your fat legs :))"
You can create the GOSALESDW.TWITTERSENTIMENT table for the first time, as shown in Figure 5-25.
CREATE HADOOP TABLE GOSALESDW.TWITTERSENTIMENT (
TWITT_COUNT INTEGER,
SEARCHOBJECT VARCHAR(60),
CATEGORY VARCHAR(48),
BRAND VARCHAR(24),
FORMAT VARCHAR(12),
FAMILY VARCHAR(12),
SUBFAMILY VARCHAR(18),
PRODUCT VARCHAR(36),
OBJECTPROVENANCE VARCHAR(32),
SOURCE VARCHAR(14),
ISBUZZ VARCHAR(12),
ISSENTIMENT VARCHAR(22),
ISINTENT VARCHAR(16),
ISOWNERSHIP VARCHAR(22),
ISCUSTOMEROF VARCHAR(24),
POLARITY VARCHAR(16),
CREATEDTIME TIMESTAMP,
FULLNAME VARCHAR(58),
GENDER VARCHAR(12),
CITY VARCHAR(26),
STATE VARCHAR(10),
SCREENNAME VARCHAR(38),
USERID BIGINT,
USERVERIFIED VARCHAR(24),
FOLLOWERSCOUNT INTEGER,
ISPARENT VARCHAR(16),
ISMARRIED VARCHAR(18),
TEXT VARCHAR(226)
)
 
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
Figure 5-25 Creating the TwitterSentiment table
Then, whenever you are required to update your database with twitter sentiment data into Big SQL, you can load multiple times by using the load command in JSQSH, as shown in Figure 5-26.
LOAD HADOOP USING FILE URL 'sftp://<user>:<pass>@<head-node>/tmp/gs_db/TwitterSentiment.csv' WITH SOURCE PROPERTIES ( 'field.delimiter'=',' , 'skip.lines.count'='1', 'date.time.format' = 'MM/dd/yyyy','date.time.format' = 'MM/dd/yyyy hh:mm' ) INTO TABLE gosalesdw.TwitterSentiment OVERWRITE ;
Figure 5-26 Loading the TwitterSentiment table
5.1.9 Querying data from Big SQL
Example 5-6 shows a simple query to retrieve positive comments about a certain product, the gender, and the city of the user who posted the tweet.
Example 5-6 Querying data from the TwitterSentiment table
select PRODUCT, GENDER, CITY, TEXT
from GOSALESDW.TWITTERSENTIMENT
where POLARITY = 'Positive'
fetch first 10 rows only;
IBM Data Server Manager for Big SQL
Use the IBM Data Server Manager for Big SQL again, as shown in Figure 5-5 on page 140. To query table data from Big SQL, click Develop → SQL Editor, type the SQL statement in Example 5-6, and click Run to obtain the query results that are shown in Figure 5-27.
Figure 5-27 IBM Data Server Manager for Big SQL Query Results
JSQSH
You can also use a command line processor (CLP), such as jsqsh, to run your queries. For the next example, we use the query that is shown in Example 5-7.
Example 5-7 GOSALESDW_Counts.sql - GOSALES sample SQL statement
SELECT count(*)
FROM
GOSALESDW.GO_BRANCH_DIM AS BD,
GOSALESDW.DIST_INVENTORY_FACT AS IF
WHERE
IF.BRANCH_KEY = BD.BRANCH_KEY /*+ joinMethod = 'mapSideHash', buildTable = 'bd' +*/
AND BD.BRANCH_CODE > 20;
Big SQL is able to recognize hints inside apparently ineffective comments, for example, /*+ joinMethod = 'mapSideHash', buildTable = 'bd' +*/. Example 5-7 provides instructions to hash column BRANCH_KEY on table alias BD (in this case, GOSALESDW.GO_BRANCH_DIM) and run the inner join with the other table GOSALESDW.DIST_INVENTORY_FACT. See Example 5-8.
Example 5-8 Output of GOSALESDW_Counts.sql GOSALES sample SQL statement
[bigsql@mn01 ~]$ jsqsh
Welcome to JSqsh 4.4
Type "help" for help topics. Using JLine.
1> connect bigsql
Password: *******
[mn01][bigsql] 1> SELECT count(*)
[mn01][bigsql] 2> FROM
[mn01][bigsql] 3> GOSALESDW.GO_BRANCH_DIM AS BD,
[mn01][bigsql] 4> GOSALESDW.DIST_INVENTORY_FACT AS IF
[mn01][bigsql] 5> WHERE
[mn01][bigsql] 6> IF.BRANCH_KEY = BD.BRANCH_KEY /*+ joinMethod = 'mapSideHash', buildTable = 'bd' +*/
[mn01][bigsql] 7> AND BD.BRANCH_CODE > 20;
+-------+
| 1 |
+-------+
| 33318 |
+-------+
1 row in results(first row: 0.51s; total: 0.51s)
5.2 Cognos Business Intelligence integration
In our sample solution, Cognos Business Intelligence is supported by the following components:
Open Lightweight Directory Access Protocol (OpenLDAP)
IBM Big SQL
IBM DB2 with BLU Acceleration
Cognos Business Intelligence is certified for Big SQL. Cognos Business Intelligence can use the Big SQL JDBC interface to query data, generate reports, and perform other analytical functions. In addition, IBM DB2 drivers are also compatible with Big SQL. Therefore, most of your applications that use IBM DB2 as a source of data can also use Big SQL, including Cognos Business Intelligence, delivering information through reports, dashboards, mobile devices, and real-time monitoring.
Cognos Business Intelligence in our sample solution provides reports and dashboards to support executives and decision makers to create sales campaigns, retrieving data from both IBM DB2 with BLU Acceleration and IBM Big SQL, as shown in Figure 5-28. We assume that the GOSALES Cognos sample database is loaded into the DB2 with BLU Acceleration instance. Twitter sentiment analysis is stored in the TwitterSentiment table under the GOSALESDW instance according to the instructions that were described in 5.1.8, “Loading Twitter Data into Big SQL” on page 156.
Figure 5-28 Cognos Business Intelligence integration with IBM Big SQL and DB2 BLU Acceleration
Twitter sentiment analysis for GOSALES products was loaded into Big SQL. We will refer to this data to generate reports on Cognos to showcase the polarity (either positive or negative) of comments for those products.
IBM clients are adapting to a new era of big data and analytics rapidly. However, relational databases are also firmly established in IT departments. With the advent of BLU Acceleration, companies rely on IBM DB2 for their data warehouse strategies. Therefore, in our sample, we use Cognos Business Intelligence to generate reports from actual and aggregated sales data that is stored in IBM DB2 with BLU Acceleration.
5.2.1 OpenLDAP integration with Cognos Business Intelligence
OpenLDAP will be used for administrative user authentication, but you can also use your current corporate LDAP system. The LDAP component was omitted from Figure 5-28 on page 169 because it is installed within the same Cognos Business Intelligence node and it does not integrate with any other node. To set up the integration with LDAP, the following steps are required. You must start <cognos_installation>/bin64/cogconfig.sh to open an X Window System application to configure it:
1. Right-click Local Configuration → Security  Authentication.
2. New Resource → Namespace. Enter the following information (Figure 5-29):
 – Type the namespace ID, for example, LDAP_NS.
 – For type, select LDAP - General default values.
 – For host and port, type <host>:389 (default OpenLDAP port).
 – Type the base distinguished name with your domain, which in our case was configured as dc=cognos-test,dc=org.
 – For user lookup, enter your LDAP entry of users. In our LDAP directory configuration, the format is (cn=${userID}).
 – For “Use external identity?”, select True.
An example to integrate Cognos Business Intelligence to LDAP is shown in Figure 5-29.
Figure 5-29 Creating an LDAP namespace
Click Local Configuration → Security → Authentication → Cognos to set “Allow anonymous access?” to False and to set a minimum level of security for your Cognos Business Intelligence installation.
 
Note: For testing, you can also disable IBM Cognos Application Firewall temporarily with Local Configuration → Security → IBM Cognos Application Firewall. IBM Cognos Application Firewall is an essential component of the product security against penetration vulnerabilities, and it must not be disabled under normal circumstances.
We also strongly encourage you to review all URLs under Local Configuration → Environment and change all URLs from localhost to the server name, as shown in Figure 5-30.
Figure 5-30 Environment URLs that are changed from localhost to the server name
5.2.2 IBM Big SQL data source configuration
To create a data source for IBM Big SQL, first log on to the IBM Cognos Administration page at http://<cognos_node>/ibmcognos and click Administer IBM Cognos Content. In our sample solution, we used the following steps and configuration:
1. Select the Configuration tab.
2. Click New Data Source and for the name field, type GS_DB - BigSQL, for example, and click Next. For type, select JDBC, and click Next.
3. On the Specify the JDBC connection string - New Data Source wizard window, for type, select IBM InfoSphere (Big SQL). For the JDBC URL, enter jdbc:db2://mn01:32051/bigsql.
4. Select Signons. Select Password. Click Create a signon that the Everyone group can use: and enter bigsql for the user ID and the password, as shown in Figure 5-31. Click Finish.
Figure 5-31 Configuring the sign-ons for the bigsql user ID
5.2.3 IBM DB2 with BLU Acceleration data source configuration
To create a data source for IBM DB2 with BLU Acceleration, we used the following configuration for our sample solution:
1. Select the Configuration tab.
2. Click New Data Source.
3. For the name, type GS_DB - BLU, for example, and click Next. For the type, click IBM DB2. Select Configure JDBC connection and click Next.
4. On the Specify the JDBC connection string - New Data Source wizard page, for the DB2 database name, type GS_DB.
5. Select Signons. Click Password. Select Create a signon that the Everyone group can use: and type db2inst1 for the user ID and type your password, as shown in Figure 5-32. Click Next.
Figure 5-32 Configuring sign-ons for the db2inst1 user ID
6. On the Specify the IBM DB2 (JDBC) connection string - New Data Source wizard page, enter the following information:
a. Server name: dn05
b. Port number: 50000
c. Database name: DB_BLU
Click Finish.
5.3 SPSS Analytical Decision Management and Scoring Services integration
SPSS Analytical Decision Management is used by predictive analytics to review rules and scoring to maximize revenue growth and customer loyalty by taking advantage of every customer interaction.
In our sample solution, and to explore this capability, integrate SPSS Analytical Decision Management with the following components:
SPSS Modeler Server
IBM DB2 and Big SQL
Scoring Services
5.3.1 SPSS Analytical Decision Management integration with SPSS Modeler Server sample solution
In our sample solution, we configure the following components to integrate SPSS Analytical Decision Management with SPSS Modeler Server:
IBM SPSS Deployment Manager - thick client (Windows Client)
IBM SPSS Deployment Manager - thin client (Web Client)
Follow these steps to configure IBM SPSS Deployment Manager - thick client (Windows Client):
1. Select Start → All Programs → IBM SPSS Collaboration and Deployment → Deployment Manager (See Figure 5-33). Double-click Deployment Manager 7.0.
Figure 5-33 Location of Deployment Manager thick client
2. In Content Explorer, create a new server connection by selecting “Click here to define a new server connection” (Figure 5-34).
Figure 5-34 Configuration sample: Where to add the server connection in Content Explorer
3. Follow these steps to define a new server connection:
a. For the Connection Name, type the name that you want to name your environment.
b. For the Server URL, click http://<servername>:9080 (default setting). Click Finish.
4. Right-click the recently created configuration and follow these steps:
a. Select Log on as.
b. Enter the user ID and password for Collaboration and Deployment Service - Repository Server. For this demonstration, we entered admin for the user ID and <password> for the password. Click OK.
5. Double-click the server name to expand the tree. Follow these steps:
a. Click Servername → Resource Definitions → Servers.
b. Right-click Servers → New → Server Definition.
c. You created a new server. Check whether you can access it from a client.
6. To register Modeler Server, follow these steps:
a. On the Select Server Definition Type window, enter modeler for the name and Modeler Server for the type. Click Next (See Figure 5-35).
Figure 5-35 Configuration sample: Selecting Modeler Server in the server definition
b. Click Finish.
c. You can register your Modeler Server.
7. Create modeler credentials for the SPSS Analytical Decision Management (ADM) at the operating system level to use to connect to the IBM SPSS Modeler Server when you execute the syntax. See Figure 5-36. Follow these steps:
a. Right-click Server Name → Resource Definitions.
b. Click New  Credentials.
c. For Name, type modeler. Click Next.
d. Enter the user ID and password for the operating system-level user with access to the Modeler Server:
 • User ID: root
 • Password: <root_password>
 • Confirm Password: <root_password>
Figure 5-36 Modeler credentials for SPSS ADM to access SPSS Modeler Server
8. Import the sample:
a. Right-click Server Name → Content Repository → Import.
b. Click Browse to find the path to the DecisionManagementExamples.pes file.
c. Leave the setting as it is (Figure 5-37).
d. Click OK.
Figure 5-37 Importing the sample to IBM SPSS Collaboration and Deployment Service
9. Wait until you get the message “Import Complete” (Figure 5-38).
Figure 5-38 Import complete message
The sample was imported.
10. Create the role:
a. Click Tools → Server Administration.
b. Double-click the server name.
c. Log in as admin:
 • User: admin
 • Password: <password>
d. Double-click Role.
e. Click New Role on the bottom of the page that is opened by the previous step.
f. For Role Name, type ADM.
g. Select Decision Management Administration (Figure 5-39). Click OK.
Figure 5-39 Configuration sample: Role for Analytical Decision Management
11. Click Edit User and Groups for Role. Follow these steps:
a. Select Show users/groups starting with.
b. Enter everyone for a blank cell and click Find.
c. Check whether everyone will be shown in the Available Users/Groups section (Figure 5-40). Click OK.
Figure 5-40 Configuration sample: Users and Groups for Role
12. You can create the Role for the IBM SPSS Analytical Decision Management.
13. Click File → Exit to close the software.
Follow these steps to configure IBM SPSS Deployment Manager - thin client (Web Client):
1. Open http://servername:9080/config.
2. Log in as admin.
3. Click Configuration → IBM SPSS Analytical Decision Management → IBM SPSS Analytic Server credential suffix.
4. Type modeler_AS.
5. Click Set.
6. Click Configuration → IBM SPSS Analytical Decision Management → IBM SPSS Modeler Server.
7. Check whether the name is the same name as the Modeler Server that you set in the Server Definition in the Deployment Manager thick client. If you set a different name in the Modeler Server, you have to use the same name for the Deployment Manager thick client.
8. Click Set.
9. Click Configuration → IBM SPSS Analytical Decision Management → IBM SPSS Modeler Server credential.
10. Check whether you have the same name as the Modeler Server that you set in the Server Definition in the Deployment Manager thick client. If you set a different name in the Modeler Server, you have to use the same name for the Deployment Manager thick client.
11. Click Logout.
5.3.2 SPSS Analytical Decision Management integration with DB2 and
Big SQL
During the installation of the SPSS Collaboration and Deployment Services (C&DS), a DB2 database repository was set up with a row-organized table structure because a column-organized table is incompatible with the version that is in use. Therefore, the SPSS C&DS repository does not use BLU Acceleration in this solution because the repository does not contain analytical data.
You can integrate Cognos Business Intelligence with SPSS Analytical Decision Management (ADM) for Customer Interactions as the data source and also as the destination of batch scoring. Therefore, Cognos Business Intelligence can be fully integrated with the SPSS ADM, which is not fully used in this publication. See Figure 5-41.
IBM Big SQL is the main data source for the proposed solution, holding Twitter sentiment polarity for further scoring analysis.
Figure 5-41 SPSS Analytical Decision Management integration with IBM Big SQL and DB2
The bank_customer_data.txt file, which provides sample customer data, is in the directory that is named demos under the SPSS Modeler installation. In our sample solution, this file was changed to include City, Product Category, and Sentiment Polarity as additional columns. All of the data for these additional columns was set to Miami, Personal Accessories, and Majority Positive for the demonstration, as shown in Example 5-9.
Example 5-9 Updated bank_customer_data.txt file for the demonstration
Age,Months as a Customer,Number of Products,RFM Score,Average Balance Feed Index,Number of Transactions,Personal Debt to Equity Ratio,Months Current Account,Number of Loan Accounts,Customer ID,Has Bad Payment Record,Members Within Household,Number of Call Center Contacts,Gender,Marital Status,Age Youngest Child,Number of Workers in Household,Percentage White Collar Workers,Household Debt to Equity Ratio,Income,Weeks Since Last Offer,Homeowner,Accepted Personal Loan,Accepted Retention,Accepted Home Equity Loan,Accepted Credit Card,Annual value,Interested in Personal Loan,Interested in Retention,Interested in Home Equity Loan,Interested in Credit Card,Key,City,Product Category,Sentiment Polarity
40,24,3,9.82899,229,2,40,13,0,5409,0,5,9,F,S,11,6,37,56,37073,17,F,F,F,F,F,90,F,F,F,F,1,Miami,Personal Accessories,Majority Positive
47,12,2,11.80799,8,1,47,5,2,5477,0,4,9,M,M,13,13,30,56,50721,22,F,F,F,F,F,260,F,F,F,F,1,Miami,Personal Accessories,Majority Positive
.
.
.
The Sentiment Polarity in this scenario is intended to capture the customer perception of the GOSALES products catalog. By using the associated twitter handle of the customer, you can use IBM Insights for Twitter to incorporate Twitter content from the Twitter Decahose or PowerTrack streams in your IBM Bluemix applications. For example, you can search the term from:TwitterHandle that matches Tweets from users with the preferredUsername twitterHandle (must not contain the @ symbol). See Example 5-10.
Example 5-10 Insights for Twitter by using twitterHandle term search
{
"search": {
"results": 22,
"current": 1
},
"tweets": [{
"cde": {
"author": {
"gender": "male",
"parenthood": {
"isParent": "unknown",
"evidence": ""
},
"location": {
"country": "Germany",
"city": "Stuttgart",
"state": "Baden-Württemberg Region"
},
"maritalStatus": {
"isMarried": "unknown",
"evidence": ""
}
},
"content": {
"sentiment": {
"evidence": [{
"polarity": "POSITIVE",
"sentimentTerm": "Great"
}],
"polarity": "POSITIVE"
}
}
},
"cdeInternal": {
"tracks": [{
"id": "2713720a-f341-4ca0-acd1-f8b1e6c0d00f"
}]
},
"message": {
"twitter_quoted_status": {
"postedTime": "2015-08-25T18:55:40.000Z",
"verb": "post",
"link": "http://twitter.com/IBMAoT/statuses/636250607154765825",
"generator": {
"displayName": "Hootsuite",
"link": "http://www.hootsuite.com"
},
"body": "A breakthrough in ultra-fast Big Data processing: #IBMResearch and UC Berkeley collaborate on Tachyon http://t.co/CWjq19wePJ",
"favoritesCount": 0,
"objectType": "activity",
"actor": {
"summary": "Over the past 25 years, the IBM Academy of Technology has provided technical leadership to IBM, its customers, and many of the top universities in the world.",
"image": "https://pbs.twimg.com/profile_images/585832589107134464/g63Qlh2p_normal.jpg",
"statusesCount": 1444,
"utcOffset": "-14400",
"languages": ["en"],
"preferredUsername": "IBMAoT",
"displayName": "IBMAoT",
"postedTime": "2011-11-02T15:29:56.000Z",
"link": "http://www.twitter.com/IBMAoT",
"verified": false,
"friendsCount": 60,
"twitterTimeZone": "Eastern Time (US & Canada)",
"favoritesCount": 622,
"listedCount": 112,
"objectType": "person",
"links": [{
"rel": "me",
"href": "http://www-03.ibm.com/ibm/academy/index.html"
}],
"id": "id:twitter.com:403487741",
"followersCount": 2389
},
"provider": {
"displayName": "Twitter",
"link": "http://www.twitter.com",
"objectType": "service"
},
"twitter_filter_level": "low",
"twitter_entities": {
"urls": [{
"display_url": "ow.ly/Rj9Ie",
"indices": [103, 125],
"expanded_url": "http://ow.ly/Rj9Ie",
"url": "http://t.co/CWjq19wePJ"
}],
"hashtags": [{
"indices": [51, 63],
"text": "IBMResearch"
}],
"user_mentions": [],
"trends": [],
"symbols": []
},
"twitter_lang": "en",
"id": "tag:search.twitter.com,2005:636250607154765825",
"object": {
"summary": "A breakthrough in ultra-fast Big Data processing: #IBMResearch and UC Berkeley collaborate on Tachyon http://t.co/CWjq19wePJ",
"postedTime": "2015-08-25T18:55:40.000Z",
"link": "http://twitter.com/IBMAoT/statuses/636250607154765825",
"id": "object:search.twitter.com,2005:636250607154765825",
"objectType": "note"
}
},
"postedTime": "2015-08-25T21:21:18.000Z",
"verb": "post",
"link": "http://twitter.com/alexlang11/statuses/636287257570361345",
"generator": {
"displayName": "Twitter for Android",
"link": "http://twitter.com/download/android"
},
"body": "Great news! https://t.co/jtPCUwy8VC",
"favoritesCount": 0,
"objectType": "activity",
"actor": {
"summary": "Architect IBM Social Media Analytics, Text Analyzer and Data Miner. My tweets are my own and don't represent IBM's positions",
"image": "https://pbs.twimg.com/profile_images/590235561740394496/tOWO1Q7f_normal.jpg",
"statusesCount": 452,
"utcOffset": "7200",
"languages": ["en"],
"preferredUsername": "alexlang11",
"displayName": "Alexander Lang",
"postedTime": "2011-07-07T20:17:46.000Z",
"link": "http://www.twitter.com/alexlang11",
"verified": false,
"friendsCount": 58,
"twitterTimeZone": "Berlin",
"favoritesCount": 58,
"listedCount": 32,
"objectType": "person",
"links": [{
"rel": "me",
"href": null
}],
"location": {
"displayName": "Stuttgart, Germany",
"objectType": "place"
},
"id": "id:twitter.com:331204884",
"followersCount": 91
},
"provider": {
"displayName": "Twitter",
"link": "http://www.twitter.com",
"objectType": "service"
},
"twitter_filter_level": "low",
"twitter_entities": {
"urls": [{
"display_url": "twitter.com/IBMAoT/status/…",
"indices": [12, 35],
"expanded_url": "https://twitter.com/IBMAoT/status/636250607154765825",
"url": "https://t.co/jtPCUwy8VC"
}],
"hashtags": [],
"user_mentions": [],
"trends": [],
"symbols": []
},
"twitter_lang": "en",
"id": "tag:search.twitter.com,2005:636287257570361345",
"retweetCount": 0,
"gnip": {
"urls": [{
"expanded_url": "https://twitter.com/IBMAoT/status/636250607154765825",
"expanded_status": 200,
"url": "https://t.co/jtPCUwy8VC"
}],
"profileLocations": [{
"geo": {
"coordinates": [9.17702, 48.78232],
"type": "point"
},
"address": {
"country": "Germany",
"countryCode": "DE",
"locality": "Stuttgart",
"region": "Baden-Württemberg Region"
},
"displayName": "Stuttgart, Baden-Württemberg Region, Germany",
"objectType": "place"
}],
"language": {
"value": "en"
}
},
"object": {
"summary": "Great news! https://t.co/jtPCUwy8VC",
"postedTime": "2015-08-25T21:21:18.000Z",
"link": "http://twitter.com/alexlang11/statuses/636287257570361345",
"id": "object:search.twitter.com,2005:636287257570361345",
"objectType": "note"
}
}
}],
"related": {
"next": {
"href": "https://cdeservice.stage1.mybluemix.net:443/api/v1/messages/search?q=from%3Aalexlang11&from=1&size=1"
}
}
}
Example 5-10 on page 181 demonstrates a simple search of a particular twitter handle. The polarity of body text “Great news! https://t.co/jtPCUwy8VC” was evaluated as a Positive comment and the evidence is the word Great in this context.
For more information about referring to the query language, see Getting started with Insights for Twitter at the following website:
By using these tools, you can create your own solution to retrieve Twitter sentiment analysis data and store the data in IBM Big SQL to further feed SPSS Analytical Decision Management.
5.3.3 SPSS Scoring Services integration with Web services
IBM SPSS Collaboration and Deployment Services provides several Web services, including the Scoring Service, which allows users to retrieve a score of a predictive model in real time. A score is a computed predictive value that is based on data that was previously provided to a model. Analytical Decision Management is one of the available C&DS Scoring Providers. A Scoring Configuration is required to configure a predictive model that is associated with a specific model file. You must use Deployment Manager to create a Scoring Configuration. See Figure 5-41 on page 180.
Figure 5-42 SPSS C&DS Scoring Service integration with Web Store
Because the Scoring Configuration is in a running state, it can accept score requests. A score request must contain all of the required input data in the defined format and published by the Scoring Configuration metadata. The web developer can create a score request programmatically and access the scoring service API. The Score Provider then returns the computed score to the web application.
The Scoring Services provides access through the following types of software:
SOAP over HTTP
SOAP over Java Message Service (JMS)
Representational State Transfer REST (JavaScript Object Notation (JSON) over HTTP)
The Scoring Service documentation is available at your own SPSS C&DS installation directory:
<spss_cads_installation_directory>/documentation/en/web_services/Scoring_Service_Developers_Guide.pdf
The REST API Scoring Service documentation is available at the same directory:
<spss_cads_installation_directory>/documentation/en/web_services/Scoring_Service_REST_Developers_Guide.pdf
For each score request, the Scoring Service can return a specific score, which can represent offers, such as credit card and other sales campaigns. Customers sometimes accept or decline each offer. This action is logged through Response Service to indicate the actual effectiveness of the business strategy (Figure 5-43). The endpoint for the service is shown:
http://<host-name>:<port-number>/DM/services/ResponseService
Figure 5-43 Scoring Service and Response Service integration with online web store
Table 5-2 shows the Scoring Service and the Response Service data flow.
Table 5-2 Scoring Service and Response Service data flow with online web store
Event
Description
1 - User Authentication
The user authenticates and the historical behavior data is retrieved with the customer ID.
2 - Scoring Service request
The online web store sends customer identification and behavior data to the Scoring Service.
3 - Scoring Service response
The Scoring Service determines the best offer for the particular customer identification and sends the offer back to the online web store.
4 - Presenting offer
The online web store presents the best offer to the user.
5 - Accept/decline offer
The user accepts or declines the offer.
6 - Response Service log
The online web store sends the response to the Response Service and it is logged for future reference and analysis.
 
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.144.109.102