7.2.1 Filtering edges to identify groups within a network

When working with weighted networks such as the 115th Senate network, it is often necessary to filter out some of the edges to identify subgroups. Because every senator voted the same as every other senator at least once, choosing Show Graph results in an uninformative dark mass of connections (Figure 7.2). To make the graph more meaningful, you will want to only show edges between senators that have a high level of agreement. In other words, you want to filter out edges where the Percent_Agreement is underneath a certain threshold (e.g., 64%).

Begin by using the AutoFill Columns feature to set the Vertex Label to the Vertex column (i.e., the senator’s last name). Set the Vertex Tooltip to State. Then navigate to the Shape column on the Vertices worksheet and change the values for all vertices to Label. This will make all senator’s names clearly visible and allow you to see their State on mouseover. Use the Edges tab on the Autofill Columns feature to set the Edge Opacity and Edge Visibility to Percent_Agreement. You’ll need to use the Options for each of them. Set the Edge Visibility Options so that the edge only shows up If the source column number is: Greater than 0.64 (Figure 7.3). Using the default settings, this will only show edges between two people who have a greater agreement level than 0.64, which is the average percentage agreement between all pairs of senators. See the Advanced topic: Visibility column options for more details. Set the Edge Opacity Options to those described in Figure 7.3. Finally, change the Color of the edges on the Edges worksheet to something more distinct than gray (e.g., 128, 128, 192). After refreshing the graph and positioning the vertices (see Section 4.4.1), the result should look like Figure 7.3. The largely two-party system in the U.S. Senate becomes very apparent, with a cluster of conservative senators and a cluster of liberal senators, and a few senators in the middle.

7.2.2 Using dynamic filters

NodeXL allows you to dynamically filter out edges or vertices based on any data fields on the Edges and Vertices worksheets. This can be an excellent way of exploring a dataset, without making any permanent decisions about how to display it. Click on the Dynamic Filters button above the graph (see Figure 7.4) and you will be presented with a new window that lets you set the minimum and maximum values for each variable. Find the frequency table and sliders associated with Percent_Agreement. Notice from the graphic that there is a bimodal distribution (i.e., a group of senators with low agreement toward the middle-left side and a group of senators with high agreement on the right-hand side). Use the sliders associated with the Percent_Agreement data to explore the dataset. For example, drag the left slider to the right of the current visibility threshold (0.64) to exclude any edges below, say 0.95 (i.e., 95% agreement). This will hide the edges and vertices from being displayed, although they still will affect the layout. You can faintly show the hidden vertices by setting the Filter Opacity to 5% as shown in Figure 7.4. This type of dynamic analysis can help identify differences between the political parties, as well as sub cliques within them. For example, Figure 7.4 makes clear that the Republicans (the top cluster) voted as a block more often than the Democrats (the bottom cluster). Comparing the different Senate datasets available on the book website can reveal trends over time. Once you are done with your explorations, you can revert to the original image by clicking on Reset Filters and close out of the dialog.

7.2.3 Creating groups based on vertex attribute

Sometimes you will have data that describes attributes of the people in your network. For example, our Senate dataset includes information on the political party of each senator (see Party column on the Vertices worksheet). Values include D (Democrat), R (Republican), and I (Independent). To visually display this information, click on the Groups drop-down menu in the NodeXL ribbon and choose Group by Vertex Attribute. Choose Party from the first drop-down menu, since this is the attribute you want to group based upon. Then choose Categories from the second drop-down menu, since the data is categorical in nature. Notice, though, that groups can be created based on numerical data or date and time data. When you click OK, NodeXL will create two new worksheets. The Group Vertices worksheet includes a table showing each unique Vertex, alongside its Group and unique Vertex ID as shown in Figure 7.5. Additionally, the Groups worksheet is created, which includes one row for each unique group, alongside information on how to visually display the group, labels, and metrics, as shown in the left-hand side of Figure 7.6.

The image in Figure 7.6 is problematic in a couple of ways, which can be fixed. You will noticed that the vertices that are part of each group have been assigned different colors. This is good. However, the automatically assigned colors are not appropriate for this context, where red is typically associated with Republicans and blue is typically associated with Democrats. This can be easily fixed by changing the Vertex Color values next to R, D, and I on the Groups worksheet to be Red, Blue, Green (see Figure 7.7). Second, the Vertex Shape is now set to Disk (on the Groups worksheet), instead of Label (on the Vertices worksheet). In order to fix this, open the Group Options dialog (via the Groups drop-down menu in the NodeXL ribbon) and choose the settings shown in Figure 7.7. The result is a visualization that clearly shows each senator, their political party, and their network position.

7.3.1 Filtering out self-loops using the edge visibility column

As described in Chapter 6, Twitter networks include three types of edges in the Relationship column: Mentions, Replies to, and Tweet. In order to focus in on the relationships between people, you can Skip the messages that only connect to one person—namely the Tweet messages. These are visually displayed as an edge that circles back and points at the same vertex it originated at. Because the data in the Relationship column is not numerical, you cannot use Autofill Columns to filter out the edges. Instead, write the formula shown in Figure 7.8 to set the Visibility column to Skip if the message type is Tweet, or, if not, set it to Show. After you Refresh Graph, there should not be any self-loops remaining. This method is preferred over using Excel's built-in filtering tool (e.g., accessible via the drop-down menu in the Relationship column header). While using the filtering tool will not necessarily break anything, it will hide the rows, which may cause inadvertent mistakes to be made later.

Advanced topic

Count and merge duplicate edges

Many networks, similar to the CSCW 2018 Twitter network, have duplicate edges. For example, as shown in Figure 7.8, dh_30x replied to w_cscw two times (see rows 18 and 19). A view of the Overall Metrics worksheet reveals that there are 1593 duplicate edges in the CSCW 2018 Twitter network. This can be useful in many contexts. For example, in this case, the content of the tweet is preserved. Unfortunately, it can cause some problems. For example, you may want to visualize the number of tweets as a weighted line. However, to do this, you will need to count how many duplicates there are for each row and represent that in a column (e.g., Edge Weight). Sometimes there is no need to preserve each individual row, and they can be merged into a single unique row, to simplify the dataset.

NodeXL allows you to do these things by using the Prepare Data dropdown menu found in the NodeXL Ribbon and choosing the Count and Merge Duplicate Edges option. This will open a new dialog like the one shown in Figure 7.9. The first option indicates that after the duplicate edges are merged, a new Edge Weight column will be added to the Edges worksheet and populated with the number of edges that were combined. For example, the two rows where dlh_30x replied to w_cscw would have an Edge Weight of 2 appear. The second option will collapse the duplicate edges into a single row. For example, if you were to check the Merge duplicate edges box, the number of rows in the Edges worksheet would be reduced to the number of unique edges. In our example, the two dlh_30x replied to w_cscw rows would be collapsed into a single row. In this network, you may NOT want to do this, since the tweet content of one of the collapsed edges would be permanently erased. However, this feature can come in handy in many instances.

The section titled Columns that determine whether edges are duplicates allows you to account for cases where you have different edge types, such as the CSCW 2018 Twitter network. If you were to select the first option, then all edges from one person to another would be counted or merged, even if they were of different types (e.g., replies to, mentions, and tweet). This may be desirable, but in many cases it is not. The example in Figure 7.9 shows the Relationship column to also determine uniqueness. By checking this box, the counts and/or merged edges do NOT roll up Replies to and Mentions (values in the Relationship column). For example, even if you merge edges, there will be two rows that show caylery in Vertex1 and farbandish in Vertex2. One would be of Relationship type Mentions (with an Edge Weight of 2) and the other would be of Relationship type Replies to (with an Edge Weight of 3).

7.3.2 Grouping and visualizing connected components

Perhaps the simplest method to identify groups within a network is to group together vertices that are part of different connected components. Find the groups connected vertices that are completely separated out from other groups of connected vertices. To do this, choose Group by Connected Component from the Groups dropdown menu. This will populate the Groups worksheet with a different row for each connected component in the graph. They will be sorted in order from largest to smallest. In many cases, there will be one large connected component and many smaller components. To make sure they do not overlap one another, open the Layout Options dialog (found in the Layout dropdown menu above the graph), and check the box shown in Figure 7.10 that lays out the smaller components in boxes at the bottom of the graph pane. You can customize the maximum size of the connected components (10) and size of the invisible boxes (25). You may want to also update the Fruchterman-Reingold layout settings to those shown in Figure 7.10 to achieve a better layout.

The graph showing the entire network in Figure 7.10 provides a good sense of the overall scope and structure of the network. In this case, there seem to be many people who are connected directly or indirectly in the large blue connected component. There are also a number of smaller connected components that have not mentioned or replied to anyone in the larger network. It is also apparent that there seems to be a core group of people in the center of the large component, as well as a few key people that many people mention or reply to (e.g., the individual toward the right-hand side with a fan of individuals pointing toward her).

7.3.3 Using dynamic filters to filter based on time

One particularly useful way to use Dynamic Filters is to examine changes in a network over time. Since Twitter data has timestamps on the tweets that were sent, you can use the sliders to “play back” the discussion over time. Open Dynamic Filters and look at the first two Edge Filters called Relationship Date (UTC) and Tweet Date (UTC)(see Figure 7.11). For this network, these columns are identical, so you can use either one. Update the values in the Relationship Date (UTC) fields so the range covers only the time period of the conference as shown in Figure 7.11. Choose the Lay Out Visible Vertices Again option from the Lay Out Again drop-down menu as shown in Figure 7.11. This will make it so that the hidden vertices do not impact the layout of those that are shown.

You can explore additional ways of playing with the Relationship Date (UTC) fields. Reset the filters back to their starting point by clicking the Reset Filters button (Figure 7.11). Then drag the right-hand slider all the way to the left and slowly slide it to the right. This will show a dynamic view of the network as it unfolded over time. A more precise way of doing this is to click on the day portion of the right-hand side date value and use the up or down arrow keys to increase or decrease the day. You will likely notice significant bursts of activity, such as a post by katestarbird on Sep. 4, 2018 that was widely retweeted. Once you are done exploring, click the Reset Filters button (Figure 7.11) and close out.

7.3.4 Filtering based on vertex metrics and the visibility column

While the full network visualization is a nice overview, it is hard to focus in on groups within the large component or identify individuals who are important within it. For that analysis, it is necessary to filter out the less important vertices as determined by network metrics. Navigate to the Metrics columns on the Vertices worksheet and use the Sort Largest to Smallest feature (e.g., on the In-Degree column title). This can help you identify important individuals, as well as cut-off points for filtering. For example, after sorting on In-Degree, it becomes apparent that there are some outliers, such as the acm_cscw account that has an in-degree of 380 (i.e., 380 unique individuals who have mentioned or replied to them). You may also notice that the majority of user accounts have an in-degree of 0, 1, and 2. This power-law distribution is typical of real world social networks.

Because in-degree is a useful measure of local importance within a particular network, you can use it to filter out those with a low score. Use Autofill Columns to skip the Visibility of vertices with an In-Degree of 2 or less as shown in the Figure 7.12 Vertex Visibility Options dialog. You can also set the Size of the vertices to be based on In-Degree (checking the Use a logarithmic mapping) and set the Vertex Tooltip to be based on the Vertex column (i.e., where usernames are recorded). The resulting image shown in Figure 7.12 is a bit less overwhelming than the unfiltered version, but is still somewhat cluttered. This is largely because of the prominent position of the acm_cscw user, whose connections are overwhelming some of the other structures, making them less apparent. Try removing the user by navigating to the Vertices worksheet and manually choosing Skip in the Visibility column on the acm_cscw row as shown in Figure 7.13. This will remove acm_cscw from the graph, though you should be careful since running Autofill Columns again will overwrite this manual change. The resulting graph (see Figure 7.13) helps you realize that the network is a bit more spread out than it might have otherwise appeared.

7.3.5 Automatically identifying groups based on network clustering algorithms

NodeXL can automatically identify groups within a network based solely on network structure. In contrast to the approach of using existing data about the attributes as used in Section 7.2.3, this approach is based solely on who is connected to whom. A number of different network “clustering” (also known as “community detection”) algorithms exist, which help find subgroups of highly inter-connected vertices within a network. NodeXL includes three such algorithms: Clauset-Newman-Moore, Wakita-Tsurumi [3], and Girvan-Newman (which can take a long time to run on large graphs). In all of these algorithms, the number of clusters is not predetermined; instead the algorithm dynamically determines the number it thinks is best. Each vertex is assigned to exactly one cluster, meaning that clusters do not overlap. The number of vertices in each cluster can vary significantly. In some cases, a single cluster can encompass all vertices, whereas in other cases, a cluster can consist of a single vertex. See Newman [4] for background on some of these and other community identifying algorithms.

There is no “right” or “wrong” algorithm to use; instead, it is often useful to try out different ones and see which ones you believe provide the best results given your network. For example, in this network, the Clauset-Newman-Moore algorithm results in fewer, larger groups than the other algorithms, which provide more groups of a smaller size. Try applying the Wakita-Tsurumi clustering algorithm by clicking on the Groups dropdown menu in the NodeXL ribbon and choosing Group by Cluster and the checking the appropriate selector as shown in Figure 7.14. Notice that the data on the Groups worksheet is now updated to reflect the new groups.

7.3.6 Group properties and metrics

The Groups worksheet includes many fields that can be useful in analyzing and visualizing networks. If you click on one of the Group worksheet rows, it will highlight all vertices associated with that row’s group. The Vertex Color and Vertex Shape columns have already been introduced. It is worth reminding you that you may need to use the Graph Options dialog to determine if vertex color and/or shape should be pulled from the Groups worksheet or the Vertices worksheet (see Figure 7.7). The Visibility column can be used to Show, Skip, or Hide each group. Choose Skip for the bottom 7 groups, which are all separated from the large component (see Figure 7.15). This will filter them out of the graph, and also make it so metrics won’t be calculated for them. Choosing Yes in the Collapsed? column lets you collapse an entire group into a single shape (whatever is specified in the Vertex Shape column) with a plus sign in the middle of it. The size of the shape depends on the number of vertices in the group. However, collapsing groups may hide important information, so keep them set to the default (i.e., No) for now.

Calculate metrics for each group by opening the Graph Metrics dialog (see Chapter 6) and checking the box next to Group metrics (Figure 7.15). This will populate the Graph Metrics columns on the Groups worksheet, except for the groups that have Visibility set to Skip. Metrics include many of the metrics found on the Overall Metrics worksheet, such as the number of Vertices, Unique Edges, Total Edges, Graph Density, etc. Sorting on those columns can help identify key differences between the groups. For example, the group G5 has a very low Graph Density compared to most other groups, largely because there is one key person mentioning many other people.

7.3.7 Group layout and labels

At times, it can be useful to visualize groups separately from one another. NodeXL allows you to do this using the the Layout Options… dialog as shown in Figure 7.16. Choose the option Lay out each of the graph’s groups in its own box and set the Intergroup edges: to Hide. The resulting graph, shown in Figure 7.16 is missing data (i.e., the hidden edge connections between each group), but this allows for a more focused comparative analysis of groups. To learn more about this novel group in a box layout technique, see Ref. [6]. To better identify important individuals within the graph, enter the names of important users (based on metrics) in the Labels column on the Vertices worksheet. Additionally, add group labels by entering the group name in the Labels column on the Groups worksheet (see Figure 7.16). To place the group labels in the upper-left corner, as shown in Figure 7.16, use the Graph Options dialog, choose the Other tab, choose Labels… and set the Position: to Top Left in the Group box labels section.

Advanced topic

Group by Motif

NodeXL is able to group by different network motifs, or in other words, different common visual patterns of connections. See Ref. [7] for a full description of this novel visualization technique. To access this feature, choose Group by Motif in the Groups drop-down menu to open the Group by Motif dialog box (Figure 7.17). As with all of the different grouping options, using this will overwrite all other groups that were previously calculated.

7.3.8 Creating subgraph images

Another useful way to understand complex networks is to view individual sections, or subgraphs, of the larger graph. NodeXL allows you to create Subgraph Images for each vertex in the network. Network scientists call these egocentric networks (see Chapter 3). They provide a personalized view of the network from the perspective of an individual vertex and are useful when comparing vertices to one another. For example, vertices with a similar structure (e.g., a hub and spoke) may play similar social roles such as a question answerer in a discussion forum (see Chapter 10).

To create network subgraph images, click on the Subgraph Images button on the Analysis section of the NodeXL ribbon (top-right of Figure 7.18). The Subgraph Images dialog box will appear (Figure 7.18). The first option allows you to choose the levels of adjacent vertices to include in each subgraph. For example, the default of 1.5 will show edges connecting the source vertex with its direct neighbors, as well as any edges that connect the neighbors to one another. Choosing 2.0 will show all of those edges, plus edges connecting the source vertex’s neighbors with all of their neighbors. If the data were from a social networking site such as Facebook, a 2.0 setting would show your friends, which of your friends know one another, and all of your friends’ friends (FOAF). For now, replicate Figure 7.18 by using the default settings. This will generate a new column called Subgraph as shown in Figure 7.18. Subgraph images are positioned relative to each other based on the currently selected layout algorithm, so make sure you use an appropriate one (e.g., Fruchterman-Reingold) for your data.

Subgraphs highlight important differences between vertices. To illustrate this point, compare the subgraph images of individuals. For example, clifflampe and asbruckman are part of a densely connected active discussion group. In contrast, stbridgetathena and warcraft are mentioned or replied to by people who are otherwise not connected to one another. While the Clustering Coefficient metric leads to similar insights, a visual representation using subgraphs can often illuminate more nuances, such as the fact that hypotext is always mentioned alongside someone else.