Logic Operators

Logic operators are the same across most languages, including Python and R. The upcoming Table C-1 lists some common operators. Explore these operators by creating a vector in R:

## R
score <- c(21, 7, 0, 14)
team <- c("GB", "DEN", "KC", "NYJ")

Or, create arrays with numpy in Python:

## Python
import numpy as np
score = np.array([21, 7, 0, 14])
team = np.array(['GB', 'DEN', 'KC', 'NYJ'])

Basic operators are easy to figure out, like > for greater than or < for less than. For example, you can see which elements are greater than 7 in Python:

## Python
score > 7

Resulting in:

array([ True, False, False,  True])

As you can see, when you use these operators with an array, the operation is performed against each element individually, and all the results are placed into a new array. This is similar to the way the operations work in R, as you will see. For example, with R you can see which elements are less than 15 in R:

## R
score < 15

Resulting in:

[1] FALSE  TRUE  TRUE  TRUE

Less than or equal to, and greater than or equal to, use the equals sign plus the operator: >= is greater than or equal to, and <= is less than or equal to. For example, compare the next code example to the previous one:

## Python
score <= 14

Resulting in:

array([False,  True,  True,  True])

Other operators are less obvious. Because we already use = to define objects, == is used for equals. For example, you can find elements of team that are equal to GB. Make sure you put team in quotes ("GB"). Otherwise, the computer thinks you are trying to use an object named GB:

## Python
team == "GB"

Resulting in:

array([ True, False, False, False])

Using an in-type operator is really useful when you have multiple ways to chart a player playing a similar position. For example, DE (defensive end), OLB (outside linebacker), and ED (edge defender) mean similar things in football, and filtering a dataset for all three terms is often something you do in analysis.

In numpy, you can do this with the .isin() function:

## Python
position = np.array(['QB', 'DE', 'OLB', 'ED'])
np.isin(position, ['DE', 'OLB', 'ED'])

Resulting in:

array([False,  True,  True,  True])

The pandas package has a similar function for dataframes, covered in “Filtering and Sorting Data”.

R has a slightly different operator, an %in% function:

## R
position <- c("QB", "DE", "OLB", "ED")
position %in% c("DE", "OLB", "ED")

Resulting in:

[1] FALSE  TRUE  TRUE  TRUE

When using %in%, be careful with the order. For example, compare position %in% c("DE", "OLB", "ED") from the previous example with c("DE", "OLB", "ED") %in% position:

## R
c("DE", "OLB", "ED") %in% position

Resulting in:

[1] TRUE TRUE TRUE

You can also string together operators by using the and operator (&) or the or operator (|). Using multiple operators requires the values to be in order as pairs. Our example implies that score corresponds to team. Both vectors are of length 4 in our examples.

For example, you can see which entries are greater than or equal to 7 for score and have a team value of DEN. When working with the numpy arrays, you need to use the where() function, but this logic will be the same and use similar notation with pandas later in this chapter. The results reveal which entry meets the criteria:

## Python
np.where((score >= 7) & (team == "DEN"))

Resulting in:

(array([1]),)

You can also use an or operator for a similar comparison to see which values of score are greater than 7 or which values of team are equal to DEN:

## R
score > 7 | team == "DEN"

Resulting in:

[1]  TRUE  TRUE FALSE  TRUE

You can string together multiple conditions with parentheses. For example, you can see what has score values greater than or equal to 7 and team equal to DEN or score equal to 0:

## Python
np.where((score >= 7) & (team == "DEN") | (score == 0))

Resulting in:

(array([1, 2]),)

Likewise, similar notation may be used in R:

## R
(score >= 7 & team == "DEN") | (score == 0)

Resulting in:

[1] FALSE  TRUE  TRUE FALSE

Filtering and Sorting Data

In the previous section, you learned about logical operators. These functions serve as the foundation of filtering data. In fact, when we get stuck with filtering, we often build small test cases like the ones in “Logic Operators” to make sure we understand our data and the way our filters work (or, as is sometimes the case, do not work).

You will work with the Green Bay–Detroit data from the second week of the 2020 season. First, read in the data and do a simple filter to look at plays that had a yards-after-catch value greater than 15 yards to get an idea of where some big plays were generated.

In R, load the tidyverse and nflfastR packages and then load the data for 2020:

## R
library(tidyverse)
library(nflfastR)

# Load all data
pbp_r <- load_pbp(2020)

In Python, import the pandas, numpy, and nfl_data_py packages and then load the data for 2020:

## Python
import pandas as pd
import numpy as np
import nfl_data_py as nfl

# Load all data

pbp_py = nfl.import_pbp_data([2020])

Resulting in:

2020 done.
Downcasting floats.

In R, use the filter() function next. The first argument into filter is data. The second argument is the filter criteria. Filter out the Detroit at Green Bay game and select some passing columns:

# Filter out game data
gb_det_2020_r_pass <-
    pbp_r |>
    filter(home_team == 'GB' & away_team == 'DET') |>
    select(posteam, yards_after_catch, air_yards,
           pass_location, qb_scramble)

Next, filter() the plays with yards_after_catch that were greater than 15:

gb_det_2020_r_pass |>
filter(yards_after_catch > 15)

Resulting in:

# A tibble: 5 × 5
  posteam yards_after_catch air_yards pass_location qb_scramble
  <chr>               <dbl>     <dbl> <chr>               <dbl>
1 DET                    16        13 left                    0
2 GB                     19         3 right                   0
3 GB                     19         6 right                   0
4 DET                    16         1 middle                  0
5 DET                    20        16 middle                  0

Notice in this example that plays that generated a lot of yards after the catch come in many shapes and sizes, including short throws with 1 yard in the air, and longer throws with 16 yards in the air. You can also filter with multiple arguments by using the “and” operator, &. For example, you can filter by yards after catch being greater than 15 and Detroit on offense:

## R
gb_det_2020_r_pass  |>
filter(yards_after_catch > 15 & posteam == "DET")

Resulting in:

# A tibble: 3 × 5
  posteam yards_after_catch air_yards pass_location qb_scramble
  <chr>               <dbl>     <dbl> <chr>               <dbl>
1 DET                    16        13 left                    0
2 DET                    16         1 middle                  0
3 DET                    20        16 middle                  0

However, what if you want to look at plays with yards after catch being greater than 15 yards or air yards being greater than 20 yards and Detroit the offensive team? If you try yards_after_catch > 15 | air_yards > 20 & posteam == "DET" in the filter, you get results with both Green Bay and Detroit rather than only Detroit. This is because the order of the operations is different than you intended:

## R
gb_det_2020_r_pass |>
filter(yards_after_catch > 15 | air_yards > 20 &
       posteam == "DET")

Resulting in:

# A tibble: 9 × 5
  posteam yards_after_catch air_yards pass_location qb_scramble
  <chr>               <dbl>     <dbl> <chr>               <dbl>
1 DET                    16        13 left                    0
2 GB                     19         3 right                   0
3 DET                    NA        28 left                    0
4 DET                    NA        28 right                   0
5 GB                     19         6 right                   0
6 DET                    16         1 middle                  0
7 DET                     0        24 right                   0
8 DET                    20        16 middle                  0
9 DET                    NA        50 left                    0

You get all plays with yards after catching being greater than 15 or all plays with yards greater than 20 and Detroit starting with possession of the ball. Instead, add a set of parentheses to the filter: (yards_after_catch > 15 | air_yards > 20) & posteam == "DET".

The use of parentheses in both coding and mathematics align, so the order of operations starts with the innermost set of parentheses and then moves outward:

## R
gb_det_2020_r_pass |>
filter((yards_after_catch > 15 | air_yards > 20) &
        posteam == "DET")

Resulting in:

# A tibble: 7 × 5
  posteam yards_after_catch air_yards pass_location qb_scramble
  <chr>               <dbl>     <dbl> <chr>               <dbl>
1 DET                    16        13 left                    0
2 DET                    NA        28 left                    0
3 DET                    NA        28 right                   0
4 DET                    16         1 middle                  0
5 DET                     0        24 right                   0
6 DET                    20        16 middle                  0
7 DET                    NA        50 left                    0

You can also change the filter to look at only possession teams that are not Detroit by using the “not-equal-to” operator, !=. In this case, the “not-equal-to” operator gives you Green Bay’s admissible offensive plays, but this would not always be the case. For example, if you were working with season long data with all teams, the “not-equal-to” operator would give you data for the 31 other NFL teams:

## R
gb_det_2020_r_pass |>
filter((yards_after_catch > 15 | air_yards > 20) &
       posteam != "DET")

Resulting in:

# A tibble: 8 × 5
  posteam yards_after_catch air_yards pass_location qb_scramble
  <chr>               <dbl>     <dbl> <chr>               <dbl>
1 GB                     NA        26 left                    0
2 GB                     NA        25 left                    0
3 GB                     19         3 right                   0
4 GB                     NA        24 right                   0
5 GB                      4        26 right                   0
6 GB                     NA        28 left                    0
7 GB                     19         6 right                   0
8 GB                      7        34 right                   0

In Python with pandas, filtering is done with similar logical structure to the tidyverse in R, but with different syntax. First, Python uses a .query() function. Second, the logical operator is inside quotes:

## Python
gb_det_2020_py_pass = 
  pbp_py
    .query("home_team == 'GB' & away_team == 'DET'")
      [["posteam", "yards_after_catch","air_yards",
        "pass_location", "qb_scramble"]]

print(gb_det_2020_py_pass.query("yards_after_catch > 15"))

Resulting in:

     posteam  yards_after_catch  air_yards pass_location  qb_scramble
4034     DET               16.0       13.0          left          0.0
4077      GB               19.0        3.0         right          0.0
4156      GB               19.0        6.0         right          0.0
4171     DET               16.0        1.0        middle          0.0
4199     DET               20.0       16.0        middle          0.0

Notice that the or operator, |, works the same with both languages:

## Python
print(gb_det_2020_py_pass.query("yards_after_catch > 15 | air_yards > 20"))

Resulting in:

     posteam  yards_after_catch  air_yards pass_location  qb_scramble
4034     DET               16.0       13.0          left          0.0
4051      GB                NaN       26.0          left          0.0
4055      GB                NaN       25.0          left          0.0
4077      GB               19.0        3.0         right          0.0
4089     DET                NaN       28.0          left          0.0
4090     DET                NaN       28.0         right          0.0
4104      GB                NaN       24.0         right          0.0
4138      GB                4.0       26.0         right          0.0
4142      GB                NaN       28.0          left          0.0
4156      GB               19.0        6.0         right          0.0
4171     DET               16.0        1.0        middle          0.0
4176     DET                0.0       24.0         right          0.0
4182      GB                7.0       34.0         right          0.0
4199     DET               20.0       16.0        middle          0.0
4203     DET                NaN       50.0          left          0.0

In Python, when your code gets too long to easily read on a line, you need a backslash (), for Python to understand the line break. This is because Python treats whitespace as a special type of code, whereas R usually treats whitespace, such as spaces, indentations, or line breaks, simply as aesthetic. To a novice, this part of Python can be frustrating, but the use of whitespace is a beautiful part of the language once you gain experience to appreciate it.

Next, look at the use of parentheses with the or operator and the and operator, just as in R:

print(gb_det_2020_py_pass.query("(yards_after_catch > 15 | 
                               air_yards > 20) & 
                               posteam == 'DET'"))

Resulting in:

     posteam  yards_after_catch  air_yards pass_location  qb_scramble
4034     DET               16.0       13.0          left          0.0
4089     DET                NaN       28.0          left          0.0
4090     DET                NaN       28.0         right          0.0
4171     DET               16.0        1.0        middle          0.0
4176     DET                0.0       24.0         right          0.0
4199     DET               20.0       16.0        middle          0.0
4203     DET                NaN       50.0          left          0.0

Cleaning

Having accurate data is important for sports analytics, as the edge in sports like football can be as little as one or two percentage points over your opponents, the sportsbook, or other players in fantasy football. Cleaning data by hand via programs such as Excel can be tedious and leaves no log indicating which values were changed. Also, fixing one or two systematic errors by hand can easily be done with Excel, but fixing or reformatting thousands of cells in Excel would be difficult and time-consuming. Luckily, you can use scripting to help you clean data.

Consider this example dataframe in pandas:

wrong_number = 
  pd.DataFrame({"col1": ["a", "b"],
                "col2": ["1O", "12"],
                "col3": [2, 44]})

Notice that col2 has a 1O (“one-oh”) rather than a 10 (“one-zero,” or ten). These types of mistakes are fairly common in hand-entered data. This may be fixed using code.

Use the locate function, .loc(), to locate the wrong cell. Also, select the column, col2. Last, replace the wrong value with a 10 (ten):

## Python
wrong_number.loc[wrong_number.col2 == "1O", "col2"] = 10

Look at the dataframe’s information, though, and you will see that col2 is still an object rather than a number or integer:

## Python
wrong_number.info()

Resulting in:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   col1    2 non-null      object
 1   col2    2 non-null      object
 2   col3    2 non-null      int64
dtypes: int64(1), object(2)
memory usage: 176.0+ bytes

Change this by using the to_numeric() function from pandas and then look at the information for the dataframe. Next, save the results to col2 and rewrite the old data. If you skip this step, the computer will not save your edits:

## Python
wrong_number["col2"] = 
    pd.to_numeric(wrong_number["col2"])
wrong_number.info()

Resulting in:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   col1    2 non-null      object
 1   col2    2 non-null      int64
 2   col3    2 non-null      int64
dtypes: int64(2), object(1)
memory usage: 176.0+ bytes

Notice that now the column has been changed to an integer.

If you want to save these changes for later, you can use the to_csv() function to save the outputs. Generally, you will want to use a new filename that makes sense to you now, to others, and to your future self. Because the dataframe does not have meaningful row names or an index, tell pandas to not save this information by using index=False:

## Python
wrong_number.to_csv("wrong_number_corrected.csv", index = False)

R uses slightly different syntax. First, use the mutate() function to change the column. Next, tell R to change col2 by using col2 = .... Then use the ifelse() function to tell R to change col2 if it is equal to 1O (“one-oh”) to be 10 (“one-zero,” or ten), or to use the current value in col2:

## R
wrong_number <-
  tibble(col1 = c("a", "b"),
         col2 = c("1O", "12"),
         col3 = c(2, 44))
wrong_number <-
  wrong_number |>
  mutate(col2 = ifelse(col2 == "1O", 10, col2))

Next, just as in Python, change col2 to be numeric. In R, use the as.numeric() function. Then look at the dataframe structure by using str():

## R
wrong_number <- mutate(wrong_number, col2 = as.numeric(col2))
str(wrong_number)

Resulting in:

tibble [2 × 3] (S3: tbl_df/tbl/data.frame)
 $ col1: chr [1:2] "a" "b"
 $ col2: num [1:2] 10 12
 $ col3: num [1:2] 2 44

Finally, just as in Python, save the file by using a name that makes sense to both the current you and future you. Hopefully, this name makes sense to other people. Creating names can be one of the most difficult parts of programming. With R, use the write_csv() function:

## R
write_csv(x = wrong_number,
          file = "wrong_numbers_corrected.csv")

Piping in R

With programming, sometimes you want to pass outputs from one function to another without needing to save the intermediate outputs. In mathematics, this is called composition, and while teaching college math classes, Eric observed this to be one of the more misunderstood procedures because of the confusing notation. In computer programming, this is called piping because outputs are piped from one function to another.

Luckily, R has allowed composition through the piping operators with the tidyverse that has a pipe function, %>%. As of R version 4.1, released in 2021, base R also now includes a |> pipe operator. We use the base R pipe operator in this book, but you may see both “in the wild” when looking at other people’s code or websites.

We cover piping here for two reasons. First, you will likely see it when you start to look at other people’s code as you teach yourself. Second, piping allows you to be more efficient with coding once you get the hang of it.