Expressions

You can consider everything that is evaluated in R to be an expression. Every statement you have in your programs is also an expression that evaluates to some value (which, of course, might be NULL). This includes control structures and function bodies. You can consider everything an expression; it’s just that some expressions involve evaluating several contained expressions for their side effects before returning the result of the last expression they evaluate. From a metaprogramming perspective, though, you are most interested in expressions you can get your hands on and examine, modify, or evaluate within a program.

Believe it or not, you have already seen most of the ways to get expression objects. You can get function bodies using the body function, or you can construct expressions using quote or call. Using any of these methods, you get an object you can manipulate and evaluate from within a program. Usually, expressions are just automatically evaluated when R gets to them during a program’s execution, but if you have an expression as the kind of object you can manipulate, you have to evaluate it explicitly. If you don’t evaluate it, its value is the actual expression; if you evaluate it, you get the value it corresponds to in a given scope.

In this chapter, you will not concern yourself with the manipulation of expressions. That is the topic of Chapter 4. Instead, you will focus on how expressions are evaluated and how you can change the scope when you evaluate an expression. If you just write an expression as R source code, it will be evaluated in the scope where it is written. This is what you are used to doing. To evaluate it in a different scope, you need to use the eval function. If you don’t give eval an environment, it will just evaluate an expression in the scope where eval is called, similar to if you had just written the expression there. If you give it an environment, however, that environment determines the scope in which the expression is evaluated.

To understand how you can exploit scopes, though, you first need to understand how environments define scopes in detail.

Chains of Linked Environments

You have seen how functions have associated environments that capture where they were defined, and you have seen that when you evaluate a function call, you have an evaluation environment that is linked to this definition environment. You have also seen how this works through a linked list of parent environments. So far, though, you have claimed that this chain ends in the global environment, where you look for variables if you don’t find them in any nested scope. For all the examples you have seen so far, this might as well be true, but in any real use of R, you have packages loaded. The reason that you can find functions from packages is that these packages are also found in the chain of environments. The global environment also has a parent, and when you load packages, the last package you loaded will be the parent of the global environment, and the previous package will be the parent of the new package. This explains both how you can find variables in loaded packages and why loading new packages can overshadow variables defined in other packages.

It has to stop at some point, of course, and there is a special environment that terminates the sequence. This is known as the empty environment, and it is the only environment that doesn’t have a parent. When you start up R, the empty environment is there. Then R puts in the base environment, where all the base language functionality lives. Next it puts in an Autoload environment , responsible for loading data on demand, and on top of that, it puts in the global environment. The base environment and the empty environment are sufficiently important that you have functions to get hold of them. These are baseenv and emptying, respectively.

When you import packages, or generally attach a namespace, it gets put on the this list just below the global environment. The function search will give you the sequence of environments from the global environment and down. You can see how loading a library affects this list in this example:

search()
##  [1] ".GlobalEnv"            
##  [2] "package:ggplot2"      
##  [3] "package:purrr"        
##  [4] "package:microbenchmark"
##  [5] "package:pryr"          
##  [6] "package:magrittr"      
##  [7] "package:knitr"        
##  [8] "package:stats"        
##  [9] "package:graphics"      
## [10] "package:grDevices"     
## [11] "package:utils"         
## [12] "package:datasets"      
## [13] "Autoloads"             
## [14] "package:base"
library(MASS)
search()                                                              
##  [1] ".GlobalEnv"            
##  [2] "package:MASS"          
##  [3] "package:ggplot2"      
##  [4] "package:purrr"        
##  [5] "package:microbenchmark"
##  [6] "package:pryr"          
##  [7] "package:magrittr"      
##  [8] "package:knitr"        
##  [9] "package:stats"        
## [10] "package:graphics"      
## [11] "package:grDevices"     
## [12] "package:utils"         
## [13] "package:datasets"      
## [14] "Autoloads"             
## [15] "package:base"

The search function is an internal function, but you can write your own version to get a feeling for how it could work. While search searches from the global environment, though, you will make your function more general and give it an environment to start from. You simply need it to print out environment names and then move from the current environment to the parent until you hit the empty environment.

To get the name of an environment, you can use the function environmentName. Not all environments have names; environments you create when you nest or call functions or environments you create with new.env do not have names. However, environments created when you are loading packages do.¹ If an environment doesn’t have a name, though, environmentName will give you an empty string. In that case, you will instead just use str to get a representation of environments that you can print.

To check whether you have reached the end of the environment chain, you check identical(env, emptyenv()). You cannot compare two environments with ==, but you can use identical. Your function could look like this:

my_search <- function(env) {
  repeat {
    name <- environmentName(env)
    if (nchar(name) != 0)
      name <- paste0(name, "
")
    else          
      name <- str(env, give.attr = FALSE)
    cat(name)
    env <- parent.env(env)
    if (identical(env, emptyenv())) break          
  }
}

Calling it with the global environment as the argument should give you a result similar to search, except you are printing the environments instead of returning a list of names.

my_search(globalenv())
## R_GlobalEnv
## package:MASS
## package:ggplot2
## package:purrr
## package:microbenchmark
## package:pryr
## package:magrittr
## package:knitr
## package:stats
## package:graphics
## package:grDevices
## package:utils
## package:datasets
## Autoloads
## base

Since you can give it any environment, you can try to get hold of the environment of a function. If you write a function nested into other function scopes, you can see that you get (nameless) environments for the functions.

f <- function() {
  g <- function() {
    h <- function() {
      function(x) x
    }
    h()
  }
  g()
}
my_search(environment(f()))
## <environment: 0x7fdc0043ced0>
## <environment: 0x7fdc0043cdb8>
## <environment: 0x7fdc0043cca0>
## R_GlobalEnv
## package:MASS
## package:ggplot2
## package:purrr
## package:microbenchmark
## package:pryr
## package:magrittr
## package:knitr
## package:stats
## package:graphics
## package:grDevices
## package:utils
## package:datasets
## Autoloads
## base

You can also get hold of the environment of imported functions . For example, you can get the chain of environments starting at the ls function like this:

my_search(environment(ls))
## base
## R_GlobalEnv
## package:MASS
## package:ggplot2
## package:purrr
## package:microbenchmark
## package:pryr
## package:magrittr
## package:knitr
## package:stats
## package:graphics
## package:grDevices
## package:utils
## package:datasets
## Autoloads
## base

Here you see something that is a little weird. It looks like base is found both at the top and at the bottom of the list. Clearly, this can’t be the same environment; it would need to have, as its parent, both the global environment and the parent, and environments have only one parent. The only reason it looks like it is the same environment is that environmentName gives you the same name for two different environments. They are different.

environment(ls)
## <environment: namespace:base>
baseenv()
## <environment: base>

The environment of ls is a namespace defined by the base package . The baseenv() environment is how this package is exported into the environment below the global environment, making base functions available to you outside of the base package.

Having such extra environments is how packages manage to have private functions within a package and have other functions that are exported to users of a package. The base package is special in defining only two environments: the namespace for the package and the package environment. All other packages have three packages set up in their environment chain before the global environment: the package namespace, a namespace containing imported symbols, and then the base environment, which, as you just saw, connects to the global environment. Figure 3-1 shows the graph of environments when three packages are loaded: MASS, stats, and graphics (here graphics was loaded first, then stats, and then MASS, so MASS appears first, followed by stats and then graphics on the path from the global environment to the base environment). The solid arrows indicate parent pointers for the environments, and the dashed arrows indicate from which package symbols are exported into package environments.

Figure 3-1. Environment graph with three loaded packages: MASS, stats, and graphics

If you try to access a symbol from a package, starting in the global environment, then you get access only to the exported functions, and some of these might be overshadowed by other packages you have loaded. For functions defined inside the package, their parent environment contains all the functions (and other variables) defined in the package, and because this package namespace environment sits before the global environment in the chain, variables from other packages do not overshadow variables inside the package when you execute functions from the package.

If a package imports other packages, these go into the import environment, below the package namespace and before the base environment. So, functions inside a package can see imported symbols, but only if they aren’t overshadowed inside the package. If a user of the package imports other packages, these cannot overshadow any symbols the package functions can see, since such imports come later in the environment chain, as seen from inside the package. After the imports environment comes the base namespace. This gives all packages access to the basic R functionality, even if some of it should be overshadowed as seen from the global environment.²

There are three ways of specifying that a package depends on another in the DESCRIPTION file: using Suggests :, Depends :, and Imports :. The first doesn’t actually set up any dependencies; it is just a suggestion of other packages that might enhance the functionality of the one being defined.

The packages specified in the Depends : directive will be loaded into the search path when you load the package. For packages specified here, you will clutter up the global namespace—not the global environment but the search path below it—and you risk that functions you depend on will be overshadowed by packages that are loaded into the global namespace later. You should avoid using Depends: when you can, for these reasons.

Using Imports :, you just require that a set of other packages are installed before your own package can be installed. Those packages, however, are not put on the search path, nor are they imported in the imports environment. Using Imports: just enables you to access functions and data in another package using the package namespace prefix, so if you use Imports: to import the stats package, you know you can access stats::sd because that function is guaranteed to exist on the installation when your package is used.

When actually importing variables into the imports namespace, you need to modify the NAMESPACE file; you use the following directives for importing an entire package, functions, S4 classes, and S4 methods, respectively:

• imports()

• importFrom()

• importClassesFrom()

• importMethodsFrom()

The easiest way to handle the NAMESPACE file, though, is to use Roxygen, and here you can import names using the following:

• @import <package>

• @importFrom <package> <name>

• @importClassesFrom <package> <classes>

• @importMethodsFrom <package> <methods>

To ensure that packages you write play well with other namespaces, you should use Imports: for dependencies you absolutely need (and Suggests: for other dependencies) and either use the package prefixes for dependencies in other packages or import the dependencies in the NAMESPACE.

The function sd sits in the package stats. Its parent is namespace:stats, its grandparent is imports:stats, and its great-grandparent is namespace:base. If you access sd from the global environment, though, you find it in package:stats.

my_search(environment(sd))
## stats
## imports:stats
## base
## R_GlobalEnv
## package:MASS
## package:ggplot2
## package:purrr
## package:microbenchmark
## package:pryr
## package:magrittr
## package:knitr
## package:stats
## package:graphics
## package:grDevices
## package:utils
## package:datasets
## Autoloads
## base
environment(sd)
## <environment: namespace:stats>
parent.env(environment(sd))
## <environment: 0x7fdbffb1fbc8>
## attr(,"name")
## [1] "imports:stats"
parent.env(parent.env(environment(sd)))
## <environment: namespace:base>
parent.env(parent.env(parent.env(environment(sd))))
## <environment: R_GlobalEnv>

Figure 3-2 shows how both ls and sd sit in package and namespace environments and how their parents are the namespace rather than the package environment.

Figure 3-2. Environment graph showing the positions of stats::sd and base::ls

As you now see, this simple way of chaining environments not only gives you lexical scope in functions but also explains how namespaces in packages work.

Environments and Function Calls

How environments and package namespaces work together is interesting to understand and might give you some inspiration for how namespaces can be implemented for other uses, but in day-to-day programming, you should be more interested in how environments and functions work together. So from now on, I am going to pretend that the scope graph ends at the global environment and focus on how environments are chained together when you define and call functions.

You already know the rules for this: when you call a function, you get an evaluation environment, its parent points to the environment in which the function was defined, and if you want the environment of the caller of the function, you can get it using the parent.frame function . Just to test your knowledge, though, you should consider a more complex example of nested functions than you have done so far. Consider the following code at the point where you evaluate the i(3) call. Figure 3-3 shows how environments are chained together; here solid lines indicate parent pointers, and dashed lines are pointers from variables to their values. Figure 3-4 adds the call stack as parent frame environments . Follow along in the figures while you go through the code.

Figure 3-3. Environment graph for a complex example of nested functions

Figure 3-4. Environment graph for a complex example of nested functions highlighting the call stack as you can get it with parent.frame

f <- function(x) {
  g <- function(y, z) x + y + z
  g
}
h <- function(a) {
  g <- f(x)
  i <- function(b) g(a + b, 5)
}
x <- 2
i <- h(1)                                                              
i(3)

The first two statements in the code define the functions f and h. You don’t call them; you just define them, so you are not creating any new environments. The environment associated with both functions is the global environment. Then you set x to 2. Nothing interesting happens here either. When you call h, however, you start creating new evaluation environments. You first create one to evaluate h inside. This environment sets a to 1 since you called h with 1. It then calls f with x.

Here is where it gets a little tricky. Since you call f with a variable, which will be lazy-evaluated inside f, you are not calling f with the value of the global parameter x. So, f gets a promise it can later use to get a value for x, and this promise knows that x should be found in the scope of the h call you are currently evaluating. That x happens to be the global variable in this example, but you could assign to a local variable x inside h after you called f, and then f would be using this local x instead. When you create a promise in a function call, the promise knows it should be evaluated in the calling scope, but it doesn’t find any variables just yet; that happens only when the promise is evaluated.

Inside the call to f, you define the function g and then return it. Since g is defined inside the f call, its environment is set to the evaluation scope of the call. This will later let g know how to get a value for x. It doesn’t store x itself, but it can find it in the scope of the f call, where it will find it to be a promise that should be evaluated in the scope of the h call, where it will be found to be the global variable x. It already looks very complicated, but whenever you need a value, you should just follow the parent environment chain to see where you will eventually find it.

The h(1) call then defines a function, i, and returns it, and you save that in the global variable i. The environment of this function is the evaluation scope of the h(1) call, so this is where the function will be looking for the names a and g.

Now, finally, you call i(3) . This first creates an evaluation environment where you set the variable b to 3 since that is what the argument for b is. Then you find the g function, which is the closer you created earlier, and you call g with the parameters a+b and 5. The 5 is just passed along as a number, and you don’t translate constants into promises; however, the a+b will be lazily evaluated, so it is a promise in the call to g. Calling g, you create a new evaluation environment for it. Inside this environment, you store the variables y and z. The former is set to the promise a+b and the latter to 5. Since promises should be evaluated where they are defined, here in the calling scope, this is stored together with the promise.

You evaluate g(a + b, 5) as such: You first need to figure out what x is, so you look for it in the local evaluation environment, where you don’t find it. Then you look in the parent environment, where you do find it, but see that it is a promise from the h(1) environment, so you have to look there for it now. It isn’t there either, so you continue down the parent chain and find it in the global environment where it is 2. Then you need to figure out what y is. You can find y in the local environment where you see that it is a promise, a+b, that should be evaluated in the i(3) environment. Here you need to find a and b. You can find b directly in the environment, so that is easy, but you need to search for a in the parent environment. Here you find it, so you can evaluate a+b as 1+3. This value now replaces the promise in y. Finally, you need to find z, but at least this is easy. That is just the number 5 stored in the local environment. You now have all the values you need to compute x + y + z. They are 2 + (1+3) + 5, so when you return from the i(3) call, you get the return value 11.

The environment graphs can get rather complicated, but the rules for finding values are quite simple. You just follow environment chains. The only pitfall that tends to confuse programmers is the lazy evaluation. Here, the rules are also simple; they are just not as familiar. Promises are evaluated in the scope where they are defined. So, a default parameter will be evaluated in the environment where the function is defined, and actual parameters will be evaluated in the calling scope. They will always be evaluated when you access a parameter, so if you don’t want side effects of modifying closure environments by changing variables in other scopes, you should use force before you create closures.

Take some time to work through this example. Once you understand it, you understand how environments work. It doesn’t get more complicated than this (well, unless you start messing around with the environments!).

Manipulating Environments

So, how can you make working with environments even more complicated? You can, of course, start modifying them and chaining them up in all kinds of new ways. You see, not only can you access environments and put variables in them, but you can also modify the chain of parent environments. You can, for example, change the environment you execute a function in by changing the parent of the evaluation environment like this:

f <- function() {
  my_env <- environment()
  parent.env(my_env) <- parent.frame()
  x
}
g <- function(x) f()
g(2)
## [1] 2

You are not changing the local environment—that would be hard to do because you don’t have anywhere to put values if you don’t have that. However, you are making its parent point to the function call rather than the environment where the function was defined. If you didn’t mess with the parent environment, x would be searched for in the global environment, but because you set the parent environment to the parent frame, you will instead start the search in the caller where you find x.

The power to modify scopes in this way can be used for good but certainly also for evil. There is nothing complicated in how it works; if you understood how environment graphs work from the previous section, you will also understand how they work if you start changing parent pointers. The main problem is just that environments are mutable, so if you start modifying them in one place, it has consequences elsewhere.

Consider this example of a closure:

f <- function() {
  my_env <- environment()
  call_env <- parent.frame()
  parent.env(my_env) <- call_env
  y
}
g <- function(x) {
  closure <- function(y) {
    z <- f()
    z + x
  }
  closure
}
add1 <- g(1)
add2 <- g(2)                                                              

It is just a complicated way of writing a closure for adding numbers, but you are not going for elegance here—you are seeing how modifying environments can affect you. Here you have a function f that sets up its calling environment as its scope and then returns y. Since y is not a local variable, it must be found in the enclosing scope with a search that starts in the parent environment; this is the environment you just changed to the calling scope. In the function g, you then define a closure that takes one argument, y, and then calls f and adds x to the result of the call. Since y is a parameter of the closure, f will be able to see it when it searches from its (modified) parent scope. Since x is not local to the closure, it will be searched for in the enclosing scope, where it was a parameter of the enclosing function.

It works as you would expect it to work. Even though it is a complicated way of achieving this effect, there are no traps in the code.

add1(3)
## [1] 4
add2(4)
## [1] 6

For reasons that will soon be apparent, I just want to show you that setting a global variable x does not change the behavior.

x <- 3
add1(3)
## [1] 4
add2(4)
## [1] 6

It also shouldn’t. When you read the definition of the closure, you can see that the x it refers to is the parameter of g, not a global variable.

But now you are going to break the closure without even touching it. You just do one simple extra thing in f. Don’t just change the enclosing scope of f, and do the same for the caller of f. Set the parent of the caller of f to be its caller instead of its enclosing environment.

f <- function() {
  my_env <- environment()
  call_env <- parent.frame()
  parent.env(my_env) <- call_env
  parent.env(call_env) <- parent.frame(2)
  y                                    
}

You haven’t touched the closure, g, of the add1 and add2 functions. You have just made a small change to f. Now, however, if you don’t have a global variable for x, the addition functions do not work. This would give you an error.

rm(x)
add1(3)

Even worse, if you do have a global variable x, you don’t get an error, but you don’t get the expected results either.

x <- 3
add1(3)
## [1] 6
add2(4)
## [1] 7

What happens here, of course, is that you change the enclosing scope of the closure from the g call to the global environment (which is the calling scope of g as well as its parent environment), so this is where you now start the search for x. The evaluation environment for the g call is not on the search path any longer.

While you can modify environments in this way, you should need an excellent reason to do so. Changing the behavior of completely unrelated functions causes the worst kind of side effects. It is one thing to mess up your own function, but don’t mess up other people’s functions. Of course, you are only modifying active environments here. You have not permanently damaged any function; you have just messed up the behavior of function calls on the stack. If you want to mess up functions permanently, you can do so using the environment function as well, but doing that tends to be a more deliberate and thought-through choice.

Don’t go modifying the scope of calling functions. If you want to change the scope of expressions you evaluate, you are better off creating new environment chains for this, rather than modifying existing ones. The latter solution can easily have unforeseen consequences, while the former at least has consequences restricted to the function you are writing.

Explicitly Creating Environments

You create a new environment with the new.env function . By default, this environment will have the current environment as its parent,³ and you can use functions such as exists and get to check what it contains.

env <- new.env()
x <- 5
exists("x", env)
## [1] TRUE
get("x", env)
## [1] 5
f <- function() {
  x <- 7
  new.env()
}
env2 <- f()
get("x", env2)
## [1] 7

You can also use the $ subscript operator to access it, but in this case, R will not search up the parent list to find a variable; only if a variable is in the actual environment can you get to it.

env$x
## NULL

You can assign variables to environments using assign or through the $<- function.

assign("x", 3, envir = env)
env$x
## [1] 3
env$x <- 7
env$x
## [1] 7

Depending on what you want to do with the environment, you might not want it to have a parent environment. There is no way to achieve that.

env <- new.env(parent = NULL) # This won't work!          

All environments have a parent except the empty environment, but you can get the next best thing by making this environment the parent of your new one.

global_x <- "foo"
env <- new.env(parent = emptyenv())
exists("global_x", env)
## [1] FALSE

You can try to do something a little more interesting with manually created environments, such as building a parallel call stack you can use to implement dynamic scoping rather than lexical scoping. Lexical scoping is the scoping you already have in R, where a function call’s parent is the definition scope of the function. Dynamic scope instead has the calling environment. It is terribly hard to reason about programs in languages with dynamic scope, so I advise that you avoid them, but for the purpose of education, you can try implementing it.

Since you don’t want to mess around with the actual call stack and modify parent pointers, you need to make a parallel sequence of environments, and you need to copy the content of each call stack frame into these. You can copy an environment like this:

copy_env <- function(from, to) {
  for (name in ls(from, all.names = TRUE)) {
    assign(name, get(name, from), to)
  }
}

Just for illustration purposes, you need a function that will show you what names you see in each environment when moving down toward the global environment. You don’t want to go all the way down to the empty environment here, so you stop a little early. This function lets you do that:

show_env <- function(env) {
  if (!identical(env, globalenv())) {
    print(env)
    print(names(env))
    show_env(parent.env(env))
  }
}

Now comes the function for creating the parallel sequence of environments. It is not that difficult; you can use parent. frame to get the frames on the call stack arbitrarily deep—well, down to the first function call—and you can get the depth of the call stack using the function sys.nframe. The only thing you have to be careful about is adjusting the depth of the stack by 1 since you want to create the call stack chain for the caller of the function, not for the function itself. The rest is just a loop.

build_caller_chain <- function() {
  n <- sys.nframe() - 1
  env <- globalenv()
  for (i in seq(1,n)) {
    env <- new.env(parent = env)
    frame <- parent.frame(n - i + 1)
    copy_env(frame, env)
  }
  env
}

To see it in action, you need to set up a rather convoluted example with both nested scopes and a call stack. It doesn’t look pretty, but try to work through it and consider what the function environments must be and what the call stack must look like.

f <- function() {
  x <- 1
  function() {
    y <- 2
    function() {
      z <- 3
      print("---Enclosing environments---")
      show_env(environment())

      call_env <- build_caller_chain()
      print("---Calling environments---")
      show_env(call_env)
    }
  }
}
g <- f()()
h <- function() {
  x <- 4
  y <- 5
  g()
}
h()
## [1] "---Enclosing environments---"
## <environment: 0x1035937b8>
## [1] "z"
## <environment: 0x103596780>
## [1] "y"
## <environment: 0x1035964e0>                                                                      
## [1] "x"
## [1] "---Calling environments---"
## <environment: 0x10354b748>
## [1] "z"
## <environment: 0x103553b20>
## [1] "x" "y"

When you call h, it calls g, and you get the list of environments starting from the innermost level where you have a z and out until the outermost level, just before the global environment, where you have the x. On the (parallel) call stack, you also see a z first (it is in a copy of the function’s environment, but it is there), but this chain is only two steps long, and the second environment contains both x and y.

You can use the stack chain you constructed here to implement dynamic scoping. You simply need to evaluate expressions in the scope defined by this chain rather than the current evaluating environment. The <<- assignment operator won’t work—it would require you to write a similar function to get that behavior, and it would be a design choice to which degree changes made to this chain should be moved back into the actual call stack frames. However, as long as it comes to evaluating expressions, you can use it for dynamic scoping.

Environments and Expression Evaluation

Finally, this section covers how you combine expressions and environments to compute values. The good news is that you are past all the hard stuff, and it gets pretty simple after all of the previous stuff. All you have to do to evaluate an expression in any selected environment chain is to provide it to the eval function. You can see this in the following example that evaluates the same expression in the lexical scope and the dynamic scope:

f <- function() {
  x <- 1
  function() {
    y <- 2
    function() {
      z <- 3
      cat("Lexical scope: ", x + y + z, "
")
      call_env <- build_caller_chain()
      cat("Dynamic scope: ", eval(quote(x + y + z), call_env), "
")
    }
  }
}
g <- f()()

h <- function() {
  x <- 4
  y <- 5
  g()
}
h()
## Lexical scope:  6
## Dynamic scope:  12

The hard part of working with environments really isn’t evaluating them. It is manipulating them.