Top-down or bottom-up migration

An important point to note is that the Hackificator processes files one at a time, in undefined order. The result of the run can therefore be different depending on the order in which it ends up processing files.

To illustrate this, let’s take a reduced version of a fairly common situation. In one PHP file, we have an abstract superclass. Scattered across many other PHP files are concrete subclasses—tens or even hundreds of them. In this example, we’ll just look at one.

Suppose we have files WorkItem.php:

<?php

abstract class WorkItem {
  abstract public function doWork();
}

and AckermannWorkItem.php:

<?php

class AckermannWorkItem extends WorkItem {
  public function doWork() {
    $this->running = true;
    // ...
  }
}

The first thing to note is that if we turn both files into partial-mode Hack files, there will be errors: the concrete subclass is using a property that isn’t declared. Therefore, the best we can do is to have one file in partial mode, with the other either in decl mode or in PHP.

If hackificator processes WorkItem.php first, it will put that file in partial mode. Because the subclass is still in PHP, it’s invisible to the typechecker, and WorkItem.php by itself has no errors in partial mode.¹ Then, when it processes AckermannWorkItem.php, it can only put the file in decl mode: because the superclass is in Hack, it can analyze the whole hierarchy and determine that the property running isn’t declared, which is an error in anything other than decl mode.

If hackificator processes AckermannWorkItem.php first, it will put that file in partial mode. Its superclass is still in PHP so it’s invisible to the typechecker. The typechecker assumes that the property running is declared in the superclass, and doesn’t report an error. Then, when it tries converting WorkItem.php to Hack, undeclared property errors pop up in AckermannWorkItem.php, because its superclass is now visible to the typechecker. Then hackificator has to revert WorkItem.php back to PHP; it can’t go back to AckermannWorkItem.php to back off to decl mode (which would silence the error) after processing it.

The first pattern, migrating the superclass first, is a top-down migration to Hack. The advantage of this is that any new subclasses can start off in Hack and get the benefit of thorough typechecking with knowledge of their superclass, even while other subclasses have yet to be migrated. The fully typechecked portion of the hierarchy steadily, linearly increases from 0% to 100% as the migration proceeds.

The second pattern, migrating the superclass after all of its subclasses are in Hack, is a bottom-up migration. The advantage of this is that it gets more code into Hack sooner. However, the typechecker is handicapped in the subclasses, because it has no knowledge of their superclass. Much of the hierarchy is checked with this handicap from the beginning of the migration, with almost none of it checked without handicap until the very end.

Because of the way the Hackificator works, it’s far more likely to produce bottom-up conversions, simply because there are many subclasses and one superclass, so it’s more likely to encounter a subclass first. If you want to ensure a top-down conversion, convert the superclass manually before running the Hackificator.

Neither pattern is strictly better than the other, and you can use both within the same codebase, on different class hierarchies. We’re discussing them here mostly so that you know what to expect when using the Hackificator, and to help you make a considered choice.

Upgrading typechecker modes

There’s another conversion the Hackificator can do, which is to inspect Hack files (but not PHP files) and upgrade them to the strictest mode that doesn’t cause typechecker errors. Activate this with the command-line flag -upgrade (single hyphen).

This will often come in useful because the Hackificator’s default behavior will almost never produce a strict-mode file. This is because strict mode requires all return types to be annotated, but Hack’s return type annotation syntax is illegal in PHP (in all 5.x versions and earlier).

It can be useful to combine hackificator -upgrade with hh_server --convert, described in the next section. That tool adds annotations, which may get a partial-mode file into a state where it can be upgraded to strict mode cleanly.

Adding annotations

The tool to add annotations only works on Hack files (any mode). It’s part of the typechecker server, and you invoke it as follows:

$ hh_server --convert my_project my_project

After the --convert flag, there are two arguments: first, the directory in which to actually make modifications; and second, the top-level directory of the project. The separation of the two allows you to restrict the modifications to a subset of the project, which helps keep the work in manageably sized chunks when dealing with a large codebase. The two arguments are allowed to be the same, and it’s best if they are: the more code the tool can work with at once, the more effective it can be.

This inference process is considerably slower than the one the typechecker uses for Hack files, because it’s not function-local. For example, when processing a function with unannotated parameters, it will find that function’s callsites to see what arguments are passed. If it finds consistent argument types, it will add the appropriate annotations.

Removing incorrect annotations

Once these annotations are added, try them out. Running tests is the best starting point. The added annotations don’t change any behavior except for warnings, so they shouldn’t cause tests to fail, but running tests is a convenient way to run the code. In addition, run any command-line scripts you can; if your project is a web app, start up a web server and visit some pages. The aim here is to exercise as much of your code as possible.

While doing this, you have to capture error messages. If you’re running scripts or tests from the command line, the error messages go to standard out. You can just redirect standard out to a file:

$ hhvm testfile.php > errors.log

This will capture everything from standard out, including output from the script, but that’s not a problem. The annotation-removal tool uses regular expressions to search for very specific error messages, so the script’s output shouldn’t interfere.

If you’re running HHVM as a server, error messages again go to standard out by default. You can use a configuration option to have error messages written to a file instead:

$ hhvm -m server -d hhvm.log.file=errors.log

After running your code, if any soft type annotations failed, you’ll see error messages in the log that look like the following example—these are what the annotation-removal tool looks for:

Warning: Argument 1 to f() must be of type @int, string given in
/home/oyamauchi/hack/testfile.php on line 5
Warning: Value returned from function f() must be of type @int, string given in
/home/oyamauchi/hack/testfile.php on line 6

It’s important to note that the annotation-removal tool extracts the file path from the error message and looks for the file at exactly that path. If the file path in the logs is relative, the tool will resolve it relative to its current working directory.

HHVM outputs absolute file paths in error logs by default. This can be a problem if, for example, you gather logs from one machine and do the annotation removal on another machine with your project’s source at a different path. To deal with this, you can strip the path to the project root from the log messages using a tool like sed (the full usage of which is beyond the scope of this book):

$ sed -e 's!/home/oyamauchi/hack/!!g' < errors.log > errors-relative-paths.log

Finally, with a suitable log file, removing the incorrect annotations is very simple. If the error log has relative paths, make sure you’re in the right working directory. Then, use the command hack_remove_soft_types:

$ hack_remove_soft_types --delete-from-log errors.log

Hardening annotations

When you’re confident that the remaining annotations are correct, you can make all the remaining annotations hard. This is also done with hack_remove_soft_types:

$ hack_remove_soft_types --harden lib/core.hh

The tool only accepts a single file as an argument for now. If you want to apply the operation to all the files in a directory, you can use the find utility. This example applies it to every file whose name ends with .hh in the directory lib and all of its subdirectories, recursively:

$ find lib -type f -name '*.hh' -exec hack_remove_soft_types --harden '{}' ';'