Chapter 7
Shells and Native Applications
I was like a boy playing on the sea-shore, and diverting myself now and then finding a smoother pebble or a prettier shell than ordinary, whilst the great ocean of truth lay all undiscovered before me.
—Sir Isaac Newton
A shell is a very convenient place to run applications, and it allows both developers and users great access to all the hardware in a platform, and all the shells/applications they are sitting upon. Many shells have significantly lower overhead to them than a modern operating system. These two features combine to make them an excellent place to develop and test new hardware and low-level drivers for that hardware, as well as a place to run diagnostics.
The common features that most shells share are the ability to run external executable images, the ability to be automated, file system access, environment access, and system configuration.
In Figure 7.1 you can see how a shell command is translated from a human readable string down to a hardware command in the EFI environment.
The running of external shell applications is critical since that is the method most used to add a new feature into the shell. This new feature can be anything from the simple printing of a string to the console for user input all the way up to a complex program that runs the entire platform. This is the only way to add a completely new feature to the shell without editing the shell’s own code. This is also commonly used to perform proprietary tasks, such as manufacturing configuration, where the executable is not found on the system afterwards.
Automation is accomplished through script files. These are a set of sequential commands that can be started automatically upon entering the shell so that they happen whenever the shell is launched. Some shells allow a user to abort the running of these automatically started scripts.
These two sets of extension abilities also can be combined. It is possible for a script file to call some of the extended executables, some commands internal to the shell, and even a different script file.
The features that make the UEFI Shell 2.0 unique have to do with the features that make it especially useful for firmware. This means specifically that the shell can be heavily configured such that the platform can have a shell with a reduced feature set and a similarly reduced footprint size. At least 64 combinations of size and feature set are available in the UEFI Shell, with more available via extensions. This allows for the UEFI Shell to vary in size from ~300 to almost 1000 KB.
The other effect that this reduction in the size of the feature set can have is security. If the end user of the platform is not expected to use the shell, it is possible to restrict the features available to eliminate some risk that they can harm the system, but still leave enough features that the limited shell could be used to initiate a platform debug session. It is even possible to have a limited built-in-shell (with a correspondingly small binary image footprint) launch a large and feature-rich shell from a peripheral storage device, such as USB, DVD, or even over the network.
When in early testing of a new platform, a common use of the shell is as a boot target, normally before the hardware can boot to a full modern operating system. This allows for lots of hardware testing via the internal commands and custom-designed shell applications. Since custom applications have access to the full system, they can easily test memory, run new assembly instructions, test a new peripheral media device, or simply examine the contents of the ACPI table.
Since the EFI and UEFI shells have built-in commands to examine memory, examine drive contents, verify device configuration, use the network, and output logs of what was found, much early testing can be accomplished in this environment. When this is combined with the ability of a shell to run itself with minimal features from the underlying system, it is a clear advantage to use the shell to test and debug new hardware of unknown quality.
The logical continuation of this is that in a system where the hardware is expected to be of high quality, but the side effect of the usage model dictates that testing be done still (such as a manufacturing line), it makes a lot of sense to first boot to the shell to do some level of testing and then “continue” the boot onto the operating system. This is easily done from any EFI or UEFI Shell since in both of these cases the operating system loader is just another UEFI application that can be launched from the shell. To do this automatically, you would configure the startup.nsh script file for the UEFI Shell and have that script file do a series of tests and, if they all pass, then launch the operating system loader application. There are features that directly support this type of behavior with the ability to get and set environment variables and use conditional expressions right inside the script files.
Here is an example script file:
Pre-OS Shells
Both the EFI Shell and the UEFI Shell are UEFI applications. This means that they will run on any current UEFI platform (the EFI Shell will run on older implementations of the UEFI and EFI specifications). These applications have very low requirements from the system. They both need memory, SimpleTextInput, SimpleTextOutput, Device-PathToText, and UnicodeCollation, and will use lots more (for example, Simple-FileSystem or BlockIO) if they are present. See the documentation for each UEFI application to verify exactly what protocols are required for that application.
The EFI Shell is a nonstandard shell. That means there is no predefined behavior for a given command and that there is no predefined set of commands that must be present. In practice, there is a de-facto standard for both risk factors due to the prevalence of a single implementation of the EFI Shell dominating in the marketplace. The EFI Shell provides all the standard features already discussed above, but unlike the UEFI Shell there are only two different versions of the EFI Shell to allow for customizing the size requirement. This means that fine-tuning of the binary size cannot occur.
The UEFI Shell 2.0 is the successor to the EFI Shell. This is a standards- based version of a UEFI-compliant shell. It has all the commands the EFI Shell had, but many of these commands have been extended and enhanced and some new commands have also been added to the UEFI Shell. This means that all script files that were written for the EFI Shell will work on the UEFI Shell, but this is not always the case in reverse. It is possible (and recommended) to write the scripts using the new features if the target shell will always be the UEFI Shell, as they greatly simplify and enhance the capabilities of the script files.
Two changes in the UEFI Shell are very significant for script files. The first is the concept of levels and profiles for the shell. These are sets of commands that can be queried for availability before they are called. This is important because if you can’t call the shell command GetMtc to get the monotonic tic count or DrvDiag to initiate a driver diagnostic then you may want to verify that it is present in the shell before you call it. With the old EFI Shells you couldn’t test for this condition and had to rely on the platform to have the correct version of the shell, but in the newer UEFI Shells, you can query for the presence of the Driver1 profile and thereby know that the DrvDiag command is required to be present. The second change directly affecting script files is the concept of Standard Format Output or -sfo. This is a feature, and a parameter, present on some shell commands (like the ls command), which have a lot of columnar output. By specifying -sfo, the output will be output in a specific format defined in the specification so output between different implementations will have no effect. This output is comma delimited and can be redirected to a file and then easily parsed with the parse shell command which did not exist in the EFI Shell. These two changes mean that a script file that works in the UEFI Shell but not in the EFI Shell can have a lot more logic in it and can have multiple methods for getting information such that if one shell command is not present, it could use another, while the same script file, if done in a cross-version method, would have to do a lot more work.
UEFI Shell Application
The UEFI and EFI Shells are themselves UEFI applications. What does this actually mean? Let’s start with the word application. In the UEFI environment, applications are binary files that are loaded into memory, have their entry-point function called, and are then unloaded from memory. This is different from a driver, which is not unloaded after running the entry-point function. The second part of it means that the application depends on some of the elements of UEFI. There is no defined method to know exactly which part of the interface is defined, but a well-designed application will have some documentation or a clear message about what it requires. The elements of UEFI that can be required are all accessed via the System Table, either through Boot Services or Runtime Services. Note that outside of Operating System Loader applications there are no UEFI applications that operate in the runtime mode of the UEFI environment.
What makes a shell application different? A shell application is one that must run after the shell application (itself a UEFI application) has started. There are a few benefits to this: parameters, file system, environment. We’ll go into each of those.
When a UEFI application starts, it gets its parameters as a single long array of CHAR16. This means that the application must carry the logic for parsing this into parameters and then determining associations between the parameters (for example, the use of quotes). When a UEFI Shell application starts, the shell has already performed this parsing and the application is passed Argc and Argv almost as if it were a standard C application. This allows for much smaller applications, much faster application creation, and generally easier to maintain applications.
The file system in the UEFI environment is accessed via DevicePaths. These are not especially easy or fun for a human to read and not especially easy to write. For example:
is the DevicePath for a USB hard disk in the system. When the shell runs, it creates a human readable map name for this called FS0: and a consistent map name (stays the same across multiple platforms) called f17b0:. These are much easier to use. For example, echo hello world, fs0:hi.txt makes a lot of sense and is easily understood by almost any user. However, the UEFI application must interpret the full device path from its command line and try to find the file system represented on that device path and find the required file at the end. This is a compound problem since there are usually multiple file systems and the effort to decode each one will repeat all the work each time the decoding must take place.
The shell environment features things like path, which is a list of places to search for a specified file by default, a series of aliases so that ls and dir are the same command, and environment variables so that %foo% can be automatically replaced by a predetermined (and configurable) value. There are also useful functions for finding files using wild card characters (? and *), finding the names of devices, and getting additional information on files. Configurable elements of these environment features can be changed and configured on the command line of the shell.
A complicating factor is that the method used to access these features from the EFI Shell differs from the method used from the UEFI Shell 2.0. This means you may have to do extra work to support both types of UEFI Shells (or use the UDK 2010 Shell Library)
You have to decide on tradeoffs:
–Is it worth the feature set that the shell provides to require that the application run under it at all? Maybe writing a UEFI application instead of a shell application would be a better solution.
–Is it worth the binary size increase to use the UDK 2010 library and support either of the two (EFI and UEFI) shells? The library will automatically detect the shell version and seamlessly handle the differences.
–Is the target environment a single type of shell such that the library overhead could be minimized by directly using the protocols produced by the shell? Direct access will reduce the binary size, but it also increases the amount of work required for some actions (printing with color is a good example).
The code for a simple Hello World UEFI application follows:
The same done as a UEFI Shell application would replace the call to
OutputString with a call to ShellPrintEx(-1,-1,L“Hello World”).
As you can see the application in its simplest form is actually quite similar in the UEFI application and in the UEFI Shell application forms. The difference becomes more apparent as we look at the code used to open a file.
The UEFI Shell application opening a file named file.txt:
The UEFI application opening a file named file.txt and for simplicity we are not even trying to search a path for the file (pseudo code):
The point here is that the shell does not do anything that any other application (or driver for that matter) cannot do. It just makes the repeating of these tasks very easy to do. Don’t think that you can’t do what the shell does. Think how much you don’t need to do that the shell already is doing for you.
EFI/UEFI Script File
UEFI applications and UEFI Shell applications are pretty well defined up to this point in terms of their capabilities and their restrictions. They are the more powerful of the two types of files that interact with the shell. The other type of file is the shell script file.
Different Features between Script and App
A Script file is always a text file, either UNICODE or ASCII, and because of this can be changed quite easily with no special programs. A small sample shell script is:
This script is a simple functional test of the goto script-only command. The significant limitation of script files are that they cannot do anything that is not already done in either a shell command or an existing shell application. This means that script files are good for repetitive tasks, but they have their limitations. For example, to output the current memory information to a log file and then compare that with a known good version of the file is an excellent task for completion by a script file. On the other hand, opening a network connection and sending some information to a remote platform is something that is not already encapsulated into a shell command (and assuming there is no special application) means that this would be better done with an application and not a script.
The power behind applications is that they can open and interact with any protocol and any handle that is present in the system. An application has all of the same privileges and rights as a UEFI driver—this means that the application can do pretty much anything. A script, on the other hand, cannot open any protocols; it can interact only with shell commands and applications.
The power behind script files is that they can do large amounts of repetitive work with less overhead than required for an application and that they can fully exercise shell commands and applications with much less development overhead than an application. Script files require no compiler or any development tools at all. They can even be edited via the edit command in the shell and then rerun without using any other software.
For example, the following loop will do some desired action 300 times in a row:
This same script could be made more generic by modifying it to use the first parameter to the script itself as the termination condition. This means that the script will run some arbitrary number of times that is controllable at the time the script is launched.
Customizing the UEFI Shell
The UEFI Shell was both specified and designed with the goal of allowing lots of customization. We already covered the 64 possible combinations of shell levels and shell profiles. For initial silicon bring-up and debugging, it would be best to try to use a level 3 shell with all the profiles installed. This would mean that the binary size would be the biggest of all the combinations, but also that the feature set would be the biggest. There is a defined method for adding additional (custom) profiles into the shell that include any custom shell commands you have developed.
In Figure 7.2 you can see how the UEFI Shell runs on top of the UEFI drivers. You can also see the separation of required and optional components of the shell, which is what allows for the easy customization and modification of the shell.
To add a command set to the UEFI Shell 2.0, located in the UDK2010 available at www.tianocore.org, add a NULL-Named-Library to the shell via your DSC file. This new library must then use the API from UefiShellCommandLib to register its commands with the shell core. The shell core will then call into this new library when the registered commands are invoked from the command line of the shell. This registration will include the name of the command (don’t overlap with existing commands in the shell), a function pointer (of a specific prototype) to be called when the command is run, a required level (zero in this case), the name of this profile, and the identifier for the help content in HII. This information is used to populate required shell features:
–The name is used for the required environment variable profiles.
–The help content is used if the help command is invoked.
–The level would be used to remove the command if the build is set for a lower level.
–The function pointer is for the core to call your library for implementation of the command.
Note: Per the UEFI Shell 2.0 specification, the distribution is always two files minimum.
Adding a shell application is almost as much overhead in terms of coding work. Although the work to build the library is reduced, more work is required in the distribution of the separate files that are not part of the shell binary itself.
The shell will by default add the root of each drive, as well as the efi ools and efioot directories. This means that any tool that wants to appear to be an internal command should reside in one of those three directories.
Note: For automatic help system integration, there should be a help file named the same as the application file and with contents contained in .man file format the same as the shell internal commands (except their information is stored via HII).
Once the internal command (in a profile) or the external application has started, they have the same privileges and almost the same access. A few actions can only be accessed via the UefiShellCommandLib. These actions are centered on internal shell functionality that a few commands need to manipulate:
–Shell script interaction
–Shell alias interaction
–Shell map interaction
Note: Linking a shell application with the library will work, but (per definition) the library functions only when also linked to the shell core so it functions (completely) only for internal shell commands.
The help system built into the UEFI Shell 2.0 core automatically parses a file when someone uses the help command (for example, help <Your_App_ Name>). This file must be in the same location as the application file. The Unicode text file must have file format similar to this:
Once you have this file in place, there is little distinction to a normal user of the shell between your command and an internal command. There are two differences the user can notice:
–Internal commands take precedence over applications with the same name. Specify the file name with the .efi extension to override this behavior. This can also happen with script files; use .nsh to override in this case.
–Internal commands are listed via help command. External applications are not.
Where to Get Shells
Shells are frequently distributed by one of two common methods: binary distribution and code distribution. The benefits of binary distribution are simplicity and convenience without any overhead. The benefit of code distribution is customization, but this brings with it a higher overhead to get the product included in your firmware image.
The binary EFI Shell is distributed in binary form via SVN in the EDK and EDK II standard distributions at http://www.tianocore.org. This binary comes in two sizes, with the larger image having more commands available.
The binary UEFI Shell will be distributed in binary form via SVN in the EDK II and via UDK releases, both located at http://www.tianocore.org.
The source for the EFI Shell is located in an EFI Shell subproject called efi-shell. This is at http://efi-shell.tianocore.org. There are two build files that support the two different versions.
The source for the UEFI Shell is located via SVN in the EDK II and in the UDK distributions at http://edk2.tianocore.org. The customization is controlled by Platform Configuration Database entries and configured in the DSC used for building the shell.
GUIs and the UEFI Shell
There is no inherent support for graphical user interfaces (GUIs) in any of the shells discussed. There are methods in human interface infrastructure (HII) that are possible to use for configuring and performing localized displays. This would be done via the HiiLib (in UDKEDK II) or directly via the HII protocols (See Chapters 29 and 30 of UEFI specification).
Another possibility is to use libraries to achieve this functionality. You could use the C Library (in UDKEDK II); it should be possible to compile some standard graphics libraries and use them in the shell.
There are libraries that some companies have developed for application development. Some of these are available for purchase and others are used internally only. One example of a publicly available solution is the AMI Provisioning solution (see ami.com/prov).
Remote Control of the UEFI Shell
Both the EFI and UEFI Shells are remote controllable since all EFI implementations have support for remote control over the serial port of the keyboard and console. This is not a modern over-the-Internet remote desktop style of control, but it is certainly sufficient. This means remote control of a system running a shell requires a second computer to be the host for the testing. A very common technique for remote debugging a platform involves connecting both a hardware debugger and a serial port between a host and the platform-under-development (PUD). Then the engineer doing the debugging remotely (via the modern remote desktop connection of your choice) controls the host computer and then controls the PUD via the hardware debugger and the serial connection. The biggest challenge in this scenario is changing the copy of an application the PUD has access to. I have found that there are KVMs that have a built-in USB switching capability and using that in conjunction with a standard USB drive allows for the UEFI application under debug to be updated without anyone having to physically change anything.
Obviously, if having the PUD physically right next to you is possible, that may be faster and easier, although for some systems the power, sharing, or noise requirements may make that prohibitive to do.
Debugging Drivers and Applications in the EFI and UEFI Shells
Driver and application debugging is an excellent use of shells. Many commands are there explicitly to enable and assist in debugging. In the UEFI Shell these are contained in the “Drivers1” and “Debug1” profiles; in the EFI Shell these are in the Full image.
The basic parts of a driver that can be tested depend on the type of driver. Some drivers are much easier to get higher test coverage than others.
For a driver with a storage medium present, you can verify that the driver installed the correct protocols, verify that the platform built the upper parts of the driver stack, verify writing and reading to/from the storage medium, and verify that the driver frees all the memory it allocated upon unloading. At the same time that you can test all of these things manually, the better method would be to test by running the installation program from a UEFI-aware operating system.
For a driver with no storage medium present, such as a network driver, there may be limited coverage. The “Network1” profile has a ping command that can verify that the network controller can send and receive data, but not much more than that. The best test for this specific case would be a PXE boot test.
If the driver you are testing is for a bus, not for a device, then you need to take into account it needs to create child controllers and usually an intermediate I/O protocol. Then you would use commands like DevTree, Devices, and Drivers to see that the parent-child relationships are correctly established.
Testing an application, even a complex application such as a shell itself, requires a lot of work. The first thing is that an application should, when finished, leave the system with the same amount of memory. The second is that testing an application can utilize StdIn redirection to control the input to the application and use StdOut redirection, combined with script files, to automatically verify the output.
The shell command MemMap can display memory information about the
platform. The actual map is, in this case, not the information that is of interest. The interesting information is the other part—about how many bytes of memory are allocated for each type (that is, BootServiceData). The MemMap command is run (recording the data) before and after running an application or loading and unloading a driver and then run again, and the end results are compared with the first run.
The commands Connect, Disconnect, and Map are required for setup and configuration of drivers for devices that are disk drives. Many commands can be used for write, read, and verify on a disk drive. A simple version would be to echo a text file to the new drive and then use the Compare command to verify it.
The End for the Shell
There are many possible end-game scenarios for the shell. They can range from exiting the shell to having it run forever as a background program. They are (vaguely organized from simple to less simple):
–Exit command
–OS Loader
–Shell application as platform
–Runtime application
The simplest example is that a user can run the exit command and return control to the caller (of the shell). This could be done so that the next application in the boot list is called, to un-nest from one instance of the shell to another, or to return control directly to the boot manager or setup screen.
The next simplest of these is when the OS Loader application is run. At some point during the OS Loader application, the ExitBootServices API is called (via the system table), causing the shell to end indirectly. In this case, all memory in the system not marked as Runtime is allowed to be overwritten, including the shell. The goal is that the OS will take over memory management and thus the boot service memory is no longer required.
The next possible method is that a shell application (or UEFI application) is the end-goal of the platform. This means there is no operating system at all and that the platform will forever (until powered off) just run the application. The best example of this scenario is how some Hewlett-Packard printers use an application to do all the work of handling print jobs, controlling the queue, monitoring ink, and so on. This embedded platform has no operating system whatsoever. All it has is an application that runs forever.
The most complex is somewhat of a hybrid of the previous two types. This is an application that handles everything like the preceding example, but uses the Exit-BootServices API as in the second example. This does allow the application to directly access hardware, but it also requires that this new application have drivers to handle this interaction. This distinction between this type of application and a true operating system may be more in the mind of the creator than anything else. It could be easily used for a hardware test where the boot time drivers need to stop interacting with the hardware, but where there is no real replacement of the features provided by an operating system. I do not know of any examples of this type of application.
Summary
A shell is a very convenient place to run applications and provide great access to all of the hardware. New UEFI shell interfaces and scripts provide a convenient, cheap, and flexible solution for simple applications for diagnostics and manufacturing and even simple user applications. The UEFI shell basics are downloadable as open source from www.Tianocore.org and can be quickly implemented on top of a UEFI firmware solution.
For more in depth information, please see the book Harnessing the UEFI Shell.