How many times have you installed the application you need and found out that, as a side effect, it created a lot of files under different locations? The beginnings are usually promising—you download a single-file executable installer and start it, only to see that it extracts its contents to the location that the file was ran from. And have you ever tried to change the location of the already installed software? If you are looking for guaranteed troubles, go ahead. Then, you decide that you need to remove it, only to find out that even after following the recommended way of uninstalling it, some files and other data—such as application data, registry entries, file associations—remain on the system.

Why is there such a lack of flexibility? Why are there a lot of self-executable installers, but not ready-to-use applications? Is it really so hard to create a portable, one-file application? Have you ever wondered how such binaries are being developed and what their internal structure is? If you have, this chapter will bring you answers to these questions, explain how to do this using the Tcl language.

A few years ago, when Tcl was nothing more than just another scripting language for me, a friend showed me an application he had made. It was a normal .exe file, looking completely identical to any other software, and I was astonished to learn that it was developed entirely in Tcl in less than two hours! In fact, seeing that application inspired me to learn Tcl. Years have passed, but I still admire the charm and the elegant simplicity of this technology.

Our main objective, the Holy Grail of this chapter, is a self-contained, fully-functional single-file binary application. We will learn through step-by-step tutorials how to reach that goal using the most popular solution—the Starpacks technology which was developed by the Netherlands based company Equi4 Software (http://www.equi4.com). The choice of Starpacks is obvious when you realize that not only are the components freely available as open source software, they also provide a high degree of flexibility and support for a wide range of platforms.

First, the internal structure of such files will be presented with an explanation of how one file may contain many files in it. The underlying Metakit database will be introduced, but the real 'magic' is related to the Virtual File System (VFS), which allows us to treat the content of the database as a normal file system that contains a number of files.

We will also answer the question of how it is possible for a Tcl-based application to run on a system where no Tcl interpreter is installed. The trick is that the interpreter will now be a part of that application itself, and in our case, the solution is called Tclkit.

Following that, the Starkit, Starkit Developer eXtensions (SDX), and Starpack mechanisms will be explained and discussed in detail in order to finally describe how create our standalone application.

Finally, having learned about all the technologies that we have in hand, we will review all the constraints and possible workarounds.

We will also mention alternative solutions to SDX.

This chapter will conclude with a real-life scenario example utilizing all the described techniques.

Understanding the executable file structure

In the context of normal files, two basic types exist—data and executable.

1. Data: These files contain raw data, which can be interpreted and used by software which understands the format of the data. Although the format of such files may be platform-specific, there is a pretty good chance that it is portable and can be used under other operating systems, either directly or after some conversion. Plain Tcl scripts (that is *.tcl files) are considered to be data files, because they need to be interpreted by the actual Tcl binaries in order to be run. On Unix systems, such scripts are correctly run by specifying the binary name, but the script itself cannot be run on its own.
2. Executable: An executable file contains a set of instructions that are executed by the system when you run the file. The instructions are usually formatted in machine code (assembler), which is directly related to the type of hardware platform and operating system you are using. Yes, there are also other kinds of files that appear to be executable, such as scripts, but in fact, these are just data files containing instructions for the scripting language's interpreter (for example Tcl), not for the CPU itself. The format of binary files is complex and depends on the operating system; describing it in detail is beyond the scope of this book.

In reality, an executable file contains not only binary instructions, but also various kinds of data (metadata) such as headers. At the end of this chapter, you will learn how to create an executable binary that will contain both the Tcl interpreter (executable) and your custom data (such as scripts, Tcl extensions, images, icons, and so on ). To be more specific, these data are stored in a convenient structure referred to as the virtual file system (VFS). Just like a normal file system, the VFS allows us to have files and directories, but the difference is that they are not stored directly on the hard drive, but inside the file it contains, in a fashion similar to ZIP archives.

The overall structure of the executable file we are targeting will be similar to the following image:

Looking at this, a basic question arises—how will the software that is intended to use this custom data be able to determine where it starts? At the end of entire file, the size of the custom data is written, so if you know the total size of the file and the size of the data part, it is easy to calculate the offset where the data starts.

The data could written in almost any format, but as you may have noticed already, Tcl-ers do not like to reinvent the wheel, and they also tend to reuse existing solutions and adapt them to their needs. In this case, the Metakit database was chosen.

Before you start creating your own standalone executable programs, we will explain the core technologies required to achieve this goal, starting with a description of Metakit as a database. Then we will talk about the Tcl's virtual file system, which allows us to use the Metakit database as a container for files.