Chapter 10. Process Management

Processes are the primary abstraction an operating system uses to represent running programs. Unfortunately, the meaning of the term process varies widely. On most general-purpose operating systems, such as UNIX and Windows, a process is seen as a resource container that manages the address space and other resources of a program. This is the abstraction that is supported in ACE. Some operating systems, such as VxWorks, do not have processes at all but instead have one large address space in which tasks run. The ACE process classes are not pertinent for these operating systems.

In this chapter, we first explain how to use the simple ACE_Process wrapper class to create a process and then manage child process termination. Next, we discuss how you can protect globally shared system resources from concurrent access by one or more processes, using special mutexes for process-level locking. Finally, we look at the high-level process manager class that offers integration with the Reactor framework.

10.1 Spawning a New Process

ACE hides all process creation and control APIs from the user in the ACE_Process wrapper class. This wrapper class allows a programmer to spawn new processes and subsequently wait for their termination. You usually use one ACE_Process object for each new process and are allowed to set several options for the child process:

• Setting standard I/O handles

• Specifying how handle inheritance will work between the two processes

• Setting the child's environment block and command line

• Specifying security attributes on Windows or set uid/gid/euid on UNIX.

For those of you from the UNIX world, the spawn() method does not have semantics similar to the fork() system call but is instead similar to the system() function available on most UNIX systems. You can force ACE_Process to do a simple fork() but in most cases are better off using ACE_OS::fork() to accomplish a simple process fork, if you need it.

Spawning a process using the ACE_Process class is a two-step process.

1. Create a new ACE_Process_Options object specifying the desired properties of the new child process.
2. Spawn a new process using the ACE_Process::spawn() method.

In the next example, we illustrate creating a slave process and then waiting for it to terminate. To do this, we create an object of type Manager, which spawns another process running the same example program, albeit with different options. Once the slave process is created, it creates an object of type Slave, which performs some artificial work and exits. Meanwhile, the master process waits for the slave process to complete before it too exits.

Because the same program is run as both the master and the slave, the command line arguments are used to distinguish which mode it is to run in; if there are arguments, the program knows to run in slave mode and otherwise runs in master mode:

int ACE_TMAIN (int argc, ACE_TCHAR *argv[])
{
  if (argc > 1)   // Slave mode
    {
      Slave s;
      return s.doWork ();
    }

  // Else, Master mode
  Manager m (argv[0]);
  return m.doWork ();
}

The Manager class has a single public method that is responsible for setting the options for the new slave process, spawning it, and then waiting for its termination:

class Manager : public ACE_Process
{
public:
  Manager (const ACE_TCHAR* program_name)
  {
    ACE_TRACE ("Manager::Manager");
    ACE_OS::strcpy (programName_, program_name);
  }

  int doWork (void)
  {
    ACE_TRACE ("Manager::doWork");


    // Spawn the new process; prepare() hook is called first.
    ACE_Process_Options options;
    pid_t pid = this->spawn (options);
    if (pid == ACE_INVALID_PID)
      ACE_ERROR_RETURN((LM_ERROR, ACE_TEXT ("%p "),
                       ACE_TEXT ("spawn")), -1);


    // Wait forever for my child to exit.
    if (this->wait () == -1)
      ACE_ERROR_RETURN ((LM_ERROR, ACE_TEXT ("%p "),
                        ACE_TEXT ("wait")), -1);

    // Dump whatever happened.
    this->dumpRun ();
    return 0;
  }

An ACE_Process_Options object carries the options for the new process. A single options object can be used for multiple ACE_Process objects if desired. The ACE_Process object on the stack represents the process to be spawned. The process object is used to spawn a new process, based on the process options passed in. The spawn() method uses execvp() on UNIX and CreateProcess() on Windows. Once the child process has been spawned successfully, the parent process uses the wait() method to wait for the child to finish and exit. The wait() method collects the exit status of the child process and avoids zombie processes on UNIX. On Windows, the wait() method causes the closing of the process and thread HANDLEs that CreateProcess() created. Once the slave returns, the master prints out the activity performed by the slave and the master to the standard output stream.

Let's take a closer look at how we set up the options for the child process. The ACE_Process::spawn() method calls the ACE_Process::prepare() hook method on the process object before creating the new process. The prepare() method enables us to inspect and modify the options for the new process. This method is a very convenient place to set platform-specific options. Our example's prepare() hook method follows:

// prepare() is inherited from ACE_Process.
int prepare (ACE_Process_Options &options)
{
  ACE_TRACE ("Manager::prepare");


  options.command_line ("%s 1", this->programName_);
  if (this->setStdHandles (options) == -1 ||
      this->setEnvVariable (options) == -1)
    return -1;
#if !defined (ACE_WIN32)
  return this->setUserID (options);
#else
  return 0;
#endif
}


int setStdHandles (ACE_Process_Options &options)
{
  ACE_TRACE("Manager::setStdHandles");


  ACE_OS::unlink ("output.dat");
  this->outputfd_ =
    ACE_OS::open ("output.dat", O_RDWR | O_CREAT);
  return options.set_handles
    (ACE_STDIN, ACE_STDOUT, this->outputfd_);
}


int setEnvVariable (ACE_Process_Options &options)
{
  ACE_TRACE ("Manager::setEnvVariables");
  return options.setenv ("PRIVATE_VAR=/that/seems/to/be/it");
}

First, we set the command line to be the same program name as the current program (as the child and parent processes are both represented by the same program) plus a single argument. The extra argument indicates that the program is to run in slave mode. After this, the standard input, output, and error handles for the child process are set up. The input and output handles will be shared between the two processes, whereas the STDERR handle for the child is set to point to a newly created file, output.dat. We also show how to set up an environment variable in the parent process that the child process should be able to see and use.

For non-Windows runs, we also set the effective user ID we want the child process to run as. We discuss this in a little detail later.

If the prepare() hook returns 0, ACE_Process::spawn() continues to spawn the new process. If prepare() returns –1, the spawn() method will not attempt to spawn the new process.

To reiterate, the Manager goes through the following steps:

1. Calls spawn() to spawn the child process
2. ACE_Process::spawn() calls back to the prepare() hook to set up process options, including the command line, environment variables, and I/O streams
3. Waits for the child to exit
4. Displays the results of the child process run.

Now let's look at the how the Slave runs. The Slave class has a single method that exercises the validity of the input, output, and error handles, along with the environment variable that was created, and displays its own and its parent's process ID:

class Slave
{
public:
  Slave ()
  {
    ACE_TRACE ("Slave::Slave");
  }


  int doWork (void)
  {
    ACE_TRACE ("Slave::doWork");


    ACE_DEBUG ((LM_INFO,
                ACE_TEXT ("(%P) started at %T, parent is %d "),
                ACE_OS::getppid ()));

    this->showWho ();
    ACE_DEBUG ((LM_INFO,
                ACE_TEXT ("(%P) the private environment is %s "),
                ACE_OS::getenv ("PRIVATE_VAR")));

    ACE_TCHAR str[128];
    ACE_OS::sprintf (str, ACE_TEXT ("(%d) Enter your command "),
                     ACE_OS::getpid ());
    ACE_OS::write (ACE_STDOUT, str, ACE_OS::strlen (str));
    this->readLine (str);
    ACE_DEBUG ((LM_DEBUG, ACE_TEXT ("(%P) Executed: %C "),
                str));
    return 0;
  }

After the slave process is created, the doWork() method is called. This method

• Checks what the effective user ID of the program is.

• Checks and prints the private environment variable PRIVATE_VAR.

• Asks the user for a string command.

• Reads the string command from standard input.

• Prints the string back out again to the standard error stream. (Remember that all ACE_DEBUG() messages are set to go to the standard error stream by default.)

After determining that the slave has completed and exited, the master displays the output that the slave generated in the debug log. Because the standard error stream was set to the file output.dat, all the Manager object needs to do is dump this file. Note that when we set the STDERR handle for the child, we kept a reference to it open in the Master. We can use this open handle to do our dump. Because the file handle was shared between the slave and the master, we first seek back to the beginning of the file and then move forward:

int dumpRun (void)
{
  ACE_TRACE ("Manager::dumpRun");

  if (ACE_OS::lseek (this->outputfd_, 0, SEEK_SET) == -1)
    ACE_ERROR_RETURN ((LM_ERROR, ACE_TEXT ("%p "),
                       ACE_TEXT ("lseek")), -1);

  char buf[1024];
  int length = 0;

  // Read the contents of the error stream written
  // by the child and print it out.
  while ((length = ACE_OS::read (this->outputfd_,
                                 buf, sizeof(buf)-1)) > 0)
    {
      buf[length] = 0;
      ACE_DEBUG ((LM_DEBUG, ACE_TEXT ("%C "), buf));
    }

  ACE_OS::close (this->outputfd_);
  return 0;
}

10.1.1 Security Parameters

As mentioned earlier, ACE_Process_Options allows you to specify the effective, real, and group IDs that you want the child process to run with. Continuing with the previous example, the following code illustrates setting the effective user ID of the child process to be the user ID of the user nobody. Of course, for this to run on your system, you must make sure that there is a nobody account and that the user running the program has permission to perform the effective user ID switch.

int setUserID (ACE_Process_Options &options)
{
  ACE_TRACE ("Manager::setUserID");
  passwd* pw = ACE_OS::getpwnam ("nobody");
  if (pw == 0)
    return -1;
  options.seteuid (pw->pw_uid);
  return 0;
}

Note that this code works only for those systems that have a UNIX-like notion of these IDs. If you are using a Windows system, you can instead use the get_process_attributes() and set_process_attributes() methods to specify the SECURITY_ATTRIBUTES for the new process and its primary thread; however, you cannot use ACE_Process to spawn a process for client impersonation.

10.1.2 Other ACE_Process Hook Methods

In addition to the prepare() hook method that prepares process options, the ACE_Process class offers two other hook methods that can be overridden to customize processing:

1. parent(pid_t child) is called back in the parent process immediately after the fork() call on UNIX platforms or the CreateProcess() call on Windows.
2. child(pid_t parent) is called back in the child process after fork() completes but before the subsequent exec() call, which occurs if you do not specify ACE_Process_Options::NO_EXEC in the creation flags for the process options, the default case. At this point, the new enviroment, including handles and working directory, are not set. This method is not called back on Windows platforms, as there is no concept of a fork() and subsequent exec() here.

10.2 Using the ACE_Process_Manager

Besides the relatively simple ACE_Process wrapper, ACE also provides a sophisticated process manager, ACE_Process_Manager, which allows a user, with a single call, to spawn and wait for the termination of multiple processes. You can also register event handlers that are called back when a child process terminates.

10.2.1 Spawning and Terminating Processes

The spawn() methods available in the ACE_Process_Manager class are similar to those available with ACE_Process. Using them entails creating an ACE_Process_Options object and passing it to the spawn() method to create the process. With ACE_Process_Manager, you can additionally spawn multiple processes at once, using the spawn_n() method. You can also wait for all these processes to exit and correctly remove all the resources held by them. In addition, you can forcibly terminate a process that was previously spawned by ACE_Process_Manager.

The following example illustrates some of these new process manager features:

#include "ace/Process_Manager.h"

static const int NCHILDREN = 2;

int ACE_TMAIN (int argc, ACE_TCHAR *argv[])
{
  if (argc > 1)     // Running as a child.
    {
      ACE_OS::sleep (10);
    }
  else             // Running as a parent.
    {
      // Get the processwide process manager.
      ACE_Process_Manager* pm = ACE_Process_Manager::instance ();

      // Specify the options for the new processes
      // to be spawned.
      ACE_Process_Options options;
      options.command_line (ACE_TEXT ("%s a"), argv[0]);

      // Spawn two child processes.
      pid_t pids[NCHILDREN];
      pm->spawn_n (NCHILDREN, options, pids);

      // Destroy the first child.
      pm->terminate (pids[0]);

      // Wait for the child we just terminated.
      ACE_exitcode status;
      pm->wait (pids[0], &status);

      // Get the results of the termination.

#if !defined(ACE_WIN32)
      if (WIFSIGNALED (status) != 0)
        ACE_DEBUG ((LM_DEBUG,
                    ACE_TEXT ("%d died because of a signal ")
                    ACE_TEXT ("of type %d "),
                    pids[0], WTERMSIG (status)));
#else
      ACE_DEBUG
        ((LM_DEBUG,
          ACE_TEXT ("The process terminated with exit code %d "),
          status));
#endif /*ACE_WIN32*/
      // Wait for all (only one left) of the
      // children to exit.
      pm->wait (0);
    }

  return 0;
}

The ACE_Process_Manager is used to spawn NCHILDREN child processes. (Note that once again, we spawn the same program.) Once the child processes start, they immediately fall asleep. The parent process then explicitly terminates the first child, using the terminate() method. This should cause the child process to abort immediately. The only argument to this call is the process ID of the process that you wish to terminate. If you pass in the process ID 0, the process manager waits for any of its managed processes to exit. On UNIX platforms, that does not work as well as one would hope, and you may end up collecting the status for a process that is not managed by your process manager. (For more on this, see the ACE reference documentation.) Immediately after issuing the termination call on the child process, the parent uses the process manager to do a blocking wait() on the exit of that child. Once the child exits, the wait() call returns with the termination code of the child process.

Note that on UNIX systems, ACE_Process_Manager issues a signal to terminate the child once you invoke the terminate() method. You can observe this by examining the termination status of the process.

After completing the wait on the first process, the parent process waits for all the rest of its children by using another blocking wait() call on the process manager. To indicate this, we pass a 0 timeout value to the wait() method. Note that you can also specify a relative timeout value after which the blocking wait will return. If the wait is unsuccessful and a timeout does occur, the method returns 0.

10.2.2 Event Handling

In the previous example, we showed how a parent can block, waiting for all its children to complete. Most of the time, you will find that your parent process has other work to do besides waiting for terminating children, especially if you have implemented a traditional network server that spawns child processes to handle network requests. In this case, you will want to keep the parent process free to handle further requests besides reaping your child processes.

To handle this use case, the ACE_Process_Manager exit-handling methods have been designed to work in conjunction with the ACE Reactor framework. This next example illustrates how you can set up a termination callback handler that is called back whenever a process is terminated:

class DeathHandler: public ACE_Event_Handler
{
public:
  DeathHandler () : count_(0)
  {
    ACE_TRACE (ACE_TEXT ("DeathHandler::DeathHandler"));
  }

  virtual int handle_exit (ACE_Process * process)
  {
    ACE_TRACE (ACE_TEXT ("DeathHandler::handle_exit"));

    ACE_DEBUG
      ((LM_DEBUG,
        ACE_TEXT ("Process %d exited with exit code %d "),
        process->getpid (), process->return_value ()));

    if (++count_ == NCHILDREN)
      ACE_Reactor::instance ()->end_reactor_event_loop ();

    return 0;
  }

private:
  int count_;
};

In this program, we create an ACE_Event_Handler subclass called DeathHandler that is used to handle process termination events for all the NCHILDREN processes that are spawned by the process manager. When a process exits, the reactor synchronously invokes the handle_exit() method on the event handler, passing in a pointer to the ACE_Process object representing the process that has just exited. This works under the hood as follows.

• On POSIX platforms, ACE_Process_Manager is registered to receive the SIGCHLD signal. On receipt of the signal, ACE_Process_Manager uses the reactor's notification mechanism to regain control in normal process context.

• On Windows platforms, ACE_Process_Manager is registered to receive event notifications on the process handle via the Reactor framework. Because only the ACE_WFMO_Reactor reactor implementation supports the handle notification capability, child exit notification does not work if you change the reactor implementation on Windows.

When ACE_Process_Manager is notified that the child process has exited, it invokes the handler's handle_exit() method:

int ACE_TMAIN (int argc, ACE_TCHAR *argv[])
{
  if (argc > 1)      // Running as a child.
    return 0;

  // Instantiate a process manager with space for
  // 10 processes.
  ACE_Process_Manager pm (10, ACE_Reactor::instance ());

  // Create a process termination handler.
  DeathHandler handler;

  // Specify the options for the new processes to be spawned.
  ACE_Process_Options options;
  options.command_line (ACE_TEXT ("%s a"), argv[0]);

  // Spawn two child processes.
  pid_t pids[NCHILDREN];
  pm.spawn_n (NCHILDREN, options, pids);

  // Register handler to be called when these processes exit.
  for (int i = 0; i < NCHILDREN; i++)
    pm.register_handler (&handler, pids[i]);

  // Run the reactor event loop waiting for events to occur.
  ACE_Reactor::instance ()->run_reactor_event_loop ();

  return 0;
}

10.3 Synchronization Using ACE_Process_Mutex

To synchronize threads, you need synchronization primitives, such as mutexes or semaphores. (We discuss these primitives in significant detail in Section 12.2.) When executing in separate processes, the threads are running in different address spaces. Synchronization between such threads becomes a little more difficult. In such cases, you can either

• Create the synchronization primitives that you are using in shared memory and set the appropriate options to ensure that they work between processes

• Use the special process synchronization primitives that are provided as a part of the ACE library

ACE provides a number of process-scope synchronization classes that are analogous to the thread-scope wrappers discussed in Chapter 14. This section explains how you can use the ACE_Process_Mutex class to ensure synchronization between threads running in different processes.

ACE provides, in the form of the ACE_Process_Mutex class, for named mutexes that can be used across address spaces. Because the mutex is named, you can recreate an object representing the same mutex by passing in the same name to the constructor of ACE_Process_Mutex.

In the next example, we create a named mutex: GlobalMutex. We then create two processes that cooperatively share an imaginary global resource, coordinating their access by using the mutex. The two processes both do this by creating an instance of the GResourceUser, an object that intermittently uses the globally shared resource.

We use the same argument-length trick that we used in the previous example to start the same program in different modes. If the program is started to be the parent, it spawns two child processes. If started as a child process, the program gets the named mutex GlobalMutex from the OS by instantiating an ACE_Process_Mutex object, passing it the name GlobalMutex. This either creates the named mutex—if this is the first time we asked for it—or attaches to the existing mutex—if the second process does the construction. The mutex is passed to a resource-acquirer object that uses it to ensure protected access to a global resource.

Again, note that even though we create separate ACE_Process_Mutex objects—each child process creates one—they both refer to the same shared mutex. The mutex itself is managed by the operating system, which recognizes that both mutex instances refer to the same GlobalMutex:

int ACE_TMAIN (int argc, ACE_TCHAR *argv[])
{
  if (argc > 1)      // Run as the child.
    {
      // Create or get the global mutex.
      ACE_Process_Mutex mutex ("GlobalMutex");

      GResourceUser acquirer (mutex);
      acquirer.run ();
    }
  else              // Run as the parent.
    {
      ACE_Process_Options options;
      options.command_line ("%s a", argv[0]);
      ACE_Process processa, processb;

      pid_t pida = processa.spawn (options);
      pid_t pidb = processb.spawn (options);

      ACE_DEBUG ((LM_DEBUG,
                  ACE_TEXT ("Spawned processes; pids %d:%d "),
                  pida, pidb));

      if (processa.wait() == -1)
        ACE_ERROR_RETURN ((LM_ERROR, ACE_TEXT ("%p "),
                           ACE_TEXT ("processa wait")), -1);

      if (processb.wait() == -1)
        ACE_ERROR_RETURN ((LM_ERROR, ACE_TEXT ("%p "),
                           ACE_TEXT ("processb wait")), -1);
    }

  return 0;
}

The GResourceUser class represents a user of an unspecified global resource that is protected by a mutex. When this class's run() method is called, it intermittently acquires the global mutex, works with the global resource, and then releases the mutex. Because it releases the resource between runs, the second process a chance to acquire it:

class GResourceUser
{
public:
  GResourceUser (ACE_Process_Mutex &mutex) : gmutex_(mutex)
  {
    ACE_TRACE (ACE_TEXT ("GResourceUser::GResourceUser"));
  }

  void run (void)
  {
    ACE_TRACE (ACE_TEXT ("GResourceUser::run"));

    int count = 0;
    while (count++ < 10)
      {
        int result = this->gmutex_.acquire ();
        ACE_ASSERT (result == 0);

        ACE_DEBUG ((LM_DEBUG,
                    ACE_TEXT ("(%P| %t) has the mutex ")));

        // Access Global resource
        ACE_OS::sleep (1);

        result = this->gmutex_.release ();
        ACE_ASSERT (result == 0);
        ACE_OS::sleep (1);     // Give other process a chance.
      }
  }

private:
  ACE_Process_Mutex &gmutex_;
};

The results from running the program show the two processes competing with each other to acquire the shared resource:

(8077| 1024) has the mutex
Spawned processes; pids 8077:8078

(8078| 1024) has the mutex
(8077| 1024) has the mutex
(8078| 1024) has the mutex
(8077| 1024) has the mutex
(8078| 1024) has the mutex
(8077| 1024) has the mutex
(8078| 1024) has the mutex
(8077| 1024) has the mutex
(8078| 1024) has the mutex
(8077| 1024) has the mutex
(8078| 1024) has the mutex
(8077| 1024) has the mutex
(8078| 1024) has the mutex
(8077| 1024) has the mutex
(8078| 1024) has the mutex
(8077| 1024) has the mutex
(8078| 1024) has the mutex
(8077| 1024) has the mutex
(8078| 1024) has the mutex

For UNIX users, the ACE_Process_Mutex maps to a System V shared semaphore. Unlike most resources, these semaphores are not automatically released once all references to it are destroyed. Therefore, be careful when using this class, and make sure that the destructor of the ACE_Process_Mutex is called, even in the case of abnormal exits. Another option for UNIX users is to use ACE_SV_Semaphore_Complex, which will automatically reference count and remove the mutex once all processes referencing it have exited. This automatic reference counting/removal process will work if any process other than the last does an exit, intentional or unintentional, without properly closing the mutex.

10.4 Summary

In this chapter, we introduced the ACE classes that support process creation, life-cycle management, and synchronization. We looked at the simple ACE_Process wrapper and the sophisticated ACE_Process_Manager. We also looked at synchronization primitives that can be used to synchronize threads that are running in separate processes.