Usage of the libvirt virCommand APIs for process spawning

The previous blog post looked at the history of libvirt APIs for spawning processes, up to the current day where there is a single virCommand object + APIs for spawning processes in a very flexible manner. This blog post will now look at the key features of this API and how it is used in practice.

Example usage

Before going into the dry details, lets consider a couple of real world examples where libvirt uses these APIs.

“system” replacement

As a first example, the “virNodeSuspendSetNodeWakeup” method is a place where libvirt might have traditionally used the ‘system’ call.

The goal is to suspend the host, setting a pre-defined wakeup alarm, for which libvirt needs to run the ‘rtcwake’ command. The wakeup time is provided to the API in terms of a unsigned long long which needs to be converted to a string.

If attempting this with the ‘system’ call the code might look like this:

static int virNodeSuspendSetNodeWakeup(unsigned long long alarmTime)
{
    char *setAlarmCmd;
    int ret = -1;

    if (asprintf(&setAlarmCmd, "rtcwake -m no -s %lld", alarmTime) < 0)
        goto cleanup;

    if (system(setAlarmCmd) < 0)
        goto cleanup;

    ret = 0;

 cleanup:
    free(setAlarmCmd);
    return ret;
}

Now consider what this would look like when using the virCommand APIs:

static int virNodeSuspendSetNodeWakeup(unsigned long long alarmTime)
{
    virCommandPtr setAlarmCmd;
    int ret = -1;

    setAlarmCmd = virCommandNewArgList("rtcwake", "-m", "no", "-s", NULL);
    virCommandAddArgFormat(setAlarmCmd, "%lld", alarmTime);

    if (virCommandRun(setAlarmCmd, NULL) < 0)
        goto cleanup;

    ret = 0;

 cleanup:
    virCommandFree(setAlarmCmd);
    return ret;
}

The difference in code complexity is negligible, but the difference in the quality of the implementation is significant.

“popen” replacement

As a second example, the “virStorageBackendIQNFound” method is a place where libvirt might have traditionally used the ‘popen’ call.

The goal this time is to run the iscsiadm command with a number of arguments and parse its stdout to look for a particular iSCSI target.

First consider what this might look like when using ‘popen’

static int virStorageBackendIQNFound(const char *initiatoriqn,
                                          char **ifacename)
{
    int ret = -1;
    FILE *fp = NULL;
    int fd = -1;
    char line[4096]

    if ((fp = popen("iscsiadm --mode iface")) == NULL)
        goto cleanup;

    while (fgets(line, 4096, fp) != NULL) {
       ...analyse line for a match... 
    }

    ret = 0;

cleanup:
    pclose(fp);
    close(fd);
    virCommandFree(cmd);

    return ret;
}

Now consider the re-write to use virCommand APIs

static int virStorageBackendIQNFound(const char *initiatoriqn,
                                          char **ifacename)
{
    int ret = -1;
    FILE *fp = NULL;
    int fd = -1;
    char line[4096]
    virCommandPtr cmd = virCommandNewArgList("iscsiadm", "--mode", "iface", NULL);

    virCommandSetOutputFD(cmd, &fd);
    if (virCommandRunAsync(cmd, NULL) < 0)
        goto cleanup;

    if ((fp = fdopen(fd, "r")) == NULL)
        goto cleanup;

    while (fgets(line, 4096, fp) != NULL) {
       ...analyse line for a match... 
    }
 
    if (virCommandWait(cmd, NULL) < 0)
        goto cleanup;
 
    ret = 0;

cleanup:
    fclose(fp);
    close(fd);
    virCommandFree(cmd);

    return ret;
}

There is a little more work todo for virCommand in terms of initial setup in this example. Technically the call to ‘virCommandWait’ could have been omitted here, since we don’t care about the exit status, but it is good practice to included it. If we had extra dynamic arguments to be provided to ‘iscsiadm’ that needed string formatting the two examples would have been nearer parity in terms of complexity. Even with the slightly longer code for virCommand, the result is a clear win from the quality POV avoiding the many flaws of popen’s implementation.

Detailed API examination

Now that the two examples have given a taste of what the virCommand APIs can do to replace popen/system, lets consider the full set of features exposed. After this it should be clear that the flexibility of the virCommand means there is never any need to delve into fork+exec anymore, let alone popen/system.

Constructing the command arguments

Probably the first task when spawning a command is actually construct the array of arguments and environment variables. In simple cases this can be done immediately when allocating the new virCommad object instance, for example using var-args

virCommandPtr cmd = virCommandNewArgList("touch", "/tmp/foo", NULL);

there is no need to check the return value of virCommandNew* for NULL. The later virCommandRun() API will look for a NULL pointer and report the OOM error at that point instead. The same is true for all error reporting in these APIs – virCommandRun is generally the only place where an error check is needed. This simplification in error handling is a major contributor to making this APIs hard to mis-use in calling code and thus minimizing errors.

Sometimes the list of arguments is not so simple that it can be initialized in one go via var-args. To deal with this it is possible to add arguments to an existing constructed command in a variety of ways

virCommandAddArgFormat(cmd, "--size=%d", 1025);
virCommandAddArgPair(cmd, "--user", "fred");
virCommandAddArgList(cmd, "some", "extra", "args", NULL);

This is handy because for complex command lines (eg those used with QEMU) it allows construction of the virCommand to be split up across multiple functions, each adding their own piece of the command line.

Setting up the environment

By default a process spawned will inherit the full environment of the parent process (almost always libvirtd in libvirt code). With things like QEMU though libvirt wants to be in complete control of the environment it runs under, so it will filter the environment to a subset of names. There are a couple of env variables that are always desired to pass down LD_PRELOAD, LD_LIBRARY_PATH, PATH, USER, HOME, LOGNAME & TMPDIR. If this set is desired there is a convenient method to request passthrough of this set

virCommandAddEnvPassCommand(cmd);

Additional environment variables can be set for passthrough from libvirtd. When passing through environment variables libvirt requires an explicit decision on whether the env variable is safe to pass when running setuid. If an env variable is considered unsafe for a setuid application, there is the option of passing a default value to substitute. The “PATH” variable is unsafe to pass when setuid, and should be set to a known safe value when running setuid:

virCommandAddEnvPassBlockSUID(cmd, "PATH", "/bin:/usr/bin");

The “LD_LIBRARY_PATH” variable is also unsafe when running setuid and should simply be dropped from the environment entirely:

virCommandAddEnvPassBlockSUID(cmd, "LD_LIBRARY_PATH", NULL);

Finally the “LOGNAME” is fine to allow even when setuid so can be left unchanged

virCommandAddEnvPassAllowSUID(cmd, "LOGNAME");

It is not always sufficient to just passthrough existing environment variables, so there are of course APIs to set them directly

virCommandAddEnvPair(cmd, "LOGNAME", "fred");
virCommandAddEnvFormat(cmd, "LOGNAME=%s", fred);
virCommandAddEnvString(cmd, "LOGNAME=fred");

Setting security attributes

Under UNIX a program will inherit process limits, umask and working directory from the parent. It is thus desirable to be able to override this, for example:

virCommandSetMaxFiles(cmd, 65536);

Along a similar vein the child’s umask can also be set

virCommandSetUmask(cmd, 0007);

If the current process’ working directory is unknown, it is a good idea to force an explicit working directory:

virCommandSetWorkingDirectory(cmd, "/");

It may sometimes be desirable to control what capabilities bits a child process has, to override the default behaviour. In such cases the command would to be initialized with the empty set and then the desired bits whitelisted.

virCommandClearCaps(cmd);
virCommandAllowCap(cmd, CAP_NET_RAW);

Finally when interacting with mandatory access control systems like SELinux or AppArmour it is possible to configure an explicit label for the child

virCommandSetSELinuxLabel(cmd, "system_u:system_r:svirt_t:s0:c135,c275");
virCommandSetAppArmorProfile(cmd, "2cb0e828-e6f6-40d1-b0f5-c50cdf34f5c9");

Interacting with stdio

A common requirement when spawning processes is to be able to interact with the child’s stdio in some manner. By default with the virCommand APIs, a process will get its stdin, stdout & stderr connected to /dev/null. For stdin there is a choice of feeding it a fixed length string, or connecting it up to the read end of an existing file descriptor, typically a pipe

virCommandSetInputBuffer(cmd, "Feed me brains");
virCommandSetInputFD(cmd, pipefd);

There are a similar pair of choices for receiving the stdout/stderr from the child. It is possible to supply a pointer to a ‘char *’ which will be filled with the child’s output upon exit. Alternatively a pointer to an ‘int’ can be provided, which can either specify an existing file descriptor, or if ‘-1’ a new anonymous pipe will be created.

char *child_out, *child_err;
virCommandSetOutputBuffer(cmd, &child_out);
virCommandSetErrorBuffer(cmd, &child_err);

int child_out -1, child_err = -1;
virCommandSetOutputFD(cmd, &child_out);
virCommandSetOutputFD(cmd, &child_err);

Passing file descriptors

Aside from any associated with stdio, all file descriptors will be closed when the child process is launched. This is generally a good thing since it prevents any leakage of file descriptors to the child. Such leakage can be a security flaw, and unless using glibc extensions to POSIX, it is not possible to avoid the race condition with setting O_CLOEXEC, so an explicit mass close is a very desirable approach. There can be times when it is is necessary to pass other arbitrary file descriptors to a child process. When passing a file descriptor it may also be desirable to close it in the parent process.

virCommandPassFD(cmd, 8, 0);
virCommandPassFD(cmd, 8, VIR_COMMAND_PASS_FD_CLOSE_PARENT);

Pre-exec callback hooks

Despite its wide range of features, there are times when the virCommand API is not sufficient for the job. In these cases there is the ability to request that a callback function be invoked immediately before exec’ing the child binary. This allows the caller to do arbitrary extra work, though of course bearing in mind POSIX’s rules about which functions are safe to use between fork+exec in a threaded application.

int my_hook(void *data)
{
    if (sometask(data) < 0)
        return -1;
    return 0;
}

virCommandSetPreExecHook(cmd, my_hook, "thefilename");

Executing the command

Everything until this point has been about setting up the command args and the constraints under which it will execute. None of the APIs shown so far require any kind of error checking of return values. Only now that it is time to execute the command will errors be reported and checked by the caller. The simplest way to execute is to use ‘virCommandRun’ which will block until the command finishes running and report the exit status

int status;
if (virCommandRun(cmd, &status) < 0) {
    virCommandFree(cmd);
    return -1;
}

virCommandFree(cmd);

It is possible to leave the ‘status’ arg as NULL in which case the API will turn any non-zero exit status into a fatal error. If the intention is to interact with the command via one or more file descriptors connected to stdio, then a slightly more flexible ‘virCommandRunAsync’ API is required. This call will only block until the process is actually running. The parent can then interact with it and when ready call ‘virCommandWait’.

int status;
pid_t child;
if (virCommandRunAsync(cmd, &child) < 0) {
    virCommandFree(cmd);
    return -1;
}

...interact with child via stdio or other means...
if (virCommandWait(cmd, &status) < 0) {
    virCommandFree(cmd);
    return -1;
}

virCommandFree(cmd);

If something goes wrong during interaction, it is possible to terminate the process with prejudice by using ‘virCommandAbort’ instead of ‘virCommandWait’.

Synchronizing with the child

Normally, once a child has fork’d off, the child & parent will continue execution in parallel, with the parent having no idea at which point the final exec() will have been performed. There can be cases where the parent needs to do some work in between the fork and exec. A pre-exec hook can often be used for this, but the work needs to take place in the parent process another solution is required. To deal with this it is possible to request a handshake take place with the child process. Before the child exec’s its binary, it will notify the parent process that is wants to handshake and wait for a reply. The parent process meanwhile will wait to be notify, then do its work and finally reply to the child again

virCommandRequireHandshake(cmd);

if (virCommandRunAsync(cmd) < 0)
    return -1;

virCommandHandshakeWait(cmd);
...do setup work...
virCommandHandshakeNotify(cmd);

Simplified command execution

The examples above are very powerful, but in the simplest use cases it is possible to combine the virCommandNew + virCommandRun + virCommandFree calls into a single API call

const char *args[] = { "/bin/program", "arg1", "arg2", NULL };
if (virRun(args, NULL) < 0)
  return -1;

This is pretty much equivalent to ‘system’ in terms of complexity, but much safer as it avoids the shell and many other problems mentioned previously.

Integration with unit tests

One of the particularly interesting features of the virCommand APIs is the ability to do unit testing of code that otherwise spawns external commands. The test suite can define a callback that will be invoked any time an attempt is made to run a command. This callback can analyse stdin string, fill in stdout/stderr strings and set the exit status. This is sufficient to avoid the need to run the real command in the context of unit tests in most cases.

static void testCommandCallback(const char *const*args,
                                const char *const*env,
                                const char *input,
                                char **output,
                                char **error,
                                int *status,
                                void *opaque)
{
    ....fake the exit status and fill in **output or **error...
}

virCommandSetDryRun(NULL, testCommandCallback, NULL);

The End

That completes our whirlwind tour of libvirts APIs for spawning child processes. It should be clear that a lot of thought & effort has gone into designing a set of APIs that maximize safety without compromising on ease of use. There can really be no excuse for using either the popen or system calls for spawning programs & thus leaving yourself vulnerable to flaws like shellshock. The libvirt code described in this post is all available under the terms of the LGPLv2+ should anyone wish to pull out & adapt the virCommand APIs for their own programs. I look forward to the day when it is possible to use a Linux system with no reliance on shell by any program. Shell should be exclusively for use by interactive login sessions and administrator local scripting work, not a part of applications where it only ever leads to misery & insecurity.