Using GObject Introspection + Gjs to provide a JavaScript plugin engine

In writing the Capa photo capture application, one of the things I wanted to support was some form of plugin engine to allow 3rd parties to easily extend its functionality. The core application code itself is designed to have a formal separation of backend and frontend logic. The backend is focused on providing the core object model & operation, typically wrapping external libraries like HAL, libgphoto, lcms in GObject classes, with no use of GTK allowed here. The primary frontend builds on this backend, to produce a GTK based user interface. It is also intended to build another frontend that provides a GIMP plugin.

Back to the question of plugins for the main frontend. If the goal is to allow people to easily write extensions, a plugin engine based on writing C code is not really very desirable. Firefox uses JavaScript for its plugin engine and this has been hugely successful in lowering the bar for contributors. Wouldn’t it be nice if any GTK application could provide a JavaScript plugin engine ? Yes, indeed and thanks to the recent development of GObject introspection this is incredibly easy.

GObject introspection provides a means to query the GObject type system and discover all classes, interfaces, methods, properties, signals, all data types associated with their parameters and any calling conventions. This is an incredibly powerful capability with far reaching implications, the most important being that you will never again have to write a language binding for any GObject based library. There is enough metadata available in the GObject introspection system to provide language bindings in a 100% automated fashion. Notice I said “provide”, rather than “generate” because if targetting a dynamic language (Perl, Python JavaScript) it won’t even be necessary to auto-generate code ahead of time – everything can and will happen at runtime based on the introspection data. Say goodbye to hand written language bindings. Say goodbye to Swig. Say goodbye to any other home grown code generators.

Adding support for introspection

That’s the sales pitch, how about the reality ? The Capa code is based on GObject and was thus ready & willing to be introspected. The first step in adding introspection support is to add some m4 magic to the configure.ac to look for the introspection tools & library. This is simple boilerplate code that will be identical for every application using GObject + autoconf

GOBJECT_INTROSPECTION_REQUIRED=0.6.2
AC_SUBST(GOBJECT_INTROSPECTION_REQUIRED)

AC_ARG_ENABLE([introspection],
        AS_HELP_STRING([--enable-introspection], [enable GObject introspection]),
        [], [enable_introspection=check])

if test "x$enable_introspection" != "xno" ; then
        PKG_CHECK_MODULES([GOBJECT_INTROSPECTION],
                          [gobject-introspection-1.0 >= $GOBJECT_INTROSPECTION_REQUIRED],
                          [enable_introspection=yes],
                          [
                             if test "x$enable_introspection" = "xcheck"; then
                               enable_introspection=no
                             else
                               AC_MSG_ERROR([gobject-introspection is not available])
                             fi
                          ])
        if test "x$enable_introspection" = "xyes" ; then
          AC_DEFINE([WITH_GOBJECT_INTROSPECTION], [1], [enable GObject introspection support])
          AC_SUBST(GOBJECT_INTROSPECTION_CFLAGS)
          AC_SUBST(GOBJECT_INTROSPECTION_LIBS)
          AC_SUBST([G_IR_SCANNER], [$($PKG_CONFIG --variable=g_ir_scanner gobject-introspection-1.0)])
          AC_SUBST([G_IR_COMPILER], [$($PKG_CONFIG --variable=g_ir_compiler gobject-introspection-1.0)])
        fi
fi
AM_CONDITIONAL([WITH_GOBJECT_INTROSPECTION], [test "x$enable_introspection" = "xyes"])

The next step is to add Makefile.am rules to extract the introspection data. This is a two step process, the first step runs g-ir-scanner across all the source code and the actual compiled binary / library to generate a .gir file. This is an XML representation of the introspection data. The second step runs g-ir-compiler to turn the XML data into a machine usable binary format so it can be efficiently accessed. When running g-ir-scanner on a binary, as opposed to a library, it is necessary for that binary to support an extra command line flag called --introspect-dump. I add this code the main.c source file to support that

#if WITH_GOBJECT_INTROSPECTION
    static gchar *introspect = NULL;
#endif

    static const GOptionEntry entries[] = {
        ...snip other options...
#if WITH_GOBJECT_INTROSPECTION
        { "introspect-dump", 'i', 0, G_OPTION_ARG_STRING, &introspect;, "Dump introspection data", NULL },
#endif
        { NULL, 0, 0, 0, NULL, NULL, NULL },
    };

    ...parse command line args...

#if WITH_GOBJECT_INTROSPECTION
    if (introspect) {
        g_irepository_dump(introspect, NULL);
        return 0;
    }
#endif

Back to the Makefile.am rules. g-ir-scanner has quite a few arguments you need to set. The --include args provide the names of introspection metadata files for any libraries depended on. The -I args provide the CPP include paths to the application’s header files. The --pkg args provide the names of any pkg-config files that code builds against. There are a few others too which I won’t cover – they’re all in the man page. The upshot is that the Makefile.am gained rules

if WITH_GOBJECT_INTROSPECTION
Capa-0.1.gir: capa $(G_IR_SCANNER) Makefile.am
        $(G_IR_SCANNER) -v \
                --namespace Capa \
                --nsversion 0.1 \
                --include GObject-2.0 \
                --include Gtk-2.0 \
                --include GPhoto-2.0 \
                --program=$(builddir)/capa \
                --add-include-path=$(srcdir) \
                --add-include-path=$(builddir) \
                --output $@ \
                -I$(srcdir)/backend \
                -I$(srcdir)/frontend \
                --verbose \
                --pkg=glib-2.0 \
                --pkg=gthread-2.0 \
                --pkg=gdk-pixbuf-2.0 \
                --pkg=gobject-2.0 \
                --pkg=gtk+-2.0 \
                --pkg=libgphoto2 \
                --pkg=libglade-2.0 \
                --pkg=hal \
                --pkg=dbus-glib-1 \
                $(libcapa_backend_la_SOURCES:%=$(srcdir)/%) \
                $(libcapa_frontend_la_SOURCES:%=$(srcdir)/%) \
                $(capa_SOURCES:%=$(srcdir)/%)

girdir = $(datadir)/gir-1.0
gir_DATA = Capa-0.1.gir

typelibsdir = $(libdir)/girepository-1.0
typelibs_DATA = Capa-0.1.typelib

%.typelib: %.gir
        g-ir-compiler \
                --includedir=$(srcdir) \
                --includedir=$(builddir) \
                -o $@ $<

CLEANFILES += Capa-0.1.gir $(typelibs_DATA)

endif # WITH_GOBJECT_INTROSPECTION

After making those changes & rebuilding, it is wise to check the .gir file, since the g-ir-scanner doesn't always get everything correct. It may be necessary to provide annotations in the source files to help it out. For example, it got object ownership wrong on some getters, requiring annotations n the return values such as

/**
 * capa_app_get_plugin_manager: Retrieve the plugin manager
 *
 * Returns: (transfer none): the plugin manager
 */

The final step was add rules to the RPM specfile, which are fairly self-explanatory

%define with_introspection 0

%if 0%{?fedora} >= 12
%define with_introspection 1
%endif
%if 0%{?rhel} >= 6
%define with_introspection 1
%endif

%if %{with_introspection}
BuildRequires: gobject-introspection-devel
BuildRequires: gir-repository-devel
%endif


%prep
....
%if %{with_introspection}
%define introspection_arg --enable-introspection
%else
%define introspection_arg --disable-introspection
%endif

%configure %{introspection_arg}

%files
....
%if %{with_introspection}
%{_datadir}/gir-1.0/Capa-0.1.gir
%{_libdir}/girepository-1.0/Capa-0.1.typelib
%endif

That is all. The entire API is now accessible from Perl, JavaScript, Python without ever having written a line of code for those languages. It is also possible to generate a .jar file to make it accessible from Java.

Adding support for a JavaScript plugin engine

Since the API is now accessible from JavaScript, adding a JavaScript plugin engine ought to be easy at this point. There are in fact 2 competing JavaScript engines supporting GObject introspection, Gjs and Seed. Seed looks more advanced, documented & polished, but Gjs was what's in Fedora currently, so I used that. Again the first step was checking for it in configure.ac

AC_ARG_WITH([javascript],
      AS_HELP_STRING([--with-javascript],[enable JavaScript plugins]),
      [], [with_javascript=check])

if test "x$with_javascript" != "xno" ; then
  if test "x$enable_introspection" = "xno" ; then
    if test "x$with_javascript" = "xyes"; then
      AC_MSG_ERROR([gobject-introspection is requird for javascript plugins])
    fi
  fi

  PKG_CHECK_MODULES(GJS, gjs-1.0 >= $GJS_REQUIRED)
  AC_SUBST(GJS_CFLAGS)
  AC_SUBST(GJS_LIBS)

  PKG_CHECK_MODULES(GJS_GI, gjs-gi-1.0 >= $GJS_REQUIRED)
  AC_SUBST(GJS_GI_CFLAGS)
  AC_SUBST(GJS_GI_LIBS)

  with_javascript=yes
  AC_DEFINE([WITH_JAVASCRIPT], [1], [enable JavaScript plugins])
fi
AM_CONDITIONAL([WITH_JAVASCRIPT], [test "x$with_javascript" = "xyes"])

I won't go into any details on the way Capa scans for plugins (it uses $HOME/.local/share/capa/plugins//main.js), merely illustrate how to execute a plugin once it has been located. The important object in the Gjs API is GjsContext, providing the execution context for the javascript code. It is possible to have multiple contexts, so each plugin is independent and potentially able to be sandboxed. The JavaScript file to be invoked is main.js in the plugin's base directory. The first step is to setup the context's search path to point to the plugin base directory:


void runplugin(const gchar *plugindir) {
    const gchar *searchpath[2];
    GjsContext *context;

    searchpath[0] = plugindir;
    searchpath[1] = NULL;

    context = gjs_context_new_with_search_path((gchar **)searchpath);

The context is now ready to execute some javascript code. The Capa plugin system expects the main.js file to contain a method called activate. To start the plugin, we can thus simply evaluate const Main = imports.main; Main.activate();

   const gchar *script = "const Main = imports.main; Main.activate();";

   gjs_context_eval(context,
                     script,
                     -1,
                     "main.js",
                     &status;,
                     NULL);

   if (status !=0) {
     fprintf(stderr, "Loading plugin failed\n");
   }

Presto, you now have a javascript plugin running, having written no JavaScript at any point in the process. There is one slight issue in this though - how does the plugin get access to the application instance ? One way would be to provide a static method in your API to get hold of the application's main object, but I really wanted to pass the object into the plugin's activate method. This is where I hit Gjs's limitations - there appears to be no official API to set any global variable except for ARGV. After much poking around in the Gjs code though I discovered an exported method, which wasn't in the header files

JSContext* gjs_context_get_context(GjsContext *js_context);

And decided to (temporarily) abuse that until a better way could be found. I have an object instance of the CapaApp class which I wanted to pass into the activate method. The first step was to set this in the global namespace of the script being evaluated. Gjs comes with an API for converting a GObject instance into a JSObject instance which the runtime needs. Thus I wrote a simple helper

static void set_global(GjsContext *context,
                       const char *name,
                       GObject *value)
{
    JSContext *jscontext;
    JSObject *jsglobal;
    JSObject *jsvalue;

    jscontext = gjs_context_get_context(context);
    jsglobal = JS_GetGlobalObject(jscontext);
    JS_EnterLocalRootScope(jscontext);
    jsvalue = gjs_object_from_g_object(jscontext, value);
    JS_DefineProperty(jscontext, jsglobal,
                      name, OBJECT_TO_JSVAL(jsvalue),
                      NULL, NULL,
                      JSPROP_READONLY | JSPROP_PERMANENT);
    JS_LeaveLocalRootScope(jscontext);
}

There was one little surprise in this though. The gjs_object_from_g_object method will only succeed if the current Gjs context has the introspection data for that object loaded. So it was necessary to import my application's introspection data by eval'ing const Capa = imports.gi.Capa. That done, it was now possible to pass variables into the plugin. The complete revised plugin loading code looks like

void runplugin(CapaApp *application, const gchar *plugindir) {
    const gchar *script = "const Main = imports.main; Main.activate(app);";
    const gchar *searchpath[2];
    GjsContext *context;

    searchpath[0] = plugindir;
    searchpath[1] = NULL;

    context = gjs_context_new_with_search_path((gchar **)searchpath);

    gjs_context_eval(context,
                     "const Capa = imports.gi.Capa",
                     -1,
                     "dummy.js",
                     &status;,
                     NULL);

    set_global(context, plugin, "app", application);

    gjs_context_eval(context,
                     script,
                     -1,
                     "main.js",
                     &status;,
                     NULL);

    if (status !=0) {
      fprintf(stderr, "Loading plugin failed\n");
    }

This code is slightly simplified, omitting error handling, for purposes of this blog post, but the real thing is not much harder. Looking at the code again, there is really very little (if anything) about the code which is specific to my application. It would be quite easy to pull out the code which finds & loads plugins into a library (eg "libgplugin"). This would make it possible for any existing GTK applications to be retrofitted with support plugins simply by generating introspection data for their internal APIs, and then instantiating a "PluginManager" object instance.

In summary, GObject Introspection is an incredibly compelling addition to GLib. With a mere handful of additions to configure.ac and Makefile.am, it completely solves "language bindings" problem for you. I'd go as far as to say that this is a single most compelling reason to write any new C libraries using GLib/GObject. Furthermore if there are existing C libraries not using GObject, then provide a GObject wrapper for them as a top priority. Don't ever write or auto-generate a language binding again. Writing GTK applications either entirely in JavaScript, or in a mix of C + JavaScript plugins is also a really nice development, avoiding the issue of "clashing runtime environments" seen when using Python + GTK. The Gjs/Seed/GObject developers deserve warm praise for these great enhancements.