Performing side effects after each edit in TopBraid EVN

TopBraid 4.3 introduces a small Event-Condition-Action (ECA) rule engine that can be used to trigger rules whenever a change was performed on an EVN vocabulary or ontology. Example use cases of this feature include anything that needs to happen as a side effect of edit operations: updating an external index (such as Solr), creating automatically generated triples, aligning inverse properties, sending automated notifications to co-workers etc.

Let me introduce this feature with an example rule definition. The rule below inserts an rdfs:comment whenever a new owl:DatatypeProperty has been created by someone.

As shown above, these ECA rules are essentially SWP elements that define their behavior is their ui:prototype. To create your own rule, simply create a globally registered .ui.* file that imports the teamworkrules system ontology, and then create a subclass of teamwork:EditRules. The rule engine of EVN will execute the ui:prototype of any globally registered edit rule after each edit. During the execution of those prototypes, there are two dedicated named graphs with a special meaning: The graph ui:addedGraph will contain the newly added triples, and ui:deletedGraph will contain those triples that have just been deleted. For example, if someone has pressed Save Changes on an EVN form, replacing one skos:altLabel with another, then the ui:addedGraph will contain the new value, and ui:deletedGraph will contain the old value. You can use SPARQL to query those change triples and perform side effects with ui:update or SPARQLMotion modules.

EVN uses this rule engine out of the box to update a Solr index after each edit (if Solr has been activated for the current project). The implementation of this Solr updater rule is shown below.

As this blog may be out of date when you read it, you can find the current implementation of this rule in the file teamworkrules.ui.ttlx. Note that this rule has a guard clause in the first ui:if statement, to make sure that the rule only fires for the master graph, i.e. the tag/working copy must be unbound. If this condition is true, then a helper SPIN template is called that will deliver all subjects that have been mentioned in the current change:

It is possible to distinguish between normal edit operations (teamwork:EditRule) and commit operations, in which a working copy has been merged into the master copy of an EVN project (teamwork:CommitRule).

Creating Web Services with the TopBraid Platform

TopBraid Live (TBL) is TopQuadrant's server platform for semantic data processing and editing. In addition to being the foundation of various out-of-the-box solution packages such as EVN, TopBraid Live can also be used "stand-alone". The features of the platform can be accessed in various ways, for example via a built-in SPARQL end point. In addition to those bundled web services, TopBraid Live can also host any number of custom web services created by users/administrators. This blog entry enumerates a few options on how to create such web services.

Common to all TopBraid custom web services is that they can be called either via GET or POST requests, and that they are created declaratively: the web services are stored in RDF and typically created with TopBraid Composer. This ensures that web services are seamlessly integrated with the data models/ontologies that they expose. Changes to an ontology will automatically update the web services etc. Furthermore, no "programming" in the traditional sense is needed to create those services.

SPARQLMotion

SPARQLMotion is an RDF-based scripting language with a graphical notations. SPARQLMotion scripts are data processing pipelines that can take some input (e.g. web service arguments), import and aggregate data from external sources, perform some processing (SPARQL etc) and export results in various formats. I had written about creating web services as early as 2008 but my colleagues have since then created much better tutorials on this topic. Note that SWP (see below) provides an alternative syntax for exposing SPARQLMotion functionality as web services.

SPIN Templates

In a nutshell, a SPIN Template is a SPARQL query that can take parameters which are mapped to pre-bound variables. A template is a bit like a stored procedure from SQL databases in that it encapsulates a complex query and gives it a name. Here is an example SPIN Template that provides information about the properties of a given class. The class itself comes in as an argument (arg:class) which is mapped to the pre-bound variable ?class in the body of the template.

Any SPIN Template saved in a .spin.* file can be called as a web service, via the tbl/template servlet. The following example calls the template above against the personal test server bundled with TopBraid Composer ME:

http://localhost:8083/tbl/template?
    _template=http://topbraid.org/swa%23GetRelevantPropertiesOfClass&
    _base=http://topbraid.org/examples/kennedys&
    class=http://topbraid.org/examples/kennedys%23Person&
   _format=application/sparql-results%2Bjson

The parameter _base points at the default query graph (if required) and _format can be any of the SPARQL result set formats including CSV, TSV, XML and JSON.

TopBraid 4.3 introduces another simpler result format, called JSON-simple, which is easy to handle by clients. Example output from the query above in JSON-simple is shown here:

[ { "propertyLabel" : "MI" ,
    "propertyURI" : "http://topbraid.org/examples/kennedys#middleInitial" ,
    "rangeURI" : "http://www.w3.org/2001/XMLSchema#string"
  },
  { "propertyLabel" : "alma mater" ,
    "propertyURI" : "http://topbraid.org/examples/kennedys#almaMater" ,
    "rangeURI" : "http://topbraid.org/examples/kennedys#College"
  },
   ...

Also new in TopBraid 4.3 is syntactic sugar that makes the query URLs easier to write:

http://localhost:8083/tbl/template/swa/GetRelevantPropertiesOfClass/kennedys?
    class=http://topbraid.org/examples/kennedys%23Person

More information on this topic can be found in the TBC help, under "TopBraid Live Integration".

SWP Services

SPARQL Web Pages (SWP) is probably best known for its ability to create HTML renderings of RDF data, but it can also produce any other textual serialization including JSON or XML. An SWP web service looks almost like a SPARQLMotion or SPIN template service in that it is a class that declares the arguments as spl:Arguments. However, SWP services also need to declare their result type, e.g. ui:JSON as value of the ui:responseType property. Here is the upper part of an example SWP web service that produces some JSON consumed by the SWA tree component:

(The ui:responseType of that service is "inherited" from the superclass ui:JSONServices.) The actual work that the service has to perform when called is declared in the lower section of that service definition:

This particular service is using some helper template to compute the shortest path from a given node to a given root, and then serializes the whole result set to a JSON array using the built-in helper element swon:RSArray. SWP is much more powerful than SPIN templates because it includes sophisticated control elements such as if-then-else and forEach. Furthermore, SWP services can also include most of the SPARQLMotion modules in a compact textual notation.

To learn more about SWP services, browse through some of the examples bundled with TopBraid. As of TopBraid 4.3 (beta 2), the TopBraid Live Web Services overview page shown below lists all available SPARQLMotion, SPIN or SWP services - the page itself is generated on the fly using SWP:

Saved Searches

TopBraid EVN includes a powerful search form that can be used to filter instances of a given class by some properties, for example to find "All Islands that have a size smaller than 1000 square kilometres":

There is a little Save button under the search form, allowing users to save the current search form so that it can be restored later. As a side effect, any of those saved searches can also be called as a web service with the URL provided by the dialog as shown below:

You can copy and paste the URL of the service from the dialog above, and it may produce any of the standard result formats.

SPIN Vocabulary for Column Metadata

When we started implementing the EPIM ReportingHub project, we had to come up with a mechanism that allows us and others to represent tabular reporting data derived from an RDF triple store. For example, there needed to be a table that lists drilling fluid information at certain depths over time. We decided to represent those tables as parameterized SPARQL SELECT queries, and in particular as SPIN Templates. This made it possible to assemble reports consisting of multiple tables quickly. However, we noticed that a SELECT query does not provide enough information to render the tables nicely, for example to specify the text that shall appear in the header of a column or the relative widths of each column. We had added some custom properties to capture such column metadata with each SPIN template. Using this metadata, our report generator was enabled to print prettier tables.

TopBraid 4.3 introduces this as a generalized feature. I have added a couple of new classes and properties to the SPIN Modeling Vocabulary itself, because I believe this will be of general interest. The extension is now online as the SPIN Column Metadata specification.

In a nutshell, each SPIN SELECT Template can now point to one or more instances of the new class spin:Column, and each spin:Column may define a human-readable label, a width, and a column type. Here is an example of how such column can be entered with TopBraid 4.3:

Drilling down into the values of spin:column, we can see that each spin:Column is a blank node with an index and several other optional properties. In TopBraid, use "Create blank node..." to add new columns:

An example rendering of such a table using the swa:ResultSetGrid component of the SWP Application Component Library (SWA) is shown below:

The code to produce such a table with SWP is very compact:

When you instantiate a SPIN template with ui:call, the produced result set will carry any column metadata around, and the functions of the SPR namespace can be used to discover the metadata dynamically. For example, use spr:colName(?rs, 0) to get the name of the first column, or spr:colWidth(?rs, 0) to get its width.

Since all this metadata is represented in RDF, it goes without saying that anyone can query it using SPARQL and repurpose the metadata for different tools such as 3rd party reporting frameworks. The extensible nature of RDF also means that anyone is free to add extra properties such as the unit of measurement of a column or the property that it has been derived from.

Defining SPARQL Functions with SWP

One of the best kept secrets about SPIN is its ability to define new SPARQL functions that encapsulate complex sub-queries into reusable building blocks. Normal SPIN functions are limited to whatever can be expressed in SPARQL, by means of a nested spin:body query. TopBraid has extended this idea by allowing functions with other implementations yet the same general RDF vocabulary. TopBraid users have been able to define new functions using SPARQLMotion and even JavaScript for several years.

TopBraid 4.3 adds the ability to define new SPARQL functions using SPARQL Web Pages (SWP). SWP is better known for its role in generating HTML, yet it also acts as a general expression language for procedural programming. SWP includes control elements such as ui:if/ui:else and ui:forEach and even the ability to invoke SPARQLMotion modules, and all that with a compact textual notation.

Here is an example SWP function that gets the current exchange rate between two given currencies using the currency2currency.org web service:

SWP functions look almost exactly like SPIN functions, only that they have the type ui:Function and use a ui:prototype instead of a spin:body. The prototype will be evaluated with the function arguments as pre-bound variables (shown in bold-face above). The new element ui:return can be used to return the resulting RDF node. An example call is shown below:

Instead of using ui:return, SWP functions can also simply produce any text, and the resulting text elements will be concatenated into the function result. This is illustrated in the following example:

An example call sequence is shown below:

SWP functions are a more powerful alternative to normal SPIN functions, especially if the business logic is too complex to express with SPARQL alone, if SPARQLMotion callbacks are needed, or if the function's result consists of text that is easier constructed with SWP's rich text generation capabilities.

An Extended Turtle Format

Several RDF/OWL ontologies use blank node structures to represent complex objects. For example, OWL uses blank nodes to represent class expression trees consisting of intersections, unions and restrictions. Other ontologies use "reified" objects that combine a value with a unit of measurement. SPIN uses blank nodes to represent SPARQL queries that are attached to classes as rules and constraints. SWP uses blank nodes to represent HTML snippets that are attached to classes as instance views.

In common to all of those blank node structures is that they look ugly and unpredictable in RDF files. Here is an example class with an SWP blank node in Turtle notation:

schema:Person
    a       owl:Class ;
    rdfs:label "Person"^^xsd:string ;
    rdfs:subClassOf owl:Thing ;
    ui:instanceView
        [ a       html:Div ;
          default:subject spin:_this ;
          ui:child
              [ a       swa:Object ;
                arg:predicate schema:familyName ;
                ui:childIndex 1
              ] ;
          ui:child
              [ a       swa:Object ;
                arg:predicate schema:givenName ;
                ui:childIndex 0
              ]
        ] ;
.

Even in this simple example you can see that nobody would want to edit such files per hand, and the use of automated diffing tools to compare versions will be hard because the serialization may be different each time - Turtle doesn't know anything about ui:childIndex for example, and as a result the surrounding blank nodes may be moved to different places each time. Look at any more complex file and you will see how ugly those structures can become...

Many years ago I used this blog to brainstorm about a plug-in mechanism for Turtle. I never got around to implement this idea until this year when I had enough user feedback to confirm that people do want to be able to edit SWP files by hand, and especially want a format that allows them to use off-the-shelf versioning systems. So for for TopBraid 4.3 I have added a new format called Extended Turtle (*.ttlx) that looks like the following:

schema:Person
    a owl:Class ;
    rdfs:label "Person"^^xsd:string ;
    rdfs:subClassOf owl:Thing ;
    ui:instanceView """
<div default:subject=\"{= ?this }\">
    <swa:Object arg:predicate=\"{= schema:givenName }\"/>
    <swa:Object arg:predicate=\"{= schema:familyName }\"/>
</div> """^^ui:Literal ;
.
Extended Turtle uses a pre-processor that turns certain blank node structures into literals and a post-processor that parses the literals back into blank nodes as part of the loading. I have currently only implemented this idea for SWP expressions, but it applies to any blank node structure that also has a parsable text notation. For OWL this would be the Manchester Syntax, for SPIN it would be SPARQL text syntax, and for SWP it is the XML tag syntax. The post-processor simply looks at all literals that have a special datatype (here: ui:Literal) and runs the text parser over those. If this idea was ever considered for standardization, a minimum implementation could require that the datatype (here: ui:Literal) resolved to a web service that takes the literal and the prefixes as input and returns a valid Turtle blank node snippet. Obviously, real-world engines would want to have this parsing done client-side, as plugins to the Turtle parser (and this is what TopBraid does).

One advantage of this notation is that the files are still valid Turtle, so that parsers that do not have those plugins can at least display something. In any case, the main goal of this work was to address the SWP serialization issue and I believe the solution above succeeds with that.

Tutorial: Customizing SWA Forms

SPARQL Web Pages (SWP) is a declarative language that represents user interface elements with an RDF vocabulary. It is tightly integrated into TopBraid Suite: TopBraid Composer is used for editing and the TopBraid Live server executes them for web applications. Its textual notation is an extension of HTML, with additional tags to represent certain reusable building blocks.

The SWP Application Component Library (SWA) includes reusable building blocks that make it easy to configure forms. The resulting components are either used in solutions such as TopBraid EVN, or can be embedded into any other web page. Now that SWA has stabilized, I thought it was a good time to explain in-depth how such forms can be used by application builders. For that purpose, I have added a new

Tutorial on SWA Forms

that explains how to get from a generic form as the one below

to a customized layout such as this:

via a small SWP snippet that is attached to a class:

I would like to point out that the source code above is indeed just a visual representation of an RDF-based model, and that ontology can also be queried (with SPARQL) for other purposes. In a sense this SWA vocabulary can fulfill a similar role as earlier attempts such as Fresnel, only that SWP is backed by an executable and highly flexible model.

Embedding SWP Components into Web Pages with jQuery

SPARQL Web Pages is a powerful framework for defining HTML renderings of RDF data. Recent releases of TopBraid have added significant improvements to this framework, including a growing library of out-of-the-box components for recurring tasks: the SWA Gadgets. SWA includes components to display or edit details of a given resource, render hierarchies in trees, display a list of instances, provide a search form with various result viewers, faceted search etc. At TopQuadrant we are using these components in our own solutions, including EVN and customer projects. We are committed to keeping these components stable and alive and will continue to improve them.

So TopBraid customers do not need to reinvent the wheel.

In order to reuse these components, you do not need to develop your whole application with SWP. Instead you can pick your favorite framework including those based on PHP or modern JavaScript libraries, for the surrounding page. Simply use the SWP components where they make sense. For example, you may need a SKOS Concept tree to drive the navigation of your application - embed an swa:TreeGadget. Or you may want to expose semantic search over your content - embed an swa:FormSearchGadget. Here is a screenshot of "normal" HTML page that includes SWA gadgets loaded from a TopBraid Live server:

You can try this out yourself if you start TopBraid Composer ME 4.3 and go to the URL in the browser above. Let's look at the source code of how this is done.

1. You need to add a few imports to JavaScript and CSS files for jQuery and SWA. Place them in the head of your HTML document.
2. Create a small script that tells SWA about the location of the SWP servlet and the default RDF graph.
3. Insert a placeholder div element in the body of your document.
4. Write a line of JavaScript that uses jQuery.load (or a similar Ajax call) to replace the div with an HTML snippet from the server.

This is only using mainstream technology that should be familar to most web developers. The interesting bit is how these gadgets interact with each other and how they communicate with the surrounding web page. We use the OpenAjax Event Hub for that (which is also used by the OpenSocial standard). Gadgets publish certain events and any other JavaScript elements can subscribe to those events. For example the following code displays an alert box whenever the selected resource in an swa:Tree changes:

    // Subscribe to class selection event to print selected URI
    gadgets.Hub.subscribe("org.example.resourceSelected",
        function(event, data) {
            alert("Selected resource: " + data);
        }
    );

Similarly, your own JavaScript code can publish events to update SWA gadgets, assuming they use the same event name and data payload.

Please take a look at the source code of the example above - in TopBraid Composer 4.3 it is found in TopBraid/SWA/doc/swadoc.www/embeddedSWADemo.html - and do not hesitate to ask questions and provide feature requests on our mailing list.