Creating Web Services with the TopBraid Platform

TopBraid Live (TBL) is TopQuadrant's server platform for semantic data processing and editing. In addition to being the foundation of various out-of-the-box solution packages such as EVN, TopBraid Live can also be used "stand-alone". The features of the platform can be accessed in various ways, for example via a built-in SPARQL end point. In addition to those bundled web services, TopBraid Live can also host any number of custom web services created by users/administrators. This blog entry enumerates a few options on how to create such web services.

Common to all TopBraid custom web services is that they can be called either via GET or POST requests, and that they are created declaratively: the web services are stored in RDF and typically created with TopBraid Composer. This ensures that web services are seamlessly integrated with the data models/ontologies that they expose. Changes to an ontology will automatically update the web services etc. Furthermore, no "programming" in the traditional sense is needed to create those services.

SPARQLMotion

SPARQLMotion is an RDF-based scripting language with a graphical notations. SPARQLMotion scripts are data processing pipelines that can take some input (e.g. web service arguments), import and aggregate data from external sources, perform some processing (SPARQL etc) and export results in various formats. I had written about creating web services as early as 2008 but my colleagues have since then created much better tutorials on this topic. Note that SWP (see below) provides an alternative syntax for exposing SPARQLMotion functionality as web services.

SPIN Templates

In a nutshell, a SPIN Template is a SPARQL query that can take parameters which are mapped to pre-bound variables. A template is a bit like a stored procedure from SQL databases in that it encapsulates a complex query and gives it a name. Here is an example SPIN Template that provides information about the properties of a given class. The class itself comes in as an argument (arg:class) which is mapped to the pre-bound variable ?class in the body of the template.

Any SPIN Template saved in a .spin.* file can be called as a web service, via the tbl/template servlet. The following example calls the template above against the personal test server bundled with TopBraid Composer ME:

http://localhost:8083/tbl/template?
    _template=http://topbraid.org/swa%23GetRelevantPropertiesOfClass&
    _base=http://topbraid.org/examples/kennedys&
    class=http://topbraid.org/examples/kennedys%23Person&
   _format=application/sparql-results%2Bjson

The parameter _base points at the default query graph (if required) and _format can be any of the SPARQL result set formats including CSV, TSV, XML and JSON.

TopBraid 4.3 introduces another simpler result format, called JSON-simple, which is easy to handle by clients. Example output from the query above in JSON-simple is shown here:

[ { "propertyLabel" : "MI" ,
    "propertyURI" : "http://topbraid.org/examples/kennedys#middleInitial" ,
    "rangeURI" : "http://www.w3.org/2001/XMLSchema#string"
  },
  { "propertyLabel" : "alma mater" ,
    "propertyURI" : "http://topbraid.org/examples/kennedys#almaMater" ,
    "rangeURI" : "http://topbraid.org/examples/kennedys#College"
  },
   ...

Also new in TopBraid 4.3 is syntactic sugar that makes the query URLs easier to write:

http://localhost:8083/tbl/template/swa/GetRelevantPropertiesOfClass/kennedys?
    class=http://topbraid.org/examples/kennedys%23Person

More information on this topic can be found in the TBC help, under "TopBraid Live Integration".

SWP Services

SPARQL Web Pages (SWP) is probably best known for its ability to create HTML renderings of RDF data, but it can also produce any other textual serialization including JSON or XML. An SWP web service looks almost like a SPARQLMotion or SPIN template service in that it is a class that declares the arguments as spl:Arguments. However, SWP services also need to declare their result type, e.g. ui:JSON as value of the ui:responseType property. Here is the upper part of an example SWP web service that produces some JSON consumed by the SWA tree component:

(The ui:responseType of that service is "inherited" from the superclass ui:JSONServices.) The actual work that the service has to perform when called is declared in the lower section of that service definition:

This particular service is using some helper template to compute the shortest path from a given node to a given root, and then serializes the whole result set to a JSON array using the built-in helper element swon:RSArray. SWP is much more powerful than SPIN templates because it includes sophisticated control elements such as if-then-else and forEach. Furthermore, SWP services can also include most of the SPARQLMotion modules in a compact textual notation.

To learn more about SWP services, browse through some of the examples bundled with TopBraid. As of TopBraid 4.3 (beta 2), the TopBraid Live Web Services overview page shown below lists all available SPARQLMotion, SPIN or SWP services - the page itself is generated on the fly using SWP:

Saved Searches

TopBraid EVN includes a powerful search form that can be used to filter instances of a given class by some properties, for example to find "All Islands that have a size smaller than 1000 square kilometres":

There is a little Save button under the search form, allowing users to save the current search form so that it can be restored later. As a side effect, any of those saved searches can also be called as a web service with the URL provided by the dialog as shown below:

You can copy and paste the URL of the service from the dialog above, and it may produce any of the standard result formats.

SPIN Vocabulary for Column Metadata

When we started implementing the EPIM ReportingHub project, we had to come up with a mechanism that allows us and others to represent tabular reporting data derived from an RDF triple store. For example, there needed to be a table that lists drilling fluid information at certain depths over time. We decided to represent those tables as parameterized SPARQL SELECT queries, and in particular as SPIN Templates. This made it possible to assemble reports consisting of multiple tables quickly. However, we noticed that a SELECT query does not provide enough information to render the tables nicely, for example to specify the text that shall appear in the header of a column or the relative widths of each column. We had added some custom properties to capture such column metadata with each SPIN template. Using this metadata, our report generator was enabled to print prettier tables.

TopBraid 4.3 introduces this as a generalized feature. I have added a couple of new classes and properties to the SPIN Modeling Vocabulary itself, because I believe this will be of general interest. The extension is now online as the SPIN Column Metadata specification.

In a nutshell, each SPIN SELECT Template can now point to one or more instances of the new class spin:Column, and each spin:Column may define a human-readable label, a width, and a column type. Here is an example of how such column can be entered with TopBraid 4.3:

Drilling down into the values of spin:column, we can see that each spin:Column is a blank node with an index and several other optional properties. In TopBraid, use "Create blank node..." to add new columns:

An example rendering of such a table using the swa:ResultSetGrid component of the SWP Application Component Library (SWA) is shown below:

The code to produce such a table with SWP is very compact:

When you instantiate a SPIN template with ui:call, the produced result set will carry any column metadata around, and the functions of the SPR namespace can be used to discover the metadata dynamically. For example, use spr:colName(?rs, 0) to get the name of the first column, or spr:colWidth(?rs, 0) to get its width.

Since all this metadata is represented in RDF, it goes without saying that anyone can query it using SPARQL and repurpose the metadata for different tools such as 3rd party reporting frameworks. The extensible nature of RDF also means that anyone is free to add extra properties such as the unit of measurement of a column or the property that it has been derived from.

Defining SPARQL Functions with SWP

One of the best kept secrets about SPIN is its ability to define new SPARQL functions that encapsulate complex sub-queries into reusable building blocks. Normal SPIN functions are limited to whatever can be expressed in SPARQL, by means of a nested spin:body query. TopBraid has extended this idea by allowing functions with other implementations yet the same general RDF vocabulary. TopBraid users have been able to define new functions using SPARQLMotion and even JavaScript for several years.

TopBraid 4.3 adds the ability to define new SPARQL functions using SPARQL Web Pages (SWP). SWP is better known for its role in generating HTML, yet it also acts as a general expression language for procedural programming. SWP includes control elements such as ui:if/ui:else and ui:forEach and even the ability to invoke SPARQLMotion modules, and all that with a compact textual notation.

Here is an example SWP function that gets the current exchange rate between two given currencies using the currency2currency.org web service:

SWP functions look almost exactly like SPIN functions, only that they have the type ui:Function and use a ui:prototype instead of a spin:body. The prototype will be evaluated with the function arguments as pre-bound variables (shown in bold-face above). The new element ui:return can be used to return the resulting RDF node. An example call is shown below:

Instead of using ui:return, SWP functions can also simply produce any text, and the resulting text elements will be concatenated into the function result. This is illustrated in the following example:

An example call sequence is shown below:

SWP functions are a more powerful alternative to normal SPIN functions, especially if the business logic is too complex to express with SPARQL alone, if SPARQLMotion callbacks are needed, or if the function's result consists of text that is easier constructed with SWP's rich text generation capabilities.

An Extended Turtle Format

Several RDF/OWL ontologies use blank node structures to represent complex objects. For example, OWL uses blank nodes to represent class expression trees consisting of intersections, unions and restrictions. Other ontologies use "reified" objects that combine a value with a unit of measurement. SPIN uses blank nodes to represent SPARQL queries that are attached to classes as rules and constraints. SWP uses blank nodes to represent HTML snippets that are attached to classes as instance views.

In common to all of those blank node structures is that they look ugly and unpredictable in RDF files. Here is an example class with an SWP blank node in Turtle notation:

schema:Person
    a       owl:Class ;
    rdfs:label "Person"^^xsd:string ;
    rdfs:subClassOf owl:Thing ;
    ui:instanceView
        [ a       html:Div ;
          default:subject spin:_this ;
          ui:child
              [ a       swa:Object ;
                arg:predicate schema:familyName ;
                ui:childIndex 1
              ] ;
          ui:child
              [ a       swa:Object ;
                arg:predicate schema:givenName ;
                ui:childIndex 0
              ]
        ] ;
.

Even in this simple example you can see that nobody would want to edit such files per hand, and the use of automated diffing tools to compare versions will be hard because the serialization may be different each time - Turtle doesn't know anything about ui:childIndex for example, and as a result the surrounding blank nodes may be moved to different places each time. Look at any more complex file and you will see how ugly those structures can become...

Many years ago I used this blog to brainstorm about a plug-in mechanism for Turtle. I never got around to implement this idea until this year when I had enough user feedback to confirm that people do want to be able to edit SWP files by hand, and especially want a format that allows them to use off-the-shelf versioning systems. So for for TopBraid 4.3 I have added a new format called Extended Turtle (*.ttlx) that looks like the following:

schema:Person
    a owl:Class ;
    rdfs:label "Person"^^xsd:string ;
    rdfs:subClassOf owl:Thing ;
    ui:instanceView """
<div default:subject=\"{= ?this }\">
    <swa:Object arg:predicate=\"{= schema:givenName }\"/>
    <swa:Object arg:predicate=\"{= schema:familyName }\"/>
</div> """^^ui:Literal ;
.
Extended Turtle uses a pre-processor that turns certain blank node structures into literals and a post-processor that parses the literals back into blank nodes as part of the loading. I have currently only implemented this idea for SWP expressions, but it applies to any blank node structure that also has a parsable text notation. For OWL this would be the Manchester Syntax, for SPIN it would be SPARQL text syntax, and for SWP it is the XML tag syntax. The post-processor simply looks at all literals that have a special datatype (here: ui:Literal) and runs the text parser over those. If this idea was ever considered for standardization, a minimum implementation could require that the datatype (here: ui:Literal) resolved to a web service that takes the literal and the prefixes as input and returns a valid Turtle blank node snippet. Obviously, real-world engines would want to have this parsing done client-side, as plugins to the Turtle parser (and this is what TopBraid does).

One advantage of this notation is that the files are still valid Turtle, so that parsers that do not have those plugins can at least display something. In any case, the main goal of this work was to address the SWP serialization issue and I believe the solution above succeeds with that.

Tutorial: Customizing SWA Forms

SPARQL Web Pages (SWP) is a declarative language that represents user interface elements with an RDF vocabulary. It is tightly integrated into TopBraid Suite: TopBraid Composer is used for editing and the TopBraid Live server executes them for web applications. Its textual notation is an extension of HTML, with additional tags to represent certain reusable building blocks.

The SWP Application Component Library (SWA) includes reusable building blocks that make it easy to configure forms. The resulting components are either used in solutions such as TopBraid EVN, or can be embedded into any other web page. Now that SWA has stabilized, I thought it was a good time to explain in-depth how such forms can be used by application builders. For that purpose, I have added a new

Tutorial on SWA Forms

that explains how to get from a generic form as the one below

to a customized layout such as this:

via a small SWP snippet that is attached to a class:

I would like to point out that the source code above is indeed just a visual representation of an RDF-based model, and that ontology can also be queried (with SPARQL) for other purposes. In a sense this SWA vocabulary can fulfill a similar role as earlier attempts such as Fresnel, only that SWP is backed by an executable and highly flexible model.

Embedding SWP Components into Web Pages with jQuery

SPARQL Web Pages is a powerful framework for defining HTML renderings of RDF data. Recent releases of TopBraid have added significant improvements to this framework, including a growing library of out-of-the-box components for recurring tasks: the SWA Gadgets. SWA includes components to display or edit details of a given resource, render hierarchies in trees, display a list of instances, provide a search form with various result viewers, faceted search etc. At TopQuadrant we are using these components in our own solutions, including EVN and customer projects. We are committed to keeping these components stable and alive and will continue to improve them.

So TopBraid customers do not need to reinvent the wheel.

In order to reuse these components, you do not need to develop your whole application with SWP. Instead you can pick your favorite framework including those based on PHP or modern JavaScript libraries, for the surrounding page. Simply use the SWP components where they make sense. For example, you may need a SKOS Concept tree to drive the navigation of your application - embed an swa:TreeGadget. Or you may want to expose semantic search over your content - embed an swa:FormSearchGadget. Here is a screenshot of "normal" HTML page that includes SWA gadgets loaded from a TopBraid Live server:

You can try this out yourself if you start TopBraid Composer ME 4.3 and go to the URL in the browser above. Let's look at the source code of how this is done.

1. You need to add a few imports to JavaScript and CSS files for jQuery and SWA. Place them in the head of your HTML document.
2. Create a small script that tells SWA about the location of the SWP servlet and the default RDF graph.
3. Insert a placeholder div element in the body of your document.
4. Write a line of JavaScript that uses jQuery.load (or a similar Ajax call) to replace the div with an HTML snippet from the server.

This is only using mainstream technology that should be familar to most web developers. The interesting bit is how these gadgets interact with each other and how they communicate with the surrounding web page. We use the OpenAjax Event Hub for that (which is also used by the OpenSocial standard). Gadgets publish certain events and any other JavaScript elements can subscribe to those events. For example the following code displays an alert box whenever the selected resource in an swa:Tree changes:

    // Subscribe to class selection event to print selected URI
    gadgets.Hub.subscribe("org.example.resourceSelected",
        function(event, data) {
            alert("Selected resource: " + data);
        }
    );

Similarly, your own JavaScript code can publish events to update SWA gadgets, assuming they use the same event name and data payload.

Please take a look at the source code of the example above - in TopBraid Composer 4.3 it is found in TopBraid/SWA/doc/swadoc.www/embeddedSWADemo.html - and do not hesitate to ask questions and provide feature requests on our mailing list.

Faceted Search in TopBraid 4.3

Faceted search is a user interface metaphor that is best suitable for explorative scenarios: A user starts by roughly selecting a type of objects and the user interface presents characteristics (or "facets") of the available instances, allowing the user to drill down into specific property values. A faceted search engine displays the available values and the number of matching instances at each step, making it easy to explore the space of available instances without knowing the search keywords in advance.

An experimental "unpolished" faceted search component had been part of TopBraid for a while, but it had scalability issues and did not integrate well with the rest of the platform. For TopBraid 4.3, we have made a complete redesign to address those issues. A screenshot of the new faceted search capabilities is shown below (click on the image for a larger view):

 

To get to that screen, open kennedys.ttl in TopBraid Composer ME 4.3, then select the Person class and move to the Browser tab. In the drop down box above the browser, select "facetsapp:Application" and click "Open current page in external Browser". The same can of course be done for any other ontology or class.

The faceted search feature has been implemented in HTML 5/jQuery as SWA gadgets. The component consisting of a search box and the "Limit your search" area is one gadget, and the "summary" view in the center of the screenshot above is another gadget. The search gadget can also be combined with other display gadgets such as the tabular grid views or even a Google Maps view. All views can be customized with a few SWP declarations, e.g. to have certain properties pre-selected as facet fields, and to modify how results are displayed (ask our mailing list if the documentation doesn't help).

While it is possible to embed faceted search into any SWP application or HTML 5 web page, TopBraid EVN now also uses faceted search to help selecting objects. The EVN edit forms have a drop down menu that allows users to open up a faceted search dialog. Another way of opening this dialog is from the context menu under the Search gadget in the new EVN Ontology Editor.

In addition to the facets and count display, more specific results can be obtained through a free-text search box in the upper left corner:

The faceted search components can seamlessly operate in two different modes. By default, it will use a native SPARQL-based engine that will work directly on the RDF store. This is fine for small to medium-sized data sets, but may become slow when too many counts need to be computed. The alternative mode is using Apache Solr, a Lucene-based search index that is optimized for faceted search scenarios. To make use of this, you only need to configure your TopBraid EVN project to synchronize with a Solr server. TopBraid will automatically update the Solr index after each edit (to the master graph) and then automatically use Solr to do the heavy lifting of computing the facet counts. The structure of the ontology is still used to discover which properties are present, and for the detailed displays if someone clicks on an instance. This hybrid approach combines the power of ontologies with an the optimized native data structure provided by Solr.

The flexibility of the RDF data model is well suited to serve as foundation for faceted search scenarios. The TopBraid platform provides a highly optimized yet flexible platform for delivering faceted search to your end users. No programming is needed to get started - this is model-driven throughout.

The TopBraid EVN Ontology Editor

The evolution of the TopBraid Enterprise Vocabulary Net (EVN) has reached a new milestone with release 4.3. While EVN is long established as a leading web editor for SKOS-based Thesauri and controlled vocabularies, we have generalized the product so that it can also be used to edit arbitrary other RDF Schema or OWL models. The new Ontology Editor inherits most of the features of the EVN SKOS vocabulary editor, including the change tracking infrastructure and the familiar UI components. Here is a screenshot of the Ontology Editor in EVN 4.3 (click on an image for a larger view):

We have placed a mixed class-property-tree on the left, with buttons to create new classes and datatype or object properties. Under the tree, a tabular view allows users to select instances or create new ones. Auto-complete fields under the tree and instances areas support quick navigation. A powerful search form is placed on the right, with features that allow the construction of even complex queries with nested objects, regular expressions, range queries and other patterns. Search results can be used to navigate around but can also be exported to spreadsheets - columns can be selected with the provided checkboxes.

Details of the selected resource are displayed in the center area, which can be toggled between view and edit mode.  As shown below,  the edit forms have a new feature that makes it possible to edit "depending" objects such as the Address of a Person using nested sub-forms (the resulting objects are represented as blank nodes).

EVN is a complete solution for editing classes, properties and instances. For classes, we have integrated a constraint editor based on SPIN templates. As shown below, average users do not need to understand how constraints are implemented - they only need to fill in the blanks on a form. Several constraint types are delivered out of the box, but almost arbitrary constraints can be added by users who are familiar with SPARQL.

EVN has a sophisticated infrastructure for change tracking. Every ontology has its own life cycle and user permissions. The manager of an ontology can assign roles to other users, allowing them to either edit the "master" copy directly, or to create so-called "working copies" based on master. A working copy is a branch that may contain changes that need to be reviewed before they can be merged into master. EVN provides various elements for reviewing changes. As shown below, the forms can be switched into a history mode in which added and deleted statements are highlighted:

Every change in EVN is tracked in a dedicated model - we love to use RDF all the way down, so the change history itself is represented in RDF and can therefore be queried uniformly via SPARQL. The Change History view allows users to browse and search the complete change history as shown below.

The EVN user interface is able to operate meaningfully on almost every ontology because it uses the ontology's structure to discover which properties make sense in which context, and how they should be displayed. However, in some cases a generic display of properties is not the best way to render information. Therefore, one of the strengths of EVN is its customizability. EVN is built using SPARQL Web Pages, and in particular the SWA component library.  I should write many more blog entries on the various customization techniques, but for now let me enumerate a few.

• EVN used standard HTML 5 and jQuery, and well-defined CSS styles can be overloaded to change look and feel.
• The layout of forms can be customized (by attaching a different ui:instanceView to a class)
• Custom widgets can be installed to display, edit or search certain values (via subclasses of swa:Widget)
• It is possible to add new actions for the drop down menus and those actions may execute arbitrary JavaScript code or call servlets that read or update data.
• The whole layout of the application can be changed, for example to add new views. The platform underlying EVN has been generalized and its gadgets can be plugged together in different ways.

If you want to experiment with EVN, just download TopBraid Composer Maestro Edition and get started - TopBraid Composer launches a built-in web server that makes it possible for you to interact with ontologies from a web browser. We are looking forward to your feedback on the first 4.3 beta release.