Semantic Web Posts

Linked Data Entity Extraction with Zemanta and OpenCalais

2010-07-28T09:50Z

I had another look at the Named Entity Extraction APIs by Zemanta and OpenCalais for some product launch demos. My first test from last year concentrated more on the Zemanta API. This time I had a closer look at both services, trying to identify the "better one" for "BlogDB", a semi-automatic blog semantifier.

My main need is a service that receives a cleaned-up plain text version of a blog post and returns normalized tags and reusable entity identifiers. So, the findings in this post are rather technical and just related to the BlogDB requirements. I ignored features which could well be essential for others, such as Zemanta's "related articles and photos" feature, or OpenCalais' entity relations ("X hired Y" etc.).

Terms and restrictions of the free API

The API terms are pretty similar (the wording is actually almost identical). You need an API key and both services can be used commercially as long as you give attribution and don't proxy/resell the service.
OpenCalais gives you more free API calls out of the box than Zemanta (50.000 vs. 1.000 per day). You can get a free upgrade to 10.000 Zemanta calls via a simple email, though (or excessive API use; Andraž auto-upgraded my API limit when he noticed my crazy HDStreams test back then ;-).
OpenCalais lets you process larger content chunks (up to 100K, vs. 8K at Zemanta).

Calling the API

Both interfaces are simple and well-documented. Calls to the OpenCalais API are a tiny bit more complicated as you have to encode certain parameters in an XML string. Zemanta uses simple query string arguments. I've added the respective PHP snippets below, the complexity difference is negligible.

function getCalaisResult($id, $text) {
  $parms = '
    <c:params xmlns:c="http://s.opencalais.com/1/pred/"
              xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
      <c:processingDirectives
        c:contentType="TEXT/RAW"
        c:outputFormat="XML/RDF"
        c:calculateRelevanceScore="true"
        c:enableMetadataType="SocialTags"
        c:docRDFaccessible="false"
        c:omitOutputtingOriginalText="true"
        ></c:processingDirectives>
      <c:userDirectives
        c:allowDistribution="false"
        c:allowSearch="false"
        c:externalID="' . $id . '"
        c:submitter="http://semsol.com/"
        ></c:userDirectives>
      <c:externalMetadata></c:externalMetadata>
    </c:params>
  ';
  $args = array(
    'licenseID' => $this->a['calais_key'],
    'content' => urlencode($text),
    'paramsXML' => urlencode(trim($parms))
  );
  $qs = substr($this->qs($args), 1);
  $url = 'http://api.opencalais.com/enlighten/rest/';
  return $this->getAPIResult($url, $qs);
}

function getZemantaResult($id, $text) {
  $args = array(
    'method' => 'zemanta.suggest',
    'api_key' => $this->a['zemanta_key'],
    'text' => urlencode($text),
    'format' => 'rdfxml',
    'return_rdf_links' => '1',
    'return_articles' => '0',
    'return_categories' => '0',
    'return_images' => '0',
    'emphasis' => '0',
  );
  $qs = substr($this->qs($args), 1);
  $url = 'http://api.zemanta.com/services/rest/0.0/';
  return $this->getAPIResult($url, $qs);
}

The actual API call is then a simple POST:

function getAPIResult($url, $qs) {
  ARC2::inc('Reader');
  $reader = new ARC2_Reader($this->a, $this);
  $reader->setHTTPMethod('POST');
  $reader->setCustomHeaders("Content-Type: application/x-www-form-urlencoded");
  $reader->setMessageBody($qs);
  $reader->activate($url);
  $r = '';
  while ($d = $reader->readStream()) {
    $r .= $d;
  }
  $reader->closeStream();
  return $r;
}

Both APIs are fast.

API result processing

The APIs return rather verbose data, as they have to stuff in a lot of meta-data such as confidence scores, text positions, internal and external identifiers, etc. But they also offer RDF as one possible result format, so I could store the response data as a simple graph and then use SPARQL queries to extract the relevant information (tags and named entities). Below is the query code for Linked Data entity extraction from Zemanta's RDF. As you can see, the graph structure isn't trivial, but still understandable:
```
SELECT DISTINCT ?id ?obj ?cnf ?name
FROM <' . $g . '> WHERE {
  ?rec a z:Recognition ;
       z:object ?obj ;
       z:confidence ?cnf .
  ?obj z:target ?id .
  ?id z:targetType <http://s.zemanta.com/targets#rdf> ;
      z:title ?name .
  FILTER(?cnf >= 0.4)
} ORDER BY ?id
```

Extracting normalized tags

OpenCalais results contain a section with so-called "SocialTags" which are directly usable as plain-text tags.
The tag structures in the Zemanta result are called "Keywords". In my tests they only contained a subset of the detected entities, and so I decided to use the labels associated with detected entities instead. This worked well, but the respective query is more complex.

Extracting entities

In general, OpenCalais results can be directly utilized more easily. They contain stable identifiers and the identifiers come with type information and other attributes such as stock symbols. The API result directly tells you how many Persons, Companies, Products, etc. were detected. And the URIs of these entity types are all from a single (OpenCalais) namespace. If you are not a Linked Data pro, this simplifies things a lot. You only have to support a simple list of entity types to build a working semantic application. If you want to leverage the wider Linked Open Data cloud, however, the OpenCalais response is just a first entry point. It doesn't contain community URIs. You have to use the OpenCalais website to first retrieve disambiguation information, which may then (often involving another request) lead you to the decentralized Linked Data identifiers.
Zemanta responses, in contrast, do not (yet, Andraž told me they are working on it) contain entity types at all. You always need an additional request to retrieve type information (unless you are doing nasty URI inspection, which is what I did with detected URIs from Semantic CrunchBase). The retrieval of type information is done via Open Data servers, so you have to be able to deal with the usual down-times of these non-commercial services.
Zemanta results are very "webby" and full of community URIs. They even include sameAs information. This can be a bit overwhelming if you are not an RDFer, e.g. looking up a DBPedia URI will often give you dozens of entity types, and you need some experience to match them with your internal type hierarchy. But for an open data developer, the hooks provided by Zemanta are a dream come true.
With Zemanta associating shared URIs with all detected entities, I noticed network effects kicking in a couple of times. I used RWW articles for the test, and in one post, for example, OpenCalais could detect the company "Starbucks" and "Howard Schultz" as their "CEO", but their public RDF (when I looked up the "Howard Schultz" URI) didn't persist this linkage. The detection scope was limited to the passed snippet. Zemanta, on the other hand, directly gave me Linked Data URIs for both "Starbucks" and "Howard Schultz", and these identifiers make it possible to re-establish the relation between the two entities at any time. This is a very powerful feature.

Summary

Both APIs are great. The quality of the entity extractors is awesome. For the RWW posts, which deal a lot with Web topics, Zemanta seemed to have a couple of extra detections (such as "ReadWriteWeb" as company). As usual, some owl:sameAs information is wrong, and Zemanta uses incorrect Semantic CrunchBase URIs (".rdf#self" instead of "#self" // Update: to be fixed in the next Zemanta API revision), but I blame us (the RDF community), not the API providers, for not making these things easier to implement.

In the end, I decided to use both APIs in combination, with an optional post-processing step that builds a consolidated, internal ontology from the detected entities (OpenCalais has two Company types which could be merged, for example). Maybe I can make a Prospect demo from the RWW data public, not sure if they would allow this. It's really impressive how much value the entity extraction services can add to blog data, though (see the screenshot below, which shows a pivot operation on products mentioned in posts by Sarah Perez). I'll write a bit more about the possibilities in another post.

Contextual configuration - Semantic Web development for visually minded webmasters

2010-05-21T12:40Z

Let's face it, building semantic web sites and apps is still far from easy. And to some extent, this is due to the configuration overhead. The RDF stack is built around declarative languages (for simplified integration at various levels), and as a consequence, configuration directives often end up in some form of declarative format, too. While fleshing out an RDF-powered website, you have to declare a ton of things. From namespace abbreviations to data sources and API endpoints, from vocabularies to identifier mappings, from queries to object templates, and what have you.

Sadly, many of these configurations are needed to style the user interface, and because of RDF's open world context, designers have to know much more about the data model and possible variations than usually necessary. Or webmasters have to deal with design work. Not ideal either. If we want to bring RDF to mainstream web developers, we have to simplify the creation of user-optimized apps. The value proposition of semantics in the context of information overload is pretty clear, and some form of data integration is becoming mandatory for any modern website. But the entry barrier caused by large and complicated configuration files (Fresnel anyone?) is still too high. How can we get from our powerful, largely generic systems to end-user-optimized apps? Or the other way round: How can we support frontend-oriented web development with our flexible tools and freely mashable data sets? (Let me quickly mention Drupal here, which is doing a great job at near-seamlessly integrating RDF. OK, back to the post.)

Enter RDF widgets. Widgets have obvious backend-related benefits like accessing, combining and re-purposing information from remote sources within a manageable code sandbox. But they can also greatly support frontend developers. They simplify page layouting and incremental site building with instant visual feedback (add a widget, test, add another one, re-arrange, etc.). And, more importantly in the RDF case, they can offer a way to iteratively configure a system with very little technical overhead. Configuration options could not only be scoped to the widget at hand, but also to the context where the widget is currently viewed. Let's say you are building an RDF browser and need resource templates for all kinds of items. With contextual configuration, you could simply browse the site and at any position in the ontology or navigation hierarchy, you would just open a configuration dialog and define a custom template, if needed. Such an approach could enable systems that worked out of the box (raw, but usable) and which could then be continually optimized, possibly even by site users.

A lot of "could" and "would" in the paragraphs above, and the idea may sound quite abstract without actually seeing it. To illustrate the point I'm trying to make I've prepared a short video (embedded below). It uses Semantic CrunchBase and Paggr Prospect (our new faceted browser builder) as an example use case for in-context configuration.

And if you are interested in using one of our solutions for your own projects, please get in touch!

Paggr Prospect (part 1)

Paggr Prospect (part 2)

Trice' Semantic Richtext Editor

2010-05-01T16:35Z

In my previous post I mentioned that I'm building a Linked Data CMS. One of its components is a rich-text editor that allows the creation (and embedding) of structured markup.

An earlier version supported limited Microdata annotations, but now I've switched the mechanism and use an intermediate, but even simpler approach based on HTML5's handy data-* attributes. This lets you build almost arbitrary markup with the editor, including Microformats, Microdata, or RDFa. I don't know yet when the CMS will be publicly available (3 sites are under development right now), but as mentioned, I'd be happy about another pilot project or two. Below is a video demonstrating the editor and its easy customization options.

Could having two RDF-in-HTMLs actually be handy?

2010-04-15T10:30Z

Apart from grumpy rants about the complexity of W3C's RDF specs and semantic richtext editing excitement, I haven't blogged or tweeted a lot recently. That's partly because there finally is increased demand for the stuff I'm doing at semsol (agency-style SemWeb development), but also because I've been working hard on getting my tools in a state where they feel more like typical Web frameworks and apps. Talis' Fanhu.bz is an example where (I think) we found a good balance between powerful RDF capabilities (data re-purposing, remote models, data augmentation, a crazy army of inference bots) and a non-technical UI (simplistic visual browser, Twitter-based annotation interfaces).

Another example is something I've been working on during the last months: I somehow managed to combine essential parts of Paggr (a drag&drop portal system based on RDF- and SPARQL-based widgets) with an RDF CMS (I'm currently looking for pilot projects). And although I decided to switch entirely to Microdata for semantic markup after exploring it during the FanHubz project, I wonder if there might be room for having two separate semantic layers in this sort of widget-based websites. Here is why:

As mentioned, I've taken a widget-like approach for the CMS. Each page section is a resource on its own that can be defined and extended by the web developer, it can be styled by themers, and it can be re-arranged and configured by the webmaster. In the RDF CMS context, widgets can easily integrate remote data, and when the integrated information is exposed as machine-readable data in the front-end, we can get beyond the "just-visual" integration of current widget pages and bring truly connectable and reusable information to the user interface.

Ideally, both the widgets' structural data and the content can be re-purposed by other apps. Just like in the early days of the Web, we could re-introduce a copy & paste culture of things for people to include in their own sites. With the difference that RDF simplifies copy-by-reference and source attribution. And both developers and end-users could be part of the game this time.

Anyway, one technical issue I encountered is when you have a page that contains multiple page items, but describes a single resource. With a single markup layer (say Microdata), you get a single tree where the context of the hierarchy is constantly switching between structural elements and content items (page structure -> main content -> page layout -> widget structure -> widget content). If you want to describe a single resource, you have to repeatedly re-introduce the triple subject ("this is about the page structure", "this is about the main page topic"). The first screenshot below shows the different (grey) widget areas in the editing view of the CMS. In the second screenshot, you can see that the displayed information (the marked calendar date, the flyer image, and the description) in the main area and the sidebar is about a single resource (an event).

Trice CMS editing view

Trice CMS page view with inline widgets describing one resource

If I used two separate semantic layers, e.g. RDFa for the content (the event description) and Microdata for the structural elements (column widths, widget template URIs, widget instance URIs), I could describe the resource and the structure without repeating the event subject in each page item.

To be honest, I'm not sure yet if this is really a problem, but I thought writing it down could kick off some thought processes (which now tend towards "No"). Keeping triples as stand-alone-ish as possible may actually be an advantage (even if subject URIs have to be repeated). No semantic markup solution so far provides full containment for reliable copy & paste, but explicit subjects (or "itemid"s in Microdata-speak) could bring us a little closer.

Conclusions? Err.., none yet. But hey, did you see the cool CMS screenshots?

Microdata, semantic markup for both RDFers and non-RDFers

2010-01-26T12:00Z

There's been a whole lot of discussion around Microdata, a new approach for embedding machine-readable information into forthcoming HTML5. What I find most attractive about Microdata is the fact that it was designed by HTMLers, not RDFers. It's refreshingly pragmatic, free of other RDF spec legacy, but still capable of expressing most of RDF.

Unfortunately, RDFa lobbyists on the HTML WG mailing list forced the spec out of HTML5 core for the time being. This manoeuver was understandable (a lot of energy went into RDFa, after all), but in my opinion very short-sighted. How many uphill battles did we have, trying to get RDF to the broader developer community? And how many were successful? Atom, microformats, OpenID, Portable Contacts, XRDS, Activity Streams (well, not really), these are examples where RDFers tried, but failed to promote some of their infrastructure into the respective solutions. Now: HTML5, where the initial RDF lobbying actually had an effect and lead to a native mechanism for RDF-in-HTML. Yes, native, not in some separate spec. This would have become part of every HTML5 book, any HTML developer on this planet would have learned about it. Finally a battle won. And what a great one. HTML.

But no, Microdata wasn't developed by an RDF group, so they voted it out again. Now, the really sad thing is, there could have been a solution that would have served everybody sufficiently well, both HTMLers and RDFers. The RDFa group recently realized that RDFa needs to be revised anyway, there is going to be an RDFa 1.1 which will require new parsers. If they'd swallowed their pride, they would most probably have been able to define RDFa 1.1 as a proper superset of Microdata.

Here is a short overview of RDF features supported by Microdata:

Explicit resource containers, via @itemscope (in RDFa, the boundaries of a resource are often implicitly defined by @rel or @typeof)
Subject declaration, via @itemid (RDFa uses @about)
Main subject typing, via @itemtype (RDFa uses @typeof)
Predicate declaration, via @itemprop (RDFa uses @property, @rel, and @rev)
Literal objects, via node values (RDFa also allows hidden values via @content)
Non-literal objects, via @href, @src, etc. (RDFa also allows hidden values via @resource)
Object language, via @lang
Blank nodes

I won't go into details why hiding semantics in RDFa will be penalized by search engines as soon as spammers discover the possibilities, why reusing RDF/XML's attribute names was probably not a smart move with regard to attracting non-RDFers, why the new @vocab idea is impractical, or why namespace prefixes, as handy as they are in other RDF formats, are not too helpful in an HTML context. Let's simply state that there is a trade-off between extended features (RDFa) and simplicity (Microdata). So, what are the core features that an RDFer would really need beyond Microdata:

the possibility to preserve markup, but probably not necessarily as an explicit rdf:XMLLiteral
datatypes for literal objects (I personally never used them in practice in the last 6 years that I've been developing RDF apps, but I can see some use cases)

Markup preservation is currently turned on by default in RDFa and can be disabled through @datatype in RDFa, so an RDFer-satisfying RDFa 1.1 spec could probably just be Microdata + @datatype + a few extended parsing rules to end up with the intended RDF. My experience with watching RDF spec creation tells me that the RDFa group won't pick this route (there simply is no "Kill a Feature" mentality in the RDF community), but hey, hope dies last.

I've been using Microdata in two of my recent RDF apps and the CMS module of (ahem, still not documented) Trice, and it's been a great experience. ARC is going to get a "microRDF" extractor that supports the RDF-in-Microdata markup below (Note: this output still requires a 2nd extraction process, as the current Microdata draft's RDF mechanism only produces intermediate RDF triples, which then still have to be post-processed. I hope my related suggestion will become official, but I seem to be the only pro-Microdata RDFer on the HTML list right now, so it may just stay as a convention):

Microdata:

<div itemscope itemtype="http://xmlns.com/foaf/0.1/Person">

  <!-- plain props are mapped to the itemtype's context -->
  <img itemprop="img" src="mypic.jpg" alt="a pic of me" />
  My name is <span itemprop="name"><span itemprop="nick">Alec</span> Tronnick</span>
  and I blog at <a itemprop="weblog" href="http://alec-tronni.ck/">alec-tronni.ck</a>.

  <!-- other RDF vocabs can be used via full itemprop URIs -->
  <span itemprop="http://purl.org/vocab/bio/0.1/olb">
    I'm a crash test dummy for semantic HTML.
  </span>
</div>

Extracted RDF:

@base <http://host/path/>
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix bio: <http://purl.org/vocab/bio/0.1/> .
_:bn1 a foaf:Person ;
      foaf:img <mypic.jpg> ;
      foaf:name "Alec Tronnick" ;
      foaf:nick "Alec" ;
      foaf:weblog <http://alec-tronni.ck/> ;
      bio:olb "I'm a crash test dummy for semantic HTML." .

Naming Properties and Relations (comment)

2009-09-15T17:45Z

I was incapable of adding a comment to Jeni's interesting post about RDF predicate Names (markdown-related, my fault), so I'll quickly post it here, as I'm pondering similar things, too.

In her post, Jeni explores the issues around naming RDF terms. The community gathered a couple of experiences and suggestions in the last years, some entry points are:

I personally find "role-noun" easier to support in RDF apps than the older hasPropertyOf (now often considered anti-)pattern. And inverse properties are just painful, as they usually require some form of inference to streamline the user experience.

Not sure if that's helpful information, but for a project around semantic note-taking/logging, I played with different notations users might be comfortable with, for entering factoids using an unstructured input form (à la Twitter). I could identify the following patterns that still seemed to be acceptable (as shared/supported syntax). All of them can be implemented using role-noun predicates (assuming that predicate labels are similar to the predicate names):

SUBJECT'(s)? PREDICATE (:|is) OBJECT
OBJECT is SUBJECT'(s)? PREDICATE
OBJECT is (the)? PREDICATE of SUBJECT
SUBJECT has PREDICATE (:)? OBJECT
(the|a)? PREDICATE(s)? of SUBJECT (is|are) OBJECT ((,|and|&) OBJECT)*

(There are more patterns, for things like tagging and typing, but the examples above are the predicate-related grammar rules).

As soon as you add (has|is|of) to one PREDICATE, you get problems with the other notations, so role-noun seems to be a good fit.

Unfortunately, one (non-trivial) problem remains: People (and Web 2.0 apps) also like 'SUBJECT PREDICATE_VERB OBJECT' (e.g. "likes", "bookmarked", "said", "posted", "is listening to" ...) and I don't have a proper idea how to handle those automatically yet, other than hard-coding support for the typical social media verbs. It could be possible to use wordnet to detect verbs and derive a canonicalized form, and then model those patterns as activities (activity = liking, bookmarking, saying, posting, listening, plus ACTIVITY_PERSON and ACTIVITY_TARGET or somesuch). If anyone has a suggestion, I'd be happy to hear it.

New ARC2 release

2009-08-21T10:20Z

I moved ARC's codebase to a BZR repository 2 months ago but didn't really find the time to synchronize it with the way I created bundles in the past. Today I finally linked the repository and its TGZ creation feature from the main download page. This is the first bundle since March, so there are quite a number of fixes. Some tweaks were not logged, but from now on, the process should be more professional (thanks to the proper versioning system).

Here is the raw list of changes, the most interesting are probably the improved RDFa extractor (cheers to Toby Inkster and Masahide Kanzaki for code) and the new auto-cleanup of unused values/hashes in the RDF store. I received a couple of more patches which will be integrated in the coming weeks:

new component: Resource
new method: completeQuery (PREFIX-injection)
Reader: new method: getResponseHeaders
RDFa: fixes, +3 test case PASSes (thx to Toby Inkster & Masahide Kanzaki)
Class: auto-populate POST (php5 bug)
Class: refactored *PName methods
new methods: toIndex, toTriples, checkRegex
Parsers: unsetting reader object to fix garbage collection
SelectQueryHandler: improved LIKE-check for REGEX-rewriting
Class: used prefixes were not logged, leading to serialization gaps
Class: fixed root calculation bug in calcURI
Class: new methods: toDataURI/fromDataURI
ARC2_SPARQLScriptProcessor: improved automatic PREFIX injection
ARC2_RemoteStore: added automatic PREFIX injection and getResourceLabel method
ARC2_StoreSelectQueryHandler: fixed missing brackets in getExpressionSQL.
Reader: Improved timeout handling
Reader: support for port in http header (thx to Roan O'Sullivan)
Slowly starting to switch to inline PHPDoc documentation
Atom_Parser: Addition: support for link types
DeleteQueryHandler: Addition: cleanValueTables method (auto-called every 500 DELETE queries)
Class: new method: resetErrors
Class: switch from getScriptURI to getRequestURI in init()

In related news:

Tuukka Hastrup created an ARC 2 Starter Pack that simplifies the process of setting up an ARC store.
Andrew Ritz created a WordPress extension that lets you embed results from remote SPARQL endpoints directly in your blog pages.

SKOS + DC + Linked Data = Semantic Tagging?

2009-08-19T12:35Z

Still looking for a simple way to tag concrete resources (to-do items, people, locations) with personal concepts (e.g. "non-profit", "research", "semweb"), and also with other non-conceptual resources (clients, projects), I skimmed through the fresh SKOS Recommendation. I'm still a fan of SKOS and frequently wonder about semweb apps where the internal models are grounded in pluggable, personal(!) SKOS schemes, instead of coordination-intensive RDF Schemas or OWL ontologies. I don't know if such an approach could really work, I guess network effects benefit more from rather tightly defined relations and identifiers. Mainly just to have it written down somewhere (this is really not well thought out yet), here are some of the related entry points and considerations:

Tagging should be personal.
While I like the idea of grounding tags in existing dictionaries such as DBPedia, tags seem to work best when they are as user-defined and informal as possible. Last year, I experimented with a tool that allowed me to tag things with other people's delicious tags. It just felt wrong, I wanted my "own" tags. (I think the latest Faviki release is a nice example for combining the best of both worlds).
SKOS supports personal tags
Concepts in SKOS are sort-of scoped (or "namespaced"). If I describe a "Fun" concept, it is defined as seen by the creator of the concept URI, i.e. I can annotate it with ':Fun dct:creator <#me> ; dct:created "2009-08-19"' etc, even though the general idea of Fun was clearly not invented by me, and definitely before today.
Tags should be safely portable
Thanks to URIs, SKOS concepts can be ported to other applications, and they can be grouped and organized in so-called concept schemes, i.e. I could have a "Waving" in a "Dance" concept scheme, and also in a "Netiquette" scheme.
There is a need to merge tag sets
If tags are used to organize all sorts of personal things, it should be possible to merge them into a unified model. Mainly for personal use ("personal world view"), but also for sharing with other people and linking to their views. This is again possible thanks to SKOS being based on RDF, URIs, and very loose semantics.
There is a need to tag real-world objects with concepts
This is partly obvious. Tags are a means to an end. But while they are already widely used to annotate document-like resources (web pages, photos, etc), I'd also like to tag things like my projects, people in my address book, and similar non-documents. From the SKOS Primer: While the SKOS vocabulary itself does not include a mechanism for associating an arbitrary resource with a skos:Concept, implementors can turn to other vocabularies So, whatever predicate URI we are going to use, it's not going to be provided by SKOS directly.
Maybe Dublin Core terms can link non-documents to concepts
This is a slightly controversial conclusion/assumption, given that DC terms are mainly associated with document metadata. But after exploring the DCMI website, I can't find any clear evidence that their terms can't be used more generally. Both the Usage Guide (thanks to Masahide for the pointer) and the Abstract Model actually support this thought. The Usage guide mentions that "DC metadata can be applied to other resources as well" (but notes that the suitability may depend on the particular context at hand), and the Abstract Model states that the notion of a Dublin Core "resource" is equivalent to "Resource" defined in RDF Schema, which can be anything, even including Literals. So, we can most probably use dct:subject or dct:relation to tag a project or person with a SKOS concept.
There is a need to associate concepts with real-world objects
If we organize our personal concept space with SKOS, we may also want to more formally specify our personal concepts, so that other applications or people can merge them with their tags. Therefore, we need a predicate that can relate concepts to non-concepts such as DBPedia identifiers. Such a mechanism could maybe also help with RDF's general problem of URI aliases. I could have a personal, canonical concept URI for a resource and use it as a container for the resource's various aliases. Again, SKOS does not provide a predicate for this use case, so we've got to look elsewhere.
Maybe Dublin Core terms can link concepts to real-world objects
Another possibly controversial conclusion, but again there is supporting text in the DCMI specs: "A value associated with the Dublin Core Subject property is a concept (a conceptual entity) or a physical object or person (a physical entity)". So, if the value of dc:subject can be a non-document, we can say things like :Berlin a skos:Concept; dct:subject dbpedia:Berlin .. This is very interesting because it could allow us to use dct:subject in both ways: for the tagging of things, and also for grounding tags. FOAF has a handy primaryTopic term, which could work in this context, too, but unfortunately, its scope is (currently) set to foaf:Document. DanBri also suggested the creation of a dedicated skos:it (or similar) predicate which would be even better.
Sometimes I'd like to "tag" real-world objects with real-world objects
Don't know if tagging is still the right word here, but what I mean is a generic relation for arbitrary things in a common application context. Often, we can do better by specifying the relation between two resources, but in other cases, a simple, maybe just temporary link, is better than laziness leading to a completely non-annotated resource. Given the two DCMI-related findings above, we could maybe conclude that a predicate like dct:relation can also be used to relate a project to a person, or the other way round, without having to invent a new predicate.

</brain:dump>

SemWeb T-Shirt Shop closed

2009-07-24T08:50Z

Just a quick FYI: I've closed the SemWeb Spreadshirt Shop from last year. I never had a payout (you have to reach a certain amount of profit before you earn actual money), and as I plan/have to discontinue most of my many pet projects anyway (Simplify Your Life etc.), this one was rather easy to start with.

I guess my red semweb cap just became a rarity ;)

The Semantic Web - Not a piece of cake...

2009-07-08T14:55Z

For a client project I've been looking at Isometric Projection, which is not only nice for mapping 3D objects to a 2D environment, but even more so for adding a 3rd dimension to (previously) flat visual objects. The additional axis allows for much more information to be provided, without (necessarily ;) sacrificing compactness and simplicity.

While I was pushing small boxes around on a 30° grid, Jim Hendler tweeted about his Layer Cake talk from the recent Dagstuhl meeting (which is awesome, BTW. Read it, if you haven't yet) and I started to wonder if an isometric version of the tech stack could help reduce the overload resulting from the current two-dimensional ones. Not really, I fear, but it was a fun experiment nontheless. Might be worth exploring this a little further. At least the concepts can be separated from specific technologies and the application layer has a different angle than before (which I personally think makes more sense). Anyway, just wanted to share the result. Enjoy.

Feel free to use and share.

Code.semsol.org - A central home for semsol code

2009-06-22T14:00Z

The code bundles on the ARC website are generated in an inefficient manual process, and each patch has to wait for the next to-be-generated zip file. The developer community is growing (there are now 600 ARC downloads each month), I'm increasingly receiving patches and requests for a proper repository, and the Trice framework is about to get online as well. So I spent last week on building a dedicated source code site for all semsol projects at code.semsol.org.

So far, it's not much more than a directory browser with source preview and a little method navigator. But it will simplify code sharing and frequent updates for me, and hopefully also for ARC and Trice developers. You can checkout various Bazaar code branches and generate a bundle from any directory. The app can't display repository messages yet (the server doesn't have bzr installed, I'm just deploying branches using the handy FTP option), but I'll try to come up with a work-around or an alternative when time permits.

CommonTag too complicated?

2009-06-12T11:25Z

Update: I just read the spec again, I can't tag non-content with the CommonTag vocabulary. Too bad, ignore the last paragraph, please.

Sorry for raising my voice here, but some of us are really working hard to show that SemWeb technologies don't have to be complicated, and unfortunately, the new CommonTag effort seems to send exactly the opposite message.

Don't get me wrong, a widely used tagging ontology would be great. We do have 3 (or 4? 5?) tagging vocabularies already, but none really caught up, possibly because tagging is meant to be simple and the proposed solutions apparently weren't easy enough. CommonTag is promoted as being "simple" and "easy", but after looking at the examples in the QuickStart Guide, I'm not so sure:

The snippets are really off-putting (not only for Non-RDFers). Do I really need multiple nested HTML nodes to create something as simple as a tag?
Couldn't the term names be more intuitive? What could a ctag:Tag be? The actual tag or an intermediate resource that is then, err, tagged? A person ctag:tagged a resource, right? Ah, no.
Why aren't the term names at least consistent? "ctag:taggingDate" follows noun-role, "ctag:tagged" is a dunno, "ctag:means" is a present-form verb, "ctag:isAbout" sort-of follows the hasPropertyOf anti-pattern.
The vocabulary introduces aliases for well-deployed terms such as rdfs:label and dct:created, which makes its use in practical settings expensive (it'll ease things on the author side, though).

To be a little more constructive: Using the vocabulary doesn't have to lead to the complicated markup seen in the examples. I'm sure they'll soon get better snippets from someone in the RDFa community. And apart from that, there is also a handy term in the RDF Schema which might just be what you are looking for: "ctag:isAbout". It lets you directly point from a resource (default is the page) to a Linked Data identifier (e.g. from DBPedia), without the need for all those intermediate nodes (which lead to triple bloat and slow down SPARQL queries). CommonTag-consuming apps will have to implement some form of inferencing to handle "isAbout", but as the term is in the spec, I assume they plan to.

Granular modeling of tags is apparently tricky, but shouldn't there be some sweet spot? Something a little more expressive than rel-tag but less complex than a fully spec'd Tag ontology? xFolk looks promising, or maybe the CommonTag group members could have agreed on formalizing and supporting "scoped rel-tag" (rel-tags with an optional RDFa "about" container). Most rel-tag-to-RDF converters have some form of scoping already anyway (because tags can apply to reviews, pages, vcards, etc.). That would have been a cool outcome after 1 year of stealth work.

I may as well just over-stress the simplicity aspect here. Maybe CommonTag is "simple enough" for web publishers. There are some initial supporters, and for RDFers, the nested structures and bnodes will most probably be acceptable. So let's see how things evolve.

I personally think I'll have a closer look at ctag:isAbout. I'm still looking for an alternative to dc/dct:subject to tag arbitrary things with arbitrary identifiers, maybe CommonTag can provide it, although

<#me> ctag:isAbout dbpedia:Semantic_Web .
still doesn't sound right for a rich tag, and the domain is "ctag:TaggedContent" which sounds wrong for non-textual resources, too. (dct:relation is the best I could find so far for tagging things with things, but Dublin Core is coming from a publishing context and is therefore often recommended for describing publications only).

ESWC 2009 Linked Data Dashboards

2009-06-04T13:20Z

In case you missed the tweets or a local announcement: The first Paggr application went online a few days ago. This year's ESWC Technologies Team pushed things a little further, with RFID tracking during the event and extended conference data that includes detailed session and date/time information (kudos to Michael Hausenblas for RDFizing even PDFs).

Based on this dataset, we provided a conference explorer and stress-tested the "Dog Food" server while at it. The system survived, but I also learned a lot. We used about 50 RDF stores for the different public and user-specific dashboards, which basically worked nicely. However, rendering non-ugly resource summaries requires a bit of endpoint hammering, and some of the more complex path queries resulted in timeouts. Yesterday, I had to create a mirror from the data dump to route a couple of widgets through a replicated (ARC :-) endpoint. But then this is also one of the powerful possibilities that come with semantic web technologies. You can often switch or double the back-end repository in no time, and without any code changes. (And as all the Sparqlets are created in a web-based tool, I didn't even have to upload a changed configuration file. I simply tweaked a SPARQLScript parameter.)

Anyway, there are a couple of public dashboards, in case you'd like to give it a try (it's still an early version), I also embedded a short screencast below. The system is going to be moved to a DERI server when the conference is over, but the URIs and data will probably stay stable. (And no, it won't really work with IE yet.) More to come!

HQ version (quicktime, 110MB)

Simple RDFication of SPARQL SELECT results with RDFa

2009-05-26T07:50Z

A couple of weeks ago, I've written about the self-enforcing value spiral that RDF data enables. Here is an example about how RDFa can be used to support this "Repurpose-Republish" loop.

While data exchange between different semantic web sources is usually RDF-based (i.e. the data always maintain their semantics), there is one major exception: SPARQL SELECT queries. This developer-oriented operation returns tabular data (similar to record sets in SQL). Once the query result is separated from the query, the associated structural data is lost. You can't directly feed SELECT results back into a triple store, even though querying based on linked resources means that you have just created knowledge. It's a pity to show this generated information to human consumers only.

One of the demos at my NYC talk was a dynamic wiki item that pulled in competitor information from Semantic CrunchBase and injected that into a page template as HTML. The existing RDF infrastructure does not let me cache the SELECT results locally as usable RDF. And a semantic web client or crawler that indexes the wiki page will not learn how the described resource (e.g. Twitter) is related to the remote, linked entities.

However, by simply adding a single RDFa hook to the wiki item template, the RDF relation (e.g. competitor) can be made available again to apps that process my site content. This is basically how Linked Data works. But here is the really nifty thing: My site can be a consumer of its own pages, too, recursively enriching its own data.

I tweaked the wiki script which now works like this: When the page is saved, a first operation updates the wiki markup in the page's graph (i.e. the not-yet-populated template). In a second step, the page URL is retrieved via HTTP. This will return HTML with RDFa-encoded remote data, which is then parsed by ARC, and finally added to the same graph. We end up with a graph that does not only contain the wiki markup, but also the RDFized information that was integrated from remote sites. After adding this graph to the RDF store, we can use a local query to generate the page and occasionally reset the graph to enable copy-by-reference. And all this without any custom API code.

Back from New York "Semantic Web for PHP Developers" trip

2009-05-25T15:45Z

I'm back from New York, where I was given the great opportunity to talk about two of my favorite topics: Semantic Web Development with PHP, and (not necessarily semantic) Software Development using RDF Technology. I was especially looking forward to the second one, as that perspective is not only easier to understand for people from a software engineering context, but also because it is still a much neglected marketing "back-door": If RDF simplifies working with data in general (and it does), then we should not limit its use to semantic web apps. Broader data distribution and integration may naturally follow in a second or third step once people use the technology (so much for my contribution to Michael Hausenblas' list of RDF MalBest Practices ;)

The talk on Thursday at the NY Semantic Web Meetup was great fun. But the most impressive part of the event were the people there. A lot to learn from on this side of the pond. Not only very practical and professional, but also extremely positive and open. Almost felt like being invited to a family party.

The positive attitude was even true for the workshop, which I clearly could have made more effective. I didn't expect (but should have) that many people would come w/o a LAMP stack on their laptops, so we lost a lot of time setting up MAMP/LAMP/WAMP before we started hacking ARC, Trice, and SPARQL.

Marco brought up a number of illustrating use cases. He maintains an ~~(inofficial, sorry, can't provide a pointer)~~ RDF wrapper for any group on meetup.com, so the workshop participants could directly work with real data. We explored overlaps between different Meetup groups, the order in which people joined selected groups, inferred new triples from combined datasets via CONSTRUCT, and played with not-yet-standard SPARQL features like COUNT and LOAD.

And having done the workshop should finally give me the last kick to launch the Trice site now. The code is out, and it's apparently not too tricky to get started even when the documentation is still incomplete. Unfortunately, I have a strict "no more non-profits" directive, but I think Trice, despite being FOSS, will help me get some paid projects, so I'll squeeze an official launch in sometime soon-ish.

Below are the slides from the meetup. I added some screenshots, but they are probably still a bit boring without the actual demos (I think a video will be put up in a couple of days, though).

RDF and SPARQL for PHP Developers