Posts tagged with: arc

The Urban Dictionary describes SPOG as "super pimped out gangsta" or as "a weapon that (...) had a fusion reactor as a power source". Sorry to disappoint you, neither has become part of ARC. Nevertheless, the SPOG I mean is quite powerful, too. It is a constrained SPARQL XML result format from SELECT queries that was proposed by Morten Frederiksen a few months ago. SPOG enables streaming store backups/dumps, and being another RDF serialization, it can be used for streamed loading as well. Support for SPOG was added in the latest revision (2008-07-02) and extends the store and the endpoint components:

• The store got a dump() method that stream-outputs SPOG from all quads, and a createBackup($path, $alternative_query) method to write a SPOG dump (or custom SPO(G) query result) to a local file
• The SPARQL endpoint feature list accepts "dump" as a new read operation
• The SPARQL endpoint accepts "DUMP" as a query type now ("DUMP" also works via the internal query() method)
• The format detector accepts SPOG XML as an RDF format now, SPARQL+ queries will work fine with LOAD <some-spog-file.srx>. (There is now a dedicated SPOG parser for streaming LOADs.)

These additions should simplify graph exchange and store replication quite a bit.

Morten++ for the idea and an initial implementation.

Documentation - Release Notes

I just managed to trick my experimental SPARQLScript parser into accepting simple IF-branches and placeholders. Here is an example of what is going to be possible with ARC soon (and yes, I know this snippet most probably won't excite anyone but me ;)

BASE <http://sparqlbot.semsol.org/data/>
PREFIX dc: <http://purl.org/dc/elements/1.1/>

# set the endpoint
ENDPOINT <endpoint.php>

# feed still fresh?
$current = ASK FROM <graph-updates> WHERE {
  <http://planetrdf.com/index.rdf> dc:date ?date .
  FILTER (?date > ${now-1h})
}

# refresh feed and update graph log
IF (!$current) {
  DELETE FROM <http://planetrdf.com/index.rdf>
  LOAD <http://planetrdf.com/index.rdf>
  INSERT INTO <graph-updates> { <http://planetrdf.com/index.rdf> dc:date "${now}" }
} 

(Parsed Structure)

The fun thing about the whole SPARQLScript experiment is that the parser (so far) is still below 200 LOC. A lot can be re-used from the official SPARQL Grammar, e.g. IF-blocks are really just:

Script ::= ( Query | PrefixDecl | EndpointDecl | Assignment | IFBlock )*
IFBlock ::= 'IF' BrackettedExpression '{' Script '}'

Implementing the actual SPARQLScript processing engine is of course more work than the parser, but I'm making progress there, too.

I couldn't afford attending this year's ESWC in person. This is a pity as the annual "Scripting for the Semantic Web" (SFSW) Workshop is part of the main conference, and I would have loved to meet the SemWeb Scripting community. I did send in a submission to the SFSW scripting challenge, though, and if the tweets that came through the Tenerife net bottleneck are right, it was worth the effort :-)

And the 2nd prize went to SMOB, the SemWebby Twitter. Congratulations to Alexandre, Tuukka, Uldis, and John! (Cool to see ARC used in their project, too.)

I promised T-Shirts to ARC's core contributors some time ago, and I think I'll use the prize (kindly sponsored by the SemWeb masters at Talis) to finally create and send out some Thank-Yous to the community.

The latest ARC release comes with a couple of non-trivial (but also not necessarily obvious) changes. The most significant (as it involves ARC's resource indexes) is the alignment with the structures used by the Talis platform. ARC's parser output and PHP or JSON formats are now directly processable by Talis' platform tools. The documentation has been updated already, you may have to adjust your code (basically just "s/val/value/" and "s/dt/datatype/") in a few places.

The second major addition is a Remote Store component (documentation still to come) that is inspired and based on Morten Frederiksen's great RemoteEndpointPlugin. The Remote Store works like Morten's Plugin, but supports SPARQL+' LOAD, INSERT, and DELETE (i.e. write/POST) operations.

The third addition is also the reason why the Remote Store (which can be used as a SPARQL Endpoint Proxy) became a core component. I've worked on a draft for a SPARQL-based scripting language during the last months, and the latest ARC revision includes an early SPARQLScript parser and a SPARQLScript processor that can run a set of routines against remote SPARQL endpoints. What's still missing before this stuff becomes more usable (apart from documentation ;) is output templating and some other essential features such as loops. I do have an early prototype running in a local SPARQLBot version, but I probably won't have it online in time for tomorrow's Semantic Scripting Workshop (that I'll try to attend remotely at least). This is really powerful (and fun) stuff that will be available soon-ish. Can't wait to replace my hard-coded inferencer with a set of easily pluggable SPARQLScript procedures.

Other tweaks and changes include a very early hCalendar extractor and a couple of bug fixes that were reported by (among others) the SMOB project maintainers.

As usual, thanks to all who sent in patches, bug reports, feature requests, and stress-tested ARC. I think we're pretty close to a release candidate now :-)

Still a few hours left to finish my presentations, then I'll join Germany's WebDev crowd at the webinale 2008 in Karlsruhe (It's taking place at the same location as this year's ISWC). My talks are about "Semantic Web Tech 'n' Use" (mostly microformats, RDFa, SPARQL), and RDF-based "Online Social Graph Consolidation" (FOAF, XFN, SPARQLy inference, knowee etc.), and there will be more SemWeb-related talks:

• Semantic Rich Internet Applications (Raju Bitter)
• The Semantic Web: What the Heck are we Doing? (Duane Nickull)

A (personally) interesting thing about the webinale is its co-location with the International PHP Conference, and the (new) Dynamic Languages World Europe, and that registering for one conference includes free access to any of the others. It's the perfect audience to talk about practical SemWeb Scripting with ARC and PHP.

If there was a "most productive SemWeb coder" category in Danny's "This Week's Semantic Web", this week's turn would probably be Keith Alexander's. Last week, he provided no fewer than three ARC2 Plugins:

• An RDFa Serializer
• Utilities for working with ARC structures (e.g. filter, merge, or diff)
• A SPARQL (Re-)Serializer (very handy for checking/adjusting SPARQL queries before they are passed to the store)

While at it, he also implemented a SPARQL+ wrapper for Talis Platform stores.

I think I blogged about Morten's RemoteEndpoint plugin a while back (this one should really become part of the core codebase), but did I mention Peter Krantz' File System Synchronizer? It keeps an RDF Store in sync with a file system directory which enables a really nice option to implement larger RDF editing systems on top of ARC: By using editing tools that work with small RDF files (quick response times and everything) and his plugin, it becomes possible to provide rich query functionality over the whole dataset without the store getting in the way of the publishing tools. RDF index rebuilding can be slow, de-coupling read from write operations and introducing an asynchronous update process is a nice solution.

Awesome stuff.

The latest ARC revision got two new features: SPARQL Triggers and MySQL function extensions for SPARQL.

SPARQL Triggers

Triggers in ARC were suggested by Dan Brickley, who is experimenting with dynamically populated/updated Group definitions. What you can effectively do now in ARC is associating custom trigger classes with SPARQL query types, which will then be automatically called after registered query types, for example to refresh inferred Graphs:

$config = array(
  ...
  'store_triggers' => array(
    /* register LOAD triggers */
    'load' => array('updateFriendsList', 'crawlXFNLinks'),
  ),
);
$ep = ARC2::getStoreEndpoint($config);
$ep->go();

MySQL Extension Functions

Morten Frederiksen did it again. He sent in about 10 lines of code which he suggested to add to ARC's SQL rewriter. The effect? ARC suddenly has access to dozens of MySQL functions. That's CONCAT, CURDATE, MD5, UNIX_TIMESTAMP, and many more. A namespace for MySQL function URIs is now online, and queries look like this:

PREFIX mysql: <http://web-semantics.org/ns/mysql/>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>

SELECT ?person WHERE {
  ?person foaf:givenname ?n1 ;
          foaf:family_name ?n2 .
  FILTER (mysql:concat(?n1, " ", ?n2) = "Alec Tronnick") .
}

I talked a little bit more about these things with Danny Ayers in a recent podcast.

OK, you're probably already wondering if Morten and I have a link exchange contract, but anyway: He just announced a plugin for ARC that provides "access to remote SPARQL endpoints as if they were local stores." Cool stuff :-)

I guess I already pushed out enough ARC spam today, so I'll keep things short: SPARQL is now a W3C Recommendation!

What I'm personally very happy about is the Implementation Survey which features two pure-PHP implementations*. This really opens the door for mainstream Web Developers to start exploring RDF and SPARQL on off-the-shelf hosted web servers. Everything I create these days (e.g. the ARC site, including the bots and archive generators there, or this blog) is powered by SPARQL. It's an amazing productivity booster as you never have to worry about complicated JOINs or evolving database schemas again. You can just code away and it's great fun to work with. Want more Testimonials? The Data Access Working Group collected quite a number of them from W3C member organizations.

* Don't let yourself be fooled by RAP's low report scores, their SPARQL engine is quite mature, they just didn't run the whole test suite.

Together with Morten Frederiksen and Dan Brickley (who is revisiting his SparqlPress idea), I've created a WordPress extension (called "RDF Tools") that adds an (ARC-based) RDF Store and SPARQL Endpoint to the blogging system. The store is kept separate from the WP tables (i.e. it's not a wrapper), but you can use WP's nice admin screens to configure it (screenshot), and given the amount of developer-friendly hooks that WP offers, I'm curious what can be done now, possibly in combination with other extensions such as those Alexandre Passant is working on. It could perhaps also be handy as a deployment accelerator for knowee.

I'm blessed with a small but first-class community around ARC that helps me with bug reports, patches, encouraging feedback, and nifty ideas. One example for the latter was Morten Frederiksen's invention to allow ARC to be extended with third party plugins. He even demonstrated the utility by enhancing the toolkit with a remote SPARQL endpoint for his named graph exchange work. ARC plugins are not bundled with the core codebase (which is meant to stay compact), but can easily be integrated in any ARC installation (Developer documentation is now online, too).

My first own plugin was triggered by Tim Berners-Lee's suggestion to write a lightweight request handler for an RDF-powered Data Wiki, as described in a recent Tech Report (PDF) and already implemented with Algae. I had to tweak the SPARQL+ spec and ARC's Query Parser to make it compatible with Eric Prud'hommeaux's SPARQL/Update flavor. This had the nice side-effect that all three SPARQL Write proposals (SPARUL, SPARQL/Update, SPARQL+) now (almost) share a common subset for basic INSERTs and DELETEs. After these updates, writing the plugin itself became almost trivial.

The code is still experimental and limited, but it's available for download, together with setup instructions. The Data Wiki plugin doesn't require a database (unlike the other SPARQL components in ARC) and supports update calls sent by RDF editors such as the Tabulator. I've set up a demo RDF wiki and will try to add remote update functionality to my own editor (to be renamed) now as well. Hmmm, would be cool to have a selection of generic tools to collaboratively read from and write to shared RDF spaces one day.

After writing a bunch of instructions for ARC2 this week, I think/hope it is now finally ready for experiments and feedback. The site will get more documentation and code snippets in the coming weeks, and some components are not even part of the release. However, I've been waiting long enough already. So, here goes:

• Mailing list announcement
• The new website (built with PHP and ARC2)
• "Introducing ARC2"
• ARC-DEV group mailing list

Shout-outs to everyone who helped with bug reports, encouraging feedback, and suggestions for the new release. Special thanks to CivicActions and Jonathan Hendler for development support and various stress tests of earlier versions.

The new ARC site is coming along quite nicely. Last week I implemented two (low-level) agents that log IRC conversations and mails to ARC-DEV. RDF and SPARQL make such things incredibly easy. Today, I started writing documentation for the preview release of ARC2, and one the core changes to ARC1 is the removal of the API class for inserts and deletes in favour of an extended SPARQL, called SPARQL+ which enables aggregates, LOAD, INSERT, and DELETE, without the need for major query engine code additions.

LOAD is compatible with the LOAD operation introduced in the SPARUL proposal:

LOAD <http://example.com/> INTO <http://example.com/archive>

INSERT and DELETE are different, though. They re-use the LOAD and CONSTRUCT handlers which simplified the implementation and will hopefully make it easier for people who just learned SPARQL's standard syntax. INSERT and DELETE in SPARQL+ each support two different forms, one for explicit triples (with simple wildcards in DELETE queries), and one for dynamically CONSTRUCTed ones, e.g.

DELETE {
 <#foo> <bar> "baz" .
 <#foo2> <bar2> ?any .
}

or

INSERT INTO <http://example.com/inferred> CONSTRUCT {
  ?s foaf:knows ?o .
}
WHERE {
  ?s xfn:contact ?o .
}

More examples and detailed information about how exactly SPARQL+ extends the SPARQL grammar are available in ARC2's SPARQL+ documentation section

I'm still working on the new website for ARC, but I managed to set up a group mailing list yesterday. It's a little (*cough*) experimental, based on ARC2 and Trice (another forthcoming semsol product). So, this is a shout-out to ARC users and developers with an invitation to subscribe and help me test that "DIY SPARQL Mailman" before I do a proper announcement for the new site and community tools (hopefully later this week).

Thanks in advance,
Benji

Yesterday, SWEO's 3rd F2F took place at the MIT, and although I couldn't afford in-person attendance, Leo Sauermann from AI research center DFKI enabled a near-equivalent in Germany. I missed the first agenda item due to the usual Deutsche "zanks for traffelink wizz us" Bahn delays, but apart from that it was a fun event and worth the 4 + 6 hour train journey. Super-modern DFKI was impressive, and the multi-screen polycom with remote-controlled cam truly rocked.


pic by Leo

The train back had power sockets, so I could hack a bit, and when I arrived in DDorf at 7am this morning, I had a working RDFa parser for ARC's next release. Almost there, now..