Posts tagged with: twitter

An article about smesher, the semantic microblogging system I've been working on during the last few weeks, is now online at DevX.

The article explains the technical side of the MBC09 talk, a code bundle is included. The latter still lacks the just added posting functionality, I'll try to make another release available once the code is a little more stable.

OK, looks like the CrunchBase bot got some attention after Fred Wilson's post about possibly handy phone apps. So, if you discovered Semantic CrunchBase or the related bot via non-SemWebby paths, the whole "Define your own API commands with SPARQL" is probably a bit too much, tech-wise. Here are some short instructions for using the bot:

The syntax is basically just "@cbbot, command" where command has to match one of the user-defined commands. Some of them are generating HTML and might therefore not be suited for Twitter access. The main, Twitter-optimized commands are:

• ${role} (of|at) ${company}: This command is for requests like "@cbbot, Intern at TechCrunch", "@cbbot, founder of facebook", "@cbbot, board of Pandora", "@cbbot, ceo of twitter", etc. (I made the command case-insensitive today, BTW)
• link to ${keyword}: This command returns a CrunchBase link for the given keyword. The latter can be a company (name or CB identifier), product (name or CB identifier), or person (CB identifier). Examples: "@cbbot, Link to foodzie", "@cbbot, link to EC2", "@cbbot, link to michael-arrington".

Would you like to see additional twitter commands, but don't know about SPARQL or how to use the command editor? Please send command requests to me or to the bot and I'll try to add them.

Update (2008-08-25): I've written a follow-up post explaining the main commands. Heh, as John Crow points out, a cool way to look at work is to think in terms of being "between vacations". So, here is a fun/experimental hack from between my previous (bday on Santorini) and next (family visit at Lake Constance) vacation: It's a tweetsphere cousin of SPARQLBot who can give answers to user-defined CrunchBase API commands. (Instructions for the "Pimp MY API" tool.)

In order to use the bot, just send a known command call to cbbot (using the @-convention), for example "@cbbot, founder of Flickr". A tweet with the answer should appear on your "Replies" tab (or under "Recent", if you are following cbbot, see screenshot below).



The bot is implemented as a long-running PHP process (*cough*), you may have to re-start the script in case you don't get an answer within a few minutes.

In the Semantic CrunchBase announcement, we saw how SPARQL can be used to retrieve fine-grained information from the CrunchBase graph. This follow-up post explains "Pimp My API", a browser-based tool for creating tailored API calls by combining SPARQL with input parameters and output templating. The command editor consists of three tabs: "Define it", "Test it", and "Activate it".

Step 1: Define a new API command

In the 1st field ("Command") you define a (human-readable) command, with input parameters set via the ${parameter_name} notation. In the screenshot on the left, we created "${role} of ${comp_name}" which we are going to use to retrieve persons with a specific role at a given company. The command processor will automatically assign variables for a matching input string, e.g. "Editor of TechCrunch" will set the variable ${role} to "Editor", and ${comp_name} to "TechCrunch".

Now on to the 2nd field ("SPARQLScript code"):
SPARQLScript is an experiment to extend SPARQL with scripting language features such as variable assignments, loops, etc. (think Pipes for SemWeb developers). If you are familiar with SPARQL, you will notice only three differences to a standard SPARQL query: In the first line, we are setting a target SPARQL service for the following script blocks. In the second line, we assign the results form the SELECT query to a variable, and the the third difference is the use of placeholders in the query. These placeholder will be filled from matching variables before the query is sent to the target endpoint.

If you don't know SPARQL at all, here is a pseudo-translation of the query: Find resources (?comp) with a cb:name (cb is the CrunchBase namespace used for CB attributes) that equals the input parameter "comp_name", and a relationship (?rel). The relationship should have an attribute ?role which regex-matches the input parameter "role". The relationship should also have a cb:person attribute (linking to ?person). The ?person node should have the cb:first_name and cb:last_name attributes. Those should be returned by the query as "fname" and "lname" variables. The whole result set is then assigned to a variable named "rows" (Hmm, maybe the SPARQL is easier to read than my explanation ;)

The third form field lets us define an output template. Each stand-lone section surrounded by quotation marks will fill the output buffer. Thus, looping through the "rows" will create a small name snippet for each row. Again, placeholders will be filled with values from the current script scope.

Step 2: Test your new Command

Using the Test form, we can see if our command pattern works, and if the result is formatted as desired. Should anything go wrong, we can select "Show raw output" to get some debugging information. Please note, even though we are using a browser, simple HTML forms, and a friendly pattern language, the commands are sent to real Web services. A broken script usually just hurts your local machine. A distributed Semantic Web processor like this, however, may harm other people's servers, so we should be careful, start small, and improve our script incrementally. In this case, the output result is a little ugly, so we could improve the output template and inject commas:

Step 3: HTTP access activation

Our command is now defined and successfully tested, let's turn it into a public API call.

Instead of the sort-of natural language command, the API expects GET or POST arguments.

The example above generates a plain text result, but it's also possible to return markup or other formats. SPARQLScript can access GETvariables via ${GET.var_name}, this feature can be used to create different output, depending on e.g. a "format" parameter. I'm also working on support for content negotiation, where you'd simply create a "${rows}" template and the SPARQLScript processor would auto-generate an appropriate serialization including correct HTTP headers.

Step 4: Have some fun

You may wonder why the command editor allows the definiton of a human-friendly pattern, when the API itself just needs the parameters. The patterns allow the implementation of an API call detector, i.e. depending on the input stream at a generic service URL, we can auto-detect the right script to run. I've test-implemented a Twitter bot that can reply to messages that match a stored API command on Semantic CrunchBase (Inactive during the week-end, it's not tested enough. Stay tuned ;). Here is a teaser screenshot for next week:

What is a scripting language without loops, or a Web language without a template mechanism? Not really usable. Yesterday, I finally added the two missing core features to my SPARQLScript processor, and I'm excited about eventually being able to test the whole thing. This is just the beginning (there is no string concatenation yet, and no WHILE blocks), but with the basic infrastructure (and documentation) in place, it's time to start gathering feedback. I'm going to upgrade SPARQLBot in the next couple of days which should be a fun way to explore the possibilities (also, it were the bot's users who triggered the creation of SPARQLScript in the first place).

So, what is it actually good for?

Mid-term-ish, I'm dreaming of an alternative to increasingly non-RDFy specs such as RIF and OWL2 (there is definitely some need for them, they just don't seem to really work for me and my Web stuff). Things like crawling, smushing, or custom inference tasks based on wild mixtures of RDFS, OWL, and SKOS should be doable with SPARQLScript.

Simple agents are another use case, as SPARQLScript simplifies task federation across multiple endpoints and RDF sources.

What's working already today is the creation of simple mashups and widgets. Below is a script that integrates status notices from my twitter and identi.ca feeds, and then creates an HTML "lifestream" snippet. The (live!) result is embedded at the bottom of this post.

# global prefix declarations
PREFIX dc: <http://purl.org/dc/elements/1.1/>
PREFIX rss: <http://purl.org/rss/1.0/>

# the target store
ENDPOINT <http://arc.semsol.org/demos/endpoint/>

# refresh feeds every 30 minutes
$up2date = ASK FROM <script-infos> WHERE {
  <script-infos> dc:date ?date . FILTER (?date > "${NOW-30min}")
}
IF (!$up2date) {
  # load feeds
  LOAD <http://twitter.com/statuses/user_timeline/9516642.rss>
  LOAD <http://identi.ca/bengee/rss>
  # remember the update time
  INSERT INTO <script-infos> { <script-infos> dc:date "${NOW}" }
}

# retrieve items
$items = SELECT * WHERE {
  ?item a rss:item ;
        rss:title ?title ;
        dc:date ?date .
} ORDER BY DESC(?date) LIMIT 8;

# output template
"""<h4>My online lifestream:</h4>
<ul>"""
FOR ($item in $items) {
  """<li><a href="${item.item}">${item.title}</a></li>"""
}
"</ul>"

(S)mashups here we come :)