%VOSWARNING%


%META:TOPICPARENT{name="VirtuosoWhitePapers"}%
---+ Deploying Linked Data

v1.2 (Virtuoso 5.0) April 2008

This document describes the process of deploying Linked Data into the existing Web. It discusses some of the difficulties 
faced in exposing RDF data and in bridging the "Semantic Data-Web" and the traditional "Document Web". Two generic 
approaches to resolving these deployment challenges are described, content negotiation and URL rewriting, before looking 
at OpenLink Virtuoso, both from the standpoint of how it implements these solutions and how Linked Data is deployed. 

A companion document, [[RDFViewOverviewDoc][Virtuoso Linked Data Views Getting Started Guide]], focuses on Virtuoso Linked Data Views, a facility for exposing relational data as RDF. In addition, it provides useful background information for readers 
unfamiliar with RDF and outlines some of the key technologies of the Semantic Web.

%TOC%

---++ Introduction

The ubiquitous "Web," born as the "World Wide Web," is primarily experienced today as a "Web of <i>Documents</i>," where 
documents (or Web pages) are connected by simple hypertext links. When you click on a hypertext link within one document, 
the result is simply that the browser loads (or downloads) the linked document. This widely understood and accepted pattern 
of interaction with the Web is made possible by two things &mdash; the <i>Uniform Resource Identifier</i>, or <i>URI</i>, 
and the <i>Hypertext Transfer Protocol</i>, or <i>HTTP</i>.

(Uniform Resource <i>Identifier?</i>  <i>URI?</i>  Don't we mean <i>URL</i>, or Uniform Resource <i>Locator</i>?  Yes and no.  A Uniform Resource <i>Locator</i> (URL) is a particular 
kind of Uniform Resource <i>Identifier;</i> a Uniform Resource <i>Name</i>, or URN, is another.  As may be obvious from these names, 
a <i>URL</i> specifies the <i>location</i> of a resource -- like, "the piece of paper centered on the blotter on your desk," or "the third book from 
the left on the top shelf of the bookcase in the entryway."  A <i>URN</i> specifies the <i>name</i> of a resource -- like "your resume," or "the
local Chicago telephone directory."  Both of these are <i>URIs</i> -- as both can be used to <i>identify</i> the resource in question, at a given 
moment in time.  The piece of paper centered on the blotter on your desk, and where the book is shelved, may change -- and though 
the URL is the same as what once referred to your resume, the URN is now different, as it is now "the menu for the pizza place 
down the street," or "the Tom Robbins novel, <i>Still Life With Woodpecker</i>."  On the Web, URLs can only lead to HTTP-transmissible 
documents, so the paper on which your resume is printed cannot actually have a URL -- but the word processing document which was 
printed on that paper <i>can</i> have a URL.  In this example, both the word processing document and the printout are associated with 
the URN, "your resume," and each is a different <i>representation</i> thereof -- one which is only easily consumable by humans, 
and one which is easily consumable by humans or machines.  URNs are typically less transient, as "your resume" will always mean the same 
document, but that document's content will change over time -- so increasingly common practice is to have URNs that incorporate 
some sense of time into the name, often leading to two "special" URNs which are tied to the "first" and the "latest" version of the 
document.  The "first" always leads to the same content -- but the "latest" is obviously likely to change over time.)
 
The popularity of the current "Document Web" sometimes obscures the fact that from the onset, Tim Berners-Lee envisaged 
a broader and deeper Web of <i>Linked Data</i>, where URIs weren't simply URLs and therefor limited to association with
HTTP-transmissible documents, and where links between resources were not limited to simple hypertext.  URNs make it
possible to have "hyperdata" links &mdash; explicit connections between <i>Named Entities</i> or <i>Data Objects</i>, 
rather than vague connections between document locations. Hyperdata links include descriptions of the kind of link exposed 
&mdash; such as "the PDF representation of your resume," "the Microsoft Word representation of your resume," "the website 
of the company at which you worked in 1998 which was named Widgets, Inc."  There is no natural requirement that URNs
be based on HTTP, but as we discuss below this <i>is</i> necessary to enabling the Web of Linked Data.

This document describes one way to start sprinkling Linked Data into the existing Document Web, gradually bringing the 
Web closer to Tim Berners-Lee's vision, without breaking its current functionality.

---+++ What is Linked Data?

"Linked Data" is the title of a Web Design Issues Note by Berners-Lee that issues a best practice recipe for injecting data 
into the Web as part of a broader effort to evolve the current Web of interlinked documents to a Web of interlinked data 
known as the "Semantic Data-Web" (Data-Web). The principles he outlined are paraphrased as:

   * Identify things (real or abstract) in your universe-of-discourse (or "data space") using URIs (whether URNs or URLs 
&mdash; each has its place)
   * Make each URI (URN, URL, or otherwise) acessible via HTTP, so that people can discover and explore your data 
spaces via the Web
   * Use URIs to expose the context of your data (i.e., describe and provide other information about your data, using URIs)
   * Enhance your URIs by adding links to other URIs, enabling discovery of other things on the Web

---+++ Deployment Challenges

The Data-Web and the Document Web are two dimensions of the same Web separated by a common element: the URI. On the 
Document Web, URIs always point to physical resources, while in the Data-Web they point to physical things that are 
associated with physical and/or abstract things. Of course, this unveils a number of deployment challenges.

---+++ Data Object Names

In the current Document Web, resource URIs do not separate <i>identity</i> from <i>representation</i>. The Document Web 
assumes that a resource URI points to the location of a physical Web information resource. The HTTP payload that conveys 
the "GET" request for a resource also includes a mechanism for defining representation. Thus, the URI 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code> 
points to the (X)HTML document representation of the physical resource <code>ALFKI</code> located in the directory <code>/Northwind/Customer/</code> 
on host machine <code>demo.openlinksw.com</code> that accepts HTTP requests at the default port 80.

The Data-web on the other hand, seeks to use the URI scheme in a manner that separates identity from representation. A URI may 
simply identify a physical or abstract entity, aka a "data object", and so serve as a unique data object name or ID. Accessing, 
or de-referencing, the data object returns a representation of the object, not the object itself. (For instance, the object in 
question may be Paris!)

---++++ Unambiguous Reference vs Ambiguous Access

When we refer to, or identify, a data object through a data object name, that reference should be unambiguous. However, when we 
access (or de-reference) a data object, access is inherently ambiguous. Accessing an abstract data object relies on materialization 
of a description of the entity in a form compatible with the transmission medium. As the object may have many possible descriptions 
(facets), the act of accessing it is ambiguous.

---++++URIs As Unique Data Object Names

Thus, unlike in the Document Web, in the Semantic Web the same URI 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code> 
cannot serve as both the identity and representation of the Customer <code>ALFKI</code>. The Linked Data provider needs to adhere to 
a URI based naming convention in order to avoid data access ambiguity. For example, the URI 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI#this][http://demo.openlinksw.com/Northwind/Customer/ALFKI#this]]</code>, 
could always to taken to imply the ID of the Customer data object referred to as ?<code>ALFKI</code>?. The URI with the <code>#this</code> suffix is a so-called 
<i>hash URI</i>, which is a convention adopted by some practitioners.

---+++Difficulties with Hash URIs As Data Object Names

In the prior section, we established the need for disambiguating references and accesses to resources via the Data-Web, and 
highlighted the <i>hash URI</i> scheme as one scheme some practitioners have adopted when using URIs as unique data object 
names. However from the perspective of the Data-Web Server (the piece responsible for understanding Reference), the URIs 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code> and 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI#this][http://demo.openlinksw.com/Northwind/Customer/ALFKI#this]]</code> 
are identical, and thereby inherently ambiguous, because nothing following the fragment identifier, "<code>#</code>", ever leaves 
the Web Client, due to the fact that the Web Client expects to process "<code>#this</code>" locally, post resource retrieval. 
As a result, the Data-Web Server has to figure out how to dereference the Information Resource URI
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code> 
and the Identity URI
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI#this][http://demo.openlinksw.com/Northwind/Customer/ALFKI#this]]</code> 
from the HTTP <code>GET</code> request payload that will only predictably contain the URI
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code>. (Even if 
a Web client knowingly tacks the data following the "<code>#</code>" to the HTTP <code>GET</code> request it has no control over proxies along the way that 
may strip out "<code>#this</code>".) Likewise, referencing an entity via its identity URI (the act of dereferencing) is only achieved via 
interaction with an associated Web information resource that "<code>DESCRIBE</code>" the entities in question. In reality, this associative 
process is inherently ambiguous and unavoidable.

It is also important to note that descriptive Web information resources can take the form of bona fide parameterized URLs of the kind 
commonly associated with RESTful Web Services.

---+++ Resolution of the Deployment Challenge

To unobtrusively evolve the dominant Document Web usage pattern to a Data-Web usage pattern, the challenges of Data Access and Data 
Reference need to be resolved using the existing Web infrastructure. The best means of resolution is content negotiation as it 
provides the foundation for an unobtrusive mechanism known as URL rewriting.

---+++ Content Negotiation

Content negotiation is a mechanism defined in the HTTP specification that makes it possible to serve different representations of 
#BodyNote1
a document (or any resource) at the same ([[#FootNote1][Note 1]]) URL, so that software agents can choose which representation best fits their capabilities. 
The originally conceived need for this mechanism stemmed from mobile phone browsers, which were better suited to smaller page sizes, 
without many graphics and other niceties of the fully featured Web, often satisfied by a WAP representation. In the world of RDF, 
a user interacting through a traditional Web browser may want a resource represented in HTML or XHTML, whereas a Semantic Web 
application would prefer an RDF/XML representation due to its Structured Data orientation.

A browser or any other HTTP based web application indicates it resource representation preferences by packaging these preferences 
via the "Accept:" headers of each HTTP request. For example, a browser could send this HTTP request to indicate that it wants an 
HTML or XHTML version of <code>[[http://www.openlinksw.com/whitepapers/data_management][http://www.openlinksw.com/whitepapers/data_management]]</code> in English or French:

<verbatim>
GET /whitepapers/data_management HTTP/1.1 
Host: www.openlinksw.com 
Accept: text/html, application/xhtml+xml 
Accept-Language: en, fr 
</verbatim>

Here, the HTTP <code>Accept</code> header sent by the browser indicates the MIME types it wants (<code>text/html</code> or <code>application/xhtml+xml</code>). An RDF 
browser, in contrast, might stipulate a MIME type of <code>application/rdf+xml</code> or <code>application/rdf+n3</code> to receive a rendering in RDF/XML 
or N3 respectively. 

Rather than returning the content in the required format directly, servers often implement content negotiation by redirecting to 
a URL where the appropriate representation is found. For example, a server might respond with:

<verbatim>
HTTP/1.1 302 Found 
Location: http://www.openlinksw.com/whitepapers/data_management.en.html
</verbatim>

The redirect is indicated by the HTTP status code <code>302</code> (<code>Found</code>). The client would then send another HTTP request to the new URL. 
HTTP defines a number of <code>3xx</code> status codes all of which indicate the client is being redirected. Instead of <code>302</code>, servers can 
also use <code>303</code> (<code>See Other</code>) to indicate the response to the request can be found at another location (as expressed via the 
"<code>Location:</code>" response header).

---++++ <nowiki>HttpRange</nowiki> - 14 Recommendations

The problem of URI/resource-type interpretation was originally addressed by the W3C Technical Architecture Group (TAG) around 
2005 and was known as the "HttpRange-14" issue. After a good deal of deliberation, the TAG proposed the guidelines below on the 
information that can be inferred from the HTTP protocol response codes when dereferencing a URI:

|*HTTP Response Code*|*Material Returned*|*Inference*|
|<code>200</code> %BR%(<code>success</code>)|A Resource Representation and its Location|A Web information resource has been located in the desired Representation.|
|<code>303</code> %BR%(<code>see other</code>)|A Resource Location|A redirection to the Location of an associated Web information resource in a desired Representation.|
|<code>4XX</code> or <code>5XX</code> %BR% (<code>error</code>)|Nothing|No Web information resource or Resource Location is discernible from the Resource and Representation combination used in the message.|

---++++ Solving Linked Data Challenges using Content Negotiation

Returning to our earlier example URIs 
(<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code> and 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI#this][http://demo.openlinksw.com/Northwind/Customer/ALFKI#this]]</code>) 
we can construct a decision table that demonstrates how a deployer of Linked Data would leverage content negotiation en route to 
alleviating the previously outlined Data Access and Data Reference challenges.

|*URI*|*URI Type*|*Requested Representation (X)HTML*|*Requested Representation RDF*|
|<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code> | Slash based | <code>406</code> (Not available or applicable) or <code>303</code> (Redirect to an associated resource in requested representation format, e.g., <code>[[http://demo.openlinksw.com/][http://demo.openlinksw.com/]]</code>)| <code>303</code> (Redirect to URL of information resource that <code>DESCRIBEs</code> the entity <code>[[http://demo.openlinksw.com/][http://demo.openlinksw.com/]]</code> in the Data space <code>[[http://demo.openlinksw.com/][http://demo.openlinksw.com/]]</code>) |
|<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI#this][http://demo.openlinksw.com/Northwind/Customer/ALFKI#this]]</code> | Hash based | <code>200</code> OK (Since fragment ID component of the URI doesn't affect the URL for the information resource | <code>200</code> OK (Return an information resource that <code>DESCRIBEs</code> the entity <code>[[http://demo.openlinksw.com/][http://demo.openlinksw.com/]]</code> |

---+++ URL Rewriting

URL rewriting is the act of modifying a source URL prior to the final processing of that URL by a Web Server.

The ability to rewrite URLs may be desirable for many reasons that include:

   * Changing Web information resource URLs on the a Web Server without breaking existing bookmarks held in 
User Agents (e.g., Web browsers)

   * URL compaction where shorter URLs may be constructed on a conditional basis for specific User Agents 
(e.g., Email clients)

   * Construction of search engine friendly URLs that enable richer indexing since most search engines cannot 
process parameterized URLs effectively.

---++++ Using URL Rewriting to Solve Linked Data Deployment Challenges

In the previous section we demonstrated how content negotiation and HTTP response messages could be used to 
address the data access issues arising from the use of URIs associated with resource identity and representation.

We determined earlier that URI naming schemes don't resolve the challenges associated with referencing data. 
To reiterate, this is demonstrated by the fact that the URIs 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code> and 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI#this][http://demo.openlinksw.com/Northwind/Customer/ALFKI#this]]</code> 
both appear as <code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code> 
to the Web Server, since data following the fragment identifier "<code>#</code>" never makes it that far.

The only way to address data referencing is by pre-processing source URIs (e.g., via regular expression or 
sprintf substitutions) as part of a URL rewriting processing pipeline. The pipeline process has to take the 
form of a set of rules that cater for elements such as HTTP Accept headers, HTTP response code, HTTP response 
headers, and rule processing order.

An example of such a pipeline for the hash URI scheme is depicted in the table below:

|*URI Source (Regular Expression Pattern)*|*HTTP Accept Headers (Regular Expression)*|*HTTP Response Code*|*HTTP Response Headers*|*Rule Processing Order*|
|<code><nowiki>/Northwind/Customer/([^#]*)</nowiki></code>|None (meaning default)|<code>200</code> or <code>303</code> responses depending on the user agent default or server side quality of service rules via Transparent Content Negotiation.|None|Normal (order irrelevant)|
|<code><nowiki>/Northwind/Customer/([^#]*)</nowiki></code>|(<code>text/rdf.n3</code>) / (<code>application/rdf.xml</code>) | <code>200 OK</code> and return the information resource that <code>DESCRIBEs</code> the entity identified by the hash URI in the requested representation. | None | Normal (order irrelevant) |
|<code><nowiki>/Northwind/Customer/([^#]*)</nowiki></code>| (<code>text/html</code>) / (<code>application/xhtml.xml</code>) | <code>200 OK</code> and return an information resource in requested representation. | None | Normal (order irrelevant) |

A similar pipeline for the slash URI scheme would be:

|*URI Source (Regular Expression Pattern)*|*HTTP Accept Headers (Regular Expression)*|*HTTP Response Code*|*HTTP Response Headers*|*Rule Processing Order*|
|<code><nowiki>/Northwind/Customer/([^#]*)</nowiki></code>|None (meaning default)|<code>200</code> or <code>303</code> responses depending on the user agent default or server side quality of service rules via Transparent Content Negotiation.| None | Normal (order irrelevant) |
|<code><nowiki>/Northwind/Customer/([^#]*)</nowiki></code>|(<code>text/rdf.n3</code>) / (<code>application/rdf.xml</code>)| <code>303 Redirect</code> to an associated URL of an information resource that <code>DESCRIBEs</code> the entity identified by the URI | None|Normal (order irrelevant)|
|<code><nowiki>/Northwind/Customer/([^#]*)</nowiki></code>|(<code>text/html</code>) / (<code>application/xhtml.xml</code>)|<code>406</code> (Not Acceptable) or <code>303 Redirect</code> to location of resource in requested representation|<code>Vary: negotiate, accept Alternates: {"ALFKI" 0.9 {type application/rdf+xml}}</code>|Last (must be last in processing chain)|

The source URI patterns refer to virtual or physical directories at <code>[[http://demo.openlinksw.com/][http://demo.openlinksw.com/]]</code>. Rules can be 
placed at the head or tail of the pipeline, or applied in the order they are declared, by specifying a Rule 
Processing Order of <code>First</code>, <code>Last</code>, or <code>Normal</code>, respectively. The decision as to which representation to return 
for URI <code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code> is based on the MIME
type(s) specified in any <code>Accept</code> header accompanying the request.

In the case of the last rule, the <code>Alternates</code> response header applies only to response code <code>406</code>. <code>406</code> would be 
returned if there were no (X)HTML representation available for the requested resource. In the example shown, 
an alternative representation is available in RDF/XML.

When applied to matching HTTP requests, the last two rules might generate responses similar to those below:

<verbatim>

$ curl -I -H "Accept: application/rdf+xml" http://demo.openlinksw.com/Northwind/Customer/ALFKI

HTTP/1.1 303 See Other
Server: Virtuoso/05.00.3016 (Solaris) x86_64-sun-solaris2.10-64 PHP5
Connection: close
Content-Type: text/html; charset=ISO-8859-1
Date: Mon, 16 Jul 2007 22:40:03 GMT
Accept-Ranges: bytes
Location: /sparql?query=CONSTRUCT+{+%3Chttp%3A//demo.openlinksw.com/Northwind/Custom
er/ALFKI%23this%3E+%3Fp+%3Fo+}+FROM+%3Chttp%3A//demo.openlinksw.com/Northwind%3E+WHE
RE+{+%3Chttp%3A//demo.openlinksw.com/Northwind/Customer/ALFKI%23this%3E+%3Fp+%3Fo+}&
format=application/rdf%2Bxml
Content-Length: 0 
</verbatim>

In the cURL exchange depicted above, the target Virtuoso server redirects to a SPARQL endpoint that retrieves 
an RDF/XML representation of the requested entity.

<verbatim>
$ curl -I -H "Accept: text/html" http://demo.openlinksw.com/Northwind/Customer/ALFKI

HTTP/1.1 406 Not Acceptable
Server: Virtuoso/05.00.3016 (Solaris) x86_64-sun-solaris2.10-64 PHP5
Connection: close
Content-Type: text/html; charset=ISO-8859-1
Date: Mon, 16 Jul 2007 22:40:23 GMT
Accept-Ranges: bytes
Vary: negotiate,accept
Alternates: {"ALFKI" 0.9 {type application/rdf+xml}}
Content-Length: 0 
</verbatim>

In this second cURL exchange, the target Virtuoso server indicates that there is no resource to deliver in the 
requested representation. It provides hints in the form of an alternate resource representation and URI that 
may be appropriate, i.e., an RDF/XML representation of the requested entity.

---++ Deploying Linked Data using Virtuoso

The preceding sections described a generic approach to deploying linked data into the existing Web. We now turn 
our attention to Virtuoso, to describe its solution for linked data deployment.

In fact, Virtuoso's solution is to implement the generic approach outlined in the prior sections, using the twin 
pillars of Content Negotiation and URL rewriting.

---+++ The Virtuoso Rules-Based URL Rewriter

Virtuoso provides a URL rewriter that can be enabled for URLs matching specified patterns. Coupled with 
customizable HTTP response headers and response codes, Data-Web server administrators can configure highly 
flexible rules for driving content negotiation and URL rewriting. The key elements of the URL rewriter are:

   * Rewriting rule
   * Each rule describes how to parse a single source URL, and how to compose the URL of the page ultimately 
returned in the "<code>Location:</code>" response headers
   * Every rewriting rule is uniquely identified internally (using IRIs).
   * Two types of rule are supported, based on the syntax used to describe the source URL pattern matching: 
sprintf-based and regex-based.
   * Rewrite rules list
   * A named ordered list of rewrite rules or rule lists where rules of the list are processed from top to 
bottom or in line with processing pipeline precedence instructions
   * Configuration API
   * The rewriter configuration API defines functions for creating, dropping, and enumerating rules and 
rule lists.
   * Virtual hosts and virtual paths
   * URL rewriting is enabled by associating a rewrite rules list with a virtual directory

Each of these elements is described in more detail below, although complete descriptions of the features or 
functions in question are not given. The intention here is to provide an overview of Virtuoso's URL rewriting 
capabilities and their application to deploying linked data. Please refer to the Virtuoso Reference 
Documentation for full details.

---+++ Conductor UI for the URL Rewriter

Virtuoso is a full-blown HTTP server in its own right. The HTTP server functionality co-exists with the 
product core (i.e., DBMS Engine, Web Services Platform, WebDAV filesystem, and other components of the 
Universal Server). As a result, it has the ability to multi-home Web domains within a single instance 
across a variety of domain name and port combinations. In addition, it also enables the creation of 
multiple virtual directories per domain.

In addition to the basic functionality describe above, Virtuoso facilitates the association of URL Rewriting 
rules with the virtual directories associated with a hosted Web domain. 

In all cases, Virtuoso enables you to configure virtual domains, virtual directories and URL rewrite rules 
for one or more virtual directories, via the (X)HTML-based Conductor Admin User Interface or a collection 
of Virtuoso Stored Procedure Language (PL)-based APIs.

---+++ Virtual Domains (Hosts) & Directories

A Virtuoso virtual directory maps a logical path to a physical directory that is file system or WebDAV based. 
This mechanism allows physical locations to be hidden or simply reorganized. Virtual directory definitions 
are held in the system table <code><nowiki>DB.DBA.HTTP_PATH</nowiki></code>. Virtual directories can be administered in 
three basic ways:

   * Using the Visual Administration Interface via a Web browser;
   * Using the functions <code><nowiki>vhost_define</nowiki>()</code> and <code><nowiki>vhost_remove</nowiki>()</code>; and
   * Using SQL statements to directly update the <code><nowiki>HTTP_PATH</nowiki></code> system table.

---+++ "Nice" URLs vs. "Long" URLs

Although we are approaching the URL Rewriter from the perspective of deploying linked data, the Rewriter was 
developed with additional objectives in mind. These in turn have influenced the naming of some of the formal 
argument names in the Configuration API function prototypes. In the following sections, long URLs are those 
containing a query string with named parameters; nice (aka. source) URLs have data encoded in some other 
format. The primary goal of the Rewriter is to accept a nice URL from an application and convert this into 
a long URL, which then identifies the page that should actually be retrieved.

---+++ Rule Processing Mechanics

When an HTTP request is accepted by the Virtuoso HTTP server, the received nice URL is passed to an internal 
path translation function. This function takes the nice URL and, if the current virtual directory has a 
url_rewrite option set to an existing ruleset name, tries to match the corresponding rulesets and rules; 
that is, it performs a recursive traversal of any rule-list associated with it. For every rule in the rule-list, 
the same logic is applied (only the logic for regex-based rules is described; that for sprintf-based rules is 
very similar):

   * The input for the rule is the resource URL as received from the HTTP header, i.e., the portion of the URL 
from the first solidus ('<code>/</code>') after the <code>host:port</code> fields to the end of the URL.
   * The input is normalized.
   * The input is matched against the rule's regex. If the match fails, the rule is not applied and the next 
rule is tried. If the match succeeds, the result is a vector of values.
   * If the URL contains a query string, the names and values of the parameters are decoded by 
<code><nowiki>split_and_decode</nowiki>()</code>.
   * The names and values of any parameters in the request body are also decoded.
   * The destination URL is composed:
   * The value of each parameter in the destination URL is taken from (in order of priority):
      * the value of a parameter in the match result;
      * the value of a named parameter in the query string of the input nice URL;
      * if the original request was submitted by the <code>POST</code> method, the value of a named parameter in the body of 
the <code>POST</code> request; or
      * if a parameter value cannot be derived from one of these sources, the rule is not applied and the next 
rule is tried.

Note: 

The path translation function described above is internal to the Web server, so its signature is not appropriate 
for Virtuoso/PL calls and thus is not published. Virtuoso/PL developers can harness the same functionality using 
the <code>DB.DBA.<nowiki>URLREWRITE_APPLY</nowiki></code> API call.

---+++ Enabling URL Rewriting via the Virtuoso Conductor UI

The steps for configuring URL Rewrite rules via the Virtuoso Conductor are as follows:

   1. Assuming you are using the local demonstration database, load <code>[[http://localhost:8890/conductor][http://localhost:8890/conductor]]</code> into your 
browser, and then proceed through the Conductor as follows:
   1. Click the "<code><nowiki>WebDAV</nowiki> & HTTP</code>", and "<code>HTTP Hosts & Directories</code>" tabs
   1. Pick the domain that contains the virtual directories to which the rules are to be applied (in this case 
the default was taken)
   1. Click on the "<code>URL-rewrite</code>" link to create, delete, or edit a rule as shown below:
   1. Create a Rule for <code>HTML Representation Requests (via SPARQL SELECT Query)</code>
   1. Create a Rule for <code>RDF Representation Requests (via SPARQL CONSTRUCT Query)</code>
   1. Then save and exit the Conductor, and test your rules with <code>curl</code> or any other User Agent.
%BR%%BR%<img src="%ATTACHURLPATH%/conductor_url_rewrite_ui_1.png" style="wikiautogen"/>%BR%%BR%

---+++ Enabling URL Rewriting via Virtuoso PL

The <code><nowiki>vhost_define()</nowiki></code> API is used to define virtual hosts and virtual paths hosted by the Virtuoso 
HTTP server. URL rewriting is enabled through this function's <code>opts</code> parameter. <code>opts</code> is of type <code>ANY</code>, e.g., a vector 
of field-value pairs. Numerous fields are recognized for controlling different options. The field value 
<code><nowiki>url_rewrite</nowiki></code> controls URL rewriting. The corresponding field value is the IRI of a rule list 
to apply.

---++++ Configuration API

Virtuoso includes the following functions for managing URL rewriting rules and rule lists. The names are self-explanatory.

   * <code><nowiki>DB.DBA.URLREWRITE_DROP_RULE</nowiki></code> &mdash; Deletes a rewriting rule
   * <code><nowiki>DB.DBA.URLREWRITE_CREATE_SPRINTF_RULE</nowiki></code> &mdash; Creates a rewriting rule which uses sprintf-based pattern matching
   * <code><nowiki>DB.DBA.URLREWRITE_CREATE_REGEX_RULE</nowiki></code> &mdash; Creates a rewriting rule which uses regular expression (regex) based pattern matching
   * <code><nowiki>DB.DBA.URLREWRITE_DROP_RULELIST</nowiki></code> &mdash; Deletes a rewriting rule list
   * <code><nowiki>DB.DBA.URLREWRITE_CREATE_RULELIST</nowiki></code> &mdash; Creates a rewriting rule list
   * <code><nowiki>DB.DBA.URLREWRITE_ENUMERATE_RULES</nowiki></code> &mdash; Lists all the rules whose IRI match the specified 'SQL like' pattern
   * <code><nowiki>DB.DBA.URLREWRITE_ENUMERATE_RULELISTS</nowiki></code> &mdash; Lists all the rule lists whose IRIs match the specified 'SQL like' pattern

---++++ Creating Rewriting Rules

Rewriting rules take two forms: sprintf-based or regex-based. When used for nice URL to long URL conversion, the 
only difference between them is the syntax of format strings. The reverse long to nice conversion works only for 
sprintf-based rules, whereas regex-based rules are unidirectional. 

For the purposes of describing how to make dereferenceable URIs for linked data, we will stick with the nice to 
long conversion using regex-based rules.

Regex rules are created using the <code><nowiki>URLREWRITE_CREATE_REGEX_RULE</nowiki>()</code> function.

---+++++ Function Prototype:

<verbatim>
URLREWRITE_CREATE_REGEX_RULE (
rule_iri,
allow_update,
nice_match,
nice_params,
nice_min_params,
target_compose,
target_params,
target_expn := null,
accept_pattern := null,
do_not_continue := 0,
http_redirect_code := null
);
</verbatim>

---+++++ Parameters:

   * <code><nowiki>rule_iri</nowiki></code>: VARCHAR.  The rule's name / identifier
   * <code><nowiki>allow_update</nowiki></code>: INTEGER. Indicates whether the rule can be updated. <code>Non-zero</code> indicates yes; 
<code>0</code> indicates no. The update is subject to the following rules:
      * If the given <code><nowiki>rule_iri</nowiki></code> is already in use as a rule list identifier, an error is signaled.
      * If the given <code><nowiki>rule_iri</nowiki></code> is already in use as a rule identifier and <code><nowiki>allow_update</nowiki></code> 
for the existing rule is <code>0</code>, an error is signaled.
      * If the given <code><nowiki>rule_iri</nowiki></code> is already in use as a rule identifier and <code><nowiki>allow_update</nowiki></code> 
for the existing rule is non-zero, the existing rule is updated.
   * <code><nowiki>nice_match</nowiki></code>: VARCHAR. A regex match expression to parse the URL into a vector of occurrences.
   * <code><nowiki>nice_params</nowiki></code>: ANY. A vector of the names of the parsed parameters. The length of the vector should 
be equal to the number of '(...)' specifiers in the format string.
   * <code><nowiki>nice_min_params</nowiki></code>: INTEGER. Used to specify the minimum number of sprintf format patterns to be matched in order to trigger the given rule. It 
only affects sprintf rules and has no effect for regex rules.
   * <code><nowiki>target_compose</nowiki></code>: VARCHAR. A regex compose expression for the URL of the destination page.
   * <code><nowiki>target_params</nowiki></code>: ANY. A vector of names of parameters that should be passed to the compose expression (<code><nowiki>target_compose</nowiki></code>) as <code>$1</code>, <code>$2</code> and so on.
   * <code><nowiki>target_expn</nowiki></code>: VARCHAR. Optional SQL text that should be executed instead of a regex <code>compose</code> call.
   * <code><nowiki>accept_pattern</nowiki></code>: VARCHAR. A regex expression to match the HTTP <code>Accept</code> header
   * <code><nowiki>do_not_continue</nowiki></code>: INTEGER. If the given rule satisfies the match conditions, <code>1</code> signifies do not try the next rule from same rule list, and 
<code>0</code> signifies try the next rule.
   * <code><nowiki>http_redirect_code</nowiki></code>: INTEGER. <code>NULL</code> or the integer values <code>301</code>, <code>302</code>, <code>303</code>, or <code>406</code>, are currently allowed. If a <code>3xx</code> redirect code is given, an HTTP 
<code>redirect</code> response will be sent back to client. If <code>NULL</code> is specified, the server will process the redirect internally.

---+++ Example - URL Rewriting For the Northwind Linked Data View

In our Linked Data Views of SQL white paper we covered the process of declaring Linked Data Views of SQL data via the Virtuoso 
Meta-schema Language. When producing the Linked Data Views we used the Virtuoso "Demo" database, which is very similar 
to the "Northwind" database that comes as an installation bundle with Microsoft ACCESS and SQL Server.

The Northwind schema is comprised of commonly understood SQL Tables including <code>Customers</code>, <code>Orders</code>, <code>Employees</code>, 
<code>Products</code>, <code>Product Categories</code>, <code>Shippers</code>, <code>Countries</code>, <code>Provinces</code>, etc.

An Linked Data View of SQL data is an RDF Named Graph (RDF data set) comprised of RDF Linked Data (triples) stored in 
a Virtuoso Quad Store (the native RDF Data Management realm of Virtuoso).

In the example that follows, we are going interact with Linked Data deployed into the Data-Web from a live instance 
of Virtuoso, which uses the URL Rewrite rules from the prior section.

The components used in the example are as follows:

   1. Virtuoso SPARQL Endpoint: <code>[[http://demo.openlinksw.com/sparql][http://demo.openlinksw.com/sparql]]</code>
   1. Named RDF Graph: <code>[[http://demo.openlinksw.com/Northwind][http://demo.openlinksw.com/Northwind]]</code>
   1. Entity ID - <code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI#this][http://demo.openlinksw.com/Northwind/Customer/ALFKI#this]]</code>
   1. Information Resource: <code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code>
   1. Interactive SPARQL Query Builder (iSPARQL) - <code>[[http://demo.openlinksw.com/DAV/JS/isparql/index.html][http://demo.openlinksw.com/DAV/JS/isparql/index.html]]</code>

---++++ Interacting with Linked Data via RDF Browser

Steps:

   1. Start the browser - <code>[[http://demo.openlinksw.com/DAV/JS/rdfbrowser/index.html][http://demo.openlinksw.com/DAV/JS/rdfbrowser/index.html]]</code>
   1. Enter the Information Resource URI, 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code>, 
into the input field labeled "<code>URI</code>"
%BR%%BR%<img src="%ATTACHURLPATH%/rdf_browser2.png" style="wikiautogen"/>%BR%%BR%
   1. Click on the "<code>Query</code>" button or simply hit "<code>Enter</code>" after typing (or pasting in) the Information Resource URI
   1. For the purpose of this exercise, view the data returned via the "Navigator" Viewer
%BR%%BR%<img src="%ATTACHURLPATH%/rdf_browser3.png" style="wikiautogen"/>%BR%%BR%
   1. Click on the "<code>Raw Triples</code>" viewer tab and observe the exposure of the Entity 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI#this][http://demo.openlinksw.com/Northwind/Customer/ALFKI#this]]</code> 
via the Triple based (<code>Subject, Predicate, Object</code>) records in the results table.
%BR%%BR%<img src="%ATTACHURLPATH%/rdf_browser4.png" style="wikiautogen"/>%BR%%BR%

---++++ Interacting with Linked Data via iSPARQL

We can interact with the same Information Resource and associated RDF using the iSPARQL Query tool as follows:

   1. Start the Query Builder by entering the following into your browser: 
<code>[[http://demo.openlinksw.com/isparql][http://demo.openlinksw.com/isparql]]</code> 
You will be presented with a default Query By Example (QBE) canvas that includes a default Graph Pattern and 
a default URI. Change the URI to: 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code> 
(Information Resource as a Data Resource in the context of RDF)
%BR%%BR%<img src="%ATTACHURLPATH%/isparql_qbe_1.png" style="wikiautogen"/>%BR%%BR%
   1. Then execute the default query (which simply gets a list of concepts), by clicking on the "<code>&gt;</code>" button. 
Note: There is a single record in the result table. It indicates that there is a single concept, Organization, 
as defined by the FOAF schema.
%BR%%BR%<img src="%ATTACHURLPATH%/isparql_qbe_2.png" style="wikiautogen"/>%BR%%BR%
   1. Click on the <code>foaf:Organization</code> record, and you will be presented with a Data Web-optimized hyperlink that 
presents you with three options: <code>Dereference</code>, <code>Explore</code>, and <code>(X)HTML Page Open</code>.
%BR%%BR%<img src="%ATTACHURLPATH%/isparql_qbe_3.png" style="wikiautogen"/>%BR%%BR%
   1. Click <code>Explore</code> (since you are interested in "instance data" for the <code>foaf:Organization</code> concept, as opposed 
to the schema definitions of said concept). You will be presented with 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI#this][http://demo.openlinksw.com/Northwind/Customer/ALFKI#this]]</code> 
which is an RDF Entity ID of a <code>foaf:Organization</code> instance.
%BR%%BR%<img src="%ATTACHURLPATH%/isparql_qbe_4.png" style="wikiautogen"/>%BR%%BR%
   1. Click on the 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code> 
record, and you will once again be presented with the enhanced hyperlink and its options. This time, click 
<code>Dereference</code>, since you are interested in the description of the entity 
<code>[[http://demo.openlinksw.com/Northwind/Customer/ALFKI][http://demo.openlinksw.com/Northwind/Customer/ALFKI]]</code>, 
as opposed to all the records in the RDF database that are related to it.
%BR%%BR%<img src="%ATTACHURLPATH%/isparql_qbe_5.png" style="wikiautogen"/>%BR%%BR%
%BR%%BR%<img src="%ATTACHURLPATH%/isparql_qbe_6.png" style="wikiautogen"/>%BR%%BR%


---++++ Interacting with Linked Data via a standard Document Web Browser

In the prior sections, we used the OpenLink RDF Browser and iSPARQL Query-By-Example tools to interact with 
RDF Entities via associated Information Resources. Each of these tools includes a Resource Save feature that 
enables you to save an RDF Browser session or an iSPARQL Query for future reuse. In either scenario the 
end-product is a Dynamic Linked Data Page &mdash; a Web Information Resource (document) that includes links to RDF 
based Linked Data.

---+++++ Saved Browser Session

Steps:

   1. From your RDF Browser session, go to the <code>Session</code> >> <code>Save</code> menu item.
%BR%%BR%<img src="%ATTACHURLPATH%/rdf_browser5.png" style="wikiautogen"/>%BR%%BR%
   1. Select a directory location (note: this is a WebDAV location in the Virtuoso Server) and then enter 
a file name, e.g., <code><nowiki>ALFKI_Linked_Datay</nowiki></code>. The saved file will automatically be assigned the 
extension <code>.wqx</code>.
%BR%%BR%<img src="%ATTACHURLPATH%/rdf_browser6.png" style="wikiautogen"/>%BR%%BR%
   1. Open a standard browser instance on your internet device (desktop, notebook, phone, etc.), and enter 
the URL for the location into which you just saved your browser session, e.g., 
<code>[[http://demo.openlinksw.com/DAV/home/demo/Public/Queries/SQLRDFIntegraton/][http://demo.openlinksw.com/DAV/home/demo/Public/Queries/SQLRDFIntegraton/]]</code> 
(based on our example).
%BR%%BR%<img src="%ATTACHURLPATH%/saving_to_webdav_1.png" style="wikiautogen"/>%BR%%BR%
   1. Click on the file <code><nowiki>ALFKI_Linked_Data</nowiki>.wqx</code>, which will then reveal a browser session-oriented 
Linked Data page.
%BR%%BR%<img src="%ATTACHURLPATH%/saving_to_webdav_3.png" style="wikiautogen"/>%BR%%BR%
%BR%%BR%<img src="%ATTACHURLPATH%/rdf_browser8.png" style="wikiautogen"/>%BR%%BR%

---+++++ Saved iSPARQL Query

Steps:

   1. From your iSPARQL Session, pick the <code>File</code> >> <code>Save</code> (if first time) or <code>File</code> >> <code>Save As</code> (for saving to different name)
%BR%%BR%<img src="%ATTACHURLPATH%/isparql_qbe_7.png" style="wikiautogen"/>%BR%%BR%
   1. Type in a name for your saved query, e.g., <code><nowiki>ALFKI_Linked_Data</nowiki></code>. Note that you have a number of file 
type options. For this exercise, we are going to choose the <code>.isparql</code> type, since we are attempting to create a Dynamic 
Linked Data page.
%BR%%BR%<img src="%ATTACHURLPATH%/isparql_qbe_save_1.png" style="wikiautogen"/>%BR%%BR%
   1. Open a standard browser instance on your internet device (desktop, notebook, phone, etc.), and enter the URL for 
the location into which you save your browser session, e.g., 
<code>[[http://demo.openlinksw.com/DAV/home/demo/Public/Queries/SQLRDFIntegraton/][http://demo.openlinksw.com/DAV/home/demo/Public/Queries/SQLRDFIntegraton/]]</code> 
(based on our example).
%BR%%BR%<img src="%ATTACHURLPATH%/saving_to_webdav_2.png" style="wikiautogen"/>%BR%%BR%
   1. Click on the file <code><nowiki>ALFKI_Linked_Data</nowiki>.isparql</code>, and then interact with the Linked Data page.
%BR%%BR%<img src="%ATTACHURLPATH%/isparql_linked_data_page1.png" style="wikiautogen"/>%BR%%BR%

---++++ Northwind URL Rewriting Verification Using curl

As illustrated earlier, the <code>curl</code> utility provides a useful tool for verifying HTTP server responses and rewriting rules. 
The curl exchanges below show the URL rewriting rules defined for the Northwind Linked Data View being applied.

---+++++Example 1

<verbatim>
$ curl -I -H "Accept: text/html" http://demo.openlinksw.com/Northwind/Customer/ALFKI 

HTTP/1.1 303 See Other 
Server: Virtuoso/05.00.3016 (Solaris) x86_64-sun-solaris2.10-64 PHP5 
Connection: close 
Content-Type: text/html; charset=ISO-8859-1
Date: Tue, 14 Aug 2007 13:30:02 GMT 
Accept-Ranges: bytes 
Location:  /isparql/execute.html?query=SELECT%20%3Fp%20%3Fo%20FROM%20%3Chttp%3A//dem
o.openlinksw.com/Northwind%3E%20WHERE%20{%20%3Chttp%3A//demo.openlinksw.com/Northwin
d/Customer/ALFKI%23this%3E%20%3Fp%20%3Fo%20}&endpoint=/sparql
Content-Length: 0 
</verbatim>

---+++++Example 2

<verbatim>
$ curl -I -H "Accept: application/rdf+xml" http://demo.openlinksw.com/Northwind/Cust
omer/ALFKI

HTTP/1.1 303 See Other
Server: Virtuoso/05.00.3016 (Solaris) x86_64-sun-solaris2.10-64 PHP5 
Connection: close
Content-Type: text/html; charset=ISO-8859-1
Date: Tue, 14 Aug 2007 13:30:22 GMT
Accept-Ranges: bytes 
Location: /sparql?query=CONSTRUCT+{+%3Chttp%3A//demo.openlinksw.com/Northwind/Custom
er/ALFKI%23this%3E+%3Fp+%3Fo+}+FROM+%3Chttp%3A//demo.openlinksw.com/Northwind%3E+WHE
RE+{+%3Chttp%3A//demo.openlinksw.com/Northwind/Customer/ALFKI%23this%3E+%3Fp+%3Fo+}&
format=application/rdf%2Bxml
Content-Length: 0 
</verbatim>

---+++++Example 3

<verbatim>
$ curl -I -H "Accept: text/html" http://demo.openlinksw.com/Northwind/Customer/ALFKI
#this 

HTTP/1.1 404 Not Found 
Server: Virtuoso/05.00.3016 (Solaris) x86_64-sun-solaris2.10-64 PHP5 
Connection: Keep-Alive
Content-Type: text/html; charset=ISO-8859-1 
Date: Tue, 14 Aug 2007 13:31:01 GMT 
Accept-Ranges: bytes 
Content-Length: 0 
</verbatim>

The output above shows how RDF entities from the Data-Web, in this case customer ALFKI, are exposed in the Document Web. 
The power of SPARQL coupled with URL rewriting enables us to produce results in line with the desired representation. 
A SPARQL SELECT or CONSTRUCT query is used depending on whether the requested representation is <code>text/html</code> or 
<code>application/rdf+xml</code>, respectively.

The <code>404</code> response in Example 3 indicates that no HTML representation is available for entity <code>ALFKI#this</code>. In most cases, 
a URI of this form (containing a '<code>#</code>' fragment identifier) will not reach the server. This example supposes that it does, 
i.e., the RDF client and network routing allows the suffixed request. The presence of the <code>#this</code> suffix implicitly states 
#BodyNote2
that this is a request for a data resource in the Data-Web realm, not a document resource from the Document Web. [[#FootNote2][Note 2]]

Rather than return <code>404</code>, we could instead choose to construct our rewriting rules to perform a <code>303</code> redirect, so that the 
response for <code>ALFKI#this</code> in Example 3 becomes the same as that for <code>ALFKI</code> in Example 1.

---+++ Transparent Content Negotiation

So as not to overload our preceding description of Linked Data deployment with excessive detail, the description of content 
negotiation presented thus far was kept deliberately brief. This section discusses content negotiation in more detail.

---++++ HTTP/1.1 Content Negotiation

Recall that a resource (conceptual entity) identified by a URI may be associated with more than one representation (e.g., 
multiple languages, data formats, sizes, resolutions). If multiple representations are available, the resource is referred 
to as negotiable and each of its representations is termed a variant. For instance, a Web document resource, named '<code>ALFKI</code>' 
may have three variants: <code>alfki.xml</code>, <code>alfki.html</code>, and <code>alfki.txt</code>, all representing the same data. Content negotiation provides 
a mechanism for selecting the best variant.

As outlined in the earlier brief discussion of content negotiation, when a user agent requests a resource, it can include 
with the request <code>Accept</code> headers (<code>Accept</code>, <code>Accept-Language</code>, <code>Accept-Charset</code>, <code>Accept-Encoding</code>, etc.) which express the user 
preferences and user agent capabilities. The server then chooses and returns the best variant based on the <code>Accept</code> headers. 
Because the selection of the best resource representation is made by the server, this scheme is classed as server-driven 
negotiation.

---++++ Transparent Content Negotiation

An alternative content negotiation mechanism is Transparent Content Negotiation (TCN), a protocol defined by RFC2295. TCN 
offers a number of benefits over standard HTTP/1.1 negotiation, for suitably enabled user agents.

RFC2295 introduces a number of new HTTP headers including the <code>Negotiate</code> request header, and the <code>TCN</code> and <code>Alternates</code> response 
headers. (Krishnamurthy et al note that although the HTTP/1.1 specification reserved the <code>Alternates</code> header for use in agent-
driven negotiation, it was not fully specified. Consequently under a pure HTTP/1.1 implementation as defined by RFC2616, 
server-driven content negotiation is the only option. RFC2295 addresses this issue.)

---+++++ Deficiencies of HTTP/1.1 Server-Driven Negotiation

Weaknesses of server-driven negotiation highlighted by RFCs 2295 and 2616 include:

   * Inefficiency &mdash; Sending details of a user agent's capabilities and preferences with every request is very inefficient, 
not least because very few Web resources have multiple variants, and expensive in terms of the number of <code>Accept</code> headers 
required to fully describe all but the most simple browser's capabilities.
   * Server doesn't always know 'best' &mdash; Having the server decide on the 'best' variant may not always result in the most 
suitable resource representation being returned to the client. The user agent might often be better placed to decide what 
is best for its needs.

---+++++ Variant Selection By User Agent

Rather than rely on server-driven negotiation and variant selection by the server, a user agent can take full control over 
deciding the best variant by explicitly requesting transparent content negotiation through the <code>Negotiate</code> request header. 
The negotiation is 'transparent' because it makes all the variants on the server visible to the agent.

Under this scheme, the server sends the user agent a list, represented in an <code>Alternates</code> header, containing the available 
variants and their properties. The user agent can then choose the best variant itself. Consequently, the agent no longer 
needs to send large <code>Accept</code> headers describing in detail its capabilities and preferences. (However, unless caching is used, 
user-agent driven negotiation does suffer from the disadvantage of needing a second request to obtain the best representation. 
By sending its best guess as the first response, server driven negotiation avoids this second request if the initial best 
guess is acceptable.)

---+++++ Variant Selection By Server

As well as variant selection by the user agent, TCN allows the server to choose on behalf of the user agent if the user agent 
explicitly allows it through the <code>Negotiate</code> request header. This option allows the user agent to send smaller <code>Accept</code> headers 
containing enough information to allow the server to choose the best variant and return it directly. The server's choice is 
controlled by a 'remote variant selection algorithm' as defined in RFC2296.

---+++++ Variant Selection By End-User

A further option is to allow the end-user to select a variant, in case the choice made by negotiation process is not optimal. 
For instance, the user agent could display an HTML-based 'pick list' of variants constructed from the variant list returned 
by the server. Alternatively the server could generate this pick list itself and include it in the response to a user agent's 
request for a variant list. (Virtuoso currently responds this way.)

---+++ Transparent Content Negotiation in Virtuoso HTTP Server

The following section describes the Virtuoso HTTP server's TCN implementation which is based on RFC2295, but without "<code>Feature</code>"
negotiation. OpenLink's RDF rich clients, iSparql and the OpenLink RDF Browser, both support TCN. User agents which do not 
support transparent content negotiation continue to be handled using HTTP/1.1 style content negotiation (whereby server-side 
selection is the only option - the server selects the best variant and returns a list of variants in an <code>Alternates</code> response 
header).

---++++ Describing Resource Variants

In order to negotiate a resource, the server needs to be given information about each of the variants. Variant descriptions 
are held in SQL table <code><nowiki>HTTP_VARIANT_MAP</nowiki></code>. The descriptions themselves can be created, updated or deleted using 
Virtuoso/PL or through the Conductor UI.

---+++++ <code><nowiki>HTTP_VARIANT_MAP</nowiki></code> Table Definition

The table definition is as follows:

<verbatim>
create table DB.DBA.HTTP_VARIANT_MAP (
  VM_ID integer identity, -- unique ID 
  VM_RULELIST varchar, -- HTTP rule list name 
  VM_URI varchar, -- name of requested resource e.g. 'page' 
  VM_VARIANT_URI varchar, -- name of variant e.g. 'page.xml', 'page.de.html' etc. 
  VM_QS float, -- Source quality, a number in the range 0.001-1.000, with 3 digit precision 
  VM_TYPE varchar, -- Content type of the variant e.g. text/xml 
  VM_LANG varchar, -- Content language e.g. 'en', 'de' etc. 
  VM_ENC varchar, -- Content encoding e.g. 'utf-8', 'ISO-8892' etc. 
  VM_DESCRIPTION long varchar, -- a human readable description about the variant e.g. 'Profile in RDF format' 
  VM_ALGO int default 0, -- reserved for future use 
  primary key (VM_RULELIST, VM_URI, VM_VARIANT_URI)
 ) 
create unique index HTTP_VARIANT_MAP_ID on DB.DBA.HTTP_VARIANT_MAP (VM_ID)
</verbatim>


---+++++ Configuration using Virtuoso/PL

Two functions are provided for adding or updating, or removing variant descriptions using Virtuoso/PL:

---++++++ Adding or Updating a Resource Variant:

<verbatim>
DB.DBA.HTTP_VARIANT_ADD ( 
  in rulelist_uri varchar, -- HTTP rule list name 
  in uri varchar, -- Requested resource name e.g. 'page' 
  in variant_uri varchar, -- Variant name e.g. 'page.xml', 'page.de.html' etc. 
  in mime varchar, -- Content type of the variant e.g. text/xml 
  in qs float := 1.0, -- Source quality, a floating point number with 3 digit precision in 0.001-1.000 range 
  in description varchar := null, -- a human readable description of the variant e.g. 'Profile in RDF format' 
  in lang varchar := null, -- Content language e.g. 'en', 'bg'. 'de' etc. 
  in enc varchar := null -- Content encoding e.g. 'utf-8', 'ISO-8892' etc. 
)
</verbatim>

---++++++Removing a Resource Variant

<verbatim>
DB.DBA.HTTP_VARIANT_REMOVE ( 
  in rulelist_uri varchar, -- HTTP rule list name 
  in uri varchar, -- Name of requested resource e.g. 'page' 
  in variant_uri varchar := '%' -- Variant name filter 
)
</verbatim>

---+++++ Configuration using Conductor UI

The Conductor '<code>Content negotiation</code>' panel for describing resource variants and configuring content negotiation is 
depicted below. It can be reached by selecting the '<code>HTTP Hosts & Directories</code>' tab under the '<code><nowiki>WebDAV</nowiki> & HTTP</code>' menu 
item, then selecting the '<code>URL rewrite</code>' option for a logical path listed amongst those for the relevant HTTP host, 
e.g., '<code>{Default Web Site}</code>'.

The screen snapshot shows the variant descriptions created by issuing the <code><nowiki>HTTP_VARIANT_ADD</nowiki></code> and 
<code><nowiki>VHOST_DEFINE</nowiki></code> Virtuoso/PL calls detailed in the examples at the end of this section. Obviously 
these definitions could instead have been created entirely 'from scratch' through the Conductor UI.

The input fields reflect the supported 'dimensions' of negotiation which include content type, language and encoding. 
Quality values corresponding to the options for 'Source Quality' are as follows:

|*Source Quality*|*Quality Value*|
|perfect representation|1.000|
|threshold of noticeable loss of quality|0.900|
|noticeable, but acceptable quality reduction|0.800|
|barely acceptable quality|0.500|
|severely degraded quality|0.300|
|completely degraded quality|0.000|

---++++ Variant Selection Algorithm

When a user agent instructs the server to select the best variant, Virtuoso does so using the selection algorithm 
below:

If a virtual directory has URL rewriting enabled (has the '<code>url_rewrite</code>' option set), the web server:

   * Looks in <code><nowiki>DB.DBA.HTTP_VARIANT_MAP</nowiki></code> for a <code><nowiki>VM_RULELIST</nowiki></code> matching the one specified 
in the '<code><nowiki>url_rewrite</nowiki></code>' option

   * If present, it loops over all variants for which <code><nowiki>VM_URI</nowiki></code> is equal to the resource requested

   * For every variant it calculates the source quality based on the value of <code><nowiki>VM_QS</nowiki></code> and the source 
quality given by the user agent

   * If the best variant is found, it adds TCN HTTP headers to the response and passes the 
<code><nowiki>VM_VARIANT_URI</nowiki></code> to the URL rewriter

   * If the user agent has asked for a variant list, it composes such a list and returns an '<code>Alternates</code>' HTTP header 
with response code <code>300</code>

   * If no URL rewriter rules exist for the target URL, the web server returns the content of the dereferenced 
<code><nowiki>VM_VARIANT_URI</nowiki></code>.

The server may return the best-choice resource representation or a list of available resource variants. When a user 
agent requests transparent negotiation, the web server returns the TCN header "<code>choice</code>". When a user agent asks for 
a variant list, the server returns the TCN header "<code>list</code>".

---++++ Examples

In this example we assume the following files have been uploaded to the Virtuoso WebDAV server, with each containing 
the same information but in different formats:

   * <code><nowiki>/DAV/TCN/page.xml</nowiki></code> - a XML variant
   * <code><nowiki>/DAV/TCN/page.html</nowiki></code> - a HTML variant
   * <code><nowiki>/DAV/TCN/page.txt</nowiki></code> - a text variant

We add TCN rules and define a virtual directory:

<verbatim>
DB.DBA.HTTP_VARIANT_ADD ('http_rule_list_1', 'page', 'page.html', 'text/html', 
0.900000, 'HTML variant'); DB.DBA.HTTP_VARIANT_ADD ('http_rule_list_1', 'page', 
'page.txt', 'text/plain', 0.500000, 'Text document'); DB.DBA.HTTP_VARIANT_ADD 
('http_rule_list_1', 'page', 'page.xml', 'text/xml', 1.000000, 'XML variant'); 
DB.DBA.VHOST_DEFINE (lpath=>'/DAV/TCN/', ppath=>'/DAV/TCN/', is_dav=>1, 
vsp_user=>'dba', opts=>vector ('url_rewrite', 'http_rule_list_1'));
</verbatim>

Having done this we can now test the setup with a suitable HTTP client, in this case the <code>curl</code> command line utility. In 
the following examples, the curl client supplies <code>Negotiate</code> request headers containing content negotiation directives 
which include:

   * "<code>trans</code>" &mdash; The user agent supports transparent content negotiation for the current request.
   * "<code>vlist</code>" &mdash; The user agent requests that any transparently negotiated response for the current request includes an 
Alternates header with the variant list bound to the negotiable resource. Implies "<code>trans</code>".
   * "<code>*</code>" &mdash; The user agent allows servers and proxies to run any remote variant selection algorithm.

The server returns a TCN response header signaling that the resource is transparently negotiated and either a choice 
or a list response as appropriate.

In the first curl exchange, the user agent indicates to the server that, of the formats it recognizes, HTML is preferred 
and it instructs the server to perform transparent content negotiation. In the response, the <code>Vary</code> header field expresses 
the parameters the server used to select a representation, i.e., only the <code>Negotiate</code> and <code>Accept</code> header fields are considered.

<verbatim>
$ curl -i -H "Accept: text/xml;q=0.3,text/html;q=1.0,text/plain;q=0.5,*/*;q=0.3" -H 
"Negotiate: *" http://localhost:8890/DAV/TCN/page 

HTTP/1.1 200 OK 
Server: Virtuoso/05.00.3021 (Linux) i686-pc-linux-gnu VDB 
Connection: Keep-Alive 
Date: Wed, 31 Oct 2007 15:43:18 GMT 
Accept-Ranges: bytes 
TCN: choice 
Vary: negotiate,accept 
Content-Location: page.html 
Content-Type: text/html 
ETag: "14056a25c066a6e0a6e65889754a0602" 
Content-Length: 49

<html> <body> some html </body> </html>
</verbatim>


Next, the source quality values are adjusted so that the user agent indicates that XML is its preferred format.

<verbatim>
$ curl -i -H "Accept: text/xml,text/html;q=0.7,text/plain;q=0.5,*/*;q=0.3" -H "Negot
iate: *" http://localhost:8890/DAV/TCN/page 

HTTP/1.1 200 OK 
Server: Virtuoso/05.00.3021 (Linux) i686-pc-linux-gnu VDB 
Connection: Keep-Alive 
Date: Wed, 31 Oct 2007 15:44:07 GMT 
Accept-Ranges: bytes 
TCN: choice 
Vary: negotiate,accept 
Content-Location: page.xml 
Content-Type: text/xml 
ETag: "8b09f4b8e358fcb7fd1f0f8fa918973a" 
Content-Length: 39

<?xml version="1.0" ?> <a>some xml</a>
</verbatim>


In the final example, the user agent wants to decide itself which is the most suitable representation, so it asks 
for a list of variants. The server provides the list, in the form of an <code>Alternates</code> response header, and, in addition, 
sends an HTML representation of the list so that the end user can decide on the preferred variant himself if the user 
agent is unable to.

<verbatim>
$ curl -i -H "Accept: text/xml,text/html;q=0.7,text/plain;q=0.5,*/*;q=0.3" -H "Negot
iate: vlist" http://localhost:8890/DAV/TCN/page 

HTTP/1.1 300 Multiple Choices 
Server: Virtuoso/05.00.3021 (Linux) i686-pc-linux-gnu VDB 
Connection: close 
Content-Type: text/html; charset=ISO-8859-1 
Date: Wed, 31 Oct 2007 15:44:35 GMT 
Accept-Ranges: bytes 
TCN: list 
Vary: negotiate,accept 
Alternates: {"page.html" 0.900000 {type text/html}}, {"page.txt" 0.500000 {type text
/plain}}, {"page.xml" 1.000000 {type text/xml}} 
Content-Length: 368

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
<head>
<title>300 Multiple Choices</title>
</head>
<body>
<h1>Multiple Choices</h1>
Available variants: 
<ul>
<li>
<a href="page.html">HTML variant</a>, type text/html</li>
<li><a href="page.txt">Text document</a>, type text/plain</li>
<li><a href="page.xml">XML variant</a>, type text/xml</li>
</ul>
</body>
</html>
</verbatim>

---+++ Glossary

   * <b>class</b>: A concept in a domain of interest. A class describes the common attributes and behaviors shared 
by entities belonging to the same group by virtue of their common characteristics.
   * <b>content negotiation</b>: A mechanism defined in HTTP which supports serving different representations of a 
URL-addressable resource. An HTTP client can indicate which representation formats it understands and prefers.
   * <b><code>cURL</code></b>: A command line tool for transferring files to or from a URL. It writes to standard output by default 
and provides a good tool for simulating a web browser's interaction with an HTTP server. 
   * <b>data resource</b>: same as data source
   * <b>data source</b>: A source of data (e.g., a place that provides access to property values associated with one 
or more Entities).
   * <b>data space</b>: A moniker for Web-accessible atomic containers that manage and expose data, information, 
services, processes, and knowledge. Data Spaces are fundamentally problem-domain-specific database applications 
with the benefit of being data model and query language agnostic.
   * <b>dereferencing</b>: The act of accessing and retrieving data, in desired representation, from a location 
identified by URL.
   * <b>document resource</b>: A Web information resource in a specific representation that is identifiable and 
accessible via a URL. Documents are the dominant information resource form on the Document Web (i.e., the 
current Web).
   * <b>Document Web</b>: Web of Linked Documents.
   * <b>entity</b>: Something, real or conceptual, which exists apart from other things.
   * <b>entity ID</b>: A unique identifier for an entity, uniquely identifying and distinguishing a particular entity 
instance from other similar entities (typically of the same type or class).
   * <b>entity set</b>: A collection of entities all belonging to the same class.
   * <b>HTTP</b>: Hypertext Transport Protocol.
   * <b>HTTP header</b>: A text record exchanged between an HTTP client and server, which forms part of an HTTP request 
or HTTP response message. A request consists of a method (or verb), headers, and an optional message body. The 
request header fields allow the client to send additional information about the request and the client itself. 
A response consists of a status line, headers, and an optional message body. A response header typically contains 
information about the data being returned and about the server itself.
   * <b>Hypertext Transport Protocol)</b>: A communication protocol for information transfer on the World Wide Web.
   * <b>information resource</b>: An encapsulation of data and representation that forms the basic payload unit (packet) 
on the Web Information Bus.
   * <b>IRI (Internationalized Resource Identifier)</b>: An internationalized version of a Uniform Resource Identifier 
(URI). While URIs are limited to a subset of the ASCII character set, IRIs may contain any Unicode character.
   * <b>Linked Data</b>: Information resource(s) encapsulating Structured Data expressed in RDF.
   * <b>non-information resource</b>: Any resource that is not an information resource (i.e., not Web transportable in 
basic form). Structured data resource (see below) is a more accurate and preferable term.
   * <b>Semantic Data-Web</b>: Web of Linked Data.
   * <b>structured data</b>: Data organized into semantic chunks or entities, with similar entities grouped together in 
relations or classes, and presented in a patterned manner.
   * <b>structured data resource</b>: A Web accessible container of structured data representing physical and abstract 
entities.
   * <b>structured data source</b>: A repository of structured data.
   * <b>URI (Uniform Resource Identifier )</b>: An Internet naming syntax which identifies a resource. The resource may 
be abstract and consequently not dereferenceable.
   * <b>URL (Uniform Resource Locator)</b>: An Internet naming syntax, which specifies both the identity and location of 
a physical, dereferenceable Web information resource. Although URLs and URIs share the same syntax, a URI specifies 
only the identity of a resource.
   * <b>Web information resource</b>: An entity of interest, which both exists in some form and is accessible on the 
World Wide Web.


---++ Notes

#FootNote1
   1. Reiterating our earlier point, the URL identifies the resource, not its representations. [[#BodyNote1][return]]
#FootNote2
   2. Some Semantic Web (<nowiki>SemWeb</nowiki>) practitioners have argued in favor of using the URI format to 
distinguish between requests for document resources (also sometimes termed as information resources) belonging 
to the Document Web, and Data Sources, i.e., physical or abstract RDF entities (also sometimes referred to as 
non-information resources) belonging to the Data Web. (The terms information resource and non-information resource 
are disliked by many and have generated a good deal of debate.) With this so-called 'hash vs. slash' URI convention, 
the presence of a fragment identifier (a hash URI, using the "#") is taken to mean that a data source (entity) is 
being referenced; the absence of a fragment identifier (a slash URI, not using the "#") implies a document resource 
is being referenced. [[#BodyNote2][return]]

---++ Bibliography

   * OpenLink Software: Virtuoso Linked Data Views - Getting Started Guide
   * T. Berners-Lee: Linked Data
   * C. Bizer et al.: How to Publish Linked Data on the Web
   * L. Sauermann et al.: Cool URIs for the Semantic Web
   * P. Hayes: In Defence of Ambiguity
   * R. Cyganiak: Debugging Semantic Web Sites With cURL
   * W3C: Dereferencing HTTP URIs
   * W3C: What do HTTP URIs Identify?
   * W3C: What URIs Identify
   * W3C: [httpRange-14] Resolved
   * W3C: Best Practice Recipes for Publishing RDF Vocabularies
   * Wikipedia: Using HTTP URIs to identify abstract resources
   * B. Krishnamurthy et al.: Key Differences between HTTP/1.0 and HTTP/1.1
   * R. Fielding et al.: RFC2616 - <nowiki>HyperText</nowiki> Transfer Protocol - HTTP/1.1
   * K. Holtman & A. Mutz: RFC2295 - Transparent Content Negotiation in HTTP

---++ See Also 

   * [[VirtEvaluatorGuideLinkedDataDeployment][Evaluator Guide for Linked Data Deployment]]
   * [[VirtLinkedDataDeploymentTutorialDOAP][Linked Data Deployment DOAP Tutorial]]

---++ Change History

1.0 Initial draft (K. Idehen / C. Blakeley, 14 Aug 2007)

1.1 Additions covering transparent content negotiation (C. Blakeley, 07 Nov 2007)

1.2 ???

1.3 Edit and polish (T. Thibodeau, K. Idehen, etc., Nov 2008)