VirtSponger Virtuoso SpongerVirtuoso Sponger What Is The Sponger? The Virtuoso Sponger is the Linked Data middleware component of Virtuoso. It generates Linked Data from a variety of data sources, and supports a wide variety of data representation and serialization formats. The Sponger is transparently integrated into Virtuoso's SPARQL Query Processor where it delivers URI de-referencing within SPARQL query patterns, across disparate data spaces. It also delivers configurable smart HTTP caching services. Optionally, it can be used by the Virtuoso Content Crawler to periodically populate and replenish data within the native RDF Quad Store. The Sponger is also a full-fledged HTTP proxy service, directly accessible via SOAP or REST interfaces. As depicted below, OpenLink's broad portfolio of Linked-Data-aware products supports a number of routes for creating or consuming Linked Data. The Sponger provides a key platform for developers to generate quality data meshes from unstructured or semi-structured data sources.
Why is it Important? A majority of the worlds data naturally resides in non-Linked-Data form at the current time. The Sponger delivers middleware that accelerates the bootstrap of the Semantic Data Web by unobtrusively generating Linked Data (typically in RDF form, today) from non-Linked-Data data sources. This "Swiss army knife" for on-the-fly Linked Data generation provides a bridge between the traditional Document Web and the Linked Data Web ("Data Web"). Sponging non-Linked-Data Web sources and converting their data content to Linked Data exposes that data in a canonical form for querying and inference, and enables fast and easy construction of Linked-Data-driven "mesh-ups" (as opposed to code-driven Web 2.0 mash-ups). Linked Data extraction and instance data generation products that offer functionality similar to that demonstrated by the Sponger are also commonly referred to as "RDFizers." How Does It Work? Designed with a pluggable architecture, the Sponger's core functionality is provided by Cartridges. Each cartridge includes Data Extractors which extract data from one or more data sources, and Ontology Mappers which map the extracted data to one or more ontologies/schemas, en route to producing RDF Linked Data. Cartridges are highly customizable, and can be developed using any language supported by the Virtuoso Server Extensions API. This enables generation of structured linked data from virtually any resource type, rather than limiting users to resource types supported by the default Sponger Cartridge collection bundled as part of the Virtuoso Sponger VAD package (cartridges_dav.vad). UNKNOWN tag: http://www.w3.org/1999/xhtml:object (See an animation of the concept, if the embed above fails in your browser.) The Sponger also includes a pluggable name resolution mechanism that enables Custom Resolvers for naming schemes (e.g., URNs) associated with protocols beyond HTTP. Examples of custom resolvers include: <tgroup><thead /><tbody> <row /> <row><entry> DOI </entry><entry> doi:10.1038/35057062 </entry><entry> <ulink url="http://demo.openlinksw.com/about/html/doi:10.1038/35057062">HTML Representation</ulink> </entry><entry> <ulink url="http://demo.openlinksw.com/describe/?url=doi:10.1038/35057062">Linked Data View</ulink> </entry><entry> <ulink url="http://demo.openlinksw.com/ode/?uri=doi:10.1038/35057062">Data Explorer View</ulink> </entry><entry> hslookup plugin; and enabling of relevant mappers for html, pdf, xml, etc. </entry></row> <row><entry> LSID </entry><entry> urn:lsid:ubio.org:namebank:12292 </entry><entry> <ulink url="http://demo.openlinksw.com/about/html/urn:lsid:ubio.org:namebank:12292">HTML Representation</ulink> </entry><entry> <ulink url="http://demo.openlinksw.com/describe/?url=urn:lsid:ubio.org:namebank:12292">Linked Data View</ulink> </entry><entry> <ulink url="http://demo.openlinksw.com/ode/?uri=urn:lsid:ubio.org:namebank:12292">Data Explorer View</ulink> </entry><entry> None </entry></row> <row><entry> OAI </entry><entry> oai:dcmi.ischool.washington.edu:article/8 </entry><entry> <ulink url="http://demo.openlinksw.com/about/html/oai:dcmi.ischool.washington.edu:article/8">HTML Representation</ulink> </entry><entry> <ulink url="http://demo.openlinksw.com/describe/?url=oai:dcmi.ischool.washington.edu:article/8">Linked Data View</ulink> </entry><entry> <ulink url="http://demo.openlinksw.com/ode/?uri=oai:dcmi.ischool.washington.edu:article/8">Data Explorer View</ulink> </entry><entry> None </entry></row> </tbody></tgroup></table> <para> Cache expiration is managed through the <emphasis><ulink url="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_SPARQL">MinExpiration</ulink></emphasis> parameter in the virtuoso.ini file.</para> <para> </para> <bridgehead class="http://www.w3.org/1999/xhtml:h2"> Installation Steps</bridgehead> <orderedlist spacing="compact"><listitem>A default Virtuoso installation includes the <emphasis>cartridges</emphasis> VAD package, which includes all publicly-available Sponger cartridges and associated components. Check to ensure it is installed using the <emphasis>System Admin</emphasis> -> <emphasis>Packages</emphasis> tab of the Virtuoso Conductor. <itemizedlist mark="bullet" spacing="compact"><listitem>If listed as uninstalled, click the <emphasis>install</emphasis> button to the right of the package. </listitem> <listitem>If the cartridges VAD is not listed, it can be <ulink url="VOSDownload#Other%20Virtuoso-related%20Packages">downloaded now</ulink>. Install the cartridges_dav.vad package using the Conductor UI from the <emphasis>System Admin</emphasis> -> <emphasis>Packages</emphasis> tab or by using iSQL: <programlisting>SQL> DB.DBA.VAD_INSTALL('tmp/cartridges_dav.vad',0); SQL_STATE SQL_MESSAGE VARCHAR VARCHAR _______________________________________________________________________________ 00000 No errors detected 00000 Installation of "Linked Data Cartridges" is complete. 00000 Now making a final checkpoint. 00000 Final checkpoint is made. 00000 SUCCESS 6 Rows. -- 1078 msec. </programlisting></listitem> </itemizedlist></listitem> <listitem>To enable data insertion into the Quad Store via SPARQL queries, you need to <ulink url="http://docs.openlinksw.com/virtuoso/rdfsparql.html#rdfsupportedprotocolendpointuri">assign SPARQL_SPONGE privileges to user SPARQL</ulink>. (Note: more sophisticated security is provided via <ulink url="http://docs.openlinksw.com/virtuoso/rdfsparql.html#sparqloauthendpointfoafssl">WebID based ACL protection</ulink> of your SPARQL endpoint). </listitem> <listitem><ulink url="VirtConfigureCartridges">Configuring Sponger Cartridges</ulink> <itemizedlist mark="bullet" spacing="compact"><listitem><ulink url="VirtSpongerCartridge">Sponger Cartridges Registering API Key</ulink> </listitem> <listitem><ulink url="#ExtractorCartridges">Extractor Cartridges</ulink> </listitem> <listitem><ulink url="#MetaCartridges">Meta Cartridges</ulink> </listitem> <listitem><ulink url="http://docs.openlinksw.com/virtuoso/virtuososponger.html#virtuosospongercreatecustcartrrgst">See also our doc</ulink></listitem> </itemizedlist></listitem> </orderedlist><bridgehead class="http://www.w3.org/1999/xhtml:h3"> Sponger Cartridges included in a Standard Virtuoso Installation</bridgehead> <para>There are a few kinds of Cartridge, and many of each kind are included in a standard Virtuoso installation. Click <ulink url="VirtSpongerCartridgeSupportedDataSources">here</ulink> for a breakdown of <ulink url="OpenLink">OpenLink</ulink>-supported Data Sources.</para> <ulink url=""> <para> </para> <bridgehead class="http://www.w3.org/1999/xhtml:h4"> Extractor Cartridges</bridgehead> <para>An Extractor Cartridge processes a Resource of a given format, extracting RDF according to rules appropriate to that format. External data does not come into play; only the content of the Resource fed to the Sponger.</para> <bridgehead class="http://www.w3.org/1999/xhtml:h5"> Supported Standard Non-RDF Data Formats</bridgehead> <para>These Cartridges handle open formats -- typically community-developed, openly-documented, and freely-licensed data structures.</para> <itemizedlist mark="bullet" spacing="compact"><listitem><ulink url="VirtSpongerCartridgeSupportedDataSourcesNonRDF">Complete list of supported Standard Non-RDF Data Formats</ulink></listitem> </itemizedlist><bridgehead class="http://www.w3.org/1999/xhtml:h5"> Supported Vendor-specific Non-RDF Data Formats</bridgehead> <para>These Cartridges handle closed formats -- typically proprietary; sometimes undocumented; possibly licensed to no-one except the format originator. Sometimes data may not be parsed as desired or expected, as many of these Cartridges have required reverse-engineering of the data format in question.</para> <itemizedlist mark="bullet" spacing="compact"><listitem><ulink url="VirtSpongerCartridgeSupportedDataSourcesVendorNonRDF">Complete list of supported Vendor-specific Non-RDF Data Formats</ulink></listitem> </itemizedlist><ulink url=""> <para> </para> <bridgehead class="http://www.w3.org/1999/xhtml:h4"> Meta Cartridges</bridgehead> <para>A Meta Cartridge submits a Resource to a third-party Web Service for processing. Returned RDF supplements the RDF generated by Extractor and other Meta Cartridges. Locally generated RDF may also be submitted to the third-party services, instead-of or in-addition-to the original Resource itself.</para> <para>Default Sponger behavior is for all installed Meta Cartridges to be brought to bear on all submitted Resources.</para> <itemizedlist mark="bullet" spacing="compact"><listitem><ulink url="VirtSpongerCartridgeSupportedDataSourcesMeta">Complete list of supported Meta Cartridges</ulink> </listitem> <listitem><ulink url="VirtSpongerCartridgeSupportedDataSourcesMetaREST">Meta Cartridge Usage via REST Request</ulink> </listitem> <listitem><ulink url="VirtSpongerCartridgeSupportedDataSourcesMetaRESTExamples">Parametrized Examples of Meta Cartridge Usage via REST Request</ulink></listitem> </itemizedlist><para> </para> <bridgehead class="http://www.w3.org/1999/xhtml:h3"> Sponger Cartridge-based, Dynamic Linked Data Cloud</bridgehead> <para>Click the image for a full-size, clickable version!</para> <para><ulink url="ClickableVirtSpongerCloud"><figure><graphic fileref="VirtSponger/../ClickableVirtSpongerCloud/sponger-cloud.png" /></figure></ulink></para> <bridgehead class="http://www.w3.org/1999/xhtml:h2"> Sponger pragmas</bridgehead> <para>Virtuoso's Sponger is a sophisticated piece of middleware that provides full Linked Data fidelity for pre-existing data objects or resources. This Linked Data is then accessible via HTTP-based Web Services, and SPARQL is enhanced with Sponger pragmas and some optional additions to the FROM clause. See <ulink url="VirtSpongerLinkedDataHooksIntoSPARQL">full list of supported pragmas and usage examples</ulink>.</para> <bridgehead class="http://www.w3.org/1999/xhtml:h2"> Sponger Cartridge Configuration</bridgehead> <itemizedlist mark="bullet" spacing="compact"><listitem><ulink url="VirtConfigureCartridges">Sponger Cartridge Configuration and Implementation Notes</ulink></listitem> </itemizedlist><bridgehead class="http://www.w3.org/1999/xhtml:h2"> Sponger Usage Examples</bridgehead> <itemizedlist mark="bullet" spacing="compact"><listitem><ulink url="http://docs.openlinksw.com/virtuoso/virtuososponger.html#virtuosospongerusageprocessorex">SPARQL Processor Usage Example</ulink> </listitem> <listitem><ulink url="http://docs.openlinksw.com/virtuoso/virtuososponger.html#virtuosospongerusageproxyex2">RDF Proxy Service Example</ulink> </listitem> <listitem><ulink url="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtDeployingLinkedDataGuide_BrowsingNorthwindRdfView#AncMozToc2">Browsing & Exploring RDF View Example Using ODE</ulink> </listitem> <listitem><ulink url="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtDeployingLinkedDataGuide_BrowsingNorthwindRdfView#AncMozToc3">Browsing & Exploring RDF View Example Using iSPARQL</ulink> </listitem> <listitem><ulink url="http://docs.openlinksw.com/virtuoso/rdfinsertmethods.html#rdfinsertmethodplapissimpleexample">Basic Sponger Cartridge Example</ulink> </listitem> <listitem><ulink url="http://docs.openlinksw.com/virtuoso/virtuososponger.html#virtuosospongerusagebriefex">HTTP Example for Extracting Metadata using CURL</ulink> </listitem> <listitem><ulink url="http://docs.openlinksw.com/virtuoso/virtuososponger.html#virtuosospongercartridgetypesmetarestexamples">RESTFul Interaction Examples</ulink> </listitem> <listitem><ulink url="http://docs.openlinksw.com/virtuoso/sect5_virtuosospongercreatecustcartrrgstflickr.html">Flickr Cartridge Example</ulink> </listitem> <listitem><ulink url="http://docs.openlinksw.com/virtuoso/virtuososponger.html#virtuosospongercreatecustcartrexmp">MusicBrainz Metadatabase Example</ulink> </listitem> <listitem><ulink url="VirtTipsAndTricksGuideAddTriplesNamedGraph">SPARQL Tutorial -- Magic of SPARUL and Sponger</ulink></listitem> </itemizedlist><bridgehead class="http://www.w3.org/1999/xhtml:h2"> Other Related Pages</bridgehead> <itemizedlist mark="bullet" spacing="compact"><listitem><ulink url="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html">Technical White Paper</ulink> </listitem> <listitem><ulink url="VirtSpongerCartridgeSupportedDataSources">Supported Virtuoso Sponger Cartridges</ulink> </listitem> <listitem><ulink url="SPARQLSponger">SPARQL Sponger</ulink> </listitem> <listitem><ulink url="VirtInteractSpongerMiddlewareRESTPatterns">Interacting with Sponger Middleware via RESTful Patterns</ulink> </listitem> <listitem><ulink url="VirtSpongerCartridgeSupportedDataSourcesMetaRESTExamples">Interacting with Sponger Meta Cartridge via RESTful Patterns</ulink> </listitem> <listitem><ulink url="VirtSpongerCartridgeRDFExtractor">Sponger Cartridge RDF Extractor</ulink> </listitem> <listitem><ulink url="RDFMappers">Extending SPARQL IRI Dereferencing with RDF Mappers</ulink> </listitem> <listitem><ulink url="VirtSpongerCartridgeProgrammersGuide">Programmer Guide for Virtuoso Linked Data Middleware ("Sponger")</ulink> </listitem> <listitem><ulink url="VirtProgrammerGuideRDFCartridge">Create RDF Custom Cartridge Tutorial</ulink> </listitem> <listitem><ulink url="VirtSpongerCartridgeSupportedDataSources">OpenLink-supplied Virtuoso Sponger Cartridges</ulink> </listitem> <listitem><ulink url="VirtAuthServerUI">Virtuoso Authentication Server</ulink> </listitem> <listitem><ulink url="VirtOAuthSPARQL">Virtuoso SPARQL OAuth Tutorial</ulink> </listitem> <listitem><ulink url="VirtSpongerACL">Virtuoso Sponger Access Control List (ACL) Setup</ulink> </listitem> <listitem><ulink url="VirtSPARQLSecurityWebID">WebID Protocol & SPARQL Endpoint ACLs Tutorial</ulink> </listitem> <listitem><ulink url="http://docs.openlinksw.com/virtuoso/virtuososponger.html">Virtuoso Documentation</ulink></listitem> </itemizedlist><para><ulink url="CategoryEvangelism">CategoryEvangelism</ulink> <ulink url="CategoryDocumentation">CategoryDocumentation</ulink> <ulink url="CategoryPR">CategoryPR</ulink> <ulink url="CategoryVirtuoso">CategoryVirtuoso</ulink> <ulink url="CategoryRDF">CategoryRDF</ulink> <ulink url="CategorySPARQL">CategorySPARQL</ulink></para> <para> </para> </ulink></ulink></section></docbook>