%META:TOPICPARENT{name="VirtSetCrawlerJobsGuide"}%
---+Setting up a Content Crawler Job to retrieve Semantic Sitemaps


The following guide describes how to set up crawler job for getting Semantic Sitemap's content -- a variation of standard sitemap:

   1 Go to Conductor UI. For ex. at http://localhost:8890/conductor .
   1 Enter dba credentials.
   1 Go to "Web Application Server".
%BR%%BR%<a href="%ATTACHURLPATH%/cr1.png" target="_blank"><img src="%ATTACHURLPATH%/cr1.png" width="600px" /></a>%BR%%BR%
   1 Go to "Content Imports".
%BR%%BR%<a href="%ATTACHURLPATH%/cr2.png" target="_blank"><img src="%ATTACHURLPATH%/cr2.png" width="600px" /></a>%BR%%BR%
   1 Click "New Target".
%BR%%BR%<a href="%ATTACHURLPATH%/cr3.png" target="_blank"><img src="%ATTACHURLPATH%/cr3.png" width="600px" /></a>%BR%%BR%
   1 In the shown form:
      * Enter for "Crawl Job Name": 
<verbatim>
Semantic Web Sitemap Example 
</verbatim>
      * Enter for "Data Source Address (URL)": 
<verbatim>
http://www.connexfilter.com/sitemap_en.xml
</verbatim>
      * Enter the location in the Virtuoso WebDAV repository the crawled should stored in the "Local WebDAV Identifier " text-box, for example, if user demo is available, then: 
<verbatim>
/DAV/home/demo/semantic_sitemap/
</verbatim>
      * Choose the "Local resources owner" for the collection from the list box available, for ex: user demo. 
      * Hatch "Semantic Web Crawling":
         * Note: when you select this option, you can either:
            1 Leave the Store Function and Extract Function empty - in this case the system Store and Extract functions will be used for the Semantic Web Crawling Process, or:
            1 You can select your own Store and Extract Functions. [[VirtSetCrawlerJobsGuideSemanticSitemapsFuncExample][View an example of these functions]].
      * Hatch "Accept RDF"
%BR%%BR%<a href="%ATTACHURLPATH%/cr16.png" target="_blank"><img src="%ATTACHURLPATH%/cr16.png" width="600px" /></a>
%BR%<a href="%ATTACHURLPATH%/cr17.png" target="_blank"><img src="%ATTACHURLPATH%/cr17.png" width="600px" /></a>%BR%%BR%
      * Optionally you can hatch "Store metadata *" and specify which RDF Cartridges to be included from the Sponger:
%BR%%BR%<a href="%ATTACHURLPATH%/cr17a.png" target="_blank"><img src="%ATTACHURLPATH%/cr17a.png" width="600px" /></a>%BR%%BR%
   1 Click the button "Create". 
%BR%%BR%<a href="%ATTACHURLPATH%/cr18.png" target="_blank"><img src="%ATTACHURLPATH%/cr18.png" width="600px" /></a>%BR%%BR%
   1 Click "Import Queues".
%BR%%BR%<a href="%ATTACHURLPATH%/cr19.png" target="_blank"><img src="%ATTACHURLPATH%/cr19.png" width="600px" /></a>%BR%%BR%
   1 For "Robot target" with label "Semantic Web Sitemap Example" click "Run".
   1 As result should be shown the number of the pages retrieved.
%BR%%BR%<a href="%ATTACHURLPATH%/cr20.png" target="_blank"><img src="%ATTACHURLPATH%/cr20.png" width="600px" /></a>%BR%%BR%
   1 Check the retrieved RDF data from your Virtuoso instance SPARQL endpoint http://cname:port/sparql with the following query selecting all the retrieved graphs for ex:
<verbatim>
SELECT ?g 
FROM <http://localhost:8890/>
WHERE 
  { 
    graph ?g { ?s ?p ?o } . 
    FILTER ( ?g LIKE <http://www.connexfilter.com/%> ) 
  }
</verbatim>
%BR%%BR%<a href="%ATTACHURLPATH%/cr21.png" target="_blank"><img src="%ATTACHURLPATH%/cr21.png" width="600px" /></a>%BR%%BR%


---++Related

   * [[VirtSetCrawlerJobsGuide][Setting up Crawler Jobs Guide using Conductor]]
   * [[http://docs.openlinksw.com/virtuoso/rdfinsertmethods.html#rdfinsertmethodvirtuosocrawler][Setting up a Content Crawler Job to Add RDF Data to the Quad Store]]
   * [[VirtSetCrawlerJobsGuideSitemaps][Setting up a Content Crawler Job to Retrieve Sitemaps (where the source includes RDFa)]]
   * [[VirtSetCrawlerJobsGuideDirectories][Setting up a Content Crawler Job to Retrieve Content from Specific Directories]]
   * [[VirtCrawlerSPARQLEndpoints][Setting up a Content Crawler Job to Retrieve Content from SPARQL endpoint]]