%META:TOPICPARENT{name="VirtSetCrawlerJobsGuide"}%
---+Setting up a Content Crawler Job to retrieve Sitemaps

The following guide describes how to set up a crawler job for getting content of a basic Sitemap where the source includes RDFa.

   1 From the Virtuoso Conductor User Interface i.e. http://cname:port/conductor, login as the "dba" user.
   1 Go to "Web Application Server" tab.
%BR%%BR%<a href="%ATTACHURLPATH%/cr1.png" target="_blank"><img src="%ATTACHURLPATH%/cr1.png" width="600px" /></a>%BR%%BR%
   1 Go to the "Content Imports" tab.
%BR%%BR%<a href="%ATTACHURLPATH%/cr2.png" target="_blank"><img src="%ATTACHURLPATH%/cr2.png" width="600px" /></a>%BR%%BR%
   1 Click on the "New Target" button.
%BR%%BR%<a href="%ATTACHURLPATH%/cr3.png" target="_blank"><img src="%ATTACHURLPATH%/cr3.png" width="600px" /></a>%BR%%BR%
   1 In the form displayed:
      *  Enter a name of choice in the "Crawl Job Name" text-box: 
<verbatim>
Basic Sitemap Crawling Example 
</verbatim>
      * Enter the URL of the site to be crawled in the "Data Source Address (URL)" text-box: 
<verbatim>
http://psclife.pscdog.com/catalog/seo_sitemap/product/&nbsp
</verbatim>
      * Enter the location in the Virtuoso WebDAV repository the crawled should stored in the "Local WebDAV Identifier" text-box, for example, if user demo is available, then: 
<verbatim>
/DAV/home/demo/basic_sitemap/
</verbatim>
      * Choose the  "Local resources owner" for the collection from the list-box available, for ex: user demo.
      * Select the "Accept RDF" check-box. 
%BR%%BR%<a href="%ATTACHURLPATH%/cr11.png" target="_blank"><img src="%ATTACHURLPATH%/cr11.png" width="600px" /></a>%BR%<a href="%ATTACHURLPATH%/cr11a.png" target="_blank"><img src="%ATTACHURLPATH%/cr11a.png" width="600px" /></a>%BR%%BR%
   1 Click the "Create" button to create the import:
%BR%%BR%<a href="%ATTACHURLPATH%/cr12.png" target="_blank"><img src="%ATTACHURLPATH%/cr12.png" width="600px" /></a>%BR%%BR%
   1 Click the "Import Queues" button.
   1 For the "Robot targets" with label "Basic Sitemap Crawling Example " click the "Run" button.
   1 This will result in the Target site being crawled and the retrieved pages stored locally in DAV and any sponged triples in the RDF Quad store.
%BR%%BR%<a href="%ATTACHURLPATH%/cr13.png" target="_blank"><img src="%ATTACHURLPATH%/cr13.png" width="600px" /></a>%BR%%BR%
   1 Go to the "Web Application Server" -> "Content Management" tab.
%BR%%BR%<a href="%ATTACHURLPATH%/cr14.png" target="_blank"><img src="%ATTACHURLPATH%/cr14.png" width="600px" /></a>%BR%%BR%
   1 Navigate to the location of newly created DAV collection:
<verbatim>
/DAV/home/demo/basic_sitemap/
</verbatim>
   1 The retrieved content will be available in this location.
%BR%%BR%<a href="%ATTACHURLPATH%/cr15.png" target="_blank"><img src="%ATTACHURLPATH%/cr15.png" width="600px" /></a>%BR%%BR%


---++Related

   * [[VirtSetCrawlerJobsGuide][Setting up Crawler Jobs Guide using Conductor]]
   * [[http://docs.openlinksw.com/virtuoso/rdfinsertmethods.html#rdfinsertmethodvirtuosocrawler][Setting up a Content Crawler Job to Add RDF Data to the Quad Store]]
   * [[VirtSetCrawlerJobsGuideSemanticSitemaps][Setting up a Content Crawler Job to Retrieve Semantic Sitemaps (a variation of the standard sitemap)]]
   * [[VirtSetCrawlerJobsGuideDirectories][Setting up a Content Crawler Job to Retrieve Content from Specific Directories]]
   * [[VirtCrawlerSPARQLEndpoints][Setting up a Content Crawler Job to Retrieve Content from SPARQL endpoint]]