%META:TOPICPARENT{name="VirtSetCrawlerJobsGuide"}%
---+Setting up a Content Crawler Job to Retrieve Content from Specific Directories

The following guide describes how to set up crawler job for getting directories using Conductor.


   1 Go to Conductor UI. For ex. at http://localhost:8890/conductor .
   1 Enter dba credentials.
   1 Go to "Web Application Server".
%BR%%BR%<a href="%ATTACHURLPATH%/cr1.png" target="_blank"><img src="%ATTACHURLPATH%/cr1.png" width="600px" /></a>%BR%%BR%
   1 Go to "Content Imports".
%BR%%BR%<a href="%ATTACHURLPATH%/cr2.png" target="_blank"><img src="%ATTACHURLPATH%/cr2.png" width="600px" /></a>%BR%%BR%
   1 Click "New Target".
%BR%%BR%<a href="%ATTACHURLPATH%/cr3.png" target="_blank"><img src="%ATTACHURLPATH%/cr3.png" width="600px" /></a>%BR%%BR%
   1 In the shown form set respectively:
      * "Crawl Job Name": 
<verbatim>
Gov.UK data
</verbatim>
      * "Data Source Address (URL)": 
<verbatim>
http://source.data.gov.uk/data/
</verbatim>
      * "Local WebDAV Identifier" for available user, for ex. demo: 
<verbatim>
/DAV/home/demo/gov.uk/
</verbatim>
      * Choose from the available list "Local resources owner" an user, for ex. demo ;
%BR%%BR%<a href="%ATTACHURLPATH%/d1.png" target="_blank"><img src="%ATTACHURLPATH%/d1.png" width="600px" /></a>%BR%%BR%
      * Click the button "Create". 
   1 As result the Robot target will be created:
%BR%%BR%<a href="%ATTACHURLPATH%/d2.png" target="_blank"><img src="%ATTACHURLPATH%/d2.png" width="600px" /></a>%BR%%BR%
   1 Click "Import Queues".
%BR%%BR%<a href="%ATTACHURLPATH%/d3.png" target="_blank"><img src="%ATTACHURLPATH%/d3.png" width="600px" /></a>%BR%%BR%
   1 For "Robot target" with label "Gov.UK data " click "Run".
   1 As result will be shown the status of the pages: retrieved, pending or respectively waiting.
%BR%%BR%<a href="%ATTACHURLPATH%/d4.png" target="_blank"><img src="%ATTACHURLPATH%/d4.png" width="600px" /></a>%BR%%BR%
   1 Click "Retrieved Sites"
   1 As result should be shown the number of the total pages retrieved.
%BR%%BR%<a href="%ATTACHURLPATH%/d5.png" target="_blank"><img src="%ATTACHURLPATH%/d5.png" width="600px" /></a>%BR%%BR%
   1 Go to  "Web Application Server" -> "Content Management" .
   1 Enter path:
<verbatim>
DAV/home/demo/gov.uk
</verbatim>
%BR%%BR%<a href="%ATTACHURLPATH%/d6.png" target="_blank"><img src="%ATTACHURLPATH%/d6.png" width="600px" /></a>%BR%%BR%
   1 Go to path: 
<verbatim>
DAV/home/demo/gov.uk/data
</verbatim>
  1 As result the retrieved content will be shown.
%BR%%BR%<a href="%ATTACHURLPATH%/d7.png" target="_blank"><img src="%ATTACHURLPATH%/d7.png" width="600px" /></a>%BR%%BR%


---++Related

   * [[VirtSetCrawlerJobsGuide][Setting up Crawler Jobs Guide using Conductor]]
   * [[http://docs.openlinksw.com/virtuoso/rdfinsertmethods.html#rdfinsertmethodvirtuosocrawler][Setting up a Content Crawler Job to Add RDF Data to the Quad Store]]
   * [[VirtSetCrawlerJobsGuideSitemaps][Setting up a Content Crawler Job to Retrieve Sitemaps (where the source includes RDFa)]]
   * [[VirtSetCrawlerJobsGuideSemanticSitemaps][Setting up a Content Crawler Job to Retrieve Semantic Sitemaps (a variation of the standard sitemap)]]
   * [[VirtCrawlerSPARQLEndpoints][Setting up a Content Crawler Job to Retrieve Content from SPARQL endpoint]]