%META:TOPICPARENT{name="VirtInstallationEC2"}%

---+ DBpedia Snapshot (Virtuoso PAGO) EBS-backed EC2 AMI

%TOC%

---++ Introduction

In addition to the [[VirtInstallationEC2][Instance-backed EC2 AMI]] that has been available since 2008, a standard unpopulated Virtuoso instance is available as an EBS-backed EC2 AMI based on either a <i>[[https://aws.amazon.com/marketplace/pp/B00ZWMSNOG][BYOL (Bring Your Own License)]]</i> or a <i>[[https://aws.amazon.com/marketplace/pp/B011VMCZ8K/][PAGO (Pay As You Go)]]</i> basis. In each case, the AMI delivers a preconfigured Virtuoso instance. 

We also now offer two PAGO variants, both pre-loaded with DBpedia data.  

   * The <b>DBpedia Snapshot (Virtuoso PAGO)</b> (documented on this page) starts as a static instance, preloaded with the DBpedia 2016-10 dataset, mirroring the public [[http://wiki.dbpedia.org/OnlineAccess][DBpedia]] instance found at [[http://dbpedia.org/sparql][http://dbpedia.org/sparql]].  You can make changes to this data, but it will not track changes made to Wikipedia nor DBpedia-Live.
%BR%%BR%
   * The <b>[[https://aws.amazon.com/marketplace/pp/B012DSCFEK][DBpedia Live (Virtuoso PAGO)]]</b> (documented on [[VirtPayAsYouGoEBSBackedAMIDBpedia2015][another page]]) starts as a static instance, preloaded with the DBpedia 2016-04 dataset, and includes an optional switch that enables data updates based on the Wikipedia firehose, effectively giving you a mirror of the public [[http://wiki.dbpedia.org/online-access/DBpediaLive][DBpedia-Live]] instance found at [[http://live.dbpedia.org/sparql][http://live.dbpedia.org/sparql]].

This type of AMI provides several fundamental benefits including &#8212;

   * Virtuoso DBMS Server is preinstalled with basic tuning for the host operating system.  (That said, since we support many AMI machine types/sizes, you should still [[#Performance%20Notes][tune the configuration to suit the available RAM in your instance]].)
   * DBpedia Dataset is preloaded and preconfigured (and may be configurable to auto-update).
   * You can start and stop the DBpedia instance without having to terminate its host AMI.
   * With the hourly model, you pay only for the time the AMI is used.

---++ Prerequisites

   * An [[https://aws-portal.amazon.com/gp/aws/developer/registration/index.html/103-7399647-0537426?][Amazon Web Services]] (AWS) account.
   * Recently created AWS accounts will have been automatically signed up for the [[http://www.amazon.com/b/ref=sc_fe_l_2?ie=UTF8&node=201590011&no=3435361&me=A36L942TSJ2AJA][Amazon S3 and EC2 Web Service]]. If you created your AWS account a long time ago, you may now need to manually sign up for these services.
   * Ensure an AWS <code>security group</code> allowing access to ports 22 (standard SSH), 80 (standard HTTP), and 8890 (Virtuoso HTTP-based Admin) is used.  (This is the setup of the AMI offerings.)

---++ Instantiating DBpedia Snapshot (Virtuoso PAGO) via Web Interface

   1 Locate the [[https://aws.amazon.com/marketplace/pp/B012DSCFEK][DBpedia Snapshot (Virtuoso PAGO)]] image in AWS Marketplace and click the <b>Continue</b> button.
%BR%%BR%<a href="%ATTACHURLPATH%/VirtDBpedia2015PAGO01.png"><img src="%ATTACHURLPATH%/VirtDBpedia2015PAGO01.png" alt="AWS Marketplace DBpedia Snapshot (Virtuoso PAGO)" style="max-width: 600px ; min-width: 200px ; width: 100%"></a>%BR%%BR%
   1 Choose a suitable size <b>EC2 Instance Type</b> and <b>Key Pair</b>, then click on the button <b>Launch with 1-click</b> button.
%BR%%BR%<a href="%ATTACHURLPATH%/VirtDBpedia2015PAGO02.png"><img src="%ATTACHURLPATH%/VirtDBpedia2015PAGO02.png" alt="AWS Marketplace DBpedia Snapshot (Virtuoso PAGO) Launch on EC2" style="max-width: 600px ; min-width: 200px ; width: 100%"></a>%BR%%BR%
   1 A confirmation dialog will be presented indicating the image has been deployed.
%BR%%BR%<a href="%ATTACHURLPATH%/VirtDBpedia2015PAGO03.png"><img src="%ATTACHURLPATH%/VirtDBpedia2015PAGO03.png" alt="AWS Marketplace DBpedia Snapshot (Virtuoso PAGO) now Deployed" style="max-width: 600px ; min-width: 200px ; width: 100%"></a>%BR%%BR%
   1 Check the in the [[https://console.aws.amazon.com/][AWS Console]] EC2 images Web Interface that the image has been successfully instantiated.
%BR%%BR%<a href="%ATTACHURLPATH%/VirtDBpedia2015PAGO04.png"><img src="%ATTACHURLPATH%/VirtDBpedia2015PAGO04.png" alt="AWS EC2 Launched Image" style="max-width: 600px ; min-width: 200px ; width: 100%"></a>%BR%%BR%
   1 Load the Virtuoso Admin Console (a/k/a Conductor) in your browser to confirm all is running properly:
<verbatim>
http://{amazon-ec2-ami-dns-name-or-ip-address}:8890/conductor
</verbatim>

---++ First-time Setup & Usage Notes 

These steps in this section are only necessary the first time you start the Virtuoso instances on the AMI.  This section may be ignored thereafter, as it is not necessary after AMI reboots.

There are two Virtuoso instances in this AMI.  One which comes up quickly, with no significant content, so you know the AMI is basically functional; and one which comes up more slowly, with the full DBpedia dataset, which takes significant time to start due to some Amazon requirements for such AWS instances.

---+++ Basic Instance

   1 <code>ssh</code> into your instantiated AMI using a command of the form &#8212; 
<verbatim>
ssh -i {secure-pem-file} ec2-user@{amazon-ec2-dns-name-or-ip-address}
</verbatim>
   1 The Virtuoso DBMS Server for the Basic Instance will have started with the AMI.  You can verify this with --
<verbatim>
ps -ef | grep "virt*" | grep -v grep
</verbatim>
      * If you do not see a running instance, execute the following commands, and then repeat the command above.
<verbatim>
cd /opt/virtuoso
. ./virtuoso-environment.sh    # this must start dot-space-dot-slash
virtuoso-start.sh database
</verbatim>
   1 We strongly recommend you now use the Conductor to change the password for the '<code>dba</code>' user from the the AMI <b><code>instance-id</code></b>. 
%BR%%BR%
      1 Retrieve the AMI <code>instance-id</code> by either --
         * checking the AMI properties presented by the Amazon AWS console UI -- 
%BR%%BR%<a href="%ATTACHURLPATH%/VirtDBpedia2015PAGOAwsConsole.png"><img src="%ATTACHURLPATH%/VirtDBpedia2015PAGOAwsConsole.png" alt="AWS Console" style="max-width: 600px ; min-width: 200px ; width: 100%"></a>%BR%%BR%
         * executing the following command in the Linux shell -- 
<verbatim>
curl http://169.254.169.254/latest/meta-data/instance-id
</verbatim>
      1 Execute this command sequence --
<verbatim>
cd /opt/virtuoso
. ./virtuoso-environment.sh    # this must start dot-space-dot-slash
cd bin
./isql localhost:1112 dba
</verbatim>
      1 When prompted for password, provide the instance-id
      1 At the <code>SQL&gt;</code> prompt, execute these two commands --
<verbatim>
vad_install('/opt/virtuoso/vad/conductor_dav.vad',0)
quit
</verbatim>
      1 Load the Conductor interface.
<verbatim>
http://{amazon-ec2-ami-dns-name-or-ip-address}:8890/conductor
</verbatim>
         * If you get any error at this point, execute the following commands, and then re-try loading the Conductor in your web browser.
<verbatim>
cd /opt/virtuoso
. ./virtuoso-environment.sh    # this must start dot-space-dot-slash
virtuoso-stop.sh database
virtuoso-start.sh database
</verbatim>
      1 At the authentication challenge, log in as the <b><code>dba</code></b> user, with the AMI <b><code>instance-id</code></b> as the password.  <i><b>Note:</b> If unable to connect to the Virtuoso server using the  <b><code>instance-id</code></b> as password, please [[http://download.openlinksw.com:8080/download/register.vsp?hname=support][register with our Support Site]], and [[http://support.openlinksw.com/support/techadd.vsp][create a Support Case]] for fastest assistance.</i>
      1 Drill down to <b>System Admin</b> &#8594; <b>User Accounts</b>.  
      1 Locate the <b><code>dba</code></b> user, and click the associated <b><code>Edit</code></b> link.
      1 The form allows many things to be changed.  For now, just input your desired password into both <b>Password</b> and <b>Confirm Password</b> boxes, and click the <b>Save</b> button.
      1 You can now perform other administrative tasks through the Conductor interface, or return to basic use.

---+++ DBpedia Instance

   1 <code>ssh</code> into your instantiated AMI using &#8212;  
<verbatim>
ssh -i {secure-pem-file} ec2-user@{ec2-dns-name-or-ip-address}
</verbatim>
   1 Stop the Basic Instance (to conserve system and license resources) by running &#8212; 
<verbatim>
virtuoso-stop.sh dbpedia
</verbatim>
   1 Start the Virtuoso DBMS Server against the DBpedia Database by issuing the commands below.  <i><b>Note:</b> At initial launch, it takes the Virtuoso DBMS Server approximately 20 minutes to bring the DBpedia database online, due to its size. </i>
<verbatim>
cd /opt/virtuoso
. ./virtuoso-environment.sh    # this must start dot-space-dot-slash
virtuoso-start.sh dbpedia
</verbatim>
   1 We strongly recommend you now use the Conductor to change the password for the '<code>dba</code>' user from the the AMI <b><code>instance-id</code></b>. 
%BR%%BR%
      1 Retrieve the AMI <code>instance-id</code> by either --
         * checking the AMI properties presented by the Amazon AWS console UI -- 
%BR%%BR%<a href="%ATTACHURLPATH%/VirtDBpedia2015PAGOAwsConsole.png"><img src="%ATTACHURLPATH%/VirtDBpedia2015PAGOAwsConsole.png" alt="AWS Console" style="max-width: 600px ; min-width: 200px ; width: 100%"></a>%BR%%BR%
         * executing the following command in the Linux shell -- 
<verbatim>
curl http://169.254.169.254/latest/meta-data/instance-id
</verbatim>
      1 Load the Conductor interface
<verbatim>
http://{amazon-ec2-ami-dns-name-or-ip-address}/conductor
</verbatim>
         * If you get any error at this point, execute the following commands, and then re-try loading the Conductor in your web browser.
<verbatim>
cd /opt/virtuoso
. ./virtuoso-environment.sh    # this must start dot-space-dot-slash
virtuoso-stop.sh dbpedia
virtuoso-start.sh dbpedia
</verbatim>
      1 At the authentication challenge, log in as the <b><code>dba</code></b> user, with the AMI <b><code>instance-id</code></b> as the password.  <i><b>Note:</b> If unable to connect to the Virtuoso server using the  <b><code>instance-id</code></b> as password, please [[http://download.openlinksw.com:8080/download/register.vsp?hname=support][register with our Support Site]], and [[http://support.openlinksw.com/support/techadd.vsp][create a Support Case]] for fastest assistance.</i>
      1 Drill down to <b>System Admin</b> &#8594; <b>User Accounts</b>.  
      1 Locate the <b><code>dba</code></b> user, and click the associated <b><code>Edit</code></b> link.
      1 The form allows many things to be changed.  For now, just input your desired password into both <b>Password</b> and <b>Confirm Password</b> boxes, and click the <b>Save</b> button.
      1 You can now perform other administrative tasks through the Conductor interface, or return to basic DBpedia use.


---++ DBpedia Snapshot (Virtuoso PAGO) Database Interaction via Web Interface

Once online, your DBpedia Snapshot instance will be ready for use from &#8212;
%BR%%BR%
   * Basic Linked Data Exploration Page &#8212; an obvious starting point
<verbatim>
http://{amazon-ec2-ami-dns-name-or-ip-address}/resource/DBpedia
</verbatim>
   * Faceted Browsing Endpoint
<verbatim>
http://{amazon-ec2-ami-dns-name-or-ip-address}/fct
</verbatim>
   * Advanced Faceted Browsing Page
<verbatim>
http://{amazon-ec2-ami-dns-name-or-ip-address}/describe/?uri=http://dbpedia.org/resource/DBpedia
</verbatim>
   * SPARQL Query Service Endpoint
<verbatim>
http://{amazon-ec2-ami-dns-name-or-ip-address}/sparql
</verbatim>
   * Virtuoso Instance Administration Page (Virtuoso Conductor)
<verbatim>
http://{amazon-ec2-ami-dns-name-or-ip-address}/conductor
</verbatim>


---++ Administering the Virtuoso Instance via SSH

All scripts for starting and stopping the Virtuoso instance are found in the following locations &#8212;

   * <b><code><nowiki>/etc/rc.d/init.d</nowiki></code></b> &#8212; scripts that enable automatic database server instantiation at operating system (AMI) boot or reboot time
%BR%%BR%
   * <b><code><nowiki>/opt/virtuoso</nowiki></code></b> &#8212; scripts for starting and stopping the database server within a running operating system (AMI)

---+++ License Manager 

The <nop>OpenLink License Manager must be launched before you launch the Virtuoso instance, and must remain running at all times for Virtuoso to run.

   * Start the License Manager
<verbatim>
/etc/rc.d/init.d/oplmgr start
</verbatim>

   * Stop the License Manager
<verbatim>
/etc/rc.d/init.d/oplmgr stop
</verbatim>

   * Restart the License Manager
<verbatim>
/etc/rc.d/init.d/oplmgr restart
</verbatim>

---+++ Virtuoso Server

   * Start the Basic Virtuoso Server
<verbatim>
/etc/rc.d/init.d/virtuoso start
</verbatim>

   * Stop the Basic Virtuoso Server
<verbatim>
/etc/rc.d/init.d/virtuoso stop
</verbatim>

   * Restart the Basic Virtuoso Server
<verbatim>
/etc/rc.d/init.d/virtuoso restart
</verbatim>

---+++ Virtuoso DBpedia Server

   * Start the DBpedia Virtuoso Server
<verbatim>
cd /opt/virtuoso
. ./virtuoso-environment.sh    # this must start dot-space-dot-slash
virtuoso-start.sh dbpedia
</verbatim>

   * Stop the DBpedia Virtuoso Server
<verbatim>
cd /opt/virtuoso
. ./virtuoso-environment.sh    # this must start dot-space-dot-slash
virtuoso-stop.sh dbpedia
</verbatim>

   * Restart the DBpedia Virtuoso Server
<verbatim>
cd /opt/virtuoso
. ./virtuoso-environment.sh    # this must start dot-space-dot-slash
virtuoso-stop.sh dbpedia
virtuoso-start.sh dbpedia
</verbatim>

---++ Command-line Interaction with the Virtuoso Database Instance 

   1 Set the Virtuoso environment variables by running the command below.  <i><b>Note:</b> This does and must start with dot-space-slash.</i>
<verbatim>
. /opt/virtuoso/virtuoso-enterprise.sh
</verbatim>
   1 Run the Virtuoso "<b><code>isql</code></b>" command line tool to connect to the database. <i><b>Note:</b> your EC2 AMI's <b>instance-id</b> will be the <code>dba</code> user's password, until you change it (as recommended above). </i>
<verbatim>
$ isql 1111 -U dba -P {Password}
Connected to OpenLink Virtuoso
Driver: 07.10.3214 OpenLink Virtuoso ODBC Driver
OpenLink Interactive SQL (Virtuoso), version 0.9849b.
Type HELP; for help and EXIT; to exit.
SQL>
</verbatim>
   1 Run the "<b><code>tables</code></b>" command to obtain a list of tables in the default schema
<verbatim>
SQL> tables;
Showing SQLTables of tables like 'NULL.NULL.NULL', tabletype/colname like 'NULL'
TABLE_QUALIFIER  TABLE_OWNER      TABLE_NAME       TABLE_TYPE       REMARKS
VARCHAR          VARCHAR          VARCHAR          VARCHAR          VARCHAR
_______________________________________________________________________________

DB               DBA              ADMIN_SESSION    SYSTEM TABLE     NULL
DB               DBA              ADM_OPT_ARRAY_TO_RS_PVIEW  SYSTEM TABLE     NULL
DB               DBA              ADM_XML_VIEWS    SYSTEM TABLE     NULL
.
.
.
DB               DBA              SYS_SQL_INVERSE  SYSTEM TABLE     NULL
DB               DBA              SYS_TRIGGERS     SYSTEM TABLE     NULL
DB               DBA              SYS_VIEWS        SYSTEM TABLE     NULL

209 Rows. -- 1890 msec.
SQL>
</verbatim>
   1 You can stop the Virtuoso Database Server by running &#8212; 
<verbatim>
virtuoso-stop.sh dbpedia
</verbatim>
   1 You can restart the Virtuoso Database Server by running &#8212; 
<verbatim>
virtuoso-start.sh dbpedia
</verbatim>

---++ Performance Notes

Please be aware of the following, which impact the performance and utility of your AMI:

   * This AMI includes a bundled Virtuoso license which enables 10 Database Sessions and the use of 4 logical processors.  Licenses that upgrade these attributes are available as paid upgrade options. 
%BR%%BR%
   * Virtuoso always takes full advantage of the memory it's configured to use.  <i>This may be much less than is found in its host environment/AMI.</i>  This AMI is pre-configured for an <code>m3.large</code> EC2 Instance Type, so will use 7GB of RAM.  If you choose a larger EC2 Instance Type (such as the recommended <code>m3.2xlarge</code>), then the <code><nop>NumberOfBuffer</code> and <code><nop>MaxDirtyBuffers</code> parameters in the <code>/opt/virtuoso/dbpedia/dbpedia.ini</code> configuration file should be increased to correspond to the chosen Instance Type's available memory, as detailed in the [[http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPerformanceTuning#General%20Memory%20Usage%20Settings][Virtuoso Performance Tuning Guide]].  A few examples are shown below.  After changing these or any other settings in the INI file, restart the Virtuoso server [[#Virtuoso%20Server][as described above]].
%BR%%BR%
| *EC2 Instance Type* |  *System RAM*  |  *Number Of Buffers*  |  *Max Dirty Buffers*  |
| m3.large |  7 GB  |   680000 |   500000 |
| r3.large |  15 GB  |   1360000 |   1000000 |
| r3.xlarge |  30.5 GB  |   2720000 |   2000000 |
| r3.2xlarge |  61 GB  |   5450000 |   4000000 |
%BR%%BR%
   * There are a wide range of AMI choices, offering various combinations of system memory and logical processors.  To improve performance, use an <b>EC2 Instance Type</b> with more memory and more logical processors.  To make use of additional processors, you will need to also acquire an upgraded Virtuoso license.


---++ Troubleshooting

If you encounter any problems resolving the sample DBpedia URIs listed in the steps above, please:

   1. Determine whether Virtuoso is running, with this command
<verbatim>
ps -ef | grep "virt*" | grep -v grep
</verbatim>
   2. Check the log of Virtuoso's most recent activity with one of these commands
      * for the DBpedia instance
<verbatim>
tail /dbpedia/*.log
</verbatim>
      * for the basic instance
<verbatim>
tail /opt/virtuoso/dbpedia/*.log
</verbatim>

The output of those commands will show you whether the initial Virtuoso DBpedia DB setup (which can take a while due to DB size) is still in progress, the setup encountered some error, or the setup has completed but Virtuoso awaits one of the following commands:

   * Startup Commands
      * Startup of the Basic Instance
<verbatim>
cd /opt/virtuoso
. ./virtuoso-environment.sh    # this must start dot-space-dot-slash
virtuoso-start.sh database
</verbatim>
      * Startup of the DBpedia Instance
<verbatim>
cd /opt/virtuoso
. ./virtuoso-environment.sh    # this must start dot-space-dot-slash
virtuoso-start.sh dbpedia
</verbatim>

   * Restart commands
      * Restart the Basic Instance
<verbatim>
cd /opt/virtuoso
. ./virtuoso-environment.sh    # this must start dot-space-dot-slash
virtuoso-stop.sh database
virtuoso-start.sh database
</verbatim>
      * Restart the DBpedia Instance
<verbatim>
cd /opt/virtuoso
. ./virtuoso-environment.sh    # this must start dot-space-dot-slash
virtuoso-stop.sh dbpedia
virtuoso-start.sh dbpedia
</verbatim>


---++ Related Items

   * [[http://ods.openlinksw.com/wiki/ODS/ODSInstallationEC2][OpenLink Data Spaces Installation]]
   * EC2 &amp; AMI related information
      * [[VirtPayAsYouGoEBSBackedAMI][Virtuoso Pay As You Go (PAGO) EBS-backed EC2 AMI]]
      * [[VirtInstallationEC2][Instance-backed Virtuoso EC2 AMI]]
      * [[VirtAWSPublicDataSets][Pre-configured &amp; Pre-loaded Virtuoso instances (DBpedia 3.8, BBC Music &amp; Programmes, etc.)]]
      * [[VirtEC2AMINeuroCommonsInstall][Creating Your Own Neurocommons Instance]]
      * [[VirtEC2AMIBio2rdfInstall][Creating Your Own Bio2RDF Instance]]
      * [[VirtEC2AMIMusicBrainzInstall][Creating Your Own MusicBrainz Instance]]
      * [[VirtEc2ExtensionUI][Backup your Virtuoso EC2 AMI to S3]]
      * [[VirtEC2AMIElasticBlockStorageConfig][Configure your Virtuoso EC2 AMI for use with Amazon Elastic Block Storage (EBS)]]
      * [[http://calculator.s3.amazonaws.com/calc5.html][Amazon-provided AWS Simple Monthly Cost Calculator]]
   * Protecting your Virtuoso-hosted SPARQL Endpoint
      * [[VirtSPARQLEndpointProtection][Safeguarding your Virtuoso-hosted SPARQL Endpoint]]
      * [[VirtTipsAndTricksGuideSPARQLEndpointProtection][SPARQL Endpoint Protection Methods Collection]]
      * [[VirtTipsAndTricksGuideSPARQLEndpoints][Securing SPARQL endpoints]]
      * [[VirtSPARQLSecurityWebID][WebID Protocol &amp; SPARQL Endpoint ACLs Tutorial]]
      * [[VirtOAuthSPARQL][SPARQL OAuth Tutorial]]
   * [[http://docs.openlinksw.com/virtuoso/][Virtuoso documentation]]
      * [[http://docs.openlinksw.com/virtuoso/rdfsparql.html][Virtuoso's SPARQL support]]
   * [[VirtTipsAndTricksGuide][Virtuoso Tips and Tricks]]
      * [[VirtSPARQLDET][SPARQL Endpoint DET (DEV Extension Type) Configuration Guide]]


CategoryHowTo CategoryODS CategoryVirtuoso CategoryDocumentation CategoryEC2

%VOSCOPY%