Using AWS Elastic Block Storage

What is AWS Elastic Block Storage?

Amazon Elastic Block Storage (Amazon EBS) is a new type of storage designed specifically for Amazon EC2 instances. Amazon EBS allows you to create volumes that can be mounted as devices by EC2 instances; your AMI sees an EBS volume as a raw unformatted hard drive device. These can then be managed through the AWS Management Console, with snapshots taken to preserve the state at a given point in time, etc.

Typical Usage Scenarios

Preconfigured Databases

With EBS it is possible to ship whole instances (dump/restore images of virtuoso.db database files) of databases. OpenLink currently provide a few such preconfigured databases. See specific installation instructions for:

Managing Virtuoso Storage

Prerequisites

Creating larger databases

  1. Start by logging into your Amazon AWS Management Console. From here you should see the dashboard overview with an instance running:



  2. From the left menu, select Volumes to view any existing EBS volumes you might have created. In this case, we have no pre-existing volumes:



  3. Click create and fill in a size (here 1Gb) and an availability zone. Note that this must match the availability zone of the instance to which you attach it:



  4. After a short while, the new EBS volume will be created and appear in the management console accordingly:



  5. If you click *attach,* a dialog will pop-up asking to which instance the new volume should be attached. Choose the relevant one and select a device-name for the volume to use within the AMI. Here we choose /dev/sdf (a pseudo-SCSI drive name):



  6. If not already connected, ssh into the AMI as root using your key identity and check for the appearance of the new drive: 

    zsh% ssh -i MyKeyPair.pem root@ec2-67-202-58-58.compute-1.amazonaws.com
[root@domU-12-31-39-03-BE-04 ~]# ls -ltr /dev/[hs]d*
brw-r----- 1 root disk 8,  3 2009-01-09 07:48 /dev/sda3
brw-r----- 1 root disk 8,  1 2009-01-09 07:48 /dev/sda1
brw-r----- 1 root disk 8,  2 2009-01-09 07:48 /dev/sda2
brw-r----- 1 root disk 8, 80 2009-01-16 11:50 /dev/sdf

  7. Should you wish, you may partition the new block-storage device at this point using fdisk or cfdisk or similar. Here we create an ext3 filesystem using the whole device: 

    [root@domU-12-31-39-03-BE-04 ~]# mkfs.ext3 -F /dev/sdf
mke2fs 1.40.8 (13-Mar-2008)
Warning: 256-byte inodes not usable on older systems
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 20 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

  8. For you to work with the new filesystem, the system has to "mount" it.
    • This can be done manually with the mount command, here becoming visible as the directory /opt/virtuoso/data-store/: 

      [root@domU-12-31-39-03-BE-04 ~]# mount -t ext3 /dev/sdf /opt/virtuoso/data-store

    • You can also add a line in /etc/fstab, to make the filesystem come up and be automatically mounted when you reboot or restart the AMI. As above, we are using the directory /opt/virtuoso/data-store/ as the mount point: 

      [root@domU-12-31-39-03-BE-04 ~]# cat /etc/fstab
/dev/sda1       /           ext3        defaults              1 1
/dev/sda2       /mnt        ext3        defaults              1 1
/dev/sda3       swap        swap        defaults              0 0
/dev/sdf        /opt/virtuoso/data-store ext3 defaults,noatime 0 0
[root@domU-12-31-39-03-BE-04 ~]# mount

  9. To migrate the Virtuoso database directory across to this new EBS volume, first, stop the running Virtuoso instance: 

    [root@domU-12-31-39-03-BE-04 ~]# cd /opt/virtuoso/
[root@domU-12-31-39-03-BE-04 virtuoso]# . ./virtuoso-environment.sh
[root@domU-12-31-39-03-BE-04 virtuoso]# virtuoso-stop.sh
Shutting down Virtuoso instance in [database]

  10. The Virtuoso installation is based on the Personal Edition layout so the virtuoso-start.sh and virtuoso-stop.sh commands work on all subdirectories containing an ini-file by default.
  11. Now that the data-store/ directory is mounted, copy the database files across: 

    [root@domU-12-31-39-03-BE-04 virtuoso]# cp -v database/* data-store/
'database/php.ini' -> 'data-store/php.ini'
'database/virtuoso' -> 'data-store/virtuoso'
'database/virtuoso.db' -> 'data-store/virtuoso.db'
'database/virtuoso.ini' -> 'data-store/virtuoso.ini'
'database/virtuoso.log' -> 'data-store/virtuoso.log'
'database/virtuoso.pxa' -> 'data-store/virtuoso.pxa'
'database/virtuoso-temp.db' -> 'data-store/virtuoso-temp.db'
'database/virtuoso.trx' -> 'data-store/virtuoso.trx'
[root@domU-12-31-39-03-BE-04 virtuoso]#

  12. Then restart Virtuoso using the data-store directory: 

    [root@domU-12-31-39-03-BE-04 virtuoso]# virtuoso-start.sh data-store/
Starting Virtuoso instance in [data-store/]

  13. You can tail the server log file to watch its progress: 

    [root@domU-12-31-39-03-BE-04 virtuoso]# tail -f data-store/virtuoso.log
12:00:51 Issued by OpenLink Software
12:00:51 This license will expire on Fri Jan  1 00:00:00 2010 GMT
12:00:51 Database version 3016
12:00:51 SQL Optimizer enabled (max 1000 layouts)
12:00:53 Compiler unit is timed at 0.001264 msec
12:00:55 Roll forward started
12:00:55 Roll forward complete
12:00:56 Checkpoint made, log reused
12:00:56 HTTP/WebDAV server online at 80
12:00:56 Server online at 1111 (pid 14099)

  14. The Virtuoso EC2 AMI instance is now ready for use, and accessible at http://your-ec2-instance-cname/>.

Striping

If your database instance is very large and/or you have a large number of concurrent users causing access to random parts of the database simultaneously, it makes sense to distribute the storage across files on two or more devices using striping.

  1. Follow through the above, to create as many EBS storage volumes as you need, and create a filesystem and mount each of them somewhere under /opt/virtuoso/.
  2. From your ssh login session, modify the database/virtuoso.ini file. First, in the [Database] section, enable Striping overall: 

    [Database]
...
Striping = 1
MaxCheckpointRemap = 2000000

  3. Then, towards the bottom of the file, enter the files to use on EBS devices in the [Striping] section: 

    [Striping]
...
Segment1 = 60000, /opt/virtuoso/data-space1>/virt-seg1.db = q1, /opt/virtuoso/data-space2>/virt-seg1-str2.db = q2, /opt/virtuoso/data-space3>/virt-seg1-str3.db = q3

  4. Each database segment file is assigned a separate background IO thread (the =q clause). *Note:* All files on the same physical device should have the same q value; i.e., multiple stripes on a single EBS volume should have the same q value.

Further information on tuning Virtuoso is available in our documentation.

Creating a Database Snapshot

It is often helpful to be able to preserve the state of a filesystem at a point in time, either in order to restore to it later or to run a backup or similar.

With the Amazon AWS Management Console, creating a new snapshot is a matter of two clicks.

  1. Start at the console's Volumes view:



  2. Click on create snapshot and select the volume of which you wish to make a copy:



  3. After a short delay, the Volumes view will show the snapshot completed:



  4. If required, the snapshot permissions can be changed from the default of private to public by right clicking on the snapshot and selecting the public radio button option, enabling it to then be accessed by the public.
  5. You restore a snapshot by using it as a source from which to create a new volume:



  6. Continue attaching it to an existing AMI instance as previously:



Related