WARNING! This URL (in the Main cluster) is no longer the authoritative source for this page; it has been moved to the VOS or ODS cluster as appropriate instead.
See Tim Haynes in case of confusion.
Core Engine Enhancements
This update of Virtuoso is a major revision of both engine, ODS applications suite and RDF functionality.
Highlights include:
- RDF Mapping of relational data.
This means that any relational data stored in Virtuoso or any other relational database can be converted on the fly to RDF and queried with SPARQL without making an RDF copy of the data itself.
- Bitmap index support - This is a generic SQL and engine enhancement that delivers dramatic space savings and incresed performance specially in RDF triple storage and many business intelligence workloads.
- Engine enhancements including automatic background compaction for up to 30% less space usage for typical relational or RDF databases, better SMP behavior with shorter critical sections, fixes in transaction logic and more.
Relational to RDF Mapping
We introduce a language for mapping between triples and relational tables.
This feature is experimental and the language will change in the future.
Bitmap Indices and Native RDF Storage
This new release introduces bitmap indices for general SQL usage.
Special attention has been given to making these also useful to RDF storage.
This requires a very low overhead for distinct key values.
This technique allows us dramatically reduce the working set size of many RDF workloads, specially importing of large data sets.
The details of this feature are covered in the article Advances in Virtuoso RDF STorage.
Engine Updates
- Database compaction - Virtuoso will now automatically merge adjacent dirty pages when they are about to be written to disk if the content of the stretch of dirty pages will fit on fewer pages.
This is also available as an explicit db..bvacuum () stored procedure.
Even the automatic compaction produces savings of upp to 30% in applications that make large inserts or updates in random order.
- Shorter critical sections for index traverssal.
To avoid needless contention on mutexes which carry a high penalty specially on SMP systems, we have shortened some critical sections.
In specific, transaction log writing now has its own section and is not serialized together with the transaction wait graph used for detecting deadlocks.
- Smarter page layout - To accommodate situations where rows grow in length as a result of update, which is common in bitmap indices as well as text indices, the system now lays rows on pages so that they have evenly spread room to grow.
- O_DIRECT support - Some Unix platforms, including newer Linux's support bypassing the OS file cache for disk IO.
When a large part of memory is used for database buffers inside the database process, OS file caching can get in the way and even cause swapping.
In such situations, enabling O_DIRECT can help.
This is done with a setting in the ini file and takes effect at server restart.
- Multithreading API for SQL - We have documented an internal Virtuoso/PL construct for allowing a stored procedure to spawn threads.
An async_quue object allows passing procedure calls and parameters to a pool of worker threads for asynchroneous execution.
Each will execute in its own transaction.
This is useful for parallelizing any IO or compute load, for example in web scraping, bulk load of data etc.
CategoryMarketing