High performance bulk-revision of existing data, on a par with simple bulk insertion of similar data, is best achieved by finding the difference (the "delta") between an existing graph or dataset and the new graph or dataset being loaded, and then applying that differential or "graph delta" to the quad store.
Given an existing dataset hosted by Virtuoso, identified by a named graph IRI, and one that's being loaded from N-Quad files in the filesystem, Virtuoso's bulk load process can automatically determine the differences between the two datasets and quickly apply relevant
DELETEs to the existing dataset.
- A Virtuoso Commercial Edition Release 06.04.3134 or greater is required.
with_deleteoption is available in
- Release 6.x, only in cluster mode
- Release 7.x, in both cluster and single-server mode
- N-Quad datasets where every graph name is specified within the dataset.
Graphs need not be in any particular order, but all triples from each graph must be together.
Triples from different graphs cannot be intermingled.
(In SQL terms,
GROUP BYgraphname; no
ORDER BYis necessary.)
- Virtuoso must be allocated at least 200 bytes of RAM per quad in the dataset being loaded.
As may be obvious, loading large graphs with this option can have a significant impact on Virtuoso's memory use.
- The Virtuoso server must be running with a default transaction isolation level of 2,
READ COMMITTED. Ensure that the
[Parameters]section of the Virtuoso configuration file (default,
virtuoso.ini) includes the following entry, and restart the Virtuoso server.
DefaultIsolation = 2
- The following lock mode settings should be set before using the
cl_exec ('__dbf_set (''lock_escalation_pct'', 200)'); cl_exec ('__dbf_set (''enable_distinct_key_dup_no_lock'', 1)');
- The dataset files must not contain multiple graphs which have the same name but contain different triples.
Doing so will result in unpredictable triple counts, depending on which dataset file is being loaded on a given thread, which is non-deterministic.
For example --
ld_dir ('/data8/2848260', '%.gz', 'with_delete'); ld_dir_all ('/data8/', '%.gz', 'with_delete');
Once all are set run the
cl_exec('rdf_ld_srv()') commands to enable the update/reload to commence.
cl_exec('rdf_ld_srv()') commands can be invoked as threads/cores are available across the machines the Virtuoso cluster is being run on for fast parallel loading of the datasets, as would typically be done for the initial bulk load of the datasets.
Note that all RDF loader threads can be stopped using the following command at which point all currently running threads will be allowed to complete and then exit:
A diagnostic log of the
with_delete activity may be written to a file called
g_log.txt on each cluster instance.
- To enable this log, run the following command:
cl_exec ('__dbf_set (''enable_g_replace_log'',1)')
- To disable this log, run the following command:
cl_exec ('__dbf_set (''enable_g_replace_log'',0)')