Error on indexing

[skybristol]

I'm having trouble indexing anything other than something from the examples. It seems like I must be missing some critical step. I keep getting the following error lines upon running 'qlever index':

2023-12-09 01:45:53.120	- WARN:  You are trying to call calculateBlockBoundaries on an empty text vocabulary
2023-12-09 01:45:53.162	- ERROR: Assertion `!classicPostings.empty()` failed. Likely cause: A function was called with arguments that violate the contract of the function. Please report this to the developers.. In file "/app/src/index/IndexImpl.Text.cpp " at line 373

My latest attempt is a small ntriples file built from CrossRef metadata on a collection of 50 DOIs. I need to pull these records into a graph that I can combine with other data in queries that refer to the DOIs as identifiers. I have not yet done anything specific with the get-data part of the process, but I will eventually make this a call to a Python script that fetches the data and caches files for indexing. I've also tried this on a number of sources serialized as TTL, but I'm trying to narrow down the problem and figure out why indexing is not working for me.

The indexing process does run up to a point, producing a number of files in my index folder for this collection but then errors out. When I try to run 'qlever start' I get the following error:

2023-12-09 01:54:47.857	- ERROR: Tried to read from a File but too few bytes were returned

I'm happy to provide files on what I'm doing, but here is the relevant snippet from the Qleverfile along with the top of my nt file.

# Indexer settings
DB                = usgspubs
RDF_FILES         = "${DB}.nt"
CAT_FILES         = "cat ${RDF_FILES}"
WITH_TEXT_INDEX   = from_literals
STXXL_MEMORY_GB   = 1
SETTINGS_JSON     = '{ "ascii-prefixes-only": true, "num-triples-per-batch": 100000 }'
BASE_URL          = ""
GET_DATA_CMD      = ""
INDEX_DESCRIPTION = "CrossRef metadata for select USGS Series Publications"
TEXT_DESCRIPTION  = "CrossRef metadata for select USGS Series Publications"

Start of NT file built from calling CrossRef content negotiation and serializing with rdflib in Python.

<http://dx.doi.org/10.3133/gp953B> <http://purl.org/dc/terms/date> "1987"^^<http://www.w3.org/2001/XMLSc
hema#gYear> .
<http://dx.doi.org/10.3133/gp972> <http://www.w3.org/2002/07/owl#sameAs> <doi:10.3133/gp972> .
<http://dx.doi.org/10.3133/gp982> <http://purl.org/dc/terms/title> "Aerial gamma-ray contour maps of reg
ional surface concentrations of potassium, uranium, and thorium in Nevada" .
<http://dx.doi.org/10.3133/gp984> <http://purl.org/dc/terms/date> "1986"^^<http://www.w3.org/2001/XMLSch
ema#gYear> .
<http://dx.doi.org/10.3133/gp1013> <http://purl.org/dc/terms/title> "Aerial gamma-ray maps of regional s
urface concentrations of potassium, uranium, and thorium in Utah" .
<http://dx.doi.org/10.3133/gp996> <http://prismstandard.org/namespaces/basic/2.1/doi> "10.3133/gp996" .
<http://dx.doi.org/10.3133/gp993> <http://purl.org/dc/terms/identifier> "10.3133/gp993" .
<http://dx.doi.org/10.3133/gp984> <http://purl.org/dc/terms/publisher> "US Geological Survey" .
<http://dx.doi.org/10.3133/gp1000> <http://purl.org/dc/terms/title> "Filtered magnetic anomaly maps of M
issouri" .
<http://dx.doi.org/10.3133/gp1003A> <http://purl.org/dc/terms/title> "Electromagnetic survey maps showin
g apparent resistivity of the Getchell Gold Trend, Osgood Mountains, North-Central Nevada" .
<http://dx.doi.org/10.3133/gp1011> <http://purl.org/dc/terms/publisher> "US Geological Survey" .
<http://dx.doi.org/10.3133/gp1015> <http://purl.org/ontology/bibo/doi> "10.3133/gp1015" .
<http://dx.doi.org/10.3133/gp989> <http://purl.org/ontology/bibo/doi> "10.3133/gp989" .
<http://dx.doi.org/10.3133/gp999> <http://www.w3.org/2002/07/owl#sameAs> <info:doi/10.3133/gp999> .
<http://dx.doi.org/10.3133/gp994> <http://purl.org/dc/terms/title> "Aeromagnetic maps and terrace-magnet
ization map centered on the Idaho Batholith and Challis volcanic field, northwestern United States" .
<http://dx.doi.org/10.3133/gp1006> <http://purl.org/dc/terms/date> "1993"^^<http://www.w3.org/2001/XMLSc
hema#gYear> .
<http://dx.doi.org/10.3133/gp994> <http://purl.org/dc/terms/date> "1992"^^<http://www.w3.org/2001/XMLSch
ema#gYear> .

[skybristol]

I guess all I needed to do was write the issue out to find the problem. In this example, I had a problem in the file name I'd used for the nt file. Once I fixed that, I was able to get the indexing completed and start the service back-end. I could then proxy the port for that through nginx and can query the data through the Qlever UI. So, I'm still having trouble with TTL files, but I can deal with writing to n-triples for now.

[hannahbast]

@skybristol Which problems are you having with your TTL files? If you provide a link, we will be happy to try it out on our side.

[skybristol]

Cool! Thank you. The main set of content I need to pull into an index is the Geoscience Ontology. This is the most comprehensive encoding I know of that organizes things like rock types and mineral species within the Basic Formal Ontology and Common Core Ontologies. I know there are a few information issues here and there in the modules, but the TTL files seem to check out structurally using rdflib. I tried setting up a Qleverfile using get-data methods I saw you all use elsewhere to pull from the GitHub source and then index. It's probably something simple that I'm missing.

[skybristol]

@hannahbast - Thanks for engaging on this. I'm excited to get into all the things Qlever should enable us to do. I am working through some of my other use cases in terms of content that we'll be needing to index and have narrowed down a problematic piece seen in the following error:

2023-12-09 14:40:07.488	- INFO:  Integers that cannot be represented by QLever will throw an exception (this is the default behavior)
terminate called after throwing an instance of 'ParseException'
  what():   Parse error at byte position 31557: Prefix [ rdf was not previously defined using a PREFIX or @prefix declaration
The next 500 bytes are:
          gn:Feature ;
                             gn:countryCode    "US" ;
                             gn:parentCountry  <http://sws.geonames.org/6252001/>
                           ] ;
        foaf:familyName    "Horton" ;
        foaf:givenName     "Cheryl" ;
        foaf:page          <https://www.researchgate.net/profile/Cheryl_Horton> , <https://www.usgs.gov/centers/werc/> ;
        foaf:publications  <https://orcid.org/0000-0003-0471-8143#workspace-works> .

<http://sws.geonames.org/62

This is an ORCID record for a person retrieved via content negotiation for text/turtle. I tried just pulling a small handful of what will be about 6000 records we need to build into an index to see where we might run into problems. Once I remove the gn:Feature triple from that record, I'm able to index the batch of 20 that I cached and start/test the Qlever backend successfully.

Here is a curl request for that document where I seem to be tripping a parsing error trying to deal with brackets. I've also seen this error elsewhere, including a whole set of triples I tried to index from our Wikibase instance.

curl -vLH'Accept: text/turtle' https://orcid.org/0000-0003-0471-8143

[hannahbast]

@skybristol I just found this issue again (stuff tends to get overlooked here on https://github.com/ad-freiburg/qlever-control/issues, the main issue tracker is over at https://github.com/ad-freiburg/qlever/issues) and tried to download https://zenodo.org/records/4743221/files/Loop3D/GKM-1.0.1.zip . This contains many .ttl files spread over several directories. Which of these .ttl files are relevant for the dataset, all of them?