Janusgraph taking too much time or gives Timeout Error

[sharma-kunal]

I am using Janusgraph 0.6.3 with cassandra as backend and elastic search as the index backend. I have around 30 million vertices and 40 million edges in my graph, I am using a 128 GB RAM machine, with 32 core CPU which have both Janus and Cassandra installed on it. (Elastic is hosted on other machine with more better specs).

I have defined both composite and mixed index for the property I am using in my gremlin queries,

Composite Index

management.buildIndex("seeker", Vertex.class).addKey(management.getPropertyKey("p_uuid"))
                .indexOnly(management.getVertexLabel("seeker"))
                .buildCompositeIndex();
        management.buildIndex("company", Vertex.class).addKey(management.getPropertyKey("company_name"))
                .indexOnly(management.getVertexLabel("company"))
                .buildCompositeIndex();
        management.buildIndex("start_date", Edge.class).addKey(management.getPropertyKey("start_date"))
                .indexOnly(management.getEdgeLabel("worked"))
                .buildCompositeIndex();
        management.buildIndex("end_date", Edge.class).addKey(management.getPropertyKey("end_date"))
                .indexOnly(management.getEdgeLabel("worked"))
                .buildCompositeIndex();

Mixed Index

management.buildIndex("vertex", Vertex.class).addKey(management.getPropertyKey("p_uuid"))
                .addKey(management.getPropertyKey("company_name"))
                .buildMixedIndex(this.graphCreds.getIndex_index_name());
        management.buildIndex("edge", Edge.class).addKey(management.getPropertyKey("start_date"))
                .addKey(management.getPropertyKey("end_date"))
                .addKey(management.getPropertyKey("current"))
                .buildMixedIndex(graphCreds.getIndex_index_name());

The query that I am running on this graph is,

g.V().hasLabel("seeker").has("p_uuid", <p_uuid>).outE("worked").as_("e1").inV().inE("worked").as_("e2")\
                .outV().as_("b").outE("worked").where(inV().hasLabel("company").has("company_name", <company_name>)).has("current", 1)\
                .where("e1", lt("e2")).by("start_date").by("end_date").where("e1", gt("e2")).by("end_date").by("start_date")\
                .select("e1", "e2", "b").range_(0,10).toList()

As we can say, I am using 4 properties in my gremlin query,

• p_uuid
• company_name
• end_date
• start_date

and all 4 are indexed in both Composite and Mixed Index. Then why does it gives timeout.

I have checked the machine performance metrics as well, the machine does not utilize 50% of CPU. If the machine has CPU power, then why Janusgraph is not uitlizing that ?

I have kept all the Janusgraph configurations to default value.

One thing that I am also unsure about is, when I do the profiling for the above query, it hits the seeker index only (from the first part of the query), after that shouldn't it hit the company index as well ? and start_date and end_date index as well ?

[porunov]

I won’t go too deep into investing performance issues of this use-case, but will leave some tips which may help:

• hasLabel step just got multi-query support in master branch. It will be available when JanusGraph 1.0.0 is released. Alternatively, you can use latest commit release to get this optimisation. The optimisation will not be available in any of the releases prior 1.0.0.
• Prior JanusGraph 1.0.0 (which isn’t released yet) you would need to configure parallel-backend-ops thread pool because that thread pool is used for IO operations. In JanusGraph 1.0.0 the approach is moved to async driver capabilities which allows better performance and smaller resources utilisation. Again, either wait for the release or use latest “commit” release (see how to use commit releases here ). Otherwise, ensure you have big enough thread pool to handle your load.
• Read timeouts (both JanusGraph timeouts and Cassandra driver timeouts) can be increased using configuration options.
• Composite and Mixed indexes are applied to the beginning of the query only. After the first traverser hop those types of indexes are not going to be used anymore because they are not designed for such use-cases. Instead Vertex-Centric Edge index designed to traverse indexes edges of the vertex. You could probably take a look at this type of index. However, it might require you to change your model a bit or make some denormalizarion in your data to properly use your index.

Hope other community members can add more tips or help you with your performance issue.

[sharma-kunal]

Thanks for the suggestions. I am going through the points you suggested. One thing I wanted to discuss about query optimization is

g.V().hasLabel("seeker").has("p_uuid", <p_uuid>).outE("worked").as_("e1").inV().inE("worked").as_("e2")\
                .outV().as_("b").outE("worked").where(inV().hasLabel("company").has("company_name", <company_name>)).has("current", 1)\
                .where("e1", lt("e2")).by("start_date").by("end_date").where("e1", gt("e2")).by("end_date").by("start_date")\
                .select("e1", "e2", "b").range_(0,10).toList()

I have this query,

If we break down the query,

1. Look for Seeker Vertex with p_uuid = <p_uuid> (using seeker index, so fast)
2. Select the outward edge from this as e1 (edge has property, start_date, end_date and a current flag 0/1)
3. We will reach the company vertex. Now go to all the Incoming edges of this vertex and select them as e2 (same properties, start_date, end_date, and a current flag 0/1).
4. Now we again reach the Seeker Vertex and we check if the seeker vertex has the vertex with property company_name = <company_name>. For this part, I did this optimization and it helped.

company = g.V().hasLabel("company").has("company_name", <company_name>).next()

.where(inV().is_(company))

This made my response much faster.

Now, the problem is the query after this,

.where("e1", lt("e2")).by("start_date").by("end_date").where("e1", gt("e2")).by("end_date").by("start_date")

Here I check if e1.start_date < e2.end_date and e1.end_date > e2.start_date. This part is taking too much time, I am not sure why. If I have already traversed the edge and have its data, then why is it taking too much time to process this part? Is there any way I can use indexing or any other optimization to make this part faster?

I have start_date and end_date properties already indexed.

[hadoopmarc]

Because of the range() step you only retrieve a small result set. Without where steps anything goes and the 10 required results are found very fast. If the where() steps are included potentially a large resultset has to be retrieved before 10 match the requirements. What happens if you put the where steps with the start_date and end_date on the second line of your query?

[sharma-kunal]

There are 3 cases now,

1. Query 1: Original Query: where() condition at 3rd line

g.V().hasLabel("seeker").has("p_uuid", "<p_uuid>").outE("worked").as_("e1").inV().inE("worked").as_("e2") \
            .outV().as_("b").outE("worked").where(inV().is_(company)).has("current", 1) \
            .where("e1", lt("e2")).by("start_date").by("end_date").where("e1", gt("e2")).by("end_date").by("start_date") \
            .select("e1", "e2", "b").limit(10).toList()

Time it took: Graph Timeout (more than 90 seconds)

2. Query 2: Without where condition

g.V().hasLabel("seeker").has("p_uuid", "<p_uuid>").outE("worked").as_("e1").inV().inE("worked").as_("e2") \
            .outV().as_("b").outE("worked").where(inV().is_(company)).has("current", 1) \
            .select("e1", "e2", "b").limit(10).toList()

Time it took: 5-6 seconds

3. Query 3: where() condition at 2nd line

g.V().hasLabel("seeker").has("p_uuid", "<p_uuid>").outE("worked").as_("e1").inV().inE("worked").as_("e2") \
            .where("e1", lt("e2")).by("start_date").by("end_date").where("e1", gt("e2")).by("end_date").by("start_date") \
            .outV().as_("b").outE("worked").where(inV().is_(company)).has("current", 1) \
            .select("e1", "e2", "b").limit(10).toList()

Time it took: 22 seconds

I have indexed the property, start_date, and end_date in my mixed index backend (elastic). And when I do .profile() for the query, I don't see the index being called. Is the index not being used in this case ?