How to utilise sharding

[coolbung]

Is there any documentation on how sharding can be set up in the Sunbird RC database ?

@dileepbapat
cc @sujitrs

[dileepbapat]

Please refer to configuration here However you may need to check the domain dependency and customize the Shard Advisor if needed (multiple entities and deps).

[pramodkvarma]

We should definitely document this.

[gamemaker1]

Hi @coolbung,

I see the following relevant portion in the sample config file:

# Choose "none" as the propertyName if you don't want any shards. Otherwise
# implement IShardAdvisor interface in your own ways. An example, SerialNumShardAdvisor
# has been provided for reference. If you'd like to use it, set value "serialNum" here.
# Note that "serialNum" must then be part of the payload.
shardProperty: ${database_shardProperty:none}

# This property is instruction to use the shard advisor.
# Values could be io.opensaber.registry.sink.shard.DefaultShardAdvisor, OR
# io.opensaber.registry.sink.shard.SerialNumberShardAdvisor OR
# absolute class name of your advisor class.
# If this property not provided, advisor is set to DefaultShardAdvisor
shardAdvisorClassName: io.opensaber.registry.sink.shard.DefaultShardAdvisor

connectionInfo:
  - # shardId, shardlabel must be a unique identifier to each connection.
    shardId: shard1

    # shardLabel is used as prefix for a uuid. Could be any alpha numeric string
    # Example of a record identifier: 1-60f76147-0acd-4dff-b75a-2d811787d04d
    # Note '1' is the label used for record identifier
    # shardLabel is not stored in the database
    shardLabel: 1

    # The format of the URI can be learnt from the following links -
    #     * 1. Graph database (Neo4J)
    #     ** https://github.com/SteelBridgeLabs/neo4j-gremlin-bolt
    #     ** Example : bolt://localhost:7687
    #     * 2. Relational databases (Postgresql, HSQLDB, H2, MariaDB, MySQL, MSSQLServer)
    #     ** http://sqlg.org/docs/2.0.0-SNAPSHOT/
    #     ** Example - Postgres - jdbc:postgresql://localhost:5432/yourdb
    #     * 3. NoSQL  databases (Cassandra)
    #     ** Example - cassandra - jdbc:cassandra://localhost:9160/yourdb
    uri: ${connectionInfo_uri:jdbc:postgresql://localhost:5432/registry}

    username: ${connectionInfo_username:postgres}
    password: ${connectionInfo_password:postgres}

    # Any other shard information follows...

You could provide your config in JSON format using the SPRING_APPLICATION_JSON environment variable. A sample configuration is as follows:

{
	"shardProperty": "serialNum",
	"shardAdvisorClassName": "io.opensaber.registry.sink.shard.DefaultShardAdvisor",
	"connectionInfo": [
		{
			"shardId": "shard-1",
			"shardLabel": "1",
			"uri": "jdbc:postgresql://db:5432/registry",
			"username": "postgres",
			"password": "postgres"
		}
	],
	"elastic": {
		"search": {
			"connection_url": "es:9200"
		}
	},
	"search": {
		"providerName": "io.opensaber.registry.service.ElasticSearchService"
	},
	"keycloak": {
		"realm": "sunbird-rc",
		"resource": "registry-frontend",
		"auth-server-url": "http://kc:8080/auth"
	},
	"keycloak-admin": {
		"client-id": "admin-api",
		"client-secret": "<...>"
	},
	"claims": {
		"url": "http://cs:8082"
	}
}

So if you are using docker, in the docker file (like this one), you would set the following in the environment section of the registry container:

- SPRING_APPLICATION_JSON='{"shardProperty":"serialNum","shardAdvisorClassName":"io.opensaber.registry.sink.shard.DefaultShardAdvisor","connectionInfo":[{"shardId":"shard-1","shardLabel":"1","uri":"jdbc:postgresql://db:5432/registry","username":"postgres","password":"postgres"}],"elastic":{"search":{"connection_url":"es:9200"}},"search":{"providerName":"io.opensaber.registry.service.ElasticSearchService"},"keycloak":{"realm":"sunbird-rc","resource":"registry-frontend","auth-server-url":"http://kc:8080/auth"},"keycloak-admin":{"client-id":"admin-api","client-secret":"<...>"},"claims":{"url":"http://cs:8082"}}'

Hope this helps.