Skip to content

Commit

Permalink
MIR-1320 MIR support for Solr authentication and Solr cloud (#1012)
Browse files Browse the repository at this point in the history
* mir support for solr authentication and solr cloud

* add tika server to wizard

* allow admin and indexer to search and admin to index


---------

Co-authored-by: Thomas Scheffler <[email protected]>
  • Loading branch information
sebhofmann and yagee-de authored Jul 31, 2024
1 parent 703af95 commit 166ef63
Show file tree
Hide file tree
Showing 21 changed files with 528 additions and 12 deletions.
9 changes: 9 additions & 0 deletions .env
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,12 @@ MIR_XMS=1024m

SOLR_HTTP=8290
SOLR_DATA=./docker/solr-data/
SOLR_ADMIN_USER=admin
SOLR_ADMIN_PASSWORD=alleswirdgut
SOLR_SEARCH_USER=searcher
SOLR_SEARCH_PASSWORD=Alleswirdgut!?
SOLR_INDEX_USER=indexer
SOLR_INDEX_PASSWORD=Alleswirdgut!?


TIKA_PORT=8294
4 changes: 0 additions & 4 deletions .gitmodules

This file was deleted.

14 changes: 12 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,12 @@ This guide addresses developers. Thats why you run it in 'dev' profile!
</properties>
</profile>
```
- initialize solr configuration using `git submodule update --init --recursive`
- to start solr, go to mir-webapp
- install solr with the command: `mvn -Pdev solr-runner:copyHome`
- run solr with the command: `mvn -Pdev solr-runner:start`
- The default users are `admin`, `indexer` and `searcher` with password `alleswirdgut`
- In the wizard of the application you need to check `Erstelle SOLR-Kerne per Solr-Cloud rest-API` and the configure
user options with the above-mentioned users and passwords.
- stop solr with the command: `mvn -Pdev solr-runner:stop`
- update solr with the command: `mvn -Pdev solr-runner:stop solr-runner:copyHome solr-runner:start`
- to starting up a servlet container in development environment go back to mir folder
Expand All @@ -48,6 +50,14 @@ The docker container has its own install script which uses the environment varia
### Environment Variables
| Property | Default, required | Description |
|--------------------------|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ENABLE_SOLR_CLOUD | false | If true the Solr Cloud mode is enabled. (solr cores will be created on install) |
| SOLR_ADMIN_USER | none | The username for the Solr Admin. (will be used for admin commands like creating cores) |
| SOLR_ADMIN_PASSWORD | none | The password for the Solr Admin. |
| SOLR_INDEX_USER | none | The username for the Solr Indexer. (will be used for indexing) |
| SOLR_INDEX_PASSWORD | none | The password for the Solr Indexer. |
| SOLR_SEARCH_USER | none | The username for the Solr Searcher. (will be used for searching) |
| SOLR_SEARCH_PASSWORD | none | The password for the Solr Searcher. |
| TIKASERVER_URL | none | The URL to the Tika Server. Same as MCR.Solr.Tika.ServerURL in mycore.properties. (also sets `MCR.Solr.FileIndexStrategy` to `org.mycore.solr.index.file.tika.MCRTikaSolrFileStrategy`) |
| SOLR_URL | none, required | The URL to the SOLR Server. Same as MCR.Solr.ServerURL in mycore.properties. |
| SOLR_CORE | mir | The name of the Solr main core. Same as MCR.Solr.Core.main.Name in mycore.properties. |
| SOLR_CLASSIFICATION_CORE | mir-classifications | The name of the Solr classification core. Same as MCR.Solr.Core.classification.Name in mycore.properties. |
Expand All @@ -74,7 +84,7 @@ To fix this you can set the docker property `FIX_FILE_SYSTEM_RIGHTS` to `true`.
mounted volumes to `mcr` and the container will start without errors.

## `mir-solr` Docker-Container
The docker container creates the required solr cores if they do not exist.
The docker container starts solr in cloud mode. It preconfigures the users with the environment variables (see table above).

### Mount Points

Expand Down
21 changes: 21 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,19 @@ services:
- ${SOLR_DATA}:/var/solr/data
ports:
- ${SOLR_HTTP}:8983
environment:
- SOLR_ADMIN_USER=${SOLR_ADMIN_USER}
- SOLR_ADMIN_PASSWORD=${SOLR_ADMIN_PASSWORD}
- SOLR_SEARCH_USER=${SOLR_SEARCH_USER}
- SOLR_SEARCH_PASSWORD=${SOLR_SEARCH_PASSWORD}
- SOLR_INDEX_USER=${SOLR_INDEX_USER}
- SOLR_INDEX_PASSWORD=${SOLR_INDEX_PASSWORD}
tika:
container_name: ${NAME}-tika
image: apache/tika:2.9.2.1-full
restart: unless-stopped
ports:
- ${TIKA_PORT}:9998
mir:
container_name: ${NAME}-mir
build: ./
Expand All @@ -34,6 +47,14 @@ services:
- SOLR_URL=http://solr:8983
- SOLR_CORE=mir
- SOLR_CLASSIFICATION_CORE=mir-classifications
- ENABLE_SOLR_CLOUD=true
- SOLR_ADMIN_USER=${SOLR_ADMIN_USER}
- SOLR_ADMIN_PASSWORD=${SOLR_ADMIN_PASSWORD}
- SOLR_SEARCH_USER=${SOLR_SEARCH_USER}
- SOLR_SEARCH_PASSWORD=${SOLR_SEARCH_PASSWORD}
- SOLR_INDEX_USER=${SOLR_INDEX_USER}
- SOLR_INDEX_PASSWORD=${SOLR_INDEX_PASSWORD}
- TIKASERVER_URL=http://tika:9998/
- XMX=${MIR_XMX}
- XMS=${MIR_XMS}
- MIR_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8295
Expand Down
59 changes: 59 additions & 0 deletions docker-entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,15 @@ function setOrAddProperty() {
KEY=$1
VALUE=$2

if [ -z "$VALUE" ]; then
# remove property
sed -ri "/$KEY/d" "${MYCORE_PROPERTIES}"
return
elif [ -z "$KEY" ]; then
echo "No Key given. Skip setting property."
return
fi

if grep -q "$KEY=" "${MYCORE_PROPERTIES}" ; then
ESCAPED_KEY=$(echo "${KEY}" | sed 's/\//\\\//g')
ESCAPED_VALUE=$(echo "${VALUE}" | sed 's/\//\\\//g')
Expand Down Expand Up @@ -116,6 +125,45 @@ function setDockerValues() {
setOrAddProperty "MCR.JPA.DefaultSchema" "${HIBERNATE_SCHEMA}"
fi

if [ -n "${SOLR_ADMIN_USER}" ]; then
setOrAddProperty "MCR.Solr.Server.Auth.Admin.Class" "org.mycore.solr.auth.MCRSolrBasicPropertyAuthentication"
setOrAddProperty "MCR.Solr.Server.Auth.Admin.Username" "${SOLR_ADMIN_USER}"
setOrAddProperty "MCR.Solr.Server.Auth.Admin.Password" "${SOLR_ADMIN_PASSWORD}"
else
setOrAddProperty "MCR.Solr.Server.Auth.Admin.Class"
setOrAddProperty "MCR.Solr.Server.Auth.Admin.Username"
setOrAddProperty "MCR.Solr.Server.Auth.Admin.Password"
fi


if [ -n "${SOLR_INDEX_USER}" ]; then
setOrAddProperty "MCR.Solr.Server.Auth.Index.Class" "org.mycore.solr.auth.MCRSolrBasicPropertyAuthentication"
setOrAddProperty "MCR.Solr.Server.Auth.Index.Username" "${SOLR_INDEX_USER}"
setOrAddProperty "MCR.Solr.Server.Auth.Index.Password" "${SOLR_INDEX_PASSWORD}"
else
setOrAddProperty "MCR.Solr.Server.Auth.Index.Class"
setOrAddProperty "MCR.Solr.Server.Auth.Index.Username"
setOrAddProperty "MCR.Solr.Server.Auth.Index.Password"
fi

if [ -n "${SOLR_SEARCH_USER}" ]; then
setOrAddProperty "MCR.Solr.Server.Auth.Search.Class" "org.mycore.solr.auth.MCRSolrBasicPropertyAuthentication"
setOrAddProperty "MCR.Solr.Server.Auth.Search.Username" "${SOLR_SEARCH_USER}"
setOrAddProperty "MCR.Solr.Server.Auth.Search.Password" "${SOLR_SEARCH_PASSWORD}"
else
setOrAddProperty "MCR.Solr.Server.Auth.Search.Class"
setOrAddProperty "MCR.Solr.Server.Auth.Search.Username"
setOrAddProperty "MCR.Solr.Server.Auth.Search.Password"
fi

if [ -n "${TIKASERVER_URL}" ]; then
setOrAddProperty "MCR.Solr.Tika.ServerURL" "${TIKASERVER_URL}"
setOrAddProperty "MCR.Solr.FileIndexStrategy" "org.mycore.solr.index.file.tika.MCRTikaSolrFileStrategy"
else
setOrAddProperty "MCR.Solr.Tika.ServerURL"
setOrAddProperty "MCR.Solr.FileIndexStrategy"
fi

setOrAddProperty "MCR.JPA.Hbm2ddlAuto" "update"
setOrAddProperty "MCR.JPA.PersistenceUnit.mir.Class" "org.mycore.backend.jpa.MCRSimpleConfigPersistenceUnitDescriptor"
setOrAddProperty "MCR.JPA.PersistenceUnitName" "mir"
Expand Down Expand Up @@ -164,6 +212,17 @@ function setUpMyCoRe {
/opt/mir/mir/bin/mir.sh create configuration directory
setDockerValues
setupLog4jConfig

# ENABLE_SOLR_CLOUD
if [[ "$ENABLE_SOLR_CLOUD" == "true" ]]
then
echo "upload local config set for main" >> "${MCR_CONFIG_DIR}setup-solr-cloud.txt"
echo "upload local config set for classification" >> "${MCR_CONFIG_DIR}setup-solr-cloud.txt"
echo "create collection for core main" >> "${MCR_CONFIG_DIR}setup-solr-cloud.txt"
echo "create collection for core classification" >> "${MCR_CONFIG_DIR}setup-solr-cloud.txt"
/opt/mir/mir/bin/mir.sh process /mcr/home/setup-solr-cloud.txt
fi

/opt/mir/mir/bin/setup.sh
}

Expand Down
4 changes: 4 additions & 0 deletions mir-cli/src/main/config/setup-solrcloud.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
upload local config set for main
upload local config set for classification
create collection for core main
create collection for core classification
12 changes: 12 additions & 0 deletions mir-it/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,18 @@
</environmentVariables>
</configuration>
<executions>
<execution>
<id>setup-solr-cloud</id>
<goals>
<goal>exec</goal>
</goals>
<phase>pre-integration-test</phase>
<configuration>
<workingDirectory>${project.build.directory}/dependency/mir-cli-${project.version}</workingDirectory>
<executable>bin${file.separator}mir.${script.suffix}</executable>
<arguments>process ${project.build.directory}/dependency/mir-cli-${project.version}/config/setup-solrcloud.txt</arguments>
</configuration>
</execution>
<execution>
<id>setup-cli</id>
<goals>
Expand Down
12 changes: 12 additions & 0 deletions mir-it/src/test/integration/mycore.properties
Original file line number Diff line number Diff line change
@@ -1,2 +1,14 @@
MCR.Solr.ServerURL=http\://localhost\:${solr.port}/
MCR.Solr.DelayIndexing_inMS=200

MCR.Solr.Server.Auth.Admin.Class=org.mycore.solr.auth.MCRSolrBasicPropertyAuthentication
MCR.Solr.Server.Auth.Admin.Password=alleswirdgut
MCR.Solr.Server.Auth.Admin.Username=admin
MCR.Solr.Server.Auth.Index.Class=org.mycore.solr.auth.MCRSolrBasicPropertyAuthentication
MCR.Solr.Server.Auth.Index.Password=alleswirdgut
MCR.Solr.Server.Auth.Index.Username=indexer
MCR.Solr.Server.Auth.Search.Class=org.mycore.solr.auth.MCRSolrBasicPropertyAuthentication
MCR.Solr.Server.Auth.Search.Password=alleswirdgut
MCR.Solr.Server.Auth.Search.Username=searcher


Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ protected SolrClient getSolrClient() {

protected static LukeResponse getLukeResponse(Core core) throws IOException, SolrServerException {
LukeRequest request = new LukeRequest();
request.setBasicAuthCredentials("admin", "alleswirdgut");
request.setNumTerms(0);
request.setShowSchema(false);
final LukeResponse lukeResponse = request.process(SOLR_CLIENT, core.getCoreName());
Expand Down
1 change: 0 additions & 1 deletion mir-webapp/src/main/solr
Submodule solr deleted from 1f1a29
7 changes: 7 additions & 0 deletions mir-webapp/src/main/solr/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
FROM solr:8.11
USER root
RUN apt-get update && \
apt-get -y install sudo
COPY --chown=root:root docker-entrypoint.sh ./
RUN chmod 555 docker-entrypoint.sh
CMD ["bash", "docker-entrypoint.sh"]
108 changes: 108 additions & 0 deletions mir-webapp/src/main/solr/docker-entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
#!/usr/bin/bash
#
# This file is part of *** M y C o R e ***
# See http://www.mycore.de/ for details.
#
# MyCoRe is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# MyCoRe is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with MyCoRe. If not, see <http://www.gnu.org/licenses/>.
#

set -e
unset SOLR_USER SOLR_UID SOLR_GROUP SOLR_GID \
SOLR_CLOSER_URL SOLR_DIST_URL SOLR_ARCHIVE_URL SOLR_DOWNLOAD_URL SOLR_DOWNLOAD_SERVER SOLR_KEYS SOLR_SHA512

function fixDirectoryRights() {
find "$1" \! -user "$2" -exec chown "$2:$2" '{}' +
}

echo "Running solr entry script as user: $(whoami)"
if [ "$(id -u)" -eq 0 ]; then
fixDirectoryRights /var/solr/ solr
exec /usr/bin/sudo -E -u solr "PATH=$PATH" "$(pwd)/$0";
exit 0;
fi

function solrpass() {
printf "%s %s" "$(echo -n "$2$1"|openssl dgst -sha256 -binary|openssl dgst -sha256 -binary|openssl base64)" "$(echo -n "$2"|openssl base64)"
}

secruity_json=/var/solr/data/security.json;


echo "{" > $secruity_json
echo " \"authentication\":{" >> $secruity_json
echo " \"blockUnknown\": true," >> $secruity_json
echo " \"class\":\"solr.BasicAuthPlugin\"," >> $secruity_json
echo " \"credentials\": {" >> $secruity_json

if [ -n "$SOLR_SEARCH_USER" ]; then
echo " \"${SOLR_SEARCH_USER}\":\"$(solrpass $SOLR_SEARCH_PASSWORD $(openssl rand 10))\"," >> $secruity_json
fi

if [ -n "$SOLR_INDEX_USER" ]; then
echo " \"${SOLR_INDEX_USER}\":\"$(solrpass $SOLR_INDEX_PASSWORD $(openssl rand 10))\"," >> $secruity_json
fi

if [ -n "$SOLR_ADMIN_USER" ]; then
echo " \"${SOLR_ADMIN_USER}\":\"$(solrpass $SOLR_ADMIN_PASSWORD $(openssl rand 10))\"" >> $secruity_json
fi

echo " }," >> $secruity_json
echo " \"realm\":\"My Solr users\"," >> $secruity_json
echo " \"forwardCredentials\": false" >> $secruity_json
echo " }," >> $secruity_json
echo " \"authorization\":{" >> $secruity_json
echo " \"class\":\"solr.RuleBasedAuthorizationPlugin\"," >> $secruity_json
echo " \"permissions\":[" >> $secruity_json

if [ -n "$SOLR_SEARCH_USER" ]; then
echo " {" >> $secruity_json
echo " \"name\":\"read\"," >> $secruity_json
echo " \"role\":[\"searcher\",\"indexer\",\"admin\"]" >> $secruity_json
echo " }," >> $secruity_json
fi

if [ -n "$SOLR_INDEX_USER" ]; then
echo " {" >> $secruity_json
echo " \"name\":\"update\"," >> $secruity_json
echo " \"role\":[\"indexer\",\"admin\"]" >> $secruity_json
echo " }," >> $secruity_json
fi

if [ -n "$SOLR_ADMIN_USER" ]; then
echo " {" >> $secruity_json
echo " \"name\":\"all\"," >> $secruity_json
echo " \"role\":\"admin\"" >> $secruity_json
echo " }" >> $secruity_json
fi
echo "]," >> $secruity_json
echo " \"user-role\":{" >> $secruity_json

if [ -n "$SOLR_SEARCH_USER" ]; then
echo " \"${SOLR_SEARCH_USER}\":\"searcher\"," >> $secruity_json
fi

if [ -n "$SOLR_INDEX_USER" ]; then
echo " \"${SOLR_INDEX_USER}\":\"indexer\"," >> $secruity_json
fi

if [ -n "$SOLR_ADMIN_USER" ]; then
echo " \"${SOLR_ADMIN_USER}\":\"admin\"" >> $secruity_json
fi

echo " }" >> $secruity_json
echo " }" >> $secruity_json
echo "}" >> $secruity_json

(/opt/docker-solr/scripts/wait-for-solr.sh;/opt/solr/bin/solr zk cp $secruity_json zk:security.json -z localhost:9983)&
/opt/docker-solr/scripts/solr-foreground -c;
42 changes: 42 additions & 0 deletions mir-webapp/src/main/solr/security.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
{
"authentication": {
"blockUnknown": true,
"class": "solr.BasicAuthPlugin",
"credentials": {
"searcher": "0QWhQdOGUMTkjxqE0rPodHKa2gUEnRZnZ837YP/L4aw= 7A==",
"indexer": "0QWhQdOGUMTkjxqE0rPodHKa2gUEnRZnZ837YP/L4aw= 7A==",
"admin": "0QWhQdOGUMTkjxqE0rPodHKa2gUEnRZnZ837YP/L4aw= 7A=="
},
"realm": "My Solr users",
"forwardCredentials": false
},
"authorization": {
"class": "solr.RuleBasedAuthorizationPlugin",
"permissions": [
{
"name": "read",
"role": [
"searcher",
"indexer",
"admin"
]
},
{
"name": "update",
"role": [
"indexer",
"admin"
]
},
{
"name": "all",
"role": "admin"
}
],
"user-role": {
"searcher": "searcher",
"indexer": "indexer",
"admin": "admin"
}
}
}
Loading

0 comments on commit 166ef63

Please sign in to comment.