split into frontend and backend to support multiple sigma versions in parallel

[M3NIX]

This pull request tries to solve the current problem of sigconverter that different sigma backends have different dependency versions to pySigma / sigma-cli. Therefore we have to chose one version at the moment and can not include all backends or easily upgrade to newer sigma versions.

To solve this, this PR includes the following changes:

1. splitting the code into frontend and backend
2. multiple sigconverter backends will run on different ports with an isolated python environment to support all different sigma versions
3. the frontend now accepts a version parameter in the api requests and depending on that the request will be sent to the matching backend port
4. a script (backend/setup-sigma-versions.sh) is included which will automatically fetch the latest 10 versions of sigma-cli and creates an isolated python environment for them. in that environment all plugins with their latest compatible version will be installed. non compatible versions will be skipped during the installation.
5. a new python package manager is used (uv) to automatically check the compatibility of all existing versions of the plugins
6. a settings area on the webpage is included to switch between the sigma-cli versions
7. implements btw all the required api endpoints for Add More API Endpoints #24

If we would schedule to weekly (or daily) build the docker container which runs the setup script we would automatically have the latest versions of sigma-cli and the plugins installed without any further manual need to add new plugins.

Happy to get feedback from you @josehelps @nasbench @MHaggis :)

[M3NIX]

This screenshot shows the new settings area which opens when clicking the icon to the right of the cli section:

[josehelps]

@M3NIX apologies for not getting to this sooner, I tried doing a local build using docker of the branch and ran into a issue:

2024-11-05 14:36:59 Launching sigconverter backend for sigma version: 0.7.5
2024-11-05 14:36:59 Launching sigconverter backend for sigma version: 0.7.6
2024-11-05 14:36:59 Launching sigconverter backend for sigma version: 0.7.7
2024-11-05 14:36:59 Launching sigconverter backend for sigma version: 0.7.8
2024-11-05 14:36:59 Launching sigconverter backend for sigma version: 0.7.9
2024-11-05 14:36:59 Launching sigconverter backend for sigma version: 1.0.0
2024-11-05 14:36:59 Launching sigconverter backend for sigma version: 1.0.1
2024-11-05 14:36:59 Launching sigconverter backend for sigma version: 1.0.2
2024-11-05 14:36:59 Launching sigconverter backend for sigma version: 1.0.3
2024-11-05 14:36:59 Launching sigconverter backend for sigma version: 1.0.4
2024-11-05 14:36:59 Using CPython 3.11.4 interpreter at: /usr/local/bin/python
2024-11-05 14:36:59 Creating virtual environment at: .venv
2024-11-05 14:37:00 Traceback (most recent call last):
2024-11-05 14:37:00   File "/app/backend/./backend.py", line 10, in <module>
2024-11-05 14:37:00     from sigma.conversion.base import Backend
2024-11-05 14:37:00   File "/app/backend/0.7.8/.venv/lib/python3.11/site-packages/sigma/conversion/base.py", line 5, in <module>
2024-11-05 14:37:00     from pyparsing import Set
2024-11-05 14:37:00 ImportError: cannot import name 'Set' from 'pyparsing' (/app/backend/0.7.8/.venv/lib/python3.11/site-packages/pyparsing/__init__.py)
2024-11-05 14:37:00 Traceback (most recent call last):
2024-11-05 14:37:00   File "/app/backend/./backend.py", line 10, in <module>
2024-11-05 14:37:00     from sigma.conversion.base import Backend
2024-11-05 14:37:00   File "/app/backend/0.7.6/.venv/lib/python3.11/site-packages/sigma/conversion/base.py", line 5, in <module>
2024-11-05 14:37:00     from pyparsing import Set
2024-11-05 14:37:00 ImportError: cannot import name 'Set' from 'pyparsing' (/app/backend/0.7.6/.venv/lib/python3.11/site-packages/pyparsing/__init__.py)
2024-11-05 14:37:00 Traceback (most recent call last):
2024-11-05 14:37:00   File "/app/backend/./backend.py", line 10, in <module>
2024-11-05 14:37:00     from sigma.conversion.base import Backend
2024-11-05 14:37:00   File "/app/backend/0.7.7/.venv/lib/python3.11/site-packages/sigma/conversion/base.py", line 5, in <module>
2024-11-05 14:37:00     from pyparsing import Set
2024-11-05 14:37:00 ImportError: cannot import name 'Set' from 'pyparsing' (/app/backend/0.7.7/.venv/lib/python3.11/site-packages/pyparsing/__init__.py)
2024-11-05 14:37:00 Traceback (most recent call last):
2024-11-05 14:37:00   File "/app/backend/./backend.py", line 10, in <module>
2024-11-05 14:37:00     from sigma.conversion.base import Backend
2024-11-05 14:37:00   File "/app/backend/0.7.9/.venv/lib/python3.11/site-packages/sigma/conversion/base.py", line 5, in <module>
2024-11-05 14:37:00     from pyparsing import Set
2024-11-05 14:37:00 ImportError: cannot import name 'Set' from 'pyparsing' (/app/backend/0.7.9/.venv/lib/python3.11/site-packages/pyparsing/__init__.py)
2024-11-05 14:37:00 Traceback (most recent call last):
2024-11-05 14:37:00   File "/app/backend/./backend.py", line 10, in <module>
2024-11-05 14:37:00     from sigma.conversion.base import Backend
2024-11-05 14:37:00   File "/app/backend/0.7.5/.venv/lib/python3.11/site-packages/sigma/conversion/base.py", line 5, in <module>
2024-11-05 14:37:00     from pyparsing import Set
2024-11-05 14:37:00 ImportError: cannot import name 'Set' from 'pyparsing' (/app/backend/0.7.5/.venv/lib/python3.11/site-packages/pyparsing/__init__.py)
2024-11-05 14:37:00 Installed 12 packages in 347ms
2024-11-05 14:37:00 WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
2024-11-05 14:37:00  * Running on all addresses (0.0.0.0)
2024-11-05 14:37:00  * Running on http://127.0.0.1:8000
2024-11-05 14:37:00  * Running on http://172.17.0.2:8000

Looks related to some of the deps required by pysigma specifically from pyparsing import Set .. I tried messing with the backend/setup-sigma-versions.sh but had no luck, any clue?

[M3NIX]

It seems to be a bug/issue with the latest pyparsing version which was now pulled in automatically during build time.
Here is an issue related to it: pyparsing/pyparsing#576
I will look into it so that for older pySigma versions, older pyparsing version will be used. The latest pySigma version was adjusted to match the latest pyparsing lib: SigmaHQ/pySigma#289

I will ping you if a new working version is available :)

[M3NIX]

@josehelps
Should be fixed now. As Thomas stated in the issue it was an accidental unused auto import so I added a simple sed to remove it on older versions. Can you check if you can now successfully build locally? You should only have to run:

docker build -t m3nix/sigconverter .
docker run -d -p 8000:8000 m3nix/sigconverter

[josehelps]

Worked flawlessly thank you for the update.

[josehelps]

LGTM