Merge branch 'main' into BC-6620-migration-pending

docs/schulcloud-server/Coding-Guidelines/event-handling.md

@@ -0,0 +1,102 @@
+# Event Handling
+
+Internal Events are used as a mechanism for Dependency Inversion.
+
+If you are implementing an operation in a module that needs to trigger an operation in another module, that is simple if you can simply import a service. However, if that other module already has a dependency on your module, that would lead to a dependency cycle. In this case, you need to inverse one of the dependencies via events.
+
+The main thing you need to think about, is which module should know about which module(s). This is the dependency, and it only ever can point into one direction. As a general rule of thumb, the module that is more specific, or is changing more frequently, or is less central to the functionality of the system, should have the dependency on the other.
+
+In the following example, the course module has a dependency on the user module, but NOT vice versa.
+
+## How to implement Event Handling
+
+consider the following folder structure
+
+``` txt
+- users
+  - api
+  - domain
+    - services
+    - events
+      - user-deleted.event.ts
+  - repo
+- courses
+  - api
+  - domain
+    - services
+    - handlers
+      - user-deleted.handler.ts
+  - repo
+```
+
+each of the modules needs to import the CqrsModule
+
+``` ts
+// users.module.ts
+import { Module } from '@nestjs/common';
+import { CqrsModule } from '@nestjs/cqrs';
+
+@Module({
+    imports: [CqrsModule],
+    providers: [/* some things here */],
+    exports: [/* some things here */],
+})
+export class GroupModule {}
+```
+
+### Defining an Event
+
+The event is in the end simply a class, containing any data required to handle the event
+
+``` ts
+// users/domain/events/user-deleted.event.ts
+export class UserDeletedEvent {
+    id: EntityId
+
+    constructor(id: EntityId) {
+        this.id = id
+    }
+}
+```
+
+Make sure to make your event public in the index file of your module
+
+### Sending an Event
+
+``` ts
+// users/domain/services/service.ts
+import { EventBus } from '@nestjs/cqrs';
+import { UserDeletedEvent } from '../events';
+
+@Injectable()
+export class Service {
+    constructor(private readonly eventBus: EventBus) {}
+
+    public async delete(userId: EntityId): Promise<void> {
+        doStuffForDeletion()
+
+        await this.eventBus.publish(new UserDeletedEvent(userId));
+    }
+}
+```
+
+### Recieving an Event
+
+``` ts
+// courses/domain/handler/user-deleted.handler.ts
+import { UserDeletedEvent } from '@modules/users';
+import { EventsHandler, IEventHandler } from '@nestjs/cqrs';
+import { SomeService } from '../services'
+
+@Injectable()
+@EventsHandler(UserDeletedEvent)
+export class GroupDeletedHandlerService implements IEventHandler<UserDeletedEvent> {
+    constructor(private readonly someService: SomeService) {}
+
+    public async handle(event: GroupDeletedEvent): Promise<void> {
+        await someService.doSomeStuff()
+    }
+}
+```
+
+Note that the handler should not contain any logic, but only the orchestration of what needs to be done.

docs/schulcloud-server/Coding-Guidelines/modules-submodules.md

@@ -0,0 +1,76 @@
+# Implementation and usage of modules, submodule and barrel files in our project
+
+In this guide, we'll cover how to use modules, submodules and barrel files in our project. These concepts help you organize your code into separate files and directories, making it easier to manage and maintain.
+
+![Module Structure](./../img/Modules-SubModules_background.svg)
+
+## Modules and Submodules
+
+In our project, modules are a way to create separate scopes. This means that interfaces, use cases, services, etc., declared in a module are not visible outside the module unless they are explicitly exported using the `export` keyword. To import a module, you use the `import` keyword followed by the module name. Here's an example:
+
+```typescript
+import { ModuleName } from '@modules/module-name';
+```
+
+Submodules are modules that are part of a larger module. They should be used within the main module. If it is necessary to use parts of the submodule outside the main module, the main module should export this via its barrel file(index.ts):
+
+```typescript
+// @modules/module-name/index.ts
+export { SubmoduleServiceName } from './submodule-name/service.ts';
+```
+
+## Barrel Files
+
+Barrel files are a way to rollup exports from several folders into a single convenient nest-module. The barrel itself is a module file that re-exports selected exports of other submodules.
+
+If you have several related service/interface files in a directory/module that should be publicly accessible, you can create a barrel file to export all these files from the main module again.
+
+Here's an example of a barrel file:
+
+```typescript
+// @modules/module-name/index.ts
+export { PublicService } from './services/public-service.ts';
+export { ServiceInterfaceA, InterfaceB } from './interfaces';
+export { InterfaceC } from './submodule-name/interfaces';
+```
+
+!!! Please don't export everything from a module in the barrel file. Only export the public API of the module. This will make it easier to understand what the module provides and avoid unnecessary dependencies. And don't use wildcard exports like `export * from './services'` in the barrel file.
+
+And here's how you can import from the barrel:
+
+```typescript
+// @modules/other-module-name/service.ts
+import { PublicService, InterfaceOfModule, InterfaceOfSubmodule } from '@modules/module-name';
+```
+
+## Handling Circular Dependencies
+
+Circular dependencies occur when Module A depends on Module B, and Module B also depends on Module A. This can lead to unexpected behavior and hard-to-diagnose bugs.
+
+![Module Structure](./../img/server_area_dependency_v2.svg)
+
+Here are some strategies to handle circular dependencies:
+
+1. **Refactor Your Code**: The best way to handle circular dependencies is to refactor your code to remove them. This might involve moving some code to a new module to break the dependency cycle.
+
+```typescript
+// @modules/moduleC/service.ts
+import { PublicService, InterfaceOfModule, InterfaceOfSubmodule } from '@modules/moduleA';
+import { PublicService, InterfaceOfModule, InterfaceOfSubmodule } from '@modules/moduleB';
+```
+
+2. **Use Interfaces**: If the circular dependency is due to types, you can use interfaces and type-only imports to break the cycle.
+
+```typescript
+// @modules/moduleC/service.ts
+import { type PublicService } from '@modules/moduleA';
+import { type PublicService } from '@modules/moduleB';
+```
+
+3. **Use Events**: If you have a circular dependency between two modules that need to communicate with each other, consider using events to decouple them. This way, one module can emit an event that the other module listens to, without directly importing it.
+
+- <https://documentation.dbildungscloud.dev/docs/schulcloud-server/Coding-Guidelines/event-handling>
+
+- <https://docs.nestjs.com/recipes/cqrs>
+
+Remember, circular dependencies are usually a sign of tightly coupled code and can lead to maintenance issues down the line. It's best to refactor your code to avoid them if possible.

docs/services/tldraw/How it works.md

@@ -0,0 +1,81 @@
+# How it works
+
+## Configuration
+
+- NEST_LOG_LEVEL - logging level
+- FEATURE_TLDRAW_ENABLED - flag determining if tldraw is enabled
+- TLDRAW_URI - address of tldraw server 
+- INCOMING_REQUEST_TIMEOUT - request timeout
+- TLDRAW_DB_URL - mongoDB connection string
+- TLDRAW__SOCKET_PORT - port number for websockets connection
+- TLDRAW__PING_TIMEOUT - timeout for ping-pong during establishing websockets connection
+- TLDRAW__FINALIZE_DELAY - delay in ms before checking if can finalize a tldraw board
+- TLDRAW__GC_ENABLED - if tldraw garbage collector should be enabled
+- TLDRAW__DB_COMPRESS_THRESHOLD - threshold size for tldraw mongo documents compression
+- TLDRAW__MAX_DOCUMENT_SIZE - max size of a single tldraw document in mongo
+- TLDRAW__ASSETS_ENABLED - enables uploading assets to tldraw board
+- TLDRAW__ASSETS_SYNC_ENABLED - enables synch of tldraw board assets with filestorage (no longer used)
+- TLDRAW__ASSETS_MAX_SIZE - maximum asset size in bytes
+- TLDRAW__ASSETS_ALLOWED_MIME_TYPES_LIST - listy of allowed assets MIME types
+- REDIS_URI - Redis connection string
+- TLDRAW_CLIENT_REPLICAS - number of pods for tldraw-client 
+- TLDRAW_SERVER_REPLICAS - number of pods for tldraw-server
+- TLDRAW_ADMIN_API_CLIENT__API_KEY - authorization API key for accessing tldraw controller (delete flow) 
+- TLDRAW_ADMIN_API_CLIENT__BASE_URL - address of tldraw controller (delete flow) 
+
+In order to have deletion functionality fully working you have to fill those feature flags, e.g.:
+- ADMIN_API__ALLOWED_API_KEYS=["7ccd4e11-c6f6-48b0-81eb-abcdef123456"]
+- TLDRAW_ADMIN_API_CLIENT__API_KEY="7ccd4e11-c6f6-48b0-81eb-abcdef123456"
+- TLDRAW_ADMIN_API_CLIENT__BASE_URL="http://localhost:3349"
+
+## Create
+![Create tldraw workflow](./assets/Create TLDRAW.drawio.svg)
+
+Creation of Tldraw starts with creation proccess for Courses and CourseBoard. It has Representation in CourseBoard as card's element (BoardNode in db). After creating Representation of drawing we can enter actual tldraw SPA client (left side of picture).
+
+1. User enters CourseBoard and creates Representation of whiteboard (tldraw) in CourseCard.
+2. Data is saved and feedback with proper creation is given - user can see Representation and can enter whiteboard.
+3. By entering whiteboard user is redirected to SPA tldraw-client.
+4. Tldraw-client is starting WS connection with tldraw-server.
+5. Tldraw-server firstly checks if user has permission to this resource (by checking if user has a permission to Representation of whiteboard -BoardNode).
+    Id of Representation is same as drawingName, which is visible in tldraw-client url.
+6. If user has permission tldraw-server is allowing to remain connection and getting drawing data from separate tldraw-db. If there were no drawing data saved tldraw-server will create it automatically. 
+
+## Usage
+![Usage tldraw workflow](./assets/Use tldraw.drawio.svg)
+
+### Connection
+
+1. user joins tldraw board
+2. tldraw-client connects to one of the tldraw-server pods and tries to establish websocket connection
+3. tldraw-server calls schulcloud-server via HTTP requests to check user permissions, if everything is fine the websocket connection is established
+4. tldraw-server gets stored tldraw board data from mongodb and sends it via websocket to connected users
+5. tldraw-server starts subscribing to Redis PUBSUB channel corresponding to tldraw board name to listen to changes from other pods
+
+### Sending updates/storing data
+
+1. tldraw-client sends user's drawing changes to the tldraw-server via websocket connection
+2. tldraw-server stores the board update in the mongodb - basically creates a diff between what's already stored and what's being updated
+3. tldraw-server pushes the update to correct Redis channel so that clients connected to different pods have synchronized board data
+4. other pods subscribing to Redis channel send updates to their connected clients via websocket whenever they see a new message on Redis channel
+
+## Delete
+![Delete tldraw workflow](./assets/Delete TLDRAW.drawio.svg)
+
+1. User from schulcloud app in CourseBoard deletes whiteboard (tldraw) instance form CardBoard.
+2. Having drawingName sc-server is removing Representation data in sc-database - BoardNodes collection ( drawingName === BoardNode id)
+3. Sc-server is calling tldraw-server via tldraw-management rules in tldraw-server-svc to delete all data that has given id).
+4. After deletion user sees refreshed state of CourseBoard. 
+
+## Assets
+### files upload
+
+Images/GIFs can be uploaded into tldraw whiteboard by every user with access to the board. We use s3 storage to physically store uploaded assets while tldraw only holds URL to a resource.
+
+The files are uploaded by calling schulcloud-api's fileController upload endpoint. This is possible because tldraw is represented as a boardnode called drawing-element. Mongo id of this drawing-element is a roomId used in URL param when connecting to a specific board.
+
+### files deletion
+
+Because of the undo/redo functionality of tldraw (user can basically undo an upload of an image, undo a deletion, then redo upload etc.) we needed a way to clean up unused assets from the storage. We could not use soft delete/restore endpoints every time undo/redo happens due to various issues with performance/user experience and technical challenges that arose when testing different scenarios. We decided to go with cron job solution: once per day, at midnight by default, we would go through each board stored in database, get every asset that's stored as URL but no longer used as an active drawing and then delete all of them via amqp call to filesStorage service.
+
+For implementation details, take a look at: tldraw-files.console.ts.

docs/services/tldraw/Local setup.md

@@ -0,0 +1,18 @@
+# Local setup
+
+### To run tldraw locally:
+
+1. Run all of the apps needed for schulcloud like mongodb, backend, frontend, file storage etc.
+2. Run redis i.e. in a docker container, it will work on localhost:6379 by default which is what the REDIS_URI env var is set to
+3. On schulcloud-server repo: npm run nest:start:tldraw:dev
+4. On tldraw-client repo: npm run dev
+
+### Create new whiteboard:
+
+1. Go to a course
+2. Go to 'Column board'
+3. Create a new card and a new 'Whiteboard' element within it, then click it
+4. A new browser tab with URL like: http://localhost:4000/tldraw?roomName=65c37329b2f97cc714d31c00 will open
+5. Change the port part from 4000 to 3046, which is the default port of tldraw-client app
+6. You should see a working tldraw whiteboard now
+

docs/services/tldraw/Technical details.md

@@ -0,0 +1,90 @@
+# Technical details
+
+## Backend
+We are using pure Web Sockets implementation to establish connection between client and server. The reason why we used pure websockets is because tldraw-client is using y-websockets from Yjs library, that does not connect with socket.io web sockets. We also have to implement broadcasting mechanism to provide stateless solution. To achive that goal we decided to use Redis. We used ioredis library to connect to our Redis instance. Everytime user make some changes at first it is handled to the server instance that he is connected to an then this change is send to the channel with the name of the board and servers that also operate that board are listening on this channel so they can recive messages from other servers and provide those changes to users that are connected to this pod. We added the same mechanism for awareness channel so every user from every pod can see other users cursours.
+
+
+Tldraw is deployed as a separate application from schoulcloud-server working on the same namespace as schoulcloud-server. On the backend side we have added couple new resources:
+
+- tldraw-deployment - deployment for tldraw-server's instances.
+- tldraw-server-svc - service for tldraw-service to communicate with tldraw-client (WS) and schoulcloud-server (management e.g. deletion of data)
+- tldraw-svc-monitor - service to collect metrics from tldraw. Apart from typical metrics like request time we also added two application-level metrics:
+  - sc_tldraw_users - number of active users on boards
+  - sc_tldraw_boards - number of active boards
+- tldraw-ingress - for steering web external traffic to tldraw-server (for now management rules in tldraw-server-svc are closed from external clients) 
+
+
+### Tldraw-server code structure
+
+- tldraw.ws.service.ts - main service responsible for establishing web socket connection as well as saving data to database. Responsibe for Redis communication.
+- tldraw.controller.ts - controller that expose HTTP deletion method outside the tldraw-server application.
+- tldraw.server.ts - service used by TldrawController.
+- y-mongodb.ts - main adapter to connect with mongodb, provides transaction mechanism, calucalate diffs between revision and to apply updates.
+- tldraw-board.repo.ts - repository object to connect TldrawWsService and YMongodb.
+- tldraw.repo.ts - repository used by TldrawService to find and delete boards from database.
+- ws-shared-doc.do.ts - main structure representing tldraw drawing during web socket communication. it holds all the web-socket addresses that are connected to this board, so we can inform all the connected clients about changes.  
+- tldraw-drawing.entity.ts - object representing tldraw drawingn entity in database.
+- metrics.service.ts - service resonsible for storing application-level metrics.
+
+
+On the backend side we are also using Yjs library to store tldraw board in memory and to calculate diffs after the board is changed.
+
+## Frontend
+
+### Key Files
+- stores/setup.ts – this file provides a real-time collaboration environment for a drawing application using the WebSocket and Yjs libraries.
+- hooks/useMultiplayerState.ts – custom hook for managing multiplayer state.
+- App.tsx – main application component integrating Tldraw and multiplayer state.
+
+### Frontend Technologies
+
+The frontend of the project is built using React and leverages various libraries and tools for enhanced functionality. Here is an overview of the key frontend technologies:
+
+- React: A JavaScript library for building user interfaces.
+- Yjs: A real-time collaboration framework for synchronizing shared state.
+- Tldraw: A library for drawing functionalities in the application. We use the old version of tldraw:  https://github.com/tldraw/tldraw-v1, after the tldraw team releases the official update of the new version, we will work on the new version and integrate it with the needs of our users. 
+
+### State Managment
+
+1. Yjs is integrated into the project for real-time collaboration. The central state (shapes, bindings, assets) is managed using Yjs maps.
+2. store.ts handles the configuration of Yjs, WebSocket connections, and provides centralized maps for shapes, bindings, and assets
+3. useMultiplayerState.ts -This hook manages the multiplayer state, including loading rooms, handling file system operations, and updating Yjs maps:
+   - Mounting and handling changes in Tldraw App.
+   - Presence management and user updates.
+   - File system operations like opening and saving projects.
+
+#### useTldrawUiSanitizer.ts 
+This hook is designed to observe changes in the DOM, specifically targeting certain buttons and a horizontal rule (< hr>), and hides them if they match a specific ID pattern. We hide this elements and left just only Language and Keyboard shortcuts.
+
+#### Event Handling
+   - onMount: Handles mounting of the Tldraw app.
+   - onChangePage: Manages page changes and updates Yjs maps.
+   - onUndo and onRedo: Handle undo and redo operations.
+   - onChangePresence: Manages presence changes in the collaborative environment.
+   - onAssetCreate: This function is triggered when a user attempts to upload an asset (like an image or a file).
+
+#### Useful links
+- https://tldraw.dev/  - documentation for the new version of tldraw
+
+- https://old.tldraw.com/ - tldraw live application
+
+- https://github.com/tldraw/tldraw-v1 - tldraw v1 repo
+
+- https://github.com/MaxNoetzold/y-mongodb-provider - code from this package was used to add mongodb as a persistence to tldraw
+
+- https://discord.com/invite/SBBEVCA4PG discord channel with open questions and answers
+
+- https://grafana.dbildungscloud.dev/d/b6b28b2b-3129-4772-8102-e32981d2c2e3/devops-tldraw-metrics?orgId=1&refresh=1m&var-source=sc-dev-dbc&var-env=main&var-env=tldraw-debugging - grafana v
+
+- https://grafana.dbildungscloud.org/d/b6b28b2b-3129-4772-8102-e32981d2c2e0/devops-tldraw?orgId=1&from=now-6h&to=now&refresh=1m - grafana metrics
+
+- https://github.com/nimeshnayaju/yjs-tldraw - yjs with tldraw POC
+
+- https://github.com/yjs/y-websocket/tree/master/bin - yjs/y-websocket repo
+
+- https://github.com/erdtool/yjs-scalable-ws-backend/tree/main - Yjs scalable WS backend with redis example
+
+- https://teamchat.dbildungscloud.de/channel/G9hJWv92zXEESKK3X - rocketchat discussion "tldraw syncronisation for release again"
+
+- https://teamchat.dbildungscloud.de/group/SagK4sCyujhu6yZr8 - rocketchat discussion "Tldraw deployment"s
+