Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EW-1061 TBA #5330

Draft
wants to merge 141 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
cc5da6f
EW-1060 testing the new export via a test endpoint
Fshmit Nov 7, 2024
7729acc
EW-1060 moved linked task to lesson dto
Fshmit Nov 7, 2024
8f27d25
updating esbuild and plugins for esbuild
psachmann Nov 8, 2024
837b4eb
generating files storage api client
psachmann Nov 8, 2024
8015b36
adding files storage client factory and module infrastructure
psachmann Nov 8, 2024
7e4dff8
working on export service
psachmann Nov 8, 2024
414b696
EW-1060 implemented new cc export service via microservice
Fshmit Nov 11, 2024
cf66e8f
EW-1060 resolved dependecy of the new cc mapper
Fshmit Nov 11, 2024
b456cb8
EW-1060 deleted testing dto of cc export
Fshmit Nov 11, 2024
4eb2890
EW-1060 created some classes instead of interfaces
Fshmit Nov 12, 2024
c403e2e
Regenerate lesson api client.
mkreuzkam-cap Nov 12, 2024
a531b70
EW-1060 implemented new mapping of lesson content
Fshmit Nov 13, 2024
718f309
EW-1060 modified mapper
Fshmit Nov 13, 2024
f980547
Merge branch 'main' into EW-1060
Fshmit Nov 13, 2024
4d198a5
EW-1060 modified mapper of lesson dto
Fshmit Nov 13, 2024
0d53c29
EW-1060 removed items from api property
Fshmit Nov 13, 2024
224c556
Regenerate lesson client.
mkreuzkam-cap Nov 13, 2024
ef5334b
regen lesson api.
mkreuzkam-cap Nov 13, 2024
3e2a730
EW-1060 modified mapper of lesson content
Fshmit Nov 13, 2024
709ea25
EW-1060 edited common cartridge mapper
Fshmit Nov 13, 2024
a0e6a54
changing factory to adapter
psachmann Nov 13, 2024
3180245
adding file resource to common cartridge
psachmann Nov 14, 2024
0329142
trying file export for boards
psachmann Nov 14, 2024
fc5d3e3
EW-1060 added control options of course element
Fshmit Nov 14, 2024
93b64f2
adding file export for tasks and boards
psachmann Nov 14, 2024
43a870d
adding modules to the learn room
psachmann Nov 14, 2024
0a0815b
Merge branch 'main' into EW-1060
Fshmit Nov 14, 2024
adc14a1
added logging
psachmann Nov 14, 2024
5f0d0d1
changing parent id
psachmann Nov 14, 2024
4c0f984
changing search id for download files
psachmann Nov 14, 2024
dafc36f
changing download mechanics
psachmann Nov 15, 2024
d12111c
EW-1060 changed endpoint name of new export
Fshmit Nov 14, 2024
9a145fa
adding logging for files storage service
psachmann Nov 15, 2024
8f7f5b8
EW-1060 changed export endpoint to POST
Fshmit Nov 15, 2024
b2c2efd
adding debug logging
psachmann Nov 15, 2024
b62b481
adding more debug logging
psachmann Nov 15, 2024
693f7a4
removing some debug logs
psachmann Nov 15, 2024
7686177
changing async code execution
psachmann Nov 15, 2024
794c806
EW-1060 modified mapping of some cc elements
Fshmit Nov 15, 2024
1a29854
temp changes
psachmann Nov 15, 2024
3b0217e
fixing imports for a module
psachmann Nov 15, 2024
cc05476
changing the base path
psachmann Nov 15, 2024
eaf41d7
changing logging
psachmann Nov 15, 2024
0140db4
changing logging
psachmann Nov 15, 2024
02cc85c
EW-1060 deleted the old export endpoint of cc- microservice
Fshmit Nov 15, 2024
9d9785d
logging
psachmann Nov 15, 2024
5a7daf3
changing logging
psachmann Nov 15, 2024
650f810
changing download mechanics
psachmann Nov 15, 2024
0f4a3be
Merge branch 'main' into EW-1060
Fshmit Nov 18, 2024
2782c16
updating open api definitions
psachmann Nov 18, 2024
5071702
adding some logging
psachmann Nov 18, 2024
7f5f981
changing factory
psachmann Nov 18, 2024
5578a04
changing logging
psachmann Nov 18, 2024
ad4c11e
changing logging
psachmann Nov 18, 2024
f6dac90
adding logging
psachmann Nov 18, 2024
21379f7
some changes
psachmann Nov 18, 2024
4f89b36
some changes
psachmann Nov 18, 2024
c20152e
some changes
psachmann Nov 18, 2024
37af823
changing something
psachmann Nov 18, 2024
94c3e50
changing something
psachmann Nov 18, 2024
9bb6d90
changes
psachmann Nov 18, 2024
5a7cc81
changes
psachmann Nov 18, 2024
3534737
some changes
psachmann Nov 18, 2024
ae47b7b
some changes
psachmann Nov 18, 2024
a6cab7f
changes
psachmann Nov 18, 2024
258a86b
changes
psachmann Nov 18, 2024
2a019fc
changes
psachmann Nov 18, 2024
34bd381
changes
psachmann Nov 18, 2024
b7b79bb
changes
psachmann Nov 18, 2024
a632b43
changes
psachmann Nov 18, 2024
d4c975f
changes
psachmann Nov 18, 2024
6a443d2
changing controller
psachmann Nov 18, 2024
45a9835
some changes
psachmann Nov 19, 2024
71c464e
some changes
psachmann Nov 19, 2024
6ce857e
updating files storage api
psachmann Nov 19, 2024
c2b7542
some changes
psachmann Nov 19, 2024
0c0377c
Merge branch 'EW-1060' into EW-1061
psachmann Nov 19, 2024
c7c5851
adding logging
psachmann Nov 19, 2024
bf2160a
some changes
psachmann Nov 19, 2024
e81ff22
EW-1060 fixed export of cards
Fshmit Nov 19, 2024
1dbe5e9
changing file resource manifest type
psachmann Nov 19, 2024
21e7abe
some bug fixes, probably
psachmann Nov 19, 2024
6dce2d3
hopefully fixing encoding errors
psachmann Nov 19, 2024
17e7942
changing to streams
psachmann Nov 19, 2024
fc9689c
logging
psachmann Nov 20, 2024
a7a930a
logging
psachmann Nov 20, 2024
780159a
some changes
psachmann Nov 20, 2024
32d7989
changes
psachmann Nov 20, 2024
0f0c026
changes
psachmann Nov 20, 2024
4a4f81c
changes
psachmann Nov 20, 2024
d39d76b
changes
psachmann Nov 20, 2024
fc75f8b
changes
psachmann Nov 20, 2024
c3ca32d
changes
psachmann Nov 20, 2024
00018e9
changes
psachmann Nov 20, 2024
e2e87ed
updating the cc micro service
psachmann Nov 20, 2024
d692683
EW-1060 fixed export of cards
Fshmit Nov 19, 2024
5487ca8
updating dependencies for c module
psachmann Nov 20, 2024
4a69fb5
naming resources
psachmann Nov 21, 2024
ae156b7
EW-1060 changed showen title of a card
Fshmit Nov 21, 2024
bc495cc
changes
psachmann Nov 21, 2024
e1e1a3e
changing mappers
psachmann Nov 21, 2024
988fcf2
trimming file captions
psachmann Nov 21, 2024
41af4d7
removing temp folder
psachmann Nov 22, 2024
62fa3b6
EW-1060 modified test of export uc
Fshmit Nov 22, 2024
dfd978c
removing dead code and linting
psachmann Nov 22, 2024
152a54a
reverting two files
psachmann Nov 22, 2024
9655871
adding tests
psachmann Nov 22, 2024
1f13353
EW-1060 added some logs
Fshmit Nov 22, 2024
1a2323c
Merge branch 'main' into EW-1060
Fshmit Nov 22, 2024
c882d53
changes
psachmann Nov 22, 2024
c28b45b
some changes
psachmann Nov 22, 2024
aa8f20f
Merge branch 'main' into EW-1061
psachmann Nov 25, 2024
f799b1d
EW-1060 added test for cc controller
Fshmit Nov 25, 2024
32f2fdd
reverting some files and code cleanup
psachmann Nov 25, 2024
ae2804d
reverting file and adding unit tests
psachmann Nov 25, 2024
d6102ef
EW-1060 changed some variables name
Fshmit Nov 27, 2024
e50495d
EW-1060 added logger module to imports
Fshmit Nov 27, 2024
a8789fe
adding unit tests
psachmann Nov 27, 2024
fa5b609
Merge branch 'main' into EW-1061
psachmann Nov 27, 2024
485741c
EW-1060 merge main into 1060
Fshmit Nov 27, 2024
e502948
adding unit tests and test factories
psachmann Nov 27, 2024
6f6929a
adding unit tests
psachmann Nov 27, 2024
1de75b3
moving method into adapter
psachmann Nov 27, 2024
131bf44
EW-1060 merge main
Fshmit Nov 27, 2024
0599ed3
EW-1060 added tests for cc export service
Fshmit Nov 27, 2024
8823b3b
fixing compile errors
psachmann Nov 28, 2024
cf76d3f
adding unit tests
psachmann Nov 28, 2024
034c819
code refactoring
psachmann Nov 28, 2024
4db3c4b
adding check and check:watch npm scripts
psachmann Nov 28, 2024
15d468e
Merge branch 'EW-1060' into EW-1061
psachmann Nov 28, 2024
69f8718
Merge branch 'main' into EW-1061
psachmann Nov 28, 2024
594b5ed
fixing imports
psachmann Nov 29, 2024
6fb8493
linting
psachmann Nov 29, 2024
e94d6bc
skipping one test suit
psachmann Nov 29, 2024
780540d
skipping another test suit
psachmann Nov 29, 2024
a050214
removing unused test
psachmann Nov 29, 2024
7ec329b
Merge branch 'main' into EW-1061
psachmann Dec 17, 2024
6a209e9
Merge remote-tracking branch 'origin/main' into EW-1061
psachmann Dec 18, 2024
89ffd9b
fixing merge error
psachmann Dec 19, 2024
c1ed7ce
Merge branch 'main' into EW-1061
psachmann Dec 19, 2024
d08f33e
removing linter warnings
psachmann Dec 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
moving method into adapter
psachmann committed Nov 27, 2024
commit 1de75b3974ca03f5002042a86682bf89e452e0b7
Original file line number Diff line number Diff line change
@@ -1,21 +1,47 @@
import { Injectable } from '@nestjs/common';
import { Inject, Injectable } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { REQUEST } from '@nestjs/core';
import { extractJwtFromRequest } from '@shared/common/utils/jwt';
import { AxiosErrorLoggable } from '@src/core/error/loggable';
import { ErrorLogger, Logger } from '@src/core/logger';
import { AxiosError } from 'axios';
import axios, { AxiosError, AxiosResponse } from 'axios';
import { Request } from 'express';
import { FilesStorageRestClientConfig } from './files-storage-rest-client.config';
import { FileApi } from './generated';

@Injectable()
export class FilesStorageRestClientAdapter {
constructor(
private readonly api: FileApi,
private readonly logger: Logger,
private readonly errorLogger: ErrorLogger
private readonly errorLogger: ErrorLogger,
// these should be removed when the generated client supports downloading files as arraybuffer
private readonly configService: ConfigService<FilesStorageRestClientConfig, true>,
@Inject(REQUEST) private readonly req: Request
) {}

public async download(fileRecordId: string, fileName: string): Promise<Buffer | null> {
try {
const response = await this.api.download(fileRecordId, fileName, undefined, {
// INFO: we need to download the file from the files storage service without using the generated client,
// because the generated client does not support downloading files as arraybuffer. Otherwise files with
// binary content would be corrupted like pdfs, zip files, etc. Setting the responseType to 'arraybuffer'
// will not work with the generated client.
// const response = await this.api.download(fileRecordId, fileName, undefined, {
// responseType: 'arraybuffer',
// });
const token = extractJwtFromRequest(this.req);
const url = new URL(
`${this.configService.getOrThrow<string>(
'FILES_STORAGE__SERVICE_BASE_URL'
)}/api/v3/file/download/${fileRecordId}/${fileName}`
);
const response: AxiosResponse<Buffer> = await axios.request({
method: 'GET',
url: url.toString(),
responseType: 'arraybuffer',
headers: {
Authorization: `Bearer ${token}`,
},
});

this.logger.warning({
Original file line number Diff line number Diff line change
@@ -8,11 +8,11 @@ import { Request } from 'express';
import { FilesStorageRestClientAdapter } from './files-storage-rest-client.adapter';
import { FilesStorageRestClientModule } from './files-storage-rest-client.module';

describe.skip(FilesStorageRestClientModule.name, () => {
const configServiceMock = createMock<ConfigService>();

describe(FilesStorageRestClientModule.name, () => {
let module: TestingModule;

const configServiceMock = createMock<ConfigService>();

beforeAll(async () => {
module = await Test.createTestingModule({
imports: [FilesStorageRestClientModule, ConfigModule.forRoot({ isGlobal: true })],
@@ -43,10 +43,10 @@ describe.skip(FilesStorageRestClientModule.name, () => {

describe('resolve providers', () => {
describe('when resolving FilesStorageRestClientAdapter', () => {
it('should resolve FilesStorageRestClientAdapter', () => {
const provider = module.resolve(FilesStorageRestClientAdapter);
it('should resolve FilesStorageRestClientAdapter', async () => {
const provider = await module.resolve(FilesStorageRestClientAdapter);

expect(provider).toBeDefined();
expect(provider).toBeInstanceOf(FilesStorageRestClientAdapter);
});
});
});
Original file line number Diff line number Diff line change
@@ -1,12 +1,7 @@
import { FilesStorageRestClientConfig } from '@infra/files-storage-client';
import { FilesStorageRestClientAdapter } from '@infra/files-storage-client';
import { FileDto, FilesStorageClientAdapterService } from '@modules/files-storage-client';
import { Inject, Injectable } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { REQUEST } from '@nestjs/core';
import { extractJwtFromRequest } from '@shared/common/utils/jwt';
import { Injectable } from '@nestjs/common';
import { ErrorLogger } from '@src/core/logger';
import axios, { AxiosResponse } from 'axios';
import { Request } from 'express';
import { BoardClientAdapter, BoardSkeletonDto, ColumnSkeletonDto } from '../common-cartridge-client/board-client';
import { CardClientAdapter } from '../common-cartridge-client/card-client/card-client.adapter';
import { CardListResponseDto } from '../common-cartridge-client/card-client/dto/card-list-response.dto';
@@ -50,10 +45,9 @@ export class CommonCartridgeExportService {
private readonly courseRoomsClientAdapter: CourseRoomsClientAdapter,
private readonly lessonClientAdapter: LessonClientAdapter,
private readonly filesStorageClient: FilesStorageClientAdapterService,
private readonly filesStorageAdapter: FilesStorageRestClientAdapter,
private readonly mapper: CommonCartridgeExportMapper,
private readonly errorLogger: ErrorLogger,
private readonly configService: ConfigService<FilesStorageRestClientConfig, true>,
@Inject(REQUEST) private readonly req: Request
private readonly errorLogger: ErrorLogger
) {}

public async findCourseCommonCartridgeMetadata(courseId: string): Promise<CourseCommonCartridgeMetadataDto> {
@@ -316,10 +310,10 @@ export class CommonCartridgeExportService {
const files = new Array<{ fileRecord: FileDto; file: Buffer }>();

for await (const fileRecord of fileRecords) {
const file = await this.downloadFile(fileRecord);
const file = await this.filesStorageAdapter.download(fileRecord.id, fileRecord.name);

if (file) {
files.push({ fileRecord, file: file.data });
files.push({ fileRecord, file });
}
}

@@ -337,27 +331,4 @@ export class CommonCartridgeExportService {
return [];
}
}

// INFO: we need to download the file from the files storage service without using the generated client,
// because the generated client does not support downloading files as arraybuffer. Otherwise files with
// binary content would be corrupted like pdfs, zip files, etc. Setting the responseType to 'arraybuffer'
// will not work with the generated client.
private async downloadFile(fileRecord: FileDto): Promise<AxiosResponse<Buffer>> {
const token = extractJwtFromRequest(this.req);
const url = new URL(
`${this.configService.getOrThrow<string>('FILES_STORAGE__SERVICE_BASE_URL')}/api/v3/file/download/${
fileRecord.id
}/${fileRecord.name}`
);
const response: AxiosResponse<Buffer> = await axios.request({
method: 'GET',
url: url.toString(),
responseType: 'arraybuffer',
headers: {
Authorization: `Bearer ${token}`,
},
});

return response;
}
}

Unchanged files with check annotations Beta

// application imports
import { FilesStorageApiModule } from '@modules/files-storage/files-storage-api.module';
import { API_VERSION_PATH } from '@modules/files-storage/files-storage.const';

Check warning on line 12 in apps/server/src/apps/files-storage.app.ts

GitHub Actions / nest_lint

'@modules/files-storage/files-storage.const' import is restricted from being used by a pattern. Do not deep import from a module
import { SwaggerDocumentOptions } from '@nestjs/swagger';
import { LegacyLogger } from '@src/core/logger';
import { enableOpenApiDocs } from '@src/shared/controller/swagger';
import { SystemRule } from '@modules/authorization-rules';
import { ColumnBoardService } from '@modules/board';
import { ContextExternalToolService } from '@src/modules/tool/context-external-tool';
import { CollaborativeStorageUc } from '@modules/collaborative-storage/uc/collaborative-storage.uc';

Check warning on line 10 in apps/server/src/apps/server.app.ts

GitHub Actions / nest_lint

'@modules/collaborative-storage/uc/collaborative-storage.uc' import is restricted from being used by a pattern. Do not deep import from a module
import { GroupService } from '@modules/group';
import { InternalServerModule } from '@modules/internal-server';
import { RocketChatService } from '@modules/rocketchat';
import { FeathersRosterService } from '@modules/roster';
import { ServerModule } from '@modules/server';
import { TeamService } from '@modules/teams/service/team.service';

Check warning on line 16 in apps/server/src/apps/server.app.ts

GitHub Actions / nest_lint

'@modules/teams/service/team.service' import is restricted from being used by a pattern. Do not deep import from a module
import { NestFactory } from '@nestjs/core';
import { ExpressAdapter } from '@nestjs/platform-express';
import { enableOpenApiDocs } from '@shared/controller/swagger';
import type { MikroOrmModuleSyncOptions } from '@mikro-orm/nestjs/typings';
import { ALL_ENTITIES } from '@shared/domain/entity';
import { FileEntity } from '@modules/files/entity';

Check warning on line 3 in apps/server/src/config/mikro-orm-cli.config.ts

GitHub Actions / nest_lint

'@modules/files/entity' import is restricted from being used by a pattern. Do not deep import from a module
import { FileRecord } from '@modules/files-storage/entity';

Check warning on line 4 in apps/server/src/config/mikro-orm-cli.config.ts

GitHub Actions / nest_lint

'@modules/files-storage/entity' import is restricted from being used by a pattern. Do not deep import from a module
import path from 'path';
import { DB_PASSWORD, DB_URL, DB_USERNAME } from './index';
import { createMock } from '@golevelup/ts-jest';
import { Test, TestingModule } from '@nestjs/testing';
import { ConsoleWriterService } from '@infra/console';
import { DatabaseManagementUc } from '@modules/management/uc/database-management.uc';

Check warning on line 4 in apps/server/src/console/api-test/test-bootstrap.console.ts

GitHub Actions / nest_lint

'@modules/management/uc/database-management.uc' import is restricted from being used by a pattern. Do not deep import from a module
import { AbstractBootstrapConsole, BootstrapConsole } from 'nestjs-console';
export class TestBootstrapConsole extends AbstractBootstrapConsole<TestingModule> {
import { Configuration } from '@hpi-schul-cloud/commons/lib';
import { ConsoleWriterModule } from '@infra/console/console-writer/console-writer.module';

Check warning on line 2 in apps/server/src/console/console.module.ts

GitHub Actions / nest_lint

'@infra/console/console-writer/console-writer.module' import is restricted from being used by a pattern. Do not deep import from a module
import { KeycloakModule } from '@infra/identity-management/keycloak/keycloak.module';

Check warning on line 3 in apps/server/src/console/console.module.ts

GitHub Actions / nest_lint

'@infra/identity-management/keycloak/keycloak.module' import is restricted from being used by a pattern. Do not deep import from a module
import { MikroOrmModule } from '@mikro-orm/nestjs';
import { FilesModule } from '@modules/files';
import { ManagementModule } from '@modules/management/management.module';
describe('GlobalErrorFilter', () => {
let module: TestingModule;
let service: GlobalErrorFilter<any>;

Check warning on line 48 in apps/server/src/core/error/filter/global-error.filter.spec.ts

GitHub Actions / nest_lint

Unexpected any. Specify a different type
let domainErrorHandler: DeepMocked<DomainErrorHandler>;
beforeAll(async () => {
return stringifiedMessage;
}
static isInstanceOfLoggable(object: any): object is Loggable {

Check warning on line 20 in apps/server/src/core/logger/logging.utils.ts

GitHub Actions / nest_lint

Unexpected any. Specify a different type
return 'getLogMessage' in object;
}
}
import { ALL_ENTITIES } from '@shared/domain/entity';
import { DB_PASSWORD, DB_URL, DB_USERNAME } from '@src/config';
import { LoggerModule } from '@src/core/logger';
import { RabbitMQWrapperModule } from '@src/infra/rabbitmq';

Check failure on line 10 in apps/server/src/modules/common-cartridge/common-cartridge.module.ts

GitHub Actions / nest_lint

'RabbitMQWrapperModule' is already defined
import { defaultMikroOrmOptions } from '@shared/common';
import { BoardClientModule } from './common-cartridge-client/board-client';
import { CardClientModule } from './common-cartridge-client/card-client/card-client.module';