Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BC-5629 batch deletion mechanism #4521

Merged
merged 93 commits into from
Nov 15, 2023
Merged
Show file tree
Hide file tree
Changes from 73 commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
ca59c48
first commit
WojciechGrancow Oct 19, 2023
5838b81
add some tests
WojciechGrancow Oct 20, 2023
7df7c23
add test cases and services
WojciechGrancow Oct 27, 2023
1b07d3f
add new (almost empty for now) batch deletion app
bn-pass Oct 30, 2023
dec5292
refactor config vars
bn-pass Oct 31, 2023
29c58a0
add optional env var for specifying delay between the API calls
bn-pass Oct 31, 2023
a853ea7
add usecases and test cases
WojciechGrancow Nov 1, 2023
72c84b4
Merge branch 'main' into BC-5521-implementation-KNL-Module
WojciechGrancow Nov 1, 2023
c56ae3d
fix importing
WojciechGrancow Nov 1, 2023
cad4491
add type in uc
WojciechGrancow Nov 2, 2023
b80ff0e
fix import
WojciechGrancow Nov 2, 2023
5d764d0
add references service that'll load all the references to the data we…
bn-pass Nov 2, 2023
c226c6d
fix most of issue form review
WojciechGrancow Nov 3, 2023
31a225c
Merge branch 'BC-5521-implementation-KNL-Module' into BC-5629-batch-d…
bn-pass Nov 3, 2023
fdbe8c9
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 3, 2023
78ed3b9
add deletion API client with just a single method for now that allows…
bn-pass Nov 4, 2023
3acf266
refactor the env vars for configurting the Admin API
bn-pass Nov 4, 2023
f2ac1bc
add exporting DeletionClientConfig
bn-pass Nov 4, 2023
b69741a
move references service to the deletion module
bn-pass Nov 4, 2023
0c02d6d
delete unused code
bn-pass Nov 4, 2023
30412f9
add batch deletion service that makes it possible ot queue deletion f…
bn-pass Nov 4, 2023
13f7b57
move some parts of the interface to the interface subdir
bn-pass Nov 4, 2023
61a0814
add an interface for the batch deletion summary
bn-pass Nov 4, 2023
6c72715
move some interfaces to a separate subdir
bn-pass Nov 4, 2023
46395c0
refactor the batch deletion summary interface
bn-pass Nov 4, 2023
f204220
add uc for the batch deletion
bn-pass Nov 4, 2023
3ecc282
remove unused annotation
bn-pass Nov 6, 2023
78dedbf
refactor deletion client implementation
bn-pass Nov 6, 2023
6c66cf6
add batch deletion service implementation
bn-pass Nov 6, 2023
1c8cae9
add UC for the batch deletion
bn-pass Nov 7, 2023
90aca19
add a console app for the deletion module and a console command to ma…
bn-pass Nov 7, 2023
20a04e9
remove no longer used app, add param to make it possible to define de…
bn-pass Nov 7, 2023
9e0d281
remove no longer used separate batch-deletion module (it became a par…
bn-pass Nov 7, 2023
ab14b19
fix invalid key
bn-pass Nov 7, 2023
4d13aa5
remove no longer used config vars
bn-pass Nov 7, 2023
ee43ec2
remove no longer used commands
bn-pass Nov 7, 2023
8dbeeb7
remove no longer used Nest cli config
bn-pass Nov 7, 2023
8018c6e
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 7, 2023
71d0b40
remove no longer used code
bn-pass Nov 7, 2023
4c5de4e
change name of the method that prepares default headers
bn-pass Nov 7, 2023
02c5ed4
add builders for most of the interfaces
bn-pass Nov 7, 2023
df4124e
add builders for the remaining interfaces
bn-pass Nov 7, 2023
0d0fdb2
add type in catch clause
bn-pass Nov 7, 2023
62695f5
do some adjustments, move PushDeletionRequestsOptions interface to a …
bn-pass Nov 7, 2023
a1c7477
remove unused import
bn-pass Nov 7, 2023
141d5d2
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 7, 2023
5d19f5d
rollback
bn-pass Nov 7, 2023
a63c594
remove unnecessary indent
bn-pass Nov 7, 2023
2f52adc
remove unnecessary indents
bn-pass Nov 7, 2023
9001592
remove empty line
bn-pass Nov 7, 2023
da9c2a6
remove repeated imports
bn-pass Nov 7, 2023
66daec1
refactor some imports to omit calling Configuration.get() on every su…
bn-pass Nov 7, 2023
2bffa6d
add builder for the DeletionRequestOutput class
bn-pass Nov 8, 2023
06ef50b
add unit tests for the batch deletion service
bn-pass Nov 8, 2023
5317dae
add unit tests for the BatchDeletionUc
bn-pass Nov 8, 2023
e9d5559
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 8, 2023
a1b117b
modify env keys for the Admin API client configuration, refactor the …
bn-pass Nov 8, 2023
5dd40a7
Merge branch 'BC-5629-batch-deletion-mechanism' of https://github.com…
bn-pass Nov 8, 2023
aa378b0
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 8, 2023
ebfe6bd
fix invalid import, remove unused undefined arg
bn-pass Nov 8, 2023
9f62aa7
add comment to ignore console.ts file for coverage
bn-pass Nov 8, 2023
8706f50
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 8, 2023
abc6656
move deletion client config interface to a separate file, refactor fu…
bn-pass Nov 8, 2023
e057fe0
Merge branch 'BC-5629-batch-deletion-mechanism' of https://github.com…
bn-pass Nov 8, 2023
638d1e9
fix invalid import
bn-pass Nov 8, 2023
a5180ae
add more test cases to the deletion client unit tests
bn-pass Nov 8, 2023
e1c9312
change invalid import
bn-pass Nov 8, 2023
b35aa9c
fix invalid import
bn-pass Nov 8, 2023
667427f
add builder for the PushDeletionRequestsOptions class, add unit tests…
bn-pass Nov 8, 2023
02bd3a2
rename the file containing the deletion module console to deletion.co…
bn-pass Nov 8, 2023
88b0b28
remove deletion.console.ts from the sonar.coverage.exclusions param a…
bn-pass Nov 8, 2023
7f06aa6
add deletion.console.ts file to the coverage exclusions (another try …
bn-pass Nov 8, 2023
4e26016
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 9, 2023
009bdd1
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 9, 2023
4d18313
change name of the file containing the deletion console app
bn-pass Nov 9, 2023
b12527c
Merge branch 'BC-5629-batch-deletion-mechanism' of https://github.com…
bn-pass Nov 9, 2023
d17c097
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 9, 2023
acf51b9
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 9, 2023
cca5fff
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 10, 2023
a3037ea
fix some imports
bn-pass Nov 13, 2023
1217640
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 13, 2023
d8ff82f
move default value for the ADMIN_API_CLIENT object to default.schema.…
bn-pass Nov 13, 2023
5624dc5
move default for the BASE_URL
bn-pass Nov 13, 2023
8ec624f
move Deletion module console app to the apps/ dir
bn-pass Nov 13, 2023
e87ede7
add separate functino to log error and set exit code
bn-pass Nov 13, 2023
abc83e5
add handling of the case that only CR chars are used as a line separa…
bn-pass Nov 13, 2023
3cd6e25
add use of the BatchDeletionSummaryBuilder in place of an anonymous o…
bn-pass Nov 13, 2023
3bf6813
fix some imports/exports
bn-pass Nov 13, 2023
28753a5
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 14, 2023
30cfe39
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 14, 2023
d61d6c8
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 14, 2023
cfa6ac9
refactor console app flow
bn-pass Nov 15, 2023
2ffa8ad
Merge branch 'main' into BC-5629-batch-deletion-mechanism
bn-pass Nov 15, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
import { ObjectId } from 'bson';
import { DeletionRequestInput } from '../interface';
import { DeletionRequestInputBuilder } from './deletion-request-input.builder';

describe(DeletionRequestInputBuilder.name, () => {
describe(DeletionRequestInputBuilder.build.name, () => {
describe('when called with proper arguments', () => {
const setup = () => {
const targetRefDomain = 'school';
const targetRefId = new ObjectId().toHexString();
const deleteInMinutes = 43200;

const expectedOutput: DeletionRequestInput = {
targetRef: {
domain: targetRefDomain,
id: targetRefId,
},
deleteInMinutes,
};

return { targetRefDomain, targetRefId, deleteInMinutes, expectedOutput };
};

it('should return valid object with expected values', () => {
const { targetRefDomain, targetRefId, deleteInMinutes, expectedOutput } = setup();

const output = DeletionRequestInputBuilder.build(targetRefDomain, targetRefId, deleteInMinutes);

expect(output).toStrictEqual(expectedOutput);
});
});
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import { DeletionRequestInput } from '../interface';
import { DeletionRequestTargetRefInputBuilder } from './deletion-request-target-ref-input.builder';

export class DeletionRequestInputBuilder {
static build(targetRefDomain: string, targetRefId: string, deleteInMinutes?: number): DeletionRequestInput {
return {
targetRef: DeletionRequestTargetRefInputBuilder.build(targetRefDomain, targetRefId),
deleteInMinutes,
};
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import { ObjectId } from 'bson';
import { DeletionRequestOutput } from '../interface';
import { DeletionRequestOutputBuilder } from './deletion-request-output.builder';

describe(DeletionRequestOutputBuilder.name, () => {
describe(DeletionRequestOutputBuilder.build.name, () => {
describe('when called with proper arguments', () => {
const setup = () => {
const requestId = new ObjectId().toHexString();
const deletionPlannedAt = new Date();

const expectedOutput: DeletionRequestOutput = {
requestId,
deletionPlannedAt,
};

return { requestId, deletionPlannedAt, expectedOutput };
};

it('should return valid object with expected values', () => {
const { requestId, deletionPlannedAt, expectedOutput } = setup();

const output = DeletionRequestOutputBuilder.build(requestId, deletionPlannedAt);

expect(output).toStrictEqual(expectedOutput);
});
});
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
import { DeletionRequestOutput } from '../interface';

export class DeletionRequestOutputBuilder {
static build(requestId: string, deletionPlannedAt: Date): DeletionRequestOutput {
return {
requestId,
deletionPlannedAt,
};
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import { ObjectId } from 'bson';
import { DeletionRequestTargetRefInput } from '../interface';
import { DeletionRequestTargetRefInputBuilder } from './deletion-request-target-ref-input.builder';

describe(DeletionRequestTargetRefInputBuilder.name, () => {
describe(DeletionRequestTargetRefInputBuilder.build.name, () => {
describe('when called with proper arguments', () => {
const setup = () => {
const domain = 'user';
const id = new ObjectId().toHexString();

const expectedOutput: DeletionRequestTargetRefInput = { domain, id };

return { domain, id, expectedOutput };
};

it('should return valid object with expected values', () => {
const { domain, id, expectedOutput } = setup();

const output = DeletionRequestTargetRefInputBuilder.build(domain, id);

expect(output).toStrictEqual(expectedOutput);
});
});
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
import { DeletionRequestTargetRefInput } from '../interface';

export class DeletionRequestTargetRefInputBuilder {
static build(domain: string, id: string): DeletionRequestTargetRefInput {
return { domain, id };
}
}
3 changes: 3 additions & 0 deletions apps/server/src/modules/deletion/client/builder/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
export * from './deletion-request-target-ref-input.builder';
export * from './deletion-request-input.builder';
export * from './deletion-request-output.builder';
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
import { IConfig } from '@hpi-schul-cloud/commons/lib/interfaces/IConfig';
import { Configuration } from '@hpi-schul-cloud/commons/lib';
import { DeletionClientConfig } from './interface';
import { getDeletionClientConfig } from './deletion-client.config';

describe(getDeletionClientConfig.name, () => {
let configBefore: IConfig;

beforeAll(() => {
configBefore = Configuration.toObject({ plainSecrets: true });
});

afterEach(() => {
Configuration.reset(configBefore);
});

describe('when called', () => {
const setup = () => {
const baseUrl = 'http://api-admin:4030';
const apiKey = '652559c2-93da-42ad-94e1-640e3afbaca0';

Configuration.set('ADMIN_API_CLIENT__BASE_URL', baseUrl);
Configuration.set('ADMIN_API_CLIENT__API_KEY', apiKey);

const expectedConfig: DeletionClientConfig = {
ADMIN_API_CLIENT_BASE_URL: baseUrl,
ADMIN_API_CLIENT_API_KEY: apiKey,
};

return { expectedConfig };
};

it('should return config with proper values', () => {
const { expectedConfig } = setup();

const config = getDeletionClientConfig();

expect(config).toEqual(expectedConfig);
});
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
import { Configuration } from '@hpi-schul-cloud/commons/lib';
import { DeletionClientConfig } from './interface';

export const getDeletionClientConfig = (): DeletionClientConfig => {
return {
ADMIN_API_CLIENT_BASE_URL: Configuration.get('ADMIN_API_CLIENT__BASE_URL') as string,
ADMIN_API_CLIENT_API_KEY: Configuration.get('ADMIN_API_CLIENT__API_KEY') as string,
};
};
154 changes: 154 additions & 0 deletions apps/server/src/modules/deletion/client/deletion.client.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
import { of } from 'rxjs';
import { AxiosResponse } from 'axios';
import { HttpService } from '@nestjs/axios';
import { ConfigService } from '@nestjs/config';
import { Test, TestingModule } from '@nestjs/testing';
import { createMock, DeepMocked } from '@golevelup/ts-jest';
import { axiosResponseFactory } from '@shared/testing';
import { DeletionRequestInputBuilder, DeletionRequestOutputBuilder } from '@modules/deletion';
bn-pass marked this conversation as resolved.
Show resolved Hide resolved
import { DeletionRequestOutput } from './interface';
import { DeletionClient } from './deletion.client';

describe(DeletionClient.name, () => {
let module: TestingModule;
let client: DeletionClient;
let httpService: DeepMocked<HttpService>;

beforeEach(async () => {
module = await Test.createTestingModule({
providers: [
DeletionClient,
{
provide: ConfigService,
useValue: createMock<ConfigService>({
get: jest.fn((key: string) => {
if (key === 'ADMIN_API_CLIENT_BASE_URL') {
return 'http://localhost:4030';
}

// Default is for the Admin APIs API Key.
return '6b3df003-61e9-467c-9e6b-579634801896';
}),
}),
},
{
provide: HttpService,
useValue: createMock<HttpService>(),
},
],
}).compile();

client = module.get(DeletionClient);
httpService = module.get(HttpService);
});

afterEach(() => {
jest.resetAllMocks();
});

afterAll(async () => {
await module.close();
});

describe('queueDeletionRequest', () => {
describe('when received valid response with expected HTTP status code', () => {
const setup = () => {
const input = DeletionRequestInputBuilder.build('user', '652f1625e9bc1a13bdaae48b');

const output: DeletionRequestOutput = DeletionRequestOutputBuilder.build(
'6536ce29b595d7c8e5faf200',
new Date('2024-10-15T12:42:50.521Z')
);

const response: AxiosResponse<DeletionRequestOutput> = axiosResponseFactory.build({
data: output,
status: 202,
});

httpService.post.mockReturnValueOnce(of(response));

return { input, output };
};

it('should return proper output', async () => {
const { input, output } = setup();

const result = await client.queueDeletionRequest(input);

expect(result).toEqual(output);
});
});

describe('when received invalid HTTP status code in a response', () => {
const setup = () => {
const input = DeletionRequestInputBuilder.build('user', '652f1625e9bc1a13bdaae48b');

const output: DeletionRequestOutput = DeletionRequestOutputBuilder.build('', new Date());

const response: AxiosResponse<DeletionRequestOutput> = axiosResponseFactory.build({
data: output,
status: 200,
});

httpService.post.mockReturnValueOnce(of(response));

return { input };
};

it('should throw an exception', async () => {
const { input } = setup();

await expect(client.queueDeletionRequest(input)).rejects.toThrow(Error);
});
});

describe('when received no requestId in a response', () => {
const setup = () => {
const input = DeletionRequestInputBuilder.build('user', '652f1625e9bc1a13bdaae48b');

const output: DeletionRequestOutput = DeletionRequestOutputBuilder.build(
'',
new Date('2024-10-15T12:42:50.521Z')
);

const response: AxiosResponse<DeletionRequestOutput> = axiosResponseFactory.build({
data: output,
status: 202,
});

httpService.post.mockReturnValueOnce(of(response));

return { input };
};

it('should throw an exception', async () => {
const { input } = setup();

await expect(client.queueDeletionRequest(input)).rejects.toThrow(Error);
});
});

describe('when received no deletionPlannedAt in a response', () => {
const setup = () => {
const input = DeletionRequestInputBuilder.build('user', '652f1625e9bc1a13bdaae48b');

const response: AxiosResponse<DeletionRequestOutput> = axiosResponseFactory.build({
data: {
requestId: '6536ce29b595d7c8e5faf200',
},
status: 202,
});

httpService.post.mockReturnValueOnce(of(response));

return { input };
};

it('should throw an exception', async () => {
const { input } = setup();

await expect(client.queueDeletionRequest(input)).rejects.toThrow(Error);
});
});
});
});
66 changes: 66 additions & 0 deletions apps/server/src/modules/deletion/client/deletion.client.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
import { firstValueFrom } from 'rxjs';
import { AxiosResponse } from 'axios';
import { Injectable } from '@nestjs/common';
import { HttpService } from '@nestjs/axios';
import { ConfigService } from '@nestjs/config';
import { DeletionRequestInput, DeletionRequestOutput, DeletionClientConfig } from './interface';

@Injectable()
export class DeletionClient {
private readonly baseUrl: string;

private readonly apiKey: string;

private readonly postDeletionRequestsEndpoint: string;

constructor(
private readonly httpService: HttpService,
private readonly configService: ConfigService<DeletionClientConfig, true>
) {
this.baseUrl = this.configService.get<string>('ADMIN_API_CLIENT_BASE_URL');
this.apiKey = this.configService.get<string>('ADMIN_API_CLIENT_API_KEY');

// Prepare the POST /deletionRequests endpoint beforehand to not do it on every client call.
this.postDeletionRequestsEndpoint = new URL('/admin/api/v1/deletionRequests', this.baseUrl).toString();
}

async queueDeletionRequest(input: DeletionRequestInput): Promise<DeletionRequestOutput> {
const request = this.httpService.post(this.postDeletionRequestsEndpoint, input, this.defaultHeaders());

return firstValueFrom(request)
.then((resp: AxiosResponse<DeletionRequestOutput>) => {
// Throw an error if any other status code (other than expected "202 Accepted" is returned).
if (resp.status !== 202) {
throw new Error(`invalid HTTP status code in a response from the server - ${resp.status} instead of 202`);
}

// Throw an error if server didn't return a requestId in a response (and it is
// required as it gives client the reference to the created deletion request).
if (!resp.data.requestId) {
throw new Error('no valid requestId returned from the server');
}

// Throw an error if server didn't return a deletionPlannedAt timestamp so the user
// will not be aware after which date the deletion request's execution will begin.
if (!resp.data.deletionPlannedAt) {
throw new Error('no valid deletionPlannedAt returned from the server');
}

return resp.data;
})
.catch((err: Error) => {
// Throw an error if sending/processing deletion request by the client failed in any way.
throw new Error(`failed to send/process a deletion request: ${err.toString()}`);
});
}

private apiKeyHeader() {
return { 'X-Api-Key': this.apiKey };
}

private defaultHeaders() {
return {
headers: this.apiKeyHeader(),
};
}
}
Loading
Loading