Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add integration with: Sharepoint #693

Merged
merged 9 commits into from
Sep 13, 2024
37 changes: 22 additions & 15 deletions packages/api/scripts/connectorUpdate.js
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,10 @@ function updateModuleFileForMapper(moduleFile, newServiceDirs, objectType) {
// Generate and insert new service imports
newServiceDirs.forEach((serviceName) => {
const mapperClass =
serviceName.charAt(0).toUpperCase() + serviceName.slice(1) + objectType + 'Mapper';
serviceName.charAt(0).toUpperCase() +
serviceName.slice(1) +
objectType +
'Mapper';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use template literals for string concatenation.

To enhance readability and maintainability, consider using template literals instead of string concatenation. This change aligns with modern JavaScript practices and addresses the static analysis tool's suggestion.

Here's the suggested change:

-const mapperClass =
-  serviceName.charAt(0).toUpperCase() +
-  serviceName.slice(1) +
-  objectType +
-  'Mapper';
+const mapperClass = `${serviceName.charAt(0).toUpperCase()}${serviceName.slice(1)}${objectType}Mapper`;
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
serviceName.charAt(0).toUpperCase() +
serviceName.slice(1) +
objectType +
'Mapper';
const mapperClass = `${serviceName.charAt(0).toUpperCase()}${serviceName.slice(1)}${objectType}Mapper`;
Tools
Biome

[error] 268-271: Template literals are preferred over string concatenation.

Unsafe fix: Use a template literal.

(lint/style/useTemplate)

const importStatement = `import { ${mapperClass} } from './services/${serviceName}/mappers';\n`;
if (!moduleFileContent.includes(importStatement)) {
moduleFileContent = importStatement + moduleFileContent;
Expand Down Expand Up @@ -404,17 +407,20 @@ function updateSeedSQLFile(seedSQLFile, newServiceDirs, vertical) {
fileContent = fileContent.replace(lastMatch[1], newColumnsSection);

// Update each VALUES section
fileContent = fileContent.replace(/INSERT INTO connector_sets \(([^)]+)\) VALUES(.*?);/gs, (match) => {
return match
.replace(/\),\s*\(/g, '),\n (') // Fix line formatting
.replace(/\([^\)]+\)/g, (values, index) => {
if (values.startsWith('(id_connector_set')) {
return values
}
let newValues = newColumns.map(() => 'TRUE').join(', ');
return values.slice(0, -1) + ', ' + newValues + ')';
});
});
fileContent = fileContent.replace(
/INSERT INTO connector_sets \(([^)]+)\) VALUES(.*?);/gs,
(match) => {
return match
.replace(/\),\s*\(/g, '),\n (') // Fix line formatting
.replace(/\([^\)]+\)/g, (values, index) => {
if (values.startsWith('(id_connector_set')) {
return values;
}
let newValues = newColumns.map(() => 'TRUE').join(', ');
return values.slice(0, -1) + ', ' + newValues + ')';
});
},
);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactor to use template literals in SQL string manipulation.

The current implementation uses complex string concatenation which can be simplified using template literals. This will improve the readability and maintainability of the code.

Here's the suggested change:

-let newValues = newColumns.map(() => 'TRUE').join(', ');
-return values.slice(0, -1) + ', ' + newValues + ')';
+let newValues = newColumns.map(() => 'TRUE').join(', ');
+return `${values.slice(0, -1)}, ${newValues})`;
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
fileContent = fileContent.replace(
/INSERT INTO connector_sets \(([^)]+)\) VALUES(.*?);/gs,
(match) => {
return match
.replace(/\),\s*\(/g, '),\n (') // Fix line formatting
.replace(/\([^\)]+\)/g, (values, index) => {
if (values.startsWith('(id_connector_set')) {
return values;
}
let newValues = newColumns.map(() => 'TRUE').join(', ');
return values.slice(0, -1) + ', ' + newValues + ')';
});
},
);
fileContent = fileContent.replace(
/INSERT INTO connector_sets \(([^)]+)\) VALUES(.*?);/gs,
(match) => {
return match
.replace(/\),\s*\(/g, '),\n (') // Fix line formatting
.replace(/\([^\)]+\)/g, (values, index) => {
if (values.startsWith('(id_connector_set')) {
return values;
}
let newValues = newColumns.map(() => 'TRUE').join(', ');
return `${values.slice(0, -1)}, ${newValues})`;
});
},
);
Tools
Biome

[error] 420-420: Template literals are preferred over string concatenation.

Unsafe fix: Use a template literal.

(lint/style/useTemplate)

}
// Write the modified content back to the file
console.log(fileContent);
Expand All @@ -427,9 +433,10 @@ function updateSeedSQLFile(seedSQLFile, newServiceDirs, vertical) {
function updateObjectTypes(baseDir, objectType, vertical) {
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const servicesDir = path.join(__dirname, baseDir);
const targetFilename = vertical == 'filestorage' ? 'file-storage' : vertical;
const targetFile = path.join(
__dirname,
`../src/@core/utils/types/original/original.${vertical}.ts`,
`../src/@core/utils/types/original/original.${targetFilename}.ts`,
);

const newServiceDirs = scanDirectory(servicesDir);
Expand Down Expand Up @@ -468,7 +475,7 @@ function updateObjectTypes(baseDir, objectType, vertical) {
);

updateModuleFileForService(moduleFile, newServiceDirs);
updateModuleFileForMapper(moduleFile, newServiceDirs, objectType)
updateModuleFileForMapper(moduleFile, newServiceDirs, objectType);

// Path to the mappings file
// const mappingsFile = path.join(
Expand Down Expand Up @@ -522,4 +529,4 @@ if (import.meta.url === process.argv[1]) {

const argv = yargs(hideBin(process.argv)).argv;
const baseDir = `../src/${argv.vertical.toLowerCase()}/${argv.objectType.toLowerCase()}/services`;
updateObjectTypes(baseDir, argv.objectType, argv.vertical);
updateObjectTypes(baseDir, argv.objectType, argv.vertical);
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,8 @@ export class SharepointConnectionService extends AbstractBaseConnectionService {
grant_type: 'authorization_code',
});
const res = await axios.post(
`https://app.sharepoint.com/oauth2/tokens`,
// `https://app.sharepoint.com/oauth2/tokens`,
`https://login.microsoftonline.com/common/oauth2/v2.0/token`,
formData.toString(),
{
headers: {
Expand Down Expand Up @@ -214,7 +215,8 @@ export class SharepointConnectionService extends AbstractBaseConnectionService {
)) as OAuth2AuthData;

const res = await axios.post(
`https://app.sharepoint.com/oauth2/tokens`,
// `https://app.sharepoint.com/oauth2/tokens`,
`https://login.microsoftonline.com/common/oauth2/v2.0/token`,
formData.toString(),
{
headers: {
Expand Down
1 change: 1 addition & 0 deletions packages/api/src/@core/sync/sync.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -538,6 +538,7 @@ export class CoreSyncService {
try {
await task();
} catch (error) {
console.log(error);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider removing console.log for error logging in production.

Using console.log can lead to performance issues and does not align with best practices for error handling in production environments. It is recommended to rely on the existing logger.error method, which integrates with the application's centralized logging system.

this.logger.error(`File Storage Task failed: ${error.message}`, error);
}
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,42 @@
import {
BoxSharedLinkInput,
BoxSharedLinkOutput,
} from '@filestorage/sharedlink/services/box/types';
import {
SharepointSharedLinkInput,
SharepointSharedLinkOutput,
} from '@filestorage/sharedlink/services/sharepoint/types';

import {
SharepointPermissionInput,
SharepointPermissionOutput,
} from '@filestorage/permission/services/sharepoint/types';

import {
SharepointGroupInput,
SharepointGroupOutput,
} from '@filestorage/group/services/sharepoint/types';

import {
SharepointUserInput,
SharepointUserOutput,
} from '@filestorage/user/services/sharepoint/types';

import {
SharepointFolderInput,
SharepointFolderOutput,
} from '@filestorage/folder/services/sharepoint/types';

import {
SharepointFileInput,
SharepointFileOutput,
} from '@filestorage/file/services/sharepoint/types';

import {
SharepointDriveInput,
SharepointDriveOutput,
} from '@filestorage/drive/services/sharepoint/types';

/* INPUT */

import {
Expand All @@ -18,25 +57,25 @@ import {
} from '@filestorage/user/services/box/types';

/* file */
export type OriginalFileInput = BoxFileInput;
export type OriginalFileInput = BoxFileInput | SharepointFileInput;

/* folder */
export type OriginalFolderInput = BoxFolderInput;
export type OriginalFolderInput = BoxFolderInput | SharepointFolderInput;

/* permission */
export type OriginalPermissionInput = any;
export type OriginalPermissionInput = any | SharepointPermissionInput;

/* shared link */
export type OriginalSharedLinkInput = any;

/* drive */
export type OriginalDriveInput = any;
export type OriginalDriveInput = any | SharepointDriveInput;

/* group */
export type OriginalGroupInput = BoxGroupInput;
export type OriginalGroupInput = BoxGroupInput | SharepointGroupInput;

/* user */
export type OriginalUserInput = BoxUserInput;
export type OriginalUserInput = BoxUserInput | SharepointUserInput;

export type FileStorageObjectInput =
| OriginalFileInput
Expand All @@ -50,25 +89,25 @@ export type FileStorageObjectInput =
/* OUTPUT */

/* file */
export type OriginalFileOutput = BoxFileOutput;
export type OriginalFileOutput = BoxFileOutput | SharepointFileOutput;

/* folder */
export type OriginalFolderOutput = BoxFolderOutput;
export type OriginalFolderOutput = BoxFolderOutput | SharepointFolderOutput;

/* permission */
export type OriginalPermissionOutput = any;
export type OriginalPermissionOutput = any | SharepointPermissionOutput;

/* shared link */
export type OriginalSharedLinkOutput = any;

/* drive */
export type OriginalDriveOutput = any;
export type OriginalDriveOutput = any | SharepointDriveOutput;

/* group */
export type OriginalGroupOutput = BoxGroupOutput;
export type OriginalGroupOutput = BoxGroupOutput | SharepointGroupOutput;

/* user */
export type OriginalUserOutput = BoxUserOutput;
export type OriginalUserOutput = BoxUserOutput | SharepointUserOutput;

export type FileStorageObjectOutput =
| OriginalFileOutput
Expand All @@ -78,3 +117,11 @@ export type FileStorageObjectOutput =
| OriginalDriveOutput
| OriginalGroupOutput
| OriginalUserOutput;

export type OriginalSharedlinkInput =
| BoxSharedLinkInput
| SharepointSharedLinkInput;

export type OriginalSharedlinkOutput =
| BoxSharedLinkOutput
| SharepointSharedLinkOutput;
4 changes: 4 additions & 0 deletions packages/api/src/filestorage/drive/drive.module.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import { SharepointDriveMapper } from './services/sharepoint/mappers';
import { SharepointService } from './services/sharepoint';
import { BullQueueModule } from '@@core/@core-services/queues/queue.module';
import { WebhookService } from '@@core/@core-services/webhooks/panora-webhooks/webhook.service';
import { Module } from '@nestjs/common';
Expand All @@ -17,6 +19,8 @@ import { Utils } from '@filestorage/@lib/@utils';
ServiceRegistry,
Utils,
/* PROVIDERS SERVICES */
SharepointService,
SharepointDriveMapper,
],
exports: [SyncService],
})
Expand Down
71 changes: 71 additions & 0 deletions packages/api/src/filestorage/drive/services/sharepoint/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
import { EncryptionService } from '@@core/@core-services/encryption/encryption.service';
import { LoggerService } from '@@core/@core-services/logger/logger.service';
import { PrismaService } from '@@core/@core-services/prisma/prisma.service';
import { ApiResponse } from '@@core/utils/types';
import { SyncParam } from '@@core/utils/types/interface';
import { FileStorageObject } from '@filestorage/@lib/@types';
import { IDriveService } from '@filestorage/drive/types';
import { Injectable } from '@nestjs/common';
import axios from 'axios';
import { ServiceRegistry } from '../registry.service';
import { SharepointDriveOutput } from './types';
import { DesunifyReturnType } from '@@core/utils/types/desunify.input';
import { OriginalDriveOutput } from '@@core/utils/types/original/original.file-storage';

@Injectable()
export class SharepointService implements IDriveService {
constructor(
private prisma: PrismaService,
private logger: LoggerService,
private cryptoService: EncryptionService,
private registry: ServiceRegistry,
) {
this.logger.setContext(
`${FileStorageObject.file.toUpperCase()}:${SharepointService.name}`,
);
this.registry.registerService('sharepoint', this);
}

async addDrive(
driveData: DesunifyReturnType,
linkedUserId: string,
): Promise<ApiResponse<OriginalDriveOutput>> {
// No API to add drive in Sharepoint
return;
}
Comment on lines +29 to +35
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarify the stub for addDrive.

The addDrive method is a stub with a comment indicating no API for adding drives in SharePoint. If this is accurate, consider adding more detail or documentation to explain how this situation is handled or if alternative methods are used.


async sync(data: SyncParam): Promise<ApiResponse<SharepointDriveOutput[]>> {
try {
const { linkedUserId } = data;

const connection = await this.prisma.connections.findFirst({
where: {
id_linked_user: linkedUserId,
provider_slug: 'sharepoint',
vertical: 'filestorage',
},
});

const resp = await axios.get(`${connection.account_url}/drives`, {
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${this.cryptoService.decrypt(
connection.access_token,
)}`,
},
});

const drives: SharepointDriveOutput[] = resp.data.value;
this.logger.log(`Synced sharepoint drives !`);

return {
data: drives,
message: 'Sharepoint drives retrived',
statusCode: 200,
};
} catch (error) {
console.log(error.response);
throw error;
}
}
}
86 changes: 86 additions & 0 deletions packages/api/src/filestorage/drive/services/sharepoint/mappers.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
import { MappersRegistry } from '@@core/@core-services/registries/mappers.registry';
import { CoreUnification } from '@@core/@core-services/unification/core-unification.service';
import { Utils } from '@filestorage/@lib/@utils';
import {
UnifiedFilestorageDriveInput,
UnifiedFilestorageDriveOutput,
} from '@filestorage/drive/types/model.unified';
import { Injectable } from '@nestjs/common';
import { SharepointDriveInput, SharepointDriveOutput } from './types';
import { IDriveMapper } from '@filestorage/drive/types';

@Injectable()
export class SharepointDriveMapper implements IDriveMapper {
constructor(
private mappersRegistry: MappersRegistry,
private utils: Utils,
private coreUnificationService: CoreUnification,
) {
this.mappersRegistry.registerService(
'filestorage',
'drive',
'sharepoint',
this,
);
}

async desunify(
source: UnifiedFilestorageDriveInput,
customFieldMappings?: {
slug: string;
remote_id: string;
}[],
): Promise<SharepointDriveInput> {
return;
Comment on lines +27 to +34
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incomplete implementation of desunify method.

The desunify method lacks implementation, which might be intentional as a placeholder or an oversight. Please confirm if this method is supposed to be implemented later or if it was missed.

Would you like me to help draft the implementation or should this be tracked as a task in your project management tool?

}

async unify(
source: SharepointDriveOutput | SharepointDriveOutput[],
connectionId: string,
customFieldMappings?: {
slug: string;
remote_id: string;
}[],
): Promise<UnifiedFilestorageDriveOutput | UnifiedFilestorageDriveOutput[]> {
if (!Array.isArray(source)) {
return await this.mapSingleDriveToUnified(
source,
connectionId,
customFieldMappings,
);
}
// Handling array of SharepointDriveOutput
return Promise.all(
source.map((drive) =>
this.mapSingleDriveToUnified(drive, connectionId, customFieldMappings),
),
);
}

private async mapSingleDriveToUnified(
drive: SharepointDriveOutput,
connectionId: string,
customFieldMappings?: {
slug: string;
remote_id: string;
}[],
): Promise<UnifiedFilestorageDriveOutput> {
const field_mappings: { [key: string]: any } = {};
if (customFieldMappings) {
for (const mapping of customFieldMappings) {
field_mappings[mapping.slug] = drive[mapping.remote_id];
}
}

const result: UnifiedFilestorageDriveOutput = {
remote_id: drive.id,
remote_data: drive,
name: drive.name,
remote_created_at: drive.createdDateTime,
drive_url: drive.webUrl,
field_mappings,
};

return result;
}
}
Loading
Loading