[Xlsx-Import] How can I read xlsx from S3 ?

[love8587]

Hi, I've using XToolset and love it.
I'my trying to read xlsx file from S3 not from file system.

Is it avilable to read xlsx from S3 ?
I've not found any code reading from S3 in ImporterFactory .

Should I contrinute for it?

[Siemienik]

Hi @love8587 ! Thank you for your nice words ;)

Backend

As far as I know, AWS provides NodeJS client which provides getting data from S3 as a Buffer. So it is able to read by following function:

const buildImporterFromBuffer = (buffer) => {
  // 1. Create ExcelJS workbook instance
  const wb = new Workbook(); 
  // 2. Read the data from the Buffer
  await wb.xlsx.load(buffer);
  
  // 3. Build and return Importer
  return new Importer(wb); 
}

Having this function, you can use it with AWS Node.js API
(code based on https://stackoverflow.com/a/16903730/4747028 )

var AWS = require('aws-sdk');
AWS.config.update(
  {
    accessKeyId: ".. your key ..",
    secretAccessKey: ".. your secret key ..",
  }
);
var s3 = new AWS.S3();
s3.getObject(
  { Bucket: "my-bucket", Key: "my-picture.jpg" },
  function (error, data) {
    if (error != null) {
      console.error("Failed to retrieve an object: " + error);
    } else {
      console.log("Loaded " + data.ContentLength + " bytes");


     // 4. Build importer 
     const importer = buildImporterFromBuffer(data.Body)

     // 5. And finally, import what you need. ;-) Good luck.

    }
  }
);

Frontend

However, please keep in mind. That using accessKeyId, and secretAccessKey on the frontend is highly unsecured. 🙈 🙉 🙊

For front-end usages, I recommend using AWS Cloud Front, which allows us to publish S3 data as public resources. It is also an additional cost. Another solution is writing a microservice that will manage a connection with S3 and serve this file.

Additionally

• We have a topic about increasing the functionality of ImporterFactory Adding more sources types into ImportFactor.from #52 - any suggestions are welcome.

• I would also invite you to our Gitter channel where it is able to chat in realtime, ask and answer. Additionally, this is a place where I share updates about project progress.

• I hope that my answer was constructive 😄 if yes, please left a star for this project and share XToolset with your friends and colleagues, what really help promote this project, and what gets me a lot of motivation to do XToolset best.

Bonus: with a Promise support:

(I really like this one)

const importer = await s3.getObject(params)
  .promise()
  .then((data) => buildImporterFromBuffer(data.Body) )
  .catch((error) => {console.error("Failed to retrieve an object: " + error);} );

(Based on AWS official API Reference )