Error when replacing ResumeableUpload with ResumableUpload2 #10

SysEngDan · 2024-11-06T14:26:31Z

Hi there,

First and foremost, thank you so very much for your work and open sourcing it for our community. I am personally very grateful for you!

I've successfully deployed a GAS that uploads 10 files concurrently using your ResumeableUploadForGoogleDrive (the first version which stores each file in RAM). I've had some users report that their uploads stop at a certain percentage (every person's percentage was different). These users are usually uploading >1,000 files that are 30-80Mb each. I assume that they're hit some out-of-memory error on their machine.

I was hoping that replacing ResumeableUploadForGoogleDrive could be swapped with ResumeableUploadForGoogleDrive2, but I didn't have success with that. I received the console error There are no required parameters. accessToken, fileName, fileSize, fileType and fileBuffer are required. But, upon code review, it appears to me that these are being provided in the resource. I see it listed in my code as:

                const resource = {
                    fileName: f.fileName,
                    fileSize: f.fileSize,
                    fileType: f.fileType,
                    fileBuffer: f.result,
                    accessToken: accessToken,
                    folderId: newFolderId,
                };

Could you please help shed some light on this? Note: In the code below, I copied your unminified v2 code directly into script blocks and removed the 2 at the end of each ResumableUploadForGoogleDrive to maintain compatibility with the exist HTML code.

<!DOCTYPE html>
<html>
  <head>
    <base target="_blank">
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Drive Multi Large File Upload</title>
    <link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/materialize/0.97.5/css/materialize.min.css">
    <style>
      .disclaimer{width: 480px; color:#646464;margin:20px auto;padding:0 16px;text-align:center;font:400 12px Roboto,Helvetica,Arial,sans-serif}.disclaimer a{color:#009688}#credit{display:none}
    </style>
  </head>
  <body>
    <form class="main" id="form" novalidate="novalidate" style="max-width: 480px;margin: 40px auto;">
      <div id="forminner">
        <div class="row">
          <div class="col s12">
            <h5 class="center-align teal-text">Vendor Upload Form</h5>
          </div>
        </div>
        <div class="row">
          <? if (!event_Date) { ?>
            <span style="color: red;">Warning: You are viewing the generic version of this page. A unique, event-specific URL should have been provided for uploading your files. Please contact our vendor team if you need assistance obtaining the correct link.
            <br><br>
            If your event does not require a customized URL, you may proceed with the form below.</span>
            <br><br>
          <? } ?>
          <div class="input-field col s12">
            <label for="eventDate" style="top: -20px;">Event Date</label>
            <input id="eventDate" type="date" name="Event Date" class="validate" required="required" aria-required="true"
                <? if (event_Date_Formatted) { ?>
                value="<?= event_Date_Formatted ?>"
                disabled
                <? } ?>>
          </div>
            <div class="input-field col s12">
              <input id="eventName" type="text" name="Event Name" class="validate" required="required" aria-required="true"
                <? if (event_Name) { ?>
                value="<?= event_Name ?>"
                disabled
                <? } ?> oninput="this.value = this.value.replace(/\s+/g, '')">
              <label for="eventName">Event Name</label>
            </div>
        </div>

        <div class="row">
          <div class="input-field col s12">
            <select id="brand" name="Brand" class="browser-default" required="required" aria-required="true">
              <label for="name">Brand</label>
              </div><script>
                const brandEnum = {
                  "Brand 1": "B1",
                  "Brand 2": "B2",
                };
              </script>

              <? if (event_Brand) { ?>
                <script>
                  document.addEventListener('DOMContentLoaded', function() {
                    const eventBrand = "<?= event_Brand ?>"; // Assuming eventBrand is available in the global scope
                    if (eventBrand) {
                      const brandValue = brandEnum[eventBrand];
                      const option = document.createElement('option');
                      option.value = brandValue;
                      option.selected = true;
                      option.textContent = eventBrand;
                      document.getElementById('brand').appendChild(option);
                    }
                  });
                </script>
              <? } else { ?>
                <option value="" disabled selected>Choose a brand</option>
                <option value="B1">Brand 1</option>
                <option value="B2">Brand 2</option>
              <? } ?>
            </select>
          </div>
        </div>

        <div class="row">
          <div class="input-field col s12">
            <select id="type" name="Event Type" class="browser-default" required="required" aria-required="true">
              <label for="type">Brand</label>
                <option value="" disabled selected>Choose an event type</option>
                <option value="Wedding">Wedding</option>
                <option value="Pre-Wedding">Pre-Wedding Event / Getting Ready</option>
                <option value="Post-Wedding">Post-Wedding Event / Gather After</option>
                <option value="Business_Event">Business Event</option>
                <option value="Engagement">Engagement</option>
                <option value="Family">Family</option>
                <option value="Maternity">Maternity</option>
                <option value="Newborn">Newborn</option>
                <option value="Senior">Senior</option>
                <option value="Portrait">Portrait</option>
                <option value="Editorial">Editorial</option> 
                <option value="Other">Other</option>
            </select>
          </div>
        </div>

        <div class="row" id="folderPathRow" style="display: none;">
          <p style="white-space: nowrap;">The folder path for this upload is:</br><strong>Products & Services/<span id="folderPathDisplay"></span>/<span id="folderNameDisplay"></span>/Unedited/<span id="typeNameDisplay"></span>/</strong></p>
            <script>
            document.addEventListener('DOMContentLoaded', function() {
              var brandField = document.getElementById("brand");
              var dateField = document.getElementById("eventDate");
              var nameField = document.getElementById("eventName");
              
              function updateEventName() {
              var dateValue = dateField.value.replace(/-/g, '');
              if (dateValue) {
                nameField.value = dateValue + '_' + nameField.value.replace(/^\d{8}_/, '');
              }
              }

              nameField.addEventListener('input', updateEventName);
              dateField.addEventListener('input', updateEventName);

              var typeField = document.getElementById("type");
              var folderPathRow = document.getElementById("folderPathRow");

              function updateFolderPath() {
              if (brandField.value && nameField.value && typeField.value && dateField.value) {
                var folderHtmlPath = brandField.value + "/" + "Client Content" + "/" + dateField.value.substring(0, 4)+ "/" + dateField.value.substring(5, 7);
                folderPathRow.style.display = "block";
                document.getElementById("folderNameDisplay").textContent = nameField.value;
                document.getElementById("typeNameDisplay").textContent = typeField.value;
                document.getElementById("folderPathDisplay").textContent = folderHtmlPath;
              } else {
                folderPathRow.style.display = "none";
              }
              }

              updateFolderPath();
              nameField.addEventListener('input', updateFolderPath);
              brandField.addEventListener('change', updateFolderPath);
              typeField.addEventListener('change', updateFolderPath);
              dateField.addEventListener('change', updateFolderPath);

              // Update eventName field with dateValue when the form loads
              updateEventName();
            });
            </script>
        </div>

        <div class="row">
          <div class="file-field input-field col s12">
            <div id="input-btn" class="btn">
              <span>File(s)</span>
              <input id="file" type="file" multiple>
            </div>
            <div class="file-path-wrapper">
              <input class="file-path validate" type="text" placeholder="Select one or more files">
            </div>
          </div>
        </div>

        <div class="row">
          <div class="input-field col s6">
            <button id="submit-btn" class="waves-effect waves-light btn submit-btn" type="submit" onclick="submitForm(); return false;">Upload</button>
          </div>
        </div>

        <div class="row">
          <div class="input-field col s12 hide" id="update">
            <hr>
            <p>
              Please wait as your file is being uploaded. Do not close or refresh the window while uploading.
            </p>
          </div>
        </div>
        <div class="row">
            <div class="input-field col s12" id="total">
            </div>
          </div>
        <div class="row">
          <div class="input-field col s12" id="progress">
          </div>
        </div>
      </div>
      <div id="success" style="display:none">
        <h5 class="left-align teal-text">File(s) Uploaded!</h5>
        <p>Everything has been uploaded</p>
      </div>
    </form>

    <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/materialize/0.97.5/js/materialize.min.js"></script>
    
    <script src="https://gumroad.com/js/gumroad.js"></script>


    <script>
      /**
 * ResumableUploadToGoogleDrive for Javascript library
 * GitHub  https://github.com/tanaikech/ResumableUploadForGoogleDrive2_js<br>
 * In this Class ResumableUploadToGoogleDrive2, the selected file is uploaded by splitting data on the disk. By this, the large file can be uploaded.<br>
 */
(function (r) {
  let ResumableUploadToGoogleDrive;
  ResumableUploadToGoogleDrive = (function () {
    function ResumableUploadToGoogleDrive() {
      this.obj = {};
      this.chunkSize = 52428800;
      this.endpoint =
        "https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable&supportsAllDrives=true";
    }

    /**
     * Run resumable upload a file.
     * @param {Object} resource the object for resumable uploading a file.
     */
    ResumableUploadToGoogleDrive.prototype.Do = async function (
      resource,
      callback
    ) {
      callback({ status: "initialize" }, null);
      try {
        this.obj = await init.call(this, resource);
      } catch (err) {
        callback(null, err);
        return;
      }
      try {
        const head = await getLocation.call(this);
        this.location = head.get("location");
        callback({ status: "getLocation" }, null);
        callback({ status: "start" }, null);
        const fileSize = resource.file.size;
        const len = Math.ceil(fileSize / this.chunkSize);
        for (let i = 0; i < len; i++) {
          let start = i * this.chunkSize;
          let end =
            fileSize < start + this.chunkSize
              ? fileSize
              : start + this.chunkSize;
          let data = resource.file.slice(start, end);
          end -= 1;
          callback(
            {
              status: "Uploading",
              progressNumber: { current: i, end: len },
              progressByte: {
                current: start,
                end: end,
                total: fileSize,
              },
            },
            null
          );
          try {
            const res = await getFile.call(this, {
              fileSize,
              len,
              start,
              end,
              data,
              i,
            });
            if (
              res.status == "Next" ||
              (res.status == "Done" && i == len - 1)
            ) {
              callback(res, null);
            } else {
              callback(null, "Internal error.");
              return;
            }
          } catch (err) {
            callback(null, err);
            return;
          }
        }
      } catch (err) {
        callback(null, err);
        return;
      }
    };

    const init = function (resource) {
      return new Promise((resolve, reject) => {
        if (!("accessToken" in resource) || !("file" in resource)) {
          reject({
            Error:
              "There are no required parameters. accessToken, fileName, fileSize, fileType and fileBuffer are required.",
          });
          return;
        }
        let object = {};
        object.resource = resource;
        if (
          "chunkSize" in resource &&
          resource.chunkSize >= 262144 &&
          resource.chunkSize % 262144 == 0
        ) {
          this.chunkSize = resource.chunkSize;
        }
        if ("fields" in resource && resource.fields != "") {
          this.endpoint += "&fields=" + encodeURIComponent(resource.fields);
        }
        if ("convertToGoogleDocs" in resource && resource.convertToGoogleDocs) {
          fetch(
            "https://www.googleapis.com/drive/v3/about?fields=importFormats",
            {
              method: "GET",
              headers: { Authorization: "Bearer " + resource.accessToken },
            }
          )
            .then((res) => {
              if (res.status != 200) {
                res.json().then((e) => reject(e));
                return;
              }
              res.json().then((res) => {
                if (resource.file.type in res.importFormats) {
                  object.resource.fileType =
                    res.importFormats[resource.fileType][0];
                }
                resolve(object);
              });
            })
            .catch((err) => {
              reject(err);
            });
        } else {
          resolve(object);
        }
      });
    };

    const getLocation = function () {
      return new Promise((resolve, reject) => {
        const resource = this.obj.resource;
        const accessToken = resource.accessToken;
        let metadata = {
          mimeType: resource.file.type,
          name: resource.file.name,
        };
        if ("folderId" in resource && resource.folderId != "") {
          metadata.parents = [resource.folderId];
        }
        fetch(this.endpoint, {
          method: "POST",
          body: JSON.stringify(metadata),
          headers: {
            Authorization: "Bearer " + accessToken,
            "Content-Type": "application/json",
          },
        })
          .then((res) => {
            if (res.status != 200) {
              res.json().then((e) => reject(e));
              return;
            }
            resolve(res.headers);
          })
          .catch((err) => {
            reject(err);
          });
      });
    };

    const getFile = function ({ fileSize, len, start, end, data, i }) {
      const location = this.location;
      return new Promise(function (resolve, reject) {
        const fr = new FileReader();
        fr.onload = async function () {
          const buf = fr.result;
          const obj = {
            data: new Uint8Array(buf),
            length: end - start + 1,
            range: "bytes " + start + "-" + end + "/" + fileSize,
            startByte: start,
            endByte: end,
            total: fileSize,
            cnt: i,
            totalChunkNumber: len,
          };
          await doUpload(obj, location)
            .then((res) => resolve(res))
            .catch((err) => reject(err));
        };
        fr.readAsArrayBuffer(data);
      });
    };

    const doUpload = function (e, url) {
      return new Promise(function (resolve, reject) {
        fetch(url, {
          method: "PUT",
          body: e.data,
          headers: { "Content-Range": e.range },
        })
          .then((res) => {
            const status = res.status;
            if (status == 308) {
              resolve({ status: "Next", result: r });
            } else if (status == 200) {
              res.json().then((r) => resolve({ status: "Done", result: r }));
            } else {
              res.json().then((err) => {
                err.additionalInformation =
                  "When the file size is large, there is the case that the file cannot be converted to Google Docs. Please be careful this.";
                reject(err);
                return;
              });
              return;
            }
          })
          .catch((err) => {
            reject(err);
            return;
          });
      });
    };

    return ResumableUploadToGoogleDrive;
  })();

  return (r.ResumableUploadToGoogleDrive = ResumableUploadToGoogleDrive);
})(this);
    </script>
    <script>

      //const chunkSize = 52428800;
      const uploadParentFolderId = "0AJYnVPHJrxDBUk9PVA"; // creates a folder inside of this folder

        function submitForm() {

            if ($('#submit-btn.disabled')[0]) return; // short circuit
            var date = $('#eventDate').val();
            var name = $('#eventName').val();
            var brand = $('#brand').val();
            var files = [...$('#file')[0].files]; // convert from FileList to array

            // Error validation
            if (!date || !name || !brand || files.length === 0) {
              showError("Please fill in all required fields and select a file.");
              return;
            }

            disableForm(); // prevent re submission

            var folderPath    = document.getElementById("brand").value + "/" +
                                "Client Content" + "/" +
                                document.getElementById("eventDate").value.substring(0, 4) + "/" +
                                document.getElementById("eventDate").value.substring(5, 7);
            var folderName = document.getElementById("eventName").value;
            var typeName = document.getElementById("type").value;
            console.log("The folderPath is: " + folderPath)

            const newFolderObj = createOrGetFolder(folderPath, folderName, typeName, uploadParentFolderId).then(newFolderId => {
                console.log("Found or created guest folder with id: " + newFolderId);
                google.script.run.withSuccessHandler(accessToken => ResumableUploadForGoogleDrive(accessToken, newFolderId)).getOAuthToken();
            });
        
        }

        function upload({ accessToken, file, idx, newFolderId }) {
            return new Promise((resolve, reject) => {
                let fr = new FileReader();
                fr.fileName = file.name;
                fr.fileSize = file.size;
                fr.fileType = file.type;
                fr.readAsArrayBuffer(file);
                fr.onload = e => {
                var id = `p_${idx}`;
                var div = document.createElement("div");
                div.id = id;
                document.getElementById("progress").appendChild(div);
                document.getElementById(id).innerHTML = "Initializing.";
                const f = e.target;
                const resource = {
                    fileName: f.fileName,
                    fileSize: f.fileSize,
                    fileType: f.fileType,
                    fileBuffer: f.result,
                    accessToken: accessToken,
                    folderId: newFolderId,
                };
                const ru = new ResumableUploadToGoogleDrive();
                ru.Do(resource, function (res, err) {
                    if (err) {
                    reject(err);
                    return;
                    }
                    console.log(res);
                    let msg = "";
                    if (res.status == "Uploading") {
                    msg = Math.round((res.progressNumber.current / res.progressNumber.end) * 100) + `% (${f.fileName})`;
                    } else {
                    msg = `${res.status} (${f.fileName})`;
                    }
                    if (res.status == "Done") {
                    resolve(res.result);
                    }
                    document.getElementById(id).innerText = msg;
                });
                };
            });
        }

        async function ResumableUploadForGoogleDrive(accessToken, newFolderId) {

            const n = 10; // You can adjust the chunk size.
            const f = document.getElementById("file");
            const files = [...f.files].sort((a, b) => a.name.localeCompare(b.name));
            var totalFiles = files.length.toString();
            const splitFiles = [...Array(Math.ceil(files.length / n))].map((_) => files.splice(0, n));
            console.log("The splitFiles are: " + splitFiles)
            console.log("The total number of files is: " + totalFiles);
            document.getElementById("total").innerHTML = `Total Files: ${totalFiles}`;

            for (let i = 0; i < splitFiles.length; i++) {
                const percentCompletion = Math.floor((i / splitFiles.length) * 100);
                document.getElementById("progress").innerHTML = `Upload Progress: ${percentCompletion}&percnt; (Refreshs after upload group of ${n})`;
                const res = await Promise.all(splitFiles[i].map(async (file, j) => await upload({ accessToken, file, idx: `${i}_${j}`, newFolderId })));
                console.log(res);
            }
            document.getElementById("progress").innerHTML = `Upload Progress: 100&percnt;`;
        }

        function disableForm() {
            $('#submit-btn').addClass('disabled');
            $('#input-btn').addClass('disabled');
            $('#update').removeClass('hide');
        }
    
        function createOrGetFolder(folderPath, folderName, typeName, uploadParentFolderId) {
            return new Promise((resolve, reject) => {
            google.script.run.withSuccessHandler(response => {
                console.log("createOrGetFolder response: ", response);
                if (response && response.length) {
                resolve(response);
                }
                reject(response);
            }).createOrGetFolder(folderPath, folderName, typeName, uploadParentFolderId);
            });
        }

        function showSuccess() {
            $('#forminner').hide();
            $('#success').show();
        }

        function showError(e) {
            $('#progress').addClass('red-text').html(e);
        }

        function showMessage(e) {
            $('#update').html(e);
        }

        function showProgressMessage(e) {
        $('#progress').removeClass('red-text').html(e);
      }

    </script>

  </body>

</html>

The text was updated successfully, but these errors were encountered:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error when replacing ResumeableUpload with ResumableUpload2 #10

Error when replacing ResumeableUpload with ResumableUpload2 #10

SysEngDan commented Nov 6, 2024 •

edited

Loading

Error when replacing ResumeableUpload with ResumableUpload2 #10

Error when replacing ResumeableUpload with ResumableUpload2 #10

Comments

SysEngDan commented Nov 6, 2024 • edited Loading

SysEngDan commented Nov 6, 2024 •

edited

Loading