Merge pull request #193 from VeliovGroup/dev

v1.7.0 - Compatibility with [email protected] - Fix memory leak caused by unsatisfied queue in `writeStream` class - New authentication model for `http` uploads based on more secure and frequently changed `connectionId` see [#159](#159) - Fixed overwhelming requests to `Meteor.users` on every received chunk - Minimised amount of data transferred between WebWorker and MainThread - Use minified version of WebWorker - `http` upload uses HTTP requests for all tasks `start`, `continue`, `EOF`, previously Meteor methods was involved - Overall code enhancements, DRY - Game changing upload via [webrtc-data-channel](https://github.com/VeliovGroup/Meteor-Files/tree/webrtc-data-channel), It’s on separate branch, I’m waiting for your feedback on this, please test it on your end
veliovgroup · Aug 20, 2016 · 7a63327 · 7a63327
2 parents 88df884 + 4f7c0a8
commit 7a63327
Show file tree

Hide file tree

Showing 15 changed files with 217 additions and 123 deletions.
diff --git a/.versions b/.versions
@@ -38,7 +38,7 @@ [email protected]
 [email protected]
 [email protected]
 ostrio:[email protected]
-ostrio:files@1.6.10
+ostrio:files@1.7.0
 [email protected]
 [email protected]
 [email protected]

diff --git a/README.md b/README.md
@@ -6,6 +6,8 @@
 [![Twitter](https://img.shields.io/twitter/url/https/github.com/VeliovGroup/Meteor-Files.svg?style=social)](https://twitter.com/intent/tweet?url=https%3A%2F%2Fgithub.com%2FVeliovGroup%2FMeteor-Files)
 [![Codewake](https://www.codewake.com/badges/codewake2.svg)](https://www.codewake.com/p/meteor-files)
 
+#### Please see our experimental [webrtc-data-channel](https://github.com/VeliovGroup/Meteor-Files/tree/webrtc-data-channel) branch. Feedback is highly appreciated!
+
 ToC:
 ========
  - [Wiki](https://github.com/VeliovGroup/Meteor-Files/wiki) - Full documentation
@@ -74,7 +76,7 @@ Related Packages:
 Why `Meteor-Files`?
 ========
 The `cfs` is a well known package, but it's huge monster which combines everything. In `Meteor-Files` is nothing to broke, it's simply upload/store/serve files to/from server.
- - Support for both `HTTP` and `DDP` transports for upload, [read about difference](https://github.com/VeliovGroup/Meteor-Files/wiki/About-Upload-Transports)
+ - Support for `HTTP`, [RTC/DC](https://github.com/VeliovGroup/Meteor-Files/tree/webrtc-data-channel) and `DDP` transports for upload, [read about difference](https://github.com/VeliovGroup/Meteor-Files/wiki/About-Upload-Transports)
  - You need store to *[GridFS](https://github.com/VeliovGroup/Meteor-Files/wiki/GridFS-Integration)*, *[AWS S3](https://github.com/VeliovGroup/Meteor-Files/wiki/AWS-S3-Integration)*, *[Google Storage](https://github.com/VeliovGroup/Meteor-Files/wiki/Google-Cloud-Storage-Integration)* or *[DropBox](https://github.com/VeliovGroup/Meteor-Files/wiki/DropBox-Integration)*? (*[Use 3rd-party storage](https://github.com/VeliovGroup/Meteor-Files/wiki/Third-party-storage)*) - *Add it yourself*
  - You need to check file mime-type, size or extension? (*[`onBeforeUpload`](https://github.com/VeliovGroup/Meteor-Files/wiki/Constructor)*) - *Add it yourself*
  - You need to [resize images](https://github.com/VeliovGroup/Meteor-Files/blob/master/demo/server/image-processing.coffee) after upload? (*[`onAfterUpload`](https://github.com/VeliovGroup/Meteor-Files/wiki/Constructor)*, *[file's subversions](https://github.com/VeliovGroup/Meteor-Files/wiki/Create-and-Manage-Subversions)*) - *Add it yourself*
@@ -305,6 +307,7 @@ For more expressive example see [Download demo](https://github.com/VeliovGroup/M
 Supporters:
 ========
 I would like to thank everyone who support this project. *Because of those guys this project can have 100% of our attention*.
+ - [@Neobii](https://github.com/Neobii)
  - [@themeteorchef](https://github.com/themeteorchef)
  - [@MeDBejoHok](https://github.com/medbejohok)
  - [@martunta](https://github.com/martunta)

diff --git a/demo/.meteor/.finished-upgraders b/demo/.meteor/.finished-upgraders
@@ -13,3 +13,4 @@ notices-for-facebook-graph-api-2
 1.3.0-split-minifiers-package
 1.3.5-remove-old-dev-bundle-link
 1.4.0-remove-old-dev-bundle-link
+1.4.1-add-shell-server-package
diff --git a/demo/.meteor/packages b/demo/.meteor/packages
@@ -1,23 +1,23 @@
 # Basic packages
 [email protected]
 [email protected]
-[email protected].10
+[email protected].11
 [email protected]
 [email protected]
 [email protected]
 [email protected]
 
 seba:minifiers-autoprefixer
-standard-minifier-js@1.1.8
+standard-minifier-js@1.2.0
 
 # Additional MDG packages
 [email protected]
 [email protected]
-[email protected].3
+[email protected].4
 [email protected]
 [email protected]
 [email protected]
-[email protected].8
+[email protected].9
 
 # UI/UX packages
 [email protected]
@@ -48,9 +48,10 @@ ostrio:flow-router-meta
 ostrio:spiderable-middleware
 
 # Accounts Packages
-[email protected].9
+[email protected].11
 [email protected]
 [email protected]
 [email protected]
 [email protected]
 [email protected]
+shell-server
diff --git a/demo/.meteor/release b/demo/.meteor/release
@@ -1 +1 @@
-[email protected].0.1
+[email protected]
diff --git a/demo/.meteor/versions b/demo/.meteor/versions
@@ -1,4 +1,4 @@
-[email protected].10
+[email protected].11
 [email protected]
 [email protected]
 [email protected]
@@ -13,31 +13,31 @@ [email protected]
 arillo:[email protected]
 [email protected]
 [email protected]
-[email protected].0
-[email protected].10
+[email protected].1
+[email protected].11
 [email protected]
 [email protected]
 [email protected]
 [email protected]
 [email protected]
 [email protected]
-[email protected].6
+[email protected].7
 [email protected]
 [email protected]
 cfs:[email protected]
 [email protected]
-[email protected].3
+[email protected].4
 [email protected]
-[email protected].0
+[email protected].1
 [email protected]
 [email protected]
-[email protected].9
+[email protected].10
 [email protected]
 [email protected]
-[email protected].7
-[email protected].13
+[email protected].8
+[email protected].14
 [email protected]
-[email protected].8
+[email protected].9
 [email protected]
 fortawesome:[email protected]
 fourseven:[email protected]
@@ -46,49 +46,49 @@ [email protected]
 [email protected]
 [email protected]
 [email protected]
-[email protected].8
+[email protected].9
 [email protected]
 [email protected]
 [email protected]
 [email protected]
 [email protected]
-[email protected].14
+[email protected].15
 [email protected]
-mdg:validation-error@0.2.0
-[email protected].16
+mdg:validation-error@0.5.1
+[email protected].17
 [email protected]
 [email protected]
 meteorhacks:[email protected]
-[email protected].13
-[email protected].13
+[email protected].14
+[email protected].14
 [email protected]
 [email protected]
 [email protected]
 [email protected]
-[email protected].5
-[email protected].5
+[email protected].6
+[email protected].6
 momentjs:[email protected]
-[email protected].10
+[email protected].11
 [email protected]
 mquandalle:[email protected]
 mquandalle:[email protected]
 mrt:[email protected]
-[email protected].45
+[email protected].46
 [email protected]
 [email protected]
 [email protected]
 [email protected]
 [email protected]
 ostrio:[email protected]
 ostrio:[email protected]
-ostrio:files@1.6.10
+ostrio:files@1.7.0
 ostrio:[email protected]
 ostrio:[email protected]
 ostrio:[email protected]
 ostrio:[email protected]
 ostrio:[email protected]
 perak:[email protected]
-[email protected].3
+[email protected].4
 raix:[email protected]
 [email protected]
 [email protected]
@@ -100,17 +100,18 @@ [email protected]
 seba:[email protected]
 [email protected]
 [email protected]
+[email protected]
 simple:[email protected]
 [email protected]
 [email protected]
-standard-minifier-js@1.1.8
-[email protected].13
+standard-minifier-js@1.2.0
+[email protected].14
 [email protected]
 [email protected]
 [email protected]
 [email protected]
 [email protected]
 [email protected]
-[email protected].10
+[email protected].11
 [email protected]
 zimme:[email protected]
diff --git a/demo/client/upload/upload-form.jade b/demo/client/upload/upload-form.jade
@@ -13,7 +13,7 @@ template(name="uploadForm")
               | DDP
             label.radio-inline
               input(type="radio" name="transport" value="http" checked="{{#if compare uploadTransport 'is' 'http'}}checked{{/if}}")
-              | HTTP (<em>3x performance</em>)
+              | HTTP (<em>3x</em>)
 
           p.annotation Any file-type.<br />With size less or equal to 128MB,<br />up to 6 files.
           p.annotation This form supports drag'n'drop.

diff --git a/demo/server/service-configurations.coffee b/demo/server/service-configurations.coffee
@@ -42,5 +42,5 @@ if process.env['ACCOUNTS_FACEBOOK_ID'] and process.env['ACCOUNTS_FACEBOOK_SEC']
       loginStyle: 'redirect'
 
 Meteor.methods
-  'getServiceConfiguration': ->
+  getServiceConfiguration: ->
     return _sc
diff --git a/docs/about-transports.md b/docs/about-transports.md
@@ -11,7 +11,7 @@ The pros:
 The cons: 
 
  - “magic” overhead inside DDP and EJSON;
- - Only one data-transfer per time unit;
+ - Only one data-transfer per time unit (*blocks other DDP requests, like methods, subs, etc.*);
  - It's synchronous.
 
 #### HTTP (TCP/IP)
@@ -32,4 +32,18 @@ The cons:
 
 
 #### RTC Data Chanel (UDP)
-*We're working on it*
+This transport supported only in [webrtc-data-channel](https://github.com/VeliovGroup/Meteor-Files/tree/webrtc-data-channel) branch. It's in testing mode, we're waiting for community feedback, before merging to `master`. If you're interested in RTC/DC uploads, try this branch locally. Any feedback on RTC/DC usage for uploads is highly appreciated!
+
+The pros:
+
+ - Single socket connection;
+ - Direct tunneled connection from Client to Server;
+ - Pure binary data support;
+ - Native implementation and support on mobile devices;
+ - It's UDP.
+
+The cons:
+
+ - No mobile browsers support;
+ - Chunk size limited to 64KB;
+ - Only single stream is supported (*so, it's currently uses synchronous chunk uploads*. If `RTC/DC` will be accepted by community we will implement asynchronous chunks upload).
diff --git a/docs/insert.md b/docs/insert.md
@@ -1,5 +1,5 @@
 ##### `insert(settings[, autoStart])` [*Client*]
-*Upload file to Server via DDP.*
+*Upload file to Server via DDP, [RTC/DC](https://github.com/VeliovGroup/Meteor-Files/tree/webrtc-data-channel) or HTTP.*
 
 <table>
   <thead>
@@ -126,7 +126,7 @@
         Callback, triggered after chunk is sent<br />
         <strong>Arguments</strong>:
         <ul>
-          <li><code>progress</code> {<em>Number</em>} - Current progress from `0` to `100`</li>
+          <li><code>progress</code> {<em>Number</em>} - Current progress from <code>0</code> to <code>100</code></li>
           <li><code>fileData</code> {<em>Object</em>}</li>
         </ul>
       </td>
@@ -281,6 +281,18 @@
       </td>
       <td></td>
     </tr>
+    <tr>
+      <td align="right">
+        <code>pipe()</code> {<em>Function</em>}
+      </td>
+      <td>
+        Pipe data before upload
+      </td>
+      <td>
+        All data must be in <code>data URI</code> scheme (*Base64*)<br />
+        Only for <code>webrtc</code> transport data represented as <code>ArrayBuffer</code>.
+      </td>
+    </tr>
     <tr>
       <td align="right">
         <code>estimateTime</code> {<em>ReactiveVar</em>}
@@ -321,7 +333,7 @@
   </tbody>
 </table>
 
-### Event map:
+### Events map:
 <table>
   <thead>
     <tr>
@@ -387,7 +399,7 @@
         Triggered after each chunk is sent.<br />
         <strong>Arguments</strong>:
         <ul>
-          <li><code>progress</code> {<em>Number</em>} - Current progress from `0` to `100`</li>
+          <li><code>progress</code> {<em>Number</em>} - Current progress from <code>0</code> to <code>100</code></li>
           <li><code>fileData</code> {<em>Object</em>}</li>
         </ul>
       </td>
@@ -609,6 +621,8 @@ Template.uploadForm.events({
 ```
 
 ##### Piping:
+Note: data flow in `webrtc` transport uses ArrayBuffer, while `ddp` and `http` uses dataURI (*Base64*). `webrtc` is available only on [webrtc-data-channel](https://github.com/VeliovGroup/Meteor-Files/tree/webrtc-data-channel) branch.
+
 ```javascript
 var encrypt = function encrypt(data) {
   return someHowEncryptAndReturnAsBase64(data);

diff --git a/docs/toc.md b/docs/toc.md
@@ -14,11 +14,12 @@ Meteor-Files
   </tbody>
 </table>
 
+Please see our experimental [webrtc-data-channel](https://github.com/VeliovGroup/Meteor-Files/tree/webrtc-data-channel) branch. Feedback is highly appreciated!
 
 ### About:
  - Event-driven API
  - Upload / Read files in Cordova app: __Cordva support__ (Any with support of `FileReader`)
- - Upload via *HTTP* or *DDP*, [read about difference](https://github.com/VeliovGroup/Meteor-Files/wiki/About-Upload-Transports)
+ - Upload via *HTTP*, [*RTC/DC*](https://github.com/VeliovGroup/Meteor-Files/tree/webrtc-data-channel) or *DDP*, [read about difference](https://github.com/VeliovGroup/Meteor-Files/wiki/About-Upload-Transports)
  - File upload:
     * Ready for small and large files (RAM used only for chunk reading - [read about `chunkSize`](https://github.com/VeliovGroup/Meteor-Files/wiki/Insert-(Upload)))
     * Pause / Resume upload