size of chcp.manifest file

[lorenzyannick]

I tried to benchmark your plugin by adding approx 5000 dummy empty files to the project. So the size of the file chcp.manifest explode (normal)

• classic generation output : 450KB
• minimified version : 360KB
• zip version : 17KB

So if there a a big amount of file, to improve the update process, we maybe first need to minimify the file or zip it.
For the min version everyone can use nodeJS like json-minify
For the zip solution the plugin should handle it.
Is there a way to do it ?

The installation phase took about 1minute now, just to update one file. (on my iPhone 6, should be worse on iPhone 5 or even old Android device)
It's because the checksum is calculated 5000 times "for nothing" and also moving the local files 5000 times

[Wed, 13 Jan 2016 14:42:01 GMT] "GET /chcp.json" "myAppHotCode/70 CFNetwork/711.5.6 Darwin/14.0.0"
[Wed, 13 Jan 2016 14:42:01 GMT] "GET /chcp.manifest" "myAppHotCode/70 CFNetwork/711.5.6 Darwin/14.0.0"
[Wed, 13 Jan 2016 14:42:11 GMT] "GET /css/style.css" "myAppHotCode/70 CFNetwork/711.5.6 Darwin/14.0.0"
[Wed, 13 Jan 2016 14:43:07 GMT] "GET /chcp.json" "myAppHotCode/70 CFNetwork/711.5.6 Darwin/14.0.0"

Maybe we can think about an improvement to prevent that time to explode ?
One thing is to reduce the project file first before using this plugin

thx @nikDemyankov for all your great work :)

[nikDemyankov]

Thanks for benchmarking, quite an interesting numbers.

Maybe we can think about an improvement to prevent that time to explode ?

Yes, there is always something that can be optimised and improved. Never sad, that it is perfect :) Right now I've been focused on making plugin more stable, postponing new features and performance improvements. One of the suggestions from the people is using gzip: #69 . It should increase the speed of the download process.

Regarding minify:
Yes, developer should do that, if he has a lot of project files. The only thing he need to keep in mind - minified version should be placed both on the server and in the app, and manifest file should be build for minified files.

Regarding using archives:
At the very beginning of the plugin development we were choosing between using archives and manifest file. We decided to go with the manifest, since it is simpler from the logic perspective and doesn't require additional maintenance from the developers perspective. Let's say you store in archives only updated files from the previous release. And there were version A, B, C, D, E, F. As a result, you will have to store on your server archives with changed files for each of them. Let's say, some user has version A on his device. In order to update to F - plugin need to load all previous versions from the server, before he can install the new one. With the manifest approach we remove this dependency: plugin compares manifests and loads only missing/changed files.

It's because the checksum is calculated 5000 times "for nothing"

That's strange, because it should have calculate checksum only for the updated files. If you changed only one file - it should have calculated it only once for that particular file.

and also moving the local files 5000 times

Maybe moving 5000 files for one time? :)

[lorenzyannick]

So you mean the checksum is not re-calculated on smartphone side, the plugin just compare the 2 manifest files ?

The fact is that there is something with the 5000 files
5000 moves ? 5000 files calcultated checksum on smartphone ? 5000 comparaison betwwen 2 manifest files ?
something that took time related to the 5000 dummy files

I agree with the approch of the manifest file that remove the dependency :)

[nikDemyankov]

So you mean the checksum is not re-calculated on smartphone side, the plugin just compare the 2 manifest files ?

On the update fetching step plugin loads manifest from the server and compares it with the manifest of his current working directory. It searches for new/changed files and then loads them from the server. When each file is loaded from the server - plugin checks his md5 hash to make sure, that it is not corrupted.

On the installation step plugin once more checks loaded files by calculating their checksum. Just in case.

So yes, it is calculated only 2 times for each changed/new file: then file is loaded from the server and just before the installation. So, if you changed 1 file - it should have been executed only two times.

something that took time related to the 5000 dummy files

At the end of this part of the docs you can find a bit more details about how updates are prepared and installed. In short, for each new release new folder is created. In it plugin places loaded files from the server, plus all unchanged files from the current working release on users device. This is why it copies 5000 files for the new release.

Before v1.2 there was another approach: there was only one directory for each release (with some tweaks). As a result, plugin didn't have to copy all the files for each update. But this led to some critical issues:

1. On iOS .css files was not updating properly, because they were cached by the UIWebView. And to drop the cache you either need to force-kill the app, or add a hack on JS side, that would add timestamp to the .css files path. Now it is solved, since for each release we have new folder -> new path to .css files -> cache is dropped automatically.
2. Since the updated folder was the same, as the one, that is displayed to the user - during the update it could break and user would end up with not working app. Now - release is installed in a totally different folder and even if it breaks - user is not effected. Plus in the future we can add something like chcp.rollbackTo(version).

So, moving to "new release - new folder" scheme add's some additional time on the installation process, but makes it safer for the user and developer.

[lorenzyannick]

Fully understand, that make sense, thx

[dwhitmore]

I'm kind of on the same path. I'm at a point where I keep getting error code -4 due to timing out. Where can this setting be updated?

[nikDemyankov]

@dwhitmore can you provide more details about it? -4 error include different stuff. For example, it can occur if plugin can't find some file on the server, that exists in the manifest file.

[dwhitmore]

Description is ''the request timed out'. I've verified the files are in the correct location. It happened when I loaded some videos in an update for offline viewing. I was assuming the download was too large.

[nikDemyankov]

@dwhitmore strange, will check it out. Didn't test it on large files, since didn't plan to use it like so. As wrote before, If you need to load some large files from the server, that user want to see - you should, probably, not add it in the app itself, but download when application starts, providing some loading progress to the user.

[dwhitmore]

These files need to be installed with the app with occasional updates as
there will be no guarantee when/if they're online. Tried a stress test of
around 300MB when this happened, which is entirely possible for this app.
Thanks for looking into it, I was hoping there was a simple timeout
somewhere but I couldn't find it.

On Thu, Jan 14, 2016 at 10:59 AM, Nikolay Demyankov <
[email protected]> wrote:

@dwhitmore https://github.com/dwhitmore strange, will check it out.
Didn't test it on large files, since didn't plan to use it like so. As
wrote before, If you need to load some large files from the server, that
user want to see - you should, probably, not add it in the app itself, but
download when application starts, providing some loading progress to the
user.

—
Reply to this email directly or view it on GitHub
#89 (comment)
.

[dwhitmore]

Ipad shows 4GB remaining in memory. App is around 500MB. Updates around 200MB mostly work, 300MB updates reliably crash:

Console shows: malloc: *** mach_vm_map(size=147456) failed (error code=3)
Logs show: EXC_BAD_ACCESS (SIGSERV) usually triggered around an com.apple.NSURLConnectionLoader

This is during the fetch, before the download. How exactly does the process work, I'm assuming every update creates a new container, fetches new files there and then copies over unchanged files. When is the old container removed? Or does it not get removed?

Thanks for all your help.

[nikDemyankov]

Not exactly. It is described in this part of documentation, starting from:

For example, lets say that currently in the app we are running version 2015.12.01-12.01.33. That means the following:
...

Also, some info can be found in the messages above.

On the fetch step plugin creates a cache folder on file system and downloads in it new content. On the install step plugin creates a copy of the existing www folder and copies in it new/changed files. Loaded files removed from cache folder. But previous version of the content is not removed: it is a backup in case if current working version breaks. On the next app launch plugin removes old releases from the file system, except current and a backup.

So, if your www folder is 500Mb - then after few releases app will consume: 500Mb for the www folder in the bundle, 500Mb for current working www on the external storage , 500Mb of backup. In total: 1500Mb.

[dwhitmore]

Looking at the logs, it looks to be a memory error when downloading. Are
the files downloaded synchronously and how is space allocated? This
particular download had 1 html file and 2 mp4's changed in it. The crash
happens during the second mp4 download, server logs show request was
stopped. How is memory allocation handled during fetches?

Thanks again.

On Fri, Jan 15, 2016 at 10:42 AM, Nikolay Demyankov <
[email protected]> wrote:

Not exactly. It is described in this part of documentation
https://github.com/nordnet/cordova-hot-code-push#how-web-project-files-are-stored-and-updated,
starting from:

For example, lets say that currently in the app we are running version 2015.12.01-12.01.33. That means the following:
...

Also, some info can be found in the messages above.

On the fetch step plugin creates a cache folder on file system and
downloads in it new content. On the install step plugin creates a copy of
the existing www folder and copies in it new/changed files. Loaded files
removed from cache folder. But previous version of the content is not
removed: it is a backup in case if current working version breaks. On the
next app launch plugin removes old releases from the file system, except
current and a backup.

So, if your www folder is 500Mb - then after few releases app will
consume: 500Mb for the www folder in the bundle, 500Mb for current
working www on the external storage , 500Mb of backup. In total: 1500Mb.

—
Reply to this email directly or view it on GitHub
#89 (comment)
.

[nikDemyankov]

For download on iOS plugin uses NSURLSession. Basically, with default configuration. So, I think it just loads data into device RAM, and then calls a completion block with loaded NSData. Probably that is why it is dying.

To fix this, it should load files by small data chunks and write them into file. As said before: didn't plan to use plugin for loading large files, so this is not implemented.

[nikDemyankov]

Probably, use this delegate method.

[nordnet-deprecation-bot]

👋 Hi! Thank you for your interest in this repo.

😢 We are not using nordnet/cordova-hot-code-push anymore, and we lack the manpower and the experience needed to maintain it. We are aware of the inconveniece that this may cause you. Feel free to use it as is, or create your own fork.

🔒 This will now be closed & locked.

ℹ️ Please see #371 for more information.