Skip to content
This repository has been archived by the owner on Jun 21, 2024. It is now read-only.

Export/Import Versions #39

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Export/Import Versions #39

wants to merge 7 commits into from

Conversation

jvillafanez
Copy link
Member

@jvillafanez jvillafanez commented Oct 17, 2018

Summing up the list of changes:

  • Refactor models according to Model changes #35
  • Include and use the FSAccess class to handle FS access instead of symfony's filesystem. For the extra features added for now:
    • recursive folder creation on some operations
    • stream usage for big file handling
    • jail operations inside a specific root folder (needs hardening)
  • Import and export versions
  • Adjust code in the serializer and nodeIteratorFactory for these changes

Known issues:

  • Version timestamps aren't kept. For local storage this means that some versions might disappear due to policy constraints.
  • On S3, an extra dummy version of 0 bytes is always created regardless of the file having versions or not, or the number of versions.

Pending things on this PR:

  • code style fixes
  • unittest coverage

Additional things pending to do later (they won't be done in this PR):

  • Refactor importers to be aligned with the new VersionImporter (change proposal still to be created)
  • Refactor extractors (change proposal still to be created)

Closes #28

@codecov-io
Copy link

codecov-io commented Oct 17, 2018

Codecov Report

Merging #39 into master will decrease coverage by 3.98%.
The diff coverage is 17.97%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master      #39      +/-   ##
============================================
- Coverage     41.16%   37.18%   -3.99%     
- Complexity      204      260      +56     
============================================
  Files            29       35       +6     
  Lines           838     1003     +165     
============================================
+ Hits            345      373      +28     
- Misses          493      630     +137
Impacted Files Coverage Δ Complexity Δ
...xporter/MetadataExtractor/PreferencesExtractor.php 100% <ø> (ø) 4 <0> (ø) ⬇️
lib/Importer/MetadataImporter/ShareImporter.php 0% <ø> (ø) 25 <0> (ø) ⬇️
lib/Model/UserMetadata/User/Preference.php 100% <ø> (ø) 6 <0> (?)
lib/Exporter/MetadataExtractor/UserExtractor.php 93.75% <ø> (ø) 3 <0> (ø) ⬇️
lib/Importer/MetadataImporter/UserImporter.php 84.37% <ø> (ø) 12 <0> (ø) ⬇️
.../Importer/MetadataImporter/PreferencesImporter.php 100% <ø> (ø) 3 <0> (ø) ⬇️
lib/Exporter/MetadataExtractor/SharesExtractor.php 65.76% <ø> (ø) 16 <0> (ø) ⬇️
lib/Model/UserMetadata/User/Share.php 0% <ø> (ø) 18 <0> (?)
lib/Command/ExportUser.php 0% <0%> (ø) 5 <1> (+1) ⬆️
lib/Importer/FilesImporter.php 0% <0%> (ø) 10 <10> (+4) ⬆️
... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c2476b5...da0098b. Read the comment docs.

Copy link
Contributor

@IljaN IljaN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, except a few small things. 👍

* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
*
*/
namespace OCA\DataExporter\Utilities\FSAccess;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move FSAccess out of Utility namespace in to the root. I dislike "Utilities" namespaces or similar because such namespace tend to be poluted with different classes where devs are too lazy to think about relationships.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also rename to FilesystemAccess if possible

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

* Also note that concurrency isn't considered, so using this class could cause
* bugs in a concurrent environment.
*/
class FSAccess {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 For this wrapper, tried to unit-test my stream_copy changes -> very ugly

*/
namespace OCA\DataExporter\Utilities\FSAccess;

class FSAccessFactory {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tend to create factories only if we need different constructions, don't see why we can't construct directly, but 🆗

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think more than 90% of the services registered in the DIContainer behave like a singleton class: although the class isn't designed as singleton, normally all the objects receive the same instance.

It's unlikely that we want the same instance in any place, so this means that we either define each service with the specific FSAccess instance to be used (increasing the boilerplate per class requiring this) or we define only one fixed instance to be used as service.
Note that the FSAccess instance is designed in a way that prevents changing the base root folder.

In addition, all the dependency wiring is defined before the commands are executed. The problem is that the target path where the exported info resides is known during the command execution. We can't place the jail during the dependency wiring because at that point is unknown.

@IljaN IljaN added this to the development milestone Oct 19, 2018
@IljaN IljaN mentioned this pull request Oct 19, 2018
9 tasks
@jvillafanez jvillafanez mentioned this pull request Oct 19, 2018
@micbar
Copy link
Contributor

micbar commented Apr 24, 2019

@jvillafanez @IljaN

This PR has been approved already but is falling behind the quickly changing master branch.
We would need to make a decision on that.

@micbar
Copy link
Contributor

micbar commented Apr 24, 2019

@jvillafanez Maybe we need to split this PR into smaller parts.

@jvillafanez
Copy link
Member Author

From my point of view, this should have been merged before starting working on things because it contains a refactor that touches and modifies a lot of things. At this point, after having been merged things, it's too late to do anything with this branch.
If there is anything useful there, feel free to copy whatever you need in a fresh branch before closing this

@IljaN IljaN mentioned this pull request Jul 29, 2019
9 tasks
@felixboehm felixboehm changed the title Versions import export Export / Import Versions Oct 14, 2019
@felixboehm felixboehm changed the title Export / Import Versions Export/Import Versions Oct 14, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Don't read whole file in to memory on importing/exporting
4 participants