Cleanup UX (#31)

* echo context as yaml

* rename echo as context

* rename call to pkg

* update GitIgnore

* qp depend

* refactor update_config

add view, revise

* qp catalog

* auto-linter fixes

* cleanup imports

* PYTHON_MYPY

* -U saves all URIs

* anyio=3.6.1.post111

* cleanup imports

* test_push_patch in TemporaryDirectory

* move test_cli up

* auto-linter

* flake8 fixes

* autofix

* refix stage typo

* flake8

* Create CLI.md

* Git-Friendly Workflow

* get -d

* quilt pull, -I

* streamline with quilt stat

* QuickStart

* stat cleanup

* DISABLE_LINTERS

* linter fixes

* linter auto-fix

* tweak CLI

* Update CLI.md

* Create CLI3.md

* Update CLI3.md

add link, context

.github/workflows/mega-linter.yml

@@ -11,8 +11,7 @@ env: # Comment env block if you do not want to apply fixes
   APPLY_FIXES: all # When active, APPLY_FIXES must also be defined as environment variable (in github/workflows/mega-linter.yml or other CI tool)
   #APPLY_FIXES_EVENT: pull_request # Decide which event triggers application of fixes in a commit or a PR (pull_request, push, all)
   #APPLY_FIXES_MODE: pull_request # If APPLY_FIXES is used, defines if the fixes are directly committed (commit) or posted in a PR (pull_request)
-  DISABLE_LINTERS: SPELL_CSPELL,COPYPASTE_JSCPD
-  VALIDATE_ALL_CODEBASE: true  
+  DISABLE_LINTERS: SPELL_CSPELL,COPYPASTE_JSCPD,PYTHON_BANDIT,PYTHON_MYPY,PYTHON_PYRIGHT,PYTHON_PYLINT
 
 concurrency:
   group: ${{ github.ref }}-${{ github.workflow }}
@@ -39,7 +38,8 @@ jobs:
         env:
           # All available variables are described in documentation
           # https://megalinter.github.io/configuration/
-          VALIDATE_ALL_CODEBASE: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} # Validates all source when push on main, else just the git diff with main. Override with true if you always want to lint all sources
+          VALIDATE_ALL_CODEBASE: true  
+          # VALIDATE_ALL_CODEBASE: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} # Validates all source when push on main, else just the git diff with main. Override with true if you always want to lint all sources
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           # ADD YOUR CUSTOM ENV VARIABLES HERE OR DEFINE THEM IN A FILE .mega-linter.yml AT THE ROOT OF YOUR REPOSITORY
           DISABLE: COPYPASTE,SPELL # Uncomment to disable copy-paste and spell checks

.github/workflows/python-package.yml

@@ -49,5 +49,5 @@ jobs:
           coverageFile: coverage.xml
           token: ${{ secrets.GITHUB_TOKEN }}
           thresholdAll: 0.8
-          thresholdNew: 0.8
-          thresholdModified: 0.8 
+          # thresholdNew: 0.8
+          # thresholdModified: 0.8 

CLI.md

@@ -0,0 +1,117 @@
+# Proposed Git-Friendly Workflow
+
+* create/manage local configuration file that can be checked into git
+* explicitly manage and track status of local registry
+* do it all via the command-line (without worrying about browse, et al)
+* use Quilt+ URIs as the common currency
+* optimize for the one-package one-folder one-git-repo use case ("sync")
+
+## QuickStart
+```bash
+# From a working local pipeline
+quilt init "quilt+s3://ml-demos#package=ml-demo/first-test"
+quilt add data/* --push
+git add *
+git commmit -m "Ready to run"
+
+# In a production workflow
+git pull https://github.com/ml-demo/first-test
+quilt pull
+poetry run main.py # or equivalent
+```
+
+## quilt init: new package for this repository
+```bash
+quilt init $NEW_URI
+```
+* creates .quilt/config.yaml configuration file (error if already exists)
+* creates empty package on the remote registry (error if already exists)
+* creates empty package on the local registry (warn if already exists)
+* creates .quiltignore that ignores the configuration folder (error if already exists)
+* creates .gitignore if it does not already exist
+
+## quilt get: existing package for this repository
+```bash
+quilt get $OLD_URI  [-r | -t | -d | -b]
+```
+* downloads package from remote registry to local directory (unless '-b' browse-only)
+* caches package in local registry
+* creates .quilt/config.yaml configuration file (unless -t temporary)
+* (error if already exists)
+* if '-r', recursively gets 'deps' from local configuration file
+* (warn if file/deps don't exists)
+* if -d, add as 'deps' to config.yaml
+
+## quilt pull: previous package for this repository
+```bash
+quilt pull [-n]
+```
+* downloads package from remote registry to local directory
+* (error if config.yaml does not exist)
+* caches package in local registry
+* recursively gets 'deps' from local configuration file (unless -n)
+
+## quilt add
+```bash
+quilt add [$FILES | -a | -s ] [-p]
+```
+* add files from local filesystem to local package (error if no config.yaml)
+* automatically updates .gitignore
+* if -a, instead add all files from current directory
+* if -s, sync directory: add new/modified files, remove deleted files
+* if -p, immediately push to remote
+
+## quilt push
+```bash
+quilt push [-u $NEW_URI]
+```
+* sync local package to remote
+* may need to browse first
+* (or NEW_URI if -u, else error if no config.yaml)
+
+## quilt config
+```bash
+quilt config [-d]
+```
+* print local configuration file
+* delete if -d
+* (warn if no config.yaml)
+
+
+## quilt stat
+```bash
+quilt stat [-r* | -R | -l | -L | -i -I]
+```
+* "stat" current files
+* returns list of files, with status:
+  * A: added
+  * D: deleted
+  * M: modified
+  * X: unknown/missing
+* (error if no config.yaml)
+
+### -r, --remote-diff (default)
+
+* lists only files that differ between local and remote package
+* (previews `push`)
+
+### -R, --remote-all
+
+* lists all files currently in remote package, by local status
+
+### -l, --local-diff
+
+* lists only files that differ between current directory and local package
+* (previews `add -s`)
+
+### -L, --local-all
+* lists all files currently in local package, by diff status
+
+### -i, --ignored-diff
+
+* lists only files in local package that are NOT in .gitignore
+
+### -I, --ignored-all
+* lists all files in local package, by .gitignore status:
+* I: ignored, U: un-ignored
+

CLI3.md

@@ -0,0 +1,57 @@
+# Proposed quilt3+git MVP Workflow
+
+## Objectives
+* create and use local quilt.yaml file that can be checked into git
+* do it all via the command-line (without worrying about browse, et al)
+* build on top of `quilt3` install and push
+* do NOT change existing semantics of any operation
+* address urgent customer use cases now while enabling richer functionality later
+
+Inspired by [this gist](https://gist.github.com/akarve/db4b8d5c032030df30b371127bc82e15)
+
+## QuickStart
+```bash
+# From a working local pipeline 
+git init
+quilt push package/name --registry reg -dir data --init-config # new package from data
+# OR  
+quilt install package/name --registry reg -dir data --init-config existing package to data:
+git add *
+git commmit -m "Ready to run"
+
+# In a production workflow
+git pull https://github.com/ml-demo/first-test
+quilt install --using-config
+poetry run main.py # or equivalent
+```
+
+## --init-config: create and configure quilt.yaml
+* new option on both `install` (existing packages) and `push` (new packages)
+* creates ./quilt.yaml configuration file
+* stores the fully-qualified Quilt+ URI for that package and registry
+* creates .quiltignore that ignores the configuration folder (error if already exists)
+* automatially adds package files to .gitignore (create if it does not exist)
+* optional argument -> alternate path to config file
+* error if quilt.yaml already exists: suggest "--extend-config" for appending packages
+
+
+## --using-config: get default package from quilt.yaml
+* new option on both `install` (existing packages) and `push` (existing packages)
+* error if user also specifies a package name
+* reads ./quilt.yaml configuration file (error if not exists)
+* uses first URI entry from packages as the default
+* automatially adds package files to .gitignore (error if it does not exist)
+* optional argument -> alternate path to config file
+
+## quilt.yaml: configuration file formal
+
+Propose using Quilt+ URIs as our canonical storage format for fully-qualified packages.
+Users would continue to use the existing CLI to create them.
+```yaml
+quilt_config:
+  version: 0.1.0
+  packages:
+  - quilt+s3://reg#package=package/name
+```
+
+

LICENSE.md

@@ -1,4 +1,4 @@
-MIT License
+# MIT License
 
 Copyright (c) 2022 Quilt Data
 

README.md

@@ -5,8 +5,21 @@ Resource-oriented API for Quilt's decentralized social knowledge platform
 
 ```bash
 pip install quiltplus
-qp get "quilt+s3://quilt-example#package=examples/echarts"
+qp -U "quilt+s3://quilt-example#package=examples/echarts" pkg # get
 qp list
+qp --help
+```
+
+### Detailed Command-Line
+
+```bash
+export WRITE_BUCKET=writeable_s3_bucket
+# create empty package and save to config
+qp -U "quilt+s3://$(WRITE_BUCKET)#package=test/quiltplus" pkg -x post
+time > README.md
+qp stage -a README.md
+qp stage # displays staged files
+qp pkg -x patch # uploads staged files
 ```
 
 ## Developmment
@@ -19,6 +32,7 @@ cd quiltplus
 poetry self update
 poetry install
 export WRITE_BUCKET=writeable_s3_bucket
+poetry run pytest --cov-report html && open htmlcov/index.html
 poetry run ptw --now .
 ```
 ## Pushing Changes