Key/value backend

[moretea]

There are quite some ideas in the issue tracker about how we could improve nixops radically.

I would like to start out with a feasible iterative plan. As far as I can see, only very trivial SQL operations are every executed against the sqlite3 database; the main use case is to use transactions to support rollback in the case of exceptions. This should not be to hard to implement ourselves in memory, in combination with a key/value server that supports locking certain keys.

• Refactor the statefile and deployment lock into a specific local implementation
• Introduce different URI schema's for future backends
• Refactor all the sqlite3 queries into concrete operations implemented in file.Statefile
  • Deployment
  • Resource
  • VirtualBox
  • GCE done, needs to be verified
  • AWS done, needs to be verified
  • Hetzner done, needs to be verified
• Implement a local json file backend

[bjornfor]

Nitpick: wrong indentation

[bjornfor]

I'm not a nixops developer, but IMHO this looks nice.

[moretea]

Yes, just found that out myself when trying to run the tests one final time; the issue was that I invoked the tests in another way that did not trigger this :(. The commit that I pushed after your comment should fix that, I'm running the vbox tests again at the moment.

[grahamc]

Ok I pulled again and got further, however:

$ nixops deploy --include builder-19 
Traceback (most recent call last):
  File "/nix/store/59qbcipcy3hrykqbxdbiwdzpj8vpxrq8-nixops-1.5.1pre0_abcdef/bin/..nixops-wrapped-wrapped", line 9, in <module>
    import nixops.state.file
ImportError: No module named state.file

[moretea]

@grahamc that should fix it!

[grahamc]

Not sure:

$ nixops deploy --include builder-19 
Traceback (most recent call last):
  File "/nix/store/xdl2ql51sclf1yr5xi3cab2qgx03mi0s-nixops-1.5.1pre0_abcdef/bin/..nixops-wrapped-wrapped", line 9, in <module>
    import nixops.state.sqlite3_file
ImportError: No module named state.sqlite3_file

python's import system is a gift that keeps on giving :D

[ip1981]

Wish: don't store in state file parameters that can be extracted from nix expressions or not used anyway: e. g. EC2 instance type, keys, toplevel, storeKeysOnMachine, configsPath, etc.

[nh2]

What's the motiviation behind this motion? Is it to be able to support other backends than sqlite for the purpose of having the state stored e.g. in a networked database?

[moretea]

Exactly.

[rbvermaa]

Thanks, committed in 308ac51

[domenkozar]

@rbvermaa wrong issue?

[bachp]

I would like to test this for AWS, what's the easiest way to get this version in my setup?

[bachp]

@moretea @grahamc I got as far that I now also getting the error

Traceback (most recent call last):
  File "/nix/store/czy2gz21dp72i6lfm1k2gx98ss8kl51x-nixops-1.6pre0_abcdef/bin/..nixops-wrapped-wrapped", line 9, in <module>
    import nixops.state.sqlite3_file
ImportError: No module named state.sqlite3_file

Have you been able to solve this?

[mogorman]

fixed merge conflicts in my clone of the fork and will continue modifying the source for this patch there.

[andrewchambers]

Some explanation of the reason/point of this change would be nice. The original post provides no rationale.

[moretea]

The point is that you'll be able to store state remotely eventually, instead of a local sqlite3 file.

Being able to have a key/value that stores to JSON will be a first step to be able to store settings in a remote implementation of that

[bachp]

@moretea are you still working on this one?

[comozo]

Hi,
@moretea this feature would be great for us. Is that possible to merge it ?
Ta.

[domenkozar]

See #1264

[grahamc]

Hello!

Thank you for this PR.

In the past several months, some major changes have taken place in
NixOps:

1. Backends have been removed, preferring a plugin-based architecture.
   Here are some of them:

   • https://github.com/NixOS/nixops-aws
   • https://github.com/NixOS/nixops-hetzner
   • https://github.com/nix-community/nixops-vbox
   • https://github.com/nix-community/nixops-libvirtd
   • https://github.com/nix-community/nixops-datadog

2. NixOps Core has been updated to be Python 3 only, and at the
   same time, MyPy type hints have been added and are now strictly
   required during CI.

This is all accumulating in to what I hope will be a NixOps 2.0
release. There is a tracking issue for that:
#1242 . It is possible that
more core changes will be made to NixOps for this release, with a
focus on simplifying NixOps core and making it easier to use and work
on.

My hope is that by adding types and more thorough automated testing,
it will be easier for contributors to make improvements, and for
contributions like this one to merge in the future.

However, because of the major changes, it has become likely that this
PR cannot merge right now as it is. The backlog of now-unmergable PRs
makes it hard to see which ones are being kept up to date.

If you would like to see this merge, please bring it up to date with
master and reopen it. If the or mypy type checking fails, please
correct any issues and then reopen it. I will be looking primarily at
open PRs whose tests are all green.

Thank you again for the work you've done here, I am sorry to be
closing it now.

Graham