eval-machine-info.nix: ported to modules

[pasqui23]

Fixes #1486
Done:

• Made requires an error.
• Implemented the nodes option.
• Made a fallback for the old way of writing nodes to the
• Added network.nodesExtraArgs
  To do:
• Implemented all resources options
• Implemented all network options
• Tested it
• Wire up the NixOS assertions and warnings
• Declare lock and storage with plugin (assigned @roberth)

[pasqui23]

@roberth you mentioned that the module system supports requires,how?

[roberth]

require has the same meaning as imports: https://github.com/NixOS/nixpkgs/blob/d81f048ef4ce360ba2d11d3d7c94398be134b380/lib/modules.nix#L368

[roberth]

We should use this exclusively for getting the nixpkgs option value, and then use its lib for the rest of the evaluation.

Suggested change

	nixpkgs1 = <nixpkgs>; # this will be replaced on install by nixops' nixpkgs input
	lib = import "${nixpkgs1}/lib";
	libBootNixpkgs = <nixpkgs>; # this will be replaced on install by nixops' nixpkgs input
	libBootLib = import "${nixpkgs1}/lib";

[roberth]

I think we'll want a type that doesn't prohibit merging, but returns a list or a mkMerge instead.

This option is actually somewhat redundant, as you can type-merge modules into all nodes by redeclaring options.nodes, but defaults is expected and convenient.

(edited to say type-merge)

[roberth]

This extendModules comes from libBoot, so this should use lib.evalModules from scratch instead.

(Sorry if I'm repetitive; made some commit comments by accident earlier)

[pasqui23]

Ok I've tried and failed to build a network, it goes in infinite recursion while evaluating scrubOptionValues in info.machines

[pasqui23]

It still goes in infinite recursion, I need to know what nixops expects exactly from this file and what can be safely removed.

[roberth]

it goes in infinite recursion

Maybe the attrsOfs need to be lazyAttrsOfs.
attrsOf can only return its attrset by inspecting the children for mkIf false values and filtering out those attributes. That makes it more strict than lazyAttrsOf, which will return the spine of the attribute set (attr names + thunks) without inspecting those thunks.

[roberth]

I need to know what nixops expects exactly

I don't expect this to be documented anywhere.

[pasqui23]

I need to know what nixops expects exactly

I don't expect this to be documented anywhere.

I don't need to be fully documented,only to tell me what the nixops cli and plugins take from the eval-machine-info.nix file right now. If, after merging this, you go and change that I would not care abut it.

[pasqui23]

it goes in infinite recursion

Maybe the attrsOfs need to be lazyAttrsOfs. attrsOf can only return its attrset by inspecting the children for mkIf false values and filtering out those attributes. That makes it more strict than lazyAttrsOf, which will return the spine of the attribute set (attr names + thunks) without inspecting those thunks.

Tried with lazyAttrsOf but it still gives me the same problems.

[pasqui23]

Ok it was an error on my part, I forgot network.nodesExtraArgs = { inherit inputs; }; in my config. 🤦🤦🤦🤦

[pasqui23]

Ok now the error I get is

; nixops list
error: The option `resources' is used but not defined.

Which I don't know where even to begin

[roberth]

; nixops list
error: The option `resources' is used but not defined.

Just add a default = { }; to the resources option.

It's some silly ux :/

[pasqui23]

Ok latest error is definetly on the python side

; nixops list                       
def-matcher: command not found
Traceback (most recent call last):
  File "/nix/store/nbrhfv3wcdsiish3y3k64jgs9826k2xc-python3.8-nixops-2.0.0/bin/.nixops-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/nbrhfv3wcdsiish3y3k64jgs9826k2xc-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/__main__.py", line 56, in main
    args.op(args)
  File "/nix/store/nbrhfv3wcdsiish3y3k64jgs9826k2xc-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/script_defs.py", line 200, in op_list_deployments
    depl.evaluate()
  File "/nix/store/nbrhfv3wcdsiish3y3k64jgs9826k2xc-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/deployment.py", line 431, in evaluate
    self.evaluate_network()
  File "/nix/store/nbrhfv3wcdsiish3y3k64jgs9826k2xc-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/deployment.py", line 423, in evaluate_network
    self.description = config.get("description", self.default_description)
  File "/nix/store/nbrhfv3wcdsiish3y3k64jgs9826k2xc-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/util.py", line 493, in set
    self._set_attr(name, x)
  File "/nix/store/nbrhfv3wcdsiish3y3k64jgs9826k2xc-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/deployment.py", line 250, in _set_attr
    self._set_attrs({name: value})
  File "/nix/store/nbrhfv3wcdsiish3y3k64jgs9826k2xc-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/deployment.py", line 243, in _set_attrs
    c.execute(
sqlite3.OperationalError: attempt to write a readonly database

[roberth]

sqlite3.OperationalError: attempt to write a readonly database

This is a known issue. #1490

At this time, nixops deploy is more suitable for testing, as this command does request write access to the db.

[pasqui23]

; nixops deploy                  
def-matcher: command not found
flatwoody> generating new SSH key pair... done
Password: 
flatwoody> setting state version to 22.05
flatwoody> waiting for SSH...
building all machine configurations...
trace: warning: Please use the actual nodes.* option instead of assigning machines to the config's top level
error: 'flatwoody' at /tmp/nixops-tmpnvee6p43/physical.nix:2:15 called without required argument 'pkgs'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:199:25:

          198|                 file = def.file;
          199|                 value = def.value arg;
             |                         ^
          200|               }) defs);

       … while evaluating the attribute 'value'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:199:17:

          198|                 file = def.file;
          199|                 value = def.value arg;
             |                 ^
          200|               }) defs);

       … while evaluating 'getType'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:171:21:

          170|         let
          171|           getType = value:
             |                     ^
          172|             if isAttrs value && isCoercibleToString value

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:179:16:

          178|           commonType = foldl' (type: def:
          179|             if getType def.value == type
             |                ^
          180|             then type

       … while evaluating anonymous lambda

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:178:38:

          177|           # don't have the same type
          178|           commonType = foldl' (type: def:
             |                                      ^
          179|             if getType def.value == type

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:178:24:

          177|           # don't have the same type
          178|           commonType = foldl' (type: def:
             |                        ^
          179|             if getType def.value == type

       … while evaluating 'merge'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:169:20:

          168|       check = value: true;
          169|       merge = loc: defs:
             |                    ^
          170|         let

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:195:38:

          194|             stringCoercibleSet = mergeOneOption;
          195|             lambda = loc: defs: arg: anything.merge
             |                                      ^
          196|               (loc ++ [ "<function body>" ])

       … while evaluating 'lambda'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:195:33:

          194|             stringCoercibleSet = mergeOneOption;
          195|             lambda = loc: defs: arg: anything.merge
             |                                 ^
          196|               (loc ++ [ "<function body>" ])

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:530:46:

          529|         coerce = unify: value: if isFunction value
          530|           then setFunctionArgs (args: unify (value args)) (functionArgs value)
             |                                              ^
          531|           else unify (if shorthandOnlyDefinesConfig then { config = value; } else value);

       … while evaluating 'unifyModuleSyntax'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:343:34:

          342|      of ‘options’, ‘config’ and ‘imports’ attributes. */
          343|   unifyModuleSyntax = file: key: m:
             |                                  ^
          344|     let

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:530:39:

          529|         coerce = unify: value: if isFunction value
          530|           then setFunctionArgs (args: unify (value args)) (functionArgs value)
             |                                       ^
          531|           else unify (if shorthandOnlyDefinesConfig then { config = value; } else value);

       … while evaluating anonymous lambda

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:530:33:

          529|         coerce = unify: value: if isFunction value
          530|           then setFunctionArgs (args: unify (value args)) (functionArgs value)
             |                                 ^
          531|           else unify (if shorthandOnlyDefinesConfig then { config = value; } else value);

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:397:8:

          396|       # works.
          397|     in f (args // extraArgs)
             |        ^
          398|   else

       … while evaluating 'applyIfFunction'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:373:29:

          372|
          373|   applyIfFunction = key: f: args@{ config, options, lib, ... }: if isFunction f then
             |                             ^
          374|     let

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:272:55:

          271|         if isFunction m || isAttrs m then
          272|           unifyModuleSyntax fallbackFile fallbackKey (applyIfFunction fallbackKey m args)
             |                                                       ^
          273|         else if isList m then

       … while evaluating 'unifyModuleSyntax'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:343:34:

          342|      of ‘options’, ‘config’ and ‘imports’ attributes. */
          343|   unifyModuleSyntax = file: key: m:
             |                                  ^
          344|     let

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:272:11:

          271|         if isFunction m || isAttrs m then
          272|           unifyModuleSyntax fallbackFile fallbackKey (applyIfFunction fallbackKey m args)
             |           ^
          273|         else if isList m then

       … while evaluating 'loadModule'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:270:53:

          269|       # Like unifyModuleSyntax, but also imports paths and calls functions if necessary
          270|       loadModule = args: fallbackFile: fallbackKey: m:
             |                                                     ^
          271|         if isFunction m || isAttrs m then

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:311:22:

          310|           let
          311|             module = loadModule args parentFile "${parentKey}:anon-${toString n}" x;
             |                      ^
          312|             collectedImports = collectStructuredModules module._file module.key module.imports args;

       … while evaluating anonymous lambda

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:328:31:

          327|           disabledKeys = map moduleKey disabled;
          328|           keyFilter = filter (attrs: ! elem attrs.key disabledKeys);
             |                               ^
          329|         in map (attrs: attrs.module) (builtins.genericClosure {

       … from call site

       … while evaluating 'filterModules'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:324:36:

          323|       # modules recursively. It returns the final list of unique-by-key modules
          324|       filterModules = modulesPath: { disabled, modules }:
             |                                    ^
          325|         let

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:335:7:

          334|     in modulesPath: initialModules: args:
          335|       filterModules modulesPath (collectStructuredModules unknownModule "" initialModules args);
             |       ^
          336|

       … while evaluating anonymous lambda

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:334:37:

          333|
          334|     in modulesPath: initialModules: args:
             |                                     ^
          335|       filterModules modulesPath (collectStructuredModules unknownModule "" initialModules args);

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:180:25:

          179|       merged =
          180|         let collected = collectModules
             |                         ^
          181|           (specialArgs.modulesPath or "")

       … while evaluating 'reverseList'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/lists.nix:393:17:

          392|   */
          393|   reverseList = xs:
             |                 ^
          394|     let l = length xs; in genList (n: elemAt xs (l - n - 1)) l;

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:184:33:

          183|           ({ inherit lib options config specialArgs; } // specialArgs);
          184|         in mergeModules prefix (reverseList collected);
             |                                 ^
          185|

       … while evaluating 'byName'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:449:25:

          448|       */
          449|       byName = attr: f: modules:
             |                         ^
          450|         zipAttrsWith (n: concatLists)

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:466:21:

          465|       # an attrset 'name' => list of submodules that declare ‘name’.
          466|       declsByName = byName "options" (module: option:
             |                     ^
          467|           [{ inherit (module) _file; options = option; }]

       … while evaluating the attribute 'matchedOptions'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:510:14:

          509|     in {
          510|       inherit matchedOptions;
             |              ^
          511|

       … while evaluating 'mapAttrsRecursiveCond'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/attrsets.nix:296:36:

          295|   */
          296|   mapAttrsRecursiveCond = cond: f: set:
             |                                    ^
          297|     let

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:192:28:

          191|           # For definitions that have an associated option
          192|           declaredConfig = mapAttrsRecursiveCond (v: ! isOption v) (_: v: v.value) options;
             |                            ^
          193|

       … while evaluating the attribute 'config'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:257:9:

          256|         options = checked options;
          257|         config = checked (removeAttrs config [ "_module" ]);
             |         ^
          258|         _module = checked (config._module);

       … while evaluating 'merge'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:569:22:

          568|         check = x: isAttrs x || isFunction x || path.check x;
          569|         merge = loc: defs:
             |                      ^
          570|           (base.extendModules {

       … from call site

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:649:59:

          648|       if isDefined then
          649|         if all (def: type.check def.value) defsFinal then type.merge loc defsFinal
             |                                                           ^
          650|         else let allInvalid = filter (def: ! type.check def.value) defsFinal;

       … while evaluating the attribute 'value'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/modules.nix:660:27:

          659|     optionalValue =
          660|       if isDefined then { value = mergedValue; }
             |                           ^
          661|       else {};

       … while evaluating anonymous lambda

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/lib/types.nix:413:22:

          412|       merge = loc: defs:
          413|         mapAttrs (n: v: v.value) (filterAttrs (n: v: v ? value) (zipAttrsWith (name: defs:
             |                      ^
          414|             (mergeDefinitions (loc ++ [name]) elemType defs).optionalValue

       … from call site

       … while evaluating the attribute 'config.system.build.toplevel'

       at /nix/store/kxlj5h6jss1szi1rs0lkccggr18g0lza-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nix/eval-machine-info.nix:161:32:

          160|   #TODO: take options and auter modules outputs for each node
          161|   nodes = lib.mapAttrs (n: v: {config = v;}) net.config.nodes;
             |                                ^
          162|

       … while evaluating anonymous lambda

       at /nix/store/kxlj5h6jss1szi1rs0lkccggr18g0lza-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nix/eval-machine-info.nix:309:54:

          308|         mkdir -p $out
          309|         ${toString (lib.attrValues (lib.mapAttrs (n: v: ''
             |                                                      ^
          310|           ln -s ${v.config.system.build.toplevel} $out/${n}

       … from call site

       … while evaluating the attribute 'buildCommand' of the derivation 'nixops-machines'

       at /nix/store/531v6sf2db04x3xqphfkh7zg2dgw135k-source/pkgs/stdenv/generic/make-derivation.nix:205:7:

          204|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          205|       name =
             |       ^
          206|         let
error: evaluation of the deployment specification failed

[pasqui23]

Where it would ask for a pkgs param?

[roberth]

The error seems to originate from the "physical" expression that's generated by nixops. You can see that expression with nixops show-physical.

[pasqui23]

; nixops show-physical
def-matcher: command not found
Traceback (most recent call last):
  File "/nix/store/3y06h6hsw56r0bszn9pl7cdpz91qjbh1-python3.8-nixops-2.0.0/bin/.nixops-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/3y06h6hsw56r0bszn9pl7cdpz91qjbh1-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/__main__.py", line 56, in main
    args.op(args)
  File "/nix/store/3y06h6hsw56r0bszn9pl7cdpz91qjbh1-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/script_defs.py", line 813, in op_show_physical
    depl.evaluate()
  File "/nix/store/3y06h6hsw56r0bszn9pl7cdpz91qjbh1-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/deployment.py", line 431, in evaluate
    self.evaluate_network()
  File "/nix/store/3y06h6hsw56r0bszn9pl7cdpz91qjbh1-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/deployment.py", line 423, in evaluate_network
    self.description = config.get("description", self.default_description)
  File "/nix/store/3y06h6hsw56r0bszn9pl7cdpz91qjbh1-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/util.py", line 493, in set
    self._set_attr(name, x)
  File "/nix/store/3y06h6hsw56r0bszn9pl7cdpz91qjbh1-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/deployment.py", line 250, in _set_attr
    self._set_attrs({name: value})
  File "/nix/store/3y06h6hsw56r0bszn9pl7cdpz91qjbh1-python3.8-nixops-2.0.0/lib/python3.8/site-packages/nixops/deployment.py", line 243, in _set_attrs
    c.execute(
sqlite3.OperationalError: attempt to write a readonly database

[pasqui23]

#1490 strikes again☹

[roberth]

#1490 strikes againfrowning_face

#1511 should work around it.

[pasqui23]

Ok why is TestSSHKeyPairResource failing?

[roberth]

Ok why is TestSSHKeyPairResource failing?

I've made some fixes so that a nixops_aws deployment with resources now works.

[roberth]

net.nix: removed nodes options, all network nodes are now defined in resources.machines

This seems a bit too radical to me. nodes used to be consistent with the NixOS tests, and I believe the distinction between nodes and machine resources may have been greater in the past, and completely merging the two may be a very grave mistake. There's no easy way out of such a decision.

Besides the consistency and the risk, I feel like nodes is actually an ingredient of a good solution, which is the idea that resources are somewhat of a low level interface to the cloud, whereas almost all the other options can be more high level. NixOS itself is actually one of these more high level options. For instance, it can select between multiple backends.
When we support a generic resource backend like terraform, cloud support would be largely be implemented in the module system, and these NixOS configurations won't be represented by a single resource that was specifically written to be this all-encompassing NixOS + VM + SSH + whatnot python object.

So I'm a bit hesitant, but I can be convinced, maybe?

[pasqui23]

On 20/03/23 08:01, Robert Hensing ***@***.***> wrote: `nodes` used to be consistent with the NixOS tests

`nodes` is the only options nixops and the nixos test have in commons, I do not even belive that the nixos tests are implemented with the modules framework so it would only cause further confusions if we were to make the 2 more similar.

and I believe the distinction between nodes and machine resources may have been greater in the past, and completely merging the two may be a very grave mistake.

On the contrary i joined the 2 for a reason of Single Source of Truth: the users are asked to set the nodes options yet the nixops script goes to look into a completely different option. It's not hard to imagine a bug where the 2 begin to diverge. Worse such bug would be hard to even detect,let alone debug, as the user begin to look in the complete wrong position and the nix language support is extremely poor.

There's no easy way out of such a decision.

`mkRenamedModule` and `mkRemovedModule` covers a multitude of sins

Besides the consistency and the risk, I feel like `nodes` is actually an ingredient of a good solution, which is the idea that `resources` are somewhat of a low level interface to the cloud, whereas almost all the other options can be more high level.

I would go the opposite direction, with `machines` being merely one kind among many of cloud resources, and even being able to configure a network that has no machines in it. It could even intrude into disnix's role and allow to configure,for example, a network of Java servlets on something like a Servlet as a Service or something.

NixOS itself is actually one of these more high level options. For instance, it can select between multiple backends. When we support a generic resource backend like terraform, cloud support would be largely be implemented in the module system,

Even now the we can access `resources.machines.*.system.build.toplevel` from inside the net module. It is still possible to implement the cloud supposrt from the module itself.

and these NixOS configurations won't be represented by a single resource that was specifically written to be this all-encompassing NixOS + VM + SSH + whatnot python object.

So we can still do all kind of object shuffling with the right `networkExpr` and they can still have all kind of backend-dependent magic that can be referred from the nodes themselves. Hell we could even steal from `deploy.rs` and have `resource.upgradeScripts` all from the module system.

So I'm a bit hesitant, but I can be convinced, maybe?

Remember that I have little experience with colud systems, let alone nixops specifically, so if any of the above sound wrong don't hesitate to call me out on it ☺

[roberth]

I'll share a response with my thoughts so far. Haven't been able to look into it more deeply yet.

I do not even belive that the nixos tests are implemented with the modules framework

They are.

On the contrary i joined the 2 for a reason of Single Source of Truth: the users are asked to set the nodes options yet the nixops script goes to look into a completely different option. It's not hard to imagine a bug where the 2 begin to diverge.

If we have option definitions that assign the right thing, I don't see how divergence could reasonably occur, except perhaps through mkForce if at all possible, which users would avoid or do for a good reason.

I would prefer to keep the delta small until we can say something definitive about this nodes vs resources.machines choice.

What worries me is a problem that we have in NixOps evaluation, where it could take an arbitrary number of nix-instantiate invocations to properly wire each value from one resource's outputs to another resource's inputs, which produces an output that is then wired to yet another resource's inputs, and so forth. This is a serious issue that requires some rearchitecting to get right. I did not plan to solve this for a NixOps 2 release, as it depends on having proper Nix python bindings (in progress but will take a while to be sufficiently usable). So iiuc for now we're stuck with NixOps 1's phased evaluated that invokes the evaluator twice, and no more than that. It has some special logic for wiring up values between resources by name, so without going through the evaluator. NixOS basically does not fit within this resources paradigm.
For a more thorough and confident explanation I'd have to do more research, but realistically we're should not solve this specific problem for the NixOps 2 release, because we have more pressing issues and very limited resources.
So anyway, in the future we'll want to essentially replace the resource modules by a custom primop invocation backed by e.g. a terraform resource, and not much else. Having a whole NixOS configuration in there as a resource would not make much sense. It'd be like declaring your database configuration in systemd.services.

Anyway, I'll do a bit more research and some testing before I actually make a decision.

[pasqui23]

What worries me is a problem that we have in NixOps evaluation, where it could take an arbitrary number of nix-instantiate invocations to properly wire each value from one resource's outputs to another resource's inputs, which produces an output that is then wired to yet another resource's inputs, and so forth. This is a serious issue that requires some rearchitecting to get right.

Doesn't the module system take care of that? Just evaluate once and all the stuff needed is in net.

Or maybe there is some black magic in the python code that I am missing.

I did not plan to solve this for a NixOps 2 release, as it depends on having proper Nix python bindings (in progress but will take a while to be sufficiently usable). So iiuc for now we're stuck with NixOps 1's phased evaluated that invokes the evaluator twice, and no more than that. It has some special logic for wiring up values between resources by name, so without going through the evaluator. NixOS basically does not fit within this resources paradigm.

So what are resources specifically? I tried to implement that resources.script idea from above but apparently from the errors that nixops gave me they require some glue from the Python side, what does that python glue do specifically?

For a more thorough and confident explanation I'd have to do more research, but realistically we're should not solve this specific problem for the NixOps 2 release, because we have more pressing issues and very limited resources. So anyway, in the future we'll want to essentially replace the resource modules by a custom primop invocation backed by e.g. a terraform resource, and not much else.

Would this allow nixops to interface for eg with Kubernetes in a remote future? Making nixops a mere terraform front-end seem like a kind of waste even as nice as both nixops and terraform are.