diff --git a/README.md b/README.md index d086742..ee3c9cb 100644 --- a/README.md +++ b/README.md @@ -10,13 +10,29 @@ --- -**NOTE**: *Active development on [spiff](https://github.com/cloudfoundry-incubator/spiff) is currently paused and does not accept feature pull requests anymore. `spiff++` is a fork of spiff that provides a compatible extension to spiff based on the latest version offering a rich set of new features not yet available in spiff. All fixes provided by the original spiff project will be incorporated into spiff++, also. If development is opened again the new features will be proposed for the original project* +**NOTE**: *Active development on spiff is currently paused, including Pull Requests. Very severe issues will be addressed, and we will still be actively responding to requests for help via Issues. `spiff++` is a fork of spiff that provides a compatible extension to spiff based on the latest version offering a rich set of new features not yet available in spiff. All fixes provided by the original spiff project will be incorporated into spiff++, also. This is the last fork version of the original spiff tool. Because there will be no way back to the spiff source base, this version will be the basis for the first version for a new independent spiff++ repository.* --- -spiff is a command line tool and declarative YAML templating system, specially designed for generating BOSH deployment manifests. +*spiff* is a command line tool and declarative in-domain hybrid YAML templating system. While regular templating systems process a template file by substituting the template expressions by values taken from +external data sources, in-domain means that the templating engine knows about the syntax and structure of the processed template. It therefore can take the values for the template expressions directly +from the document processed, including those parts denoted by the template expressions itself. + +For example: +```yaml +resource: + name: bosh deployment + version: 25 + url: (( "http://resource.location/bosh?version=" version )) + description: (( "This document describes a " name " located at " url )) +``` + +Hybrid mean that the template processing is not restricted to the template itself. Additionally +*spiff* is able to merge the template with information from additional yaml files, so-called stubs, that again may contain template expressions. + Contents: + - [Installation](#installation) - [Usage](#usage) - [dynaml Templating Language](#dynaml-templating-language) @@ -130,18 +146,25 @@ It is possible to read one file from standard input by using the file name `-`. Show structural differences between two deployment manifests. -Unlike 'bosh diff', this command has semantic knowledge of a deployment -manifest, and is not just text-based. It also doesn't modify either file. +Unlike basic diffing tools and even `bosh diff`, this command has semantic +knowledge of a deployment manifest, and is not just text-based. For example, +if two manifests are the same except they have some jobs listed in different +orders, `spiff diff` will detect this, since job order matters in a manifest. +On the other hand, if two manifests differ only in the order of their +resource pools, for instance, then it will yield and empty diff since +resource pool order doesn't actually matter for a deployment. -It's tailed for checking differences between one deployment and the next. +Also unlike `bosh diff`, this command doesn't modify either file. + +It's tailored for checking differences between one deployment and the next. Typical flow: ```sh -$ spiff merge template.yml [templates...] > deployment.yml +$ spiff merge template.yml [templates...] > upgrade.yml $ bosh download manifest [deployment] current.yml -$ spiff diff deployment.yml current.yml -$ bosh deployment deployment.yml +$ spiff diff upgrade.yml current.yml +$ bosh deployment upgrade.yml $ bosh deploy ``` @@ -862,7 +885,8 @@ The result is the string `3 times 2 yields 6`. ## `(( "10.10.10.10" - 11 ))` -Besides arithmetic on integers it is also possible to use addition and subtraction on ip addresses. +Besides arithmetic on integers it is also possible to use addition and +subtraction on ip addresses, or multiplication and division on CIDRs. e.g.: @@ -878,6 +902,39 @@ ip: 10.10.10.10 range: 10.10.10.10-10.11.11.1 ``` +Subtraction also works on two IP addresses to calculate the number of +IP addresses between two IP addresses. + +e.g.: + +```yaml +diff: (( 10.0.1.0 - 10.0.0.1 + 1 )) +``` + +yields the value 256. IP address constants can be directly used in dynaml +expressions. They are implicitly converted to strings and back to IP +addresses if required by an operation. + +Multiplication and division can be used to handle IP range shifts on CIDRs. +With division a network can be partioned. The network size is increased +to allow at least a dedicated number of subnets below the original CIDR. +Multiplication then can be used to get the n-th next subnet of the same +size. + +e.g.: + +```yaml +subnet: (( "10.1.2.1/24" / 12 )) # first subnet CIDR for 16 subnets +next: (( "10.1.2.1/24" / 12 * 2)) # 2nd next (3rd) subnet CIDRS +``` + +yields + +```yaml +subnet: 10.1.2.0/28 +next: 10.1.2.32/28 +``` + Additionally there are functions working on IPv4 CIDRs: ```yaml @@ -896,19 +953,6 @@ next: 192.168.1.0 num: 192.168.0.0+256=192.168.1.0 ``` -Subtraction also works on two IP addresses to calculate the number of -IP addresses between two IP addresses. - -e.g.: - -```yaml -diff: (( 10.0.1.0 - 10.0.0.1 + 1 )) -``` - -yields the value 256. IP address constants can be directly used in dynaml -expressions. They are implicitly converted to strings and back to IP -addresses if required by an operation. - ## `(( a > 1 ? foo :bar ))` Dynaml supports the comparison operators `<`, `<=`, `==`, `!=`, `>=` and `>`. The comparison operators work on diff --git a/dynaml/division.go b/dynaml/division.go index b6ff400..6190a04 100644 --- a/dynaml/division.go +++ b/dynaml/division.go @@ -2,8 +2,14 @@ package dynaml import ( "fmt" + "log" + "net" ) +func deb(fmt string, args ...interface{}) { + log.Printf(fmt, args...) +} + type DivisionExpr struct { A Expression B Expression @@ -12,7 +18,7 @@ type DivisionExpr struct { func (e DivisionExpr) Evaluate(binding Binding, locally bool) (interface{}, EvaluationInfo, bool) { resolved := true - aint, info, ok := ResolveIntegerExpressionOrPushEvaluation(&e.A, &resolved, nil, binding, false) + a, info, ok := ResolveExpressionOrPushEvaluation(&e.A, &resolved, nil, binding, false) if !ok { return nil, info, false } @@ -29,7 +35,37 @@ func (e DivisionExpr) Evaluate(binding Binding, locally bool) (interface{}, Eval if bint == 0 { return info.Error("division by zero") } - return aint / bint, info, true + + aint, ok := a.(int64) + if ok { + return aint / bint, info, true + } + + str, ok := a.(string) + if ok { + ip, cidr, err := net.ParseCIDR(str) + if err != nil { + return info.Error("CIDR or int argument required as first argument for division: %s", err) + } + ones, bits := cidr.Mask.Size() + ip = ip.Mask(cidr.Mask) + round := false + for bint > 1 { + if bint%2 == 1 { + round = true + } + bint = bint / 2 + ones++ + } + if round { + ones++ + } + if ones > 32 { + return info.Error("divisor too large for CIDR network size") + } + return (&net.IPNet{ip, net.CIDRMask(ones, bits)}).String(), info, true + } + return info.Error("CIDR or int argument required as first argument for division") } func (e DivisionExpr) String() string { diff --git a/dynaml/division_test.go b/dynaml/division_test.go index bb760dd..61e58fb 100644 --- a/dynaml/division_test.go +++ b/dynaml/division_test.go @@ -47,4 +47,31 @@ var _ = Describe("division", func() { Expect(expr).To(FailToEvaluate(FakeBinding{})) }) }) + + Context("when the left-hand side is a CIDR", func() { + It("divides an IP range", func() { + expr := DivisionExpr{ + StringExpr{"10.1.2.1/24"}, + IntegerExpr{4}, + } + + Expect(expr).To(EvaluateAs("10.1.2.0/26", FakeBinding{})) + }) + It("rounds up divisor", func() { + expr := DivisionExpr{ + StringExpr{"10.1.2.1/24"}, + IntegerExpr{12}, + } + + Expect(expr).To(EvaluateAs("10.1.2.0/28", FakeBinding{})) + }) + It("fails for too large divisor", func() { + expr := DivisionExpr{ + StringExpr{"10.1.2.1/24"}, + IntegerExpr{257}, + } + + Expect(expr).To(FailToEvaluate(FakeBinding{})) + }) + }) }) diff --git a/dynaml/multiplication.go b/dynaml/multiplication.go index fc41d24..8cc4bb6 100644 --- a/dynaml/multiplication.go +++ b/dynaml/multiplication.go @@ -2,6 +2,7 @@ package dynaml import ( "fmt" + "net" ) type MultiplicationExpr struct { @@ -12,7 +13,7 @@ type MultiplicationExpr struct { func (e MultiplicationExpr) Evaluate(binding Binding, locally bool) (interface{}, EvaluationInfo, bool) { resolved := true - aint, info, ok := ResolveIntegerExpressionOrPushEvaluation(&e.A, &resolved, nil, binding, false) + a, info, ok := ResolveExpressionOrPushEvaluation(&e.A, &resolved, nil, binding, false) if !ok { return nil, info, false } @@ -25,7 +26,24 @@ func (e MultiplicationExpr) Evaluate(binding Binding, locally bool) (interface{} if !resolved { return e, info, true } - return aint * bint, info, true + + aint, ok := a.(int64) + if ok { + return aint * bint, info, true + } + + str, ok := a.(string) + if ok { + ip, cidr, err := net.ParseCIDR(str) + if err != nil { + return info.Error("CIDR or int argument required for multiplication: %s", err) + } + ones, _ := cidr.Mask.Size() + size := int64(1 << (32 - uint32(ones))) + ip = IPAdd(ip.Mask(cidr.Mask), size*bint) + return (&net.IPNet{ip, cidr.Mask}).String(), info, true + } + return info.Error("CIDR or int argument required as first argument for multiplication") } func (e MultiplicationExpr) String() string { diff --git a/dynaml/multiplication_test.go b/dynaml/multiplication_test.go index 60cde1e..3deb317 100644 --- a/dynaml/multiplication_test.go +++ b/dynaml/multiplication_test.go @@ -36,4 +36,15 @@ var _ = Describe("multiplication", func() { Expect(expr).To(FailToEvaluate(FakeBinding{})) }) }) + + Context("when the left-hand side is a CIDR", func() { + It("shifts the IP range", func() { + expr := MultiplicationExpr{ + StringExpr{"10.1.2.1/24"}, + IntegerExpr{3}, + } + + Expect(expr).To(EvaluateAs("10.1.5.0/24", FakeBinding{})) + }) + }) }) diff --git a/flow/flow_test.go b/flow/flow_test.go index c023729..835aebf 100644 --- a/flow/flow_test.go +++ b/flow/flow_test.go @@ -275,6 +275,22 @@ foo: (( auto )) }) }) + Context("when there are ignorable dynaml nodes start with '!'", func() { + It("ignores nodes", func() { + source := parseYAML(` +--- +foo: ((!template_only.foo)) +`) + + resolved := parseYAML(` +--- +foo: ((!template_only.foo)) +`) + + Expect(source).To(FlowAs(resolved)) + }) + }) + Context("when a reference is made to a yet-to-be-resolved node, in a || expression", func() { It("eventually resolves to the referenced node", func() { source := parseYAML(` @@ -5598,4 +5614,22 @@ result: Expect(source).To(FlowAs(resolved)) }) }) + + Describe("when shifting network ranges", func() { + Context("with arithmetic operator", func() { + It("splits and shifts", func() { + source := parseYAML(` +--- +subnet: (( "10.1.2.1/24" / 12 )) +next: (( "10.1.2.1/24" / 12 * 2 )) +`) + resolved := parseYAML(` +--- +subnet: 10.1.2.0/28 +next: 10.1.2.32/28 +`) + Expect(source).To(FlowAs(resolved)) + }) + }) + }) }) diff --git a/spiff.go b/spiff.go index bff59c8..c3d2ddd 100644 --- a/spiff.go +++ b/spiff.go @@ -22,7 +22,7 @@ func main() { app := cli.NewApp() app.Name = "spiff" app.Usage = "BOSH deployment manifest toolkit" - app.Version = "1.0.8-ms.7" + app.Version = "1.0.8-ms.8" app.Commands = []cli.Command{ { diff --git a/yaml/node.go b/yaml/node.go index ae5b2e8..50ab00e 100644 --- a/yaml/node.go +++ b/yaml/node.go @@ -336,7 +336,7 @@ func (n AnnotatedNode) EquivalentToNode(o Node) bool { return b } -var embeddedDynaml = regexp.MustCompile(`^\(\((.*)\)\)$`) +var embeddedDynaml = regexp.MustCompile(`^\(\((([^!].*)?)\)\)$`) func EmbeddedDynaml(root Node) *string { rootString := root.Value().(string)