From 1d3169748936e4891a4f6f6d082e6d0537abaca3 Mon Sep 17 00:00:00 2001 From: David Sabeti Date: Wed, 8 Jun 2016 11:05:07 -0700 Subject: [PATCH 1/5] Update README Explains that `spiff diff` does not honor list order --- README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/README.md b/README.md index 60baf10..6ef83a8 100644 --- a/README.md +++ b/README.md @@ -59,6 +59,23 @@ Show structural differences between two deployment manifests. Unlike 'bosh diff', this command has semantic knowledge of a deployment manifest, and is not just text-based. It also doesn't modify either file. +`spiff diff` ignores order in lists. For example, comparing the following files will yeild an empty diff: +```yaml +# a.yml +list: +- name: item-1 +- name: item-2 + property: foo +``` + +```yaml +# b.yml +list: +- name: item-2 + property: foo +- name: item-1 +``` + It's tailed for checking differences between one deployment and the next. Typical flow: From b0f82db870f0198fbc98d018d9edc1a40dc0e259 Mon Sep 17 00:00:00 2001 From: Amit Gupta Date: Sat, 11 Jun 2016 19:32:19 -0700 Subject: [PATCH 2/5] Update README Clarify `spiff diff` is customized to diffing deployment manifests. --- README.md | 34 ++++++++++++---------------------- 1 file changed, 12 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 6ef83a8..15a6816 100644 --- a/README.md +++ b/README.md @@ -56,35 +56,25 @@ spiff merge cf-release/templates/cf-deployment.yml my-cloud-stub.yml Show structural differences between two deployment manifests. -Unlike 'bosh diff', this command has semantic knowledge of a deployment -manifest, and is not just text-based. It also doesn't modify either file. +Unlike basic diffing tools and even `bosh diff`, this command has semantic +knowledge of a deployment manifest, and is not just text-based. For example, +if two manifests are the same except they have some jobs listed in different +orders, `spiff diff` will detect this, since job order matters in a manifest. +On the other hand, if two manifests differ only in the order of their +resource pools, for instance, then it will yield and empty diff since +resource pool order doesn't actually matter for a deployment. -`spiff diff` ignores order in lists. For example, comparing the following files will yeild an empty diff: -```yaml -# a.yml -list: -- name: item-1 -- name: item-2 - property: foo -``` - -```yaml -# b.yml -list: -- name: item-2 - property: foo -- name: item-1 -``` +Also unlike `bosh diff`, this command doesn't modify either file. -It's tailed for checking differences between one deployment and the next. +It's tailored for checking differences between one deployment and the next. Typical flow: ```sh -$ spiff merge template.yml [templates...] > deployment.yml +$ spiff merge template.yml [templates...] > upgrade.yml $ bosh download manifest [deployment] current.yml -$ spiff diff deployment.yml current.yml -$ bosh deployment deployment.yml +$ spiff diff upgrade.yml current.yml +$ bosh deployment upgrade.yml $ bosh deploy ``` From e5378cc41debe0a3442cc658c5a8a1477b08e037 Mon Sep 17 00:00:00 2001 From: dmitriy kalinin Date: Thu, 1 Sep 2016 11:52:20 -0700 Subject: [PATCH 3/5] make sure nodes that start with ! are ignored so that bosh config server integration can cooperate with spiff --- flow/flow_test.go | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/flow/flow_test.go b/flow/flow_test.go index cde019d..ea6a0f4 100644 --- a/flow/flow_test.go +++ b/flow/flow_test.go @@ -103,6 +103,22 @@ foo: (( auto )) }) }) + Context("when there are ignorable dynaml nodes start with '!'", func() { + It("ignores nodes", func() { + source := parseYAML(` +--- +foo: ((!template_only.foo)) +`) + + resolved := parseYAML(` +--- +foo: ((!template_only.foo)) +`) + + Expect(source).To(FlowAs(resolved)) + }) + }) + Context("when a reference is made to a yet-to-be-resolved node, in a || expression", func() { It("eventually resolves to the referenced node", func() { source := parseYAML(` From fb580e9ed9dfaee0d937e35ad2ad83a0f893b5fe Mon Sep 17 00:00:00 2001 From: Uwe Krueger Date: Fri, 17 Feb 2017 18:05:39 +0100 Subject: [PATCH 4/5] arithmetic on CIDRs (subnet partioning and shifts) --- README.md | 49 +++++++++++++++++++++++++---------- dynaml/division.go | 40 ++++++++++++++++++++++++++-- dynaml/division_test.go | 27 +++++++++++++++++++ dynaml/multiplication.go | 22 ++++++++++++++-- dynaml/multiplication_test.go | 11 ++++++++ flow/flow_test.go | 18 +++++++++++++ 6 files changed, 149 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index d25a0e4..224297e 100644 --- a/README.md +++ b/README.md @@ -861,7 +861,8 @@ The result is the string `3 times 2 yields 6`. ## `(( "10.10.10.10" - 11 ))` -Besides arithmetic on integers it is also possible to use addition and subtraction on ip addresses. +Besides arithmetic on integers it is also possible to use addition and +subtraction on ip addresses, or multiplication and division on CIDRs. e.g.: @@ -877,6 +878,39 @@ ip: 10.10.10.10 range: 10.10.10.10-10.11.11.1 ``` +Subtraction also works on two IP addresses to calculate the number of +IP addresses between two IP addresses. + +e.g.: + +```yaml +diff: (( 10.0.1.0 - 10.0.0.1 + 1 )) +``` + +yields the value 256. IP address constants can be directly used in dynaml +expressions. They are implicitly converted to strings and back to IP +addresses if required by an operation. + +Multiplication and division can be used to handle IP range shifts on CIDRs. +With division a network can be partioned. The network size is increased +to allow at least a dedicated number of subnets below the original CIDR. +Multiplication then can be used to get the n-th next subnet of the same +size. + +e.g.: + +```yaml +subnet: (( "10.1.2.1/24" / 12 )) # first subnet CIDR for 16 subnets +next: (( "10.1.2.1/24" / 12 * 2)) # 2nd next (3rd) subnet CIDRS +``` + +yields + +```yaml +subnet: 10.1.2.0/28 +next: 10.1.2.32/28 +``` + Additionally there are functions working on IPv4 CIDRs: ```yaml @@ -895,19 +929,6 @@ next: 192.168.1.0 num: 192.168.0.0+256=192.168.1.0 ``` -Subtraction also works on two IP addresses to calculate the number of -IP addresses between two IP addresses. - -e.g.: - -```yaml -diff: (( 10.0.1.0 - 10.0.0.1 + 1 )) -``` - -yields the value 256. IP address constants can be directly used in dynaml -expressions. They are implicitly converted to strings and back to IP -addresses if required by an operation. - ## `(( a > 1 ? foo :bar ))` Dynaml supports the comparison operators `<`, `<=`, `==`, `!=`, `>=` and `>`. The comparison operators work on diff --git a/dynaml/division.go b/dynaml/division.go index b6ff400..6190a04 100644 --- a/dynaml/division.go +++ b/dynaml/division.go @@ -2,8 +2,14 @@ package dynaml import ( "fmt" + "log" + "net" ) +func deb(fmt string, args ...interface{}) { + log.Printf(fmt, args...) +} + type DivisionExpr struct { A Expression B Expression @@ -12,7 +18,7 @@ type DivisionExpr struct { func (e DivisionExpr) Evaluate(binding Binding, locally bool) (interface{}, EvaluationInfo, bool) { resolved := true - aint, info, ok := ResolveIntegerExpressionOrPushEvaluation(&e.A, &resolved, nil, binding, false) + a, info, ok := ResolveExpressionOrPushEvaluation(&e.A, &resolved, nil, binding, false) if !ok { return nil, info, false } @@ -29,7 +35,37 @@ func (e DivisionExpr) Evaluate(binding Binding, locally bool) (interface{}, Eval if bint == 0 { return info.Error("division by zero") } - return aint / bint, info, true + + aint, ok := a.(int64) + if ok { + return aint / bint, info, true + } + + str, ok := a.(string) + if ok { + ip, cidr, err := net.ParseCIDR(str) + if err != nil { + return info.Error("CIDR or int argument required as first argument for division: %s", err) + } + ones, bits := cidr.Mask.Size() + ip = ip.Mask(cidr.Mask) + round := false + for bint > 1 { + if bint%2 == 1 { + round = true + } + bint = bint / 2 + ones++ + } + if round { + ones++ + } + if ones > 32 { + return info.Error("divisor too large for CIDR network size") + } + return (&net.IPNet{ip, net.CIDRMask(ones, bits)}).String(), info, true + } + return info.Error("CIDR or int argument required as first argument for division") } func (e DivisionExpr) String() string { diff --git a/dynaml/division_test.go b/dynaml/division_test.go index bb760dd..61e58fb 100644 --- a/dynaml/division_test.go +++ b/dynaml/division_test.go @@ -47,4 +47,31 @@ var _ = Describe("division", func() { Expect(expr).To(FailToEvaluate(FakeBinding{})) }) }) + + Context("when the left-hand side is a CIDR", func() { + It("divides an IP range", func() { + expr := DivisionExpr{ + StringExpr{"10.1.2.1/24"}, + IntegerExpr{4}, + } + + Expect(expr).To(EvaluateAs("10.1.2.0/26", FakeBinding{})) + }) + It("rounds up divisor", func() { + expr := DivisionExpr{ + StringExpr{"10.1.2.1/24"}, + IntegerExpr{12}, + } + + Expect(expr).To(EvaluateAs("10.1.2.0/28", FakeBinding{})) + }) + It("fails for too large divisor", func() { + expr := DivisionExpr{ + StringExpr{"10.1.2.1/24"}, + IntegerExpr{257}, + } + + Expect(expr).To(FailToEvaluate(FakeBinding{})) + }) + }) }) diff --git a/dynaml/multiplication.go b/dynaml/multiplication.go index fc41d24..8cc4bb6 100644 --- a/dynaml/multiplication.go +++ b/dynaml/multiplication.go @@ -2,6 +2,7 @@ package dynaml import ( "fmt" + "net" ) type MultiplicationExpr struct { @@ -12,7 +13,7 @@ type MultiplicationExpr struct { func (e MultiplicationExpr) Evaluate(binding Binding, locally bool) (interface{}, EvaluationInfo, bool) { resolved := true - aint, info, ok := ResolveIntegerExpressionOrPushEvaluation(&e.A, &resolved, nil, binding, false) + a, info, ok := ResolveExpressionOrPushEvaluation(&e.A, &resolved, nil, binding, false) if !ok { return nil, info, false } @@ -25,7 +26,24 @@ func (e MultiplicationExpr) Evaluate(binding Binding, locally bool) (interface{} if !resolved { return e, info, true } - return aint * bint, info, true + + aint, ok := a.(int64) + if ok { + return aint * bint, info, true + } + + str, ok := a.(string) + if ok { + ip, cidr, err := net.ParseCIDR(str) + if err != nil { + return info.Error("CIDR or int argument required for multiplication: %s", err) + } + ones, _ := cidr.Mask.Size() + size := int64(1 << (32 - uint32(ones))) + ip = IPAdd(ip.Mask(cidr.Mask), size*bint) + return (&net.IPNet{ip, cidr.Mask}).String(), info, true + } + return info.Error("CIDR or int argument required as first argument for multiplication") } func (e MultiplicationExpr) String() string { diff --git a/dynaml/multiplication_test.go b/dynaml/multiplication_test.go index 60cde1e..3deb317 100644 --- a/dynaml/multiplication_test.go +++ b/dynaml/multiplication_test.go @@ -36,4 +36,15 @@ var _ = Describe("multiplication", func() { Expect(expr).To(FailToEvaluate(FakeBinding{})) }) }) + + Context("when the left-hand side is a CIDR", func() { + It("shifts the IP range", func() { + expr := MultiplicationExpr{ + StringExpr{"10.1.2.1/24"}, + IntegerExpr{3}, + } + + Expect(expr).To(EvaluateAs("10.1.5.0/24", FakeBinding{})) + }) + }) }) diff --git a/flow/flow_test.go b/flow/flow_test.go index c023729..c1efccf 100644 --- a/flow/flow_test.go +++ b/flow/flow_test.go @@ -5598,4 +5598,22 @@ result: Expect(source).To(FlowAs(resolved)) }) }) + + Describe("when shifting network ranges", func() { + Context("with arithmetic operator", func() { + It("splits and shifts", func() { + source := parseYAML(` +--- +subnet: (( "10.1.2.1/24" / 12 )) +next: (( "10.1.2.1/24" / 12 * 2 )) +`) + resolved := parseYAML(` +--- +subnet: 10.1.2.0/28 +next: 10.1.2.32/28 +`) + Expect(source).To(FlowAs(resolved)) + }) + }) + }) }) From 57545b3dd6e76bc6cc321cd177bc45876acf14a2 Mon Sep 17 00:00:00 2001 From: Uwe Krueger Date: Fri, 17 Feb 2017 22:33:35 +0100 Subject: [PATCH 5/5] Announcement --- README.md | 20 ++++++++++++++++++-- spiff.go | 2 +- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 224297e..540ef4e 100644 --- a/README.md +++ b/README.md @@ -9,13 +9,29 @@ --- -**NOTE**: *Active development on spiff is currently paused, including Pull Requests. Very severe issues will be addressed, and we will still be actively responding to requests for help via Issues.* +**NOTE**: *Active development on spiff is currently paused, including Pull Requests. Very severe issues will be addressed, and we will still be actively responding to requests for help via Issues. This is the last fork version of the original spiff tool. Because there will be no way back to the spiff source base, this version will be the basis for the first version for a new independent spiff++ repository.* --- -spiff is a command line tool and declarative YAML templating system, specially designed for generating BOSH deployment manifests. +*spiff* is a command line tool and declarative in-domain hybrid YAML templating system. While regular templating systems process a template file by substituting the template expressions by values taken from +external data sources, in-domain means that the templating engine knows about the syntax and structure of the processed template. It therefore can take the values for the template expressions directly +from the document processed, including those parts denoted by the template expressions itself. + +For example: +```yaml +resource: + name: bosh deployment + version: 25 + url: (( "http://resource.location/bosh?version=" version )) + description: (( "This document describes a " name " located at " url )) +``` + +Hybrid mean that the template processing is not restricted to the template itself. Additionally +*spiff* is able to merge the template with information from additional yaml files, so-called stubs, that again may contain template expressions. + Contents: + - [Installation](#installation) - [Usage](#usage) - [dynaml Templating Language](#dynaml-templating-language) diff --git a/spiff.go b/spiff.go index 0283e10..c673df1 100644 --- a/spiff.go +++ b/spiff.go @@ -22,7 +22,7 @@ func main() { app := cli.NewApp() app.Name = "spiff" app.Usage = "BOSH deployment manifest toolkit" - app.Version = "1.0.8-dev.6" + app.Version = "1.0.8-dev.7" app.Commands = []cli.Command{ {