Filter resources NOT having a tag (#101)

* updated to allow for all resources not matching tag to be removed * better example of use * Refactor NewFilter function * move unmarshal JSON into NewFilter * return error instead of Fatal() * Add tagged filter If true, select all resources with any tag. If false, select all resources without tags. * Check len(tags) == 0 instead of tags != nil * Separate MatchTags and MatchTagged * Add notags filter * Remove notags and allow NOT(key) in tags filter * Update README * Fix using matchTags and matchNoTags together Co-authored-by: Richard Silver <[email protected]>
jckuester · May 11, 2020 · b05f0b0 · b05f0b0
1 parent 5674065
commit b05f0b0
Show file tree

Hide file tree

Showing 9 changed files with 554 additions and 231 deletions.
diff --git a/Makefile b/Makefile
@@ -33,7 +33,7 @@ test: ## Run unit tests
 .PHONY: test-all
 test-all: ## Run tests (including acceptance and integration tests)
 	go clean -testcache ${PKG_LIST}
-	./bin/go-acc ${PKG_LIST} -- -v -$(TESTARGS) -p 1 -race -timeout 30m
+	./bin/go-acc ${PKG_LIST} -- -v $(TESTARGS) -p 1 -race -timeout 30m
 
 .PHONY: build
 build: ## Build binary

diff --git a/README.md b/README.md
@@ -1,130 +1,123 @@
-# AWSweeper
-
-<p align="right">
-  <a href="https://github.com/cloudetc/awsweeper/releases/latest">
-    <img alt="Release" src="https://img.shields.io/github/release/cloudetc/awsweeper.svg?style=flat-square">
-  </a>
-  <a href="https://travis-ci.org/cloudetc/awsweeper">
-    <img alt="pipeline status" src="https://img.shields.io/travis/cloudetc/awsweeper/master.svg" />
-  </a>
-  <a href="https://goreportcard.com/report/github.com/cloudetc/awsweeper">
-    <img alt="Go Report" src="https://goreportcard.com/badge/github.com/cloudetc/awsweeper?style=flat-square" />
-  </a>
-  <a href="https://godoc.org/github.com/cloudetc/awsweeper">
-    <img alt="Go Doc" src="https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square" />
-  </a>
-  <a href="https://github.com/cloudetc/awsweeper/blob/master/LICENSE">
-    <img alt="Software License" src="https://img.shields.io/github/license/cloudetc/awsweeper.svg?style=flat-square" />
-  </a>
+<p align="center">
+  <img alt="AWSweeper Logo" src="https://github.com/cloudetc/awsweeper/blob/master/img/logo.png" height="180" />
+  <h3 align="center">AWSweeper</h3>
+  <p align="center">A tool for cleaning your AWS account</p>
 </p>
 
-AWSweeper wipes out all (or parts) of the resources in your AWS account. Resources to be deleted can be filtered by their ID, tags or
-creation date using [regular expressions](https://golang.org/pkg/regexp/syntax/) declared in a yaml file (see [config.yml](example/config.yml)).
+---
+[![Release](https://img.shields.io/github/release/cloudetc/awsweeper.svg?style=for-the-badge)](https://github.com/cloudetc/awsweeper/releases/latest)
+[![Software License](https://img.shields.io/badge/license-MIT-brightgreen.svg?style=for-the-badge)](/LICENSE.md)
+[![Travis](https://img.shields.io/travis/cloudetc/awsweeper/master.svg?style=for-the-badge)](https://travis-ci.org/cloudetc/awsweeper)
+
+AWSweeper cleans out all (or parts) of the resources in your AWS account. Resources to be deleted can be filtered by
+their ID, tags or creation date using [regular expressions](https://golang.org/pkg/regexp/syntax/) declared via a filter
+in a YAML file (see [filter.yml](example/config.yml) as an example).
 
 AWSweeper [can delete many](#supported-resources), but not all resources yet. Your help
-supporting more resources is very much appreciated ([please read this issue](https://github.com/cloudetc/awsweeper/issues/21)
- to see how easy it is). Note that AWSweeper is based on the cloud-agnostic Terraform API for deletion - so it's planned to support
- deleting Azure and Google Cloud Platform resources soon, too.
+to support more resources is very much appreciated ([please read this issue](https://github.com/cloudetc/awsweeper/issues/21)
+ to see how easy it is). 
 
 Happy erasing!
 
 [![AWSweeper tutorial](img/asciinema-tutorial.gif)](https://asciinema.org/a/149097)
 
 ## Installation
 
-It's recommended to install a specific version of awsweeper available on the
+It's recommended to install a specific version of AWSweeper available on the
 [releases page](https://github.com/cloudetc/awsweeper/releases).
 
-Here is the recommended way to install awsweeper v0.6.0:
+Here is the recommended way to install AWSweeper v0.8.0:
 
 ```bash
 # install it into ./bin/
-curl -sSfL https://raw.githubusercontent.com/cloudetc/awsweeper/master/install.sh | sh -s v0.6.0
+curl -sSfL https://raw.githubusercontent.com/cloudetc/awsweeper/master/install.sh | sh -s v0.8.0
 ```
 
 ## Usage
 
-    awsweeper [options] <config.yml>
+    awsweeper [options] <filter.yml>
 
 To see options available run `awsweeper --help`.
 
-## Filtering
+## Filter
 
-Resources to be deleted are filtered by a yaml configuration. To learn how, have a look at the following example:
+Resources are deleted via a filter declared in a YAML file.
 
     aws_instance:
+      # instance filter part 1
       - id: ^foo.*
-        tags:
-          foo: bar
-          bla: blub
         created:
-          before: 2018-06-14
-          after: 2018-10-28 12:28:39.0000
+          before: 2018-10-14
+          after: 2018-06-28 12:28:39
+            
+      # instance filter part 2   
       - tags:
           foo: bar
-         created:
-           before: 2018-06-14
-      - tags:
-          foo: NOT(bar)
-        created:
-          after: 2018-06-14
-    aws_iam_role:
-
-This config would delete all instances which ID matches `^foo.*` *AND* which have tags `foo: bar` *AND* `bla: blub`
-*AND* which have been created between `2018-10-28 12:28:39 +0000 UTC` and `2018-06-14`. Additionally, it would delete instances
-with tag `foo: bar` and which are older than `2018-06-14`.
+          NOT(owner): .*
+           
+    aws_security_groups:
 
-Furthermore, this config would delete all IAM roles, as there is no list of filters provided for this resource type.
+The filter snippet above deletes all EC2 instances that ID matches `^foo.*` and that have been created between
+ `2018-06-28 12:28:39` and `2018-10-14` UTC (instance filter part 1); additionally, EC2 instances having a tag 
+ `foo: bar` *AND* not a tag key `owner` with any value are deleted (instance filter part 2); last but not least,
+ ALL security groups are deleted by this filter.
 
-The general syntax of the filter config is as follows:
+The general filter syntax is as follows:
 
     <resource type>:
-      # filter 1
       - id: <regex to filter by id> | NOT(<regex to filter by id>)
+        tagged: bool (optional)
         tags:
-          <key>: <regex to filter value> | NOT(<regex to filter value>)
+          <key> | NOT(key): <regex to filter value> | NOT(<regex to filter value>)
           ...
         created:
           before: <timestamp> (optional)
           after: <timestamp> (optional)
-      # filter 2
+      # OR
       - ...
     <resource type>:
       ...
 
-A more detailed description of the ways to filter resources:
+Here is a more detailed description of the various ways to filter resources:
 
-##### 1) All resources of a particular type
+##### 1) Delete all resources of a particular type
 
-   [Terraform types](https://www.terraform.io/docs/providers/aws/index.html) are used to identify resources of a particular type
-   (e.g., `aws_security_group` selects all resources that are security groups, `aws_iam_role` all roles,
-   or `aws_instance` all EC2 instances).
-
-   In the example above, by simply adding `security_group:` (no further filters for IDs or tags),
-   all security groups in your account would be deleted. Use the [all.yml](./all.yml), to delete all (currently supported)
+   [Terraform resource type indentifiers](https://www.terraform.io/docs/providers/aws/index.html) are used to delete 
+   resources by type. The following filter snippet deletes *ALL* security groups, IAM roles, and EC2 instances:
+
+    aws_security_group:
+    aws_iam_role:
+    aws_instance:
+
+   Don't forget the `:` at the end of each line. Use the [all.yml](./all.yml), to delete all (currently supported)
    resources.
 
-##### 2) By tags
-
-   You can narrow down on particular types of resources by the tags they have.
+##### 2) Delete by tags
 
-   If most of your resources have tags, this is probably the best to filter them
-   for deletion. But be aware: not all resources support tags and can be filtered this way.
+   If most of your resources have tags, this is probably the best way to filter them
+   for deletion. **Be aware**: Not all resources [support tags](#supported-resources) yet and can be filtered this way.
+
+   The key and the value part of the tag filter can be negated by a surrounding `NOT(...)`. This allows for removing of 
+   all resources not matching some tag key or value. In the example below, all EC2 instances without the `owner: me`
+   tag are deleted:
 
-   In the example above, all EC2 instances are terminated that have a tag with key `foo` and value `bar` as well as
-   `bla` and value `blub`.
-
-   The tag filter can be negated by surrounding the regex with `NOT(...)`
+    aws_instance:
+      - tags:
+          NOT(Owner): me
+          
+   The flag `tagged: false` deletes all resources that have no tags. Contrary, resources with any tags can be deleted 
+   with `tagged: true`:
 
-##### 3) By ID
+    aws_instance:
+      - tagged: true
 
-   You can narrow down on particular types of resources by filtering on their IDs.
+##### 3) Delete By ID
 
-   To see what the IDs of your resources are (could be their name, ARN, a random number),
-   run awsweeper in dry-run mode: `awsweeper --dry-run all.yml`. This way, nothing is deleted but
-   all the IDs and tags of your resources are printed. Then, use this information to create the yaml file.
+   You can narrow down on particular types of resources by filtering on based their IDs.
 
-   In the example above, all roles which name starts with `foo` are deleted (the ID of roles is their name).
+   To see what the ID of a resource is (could be its name, ARN, a random number),
+   run AWSweeper in dry-run mode: `awsweeper --dry-run all.yml`. This way, nothing is deleted but
+   all the IDs and tags of your resources are printed. Then, use this information to create the YAML config file.
 
    The id filter can be negated by surrounding the regex with `NOT(...)`
 
@@ -152,8 +145,8 @@ A more detailed description of the ways to filter resources:
 
 ## Dry-run mode
 
- Use `awsweeper --dry-run <config.yml>` to only show what
-would be deleted. This way, you can fine-tune your yaml configuration until it works the way you want it to.
+ Use `awsweeper --dry-run <filter.yml>` to only show what
+would be deleted. This way, you can fine-tune your YAML filter configuration until it works the way you want it to.
 
 ## Supported resources
 

diff --git a/command/wipe.go b/command/wipe.go
@@ -62,11 +62,16 @@ func list(c *Wipe) []terradozerRes.DestroyableResource {
 // Run executes the wipe command.
 func (c *Wipe) Run(args []string) int {
 	if len(args) == 1 {
-		c.filter = resource.NewFilter(args[0])
+		filter, err := resource.NewFilter(args[0])
+		if err != nil {
+			log.WithError(err).Fatal("failed to create resource filter")
+		}
+
+		c.filter = filter
 
-		err := c.filter.Validate()
+		err = c.filter.Validate()
 		if err != nil {
-			log.WithError(err).Fatal("failed to validate filter config")
+			log.WithError(err).Fatal("invalid filter config")
 		}
 	} else {
 		fmt.Println(help())

diff --git a/command/wrapped_main.go b/command/wrapped_main.go
@@ -21,7 +21,7 @@ import (
 // WrappedMain is the actual main function that does not exit for acceptance testing purposes
 func WrappedMain() int {
 	app := "awsweeper"
-	version := "v0.7.0"
+	version := "v0.8.0"
 
 	set := flag.NewFlagSet(app, 0)
 	versionFlag := set.Bool("version", false, "Show version")

diff --git a/img/logo.png b/img/logo.png