Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decouple 3rd party integrations from Concourse #57

Closed
wants to merge 2 commits into from
Closed

Decouple 3rd party integrations from Concourse #57

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1669091
add decoupling rfc
xtreme-sameer-vohra Jun 1, 2020
be3363d
Update proposal.md
xtreme-sameer-vohra Jun 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions 056-decouple-integrations/proposal.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Summary

Concourse has several types of integrations such as metrics and credential stores. For each type, there are several implementations such as Datadog, New Relic, Prometheus, etc.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's probably worth cross-referencing var sources (#39) and prototypes (#37) as those are planned to be combined like peanut-butter and jelly to achieve decoupling of credential management in Concourse. The latest revision introduces support for encrypting which was the missing piece of the puzzle.

There'd still be a few direct integrations remaining: policy enforcement (OPA is currently the only implementation, but theoretically it's generic), metrics, and tracing. Using Prototypes for these scenarios is probably impractical since those would benefit from lower latency and high throughput.

This RFC proposes making those integrations `pluggable` to enhance the user experience.

# Motivation

Currently, an update to an existing Concourse integration or addition of a new integration requires a new release of Concourse.
This results in additional time and complexity to make changes and extend Concourse

For example, adding a new credentials store [Conjur](https://github.com/concourse/concourse/pull/4693) required getting a PR merged into the main Concourse codebase and a new release of Concourse with the changes.

Similarly, leveraging a [new batch processing](https://github.com/concourse/concourse/pull/4698) feature in the `New Relic API` required updating the main Concourse codebase and a new release of Concourse with the changes.


## Challenges
- greater concensus requirements for adding Concourse integrations
- longer turn around for enhancements and bug fixes to Concourse integrations
- leverage the large and vibrant Concourse community
- prevent the Concourse team from being a bottleneck especially when we aren't experts in a particular specific integration (eg. `Conjur`)

# Proposal

Define interfaces and externalize the integrations such that adding new integrations or updating existing integrations doesn't require a new version of Concourse.

One way to achieve this would be to use [gRP](https://grpc.io/) where each integration type would have a `service` defining the interface.
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aoldershaw also brought my attention to doing something similar for the build events storage

xtreme-sameer-vohra marked this conversation as resolved.
Show resolved Hide resolved
The interface would define the set of methods including their arguments and responses.
This would be preferred over defining a `json/http` style API due to
- code generation for clients
- transport optimized
- simpler management

This would allow users to use a new integration or update an existing integration by downloading a new version of the integration or bulding it locally and allowing Concourse to connect to it via socket connection.
This would also allow integrations to be written in other languages.

There are many examples of successfully doing this in the larger community such as the [containerd project](https://www.slideshare.net/Docker/driving-containerd-oprations-with-grpc).

# Open Questions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question I have is how we would go about supporting plugin configuration. Currently, we have specific flags for e.g. each credential manager. One approach is to simply pass along environment variables with a specific prefix (e.g. CONCOURSE_CREDS_*) to the plugin, and allow the plugin to interpret them - but we lose the ability to see what config options are available from concourse web -help (which is maybe okay?)

One idea to remediate this is to come up with some interface for plugins to expose their configuration options to Concourse (via gRPC)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, the approach of passing along the env vars with a prefix makes sense (there is precedence here with Garden + Concourse already) and would keep the plugin and Concourse decoupled.
And subsequently tackling a way to surface the plugin configuration options to the user via a flavor ofconcourse -h would be a awesome enhancement


How do we manage changes to the interface ?
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should follow Terraform's lead on this, particularly wrt how they support providers (and provisioners I believe). They use their in-house https://github.com/hashicorp/go-plugin, which supports gRPC-based plugins. From what I can tell, the benefit to using go-plugin over gRPC alone is that it supports some form of plugin versioning (I haven't looked to much into it) on both the client and server side.

Worth taking a look at how it's used in Terraform: https://github.com/hashicorp/terraform/tree/master/plugin

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool find. will take a look 👀

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For completeness, the Terraform plugin code has been extracted into a separate project. The latest version of the code can be found here: https://github.com/hashicorp/terraform-plugin-sdk

How does this work impact standardizing on [opentelemetry](https://github.com/concourse/concourse/pull/5104) for metrics ?
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this boils down to the general question: what will be the API within Concourse for a certain integration that is powerfull enough so it can support multiple external implementations as plugins?

Let's focus on this observability angle: is OpenTelemetry the API to use inside of Concourse or is a Concourse specific API needed delegating to an OpenTelemetry specific implementation? There are arguments to say on both approaches.

Looking at the OpenTelemetry website, I read:

OpenTelemetry provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application.

The client side delegates to a collector for further processing. This is your API & plugin. So for observability, I would just pick OpenTelemetry as the API to integrate in the Concourse codebase and everybody will be able to plug it to their own observability platform, whether OpenSource or commercial.

Putting yet another API in front of OpenTelemetry would be a waste of cycles and nobody would win here IMO.



# Answered Questions

# New Implications

- Defining interfaces for integration types that we wish to externalize
- Defining the method by which users may leverage new integrations or different versions of integrations than what Concourse was shipped with ( This would be very similar to how we manage built-in resources )