Upgrade components to 0.99 and use golang for config processing (#891)

[comment]: # (Note that your PR title should follow the conventional commit format: https://conventionalcommits.org/en/v1.0.0/#summary) # PR Description 1. Upgrade otelcollector and TA to 0.99 2. Remove Prometheus-Operator custom build 3. Use golang for configuration parsing instead of shell scripts 4. Use golang exe as the main exe for container start 5. Configure liveness probe from configuration-reader sidecar to accommodate removal of hot reload 6. Web handler changes to work with latest version [comment]: # (The below checklist is for PRs adding new features. If a box is not checked, add a reason why it's not needed.) # New Feature Checklist - [ ] List telemetry added about the feature. - [ ] Link to the one-pager about the feature. - [ ] List any tasks necessary for release (3P docs, AKS RP chart changes, etc.) after merging the PR. - [ ] Attach results of scale and perf testing. [comment]: # (The below checklist is for code changes. Not all boxes necessarily need to be checked. Build, doc, and template changes do not need to fill out the checklist.) # Tests Checklist - [ ] Have end-to-end Ginkgo tests been run on your cluster and passed? To bootstrap your cluster to run the tests, follow [these instructions](/otelcollector/test/README.md#bootstrap-a-dev-cluster-to-run-ginkgo-tests). - Labels used when running the tests on your cluster: - [ ] `operator` - [ ] `windows` - [ ] `arm64` - [ ] `arc-extension` - [ ] `fips` - [ ] Have new tests been added? For features, have tests been added for this feature? For fixes, is there a test that could have caught this issue and could validate that the fix works? - [ ] Is a new scrape job needed? - [ ] The scrape job was added to the folder [test-cluster-yamls](/otelcollector/test/test-cluster-yamls/) in the correct configmap or as a CR. - [ ] Was a new test label added? - [ ] A string constant for the label was added to [constants.go](/otelcollector/test/utils/constants.go). - [ ] The label and description was added to the [test README](/otelcollector/test/README.md). - [ ] The label was added to this [PR checklist](/.github/pull_request_template). - [ ] The label was added as needed to [testkube-test-crs.yaml](/otelcollector/test/testkube/testkube-test-crs.yaml). - [ ] Are additional API server permissions needed for the new tests? - [ ] These permissions have been added to [api-server-permissions.yaml](/otelcollector/test/testkube/api-server-permissions.yaml). - [ ] Was a new test suite (a new folder under `/tests`) added? - [ ] The new test suite is included in [testkube-test-crs.yaml](/otelcollector/test/testkube/testkube-test-crs.yaml). --------- Co-authored-by: Kaveesh Dubey <[email protected]> Co-authored-by: Grace Wehner <[email protected]> Co-authored-by: Vishwanath <[email protected]> Co-authored-by: Sohamdg081992 <[email protected]>
Azure · Jul 12, 2024 · 516c07e · 516c07e
1 parent 598e5e5
commit 516c07e
Show file tree

Hide file tree

Showing 1,065 changed files with 13,316 additions and 299,865 deletions.
diff --git a/.pipelines/azure-pipeline-build.yml b/.pipelines/azure-pipeline-build.yml
@@ -607,7 +607,7 @@ stages:
       - task: GoTool@0
         displayName: "Build: specify golang version"
         inputs:
-          version: '1.20'
+          version: '1.21.5'
 
       - bash: |
           mkdir -p $(Build.ArtifactStagingDirectory)/linux
@@ -782,7 +782,7 @@ stages:
       - task: GoTool@0
         displayName: "Build: specify golang version"
         inputs:
-          version: '1.20'
+          version: '1.21.5'
 
       - bash: |
           mkdir -p $(Build.ArtifactStagingDirectory)/linuxccp
@@ -1129,7 +1129,7 @@ stages:
       - task: GoTool@0
         displayName: "Build: specify golang version"
         inputs:
-          version: '1.20'
+          version: '1.21.5'
 
       - powershell: |
           ./makefile_windows.ps1
@@ -1162,7 +1162,7 @@ stages:
       - task: GoTool@0
         displayName: "Build: specify golang version"
         inputs:
-          version: '1.20'
+          version: '1.21.5'
 
       - powershell: |
           ./makefile_windows.ps1
@@ -1199,7 +1199,7 @@ stages:
       - task: GoTool@0
         displayName: "Build: specify golang version"
         inputs:
-          version: '1.20'
+          version: '1.21.5'
 
       - bash: |
           export ACR_REPOSITORY_VAR="$(ACR_REPOSITORY)"
@@ -1678,4 +1678,4 @@ stages:
 
               exit 0
             workingDirectory: $(Build.SourcesDirectory)
-            displayName: "Run tests"
+            displayName: "Run tests"
diff --git a/.trivyignore b/.trivyignore
@@ -13,6 +13,8 @@ GHSA-fr2g-9hjm-wr23
 CVE-2023-29403
 CVE-2023-45283
 # HIGH - promconfigvalidator
+GHSA-87m9-rv8p-rgmg
+CVE-2024-36129
 # HIGH - telegraf
 CVE-2023-39325
 GHSA-m425-mq94-257g
@@ -40,13 +42,15 @@ CVE-2024-24783
 CVE-2024-24784
 CVE-2024-24785
 CVE-2024-6104
+GHSA-mh55-gqvf-xfwm
 # MEDIUM - promconfigvalidator
 CVE-2023-48795
 CVE-2024-24786
 CVE-2024-24557
 CVE-2024-27281
 CVE-2020-8559
 CVE-2024-6104
+GHSA-mh55-gqvf-xfwm
 # MEDIUM - go vulnerabilities
 CVE-2023-3978
 CVE-2023-44487
@@ -71,6 +75,7 @@ CVE-2024-28110
 CVE-2024-27289
 CVE-2024-24557
 CVE-2024-29018
+CVE-2024-24791
 # MEDIUM - mariner
 CVE-2023-5678
 # MEDIUM - ruby

diff --git a/RELEASENOTES.md b/RELEASENOTES.md
@@ -3,6 +3,7 @@
 ## Pending
 * fix: Truncate DCE/DCR to 44/64 chars in ARM, Bicep, Terraform... - https://github.com/Azure/prometheus-collector/pull/922
 * fix: add single quotes to pod annotations for multiple namespace scenario - https://github.com/Azure/prometheus-collector/pull/927
+* feat: Upgrade components to 0.99 and use golang for config processing - https://github.com/Azure/prometheus-collector/pull/891
 
 ## Release 06-12-2024
 * Target Allocator Chart update to add Tls secret to the config reader sidecar container  

diff --git a/internal/docs/UpgradeotelCol.md b/internal/docs/UpgradeotelCol.md
@@ -6,76 +6,122 @@ Get latest release version and latest prometheusreceiver code:
 1. Check for the latest release here: https://github.com/open-telemetry/opentelemetry-collector-contrib/releases
 2. git clone https://github.com/open-telemetry/opentelemetry-collector-contrib.git
 3. git checkout tags/<tag_name> -b <branch_name>   tag name will be in the format of v0.x.x and branch name is your local branch name. You can name it whatever you want
-opentelemetry-collector-builder
+
+### opentelemetry-collector-builder
 * update go.mod to new collector version for all components
-update line 18 in main.go with the new collector version
-prometheus-receiver
+* update line 18 in main.go with the new collector version
+### prometheus-receiver
 * copy over new folder
-* go.mod rename module
-go.mod remove replacements at the end
-Find new version of github.com/prometheus/prometheus. Put this version in the file /otelcollector/opentelemetry-collector-builder/PROMETHEUS_VERSION
 * delete testdata directory
 * metrics_receiver.go: rename internal package "github.com/gracewehner/prometheusreceiver/internal"
+* factory.go: rename internal package "github.com/gracewehner/prometheusreceiver/internal/metadata"
+    #### go.mod 
+    * rename module to github.com/gracewehner/prometheusreceiver
+    * remove replacements at the end
+    * add at the top - 
+```
+    replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheus => ../pkgtranslatorprometheus
+```
+### Prometheus version
+Find new version of github.com/prometheus/prometheus. Put this version in the file /otelcollector/opentelemetry-collector-builder/PROMETHEUS_VERSION
+
+### web handler changes to be added 
 * metrics_receiver.go: add webhandler code in initPrometheusComponents() or Start() function
-* metrics_receiver.go: add extra import packages at the top
+* metrics_receiver.go: add extra import packages at the top - 
+	"github.com/prometheus/common/version"
+	"github.com/prometheus/prometheus/web"
 * metrics_receiver.go: add constants at the top
-internal/otlp_transaction.go: in Append() function before if len(t.externalLabels) != 0 (currently line 92) add labels = labels.Copy()
-prom-config-validator-builder
+```
+		// Use same settings as Prometheus web server
+		maxConnections     = 512
+		readTimeoutMinutes = 10
+```
+* metrics_receiver.go: In **type pReceiver struct** - add 
+```
+webHandler        *web.Handler
+```
+<!-- internal/otlp_transaction.go: in Append() function before if len(t.externalLabels) != 0 (currently line 92) add labels = labels.Copy() -->
+
+### strusthash changes 
+* Refer to the previous version and make changes or if changes are merged just take latest from collector-contrib repo
+
+### opentelemetry-collector-builder - 
+* go mod tidy
+* make
+
+
+### prom-config-validator-builder
 * update go.mod to new collector version for all components
-* try to build to check for any breaking changes to the interfaces used: run make
-opentelemetry-collector-builder
-go mod tidy
+* copy the second block of go.mod from the latest of go.mod of opentelemetry-collector-builder 
+* try to build to check for any breaking changes to the interfaces used: 
+* Run - go mod tidy
+* Run - make
+
 
-Code block
+### Update golang versions
+* Update go versions in the dockerfiles for build to use the same version in the above components
+
+Code block for web handler (This will be moved to extension)
 ```
-module github.com/gracewehner/prometheusreceiver
-    "github.com/prometheus/client_golang/prometheus"
-    "github.com/prometheus/common/version"
-    "github.com/prometheus/prometheus/web"
-    // Use same settings as Prometheus web server
-    maxConnections = 512
-    readTimeoutMinutes = 10
-    // Setup settings and logger and create Prometheus web handler
-    webOptions := web.Options{
-        ScrapeManager: r.scrapeManager,
-        Context: ctx,
-        ListenAddress: ":9090",
-        ExternalURL: &url.URL{
-            Scheme: "http",
-            Host:   "localhost:9090",
-            Path:   "",
-        },
-        RoutePrefix:    "/",
-        ReadTimeout: time.Minute * readTimeoutMinutes,
-        PageTitle: "Prometheus Receiver",
-        Version: &web.PrometheusVersion{
-            Version:   version.Version,
-            Revision:  version.Revision,
-            Branch:    version.Branch,
-            BuildUser: version.BuildUser,
-            BuildDate: version.BuildDate,
-            GoVersion: version.GoVersion,
-        },
-        Flags: make(map[string]string),
-        MaxConnections: maxConnections,
-        IsAgent: true,
-        Gatherer:   prometheus.DefaultGatherer,
-    }
-    go_kit_logger := log.NewLogfmtLogger(log.NewSyncWriter(os.Stderr))
-    webHandler := web.New(go_kit_logger, &webOptions)
-    listener, err := webHandler.Listener()
-    if err != nil {
-        return err
-    }
-    // Pass config and let the web handler know the config is ready.
-    // These are needed because Prometheus allows reloading the config without restarting.
-    webHandler.ApplyConfig(r.cfg.PrometheusConfig)
-    webHandler.SetReady(true)
-    // Uses the same context as the discovery and scrape managers for shutting down
-    go func() {
-        if err := webHandler.Run(ctx, listener, ""); err != nil {
-            r.settings.Logger.Error("Web handler failed", zap.Error(err))
-            host.ReportFatalError(err)
-        }
-    }()
+// Setup settings and logger and create Prometheus web handler
+	webOptions := web.Options{
+		ScrapeManager: r.scrapeManager,
+		Context:       ctx,
+		ListenAddress: ":9090",
+		ExternalURL: &url.URL{
+			Scheme: "http",
+			Host:   "localhost:9090",
+			Path:   "",
+		},
+		RoutePrefix: "/",
+		ReadTimeout: time.Minute * readTimeoutMinutes,
+		PageTitle:   "Prometheus Receiver",
+		Version: &web.PrometheusVersion{
+			Version:   version.Version,
+			Revision:  version.Revision,
+			Branch:    version.Branch,
+			BuildUser: version.BuildUser,
+			BuildDate: version.BuildDate,
+			GoVersion: version.GoVersion,
+		},
+		Flags:          make(map[string]string),
+		MaxConnections: maxConnections,
+		IsAgent:        true,
+		Gatherer:       prometheus.DefaultGatherer,
+	}
+	go_kit_logger := log.NewLogfmtLogger(log.NewSyncWriter(os.Stderr))
+	r.webHandler = web.New(go_kit_logger, &webOptions)
+	listener, err := r.webHandler.Listener()
+	if err != nil {
+		return err
+	}
+	// Pass config and let the web handler know the config is ready.
+	// These are needed because Prometheus allows reloading the config without restarting.
+	r.webHandler.ApplyConfig((*config.Config)(r.cfg.PrometheusConfig))
+	r.webHandler.SetReady(true)
+	// Uses the same context as the discovery and scrape managers for shutting down
+	go func() {
+		if err := r.webHandler.Run(ctx, listener, ""); err != nil {
+			r.settings.Logger.Error("Web handler failed", zap.Error(err))
+			r.settings.TelemetrySettings.ReportStatus(component.NewFatalErrorEvent(err))
+		}
+	}()
+``` 
+
+### TargetAllocator Update
+Get latest release version and latest prometheusreceiver code:
+1. Check for the latest release here: https://github.com/open-telemetry/opentelemetry-operator/releases (Pick the same version as opentelemetry-collector)
+2. git clone https://github.com/open-telemetry/opentelemetry-operator.git
+3. git checkout tags/<tag_name> -b <branch_name>   tag name will be in the format of v0.x.x and branch name is your local branch name. 
+4. Copy the folder otel-allocator
+5. Update Dockerfile with the existing Dockerfile changes accordingly(Make sure to include prometheus-operators' api group customization for build command like below)
 ```
+go build -buildmode=pie -ldflags '-linkmode external -extldflags=-Wl,-z,now -s -X github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring.GroupName=azmonitoring.coreos.com' -o main . ; else CGO_ENABLED=1 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -buildmode=pie -ldflags '-linkmode external -extldflags=-Wl,-z,now -s -X github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring.GroupName=azmonitoring.coreos.com'
+```
+6. Update main.go to include ARC EULA (lines 69-73)
+7. Update go.mod file in the otel-allocator folder with the go.mod of the opentelemetry-operator file.
+8. Run go mod tidy from the otel-allocator directory and then run make.
+
+### Configuration Reader Builder
+1. Update the version of operator in go.mod of configuration-reader-builder to match versions in go.mod of otel-allocator
+2. Run go mod tidy from configuration-reader-builder directory and then run make