Skip to content

Latest commit

 

History

History

artifactory

README.md

Artifactory

A collection of commands and scripts I use to automate some tasks with JFrog Artifactory

Run Artifactory

Artifactory with local Docker

You can easily spin up a local Artifactory container with

# Run Artifactory with mounting a local artifactory directory to container's /var/opt/jfrog/artifactory
docker run -d --name rt -p 8082:8082 -v $(pwd)/artifactory:/var/opt/jfrog/artifactory releases-docker.jfrog.io/jfrog/artifactory-pro

Open a browser to http://localhost:8082

Artifactory in Kubernetes

Examples of commands to install Artifactory in K8s with helm using various databases. See examples used here of custom values files in values-examples.

Setup Helm repository

Add JFrog's Helm charts repository

helm repo add jfrog https://charts.jfrog.io
helm repo update

Default Install

Install with Artifactory's default bundled database PostgreSQL

helm upgrade --install artifactory jfrog/artifactory

With an External PostgreSQL (recommended)

Install Artifactory with external PostgreSQL database in K8s

# Add Bitnami helm repository
helm repo add bitnami https://charts.bitnami.com/bitnami

# Install PostgreSQL
helm upgrade --install postgresql bitnami/postgresql -f values-examples/values-postgresql.yaml

# Install Artifactory (PostgreSQL driver already included in Docker image)
helm upgrade --install artifactory jfrog/artifactory -f values-examples/values-postgresql.yaml

# Open a shell to the postgresql pod and connect to the database
kubectl exec -it postgresql-0 -- bash

# Inside the container, connect to the database
psql --host postgresql -U artifactory

# You can open a client container to this database with
kubectl run pg-client --rm --tty -i --restart='Never' --image docker.io/bitnami/postgresql:15.1.0-debian-11-r13 \
    --env="PGPASSWORD=password1" --labels="postgresql-client=true" --command -- psql --host postgresql -U artifactory

With an External MySQL (deprecated)

Install Artifactory with external MySQL database in K8s (deprecated and will eventually not be supported)

# Add Bitnami helm repository
helm repo add bitnami https://charts.bitnami.com/bitnami

# Install MySQL
helm upgrade --install mysql bitnami/mysql -f values-examples/values-mysql.yaml

# Install Artifactory
helm upgrade --install artifactory jfrog/artifactory -f values-examples/values-mysql.yaml

# Open a shell to the mysql pod and connect to the database
kubectl exec -it mysql-0 -- bash

# Inside the container, connect to the database
mysql --host=localhost --user=artifactory --password=password1 artifactory

With a memory based emptyDir volume as cache-fs

Install with a memory backed emptyDir for cache-fs

IMPORTANT: The memory used by the volume is counted against your container's memory limit!
So you need to adjust the artifactory container's memory limit to + 

helm upgrade --install artifactory jfrog/artifactory -f values-examples/values-memory-cache-fs.yaml

Benchmark Artifactory

Here are a couple of ways to benchmark Artifactory. This is very useful for comparing the impact of configuration changes on the performance of Artifactory

Using Apache ab

Using Apache ab is useful for running multiple concurrent downloads of a single file.
Multiple instances of ab can be run to add more load and to combine multiple files and technologies

Downloads

First, upload a test file you want to use to a repository. After that, you can run

# From official ab docs:
# -A username:password: Supply BASIC Authentication credentials to the server.
# -c <num>: Number of multiple requests to perform at a time. Default is one request at a time.
# -n <num>: Number of requests to perform for the benchmarking session. The default is to just perform a single request which usually leads to non-representative benchmarking results.

# 10 concurrent downloads with a total of 1000 requests (basic auth)
ab -A admin:password -c 10 -n 1000 -k http://localhost/artifactory/example-repo-local/example-file.bin

# Using an access token instead of basic auth to download
ab -H "Authorization: Bearer ${TOKEN}" -c 10 -n 1000 -k http://localhost/artifactory/example-repo-local/example-file.bin

Uploads

WARNING: For upload tests, use a concurrent value of 1 to avoid errors!

First, create a local file to be uploaded

# From official ab docs:
# -A username:password: Supply BASIC Authentication credentials to the server.
# -c <num>: Number of multiple requests to perform at a time. Default is one request at a time.
# -n <num>: Number of requests to perform for the benchmarking session. The default is to just perform a single request which usually leads to non-representative benchmarking results.
# -u <file>: File containing data to PUT.

# 1 concurrent upload of a file 50 times
ab -A admin:password -u ./file.bin -c 1 -n 50 -k http://localhost/artifactory/example-repo-local/file.bin

# Use a token instead of basic auth to upload
ab -H "Authorization: Bearer ${TOKEN}" -u ./file.bin -c 1 -n 50 -k http://localhost/artifactory/example-repo-local/file.bin

Artifactory Load Helm Chart

You can use the artifactory-load helm chart to deploy one or more pods running ab once or in an infinite loop.

This chart support up to 4 different ab scenarios at the same time (using multiple jobs). The default is a single job.

Upload the files you want to use for testing. These should each be configured in its own jobX block. See example below that has 4 different scenarios with different sizes.

Create a test-value.yaml with the specific details of your run

# Set the URL and credentials to access the Artifactory server to be tested
artifactory:
  url: http://artifactory-server
  auth: true
  user: admin
  password: password

# Run 2 pods with each pulling a 1KB file 100,000 times with 2 concurrent requests
job0:
  parallelism: 2
  file: example-repo-local/file1KB.bin
  requests: "100000"
  concurrency: "2"

# Run 1 pod pulling a 1KB file 50,000 times with 2 concurrent requests
job1:
  parallelism: 1
  enabled: true
  file: example-repo-local/file10KB.bin
  requests: "50000"
  concurrency: "2"

# Run 3 pods with each pulling a 100KB file 10,000 times with 3 concurrent requests
job2:
  parallelism: 3
  enabled: true
  file: example-repo-local/file100KB.bin
  requests: "10000"
  concurrency: "3"

# Run 1 pod pulling a 10MB file 10,000 times with 1 concurrent requests
job3:
  parallelism: 1
  enabled: true
  file: example-repo-local/file10MB.bin
  requests: "10000"
  concurrency: "1"

Deploy the chart with the command

helm upgrade --install al . -f test-values.yaml

Shell Scripts

  • artifactoryBenchmark.sh - Run download, upload (or both) tests with a single file for a given size and iterations count. Results as CSV
  • artifactoryLoad.sh - Run parallel processes of artifactoryBenchmark.sh. This is useful for generating load on Artifactory
  • artifactoryDownloadsLoop.sh - Create and upload a single generic binary generic file to Artifactory and download it in loops with set iterations and parallel threads

Each script has its own usage you can get with

./<script.sh> --help

Benchmark From Within a Pod

You can run these tests from within a pod in Kubernetes

# Deploy a pod with the needed tools
kubectl apply -f https://github.com/eldada/kubernetes-scripts/raw/master/yaml/podWithTools.yaml

# Open a shell to the pod
kubectl exec -it pod-with-tools --bash

## You can use the load scripts from this repository, or directly the Apache ab utility, which is also installed in this pod

# To use the scripts, clone this repository
cd /opt
git clone https://github.com/eldada/command-examples.git

cd command-examples/artifactory

# Run the script(s) you want
# NOTE - use the internal artifactory service rather than the external load balancer
./artifactoryBenchmark.sh --help