Skip to content

Enabling FlexGroup Constituents

Jump to bottom
Hardikl edited this page Oct 18, 2023 · 4 revisions

Harvest Guide for Enabling FlexGroup Constituents

You must extend all the volume.yaml templates with the include_constituents flag to true to enable flexgroup constituents counters.

Note: Enabling FlexGroup Constituents counters may slow down data collection due to a high number of constituents metrics.

For RPM, DEB, Native and Docker Installation:

Extend Rest volume template

We are going to create a new template and add it to your custom.yaml. In the commands below, Harvest is installed in /opt/harvest. Adjust as needed.

  1. Edit vi /opt/harvest/conf/rest/custom.yaml

  2. Add below content to this file. If custom.yaml already exists then add the Volume line under the Objects: section.

objects:
  Volume: custom_volume.yaml

  1. Create the custom_volume.yaml file vi /opt/harvest/conf/rest/9.12.0/custom_volume.yaml

  2. Copy existing /opt/harvest/conf/rest/9.12.0/volume.yaml template and changed the flag include_constituents: true and save.

name:                     Volume
query:                    api/private/cli/volume
object:                   volume

counters:
  - ^^volume
  - ^^vserver                                     => svm
  - ^aggr_list                                    => aggr
  - ^clone_parent_name                            => clone_parent_volume
  - ^clone_parent_vserver                         => clone_parent_svm
  - ^clone_volume                                 => is_flexclone
  - ^is_encrypted                                 => isEncrypted
  - ^is_sis_volume                                => is_sis_volume
  - ^junction_path                                => junction_path
  - ^nodes                                        => node
  - ^snaplock_type                                => snaplock_type
  - ^snapshot_policy                              => snapshot_policy
  - ^state                                        => state
  - ^type                                         => type
  - ^volume_style_extended                        => style
  - ^vsroot                                       => svm_root
  - autosize_grow_threshold_percent               => autosize_grow_threshold_percent
  - available                                     => size_available
  - compression_space_saved                       => sis_compress_saved
  - compression_space_saved_percent               => sis_compress_saved_percent
  - dedupe_space_saved                            => sis_dedup_saved
  - dedupe_space_saved_percent                    => sis_dedup_saved_percent
  - expected_available                            => space_expected_available
  - files                                         => inode_files_total
  - files_used                                    => inode_files_used
  - filesystem_size                               => filesystem_size
  - logical_used                                  => space_logical_used
  - logical_used_by_afs                           => space_logical_used_by_afs
  - logical_used_by_snapshots                     => space_logical_used_by_snapshots
  - logical_used_percent                          => space_logical_used_percent
  - max_autosize                                  => autosize_maximum_size
  - overwrite_reserve                             => overwrite_reserve_total
  - overwrite_reserve_used                        => overwrite_reserve_used
  - percent_snapshot_space                        => snapshot_reserve_percent
  - percent_used                                  => size_used_percent
  - physical_used                                 => space_physical_used
  - physical_used_percent                         => space_physical_used_percent
  - sis_space_saved                               => sis_total_saved
  - sis_space_saved_percent                       => sis_total_saved_percent
  - size                                          => size
  - size_available_for_snapshots                  => snapshots_size_available
  - size_used_by_snapshots                        => snapshots_size_used
  - snapshot_count
  - snapshot_reserve_available                    => snapshot_reserve_available
  - snapshot_reserve_size                         => snapshot_reserve_size
  - snapshot_space_used                           => snapshot_reserve_used_percent
  - total                                         => size_total
  - used                                          => size_used
  - filter:
      - is_constituent=*

endpoints:
  - query: api/private/cli/volume/efficiency/stat
    counters:
      - ^^volume
      - ^^vserver                                 => svm
      - num_compress_attempts
      - num_compress_fail
      - filter:
          - privilege_level=diagnostic

  - query: api/private/cli/volume/footprint
    counters:
      - ^^volume
      - ^^vserver                                 => svm
      - volume_blocks_footprint_bin0              => performance_tier_footprint
      - volume_blocks_footprint_bin0_percent      => performance_tier_footprint_percent
      - volume_blocks_footprint_bin1              => capacity_tier_footprint
      - volume_blocks_footprint_bin1_percent      => capacity_tier_footprint_percent


plugins:
  - Volume:
      schedule:
        - data: 15m  # should be multiple of poll duration
      include_constituents: true
  - MetricAgent:
      compute_metric:
        - inode_used_percent PERCENT inode_files_used inode_files_total
        - snapshot_reserve_used SUBTRACT snapshot_reserve_size snapshot_reserve_available
        - overwrite_reserve_available SUBTRACT overwrite_reserve_total overwrite_reserve_used
  - LabelAgent:
      value_to_num:
        - new_status state online online `0`
      replace:
        - svm_root root_volume `false` `No`
        - svm_root root_volume `true` `Yes`
      # To prevent visibility of transient volumes, uncomment the following lines
#      exclude_regex:
#        # Exclude SnapProtect/CommVault Intellisnap, Clone volumes have a “_CVclone” suffix
#        - volume `.+_CVclone`
#        # Exclude SnapCenter, Clone volumes have a “DDMMYYhhmmss” suffix
#        - volume `.+(0[1-9]|[12][0-9]|3[01])(0[1-9]|1[012])\d\d[0-9]{6}`
#        # Exclude manually created SnapCreator clones, Clone volumes have a “cl_” prefix and a “_YYYYMMDDhhmmss” suffix
#        - volume `cl_.+_(19|20)\d\d(0[1-9]|1[012])( 0[1-9]|[12][0-9]|3[01])[0-9]{6}`
#        # Exclude SnapDrive/SnapManager, Clone volumes have a “sdw_cl_” prefix
#        - volume `sdw_cl_.+`
#        # Exclude Metadata volumes, CRS volumes in SVM-DR or MetroCluster have a “MDV_CRS_” prefix
#        - volume `MDV_CRS_.+`
#        # Exclude Metadata volumes, Audit volumes have a “MDV_aud_” prefix
#        - volume `MDV_aud_.+`
#  - ChangeLog

export_options:
  instance_keys:
    - aggr
    - node
    - style
    - svm
    - volume
  instance_labels:
    - clone_parent_snapshot
    - clone_parent_svm
    - clone_parent_volume
    - isEncrypted
    - isHardwareEncrypted
    - is_sis_volume
    - junction_path
    - root_volume
    - snaplock_type
    - snapshot_autodelete
    - snapshot_policy
    - state
    - svm_root
    - type
  1. Restart Harvest

Extend RestPerf volume template

We are going to create a new template and add it to your custom.yaml. In the commands below, Harvest is installed in /opt/harvest. Adjust as needed.

  1. Edit vi /opt/harvest/conf/restperf/custom.yaml

  2. Add below content to this file. If custom.yaml already exists then add the Volume line under the Objects: section.

objects:
  Volume: custom_volume.yaml

  1. Create the custom_volume.yaml file vi /opt/harvest/conf/restperf/9.12.0/custom_volume.yaml

  2. Copy existing /opt/harvest/conf/restperf/9.12.0/volume.yaml template and changed the flag include_constituents: true and save.

name:                     Volume
query:                    api/cluster/counter/tables/volume
object:                   volume

counters:
  - ^^uuid
  - ^name                  => volume
  - ^node.name             => node
  - ^parent_aggregate      => aggr
  - ^svm.name              => svm
  - average_latency        => avg_latency
  - bytes_read             => read_data
  - bytes_written          => write_data
  - nfs.access_latency     => nfs_access_latency
  - nfs.access_ops         => nfs_access_ops
  - nfs.getattr_latency    => nfs_getattr_latency
  - nfs.getattr_ops        => nfs_getattr_ops
  - nfs.lookup_latency     => nfs_lookup_latency
  - nfs.lookup_ops         => nfs_lookup_ops
  - nfs.other_latency      => nfs_other_latency
  - nfs.other_ops          => nfs_other_ops
  - nfs.punch_hole_latency => nfs_punch_hole_latency
  - nfs.punch_hole_ops     => nfs_punch_hole_ops
  - nfs.read_latency       => nfs_read_latency
  - nfs.read_ops           => nfs_read_ops
  - nfs.setattr_latency    => nfs_setattr_latency
  - nfs.setattr_ops        => nfs_setattr_ops
  - nfs.total_ops          => nfs_total_ops
  - nfs.write_latency      => nfs_write_latency
  - nfs.write_ops          => nfs_write_ops
  - other_latency
  - read_latency
  - total_ops
  - total_other_ops        => other_ops
  - total_read_ops         => read_ops
  - total_write_ops        => write_ops
  - write_latency

plugins:
  - Aggregator:
      # plugin will create summary/average for each object
      # any names after the object names will be treated as label names that will be added to instances
      - node
  - Volume:
      include_constituents: true
#  - LabelAgent:
#      # To prevent visibility of transient volumes, uncomment the following lines
#      exclude_regex:
#        # Exclude SnapProtect/CommVault Intellisnap, Clone volumes have a “_CVclone” suffix
#        - volume `.+_CVclone`
#        # Exclude SnapCenter, Clone volumes have a “DDMMYYhhmmss” suffix
#        - volume `.+(0[1-9]|[12][0-9]|3[01])(0[1-9]|1[012])\d\d[0-9]{6}`
#        # Exclude manually created SnapCreator clones, Clone volumes have a “cl_” prefix and a “_YYYYMMDDhhmmss” suffix
#        - volume `cl_.+_(19|20)\d\d(0[1-9]|1[012])( 0[1-9]|[12][0-9]|3[01])[0-9]{6}`
#        # Exclude SnapDrive/SnapManager, Clone volumes have a “sdw_cl_” prefix
#        - volume `sdw_cl_.+`
#        # Exclude Metadata volumes, CRS volumes in SVM-DR or MetroCluster have a “MDV_CRS_” prefix
#        - volume `MDV_CRS_.+`
#        # Exclude Metadata volumes, Audit volumes have a “MDV_aud_” prefix
#        - volume `MDV_aud_.+`

export_options:
  instance_keys:
    - aggr
    - node
    - style
    - svm
    - volume
  1. Restart Harvest

Extend Zapi volume template

We are going to create a new template and add it to your custom.yaml. In the commands below, Harvest is installed in /opt/harvest. Adjust as needed.

  1. Edit vi /opt/harvest/conf/zapi/cdot/9.8.0/custom.yaml

  2. Add below content to this file. If custom.yaml already exists then add the Volume line under the Objects: section.

objects:
  Volume: custom_volume.yaml

  1. Create the custom_volume.yaml file vi /opt/harvest/conf/zapi/cdot/9.8.0/custom_volume.yaml

  2. Add these below lines and save.

plugins:
  - Volume:
      include_constituents: true
  1. Restart Harvest

Extend ZapiPerf volume template

We are going to create a new template and add it to your custom.yaml. In the commands below, Harvest is installed in /opt/harvest. Adjust as needed.

  1. Edit vi /opt/harvest/conf/zapiperf/cdot/9.8.0/custom.yaml

  2. Add below content to this file. If custom.yaml already exists then add the Volume line under the Objects: section.

objects:
  Volume: custom_volume.yaml

  1. Create the custom_volume.yaml file vi /opt/harvest/conf/zapiperf/cdot/9.8.0/custom_volume.yaml

  2. Add these below lines and save.

plugins:
  - Volume:
      include_constituents: true
  1. Restart Harvest

For NABox Installation:

Extend Rest volume template

We are going to create a new template. In the commands below, Harvest is installed in /opt/harvest2-conf. Adjust as needed.

  1. Add file vi /opt/harvest2-conf/conf/rest/custom_volume_flexgroup.yaml

  2. Add below content to this file.

collector:          Rest
objects:
  Volume:
    - custom_volume_flexgroup.yaml

  1. Create the custom_volume_flexgroup.yaml file vi /opt/harvest2-conf/conf/rest/9.12.0/custom_volume_flexgroup.yaml

  2. Copy existing /opt/harvest2-conf/conf/rest/9.12.0/volume.yaml template and changed the flag include_constituents: true and save.

name:                     Volume
query:                    api/private/cli/volume
object:                   volume

counters:
  - ^^volume
  - ^^vserver                                     => svm
  - ^aggr_list                                    => aggr
  - ^clone_parent_name                            => clone_parent_volume
  - ^clone_parent_vserver                         => clone_parent_svm
  - ^clone_volume                                 => is_flexclone
  - ^is_encrypted                                 => isEncrypted
  - ^is_sis_volume                                => is_sis_volume
  - ^junction_path                                => junction_path
  - ^nodes                                        => node
  - ^snaplock_type                                => snaplock_type
  - ^snapshot_policy                              => snapshot_policy
  - ^state                                        => state
  - ^type                                         => type
  - ^volume_style_extended                        => style
  - ^vsroot                                       => svm_root
  - autosize_grow_threshold_percent               => autosize_grow_threshold_percent
  - available                                     => size_available
  - compression_space_saved                       => sis_compress_saved
  - compression_space_saved_percent               => sis_compress_saved_percent
  - dedupe_space_saved                            => sis_dedup_saved
  - dedupe_space_saved_percent                    => sis_dedup_saved_percent
  - expected_available                            => space_expected_available
  - files                                         => inode_files_total
  - files_used                                    => inode_files_used
  - filesystem_size                               => filesystem_size
  - logical_used                                  => space_logical_used
  - logical_used_by_afs                           => space_logical_used_by_afs
  - logical_used_by_snapshots                     => space_logical_used_by_snapshots
  - logical_used_percent                          => space_logical_used_percent
  - max_autosize                                  => autosize_maximum_size
  - overwrite_reserve                             => overwrite_reserve_total
  - overwrite_reserve_used                        => overwrite_reserve_used
  - percent_snapshot_space                        => snapshot_reserve_percent
  - percent_used                                  => size_used_percent
  - physical_used                                 => space_physical_used
  - physical_used_percent                         => space_physical_used_percent
  - sis_space_saved                               => sis_total_saved
  - sis_space_saved_percent                       => sis_total_saved_percent
  - size                                          => size
  - size_available_for_snapshots                  => snapshots_size_available
  - size_used_by_snapshots                        => snapshots_size_used
  - snapshot_count
  - snapshot_reserve_available                    => snapshot_reserve_available
  - snapshot_reserve_size                         => snapshot_reserve_size
  - snapshot_space_used                           => snapshot_reserve_used_percent
  - total                                         => size_total
  - used                                          => size_used
  - filter:
      - is_constituent=*

endpoints:
  - query: api/private/cli/volume/efficiency/stat
    counters:
      - ^^volume
      - ^^vserver                                 => svm
      - num_compress_attempts
      - num_compress_fail
      - filter:
          - privilege_level=diagnostic

  - query: api/private/cli/volume/footprint
    counters:
      - ^^volume
      - ^^vserver                                 => svm
      - volume_blocks_footprint_bin0              => performance_tier_footprint
      - volume_blocks_footprint_bin0_percent      => performance_tier_footprint_percent
      - volume_blocks_footprint_bin1              => capacity_tier_footprint
      - volume_blocks_footprint_bin1_percent      => capacity_tier_footprint_percent


plugins:
  - Volume:
      schedule:
        - data: 15m  # should be multiple of poll duration
      include_constituents: true
  - MetricAgent:
      compute_metric:
        - inode_used_percent PERCENT inode_files_used inode_files_total
        - snapshot_reserve_used SUBTRACT snapshot_reserve_size snapshot_reserve_available
        - overwrite_reserve_available SUBTRACT overwrite_reserve_total overwrite_reserve_used
  - LabelAgent:
      value_to_num:
        - new_status state online online `0`
      replace:
        - svm_root root_volume `false` `No`
        - svm_root root_volume `true` `Yes`
      # To prevent visibility of transient volumes, uncomment the following lines
#      exclude_regex:
#        # Exclude SnapProtect/CommVault Intellisnap, Clone volumes have a “_CVclone” suffix
#        - volume `.+_CVclone`
#        # Exclude SnapCenter, Clone volumes have a “DDMMYYhhmmss” suffix
#        - volume `.+(0[1-9]|[12][0-9]|3[01])(0[1-9]|1[012])\d\d[0-9]{6}`
#        # Exclude manually created SnapCreator clones, Clone volumes have a “cl_” prefix and a “_YYYYMMDDhhmmss” suffix
#        - volume `cl_.+_(19|20)\d\d(0[1-9]|1[012])( 0[1-9]|[12][0-9]|3[01])[0-9]{6}`
#        # Exclude SnapDrive/SnapManager, Clone volumes have a “sdw_cl_” prefix
#        - volume `sdw_cl_.+`
#        # Exclude Metadata volumes, CRS volumes in SVM-DR or MetroCluster have a “MDV_CRS_” prefix
#        - volume `MDV_CRS_.+`
#        # Exclude Metadata volumes, Audit volumes have a “MDV_aud_” prefix
#        - volume `MDV_aud_.+`
#  - ChangeLog

export_options:
  instance_keys:
    - aggr
    - node
    - style
    - svm
    - volume
  instance_labels:
    - clone_parent_snapshot
    - clone_parent_svm
    - clone_parent_volume
    - isEncrypted
    - isHardwareEncrypted
    - is_sis_volume
    - junction_path
    - root_volume
    - snaplock_type
    - snapshot_autodelete
    - snapshot_policy
    - state
    - svm_root
    - type
  1. Restart Harvest via dc restart command.

Extend RestPerf volume template

We are going to create a new template. In the commands below, Harvest is installed in /opt/harvest2-conf. Adjust as needed.

  1. Edit vi /opt/harvest2-conf/conf/restperf/custom_volume_flexgroup.yaml

  2. Add below content to this file.

collector:          RestPerf
objects:
  Volume:
    - custom_volume_flexgroup.yaml

  1. Create the custom_volume_flexgroup.yaml file vi /opt/harvest2-conf/conf/restperf/9.12.0/custom_volume_flexgroup.yaml

  2. Copy existing /opt/harvest2-conf/conf/restperf/9.12.0/volume.yaml template and changed the flag include_constituents: true and save.

name:                     Volume
query:                    api/cluster/counter/tables/volume
object:                   volume

counters:
  - ^^uuid
  - ^name                  => volume
  - ^node.name             => node
  - ^parent_aggregate      => aggr
  - ^svm.name              => svm
  - average_latency        => avg_latency
  - bytes_read             => read_data
  - bytes_written          => write_data
  - nfs.access_latency     => nfs_access_latency
  - nfs.access_ops         => nfs_access_ops
  - nfs.getattr_latency    => nfs_getattr_latency
  - nfs.getattr_ops        => nfs_getattr_ops
  - nfs.lookup_latency     => nfs_lookup_latency
  - nfs.lookup_ops         => nfs_lookup_ops
  - nfs.other_latency      => nfs_other_latency
  - nfs.other_ops          => nfs_other_ops
  - nfs.punch_hole_latency => nfs_punch_hole_latency
  - nfs.punch_hole_ops     => nfs_punch_hole_ops
  - nfs.read_latency       => nfs_read_latency
  - nfs.read_ops           => nfs_read_ops
  - nfs.setattr_latency    => nfs_setattr_latency
  - nfs.setattr_ops        => nfs_setattr_ops
  - nfs.total_ops          => nfs_total_ops
  - nfs.write_latency      => nfs_write_latency
  - nfs.write_ops          => nfs_write_ops
  - other_latency
  - read_latency
  - total_ops
  - total_other_ops        => other_ops
  - total_read_ops         => read_ops
  - total_write_ops        => write_ops
  - write_latency

plugins:
  - Aggregator:
      # plugin will create summary/average for each object
      # any names after the object names will be treated as label names that will be added to instances
      - node
  - Volume:
      include_constituents: true
#  - LabelAgent:
#      # To prevent visibility of transient volumes, uncomment the following lines
#      exclude_regex:
#        # Exclude SnapProtect/CommVault Intellisnap, Clone volumes have a “_CVclone” suffix
#        - volume `.+_CVclone`
#        # Exclude SnapCenter, Clone volumes have a “DDMMYYhhmmss” suffix
#        - volume `.+(0[1-9]|[12][0-9]|3[01])(0[1-9]|1[012])\d\d[0-9]{6}`
#        # Exclude manually created SnapCreator clones, Clone volumes have a “cl_” prefix and a “_YYYYMMDDhhmmss” suffix
#        - volume `cl_.+_(19|20)\d\d(0[1-9]|1[012])( 0[1-9]|[12][0-9]|3[01])[0-9]{6}`
#        # Exclude SnapDrive/SnapManager, Clone volumes have a “sdw_cl_” prefix
#        - volume `sdw_cl_.+`
#        # Exclude Metadata volumes, CRS volumes in SVM-DR or MetroCluster have a “MDV_CRS_” prefix
#        - volume `MDV_CRS_.+`
#        # Exclude Metadata volumes, Audit volumes have a “MDV_aud_” prefix
#        - volume `MDV_aud_.+`

export_options:
  instance_keys:
    - aggr
    - node
    - style
    - svm
    - volume
  1. Restart Harvest via dc restart command.

Extend Zapi volume template

We are going to create a new template. In the commands below, Harvest is installed in /opt/harvest2-conf. Adjust as needed.

  1. Add file vi /opt/harvest2-conf/conf/zapi/custom_volume_flexgroup.yaml

  2. Add below content to this file.

collector:          Zapi
objects:
  Volume:
    - custom_volume_flexgroup.yaml

  1. Create the custom_volume_flexgroup.yaml file vi /opt/harvest2-conf/conf/zapi/cdot/9.8.0/custom_volume_flexgroup.yaml

  2. Add these below lines and save.

plugins:
  - Volume:
      include_constituents: true
  1. Restart Harvest via dc restart command.

Extend ZapiPerf volume template

We are going to create a new template. In the commands below, Harvest is installed in /opt/harvest2-conf. Adjust as needed.

  1. Add file vi /opt/harvest2-conf/conf/zapiperf/custom_volume_flexgroup.yaml

  2. Add below content to this file.

collector:          ZapiPerf
objects:
  Volume:
    - custom_volume_flexgroup.yaml

  1. Create the custom_volume_flexgroup.yaml file vi /opt/harvest2-conf/conf/zapiperf/cdot/9.8.0/custom_volume_flexgroup.yaml

  2. Add these below lines and save.

plugins:
  - Volume:
      include_constituents: true
  1. Restart Harvest via dc restart command.
Clone this wiki locally