Skip to content

[Telemetry] Add support dynamic min/max node count metrics#5961

Open
kadupoornima wants to merge 21 commits into
GoogleCloudPlatform:developfrom
kadupoornima:feat/dynamic-node-metrics
Open

[Telemetry] Add support dynamic min/max node count metrics#5961
kadupoornima wants to merge 21 commits into
GoogleCloudPlatform:developfrom
kadupoornima:feat/dynamic-node-metrics

Conversation

@kadupoornima

Copy link
Copy Markdown
Contributor

Overview
This PR introduces robust schema mappings to support capturing and analyzing bounded autoscaling and dynamic machine limits through two newly added telemetry metrics: DYNAMIC_MIN_NODE_COUNTS and DYNAMIC_MAX_NODE_COUNTS. Alongside these metrics addition, it resolves a systemic bug affecting metric inflation regarding zonal application rules.

Key Changes & Implementations:

  • Dynamic Metric Collection Lists: Added tracking for dynamicMinNodeCountSettings and dynamicMaxNodeCountSettings to support properties inclusive of dynamic bounding modules across the codebase. Features evaluated include: GKE dimensions (autoscaling_max_node_count, autoscaling_total_max_nodes), Slurm nodeset ranges (node_count_dynamic_max), GKE cluster system pool dot-notated objects (system_node_pool_node_count.total_min_nodes), and HTCondor execution points (max_size).
  • Fixed Telemetry Zonal Defect: Remediated a logic bug in getTopLevelNodeCount & extractTargetNodeCount where global metrics such as autoscaling_total_max_nodes were being inaccurately multiplied by getZonalMultiplier counts.
  • Struct Refactoring: Refactored evaluation blocks relying on hardcoded parameter checks into a uniform, centralized slice definition (zonalNodeCountSettings) mapped across functions.
  • Testing Coverage: Engineered and applied explicitly mapped scenarios covering global scale-bounded variables (combined directly with implicit defaults) to assert that global variables pass untampered while strictly enforcing zonal operations exclusively for zonal variables.

Change-Id: I6a9d7ef958bc898930540ee532d5423b376370e7
.
Change-Id: I4a883566aa9bdd3a7001b3b9044f8ac6621f6b15
…educe cyclomatic complexity

Change-Id: I07ce053ad98d5047b25946cd8f07f41969b0bffb
…valuation null errors

Change-Id: Ic5b55da2ef310fffb6d41ad596b6516db0370ffb
…N/MAX_NODE_COUNTS

Change-Id: I8a99352b0ba7ff346765c0c734c6090cabf8cc42
Change-Id: I0c0b5119b8e21e8cbbbc65904115e5baae70a5b1
.
Change-Id: I5a0f508a3d8f6c8af89a2c81164129c67b191aae
@kadupoornima
kadupoornima requested a review from a team as a code owner July 16, 2026 09:37
@kadupoornima kadupoornima added the release-improvements Added to release notes under the "Improvements" heading. label Jul 16, 2026
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the telemetry collection system by introducing support for dynamic node count metrics. It enables the tracking of autoscaling and dynamic machine limits across various cloud modules while simultaneously fixing a bug that caused incorrect metric inflation for global node count settings in zonal environments.

Highlights

  • New Telemetry Metrics: Added support for tracking dynamic minimum and maximum node counts via new telemetry metrics: DYNAMIC_MIN_NODE_COUNTS and DYNAMIC_MAX_NODE_COUNTS.
  • Telemetry Bug Fix: Resolved a logic error in node count extraction where global metrics were incorrectly multiplied by zonal multipliers.
  • Refactoring: Centralized node count evaluation logic into a new zonalNodeCountSettings slice to improve maintainability and consistency across modules.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@kadupoornima
kadupoornima enabled auto-merge (squash) July 16, 2026 09:38

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces telemetry collection for dynamic minimum and maximum node counts in the Cluster Toolkit. It adds functions to parse and aggregate dynamic node settings from blueprints, including support for nested keys using dot notation, and includes comprehensive unit tests. Feedback on the changes highlights three key improvements: safely unmarking cty.Value during nested property traversal to prevent panics, simplifying the loop in getDynamicNodeCounts by directly iterating over the slice from config.GetAllBpModules, and adding a defensive check with Has("static_node_count") in getModuleDynamicNodeCounts to avoid potential runtime panics.

Comment thread pkg/telemetry/collector_util.go
Comment thread pkg/telemetry/collector.go Outdated
Comment thread pkg/telemetry/collector_util.go
kadupoornima and others added 6 commits July 16, 2026 09:41
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
.
Change-Id: I5ccf458852b20c37d5170d10ba9bfad745e5be6d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-improvements Added to release notes under the "Improvements" heading.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants