GoogleCloudPlatform · holtskinner · Jul 15, 2026 · Jul 13, 2026
@@ -95,6 +95,8 @@ The job's service account requires the following:
 - `roles/cloudidentity.groups.viewer` — list group members and verify membership
 - `roles/secretmanager.secretAccessor` — read the mounted configuration secret
 - `roles/billing.viewer` — read the purchased Gemini Enterprise subscription configurations
+- `roles/run.builder` - (Optional) When using Cloud Build to build the artifacts
+- `roles/storage.objectUser` - (Optional) When using Cloud Build, to store the built artifact
 
 **OAuth scopes:**
 - `https://www.googleapis.com/auth/cloud-platform`
@@ -104,6 +106,11 @@ The job's service account requires the following:
 - Discovery Engine API (GCP)
 - Resource Manager API (GCP)
 - Admin SDK API (Cloud Identity / Workspace)
+- Cloud Run Admin API (Optional)
+- Cloud Build API (Optional)
+- Compute Engine API (Optional)
+- Secret Manager API (Optional)
+- Cloud Scheduler API (Optional)
 
 The job uses **Application Default Credentials**. No service account key files are required.
 
@@ -150,7 +157,8 @@ gcloud run jobs deploy [name_of_job] \
   --source . \
   --region [desired_cloud_run_region] \
   --update-secrets=/run/secrets/entitlements.json=[name_of_secret_in_secret_manager]:latest \
-  --set-env-vars JOB_TYPE=[joiner_or_garbage_collection],DRY_RUN=false
+  --set-env-vars JOB_TYPE=[joiner_or_garbage_collection],DRY_RUN=false \
+  --service-account [desired_service_account_email_address]
 ```
 
 ## Job output
@@ -161,12 +169,14 @@ On completion the job exits `0` (success) or `1` (failure). Results are emitted
 {"time":"...","level":"INFO","workflow":"joiner","task_index":0,"msg":"joiner workflow complete","duration_ms":4821,"licenses_granted":42,"licenses_soft_failed":0,"groups_processed":5,"dry_run":false}
 ```
 
-If the license pool for a SKU is exhausted mid-run, a warning is emitted and the job continues:
+If the license pool for a SKU is exhausted mid-run, a warning is emitted per exhausted pool and the job continues:
 
 ```json
 {"time":"...","level":"WARN","workflow":"joiner","task_index":0,"msg":"license pool exhausted, soft-failing remaining users","project_id":"customer-project-alpha","license_config_path":"projects/123/locations/global/licenseConfigs/ent-config","available":3,"soft_failed":17}
 ```
 
+If the billing account has multiple active subscriptions for the same SKU, users who could not be seated in one pool are automatically carried forward to the next. A warning fires for each exhausted pool, but the `licenses_soft_failed` count in the final summary reflects only users who remained unseated after all pools for that SKU were tried.
+
 ```json
 {"time":"...","level":"INFO","workflow":"garbage_collection","task_index":0,"msg":"garbage collection workflow complete","duration_ms":9134,"licenses_revoked":12,"users_evaluated":500,"dry_run":false}
 ```

@@ -24,7 +24,6 @@ import (
 	"os"
 	"slices"
 
-	discoveryenginesdk "cloud.google.com/go/discoveryengine/apiv1"
 	admin "google.golang.org/api/admin/directory/v1"
 	cloudresourcemanager "google.golang.org/api/cloudresourcemanager/v3"
 	"google.golang.org/api/option"
@@ -95,20 +94,14 @@ func main() {
 		os.Exit(1)
 	}
 
-	deClient, err := discoveryenginesdk.NewUserLicenseClient(ctx)
-	if err != nil {
-		slog.Error("failed to create discovery engine client", slog.Any("error", err))
-		os.Exit(1)
-	}
-
 	crmSvc, err := cloudresourcemanager.NewService(ctx, option.WithScopes(cloudresourcemanager.CloudPlatformReadOnlyScope))
 	if err != nil {
 		slog.Error("failed to create resource manager client", slog.Any("error", err))
 		os.Exit(1)
 	}
 
 	idpAdapter := cloudidentity.New(adminSvc)
-	geminiAdapter := discoveryengine.New(deClient)
+	geminiAdapter := discoveryengine.New()
 	rmAdapter := resourcemanager.New(crmSvc)
 
 	// Step 7: run the workflow determined by JobType.

@@ -24,7 +24,7 @@

 ## **3. Data Storage & Configuration**

 The utility uses a JSON-based configuration model which is persisted to GCP Secret Manager.

 ### **3.1. Entitlement Configuration (Secret Manager)**

@@ -66,8 +66,8 @@

 | Field | Type | Required | Description |
 | ----- | ----- | ----- | ----- |
 | `billing_account_id` | string | Yes | The GCP billing account ID associated with the managed projects. |
 | `projects` | object | Yes | Map of GCP project ID → list of entitlement entries. |
 | `settings.staleness_threshold_days` | integer | No | Users inactive beyond this many days are revoked. `0` (default) disables the staleness check. |

 **Per-project entry fields:**
@@ -119,12 +119,16 @@
   3. For each member that is a `User`, it calls the **Discovery Engine API** (`batchUpdateUserLicenses`) to assign the license. This operation is idempotent.
 
 * **License Pool Exhaustion Handling:**
-  If a `batchUpdateUserLicenses` call fails because the license pool for a SKU is exhausted, the job does **not** treat this as a fatal error. Instead it:
+  A billing account may have multiple active subscriptions for the same SKU. At startup, the joiner fetches all `BillingAccountLicenseConfig` records and builds an index keyed by `(SKU, ProjectNumber, Location)`. Each key maps to an ordered slice of `LicenseConfigEntry` values — one per distinct subscription pool — so that all seat capacity is visible and no pool is silently discarded.
+
+  If a `batchUpdateUserLicenses` call fails because a pool is exhausted, the job does **not** treat this as a fatal error. Instead it:
   1. Calls `licenseConfigsUsageStats.list` for the affected project to retrieve the current `usedLicenseCount` for the relevant `licenseConfig`.
-  2. Computes `available = allocatedCount - usedLicenseCount` (the allocated count is derived from the billing account `licenseConfigDistributions` map fetched at startup).
+  2. Computes `available = allocatedCount - usedLicenseCount` (the allocated count is from the `licenseConfigDistributions` map fetched at startup).
   3. Retries the batch trimmed to `available` items, granting as many licenses as possible.
-  4. Soft-fails the remaining users: logs a `WARN` entry with the count of users who could not be assigned a license, then continues to the next batch or project. The job exits `0`.
-  The `licenses_soft_failed` count is included in the final summary log and in the `SyncAddResponse`.
+  4. Carries the remaining (ungranted) users forward to the next subscription pool for the same SKU, if one exists, and repeats steps 1–3.
+  5. After all pools for a SKU are exhausted, soft-fails any users still unseated: logs a `WARN` entry per exhausted pool and continues to the next batch or project. The job exits `0`.
+
+  Each user appears in at most one successful `batchUpdateUserLicenses` call — users do not advance to the next pool if they were already granted a seat. The `licenses_soft_failed` count in the final summary reflects only users who could not be seated across all pools for their SKU.
 
 ### **5.2. Workflow 2: Bulk "Garbage Collection" Job (Scheduled: 6hr)**
 
@@ -136,7 +140,7 @@
   1. Iterates through all licensed users via the **Discovery Engine API** (`listUserLicenses`).  
   2. For each user, the job evaluates two deprovisioning conditions:
       - **Staleness**: If `staleness_threshold_days` is greater than `0`, checks whether the user's staleness reference date is more than `X` days ago. The reference date is chosen as follows: if `lastLoginTime` is set, it is used; if `lastLoginTime` is absent (the user has never logged in), `createTime` (the license assignment timestamp) is used instead, so that a recently provisioned account is not immediately revoked before the user has had a chance to sign in. If both timestamps are absent, the user is treated as immediately stale. Both timestamps are retrieved from the `UserLicense` object provided by the **Discovery Engine API**. Setting `staleness_threshold_days` to `0` (or omitting it) disables this check entirely — only entitlement is evaluated.
-      - **Entitlement**: Calls the **Cloud Identity Admin API's** `members.hasMember` method to confirm if the user is still a member of any entitled group.
+      - **Entitlement**: Calls the **Cloud Identity Admin API's** `members.hasMember` method to confirm if the user is still a member of any entitled group. If the API returns HTTP 400 (invalid member key — e.g. a typo'd email entered manually via the Discovery Engine UI), the user is skipped with a structured `WARN` log entry and the job continues; no revocation is performed for that user.
   3. If **either** condition is met (the user is stale OR no longer entitled), the utility calls the **Discovery Engine API** (`batchUpdateUserLicenses`) to revoke the license.
 
 ### **5.3. License Precedence Logic**
@@ -185,6 +189,7 @@
     *   **Request Batching:** The utility MUST utilize the `batchUpdateUserLicenses` endpoint to consolidate up to 100 license modifications (assignments or revocations) per API request, minimizing round-trip latency and quota consumption.
     *   **Backoff:** Exponential backoff will be applied to all 429 and 5xx errors using a standard retry library.
     *   **License Pool Exhaustion:** License pool exhaustion (`ErrLicensesExhausted`) is treated as a soft failure distinct from rate limiting (`ErrAPIRateLimited`). Exhaustion triggers a `licenseConfigsUsageStats.list` lookup, a trimmed retry for remaining available seats, and a structured warning log — it does not abort the job.
+    *   **Invalid Member Key (GC only):** If `members.hasMember` returns HTTP 400 (`ErrInvalidMemberKey`) for a licensed user — indicating the stored email is malformed or otherwise unresolvable by the Admin API — the user is skipped with a `WARN` log entry. No revocation is issued and the job continues. This guards against typo'd emails entered via the Discovery Engine UI, which lacks input validation.
 * **Structured Logging:**
     *   **Format:** All logs MUST be emitted in JSON format to stdout for automatic ingestion by Cloud Logging.
     *   **Library:** Use Go's standard `log/slog` library.

@@ -164,6 +164,8 @@ func mapHTTPError(err error) error {
 	}
 
 	switch {
+	case apiErr.Code == 400:
+		return fmt.Errorf("%w: %w", models.ErrInvalidMemberKey, apiErr)
 	case apiErr.Code == 429:
 		return fmt.Errorf("%w: %w", models.ErrAPIRateLimited, apiErr)
 	case apiErr.Code == 404:

@@ -357,3 +357,23 @@ func TestHasMember_ServerError(t *testing.T) {
 	mockSvc.AssertExpectations(t)
 	hasMemberCall.AssertExpectations(t)
 }
+
+func TestHasMember_InvalidMemberKey(t *testing.T) {
+	hasMemberCall := new(MockMemberHasMemberCall)
+	hasMemberCall.On("Do").Return((*admin.MembersHasMember)(nil), makeAPIError(400))
+
+	mockSvc := new(MockMembersService)
+	mockSvc.On("HasMember", "group@example.com", "bad@@example.com").Return(hasMemberCall)
+
+	adapter := newWithMembers(mockSvc)
+	_, err := adapter.HasMember(context.Background(), "group@example.com", "bad@@example.com")
+
+	require.Error(t, err)
+	assert.True(t, errors.Is(err, models.ErrInvalidMemberKey),
+		"error chain missing ErrInvalidMemberKey; err = %v", err)
+	assert.True(t, errors.Is(err, models.ErrMembershipCheckFailed),
+		"error chain missing ErrMembershipCheckFailed; err = %v", err)
+
+	mockSvc.AssertExpectations(t)
+	hasMemberCall.AssertExpectations(t)
+}