Skip to content

RTECO-1411 Add agentplugins tests#3551

Open
udaykb2 wants to merge 11 commits into
masterfrom
RTECO-1411-Agentplugins-tests
Open

RTECO-1411 Add agentplugins tests#3551
udaykb2 wants to merge 11 commits into
masterfrom
RTECO-1411-Agentplugins-tests

Conversation

@udaykb2

@udaykb2 udaykb2 commented Jun 17, 2026

Copy link
Copy Markdown
  • All tests have passed. If this feature is not already covered by the tests, new tests have been added.
  • The pull request is targeting the master branch.
  • The code has been validated to compile successfully by running go vet ./....
  • The code has been formatted properly using go fmt ./....

Agent Plugins Test Coverage

Commands and Scenarios

publish

  • Basic publish: uploads plugin zip to correct path in Artifactory repo
  • --version flag: overrides the version declared in plugin.json
  • Version collision (CI mode): publishing an already-existing version fails with a clear "already exists" error
  • Missing plugin.json: returns a clear error when the manifest is absent
  • Invalid semver: non-semver version string is rejected before any upload attempt
  • Invalid slug: slug containing invalid characters is rejected
  • Missing <path> argument: returns a usage/validation error
  • Non-existent repo: returns a clear error from Artifactory
  • Wrong repo type: publishing to a non-agentplugins repo returns an Artifactory error
  • Prebuilt zip: a plugin that already contains a zip inside zip/ is uploaded as-is without re-zipping
  • Signing key: --signing-key attaches evidence to the artifact
  • Without signing key: publish without --signing-key succeeds; evidence step is skipped (not a failure)
  • Build info (--build-name / --build-number): build info entry is created and retrievable from Artifactory
  • No build info without flags: omitting --build-name/--build-number does not create a build info entry
  • Module override (--module): overrides the default module ID (slug) in the build info entry
  • Build properties on artifact: build-name, build-number, and module properties are stamped on the artifact
  • Build info from env vars: JFROG_BUILD_NAME/JFROG_BUILD_NUMBER are auto-collected even without explicit flags
  • Full build info round-trip: publish plugin → publish build info → retrieve build info from Artifactory

install

  • Latest version: installs latest available version into <installPath>/<slug>/; uses --path to bypass harness resolution
  • Specific version (--version): installs the exact requested version rather than latest
  • Not found: returns a clear not-found error when slug/version doesn't exist
  • --project-dir: installs into the project-relative harness directory
  • --global: installs into the agent's global harness directory (e.g. ~/.claude/plugins/<slug>)
  • --path: installs directly into the given path, bypassing harness lookup
  • Multiple harnesses (--harness a,b): installs the plugin into all specified harness directories
  • Missing slug argument: returns a clear usage error
  • Unknown harness: an unrecognised harness name (with no agent-config.json override) returns a clear error
  • Empty harness: empty or blank harness value is rejected
  • Marketplace resolution: --harness=claude without --version resolves version from claude-marketplace.json; fails if marketplace file absent; --version bypasses marketplace entirely
  • agent-config.json override: custom agent entry under plugins-agents is respected for --global (globalDir), explicit --project-dir, and default cwd (projectDir)
  • Writes plugin-info manifest: after install, a plugin-info.json is written recording slug and installed version
  • Evidence gate CI mode: when evidence is required and absent, install fails unless JFROG_AGENT_PLUGINS_DISABLE_QUIET_FAILURE is set
  • Evidence gate disabled: JFROG_AGENT_PLUGINS_DISABLE_QUIET_FAILURE=true allows install to succeed even without evidence
  • --format=json: install output is parseable JSON rather than human-readable table

update

  • Single slug (--slug): updates a specific plugin to the newest available version
  • --dry-run: shows what would be updated without changing any files
  • --force: re-installs even when the installed version is already up-to-date
  • --all: updates every installed plugin under a given harness
  • --all non-interactive (CI=true): update --all without --quiet proceeds automatically when CI=true instead of blocking on a confirmation prompt
  • --all invalid flag combinations: --all+--slug mutually exclusive; --all+--version mutually exclusive; missing both --slug and --all fails
  • Invalid slug format: rejected with a validation error
  • Plugin never installed: --slug for an uninstalled plugin returns a clear error
  • --format=json (single slug): output is valid JSON
  • --format=json (--all): output is valid JSON for all updated plugins

delete

  • Basic delete: removes the specified version folder from Artifactory
  • Dry run (--dry-run): reports what would be deleted without removing any artifact
  • Only specified version: deleting one version leaves other versions of the same plugin intact
  • Missing slug: plugin that does not exist in the repository returns a clear error
  • Missing --version: omitting --version produces a clear error rather than silently deleting all versions

list

  • Remote (--repo): lists available plugins in an Artifactory repo without error after a publish
  • Local (--harness): lists installed plugins after a local install
  • --check-updates (--harness): flag is accepted and produces no error
  • --format=json with --repo: produces valid JSON output
  • --limit: positive limit value is accepted
  • --sort-by=updated: valid sort field accepted
  • --sort-by=<unknown>: unknown sort field returns an error
  • --sort-order=desc: accepted without error
  • --sort-order=<unknown>: accepted without error (not validated by CLI)
  • --check-updates without --harness: rejected with a clear error (--repo alone is unsupported)

search

  • Basic search: returns matches by agentplugins.name property
  • No matches: empty result set is returned — not treated as an error
  • Empty query: returns a usage error
  • --format=json: produces parseable JSON output

Cross-command / round-trips

  • Round-trip (publish → install → list): installed manifest matches slug and version
  • Round-trip with update (publish v1 + v2 → install v1 → update → assert v2 installed)
  • Round-trip delete then install (publish → install → delete → install → expect not-found)
  • CI pipeline simulation: publish → install → list → update → delete in sequence

Configuration and connectivity

  • Repo from env var (JFROG_AGENT_PLUGINS_REPO): used when --repo is omitted
  • --repo flag overrides env var
  • No repo configured: clear error naming both --repo and the env var
  • --server-id valid: using a configured server ID succeeds
  • --server-id unknown: unrecognised server ID returns a clear error before any network call
  • Artifactory unreachable: install/publish to a bogus URL fails with a network/connection error
  • Proxy (HTTPS_PROXY): install and publish work when routed through a proxy
  • No-proxy (NO_PROXY): proxy is bypassed when Artifactory host is in NO_PROXY
  • Insecure TLS (--insecure-tls): install and publish work against an instance with a self-signed certificate

Checksum

  • Integrity check: published artifact SHA256 matches the locally computed checksum
  • Stored by Artifactory: Artifactory stores a non-empty, trusted SHA256 for the artifact

Misc / validation

  • Unknown flag: passing an unrecognised flag produces a usage error (no panic)
  • Invalid --format: non-"json" format values are treated as table (not an error for list; format error for other commands)

Test Name → Scenarios

Test Name Commands Scenarios Covered
TestAgentPluginsPublish publish Basic publish; zip uploaded to correct path
TestAgentPluginsVersionCollisionCI publish Duplicate publish in CI mode errors with "already exists"
TestAgentPluginsPublishWithVersion publish --version overrides plugin.json version
TestAgentPluginsPublishMissingPluginJson publish Publish without plugin.json returns error
TestAgentPluginsPublishToNonExistentRepo publish Publish to nonexistent repo returns error
TestAgentPluginsPublishInvalidSemver publish Non-semver version rejected before upload
TestAgentPluginsPublishInvalidSlug publish Invalid slug characters rejected
TestAgentPluginsPublishMissingPathArg publish Missing path argument returns usage error
TestAgentPluginsPublishToWrongRepoType publish Publishing to wrong repo type returns Artifactory error
TestAgentPluginsPublishPrebuiltZip publish Pre-built zip inside zip/ uploaded as-is without re-zipping
TestAgentPluginsPublishWithSigningKey publish --signing-key attaches evidence to artifact
TestAgentPluginsPublishWithoutSigningKey publish Omitting --signing-key succeeds; evidence step skipped
TestAgentPluginsPublishWithBuildInfo publish Build info entry created and retrievable
TestAgentPluginsNoBuildInfoWithoutFlags publish No build info created without --build-name/--build-number
TestAgentPluginsPublishBuildNameWithoutNumber publish --build-name without --build-number (and vice versa) returns an error
TestAgentPluginsModuleOverride publish --module overrides default module ID in build info
TestAgentPluginsBuildPropertiesOnArtifact publish Build properties stamped on artifact
TestAgentPluginsBuildInfoFromEnvVars publish Build name/number auto-collected from env vars
TestAgentPluginsBuildPublishRetrievable publish Full build info round-trip: publish → publish build → retrieve
TestAgentPluginsInstallLatest install Install latest version via --path
TestAgentPluginsInstallSpecificVersion install --version installs exact version
TestAgentPluginsInstallNotFound install Missing plugin returns clear not-found error
TestAgentPluginsInstallWithProjectDir install --project-dir installs into project-relative harness dir
TestAgentPluginsInstallGlobal install --global --harness=claude installs into ~/.claude/plugins/<slug>
TestAgentPluginsInstallWithPath install --path installs directly, bypassing harness lookup
TestAgentPluginsInstallPathWithVersion install --path --version=1.0.0 installs the specific version into the given dir
TestAgentPluginsInstallMultipleHarnesses install Comma-separated --harness installs into all specified dirs
TestAgentPluginsInstallMissingSlugArg install Missing slug argument returns usage error
TestAgentPluginsInstallUnknownHarness install Unrecognised harness (no config override) returns clear error
TestAgentPluginsInstallEmptyHarness install Empty/blank harness value is rejected
TestAgentPluginsInstallGlobalProjectDirMutuallyExclusive install --global and --project-dir together returns a clear error
TestAgentPluginsInstallHarnessPathMutuallyExclusive install --harness and --path together returns a clear error
TestAgentPluginsInstallMarketplace install Marketplace resolution: claude resolves via JSON; cursor fails without JSON; cursor passes with --version
TestAgentPluginsInstallAgentConfigOverride install agent-config.json overrides respected for --global, --project-dir, and default cwd
TestAgentPluginsInstallWritesPluginInfoManifest install plugin-info.json written after install recording slug and version
TestAgentPluginsInstallEvidenceGateCI install Missing evidence causes install failure in CI quiet mode
TestAgentPluginsInstallEvidenceGateDisabled install JFROG_AGENT_PLUGINS_DISABLE_QUIET_FAILURE=true bypasses evidence gate
TestAgentPluginsInstallFormatJSON install --format=json produces parseable JSON install output
TestAgentPluginsUpdateSlug update --slug updates specific plugin to newest version
TestAgentPluginsUpdateDryRun update --dry-run reports changes without writing files
TestAgentPluginsUpdateForce update --force re-installs even when already up-to-date
TestAgentPluginsUpdateAll update --all updates every installed plugin for a harness
TestAgentPluginsUpdateAllNothingInstalled update --all with empty harness dir succeeds with "nothing to update"
TestAgentPluginsUpdateAllNonInteractive update --all without --quiet auto-proceeds when CI=true
TestAgentPluginsUpdateFormatJSON update --format=json with --slug and with --all both produce valid JSON
TestAgentPluginsUpdateFlags update Flag validation: missing --slug/--all; mutually exclusive pairs (--all+--slug, --all+--version, --all+--path); invalid slug format; uninstalled slug
TestAgentPluginsDelete delete Basic delete removes specified version; --version always required
TestAgentPluginsDeleteDryRun delete --dry-run reports without deleting artifact
TestAgentPluginsDeleteDryRunMultipleVersions delete --dry-run on multi-version plugin only targets specified version; others untouched
TestAgentPluginsDeleteDryRunNotFound delete --dry-run on a non-existent plugin returns a not-found error
TestAgentPluginsDeleteOnlySpecifiedVersion delete Deleting one version leaves other versions intact
TestAgentPluginsDeleteMissing delete Deleting non-existent plugin returns clear error
TestAgentPluginsDeleteMissingVersion delete Omitting --version returns clear error instead of deleting all
TestAgentPluginsListRemote list --repo lists available plugins without error after publish
TestAgentPluginsListLocal list --harness lists installed plugins after local install
TestAgentPluginsListMultipleHarnesses list --harness=claude,cursor succeeds and produces output for each harness
TestAgentPluginsListCheckUpdates list --check-updates --harness accepted and produces no error
TestAgentPluginsListCheckUpdatesStatus list --check-updates --format=json reports status "behind" when v1 installed and v2 available
TestAgentPluginsListFlags list Flag combinations: --format, --limit, --sort-by, --sort-order valid/invalid; --check-updates without --harness rejected
TestAgentPluginsListGlobalProjectDirMutuallyExclusive list --global and --project-dir together returns a clear error
TestAgentPluginsListLimitHarnessMode list --limit truncates results in --harness mode
TestAgentPluginsListLimitZero list --limit=0 is rejected as invalid
TestAgentPluginsListCheckUpdatesCurrent list --check-updates --format=json reports status "current" when plugin is at latest version
TestAgentPluginsSearch search Basic search returns matches by agentplugins.name property
TestAgentPluginsSearchNoMatches search Empty result set returned — not an error
TestAgentPluginsSearchEmptyQuery search Empty query returns usage error
TestAgentPluginsSearchRepoFromEnvVar search JFROG_AGENT_PLUGINS_REPO env var used when --repo omitted
TestAgentPluginsSearchFormatJSON search --format=json produces parseable JSON search output
TestAgentPluginsChecksumIntegrity publish Published artifact SHA256 matches local checksum
TestAgentPluginsChecksumStoredByArtifactory publish Artifactory stores non-empty trusted SHA256 for artifact
TestAgentPluginsRoundTrip publish, install publish → install; installed manifest matches slug and version
TestAgentPluginsRoundTripWithUpdate publish, install, update publish v1+v2 → install v1 → update → assert v2 installed
TestAgentPluginsRoundTripDeleteThenInstall publish, install, delete publish → install → delete → install again → expect not-found
TestAgentPluginsCIPipeline publish, install Full CI pipeline: publish with build info → bp → install in sequence
TestAgentPluginsRepoFromEnvVar publish JFROG_AGENT_PLUGINS_REPO used when --repo omitted
TestAgentPluginsRepoFlagOverridesEnvVar publish --repo flag overrides env var
TestAgentPluginsNoRepoConfigured publish No repo configured: error names both --repo and env var
TestAgentPluginsServerIDValid publish --server-id with known server ID succeeds
TestAgentPluginsServerIDUnknown publish --server-id with unknown ID errors before any network call
TestAgentPluginsArtifactoryUnreachable install Bogus Artifactory URL fails with network/connection error
TestAgentPluginsWithProxy publish Install and publish work through HTTPS_PROXY
TestAgentPluginsNoProxy publish Proxy bypassed when host is in NO_PROXY
TestAgentPluginsInsecureTLS publish --insecure-tls allows publish against self-signed cert
TestAgentPluginsUnknownFlag publish, install, update, delete, list, search Unrecognised flag returns usage error without panic
TestAgentPluginsInvalidFormatFlag list Non-"json" format treated as table (not an error)

Total: 85 tests

@udaykb2 udaykb2 force-pushed the RTECO-1411-Agentplugins-tests branch from 21c5b58 to 7421e4c Compare June 17, 2026 17:52
@udaykb2 udaykb2 force-pushed the RTECO-1411-Agentplugins-tests branch from d9ef023 to 8c14733 Compare June 18, 2026 04:23
@udaykb2 udaykb2 added the safe to test Approve running integration tests on a pull request label Jun 18, 2026
@udaykb2 udaykb2 changed the title Rteco 1411 agentplugins tests RTECO-1411 Add agentplugins tests Jun 18, 2026
@udaykb2 udaykb2 marked this pull request as ready for review June 18, 2026 04:55
@udaykb2 udaykb2 force-pushed the RTECO-1411-Agentplugins-tests branch from 5ab34f1 to ff67c00 Compare June 18, 2026 04:58
All similar tests are refactored together.
Removed scanario number from comments
@udaykb2 udaykb2 force-pushed the RTECO-1411-Agentplugins-tests branch from ff67c00 to 149ad0d Compare June 18, 2026 05:15
@udaykb2 udaykb2 requested review from a team, agrasth, bhanurp, fluxxBot, itsmeleela and naveenku-jfrog and removed request for a team June 18, 2026 08:45
@github-actions

Copy link
Copy Markdown
Contributor

👍 Frogbot scanned this pull request and did not find any new security issues.


@udaykb2 udaykb2 requested a review from reshmifrog June 18, 2026 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to test Approve running integration tests on a pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant