Skip to content

Commit 0bab2ee

Browse files
RushabhBhansaliRushabhBhansali
authored
Enable GoCliDetector by default. Using env variable 'DisableGoCliDetector=true" to manually disable GoCliDetector. (#113)
Previously, the Go-Detector by default scanned the manifest and generated components. We were using EnableGoCliScan env. variable to activate the Go Cli Detector. With this change, the use of EnableGoCliScan is removed. The Go detector by default uses Cli scan. To manually override this behavior, new env. variable DisableGoCliScan is introduced.
1 parent 7bf1182 commit 0bab2ee

7 files changed

Lines changed: 113 additions & 39 deletions

File tree

‎docs/detectors/go.md‎

Lines changed: 17 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -6,19 +6,9 @@ Go detection depends on the following to successfully run:
66

77
- `go.mod` or `go.sum` files
88

9-
## Detection strategy
9+
## Default Detection strategy
1010

11-
Go detection is performed by parsing any `go.mod` or `go.sum` found under the scan directory.
12-
13-
Only root dependency information is generated instead of full graph.
14-
I.e. tags the top level component or explicit dependency a given transitive dependency was brought by.
15-
Given a dependency tree A -> B -> C, C's root dependency is A.
16-
17-
### Improved detection accuracy via opt-in
18-
19-
**To enable improved detection accuracy, create an environment variable named `EnableGoCliScan` with any value.**
20-
21-
Improved go detection depends on the following to successfully run:
11+
Go detection depends on the following to successfully run:
2212

2313
- Go v1.11+.
2414

@@ -27,23 +17,30 @@ If no Go v1.11+ is present, fallback detection strategy is performed.
2717

2818
Go detection is performed by parsing output from executing [go list -m -json all](1). To generate the graph, the command [go mod graph](2) is executed, this only adds edges between the components that were already registered by `go list`.
2919

30-
As we validate this opt-in behavior, we will eventually graduate it to the default detection strategy.
20+
## Fallback Detection strategy
21+
Go detection is performed by parsing any `go.mod` or `go.sum` found under the scan directory.
22+
23+
Only root dependency information is generated instead of full graph.
24+
I.e. tags the top level component or explicit dependency a given transitive dependency was brought by.
25+
Given a dependency tree A -> B -> C, C's root dependency is A.
26+
27+
**To force fallback detection strategy, create set environment variable `DisableGoCliScan=true`.**
3128

3229
## Known limitations
30+
- If the default strategy is used and go modules are not present in the system before the detector is executed, the go cli will fetch all modules to generate the dependency graph. This will incur additional detector time execution.
3331

34-
Dev dependency tagging is not supported.
32+
- Dev dependency tagging is not supported.
3533

36-
Go detection will fallback if no Go v1.11+ is present.
34+
- Go detection will fallback if no Go v1.11+ is present.
3735

38-
Due to the nature of `go.sum` containing references for all dependencies, including historical, no-longer-needed dependencies; the fallback strategy can result in over detection.
39-
Executing [go mod tidy](https://go.dev/ref/mod#go-mod-tidy) before detection via fallback is encouraged.
36+
- Due to the nature of `go.sum` containing references for all dependencies, including historical, no-longer-needed dependencies; the fallback strategy can result in over detection.
37+
Executing [go mod tidy](https://go.dev/ref/mod#go-mod-tidy) before detection via default strategy is encouraged.
4038

41-
Some legacy dependencies may report stale transitive dependencies in their manifests, in this case you can remove them safely from your binaries by using [exclude directive](https://go.dev/doc/modules/gomod-ref#exclude).
39+
- Some legacy dependencies may report stale transitive dependencies in their manifests, in this case you can remove them safely from your binaries by using [exclude directive](https://go.dev/doc/modules/gomod-ref#exclude).
4240

4341
## Environment Variables
4442

45-
If the environment variable `EnableGoCliScan` is set, to any value, the Go detector uses [`go list -m -json all`][1] to discover Go dependencies.
46-
If the environment variable is not present, we fall back to parsing `go.mod` and `go.sum` ourselves.
43+
If the environment variable `DisableGoCliScan` is set to `true`, the Go detector parses `go.mod` and `go.sum` to discover dependencies. otherwise, it executes default strategy.
4744

4845
[1]: https://go.dev/ref/mod#go-list-m
4946
[2]: https://go.dev/ref/mod#go-mod-graph

‎docs/environment-variables.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,10 @@
22

33
Environment variables are sometimes used to control experimental features or advanced options
44

5-
## `EnableGoCliScan`
5+
## `DisableGoCliScan`
66

7-
If the environment variable `EnableGoCliScan` is set, to any value, the Go detector uses [`go mod graph`][1] to discover Go dependencies.
8-
If the environment variable is not set, we fall back to parsing `go.mod` and `go.sum` ourselves.
7+
If the environment variable `DisableGoCliScan` is set to "true", we fall back to parsing `go.mod` and `go.sum` ourselves.
8+
Otherwise, the Go detector uses go-cli command: `go list -m all` to discover Go dependencies.
99

1010
## `PyPiMaxCacheEntries`
1111

‎src/Microsoft.ComponentDetection.Common/EnvironmentVariableService.cs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,5 +24,11 @@ public string GetEnvironmentVariable(string name)
2424

2525
return caseInsensitiveName != null ? Environment.GetEnvironmentVariable(caseInsensitiveName) : null;
2626
}
27+
28+
public bool IsEnvironmentVariableValueTrue(string name)
29+
{
30+
_ = bool.TryParse(GetEnvironmentVariable(name), out bool result);
31+
return result;
32+
}
2733
}
2834
}

‎src/Microsoft.ComponentDetection.Contracts/IEnvironmentVariableService.cs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,5 +5,7 @@ public interface IEnvironmentVariableService
55
bool DoesEnvironmentVariableExist(string name);
66

77
string GetEnvironmentVariable(string name);
8+
9+
bool IsEnvironmentVariableValueTrue(string name);
810
}
911
}

‎src/Microsoft.ComponentDetection.Detectors/go/GoComponentDetector.cs‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
using System.Composition;
44
using System.IO;
55
using System.Linq;
6+
using System.Security.Cryptography.X509Certificates;
67
using System.Text.RegularExpressions;
78
using System.Threading.Tasks;
89
using Microsoft.ComponentDetection.Common.Telemetry.Records;
@@ -34,7 +35,7 @@ public class GoComponentDetector : FileComponentDetector
3435

3536
public override IEnumerable<ComponentType> SupportedComponentTypes { get; } = new[] { ComponentType.Go };
3637

37-
public override int Version => 5;
38+
public override int Version => 6;
3839

3940
private HashSet<string> projectRoots = new HashSet<string>();
4041

@@ -52,11 +53,15 @@ protected override async Task OnFileFound(ProcessRequest processRequest, IDictio
5253
var wasGoCliScanSuccessful = false;
5354
try
5455
{
55-
if (IsGoCliManuallyEnabled())
56+
if (!IsGoCliManuallyDisabled())
5657
{
57-
Logger.LogInfo("Go cli scan was manually enabled");
5858
wasGoCliScanSuccessful = await UseGoCliToScan(file.Location, singleFileComponentRecorder);
5959
}
60+
else
61+
{
62+
Logger.LogInfo("Go cli scan was manually disabled, fallback strategy performed." +
63+
" More info: https://github.com/microsoft/component-detection/blob/main/docs/detectors/go.md#fallback-detection-strategy");
64+
}
6065
}
6166
catch (Exception ex)
6267
{
@@ -113,6 +118,9 @@ private async Task<bool> UseGoCliToScan(string location, ISingleFileComponentRec
113118
return false;
114119
}
115120

121+
Logger.LogInfo("Go CLI was found in system and will be used to generate dependency graph. " +
122+
"Detection time may be improved by activating fallback strategy (https://github.com/microsoft/component-detection/blob/main/docs/detectors/go.md#fallback-detection-strategy). " +
123+
"But, it will introduce noise into the detected components.");
116124
var goDependenciesProcess = await CommandLineInvocationService.ExecuteCommand("go", null, workingDirectory: projectRootDirectory, new[] { "list", "-m", "-json", "all" });
117125
if (goDependenciesProcess.ExitCode != 0)
118126
{
@@ -307,9 +315,9 @@ private bool TryCreateGoComponentFromRelationshipPart(string relationship, out G
307315
return true;
308316
}
309317

310-
private bool IsGoCliManuallyEnabled()
318+
private bool IsGoCliManuallyDisabled()
311319
{
312-
return EnvVarService.DoesEnvironmentVariableExist("EnableGoCliScan");
320+
return EnvVarService.IsEnvironmentVariableValueTrue("DisableGoCliScan");
313321
}
314322

315323
private class GoBuildModule

‎test/Microsoft.ComponentDetection.Common.Tests/EnvironmentVariableServiceTests.cs‎

Lines changed: 65 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,12 @@ namespace Microsoft.ComponentDetection.Common.Tests
99
public class EnvironmentVariableServiceTests
1010
{
1111
public const string MyEnvVar = nameof(MyEnvVar);
12+
private EnvironmentVariableService testSubject;
1213

1314
[TestInitialize]
1415
public void TestInitialize()
1516
{
17+
testSubject = new EnvironmentVariableService();
1618
Environment.SetEnvironmentVariable(EnvironmentVariableServiceTests.MyEnvVar, "true");
1719
}
1820

@@ -23,15 +25,74 @@ public void TestCleanup()
2325
}
2426

2527
[TestMethod]
26-
public void EnvironmentVariableService_ChecksAreCaseInsensitive()
27-
{
28-
var testSubject = new EnvironmentVariableService();
29-
28+
public void DoesEnvironmentVariableExist_ChecksAreCaseInsensitive()
29+
{
3030
Assert.IsFalse(testSubject.DoesEnvironmentVariableExist("THIS_ENVIRONMENT_VARIABLE_DOES_NOT_EXIST"));
3131

3232
Assert.IsTrue(testSubject.DoesEnvironmentVariableExist(MyEnvVar));
3333
Assert.IsTrue(testSubject.DoesEnvironmentVariableExist(MyEnvVar.ToLower()));
3434
Assert.IsTrue(testSubject.DoesEnvironmentVariableExist(MyEnvVar.ToUpper()));
3535
}
36+
37+
[TestMethod]
38+
public void GetEnvironmentVariable_returnNullIfVariableDoesNotExist()
39+
{
40+
Assert.IsNull(testSubject.GetEnvironmentVariable("NonExistentVar"));
41+
}
42+
43+
[TestMethod]
44+
public void GetEnvironmentVariable_returnCorrectValue()
45+
{
46+
string envVariableKey = nameof(envVariableKey);
47+
string envVariableValue = nameof(envVariableValue);
48+
Environment.SetEnvironmentVariable(envVariableKey, envVariableValue);
49+
var result = testSubject.GetEnvironmentVariable(envVariableKey);
50+
Assert.IsNotNull(result);
51+
Assert.AreEqual(envVariableValue, result);
52+
Environment.SetEnvironmentVariable(envVariableKey, null);
53+
}
54+
55+
[TestMethod]
56+
public void IsEnvironmentVariableValueTrue_returnsTrueForValidKey_caseInsensitive()
57+
{
58+
string envVariableKey1 = nameof(envVariableKey1);
59+
string envVariableKey2 = nameof(envVariableKey2);
60+
Environment.SetEnvironmentVariable(envVariableKey1, "True");
61+
Environment.SetEnvironmentVariable(envVariableKey2, "tRuE");
62+
var result1 = testSubject.IsEnvironmentVariableValueTrue(envVariableKey1);
63+
var result2 = testSubject.IsEnvironmentVariableValueTrue(envVariableKey1);
64+
Assert.IsTrue(result1);
65+
Assert.IsTrue(result2);
66+
Environment.SetEnvironmentVariable(envVariableKey1, null);
67+
Environment.SetEnvironmentVariable(envVariableKey2, null);
68+
}
69+
70+
[TestMethod]
71+
public void IsEnvironmentVariableValueTrue_returnsFalseForValidKey_caseInsensitive()
72+
{
73+
string envVariableKey1 = nameof(envVariableKey1);
74+
string envVariableKey2 = nameof(envVariableKey2);
75+
Environment.SetEnvironmentVariable(envVariableKey1, "False");
76+
Environment.SetEnvironmentVariable(envVariableKey2, "fAlSe");
77+
var result1 = testSubject.IsEnvironmentVariableValueTrue(envVariableKey1);
78+
var result2 = testSubject.IsEnvironmentVariableValueTrue(envVariableKey1);
79+
Assert.IsFalse(result1);
80+
Assert.IsFalse(result2);
81+
Environment.SetEnvironmentVariable(envVariableKey1, null);
82+
Environment.SetEnvironmentVariable(envVariableKey2, null);
83+
}
84+
85+
[TestMethod]
86+
public void IsEnvironmentVariableValueTrue_returnsFalseForInvalidAndNull()
87+
{
88+
string envVariableKey1 = nameof(envVariableKey1);
89+
string nonExistentKey = nameof(nonExistentKey);
90+
Environment.SetEnvironmentVariable(envVariableKey1, "notABoolean");
91+
var result1 = testSubject.IsEnvironmentVariableValueTrue(envVariableKey1);
92+
var result2 = testSubject.IsEnvironmentVariableValueTrue(nonExistentKey);
93+
Assert.IsFalse(result1);
94+
Assert.IsFalse(result2);
95+
Environment.SetEnvironmentVariable(envVariableKey1, null);
96+
}
3697
}
3798
}

‎test/Microsoft.ComponentDetection.Detectors.Tests/GoComponentDetectorTests.cs‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ public void TestInitialize()
3333

3434
var loggerMock = new Mock<ILogger>();
3535

36-
envVarService.Setup(x => x.DoesEnvironmentVariableExist("EnableGoCliScan")).Returns(false);
36+
envVarService.Setup(x => x.IsEnvironmentVariableValueTrue("DisableGoCliScan")).Returns(true);
3737

3838
var detector = new GoComponentDetector
3939
{
@@ -261,7 +261,7 @@ public async Task TestGoDetector_GoCommandNotFound()
261261
commandLineMock.Setup(x => x.CanCommandBeLocated("go", null, It.IsAny<DirectoryInfo>(), It.IsAny<string[]>()))
262262
.ReturnsAsync(false);
263263

264-
envVarService.Setup(x => x.DoesEnvironmentVariableExist("EnableGoCliScan")).Returns(true);
264+
envVarService.Setup(x => x.IsEnvironmentVariableValueTrue("DisableGoCliScan")).Returns(false);
265265

266266
await TestGoSumDetectorWithValidFile_ReturnsSuccessfully();
267267
}
@@ -272,7 +272,7 @@ public async Task TestGoDetector_GoCommandThrows()
272272
commandLineMock.Setup(x => x.CanCommandBeLocated("go", null, It.IsAny<DirectoryInfo>(), It.IsAny<string[]>()))
273273
.ReturnsAsync(() => throw new Exception("Some horrible error occured"));
274274

275-
envVarService.Setup(x => x.DoesEnvironmentVariableExist("EnableGoCliScan")).Returns(true);
275+
envVarService.Setup(x => x.IsEnvironmentVariableValueTrue("DisableGoCliScan")).Returns(false);
276276

277277
await TestGoSumDetectorWithValidFile_ReturnsSuccessfully();
278278
}
@@ -289,7 +289,7 @@ public async Task TestGoDetector_GoGraphCommandFails()
289289
ExitCode = 1,
290290
});
291291

292-
envVarService.Setup(x => x.DoesEnvironmentVariableExist("EnableGoCliScan")).Returns(true);
292+
envVarService.Setup(x => x.IsEnvironmentVariableValueTrue("DisableGoCliScan")).Returns(false);
293293

294294
await TestGoSumDetectorWithValidFile_ReturnsSuccessfully();
295295
}
@@ -303,7 +303,7 @@ public async Task TestGoDetector_GoGraphCommandThrows()
303303
commandLineMock.Setup(x => x.ExecuteCommand("go mod graph", null, It.IsAny<DirectoryInfo>(), It.IsAny<string>()))
304304
.ReturnsAsync(() => throw new Exception("Some horrible error occured"));
305305

306-
envVarService.Setup(x => x.DoesEnvironmentVariableExist("EnableGoCliScan")).Returns(true);
306+
envVarService.Setup(x => x.IsEnvironmentVariableValueTrue("DisableGoCliScan")).Returns(false);
307307

308308
await TestGoSumDetectorWithValidFile_ReturnsSuccessfully();
309309
}
@@ -359,7 +359,7 @@ public async Task TestGoDetector_GoGraphHappyPath()
359359
StdOut = goGraph,
360360
});
361361

362-
envVarService.Setup(x => x.DoesEnvironmentVariableExist("EnableGoCliScan")).Returns(true);
362+
envVarService.Setup(x => x.IsEnvironmentVariableValueTrue("DisableGoCliScan")).Returns(false);
363363

364364
var (scanResult, componentRecorder) = await detectorTestUtility
365365
.WithFile("go.mod", string.Empty)
@@ -421,7 +421,7 @@ public async Task TestGoDetector_GoGraphCyclicDependencies()
421421
StdOut = goGraph,
422422
});
423423

424-
envVarService.Setup(x => x.DoesEnvironmentVariableExist("EnableGoCliScan")).Returns(true);
424+
envVarService.Setup(x => x.IsEnvironmentVariableValueTrue("DisableGoCliScan")).Returns(false);
425425

426426
var (scanResult, componentRecorder) = await detectorTestUtility
427427
.WithFile("go.mod", string.Empty)

0 commit comments

Comments
 (0)