Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Repository	Step Name	Console log	Pull Request
1455160	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6370/merge to s	Log	dotnet/arcade-services#6370
1454373	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16990/merge to s	Log	#16990
1454302	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
1454285	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54122/merge to s	Log	dotnet/sdk#54122
1453193	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54630/merge to s	Log	dotnet/sdk#54630
1452646	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129068/merge to s	Log	dotnet/runtime#129068
1451912	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129051/merge to s	Log	dotnet/runtime#129051
1451780	dotnet/maui	Checkout dotnet/maui@refs/pull/34758/merge to s	Log	dotnet/maui#34758
2993279	dotnet-maui	Checkout dotnet-maui@release/10.0.1xx-sr7 to s	Log
1451203	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67034/merge to s	Log	dotnet/aspnetcore#67034
1450459	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54600/merge to s	Log	dotnet/sdk#54600
1450162	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129006/merge to s	Log	dotnet/runtime#129006
2992372	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/61841/merge to s	Log	#61841
1449786	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16968/merge to s	Log	#16968
1449607	dotnet/maui	Checkout dotnet/maui@refs/pull/35662/merge to s	Log	dotnet/maui#35662
1449750	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16962/merge to s	Log	#16962
1449385	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6349/merge to s	Log	dotnet/arcade-services#6349
1449116	dotnet/runtime	Checkout dotnet/runtime@release/10.0 to s	Log
1449010	dotnet/maui	Checkout dotnet/maui@darc-net11.0-ae17e370-1ba8-4947-9586-8bfb3adb6d5b to s	Log
1448661	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54330/merge to s	Log	dotnet/sdk#54330
1448059	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6351/merge to s	Log	dotnet/arcade-services#6351
1447916	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6347/merge to s	Log	dotnet/arcade-services#6347
1447468	dotnet/maui	Checkout dotnet/maui@release/10.0.1xx-sr7 to s	Log
1446538	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54566/merge to s	Log	dotnet/sdk#54566
1446232	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66982/merge to s	Log	dotnet/aspnetcore#66982
1446087	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128716/merge to s	Log	dotnet/runtime#128716
1445901	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123440/merge to s	Log	dotnet/runtime#123440
1445743	dotnet/maui	Checkout dotnet/maui@refs/pull/35689/merge to s	Log	dotnet/maui#35689
1445625	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16946/merge to s	Log	#16946
1445566	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16942/merge to s	Log	#16942
1445557	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128384/merge to s	Log	dotnet/runtime#128384
1445251	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128649/merge to s	Log	dotnet/runtime#128649
1444688	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128784/merge to s	Log	dotnet/runtime#128784
1444077	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128849/merge to s	Log	dotnet/runtime#128849
1443930	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128852/merge to s	Log	dotnet/runtime#128852
1443780	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128775/merge to s	Log	dotnet/runtime#128775
1443827	dotnet/xharness	Checkout dotnet/xharness@refs/pull/1605/merge to s	Log	dotnet/xharness#1605
1443779	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128775/merge to s	Log	dotnet/runtime#128775
1441480	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54510/merge to s	Log	dotnet/sdk#54510
1441063	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54504/merge to s	Log	dotnet/sdk#54504
1441408	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128782/merge to s	Log	dotnet/runtime#128782
1441055	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16918/merge to s	Log	#16918
1440774	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128763/merge to s	Log	dotnet/runtime#128763
2986945	dotnet-sdk	Checkout dotnet-sdk@main to s	Log
1439277	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54445/merge to s	Log	dotnet/sdk#54445
1437830	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54471/merge to s	Log	dotnet/sdk#54471
1437256	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128647/merge to s	Log	dotnet/runtime#128647
1437034	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128584/merge to s	Log	dotnet/runtime#128584
1437022	dotnet/maui	Checkout dotnet/maui@inflight/current to s	Log
1437023	dotnet/maui	Checkout dotnet/maui@inflight/current to s	Log
1436083	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128510/merge to s	Log	dotnet/runtime#128510
1436039	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125596/merge to s	Log	dotnet/runtime#125596
1434728	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54434/merge to s	Log	dotnet/sdk#54434
1434377	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128556/merge to s	Log	dotnet/runtime#128556
1432988	dotnet/roslyn	Checkout dotnet/roslyn@main to s	Log
1432954	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128521/merge to s	Log	dotnet/runtime#128521
1432885	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128522/merge to s	Log	dotnet/runtime#128522
1432820	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127989/merge to s	Log	dotnet/runtime#127989
1432826	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126947/merge to s	Log	dotnet/runtime#126947
1432790	dotnet/runtime	Checkout dotnet/runtime@release/8.0 to s	Log
1432774	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1432768	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1432757	dotnet/roslyn	Checkout dotnet/roslyn@refs/pull/83397/merge to s	Log	dotnet/roslyn#83397
1432751	dotnet/roslyn	Checkout dotnet/roslyn@refs/pull/83761/merge to s	Log	dotnet/roslyn#83761
1432716	dotnet/performance	Checkout dotnet/performance@refs/pull/5220/merge to s/performance	Log	dotnet/performance#5220
1432687	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128519/merge to s	Log	dotnet/runtime#128519
2981947	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/61470/merge to s	Log	#61470
2981946	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/61469/merge to s	Log	#61469
2981945	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/61469/merge to s	Log	#61469
2981923	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/61466/merge to s	Log	#61466
2981897	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/61465/merge to s	Log	#61465
1431680	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128487/merge to s	Log	dotnet/runtime#128487
1431217	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53953/merge to s	Log	dotnet/sdk#53953
1431124	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54409/merge to s	Log	dotnet/sdk#54409
1430745	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66793/merge to s	Log	dotnet/aspnetcore#66793
2981311	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/61439/merge to s	Log	#61439
1429746	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128442/merge to s	Log	dotnet/runtime#128442
1428924	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128347/merge to s	Log	dotnet/runtime#128347
1428393	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128409/merge to s	Log	dotnet/runtime#128409
2980017	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/61351/merge to s	Log	#61351
1427854	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6318/merge to s	Log	dotnet/arcade-services#6318
1427691	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128394/merge to s	Log	dotnet/runtime#128394
2979270	dotnet-maui	Checkout dotnet-maui@inflight/current to s	Log
2979009	dotnet-installer	Clone dotnet/installer	Log	#61281
2978769	dotnet-maui	Checkout dotnet-maui@release/10.0.1xx-sr7 to s	Log
1426660	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124173/merge to s	Log	dotnet/runtime#124173
1426463	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123252/merge to s	Log	dotnet/runtime#123252
2978282	dotnet-sdk	Checkout dotnet-sdk@refs/pull/61258/merge to s	Log	#61258
1425412	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128337/merge to s	Log	dotnet/runtime#128337
1424819	dotnet/maui	Checkout dotnet/maui@refs/pull/34758/merge to s	Log	dotnet/maui#34758
1424687	dotnet/maui	Checkout dotnet/maui@refs/pull/35362/merge to s	Log	dotnet/maui#35362
1423818	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66712/merge to s	Log	dotnet/aspnetcore#66712
1423733	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128298/merge to s	Log	dotnet/runtime#128298
1422707	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128264/merge to s	Log	dotnet/runtime#128264
1422251	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54318/merge to s	Log	dotnet/sdk#54318
1417748	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54266/merge to s	Log	dotnet/sdk#54266
2975817	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/61122/merge to s	Log	#61122
1421013	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128213/merge to s	Log	dotnet/runtime#128213
1421076	dotnet/runtime	Checkout dotnet/runtime@release/8.0 to s	Log
2975280	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/61099/merge to s	Log	#61099
Displaying 100 of 120 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	33	120

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?