Skip to content

Commit 8e526c0

Browse files
hallelx2hallelx2
committed
feat: Add comprehensive benchmark results and fix feature selection
1 parent 45e43a1 commit 8e526c0

8 files changed

Lines changed: 471 additions & 219 deletions

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,3 +48,6 @@ test_tuning_results*.txt
4848
data/
4949
*.png
5050
!benchmarks/*.png
51+
52+
# Paper (before submission)
53+
paper/

‎COMPREHENSIVE_EXPERIMENT_REPORT.md‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# Comprehensive Research Benchmark Report
2+
Generated on: Tue Mar 10 10:43:42 2026
3+
4+
Total Model Training Optimizers Evaluated: 60
5+
Total HPO Searchers Evaluated: 59
6+
7+
## Best Performing Algorithms by Category
8+
- **bio_inspired**: Best Trainer is SineCosine (Loss: 0.7708)
9+
- **evolutionary**: Best Trainer is CA (Loss: 0.2187)
10+
- **human_based**: Best Trainer is TLBO (Loss: 0.6901)
11+
- **physics**: Best Trainer is FPA (Loss: 0.7203)
12+
- **hybrid**: Best Trainer is Gorilla (Loss: 0.2728)
13+
- **swarm**: Best Trainer is IWOA (Loss: 0.2400)
14+
15+
## Detailed Results (Training Weight Optimization)
16+
| Algorithm | Category | Final Loss |
17+
| --- | --- | --- |
18+
| Adam | Gradient | 0.027349941432476044 |
19+
| SGD | Gradient | 0.12745100259780884 |
20+
| CA | evolutionary | 0.21874654293060303 |
21+
| PFA | evolutionary | 0.22179865837097168 |
22+
| CEM | evolutionary | 0.22524750232696533 |
23+
| IWOA | swarm | 0.2399510145187378 |
24+
| Gorilla | hybrid | 0.27279114723205566 |
25+
| GOA | swarm | 0.28064194321632385 |
26+
| DFO | swarm | 0.29363566637039185 |
27+
| HHO | swarm | 0.33173149824142456 |
28+
| IGWO | swarm | 0.35967591404914856 |
29+
| PSO | swarm | 0.36392876505851746 |
30+
| DVBA | swarm | 0.38466307520866394 |
31+
| Clonalg | swarm | 0.46932533383369446 |
32+
| Firefly | swarm | 0.47261345386505127 |
33+
| FDA | evolutionary | 0.48151740431785583 |
34+
| DE | evolutionary | 0.493569016456604 |
35+
| GA | evolutionary | 0.5120543837547302 |
36+
| SSA | swarm | 0.5476040840148926 |
37+
| ABCO | swarm | 0.5686624646186829 |
38+
| SMA | hybrid | 0.62315833568573 |
39+
| Salp | swarm | 0.6289653182029724 |
40+
| SPBO | swarm | 0.6366990804672241 |
41+
| HUS | swarm | 0.6404846906661987 |
42+
| JSO | hybrid | 0.6434437036514282 |
43+
| GWO | swarm | 0.6477985382080078 |
44+
| MBO | swarm | 0.6646429896354675 |
45+
| SOS | swarm | 0.6796358227729797 |
46+
| WOA | swarm | 0.6832001209259033 |
47+
| Dragonfly | swarm | 0.6852114796638489 |
48+
| TLBO | human_based | 0.6900981068611145 |
49+
| CSA | swarm | 0.690549910068512 |
50+
| PBIL | evolutionary | 0.6931473016738892 |
51+
| Bee | swarm | 0.7153347134590149 |
52+
| FPA | physics | 0.7202872633934021 |
53+
| SineCosine | bio_inspired | 0.7708128094673157 |
54+
| CatSwarm | hybrid | 0.7822293639183044 |
55+
| RandomSearch | swarm | 0.9576318264007568 |
56+
| HarmonySearch | human_based | 0.9678534865379333 |
57+
| BBO | bio_inspired | 1.122300624847412 |
58+
| CuckooSearch | swarm | 1.1293624639511108 |
59+
| Fish | swarm | 1.156600832939148 |
60+
| HSA | swarm | 1.1732394695281982 |
61+
| ALO | bio_inspired | 1.2519598007202148 |
62+
| AFSA | swarm | 1.3024922609329224 |
63+
| Bat | swarm | 1.3055055141448975 |
64+
| Cockroach | hybrid | 1.3235371112823486 |
65+
| DFO2 | swarm | 1.3432682752609253 |
66+
| EHO | hybrid | 1.3501793146133423 |
67+
| ACGWO | swarm | 1.3662052154541016 |
68+
| Memetic | swarm | 1.3912653923034668 |
69+
| JY | swarm | 1.3927199840545654 |
70+
| KHA | hybrid | 1.3947255611419678 |
71+
| MVO | bio_inspired | 1.4407864809036255 |
72+
| AOA | swarm | 1.4603041410446167 |
73+
| GSA | physics | 1.4768476486206055 |
74+
| ARS | evolutionary | 1.5305553674697876 |
75+
| Coati | hybrid | 1.5556957721710205 |
76+
| ChickenSwarm | hybrid | 1.5732109546661377 |
77+
| MFO | bio_inspired | 1.7682843208312988 |
78+
79+
80+
## Detailed Results (Hyperparameter Optimization)
81+
| Algorithm | Category | Best Accuracy |
82+
| --- | --- | --- |
83+
| PFASearch | evolutionary | 0.9549999833106995 |
84+
| HarmonySearchHT | human_based | 0.9549999833106995 |
85+
| PBILSearch | evolutionary | 0.9549999833106995 |
86+
| GSASearch | physics | 0.9549999833106995 |
87+
| JYSearch | swarm | 0.9549999833106995 |
88+
| FDASearch | evolutionary | 0.949999988079071 |
89+
| CockroachSearch | hybrid | 0.949999988079071 |
90+
| FishSearch | swarm | 0.949999988079071 |
91+
| GOASearch | swarm | 0.949999988079071 |
92+
| WOASearch | swarm | 0.949999988079071 |
93+
| CatSwarmSearch | hybrid | 0.949999988079071 |
94+
| ClonalgSearch | swarm | 0.949999988079071 |
95+
| ACGWOSearch | swarm | 0.949999988079071 |
96+
| CoatiSearch | hybrid | 0.9449999928474426 |
97+
| AFSASearch | swarm | 0.9449999928474426 |
98+
| IGWOSearch | swarm | 0.9449999928474426 |
99+
| IWOASearch | swarm | 0.9449999928474426 |
100+
| RandomSearchHT | swarm | 0.9449999928474426 |
101+
| ABCOSearch | swarm | 0.9449999928474426 |
102+
| HHOSearch | swarm | 0.9449999928474426 |
103+
| DragonflySearch | swarm | 0.9449999928474426 |
104+
| SSASearch | swarm | 0.9449999928474426 |
105+
| MBOSearch | swarm | 0.9449999928474426 |
106+
| ARSSearch | evolutionary | 0.9449999928474426 |
107+
| GASearch | evolutionary | 0.9449999928474426 |
108+
| JSOSearch | hybrid | 0.9449999928474426 |
109+
| SineCosineSearch | bio_inspired | 0.9449999928474426 |
110+
| MemeticSearch | swarm | 0.9449999928474426 |
111+
| GWOSearch | swarm | 0.9399999976158142 |
112+
| DVBASearch | swarm | 0.9399999976158142 |
113+
| FireflySearch | swarm | 0.9399999976158142 |
114+
| HSASearch | swarm | 0.9399999976158142 |
115+
| SalpSearch | swarm | 0.9399999976158142 |
116+
| CuckooSearchHT | swarm | 0.9399999976158142 |
117+
| SOSSearch | swarm | 0.9399999976158142 |
118+
| CASearch | evolutionary | 0.9399999976158142 |
119+
| MFOSearch | bio_inspired | 0.9399999976158142 |
120+
| ALOSearch | bio_inspired | 0.9399999976158142 |
121+
| BBOSearch | bio_inspired | 0.9399999976158142 |
122+
| DESearch | evolutionary | 0.9399999976158142 |
123+
| FPASearch | physics | 0.9399999976158142 |
124+
| CEMSearch | evolutionary | 0.9399999976158142 |
125+
| EHOSearch | hybrid | 0.9399999976158142 |
126+
| BeeSearch | swarm | 0.9399999976158142 |
127+
| BatSearch | swarm | 0.9350000023841858 |
128+
| SMASearch | hybrid | 0.9350000023841858 |
129+
| KHASearch | hybrid | 0.9350000023841858 |
130+
| TLBOSearch | human_based | 0.9350000023841858 |
131+
| ChickenSwarmSearch | hybrid | 0.9350000023841858 |
132+
| MVOSearch | bio_inspired | 0.9350000023841858 |
133+
| GorillaSearch | hybrid | 0.9350000023841858 |
134+
| DFO2Search | swarm | 0.9350000023841858 |
135+
| HUSSearch | swarm | 0.9350000023841858 |
136+
| DFOSearch | swarm | 0.9350000023841858 |
137+
| PSOSearch | swarm | 0.9300000071525574 |
138+
| SPBOSearch | swarm | 0.9300000071525574 |
139+
| AOASearch | swarm | 0.9300000071525574 |
140+
| SASearch | physics | 0.925000011920929 |
141+
| CSASearch | swarm | 0.925000011920929 |

‎benchmarks/feature_selection_benchmark.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,11 +53,11 @@ def evaluate_subset(
5353
y_train, y_val = self.y[:n_train], self.y[n_train:]
5454

5555
# Quick training with Adam
56-
n_selected = selected_indices.sum().item()
56+
n_selected = int(selected_indices.sum().item())
5757
if n_selected == 0:
5858
return 1.0, None # Worst case: no features
5959

60-
hidden = min(32, max(8, n_selected))
60+
hidden = int(min(32, max(8, n_selected)))
6161

6262
model = nn.Sequential(
6363
nn.Linear(n_selected, hidden), nn.ReLU(), nn.Linear(hidden, 1), nn.Sigmoid()

‎benchmarks/gpu_scaling.png‎

74.1 KB
Loading

‎benchmarks/nonconvex_results.json‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
{
2+
"Rastrigin": {
3+
"Adam": {
4+
"final_loss": 16.416805267333984,
5+
"std": 0.4974784851074219
6+
},
7+
"PSO": {
8+
"final_loss": 6.9647064208984375,
9+
"std": 0.9949569702148438
10+
},
11+
"GWO": {
12+
"final_loss": 0.0,
13+
"std": 0.0
14+
},
15+
"WOA": {
16+
"final_loss": 3.0087203979492188,
17+
"std": 1.4063034057617188
18+
},
19+
"HHO": {
20+
"final_loss": 24.480865478515625,
21+
"std": 3.2164459228515625
22+
},
23+
"DE": {
24+
"final_loss": 24.23912811279297,
25+
"std": 2.7301177978515625
26+
},
27+
"GA": {
28+
"final_loss": 4.691169738769531,
29+
"std": 0.46814727783203125
30+
}
31+
},
32+
"Ackley": {
33+
"Adam": {
34+
"final_loss": 3.1942930221557617,
35+
"std": 0.3799457550048828
36+
},
37+
"PSO": {
38+
"final_loss": 2.384185791015625e-05,
39+
"std": 1.1444091796875e-05
40+
},
41+
"GWO": {
42+
"final_loss": 9.5367431640625e-07,
43+
"std": 0.0
44+
},
45+
"WOA": {
46+
"final_loss": 0.035750389099121094,
47+
"std": 0.03560829162597656
48+
},
49+
"HHO": {
50+
"final_loss": 1.089625358581543,
51+
"std": 0.0299835205078125
52+
},
53+
"DE": {
54+
"final_loss": 0.027215957641601562,
55+
"std": 0.0013780593872070312
56+
},
57+
"GA": {
58+
"final_loss": 0.04723930358886719,
59+
"std": 0.0034379959106445312
60+
}
61+
},
62+
"Rosenbrock": {
63+
"Adam": {
64+
"final_loss": 88.77474784851074,
65+
"std": 80.41482734680176
66+
},
67+
"PSO": {
68+
"final_loss": 4.272107124328613,
69+
"std": 1.564004898071289
70+
},
71+
"GWO": {
72+
"final_loss": 8.767738819122314,
73+
"std": 0.0031075477600097656
74+
},
75+
"WOA": {
76+
"final_loss": 8.945044040679932,
77+
"std": 0.0012412071228027344
78+
},
79+
"HHO": {
80+
"final_loss": 30.712162017822266,
81+
"std": 10.909629821777344
82+
},
83+
"DE": {
84+
"final_loss": 2.7546653151512146,
85+
"std": 1.209701120853424
86+
},
87+
"GA": {
88+
"final_loss": 4.575098276138306,
89+
"std": 0.2811167240142822
90+
}
91+
},
92+
"Schwefel": {
93+
"Adam": {
94+
"final_loss": -15.243349075317383,
95+
"std": 1.1974658966064453
96+
},
97+
"PSO": {
98+
"final_loss": -163.9660701751709,
99+
"std": 124.5130558013916
100+
},
101+
"GWO": {
102+
"final_loss": -478.81583189964294,
103+
"std": 474.08492493629456
104+
},
105+
"WOA": {
106+
"final_loss": -2.032773381703753e+37,
107+
"std": 1.2829594460844156e+37
108+
},
109+
"HHO": {
110+
"final_loss": -95.86214828491211,
111+
"std": 56.435062408447266
112+
},
113+
"DE": {
114+
"final_loss": -123613126.625,
115+
"std": 119824729.375
116+
},
117+
"GA": {
118+
"final_loss": -614.7274684906006,
119+
"std": 578.7637424468994
120+
}
121+
},
122+
"Griewank": {
123+
"Adam": {
124+
"final_loss": 0.004618614912033081,
125+
"std": 0.003447920083999634
126+
},
127+
"PSO": {
128+
"final_loss": 0.0,
129+
"std": 0.0
130+
},
131+
"GWO": {
132+
"final_loss": 0.0,
133+
"std": 0.0
134+
},
135+
"WOA": {
136+
"final_loss": 0.0002269446849822998,
137+
"std": 0.00021627545356750488
138+
},
139+
"HHO": {
140+
"final_loss": 0.010126769542694092,
141+
"std": 0.003208756446838379
142+
},
143+
"DE": {
144+
"final_loss": 1.6510486602783203e-05,
145+
"std": 1.049041748046875e-05
146+
},
147+
"GA": {
148+
"final_loss": 9.971857070922852e-05,
149+
"std": 3.5762786865234375e-07
150+
}
151+
},
152+
"Sphere": {
153+
"Adam": {
154+
"final_loss": 0.03764493702146865,
155+
"std": 0.037596092241074075
156+
},
157+
"PSO": {
158+
"final_loss": 3.9502311144357094e-11,
159+
"std": 2.7986696293780255e-11
160+
},
161+
"GWO": {
162+
"final_loss": 0.0,
163+
"std": 0.0
164+
},
165+
"WOA": {
166+
"final_loss": 6.151113484520465e-05,
167+
"std": 1.0862546332646161e-05
168+
},
169+
"HHO": {
170+
"final_loss": 0.15951810032129288,
171+
"std": 0.03103446215391159
172+
},
173+
"DE": {
174+
"final_loss": 9.856560427579097e-05,
175+
"std": 1.6551599401282147e-05
176+
},
177+
"GA": {
178+
"final_loss": 0.0012610839039552957,
179+
"std": 0.00045184415648691356
180+
}
181+
}
182+
}

0 commit comments

Comments
 (0)