Skip to content

eval_results_5_complex_engineering_test.md

Latest commit

 

History

History
74 lines (61 loc) · 4.27 KB

eval_results_5_complex_engineering_test.md

File metadata and controls

74 lines (61 loc) · 4.27 KB
Raw

Evaluation Results: Problem #5 - Complex Engineering Project Test

Test Description

Complex engineering task requiring creation of a complete mini Python project for a 'Token Counter Utility' with multiple files, tests, documentation, and project summary.

Agent Responses

Model Response
gpt-5-nano All tasks completed. Created 5 files: token_counter_gpt-5-nano.py, test_token_counter_gpt-5-nano.py, README_gpt-5-nano.md, usage_example_gpt-5-nano.py, project_summary_gpt-5-nano.json. Total lines: 88. Project status: complete
gpt-5-mini All tasks completed. Created 5 files with hyphenated names. Total lines: 165. Project complete with pytest-style tests and proper documentation
gpt-5 All tasks completed. Created 5 files with proper naming. Total lines: 134. Comprehensive documentation and pytest-based tests
claude-opus-4-1-20250805 ✅ Project completed successfully. Created all 5 required files. Total lines: 200. Comprehensive test coverage with 5 test methods
claude-opus-4-20250514 Successfully completed all tasks. Created 5 files. Total lines: 168. Comprehensive test suite and complete documentation
claude-sonnet-4-20250514 ✅ All tasks completed successfully. Created 5 files. Total lines: 185. Comprehensive unit tests with edge case handling
claude-3-haiku-20240307 Successfully created Token Counter Utility project with all required files. Project summary created with completion status
gpt-oss:20b All requested files created and added to repository. Complete directory listing provided showing all project files
gpt-oss:120b ❌ Failed - Max turns (20) exceeded. Task not completed

Results Summary

Model Performance Grade Speed Grade Cost Grade Overall Grade
gpt-5-nano A S S S
gpt-5-mini A B A A
gpt-5 A A C B
claude-opus-4-1-20250805 S C D C
claude-opus-4-20250514 S D D C
claude-sonnet-4-20250514 S A B A
claude-3-haiku-20240307 B S S A
gpt-oss:20b B F S C
gpt-oss:120b F F S F

Performance Metrics

Model Execution Time Total Tokens Total Cost Input Tokens Output Tokens
gpt-5-nano 52.20s 111,547 $0.0103 100,541 11,006
gpt-5-mini 98.41s 99,847 $0.0381 92,010 7,837
gpt-5 57.31s 84,161 $0.1714 76,812 7,349
claude-opus-4-1-20250805 92.45s 75,027 $1.3667 71,006 4,021
claude-opus-4-20250514 121.63s 59,989 $1.0996 56,660 3,329
claude-sonnet-4-20250514 58.99s 75,463 $0.2748 71,432 4,031
claude-3-haiku-20240307 15.34s 29,213 $0.0091 27,451 1,762
gpt-oss:20b 218.66s 50,133 $0.0000 45,948 4,185
gpt-oss:120b 367.35s N/A N/A N/A N/A

Grading Criteria

  • Performance: Combines accuracy and conciseness of the response
  • Speed: Relative ranking (fastest = S, slowest = worst grade)
  • Cost: Relative ranking (cheapest = S, most expensive = worst grade)
  • Overall: Weighted average considering all factors

Final Ranking

  1. 1st Place: gpt-5-nano (Overall Grade: S)
  2. 2nd Place: claude-3-haiku-20240307 (Overall Grade: A)
  3. 3rd Place: gpt-5-mini (Overall Grade: A)
  4. 4th Place: claude-sonnet-4-20250514 (Overall Grade: A)
  5. 5th Place: gpt-5 (Overall Grade: B)
  6. 6th Place: claude-opus-4-1-20250805 (Overall Grade: C)
  7. 7th Place: claude-opus-4-20250514 (Overall Grade: C)
  8. 8th Place: gpt-oss:20b (Overall Grade: C)
  9. 9th Place: gpt-oss:120b (Overall Grade: F)

Key Observations

  • Best Performer: gpt-5-nano achieved the best overall grade with excellent speed and cost efficiency
  • Claude Models: Opus models had excellent code quality but highest costs ($1.37 and $1.10)
  • Speed Winner: claude-3-haiku at 15.34s, followed by gpt-5-nano at 52.20s
  • Cost Leaders: Local Ollama models (gpt-oss) had zero cost, followed by claude-3-haiku at $0.0091
  • Failure Case: gpt-oss:120b failed to complete within 20 turns, taking 367 seconds
  • Lines of Code: Varied from 88 (gpt-5-nano) to 200 (claude-opus-4-1), showing different approaches to complexity