Skip to content

Commit 16eef48

Browse files
simonwsimonw
committed
Tests for options, closes #6 
1 parent f4c8ec6 commit 16eef48

10 files changed

Lines changed: 517 additions & 1 deletion

‎README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,3 +40,11 @@ To run the tests:
4040
```bash
4141
python -m pytest
4242
```
43+
44+
This project uses [pytest-recording](https://github.com/kiwicom/pytest-recording) to record OpenAI API responses for the tests, and [syrupy](https://github.com/syrupy-project/syrupy) to capture snapshots of their results.
45+
46+
If you add a new test that calls the API you can capture the API response and snapshot like this:
47+
```bash
48+
PYTEST_OPENAI_API_KEY="$(llm keys get openai)" pytest --record-mode once --snapshot-update
49+
```
50+
Then review the new snapshots in `tests/__snapshots__/` to make sure they look correct.

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,4 +28,4 @@ CI = "https://github.com/simonw/llm-openai-plugin/actions"
2828
openai = "llm_openai"
2929

3030
[project.optional-dependencies]
31-
test = ["pytest"]
31+
test = ["pytest", "pytest-recording", "syrupy"]

‎tests/__snapshots__/test_openai.ambr‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# serializer version: 1
2+
# name: test_options[options0]
3+
'Hi there! How can I assist you today?'
4+
# ---
5+
# name: test_options[options1]
6+
'Hi there! How can I assist you today?'
7+
# ---
8+
# name: test_options[options2]
9+
'Hi there! How can I assist you today?'
10+
# ---
11+
# name: test_options[options3]
12+
'Hi there! How can I assist you today?'
13+
# ---
14+
# name: test_options[options4]
15+
'Hi there! How can I assist you today?'
16+
# ---

‎tests/cassettes/test_openai/test_options[options0].yaml‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
interactions:
2+
- request:
3+
body: '{"input":[{"role":"user","content":"say hi"}],"model":"gpt-4o-mini","max_output_tokens":24,"stream":false}'
4+
headers:
5+
accept:
6+
- application/json
7+
accept-encoding:
8+
- gzip, deflate
9+
connection:
10+
- keep-alive
11+
content-length:
12+
- '106'
13+
content-type:
14+
- application/json
15+
host:
16+
- api.openai.com
17+
user-agent:
18+
- OpenAI/Python 1.67.0
19+
x-stainless-arch:
20+
- arm64
21+
x-stainless-async:
22+
- 'false'
23+
x-stainless-lang:
24+
- python
25+
x-stainless-os:
26+
- MacOS
27+
x-stainless-package-version:
28+
- 1.67.0
29+
x-stainless-read-timeout:
30+
- '600'
31+
x-stainless-retry-count:
32+
- '0'
33+
x-stainless-runtime:
34+
- CPython
35+
x-stainless-runtime-version:
36+
- 3.10.10
37+
method: POST
38+
uri: https://api.openai.com/v1/responses
39+
response:
40+
body:
41+
string: !!binary |
42+
H4sIAAAAAAAAA3RTTZObMAy951e4nDcdICTQXHrd/oadHUaxReLWWIwtt5vZyX/vYAMh7e6FwXp6
43+
+niS3jdCZFplR5E59EN7qJVUctfkh0o2xbcSAIv6oHYNdmWe54e8U/VJYrnvTo3a7cvsaQxAp58o
44+
eQ5C1mOyS4fAqFoYsaKuyn1Z76sqYp6Bgx85kvrBIKNKJHSOXHYUNhgTDdrOHq1CBm38I+rZBcma
45+
7IO9h7eWAg+BW6ZfGMEype5JoRkznwfeVrTttdXbMi+rbV5vi2bqKXKzo3jZCCHEe/wKkfF1wJHb
46+
o/dwTo1GIKnY+/MiYlN3VRQx7w7q1OzkoWhAFvsPRYwxPhMlgo5MzAzea89g+Q5Jsoz2Xu264oeq
47+
Z0Xw7c5ODqPlKLJnLfiCDr+IZ/ojJFjxQ6SE4kpBMCm4fn+kgrXEMA/g5XXBbtNfsoyv1yjtAA6M
48+
QdMykWklmDhRdgET7PC3puDbeZnaKO0yWYfgyWp7zo5Tkxl2HTleOQmRndGiA8bWh74Hd53QjRC3
49+
aQHJ4TovYz+MjBDNxdd8skZhpkQduR7u75Wy0W9pNKVI/V1IyzS4wJQtQBJreg7tsM7pgpVR0ZGm
50+
tIeTmc8jxK1bCtL2ccPrp//tq7NZypYgL6juxDyVPrH/PZyi+Aj4KO4ynM9CMzGYO7hrFrWCx4e7
51+
75FBAcMY/ra5/QUAAP//AwClcSKcqwQAAA==
52+
headers:
53+
CF-RAY:
54+
- 923a57fdde462ea5-LAX
55+
Connection:
56+
- keep-alive
57+
Content-Encoding:
58+
- gzip
59+
Content-Type:
60+
- application/json
61+
Date:
62+
- Fri, 21 Mar 2025 03:25:44 GMT
63+
Server:
64+
- cloudflare
65+
Set-Cookie:
66+
- __cf_bm=JXdxQux4nrLIqjrYP9gbm4JNTlxzGLB2PWq3N_WThZE-1742527544-1.0.1.1-7dNckOa7ss5XFpwaUYgV3VxV4GFFxh8miaVnZ0mCjI0l7CLvbVeuq3s1b1ZfXjR0EV1Mu2Q6hYzowIQENjls1pDw7pIeTPVqa6G9rf5v9W0;
67+
path=/; expires=Fri, 21-Mar-25 03:55:44 GMT; domain=.api.openai.com; HttpOnly;
68+
Secure; SameSite=None
69+
- _cfuvid=26M8u12AWt8WKOwfunlmJPAzHro4wjIYkJlPJsDXGDI-1742527544760-0.0.1.1-604800000;
70+
path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None
71+
Transfer-Encoding:
72+
- chunked
73+
X-Content-Type-Options:
74+
- nosniff
75+
alt-svc:
76+
- h3=":443"; ma=86400
77+
cf-cache-status:
78+
- DYNAMIC
79+
openai-organization:
80+
- user-r3e61fpak04cbaokp5buoae4
81+
openai-processing-ms:
82+
- '739'
83+
openai-version:
84+
- '2020-10-01'
85+
strict-transport-security:
86+
- max-age=31536000; includeSubDomains; preload
87+
x-request-id:
88+
- req_26583719d8653f8dcafe210c8a2ea214
89+
status:
90+
code: 200
91+
message: OK
92+
version: 1

‎tests/cassettes/test_openai/test_options[options1].yaml‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
interactions:
2+
- request:
3+
body: '{"input":[{"role":"user","content":"say hi"}],"model":"gpt-4o-mini","stream":false,"temperature":0.5}'
4+
headers:
5+
accept:
6+
- application/json
7+
accept-encoding:
8+
- gzip, deflate
9+
connection:
10+
- keep-alive
11+
content-length:
12+
- '101'
13+
content-type:
14+
- application/json
15+
host:
16+
- api.openai.com
17+
user-agent:
18+
- OpenAI/Python 1.67.0
19+
x-stainless-arch:
20+
- arm64
21+
x-stainless-async:
22+
- 'false'
23+
x-stainless-lang:
24+
- python
25+
x-stainless-os:
26+
- MacOS
27+
x-stainless-package-version:
28+
- 1.67.0
29+
x-stainless-read-timeout:
30+
- '600'
31+
x-stainless-retry-count:
32+
- '0'
33+
x-stainless-runtime:
34+
- CPython
35+
x-stainless-runtime-version:
36+
- 3.10.10
37+
method: POST
38+
uri: https://api.openai.com/v1/responses
39+
response:
40+
body:
41+
string: !!binary |
42+
H4sIAAAAAAAAA3RTTW/bMAy951doPjeF7SS2k8uu3W8oCoOR6ESdLBoStTUo8t8HS46TdOnFsPj4
43+
+PFIfi6EyLTKdiJz6Ie2qpVUcrXNEfKm2JZbqcr9ptjvFTRypaq8kp0s8rrpOmwqUNnTGID27yj5
44+
EoSsx2SXDoFRtTBiRb0uN2W9WW8i5hk4+JEjqR8MMk7B0Dly2U7YYEw0aHvxaBUyaOPvUc8uSNZk
45+
7+w9fLQUeAjcMv3GLyApNGPuw8DLNS17bfWyzMv1Mq+XRTN1FdnZTrwuhBDiM36FyPg04Mjt0Xs4
46+
pFYjkHTs/WGWsapXMsrY7PNaVlVedahUXj+UMcb4TpYIOjIxM3ivPYPlKyTJMtprtbcV31V90QQ/
47+
ruzkMFp2InvRgo/o8Id4ob9CghW/REooThQEk4LTz3sqWEsMlxG8vs3YefpLlvH1FqUdwIExaFom
48+
Mq0EE2fKLmCCHf7RFHx7Wac2SjuPzyF4stoest3UZIZdR45vnITIDmjRAWPrQ9+DO03oQojztILk
49+
8DYvYz+MjBDN+fNmskZhpkQduR6u7xtlo9/caEqR+juSlmlwgSmbgSTW9BzaYTyS5zy9XbAyKjrS
50+
lPawN5cDCXHr5oK0vdvxsn76335zOHPZEuQR1ZWYp9In9tfTKYpHwKO483C+C83EYK7gqpnVCh7v
51+
Lr9HBgUMY/jz4vwPAAD//wMAasRMqa0EAAA=
52+
headers:
53+
CF-RAY:
54+
- 923a580439a62f08-LAX
55+
Connection:
56+
- keep-alive
57+
Content-Encoding:
58+
- gzip
59+
Content-Type:
60+
- application/json
61+
Date:
62+
- Fri, 21 Mar 2025 03:25:45 GMT
63+
Server:
64+
- cloudflare
65+
Set-Cookie:
66+
- __cf_bm=tLvGN1wstwe_KQJoRmgZgifwCcxuZ5ebkT.fnO48i08-1742527545-1.0.1.1-HExdOMXRHhHuX4s3NHFXzBarGFbN1WdV37FvhvBGIVzKt72u1B4V8pLnpjrfi4OBjeKF5LlQcOZGnYr6ZpKOxKsJcgoUYI_MKM7PtcV4MwY;
67+
path=/; expires=Fri, 21-Mar-25 03:55:45 GMT; domain=.api.openai.com; HttpOnly;
68+
Secure; SameSite=None
69+
- _cfuvid=OJXkrSaEUWlb_53aF9ZNHlCZMYobE27TFSBWPsGDFvw-1742527545618-0.0.1.1-604800000;
70+
path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None
71+
Transfer-Encoding:
72+
- chunked
73+
X-Content-Type-Options:
74+
- nosniff
75+
alt-svc:
76+
- h3=":443"; ma=86400
77+
cf-cache-status:
78+
- DYNAMIC
79+
openai-organization:
80+
- user-r3e61fpak04cbaokp5buoae4
81+
openai-processing-ms:
82+
- '573'
83+
openai-version:
84+
- '2020-10-01'
85+
strict-transport-security:
86+
- max-age=31536000; includeSubDomains; preload
87+
x-request-id:
88+
- req_4e29993d5f141ee003de817ce43ad8a5
89+
status:
90+
code: 200
91+
message: OK
92+
version: 1

‎tests/cassettes/test_openai/test_options[options2].yaml‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
interactions:
2+
- request:
3+
body: '{"input":[{"role":"user","content":"say hi"}],"model":"gpt-4o-mini","stream":false,"top_p":0.5}'
4+
headers:
5+
accept:
6+
- application/json
7+
accept-encoding:
8+
- gzip, deflate
9+
connection:
10+
- keep-alive
11+
content-length:
12+
- '95'
13+
content-type:
14+
- application/json
15+
host:
16+
- api.openai.com
17+
user-agent:
18+
- OpenAI/Python 1.67.0
19+
x-stainless-arch:
20+
- arm64
21+
x-stainless-async:
22+
- 'false'
23+
x-stainless-lang:
24+
- python
25+
x-stainless-os:
26+
- MacOS
27+
x-stainless-package-version:
28+
- 1.67.0
29+
x-stainless-read-timeout:
30+
- '600'
31+
x-stainless-retry-count:
32+
- '0'
33+
x-stainless-runtime:
34+
- CPython
35+
x-stainless-runtime-version:
36+
- 3.10.10
37+
method: POST
38+
uri: https://api.openai.com/v1/responses
39+
response:
40+
body:
41+
string: !!binary |
42+
H4sIAAAAAAAAA3RT227bMAx9z1doem4GX5I4zcteu28oCoOWmFSbLBoStTUo8u+DJdtJuvbFsHh4
43+
eDkk31dCSKPlQUiPYWh3jVZa1Y/q8Vjsy8cKiqqs951qul1XoioK6PZFo3QNWO807OTDGIC6X6h4
44+
DkIuYLYrj8CoWxixstlU26rZbrYJCwwcw8hR1A8WGXUmoffk5UG4aG0yGDd7tBoZjA33aGAfFRty
45+
d/Ye3lqKPERumX7jB5A02jH3aeD1hta9cWZdFdVmXTTrcj91ldjyIJ5XQgjxnr5CSD4POHJ7DAFO
46+
udUEZB37cJplhLqaZETc1hqKbttsatipT2VMMb6SJYGebMoMIZjA4PgKKXKM7lrtbcV3Vc+a4NuV
47+
nR1Gy0HIJyP4FT1+E0/0Vyhw4qfICcWZomDScP5xTwXniGEewfPLgl2mv2wZXy9J2gE8WIu2ZSLb
48+
KrBppuwjZtjjH0MxtPM6tUnaZXweIZAz7iQPU5MSj0fyfOMkhDyhQw+MbYh9D/48oSshLtMKksfb
49+
vIz9MDJiMpffi8mahJkSHcn3cH3fKJv8lkZzitzfKxmVBxeZ5AJksabn0A7yIIrv+TrYR6eSoiNN
50+
mwCdnQ8kpq1bCjLubser5uF/+83hLGUrUK+or8Qilz6xP55OWX4GfBZ3Gc5XoZkY7BWs94taMeDd
51+
5ffIoIFhDH9ZXf4BAAD//wMAciiVsq0EAAA=
52+
headers:
53+
CF-RAY:
54+
- 923a5808efcf7bec-LAX
55+
Connection:
56+
- keep-alive
57+
Content-Encoding:
58+
- gzip
59+
Content-Type:
60+
- application/json
61+
Date:
62+
- Fri, 21 Mar 2025 03:25:46 GMT
63+
Server:
64+
- cloudflare
65+
Set-Cookie:
66+
- __cf_bm=EXHA7qL.M6s8y_lcT5KTzdfjIWG_LJoiM5YD7hQQixs-1742527546-1.0.1.1-LxZLcYHA1p_Ok7hnST4a4uQWtfGw1o9.NExFCCUf6WIAIiPfizVdzEbxuhFmvmpx3.tUd6VtpqEvlCvGmxp3F57yepgNdsU33CMR9RDHsbc;
67+
path=/; expires=Fri, 21-Mar-25 03:55:46 GMT; domain=.api.openai.com; HttpOnly;
68+
Secure; SameSite=None
69+
- _cfuvid=5VCny7O9sqs3BgjQWm43s68Oir0n7zVqeJ6Zbv8uedw-1742527546484-0.0.1.1-604800000;
70+
path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None
71+
Transfer-Encoding:
72+
- chunked
73+
X-Content-Type-Options:
74+
- nosniff
75+
alt-svc:
76+
- h3=":443"; ma=86400
77+
cf-cache-status:
78+
- DYNAMIC
79+
openai-organization:
80+
- user-r3e61fpak04cbaokp5buoae4
81+
openai-processing-ms:
82+
- '700'
83+
openai-version:
84+
- '2020-10-01'
85+
strict-transport-security:
86+
- max-age=31536000; includeSubDomains; preload
87+
x-request-id:
88+
- req_03c8e6c7508578d460a781c0e0d9ef7d
89+
status:
90+
code: 200
91+
message: OK
92+
version: 1

0 commit comments

Comments
 (0)