Differentiate between JSON 'key = null' and 'key not set'

[miketonks]

In JSON there are effectively two types of null.

1) When the key is provided, and the value is explicitly stated as null.
2) When the key is not provided, and the value is implicitly null.

This is important when constructing a PATCH endpoint because:

1. When the key is provided, with a null value, the field should be set to null
2. When the key is not provided, the value should not be changed

There is a more detailed discussion of this issue here, with example code of how to solve the problem:

https://www.calhoun.io/how-to-determine-if-a-json-key-has-been-set-to-null-or-not-provided/

It would be nice if the null library can support this.

[sminf]

If you want to omit empty field, field need to be pointer or primitive type, I try to create a tool to solve this problem.
Giving it a try, maybe it helps.
poohvpn/ptr

[Fryuni]

Although @honeycruse example is for marshaling, the problem is at the receiving end. After unmarshaling, how to know if the field is null because it was set to nul or because it was missing. If you use a pointer it will be nil in both cases, so it does not solve the problem.

A nice solution would be to have another field in the struct in addition to the Valid field, like SetToNull or ExplicitNull, that would be set to true only when decoding a null. Since a missing key would not do any decoding the field would be false if is missing.

[timsolov]

Yep, it would be nice to implement this feature. Good implementation of you talk about I saw in pgtype.Status link

[zwass]

@guregu is there any interest in tackling this problem with this library?

[jamietanna]

We are looking at solving this as part of oapi-codegen/runtime#26 for a similar issue for use with OpenAPI. Would it be of interest to expose this code as a library on its own, so these cases (as well as #67) can be filled?

[jamietanna]

I've released this library as https://github.com/oapi-codegen/nullable - it's not at all reliant on OpenAPI, and can be used as standalone, or feel free to copy-paste the (Apache-2.0) code for your own usages!

This handles:

• field isn't set
• field is set, and is null
• field is set, and is a given value

And both marshalling and unmarshalling 👏

[juev]

@Fryuni

Although @honeycruse example is for marshaling, the problem is at the receiving end. After unmarshaling, how to know if the field is null because it was set to nul or because it was missing. If you use a pointer it will be nil in both cases, so it does not solve the problem.

What is the difference between the field that we set to null and the field that was missing? In both cases, the result will be null. Or nil. For what purposes do we need to know that the field was explicitly set to null? In my opinion, it's just an empty meaning, and whether we put it up ourselves or it's just missing, it doesn't really matter. And we just take the meaning that is, that is, nil.

[jamietanna]

What is the difference between the field that we set to null and the field that was missing? In both cases, the result will be null. Or nil. For what purposes do we need to know that the field was explicitly set to null? In my opinion, it's just an empty meaning, and whether we put it up ourselves or it's just missing, it doesn't really matter. And we just take the meaning that is, that is, nil.

Not true, there are semantic differences for some use cases. It may not be the case for everyone, but given the folks requesting it ☝️ as well as users of oapi-codegen requiring it,

I've written about it in more detail in https://www.jvt.me/posts/2024/01/09/go-json-nullable/ which goes alongside the library release, and for instance is used by https://www.rfc-editor.org/rfc/rfc7386 to mean different things:

• field isn't set: so don't make any changes to it
• field is set, and is null: so set the field's value to null
• field is set, and is a given value: so set the field's value to that value

The library we've written handles all three cases, for marshalling and unmarshalling 👍

[juev]

@jamietanna I'm sorry, but I still didn't understand the difference between not set and null. And how should we take this into account in the processing of values.

If you look at the list of types used
https://www.w3schools.com/js/js_json_datatypes.asp

Then, as I understand it, null is used to indicate that the field isn't set. And as a result, it should be used as the first case.

[jamietanna]

Via this feature request, there's the idea that these are two different expressions:

{
}

and

{
  "field": null
}

Go, by default, only allows us to say whether something is there (string), or it isn't (*string).

This library's types allow making that easier, but we still miss out on the expression of "it is there, but it is explicitly set to null".

Not sure if anyone else who's looking for this feature have their own use-cases they'd like to share?

[juev]

The result will be used in the future. To do this, we conduct Unmarshal of the received json.

But what in the end?

https://go.dev/play/p/N_dMIfMvGts

js result: main.tStruct{Name:"Denis", Type:"Name"}

jsNull result: main.tStruct{Name:"Denis", Type:""}

jsEmpty result: main.tStruct{Name:"Denis", Type:""}

Thus, the result is identical. Whether we used a null value or an empty value.

[guregu]

I can understand why people want this. Usually they have PATCH APIs where they want to differentiate between "don't touch this value" and "set this value to null". I try to avoid making these kinds of APIs but they do exist. You can get this by using e.g. *null.String but then you lose the safety aspect. Maybe the ideal situation would be to change the methods on the null.* structs to take a pointer receiver and check for nil inside them so this pattern is easier to deal with.

[hmh]

Yes, the use for this is on "PATCH"-like APIs. E.g. an endpoint where you can add / replace / delete attributes of an object, which takes its input as a JSON.

On this type of API (which is very common), to delete something you set it to null: { "deleteme": null }. To leave something unchanged, you don't mention it at all. So:

• { } as an input, leaves the object unmodified
• { "foo": 3, "bar": null } as an input, creates "foo" (or replaces if it already exists) and sets it value to 3, deletes "bar" (if it exists), and leaves everything else unchanged.