fitdump.pl: JSON output

[whitlockjc]

I'm trying to do some data analysis of .fit files and I love Garmin::FIT for helping me with this. But not being a Perl developer by trade, it would be really slick if fitdump.pl had a --json option or some way to output the data as JSON. Ideally, you'd output basically an array of objects and each object would correspond with the type of data it corresponds to. I will look into submitting a PR for this in the meantime but I figured I'd start here to get your opinion, and possibly save time by having an expert knock it out.

Thanks again for this awesome library.

[mrihtar]

I agree that the JSON version of .fit data could be very helpful, but there are two possible problems:

1. data is stored in .fit file in a certain order, which could be ignored or not - depends on what you want to do with it. Not all records have a timestamp entry.
2. mapping of .fit data types to JSON data types. One could create a simplified version of JSON (just the field name and value) or more complex, where a data type would also be printed. There are also arrays of values and units of measurement (not really in the .fit file, but defined in SDK).

The main routine used in fitdump.pl is print_all_fields() from Garmin::FIT, which is Kiyokazu's creation (he's the real expert on this :-). This routine could be used as a basis for JSON output.

[whitlockjc]

What if it just followed the FIT spec? A FIT file has a header, an array/list or record objects and a CRC. I've just started writing a parser in Go and my FIT file structure looks very similar to what is proposed below:

{
  "crc": 49512,
  "header": {
    "crc": 23380,
    "dataSize": 4430,
    "profileVersion": "20.13",
    "protocolVersion": "1.0",
    "size": 14,
  },
  "records": [

  ]
}

Note: The CRC values above are in int format instead of hex.

This would still let you have the varying data/length in the data/records section while still providing some of the FIT file details in other portions of the structure. Thoughts?

Note: My current work, and thus my proposed JSON format, do not account for the scenario where you have multiple FIT files concatenated into one FIT file (which is permissible).

[mrihtar]

I added 1st version of JSON output to fitdump.pl. You can try it with command line switch "-print_json=1". If you want to include the units, which are switched off by default for JSON, add command line switch "-without_unit=0". The format is the following (trying to cover concatenated .fit files, although I have no such example):

{
  "file": [{
    "crc": "0x37CC",
    "header": {
      "crc": "0x3B2D",
      "file_size": 135955,
      "protocol_version": "1.00",
      "profile_version": "16.72"
    },
    "records": [
      {"file_id": {
        "serial_number": 3917395546,
        "time_created": "2016-06-17T20:40:00",
        "manufacturer": "garmin",
        "garmin_product": "edge520",
        "type": "activity"
      }},
      ...
    ]
  }]
}

I also don't output invalid values, because with this simpler format there's no way to distinguish them from the valid ones.

[whitlockjc]

From what I see, it looks good. I'll give it a spin soon. If you don't hear from me, feel free to ping me.

[jameswberry]

I've been using this to build some tools in node.js. I noticed that the JSON output loses the message_number value from the original file. I've hacked my local version for my needs for now, but this would be a great addition.

[mrihtar]

It's no quite clear to me where would we put the message number and retain easy accessibility of the data. Can you give me an example? We could also add an additional mapping between message names and numbers at the beginning.

[jameswberry]

+1 for message mapping. Including the other message information would be useful too. It's lost in translation right now.

  },
  "messages": {
    "file_id": {
      "message_number": 0,
      "message_type": 0,
      "message_length": 16
    },
    "file_creator": {
      "message_number": 49,
      "message_type": 0,
      "message_length": 4
    },
    ... etc.
  },
  "records": [

Here's an example of what I'm seeing in the library.

fitdump.pl::45::2
=====
    if ($desc->{message_name} ne '') {
      print "    {\"$desc->{message_name}\": {\n";
    } else {
      print "    {\"unknown$desc->{message_number}\": {\n";
    }

Right now, I have modified this to at lease consistently apply the message number to the message name. To make it easier for parsing, I've also delimited it with "__".

fitdump.pl::45::2
=====
    if ($desc->{message_name} ne '') {
      print "    {\"$desc->{message_name}__$desc->{message_number}\": {\n";
    } else {
      print "    {\"unknown$desc->{message_number}\": {\n";
    }

_NOTE: "unknown[message_number]" and "xxx[message_number]" appear to already be reserved prefixes for those message types.

Another option, though it won't be consistent with the raw FIT output, would be to append a "message_number" attribute.

fitdump.pl::45::2
=====
    if ($desc->{message_name} ne '') {
      print "    {\"$desc->{message_name}__$desc->{message_number}\": {\n";
      print "        {\"message_number\": $desc->{message_number}{\n";
    } else {
      print "    {\"unknown$desc->{message_number}\": {\n";
      print "        {\"message_number\": $desc->{message_number}{\n";
    }

[mrihtar]

Rethinking: Current json output is actually not so much a repack of fitdump output, but more of a converter, which provides only the minimum (useful) information. Including message numbers, lengths, data types, ... would require a different json structure. Besides, message numbers are always mapped to the same message names (see SDK/FIT.pm), so this will not add any additional information.

[jameswberry]

Interesting. Since I'm using the JSON output to programmatically construct fitsed.pl expressions to modify the .fit file, not being able to retrieve the message number from all messages in the JSON is a problem.

The fact that message_number is available for messages with xxx and unknown prefixes makes this even more frustrating. :-)