Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,21 @@ the DevConsole runtime leaked.

### Added

- **Location ping tracker.** Name the location-ping endpoints your app sends with
`DevConsoleConfig.withLocationPingTracking(...)` and DevConsole records their bodies to app-private
JSON-lines files, one track per group id and day. A new **Location pings** dashboard view replays a
track on an OpenStreetMap map, colouring each ping as sent, failed, dropped on device or offline,
with device- and fix-time gap labels, status and app-state filters, a ping list, per-ping details
and a route summary. Streams are switched on from the dashboard or from new switches on the in-app
**More** screen. Pings sent through the OkHttp interceptor are recorded automatically;
`DevConsole.recordLocationPing(...)` covers the ones that never reach the network, and
`DevConsole.isLocationPingTrackingEnabled(...)` reports whether a stream is recording. The new
`devconsole-location-ping` module holds the types and is pulled in by `devconsole` and
`devconsole-noop`. Leaflet 1.9.4 now ships with the dashboard, and the dashboard's CSP allows
images from `https://tile.openstreetmap.org`. See
[docs/LOCATION_PINGS.md](docs/LOCATION_PINGS.md).
- **`NetworkTransactionRecorder.withCompletionObserver(...)`** sees every completed request before
capture rules and redaction apply. The location ping tracker is its first user.
- **`devConsole.verificationEnabled`**, also settable as `-Pdevconsole.verification.enabled=false`.
Takes verification off `assemble`/`bundle` so a build can always get out the door, warning loudly
every time and leaving `verifyDevConsoleProtectedArtifacts` runnable by name. A plugin that can
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ Then open the URL. With the default `BrowserSecurity.NONE`, the bare address is
| **Crash & ANR capture** | Uncaught exceptions and a main-looper watchdog with bounded all-thread dumps and breadcrumbs. Always delegates to any crash reporter you already have. |
| **Push timeline** | Record FCM (or any) push messages and their lifecycle: received → displayed → opened. The Firebase adapter uses reflection — no compile-time Firebase dependency. |
| **State & feature flags** | Snapshot host-registered state and override feature flags from the browser. |
| **Location ping tracker** | Record the location-ping requests you name to files on the device and replay them on a map: every ping coloured by whether it was sent, failed, dropped on the device or never left it, with device- and fix-time gaps, filters, and a route summary. See [Location pings](docs/LOCATION_PINGS.md). |
| **Remote Config inspector** | Every Remote Config value active on the device, and where each one came from — server, in-app default, static fallback, or a local override — plus last fetch time and status. Read-only. The Firebase adapter uses reflection — no compile-time Firebase dependency. |
| **Data inspectors** | Browse and edit SharedPreferences, SQLite (incl. a SQL console), and app files. All editing capabilities enabled; read-only operation is opt-in. |
| **Evidence tray & exports** | Flag anything, attach it to a bug report bundle or Markdown/Jira/GitHub clipboard text. Export HAR, Postman Collection, or a full session ZIP. |
Expand Down Expand Up @@ -659,7 +660,7 @@ Full index: [docs/README.md](docs/README.md). Highlights:
[Push](docs/PUSH.md)
- [Data inspectors and exports](docs/DATA_INSPECTORS_AND_EXPORTS.md) ·
[Evidence and bug reports](docs/EVIDENCE_AND_BUG_REPORTS.md)
- [State and feature flags](docs/STATE_AND_FLAGS.md) · [Remote Config](docs/REMOTE_CONFIG.md)
- [State and feature flags](docs/STATE_AND_FLAGS.md) · [Remote Config](docs/REMOTE_CONFIG.md) · [Location pings](docs/LOCATION_PINGS.md)
- [Crash and ANR capture](docs/CRASH_AND_ANR.md) ·
[Background keep-alive](docs/BACKGROUND_KEEPALIVE.md)
- [FAQ / troubleshooting](docs/FAQ_TROUBLESHOOTING.md)
Expand Down
127 changes: 127 additions & 0 deletions docs/LOCATION_PINGS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Location pings

Apps that report the device's position in the background (ride, delivery, fleet and tracking apps)
send a steady stream of "location ping" requests. When a route looks wrong on the server, the
question is almost always *what did the phone actually send, and when?* The location ping tracker
answers it: DevConsole records the ping requests you name to files on the device and replays them on
a map in the browser dashboard.

- Every ping is a dot on the map, coloured by what happened to it: **sent** (2xx), **failed**
(any other HTTP status), **dropped on device** (the app filtered it out before sending) or
**offline** (it never reached the server).
- The gap between pings, by device time and by location-fix time, can be shown next to each dot.
- Filter by status and by app state (foreground or background), list the pings in order, and open
any one to see every field it carried.
- Files survive app restarts, so a tester can record a trip and you can look at it later.

Nothing is recorded until you switch a stream on, either from the dashboard or from the in-app
inspector's **More → Location pings** switches. The switch state is remembered across launches.

## Set it up

Tell DevConsole which endpoints are location pings and where the values live in their bodies:

```kotlin
DevConsole.initialize(
application,
DevConsoleConfig.default().withLocationPingTracking(
LocationPingTracking(
streams = listOf(
LocationPingStream(name = "idle", path = "/v2/driver/locations"),
LocationPingStream(name = "trip", path = "/v2/trips/locations"),
),
schema = LocationPingSchema(
pingListKey = "locations",
latitudeKey = "lat",
longitudeKey = "lng",
deviceTimeKey = "recorded_at",
fixTimeKey = "fix_time",
groupIdKey = "trip_id",
appStateKey = "app_state",
groupLabel = "Trip",
),
),
),
)
```

From Java, use `DevConsoleConfig.builder().locationPingTracking(...)`.

Requests that go through the DevConsole OkHttp interceptor are recorded automatically, whatever the
response. Pings that never reach the network have to be reported by the app:

```kotlin
DevConsole.recordLocationPing(url, body, LocationPingOutcome.DroppedOnDevice)
DevConsole.recordLocationPing(url, body, LocationPingOutcome.NoConnection)
```

`DevConsole.isLocationPingTrackingEnabled("trip")` tells you whether a stream is being recorded.
Both calls are no-ops in the release (`devconsole-noop`) build.

### Streams

| Field | Meaning |
|---|---|
| `name` | Folder name and dashboard label. 1–64 letters, digits, `.`, `_` or `-`. |
| `path` | Matched against the request path only. The host and query string are ignored, so staging, production and local hosts are all captured. A request whose path *ends with* the stream path also matches (`/api/v2/trips/locations` matches `/v2/trips/locations`); the longest matching path wins. |

### Schema

Every key is looked up on the ping object first and on the request body second. A value shared by a
whole batch, such as a group id or the app state, can sit at the top level while the coordinates sit
on each ping.

| Field | Default | Meaning |
|---|---|---|
| `pingListKey` | `null` | Array of pings in the body. `null` means the body itself is one ping. |
| `latitudeKey`, `longitudeKey` | `latitude`, `longitude` | Numbers or numeric strings. Pings without valid coordinates are skipped. |
| `deviceTimeKey` | `timestamp` | When the device created the ping. Epoch milliseconds, or epoch seconds (detected automatically). |
| `fixTimeKey` | `null` | When the location fix was taken, for the "fix time gap". |
| `groupIdKey` | `null` | Splits a stream into one track per value, such as an order or trip id. A string, a number, or an array (the first entry is used). |
| `appStateKey` | `null` | Enables the foreground/background filter. Values containing `fore` or `back` are recognised. |
| `groupLabel` | `Group` | What the dashboard calls a group, such as "Order" or "Trip". |

## Storage

Pings are appended as JSON lines to the app's private files directory:

```
files/devconsole/location-pings/<stream>/<group>-<yyyy-MM-dd>.jsonl
```

Each line holds the outcome, the time it was recorded and the request body exactly as sent. A ping
without a group id is filed under the stream name. No storage permission is needed and no other app
can read the files; pull them with `adb shell run-as <your.package> ...` if you need the raw data.

Recording stops at `maxStorageBytes` (64 MB by default). The dashboard then shows why, and deleting
tracks frees the space. Location pings are not subject to the timeline's retention policy; they stay
until you delete them.

## The dashboard

**Location pings** sits under *Traffic* in the dashboard's rail. It shows:

- **Recording** switches, one per configured stream.
- Stream and group pickers. Tracks are listed newest first with their date and size.
- Status and app-state filters, the device and fix time gap labels, and an auto-refresh toggle.
- A route summary: pings shown, delivery rate, the breakdown of undelivered pings, distance and
duration.
- The map, with the first and last ping ringed, and a ping list beside it. Selecting a row centres
the map on that ping; selecting it again opens its details.
- **Delete track** and **Delete all**, both confirmed first.

The map tiles come from OpenStreetMap, so the computer viewing the dashboard needs internet access to
draw streets. The route and pings still draw without it. Leaflet (BSD-2-Clause, licence in
`sdk/server-ktor/src/main/resources/devconsole-web/leaflet-LICENSE.txt`) ships inside the dashboard,
and `https://tile.openstreetmap.org` is the only external origin the dashboard's Content Security
Policy allows, and only for images.

## Security

- Coordinates are shown as recorded; that is the point of the feature. Every other field passes
through the same `RedactionPolicy` as network captures before it reaches the dashboard, so a
`token` or `password` field in a ping body is masked.
- Every location ping route needs a browser session. Toggling a stream and deleting data also need
the CSRF token, and are written to the command audit log.
- Everything is gated by the `NETWORK` capture category. With it off, nothing is recorded and every
route answers `403 CATEGORY_DISABLED`.
14 changes: 14 additions & 0 deletions docs/PROTOCOL_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,20 @@ has completed — plus `status` and `minimumFetchIntervalSeconds`), `unavailable
`static`, `override`, `unknown`. Gated by the `state` capture category, the same one that covers
state providers and feature flags. See [REMOTE_CONFIG.md](REMOTE_CONFIG.md).

### Location pings

`GET /api/v1/location-pings` (auth) returns `configured`, `streams` (`name`, `path`, `enabled`),
`schema` (`groupLabel`, `hasFixTime`, `hasAppState`), `storage` (`usedBytes`, `maxBytes`,
`lastError`) and `tracks` (`stream`, `file`, `groupId`, `capturedDate`, `sizeBytes`,
`lastModifiedEpochMs`), newest first. `GET /api/v1/location-pings/track?stream=&file=` (auth)
returns `track`, `truncated` and `points` (`index`, `lat`, `lng`, `deviceTime`, `deviceGapSeconds`,
`fixTime`, `fixGapSeconds`, `status` — one of `sent`, `failed`, `dropped`, `offline` —
`statusCode`, `appState`, `fields`), ordered by device time; `fields` are redacted. Mutations need
auth plus Origin/CSRF and are audited: `PUT /api/v1/location-pings/streams/{name}` with a body of
`true` or `false`, `DELETE /api/v1/location-pings/track?stream=&file=`, and
`DELETE /api/v1/location-pings` (optionally `?stream=a&stream=b`) answering `filesDeleted` and
`filesLeft`. Gated by the `network` capture category. See [LOCATION_PINGS.md](LOCATION_PINGS.md).

### Composer

`POST /api/v1/composer/execute` (auth — one-off request; `method` restricted to
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ read one page here, make it the threat model, especially before you put the dash
- [Push](PUSH.md)
- [State and feature flags](STATE_AND_FLAGS.md)
- [Remote Config](REMOTE_CONFIG.md)
- [Location pings](LOCATION_PINGS.md)
- [Crash and ANR capture](CRASH_AND_ANR.md) — all-thread dumps, breadcrumbs, `CrashPolicy`, and the
Crashes surface on both web and Android
- [Data inspectors, exports, and the More screen](DATA_INSPECTORS_AND_EXPORTS.md) — preferences,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ public final class MainActivity extends Activity {

private CoroutineScope panelScope;
private OkHttpClient client;
private SampleLocationPings sampleLocationPings;
private OkHttpClient socketClient;
private TextView lastResponse;
private DevConsolePanelController panel;
Expand Down Expand Up @@ -157,6 +158,7 @@ public final class MainActivity extends Activity {
// engine DevConsole published at initialize, so no DevConsoleMockInterceptor line is needed.
client = DevConsoleOkHttp.install(new OkHttpClient.Builder(), DevConsole.networkRecorder())
.build();
sampleLocationPings = new SampleLocationPings(client);
socketClient = new OkHttpClient();
// Mock rules are SESSION-scoped and dropped on every server restart -- restoreRunningSession
// (re)installs it on each Running transition instead, so a one-shot call here isn't needed.
Expand Down Expand Up @@ -185,6 +187,8 @@ public final class MainActivity extends Activity {
findViewById(R.id.btnSimulateFirebasePush).setOnClickListener(view -> simulateFirebasePush());
findViewById(R.id.btnReadFeatureFlag).setOnClickListener(view -> readFeatureFlag());
findViewById(R.id.btnToggleMockRule).setOnClickListener(view -> toggleMockRule());
findViewById(R.id.btnSendLocationPing).setOnClickListener(view ->
ioExecutor.execute(() -> postResult(sampleLocationPings.sendNextPing())));
findViewById(R.id.btnCaptureScreenshot).setOnClickListener(view -> captureScreenshot());
findViewById(R.id.btnTriggerCrash).setOnClickListener(view -> triggerCrash());
findViewById(R.id.btnTriggerAnr).setOnClickListener(view -> triggerAnr());
Expand Down Expand Up @@ -242,6 +246,7 @@ private DevConsoleConfig buildConfig() {
.addStateProvider(stateProvider)
.addFeatureFlag(featureFlag)
.editingCapabilities(editingCapabilities)
.locationPingTracking(SampleLocationPings.tracking())
// Composer is off by default: it makes the device issue arbitrary outbound
// requests, i.e. turns it into a proxy onto its network. Every /api/v1/composer
// route (GETs included) answers 404 {"code":"COMPOSER_DISABLED"} until this is on.
Expand Down
Loading
Loading