Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 33 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,24 +234,26 @@ it report `"oidc_state": true` in `GET /meta`.
`privacy_policy_url` if configured. Clients use the capabilities to decide which
features they can use.

| Capability | Meaning |
| ------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| `encrypted_sync` | [Encrypted sync](#encrypted-sync) is available |
| `collection_changes` | Collections accept [incremental changes](#incremental-changes) |
| `live_sync` | [Change notifications and device requests](#live-sync) |
| `account_sessions` | Tokens are bound to [account sessions](#account-sessions) |
| `key_pairing` | [Secure device pairing](#secure-device-pairing) |
| `bulk_sync` | Plaintext subscriptions and watch history accept bulk uploads |
| `history_page_size` | Maximum page size for plaintext watch history |
| `seen_videos`, `seen_posts`, `seen_rss_items`, `watch_stats`, `live_reminders`, `external_media_positions`, `notification_dismissals` | The matching [collection](#collections) exists |
| Capability | Meaning |
| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| `encrypted_sync` | [Encrypted sync](#encrypted-sync) is available |
| `collection_changes` | Collections accept [incremental changes](#incremental-changes) |
| `live_sync` | [Change notifications and device requests](#live-sync) |
| `account_sessions` | Tokens are bound to [account sessions](#account-sessions) |
| `key_pairing` | [Secure device pairing](#secure-device-pairing) |
| `bulk_sync` | Plaintext subscriptions and watch history accept bulk uploads |
| `history_page_size` | Maximum page size for plaintext watch history |
| `seen_videos`, `seen_posts`, `seen_rss_items`, `watch_stats`, `live_reminders`, `external_media_positions`, `notification_dismissals` | The matching [collection](#collections) exists |
| `separate_profiles` | [Separate profiles](#separate-profiles) have their own collections |

### Encrypted sync

| Endpoint | Purpose |
| ------------------------------------- | ------------------------------------- |
| `GET /v1/encrypted_sync` | Manifest of collections and revisions |
| `GET /v1/encrypted_sync/{collection}` | Download one collection |
| `PUT /v1/encrypted_sync/{collection}` | Upload one collection |
| Endpoint | Purpose |
| ---------------------------------------- | -------------------------------------- |
| `GET /v1/encrypted_sync` | Manifest of collections and revisions |
| `GET /v1/encrypted_sync/{collection}` | Download one collection |
| `PUT /v1/encrypted_sync/{collection}` | Upload one collection |
| `DELETE /v1/encrypted_sync/{collection}` | Delete a separate profile's collection |

- Each collection has its own revision. Uploading with a stale revision fails
with `409` without blocking other collections.
Expand All @@ -268,6 +270,7 @@ features they can use.
| `settings` | 2 MiB | |
| `notificationDismissals` (read and removed notifications) | 2 MiB | `notification_dismissals` |
| `profiles`, `sessions`, `sessionsV2` | 8 MiB | |
| `separateProfiles` | 8 MiB | `separate_profiles` |
| `watchStats` (per-device totals) | 8 MiB | `watch_stats` |
| `liveReminders` | 8 MiB | `live_reminders` |
| `subscriptions`, `playlistBookmarks` | 16 MiB | |
Expand All @@ -281,6 +284,21 @@ All collections of an account, including pending changes, are limited to
128 MiB in total. `seenPosts` is separate from `seenVideos` so older clients
that sync `seenVideos` cannot erase post marks.

#### Separate profiles

Clients store the subscriptions, history, playlists and watch statistics of a
separate profile in `subscriptions.<scope>`, `history.<scope>`,
`playlists.<scope>` and `watchStats.<scope>`. Each has the size limit of its
base collection. `<scope>` is the profile ID in lowercase hex, up to 128
digits. `separateProfiles` holds the names and colors of these profiles. Older
clients never read these collections, so they can't mix a separate profile's
data into their own.

`DELETE /v1/encrypted_sync/{collection}` deletes one of a separate profile's
collections with its pending changes after the profile is deleted. It returns
`204` even when the collection does not exist, and `400` for every other
collection.

### Incremental changes

With `collection_changes`, clients can upload a small encrypted change instead
Expand Down
26 changes: 26 additions & 0 deletions src/database/encrypted_sync.rs
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,32 @@ pub async fn save(
.await
}

/// Delete a collection and its appended changes. Returns whether it existed.
pub async fn delete(
conn: &mut DbConnection,
owner_id: &str,
collection_name: &str,
) -> Result<bool, DbError> {
conn.transaction(async |conn| {
diesel::delete(
encrypted_sync_change::table
.filter(encrypted_sync_change::account_id.eq(owner_id))
.filter(encrypted_sync_change::collection.eq(collection_name)),
)
.execute(conn)
.await?;
let deleted = diesel::delete(
encrypted_sync
.filter(account_id.eq(owner_id))
.filter(collection.eq(collection_name)),
)
.execute(conn)
.await?;
Ok(deleted > 0)
})
.await
}

/// Read a collection for a client holding revision `since`. When the change log
/// covers that revision, only the later changes are returned. Otherwise the
/// full payload is returned with every change appended after it.
Expand Down
2 changes: 2 additions & 0 deletions src/dto.rs
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ pub struct SyncCapabilities {
pub live_sync: u8,
/// Collections accept appended changes and incremental reads.
pub collection_changes: u8,
/// Separate profiles have their own collections and can delete them.
pub separate_profiles: u8,
}

#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
Expand Down
166 changes: 166 additions & 0 deletions src/handlers/encrypted_sync.rs
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ impl ScopedHandler for EncryptedSyncHandler {
.service(get_legacy_encrypted_sync)
.service(get_encrypted_sync_collection)
.service(put_encrypted_sync_collection)
.service(delete_encrypted_sync_collection)
.service(append_encrypted_sync_change)
}
}
Expand All @@ -89,16 +90,42 @@ pub(crate) fn sync_capabilities() -> SyncCapabilities {
notification_dismissals: 1,
live_sync: 1,
collection_changes: 1,
separate_profiles: 1,
}
}

/// Collections that a separate profile keeps for itself, as `<base>.<scope>`.
const SCOPED_COLLECTIONS: [&str; 4] = ["history", "playlists", "subscriptions", "watchStats"];
/// Clients hex-encode the profile ID, which has at most 64 bytes.
const MAX_SCOPE_LENGTH: usize = 128;

/// The base collection of a separate profile's collection, such as `history`
/// for `history.<scope>`. The scope is lowercase hex, so it never contains
/// another separator or a path segment.
fn scoped_collection_base(collection: &str) -> Option<&str> {
let (base, scope) = collection.split_once('.')?;
let valid_scope = !scope.is_empty()
&& scope.len() <= MAX_SCOPE_LENGTH
&& scope.len() % 2 == 0
&& scope
.bytes()
.all(|byte| matches!(byte, b'0'..=b'9' | b'a'..=b'f'));
(valid_scope && SCOPED_COLLECTIONS.contains(&base)).then_some(base)
}

fn collection_limit(collection: &str) -> HandlerResult<usize> {
// A separate profile's collection has the limits of its base collection.
if let Some(base) = scoped_collection_base(collection) {
return collection_limit(base);
}
match collection {
"settings" => Ok(2 * MEBIBYTE),
// Keys of the notifications read or removed on any device, capped by clients.
"notificationDismissals" => Ok(2 * MEBIBYTE),
// `playbackSpeeds` remains readable for migration into `settings`.
"sessions" | "sessionsV2" | "profiles" | "playbackSpeeds" => Ok(8 * MEBIBYTE),
// Names and colors of the profiles that keep their data separate.
"separateProfiles" => Ok(8 * MEBIBYTE),
// Per-device watch time totals are merged by clients.
"watchStats" | "liveReminders" => Ok(8 * MEBIBYTE),
// Playback positions of media opened by URL instead of from YouTube.
Expand Down Expand Up @@ -320,6 +347,31 @@ async fn put_encrypted_sync_collection(
}))
}

/// Deletes a separate profile's collection with its changes, once the profile
/// is deleted. Other collections are shared by every profile and stay.
#[utoipa::path(responses((status = NO_CONTENT)), security(("api_jwt_token" = [])))]
#[delete("/{collection}")]
async fn delete_encrypted_sync_collection(
account: Account,
pool: WebData,
collection: web::Path<String>,
) -> HandlerResult<impl Responder> {
let collection = collection.into_inner();
if scoped_collection_base(&collection).is_none() {
return Err(HandlerError::ValidationErrorWithContext(
"only separate profile collections can be deleted".to_owned(),
));
}
let mut conn = get_db_conn!(pool);
let deleted = encrypted_sync::delete(&mut conn, &account.id, &collection)
.await
.map_err(HandlerError::database)?;
if deleted {
notify(&account.id, None);
}
Ok(HttpResponse::NoContent().finish())
}

#[utoipa::path(request_body = PutEncryptedSync, responses((status = OK, body = EncryptedSyncCollectionResponse)), security(("api_jwt_token" = [])))]
#[post("/{collection}/changes")]
async fn append_encrypted_sync_change(
Expand Down Expand Up @@ -593,6 +645,34 @@ mod tests {
assert_eq!(super::sync_capabilities().seen_videos, 1);
assert!(collection_limit("unknown").is_err());
}

#[test]
fn separate_profile_collections_have_their_base_limits() {
assert_eq!(super::sync_capabilities().separate_profiles, 1);
assert_eq!(collection_limit("separateProfiles").unwrap(), 8 * MEBIBYTE);
assert_eq!(collection_limit("history.6162").unwrap(), 64 * MEBIBYTE);
assert_eq!(collection_limit("playlists.6162").unwrap(), 64 * MEBIBYTE);
assert_eq!(
collection_limit("subscriptions.6162").unwrap(),
16 * MEBIBYTE
);
assert_eq!(collection_limit("watchStats.6162").unwrap(), 8 * MEBIBYTE);
let longest = format!("history.{}", "ab".repeat(64));
assert!(collection_limit(&longest).is_ok());
for invalid in [
"history.",
"history.616",
"history.ABCD",
"history.61.62",
"history.61/62",
"settings.6162",
"profiles.6162",
"separateProfiles.6162",
] {
assert!(collection_limit(invalid).is_err(), "{invalid}");
}
assert!(collection_limit(&format!("history.{}", "ab".repeat(65))).is_err());
}
}

#[cfg(all(test, feature = "sqlite"))]
Expand All @@ -605,6 +685,92 @@ mod migration_tests {

use crate::{DbConnection, MIGRATIONS, models::Account};

#[actix_rt::test]
async fn deleting_a_separate_profile_collection_keeps_the_shared_ones() {
let pool = Pool::builder()
.max_size(1)
.build(AsyncDieselConnectionManager::<DbConnection>::new(
":memory:",
))
.await
.unwrap();
let account = Account {
id: "owner".into(),
name_hash: "owner-hash".into(),
password_hash: None,
oidc_sub: None,
legacy_tokens_enabled: false,
session_generation: 0,
};
{
let mut conn = pool.get().await.unwrap();
conn.spawn_blocking(|conn| {
conn.run_pending_migrations(MIGRATIONS).unwrap();
conn.batch_execute(
"INSERT INTO account (id, name_hash) VALUES ('owner', 'owner-hash');",
)?;
Ok(())
})
.await
.unwrap();
}
let app = test::init_service(
App::new().app_data(web::Data::new(pool.clone())).service(
web::scope("/sync")
.service(super::get_encrypted_sync_manifest)
.service(super::get_encrypted_sync_collection)
.service(super::put_encrypted_sync_collection)
.service(super::delete_encrypted_sync_collection),
),
)
.await;
for collection in ["history", "history.6162"] {
let request = test::TestRequest::put()
.uri(&format!("/sync/{collection}"))
.set_json(serde_json::json!({ "revision": 0, "payload": "ciphertext" }))
.to_request();
request.extensions_mut().insert(account.clone());
assert_eq!(
test::call_service(&app, request).await.status().as_u16(),
200
);
}

// Shared collections belong to every profile, so they cannot be deleted.
for collection in ["history", "settings", "history.6162"] {
let request = test::TestRequest::delete()
.uri(&format!("/sync/{collection}"))
.to_request();
request.extensions_mut().insert(account.clone());
let status = test::call_service(&app, request).await.status().as_u16();
assert_eq!(
status,
if collection == "history.6162" {
204
} else {
400
}
);
}
// Deleting it again succeeds, as a retry after a lost reply would.
let request = test::TestRequest::delete()
.uri("/sync/history.6162")
.to_request();
request.extensions_mut().insert(account.clone());
assert_eq!(
test::call_service(&app, request).await.status().as_u16(),
204
);

let request = test::TestRequest::get().uri("/sync").to_request();
request.extensions_mut().insert(account.clone());
let manifest: serde_json::Value = test::call_and_read_body_json(&app, request).await;
assert_eq!(
manifest["collections"],
serde_json::json!([{ "collection": "history", "revision": 1 }])
);
}

#[actix_rt::test]
async fn old_and_new_clients_can_alternate_collection_writes_without_losing_activity() {
let pool = Pool::builder()
Expand Down
1 change: 1 addition & 0 deletions src/handlers/health.rs
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ mod tests {
assert_eq!(json["capabilities"]["external_media_positions"], 1);
assert_eq!(json["capabilities"]["seen_rss_items"], 1);
assert_eq!(json["capabilities"]["notification_dismissals"], 1);
assert_eq!(json["capabilities"]["separate_profiles"], 1);
match &policy {
Some(url) => assert_eq!(json["privacy_policy_url"], *url),
None => assert!(json.get("privacy_policy_url").is_none()),
Expand Down
Loading