rmorawski/standardnotes-app-web
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: docs build (#1203)

Browse Source
This commit is contained in:
Mo
2022-07-04 15:42:47 -05:00
committed by GitHub
parent f11b8e163c
commit 66d5dce5e1
267 changed files with 43 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files

303
packages/docs/docs/specification/auth.md Normal file
View File

@@ -0,0 +1,303 @@
---
id: auth
title: Auth API
sidebar_label: Auth
description: Authentication portion of the Standard Notes API.
keywords:
- standard notes
- docs
- notes app
- end-to-end encryption
- sync
image: /img/logo.png
hide_title: false
hide_table_of_contents: false
---
### POST /auth
**Registers a new user**
| | | |
|-------------|---------|---------------------------------------------------------------------------------------------------------------------|
| api | `string` | The API version to use. Must not be null, otherwise oldest API version used. Latest value is "20200115" |
| created | `string` | Integer timestamp representing the date the client generated the account's encryption keys. Example "1622494310383" |
| email | `string` | The account email. |
| ephemeral | `boolean` | Whether the initial session created for this account is ephemeral ("Stay signed in" unchecked). |
| identifier | `string` | Should equal the account email. |
| origination | `string` | Should equal "registration" |
| password | `string` | The server password generated by the client from the user's master password. |
| pw_nonce | `string` | The nonce generated by the client for the user's encryption key. |
| version | `string` | The protocol version the client used to generate the user's encryption key. Latest is "004". |
#### Response
```JSON
{
"session": {
"access_token": "1:457b1f4b-88c2-4328-bf32-fe7dd9431d62:WwKgTEDUoEhxECu6",
"refresh_token": "1:457b1f4b-88c2-4328-bf32-fe7dd9431d62:G-qEoZsFtoj~RLL-",
"access_expiration": 1627678312000,
"refresh_expiration": 1654051238000
},
"key_params": {
"created": "1622494310383",
"identifier": "foo@example.com",
"origination": "registration",
"pw_nonce": "d97ed41c581fe8c3e0dce7d2ee72afcb63f9f461ae875bae66e30ecf3d952900",
"version": "004"
},
"user": {
"uuid": "682f9deb-b75f-4d97-91fa-6fd82a482db1",
"email": "foo@example.com"
}
}
```
### POST /auth/sign_in
**Authenticates a user and returns a session.**
| | | |
|-------------|---------|---------------------------------------------------------------------------------------------------------------------|
| api | `string` | The API version to use. Must not be null, otherwise oldest API version used. Latest value is "20200115" |
| email | `string` | The account email. |
| ephemeral | `boolean` | Whether the session created for this account is ephemeral ("Stay signed in" unchecked). |
| password | `string` | The server password generated by the client from the user's master password. |
#### Response
```JSON
{
"session": {
"access_token": "1:ee03808d-dd05-417c-9c87-d471e49bcc49:Q~-UoIpDhdtkii_t",
"refresh_token": "1:ee03808d-dd05-417c-9c87-d471e49bcc49:Jn8UWzeHx2H5nZI5",
"access_expiration": 1627678779000,
"refresh_expiration": 1654051705000
},
"key_params": {
"created": "1622494310383",
"identifier": "foo@example.com",
"origination": "registration",
"pw_nonce": "d97ed41c581fe8c3e0dce7d2ee72afcb63f9f461ae875bae66e30ecf3d952900",
"version": "004"
},
"user": {
"uuid": "682f9deb-b75f-4d97-91fa-6fd82a482db1",
"email": "foo@example.com"
}
}
```
### GET /auth/params
**Queries the parameters used for key generation for an email address. Queried before signing into an account.**
| | | |
|-------------|---------|---------------------------------------------------------------------------------------------------------------------|
| api | `string` | The API version to use. Must not be null, otherwise oldest API version used. Latest value is "20200115" |
| email | `string` | The account email. |
#### Response
```JSON
{
"identifier": "foo@example.com",
"pw_nonce": "d97ed41c581fe8c3e0dce7d2ee72afcb63f9f461ae875bae66e30ecf3d952900",
"version": "004"
}
```
### POST /auth/change_pw
**Changes a user's password.**
| | | |
|-------------|---------|---------------------------------------------------------------------------------------------------------------------|
| api | `string` | The API version to use. Must not be null, otherwise oldest API version used. Latest value is "20200115" |
| created | `string` | Integer timestamp representing the date the client generated the account's new encryption keys. Example "1622494310383" |
| identifier | `string` | The account email. |
| origination | `string` | Should equal "password-change" |
| current_password | `string` | The old server password generated by the client from the user's master password. |
| new_password | `string` | The new server password generated by the client from the user's master password. |
| pw_nonce | `string` | The nonce generated by the client for the user's encryption key. |
| version | `string` | The protocol version the client used to generate the user's encryption key. Latest is "004". |
#### Response
```JSON
{
"session": {
"access_token": "1:27d4fd8f-b730-4e0a-afd3-1600fb466aaa:HMgmZwV5k5ePt0vj",
"refresh_token": "1:27d4fd8f-b730-4e0a-afd3-1600fb466aaa:wHuTeOB-qWGP3AE3",
"access_expiration": 1627679129000,
"refresh_expiration": 1654052055000
},
"key_params": {
"created": "1622494310383",
"identifier": "foo@example.com",
"origination": "password-change",
"pw_nonce": "be1974ff6fb1c541aa8c71fd3c66851b6492cf224b661c72daf44e0bef3096bb",
"version": "004"
},
"user": {
"uuid": "682f9deb-b75f-4d97-91fa-6fd82a482db1",
"email": "foo@example.com"
}
}
```
## Sessions
### Session Model
| Field | Type | Description |
|----------------|-----------|---------------------------------------------------|
| uuid | `string` | Unique identifier of the session |
| user_uuid | `string` | Unique identifier of the user |
| user_agent | `string` | The user agent used to create the session |
| api_version | `string` | The server API version used to create the session |
| access_token | `string` | The access token used to authenticate requests |
| refresh_token | `string` | The refresh token used to extend tokens |
| access_expiration | `datetime` | The expiration time of the access token |
| refresh_expiration | `datetime` | The expiration time of the refresh token |
| created_at | `datetime` | Date and time of creation of the session |
| updated_at | `datetime` | Last updated date and time of the session |
- Each `session` includes the API version they were created with. This way we can deny sessions for a given API version if we detect a vulnerability with that version in the future
- Sessions are created in the following cases:
- When a user signs in
- When a user registers a new account
---
### Authenticated requests
The `Authorization` request header field is used by clients to make authenticated request. `Bearer` is the only authentication scheme allowed.
The client must send the `access` token generated by the session, in the `Authorization` header. For example:
```
GET /sessions HTTP/1.1
Host: sync.standardnotes.org
Authorization: Bearer <access token>
```
---
Below is a list of endpoints related to session management:
| Method | URL | Params | Description | Successful response code |
|-----------|--------------------------|---------------------|--------------------------------------------------------|--------------------------|
|`POST` | /auth/sign_out | *None* | Terminates the current session | `204` |
|`DELETE` | /session | **uuid** | Terminates the specified session by UUID | `204` |
|`DELETE` | /sessions | *None* | Terminates all sessions, except the current one | `204` |
|`GET` | /sessions | *None* | Lists all sessions active sessions | `200` |
|`POST` | /session/token/refresh | **refresh_token** | Obtains new pair of `access_token` and `refresh_token` | `200` |
A successful request to `GET /sessions` returns the following JSON response:
```json
{
"sessions": [
{
"uuid": "xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx",
"user_agent": "<product> / <product-version> <comment>",
"api_version": "xxxxyyzz",
"current": "<boolean>",
"created_at": "2020-01-01T00:00:00.000Z"
}
...
]
}
```
#### Obtaining tokens
Tokens can be obtained every time the user performs any of the following actions:
1. When a user signs in
1. When a user registers an account
1. When the tokens are refreshed
#### Refreshing tokens
When an expired `access_token` is provided in the `Authorization` HTTP header, the following JSON response is returned:
HTTP Status Code: `498 Expired Access Token`
```json
{
"error": {
"tag": "expired-access-token",
"message": "The provided access token has expired."
}
}
```
To continue accessing resources, the `access_token` must be refreshed. That is, the current `access_token` is replaced with a new one with an extended expiration date.
To refresh an `access_token`, a valid `refresh_token` is needed. This `refresh_token` must meet the following requirements:
- It should belong to the session of the `access_token`
- It should not be expired
Since the `refresh_token` is of single-use, a new `refresh_token` is obtained when the `access_token` is refreshed.
Refreshing tokens is a process that is transparent to the user, meaning that the client will perform the requests to keep valid tokens without user intervention.
Here's how to refresh tokens:
1. Send a `POST` request to `/session/token/refresh`. The body should contain a JSON paylaod with the `refresh_token`:
```
POST /session/token/refresh HTTP/1.1
Host: sync.standardnotes.org
Authorization: Bearer <access token>
{
"refresh_token": "R_xxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx"
}
```
1. The `refresh_token` is validated on the server. Depending of the circumstances, there should be two outcomes:
1. The provided `refresh_token` is valid. If so, a new pair of tokens is generated and the following JSON response is returned:
```json
{
"token": "xxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx",
"session": {
"refresh_expiration": 1583020800,
"refresh_token": "xxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx",
}
}
```
1. The provided `refresh_token` is expired. If so, the following JSON response is returned:
HTTP Status Code: `400 Bad Request`
```json
{
"error": {
"tag": "expired-refresh-token",
"message": "The refresh token has expired."
}
}
```
User must start a new session by re-entering their credentials.
### Expiration
#### Sessions
Sessions should be terminated after a period of inactivity. This is for best security practices.
`Long-lived sessions` are a good choice for our use case, because it can build a better user experience than expiring sessions for a short idle-timeout.
- A `session` that remains inactive for `1 year` will be terminated, along with all `tokens`
#### Tokens
| Name | Type | Expiration |
|-----------|-------------|---------------|
| `access` | Short-lived | `60 days` |
| `refresh` | Long-lived | `1 year` |
- A `refresh` token is of single-use, while an `access` token can be used as long as it is not expired

162
packages/docs/docs/specification/encryption-003.md Normal file
View File

@@ -0,0 +1,162 @@
---
slug: encryption/003
id: encryption-003
title: Encryption Protocol Specification v003
sidebar_label: Encryption v003
description: Specification for the Standard Notes end-to-end encryption.
keywords:
- standard notes
- docs
- notes app
- end-to-end encryption
- encryption specification
image: /img/logo.png
hide_title: false
hide_table_of_contents: false
---
## Version 0.0.3
It is important that there exist a separation of concerns between the server and the client. That is, the client should not trust the server, and vice versa.
Encryption keys are generated by stretching the user's input password using a [key derivation function.](https://en.wikipedia.org/wiki/Key_derivation_function)
The resulting key is split in three — the first third is sent to the server as the user's password, the second third is saved locally as the user's master encryption key, and the last third is used as an authentication key. In this setup, the server is never able to compute the encryption key or the user's original password given just a fraction of the resulting key.
Note: client-server connections must be made securely through SSL/TLS.
#### Elaboration on User model encryption related fields
| name | details |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| pw_cost | The number of iterations to be used by the KDF. The minimum for version 003 is 100,000. However note that non-native clients (web clients not using WebCrypto) will not be able to handle any more than 3,000 iterations. |
| pw_nonce | A nonce for password derivation. This value is initially created by the client during registration. |
## Key Generation
### Client Instructions
Given a user inputted password `uip`, the client's job is to generate a password `pw` to send to the server, a master key `mk` that the user stores locally to encrypt/decrypt data, and an auth key `ak` for authenticating encrypted data.
#### Login Steps
1. Client makes GET request with user's email to `auth/params` to retrieve password nonce, cost, and version.
1. Client verifies cost >= minimum cost (100,000 for 003.)
1. Client computes `pw`, `mk`, and `ak` using PBKDF2 with SHA512 as the hashing function and output length of 768 bits:
```
salt = SHA256:Hexdigest([email, "SF", version, pw_cost, pw_nonce].join(":"))
key = pbkdf2(uip, salt, sha512, 768, pw_cost) // hex encoded
pw = key.substring(0, key.length/3)
mk = key.substring(key.length/3, key.length/3)
ak = key.substring(key.length/3 * 2, key.length/3)
```
1. Client sends `pw` to the server as the user's "regular" password and stores `mk` and `ak` locally. (`mk` and `ak` are never sent to the server).
#### Registration Steps
1. Client chooses default for `pw_cost` (minimum 100,000).
1. Client generates `pw_nonce`:
```
pw_nonce = random_string(256)
```
1. Client computes `pw`, `mk`, and `ak` using step (3) from Login Steps.
1. Client registers with `email`, `pw`, `pw_cost`, `pw_nonce`, and `version`.
## Item Encryption
In general, when encrypting a string, one should use an IV so that two subsequent encryptions of the same content yield different results, and one should authenticate the data as to ascertain its authenticity and lack of tampering.
In Standard Notes, two strings are encrypted for every item:
- The item's `content`.
- The item's `enc_item_key`.
## Client-side
An item is encrypted using a random key generated for each item.
### Encryption:
Note that when encrypting/decrypting data, keys should be converted to the proper format your platform function supports. It's best to convert keys to binary data before running through any encryption/hashing algorithm.
For every item:
1. Generate a random 512 bit key `item_key` (in hex format).
2. Split `item_key` in half; set item encryption key `item_ek = first_half` and item authentication key `item_ak = second_half`.
3. Encrypt `content` using `item_ek` and `item_ak` following the instructions "Encrypting a string using the 003 scheme" below and send to server as `content`.
4. Encrypt `item_key` using the global `mk` and global `ak` following the instructions "Encrypting a string using the 003 scheme" below and send to server as `enc_item_key`.
### Decryption:
Check the first 3 characters of the `content` string. This will be the encryption version.
- If it is equal to "001", which is a legacy scheme, decrypt according to the 001 instructions found [here](https://github.com/standardfile/standardfile.github.io/blob/master/doc/spec-001.md).
- If it is equal to "002" or "003", decrypt as follows:
1. Decrypt `enc_item_key` using the global `mk` and global `ak` according to the "Decrypting a string using the 003 scheme" instructions below to get `item_key`.
2. Split `item_key` in half; set encryption key `item_ek = first_half` and authentication key `item_ak = second_half`.
3. Decrypt `content` using `item_ek` and `item_ak` according to the "Decrypting a string using the 003 scheme" instructions below.
### Encrypting a string using the 003 scheme:
Given a `string_to_encrypt`, an `encryption_key`, and an `auth_key`:
1. Generate a random 128 bit string called IV.
1. Encrypt `string_to_encrypt` using `AES-CBC-256:Base64`, `encryption_key`, and `IV`:
```
ciphertext = AES-Encrypt(string_to_encrypt, encryption_key, IV)
```
1. Generate `string_to_auth` by combining the encryption version (003), the item's UUID, the IV, and the ciphertext using the colon ":" character:
```
string_to_auth = ["003", uuid, IV, ciphertext].join(":")
```
1. Compute `auth_hash = HMAC-SHA256:Hex(string_to_auth, auth_key)`.
1. Generate the final result by combining the five components into a `:` separated string:
```
result = ["003", auth_hash, uuid, IV, ciphertext].join(":")
```
### Decrypting a string using the 003 scheme:
Given a `string_to_decrypt`, an `encryption_key`, and an `auth_key`:
1. Split the string into its constituent parts: `components = string_to_decrypt.split(":")`.
1. Assign local variables:
```
version = components[0]
auth_hash = components[1]
uuid = components[2]
IV = components[3]
ciphertext = components[4]
```
1. Ensure that `uuid == item.uuid`. If not, abort decryption.
1. Generate `string_to_auth = [version, uuid, IV, ciphertext].join(":")`.
1. Compute `local_auth_hash = HMAC-SHA256(string_to_auth, auth_key)`. Compare `local_auth_hash` to `auth_hash`. If they are not the same, skip decrypting this item, as this indicates that the string has been tampered with.
1. Decrypt `ciphertext` to get final result: `result = AES-Decrypt(ciphertext, encryption_key, IV)`.
## Server-side
For every received item:
1. (Optional but recommended) Encrypt `content` using server known key and store. Decrypt before sending back to client.
## Next Steps
Join the [Slack group](https://standardnotes.com/slack) to discuss implementation details and ask any questions you may have.
You can also email [help@standardnotes.org](mailto:help@standardnotes.org).
Follow [@standardnotes on Twitter](https://twitter.com/standardnotes) for updates and announcements.

345
packages/docs/docs/specification/encryption-004.md Normal file
View File

@@ -0,0 +1,345 @@
---
slug: encryption
id: encryption
title: Client Encryption API
sidebar_label: Client Encryption
description: Specification for the Standard Notes end-to-end encryption.
keywords:
- standard notes
- docs
- notes app
- end-to-end encryption
- encryption specification
image: /img/logo.png
hide_title: false
hide_table_of_contents: false
---
The 004 protocol upgrade centers around a system that makes it easy and painless to upgrade to a future protocol version, as well as more modern cryptographic primitives.
This page is a copy of the specification file located at [github.com/standardnotes/snjs](https://github.com/standardnotes/snjs/blob/master/packages/snjs/specification.md).
## Introduction
The Standard Notes Protocol describes a set of procedures that ensure client-side encryption of data in such a way that makes it impossible for the server, which houses the data, to read or decrypt the data. It treats the server as a dumb data-store that simply saves and returns values on demand.
Even in scenarios when the server is under active attack, clients should be fully protected, and cannot be tricked into revealing any sensitive information.
The client and server communicate under two common procedures: authentication, and syncing.
Authentication is a one-time transfer of information between client and server. In short, clients generate a long secret key by stretching a user-inputted password using a KDF. The first half of that key is kept locally as the "master key" and is never revealed to the server. The second half of that key is sent to the server as the "account server password".
The master key is then used to encrypt an arbitrary number of items keys. Items keys are generated randomly and not based on the account password. Items keys are used to encrypt syncable data, like notes, tags, and user preferences. Items keys themselves are also synced to user accounts, and are encrypted directly with the master key.
When a user's master key changes, all items keys must be re-encrypted with the new master key. Accounts should generally have one items key per protocol version, so even in the event where many protocol upgrades are created, only a few KB of data must be re-encrypted when a user's credentials change (as opposed to completely re-encrypting many megabytes or gigabytes of data).
Data is also encrypted client-side for on-device storage. When an account is present, all local data is encrypted by default, including simple key-value storage (similar to a localStorage-like store). Persistence stores are always encrypted with the account master key, and the master key is stored in the device's secure keychain (when available).
Clients also have the option of configuring an application passcode, which wraps the account master key with an additional layer of encryption. Having a passcode enabled is referred to as having a "root key wrapper" enabled. When a root key is wrapped, it is stored in local storage as an encrypted payload, and the keychain is bypassed. This allows for secure key storage even in environments that don't expose a keychain, such as web browsers.
This document delineates client-side procedures for key management and generation, data encryption, and storage encryption. Concepts related to server syncing and server session management are outside the scope of this document. This document however wholly covers any values that a server would receive, so even though syncing and server session management is out of scope, the procedures outlined in this document should guarantee that no secret value is ever revealed to the server.
## Key Management
**There are three main concepts as related to keys:**
1. **A root key**—based on an account's user-inputted password. There exists only one root key per account.
2. **A root key wrapper**—_wraps_ a root key (encrypts it) with an additional layer. This is a local-only construct, and translates directly as an "application passcode" feature.
3. **Items keys**—used to encrypt items. There can exist many items keys, and one items key can encrypt many items. Each items key is encrypted with the root key. When the root key changes, all items keys must be re-encrypted using the new root key.
### Key Generation Flow
1. User registers with an email (`identifier`) and a `password`.
2. `password` is run through a KDF to generate a key, which is then split in two, as part of a single `rootKey`.
1. The first half is the `masterKey`.
2. The second half is the `serverPassword`.
3. Client registers user account with server using `email` and `rootKey.serverPassword`.
4. Client creates new random key `itemsKey`. This key is encrypted directly with `rootKey.masterKey`, and the encrypted `itemsKey` is assigned a UUID and uploaded to the user's account. (Each `itemsKey` is a traditional item, just like a note or tag.)
### Password change or protocol upgrade flow
**When a user changes their password, or when a new protocol version is available:**
1. Client generates new `rootKey` using account identifier and password, and thus generates new `rootKey.masterKey`, `rootKey.serverPassword`, and `keyParams`, which include the protocol version and other public information used to guide clients on generating the `rootKey` given a user password.
2. Client submits new `rootKey.serverPassword` and `keyParams` to server. Note that the changing the `serverPassword` does not necessarily invalidate a user's session. Sessions management is outside of the scope of this document.
3. Client loops through all `itemsKeys` and re-encrypts them with new `rootKey.masterKey`. All `itemsKeys` are then re-uploaded to server. Note that `itemsKeys` are immutable and their inner key never changes. The key is only re-encrypted using the new `masterKey`.
This flow means that when a new protocol version is available or when a user changes their password, we do not need to re-encrypt all their data, but instead only a handful of keys.
### Key Rotation
By default, upgrading an account's protocol version will create a new `itemsKey` for that version, and that key will be used to encrypt all data going forward. To prevent large-scale data modification that may take hours to complete, any data encrypted with a previous `itemsKey` will be re-encrypted with the new `itemsKey` progressively, and not all at once. This progressive re-encryption occurs when an item is explicitly modified by the user. Applications can also be designed to bulk-modify items during idle-capacity, without user interaction.
**When changing the account password:**
- If a new protocol version is available, changing the account password will also upgrade to the latest protocol version and thus generates a new default `itemsKey`.
- If no new protocol version is available, or if the user is already using the latest version, changing the account password generates a new `rootKey`, as well as generates a new `itemsKey`. The new `itemsKey` will be used as the default items encryption key, and will also be used to progressively re-encrypt previous data. Generating a new `itemsKey` on password change ensures backward secrecy in the case the previous account password is compromised.
## Encryption Flow
_For each_ item (such as a note) the client wants to encrypt:
1. Client generates random `item_key` (note: singular. Not related to `itemsKey`).
2. Client encrypts note content with `item_key` to form `content`.
3. Client encrypts `item_key` with default `itemsKey` as `enc_item_key`.
4. Client notes `itemsKey` UUID and associates it with encrypted item payload as `items_key_id`, and uploads payload to server.
To decrypt an item payload:
1. Client retrieves `itemsKey` matching `items_key_id` of payload.
2. Client decrypts item's `enc_item_key` with `itemsKey` to form `item_key`.
3. Client decrypts item's `content` using `item_key`.
## Authentication
Registering for an account involves generating a `rootKey` and respective `keyParams`, according to the key generation flow above. The key parameters are uploaded to the server, and include:
- unique identifier (email)
- salt seed
- protocol version
To sign into an account, clients first make a request to the server to retrieve the key params for a given email. This endpoint is public and non-authenticated (unless the account has two-factor authentication enabled). The client then uses the retrieved key params to generate a `rootKey`, and uses the `rootKey.serverPassword` to authenticate the account.
Note that by default, the client trusts the protocol version the server reports. The client uses this protocol version to determine which cryptographic primitives (and their parameters) to use for key generation. This raises the question of, what happens if a malicious server underreports an account's version in order to weaken key generation parameters? For example, if a user's account is 004, but the server reports 002, the client will proceed to generate a `serverPassword` using outdated primitives.
There are two safeguards against this scenario:
1. Older protocol versions are expired and become no longer supported after a certain period.
2. Clients may sign in with a flag known as "strict sign in" (SSI). SSI ensures that the client _always_ signs in with the client-side _hardcoded latest version_ of the protocol. For example, if a client with SNJS 004 support attempts to sign in with SSI enabled, and the server reports a protocol version of 002 for a given account, the client will refuse this sign-in, and will not proceed with key generation. SSI is a user-controlled option. Clients cannot be programmed to default to SSI, as otherwise, users would be unable to sign in to their account whenever a new protocol version is available.
## Root Key Wrapping
Root key wrapping is a local-only construct that pertains to how the root key is stored locally. By default, and with no root key wrapping, the `rootKey` is stored in the secure device keychain. Only the `rootKey.masterKey` is stored locally; the `rootKey.serverPassword` is never stored locally, and is only used for initial account registration. If no keychain is available (web browsers), the `rootKey` is stored in storage in necessarily plain format.
Root key wrapping allows the client to encrypt the `rootKey` before storing it to disk. Wrapping a root key consists of:
1. Client asks user to choose a "local passcode".
2. The local passcode is run through the same key generation flow as account registration (using a random UUID as the account identifier, in place of an email) to generate a separate new root key known as the `rootKeyWrappingKey` (which likewise consists of a `masterKey` and an unused `serverPassword`).
3. The `rootKeyWrappingKey` is used to encrypt the `rootKey` as `wrappedRootKey`. The `wrappedRootKey` (along with `wrappingKeyKeyParams`) is stored directly in storage, and the keychain is cleared of previous unwrapped `rootKey`. (Some keychains have fixed payload size limit, so an encrypted payload may not always fit. For this reason `wrappedRootKey` is always stored directly in storage.)
**To unwrap a root key:**
1. Client displays an "Enter your local passcode" prompt to user.
2. Client runs user-inputted password through key generation scheme (using stored `wrappingKeyKeyParams`) to generate a temporary `rootKeyWrappingKey`.
3. Client attempts to decrypt `wrappedRootKey` using `rootKeyWrappingKey`. If the decryption process succeeds (no errors are thrown), the client successfully unlocks application, and keeps the unwrapped `rootKey` in application memory to aid in encryption and decryption of items (or rather `itemsKeys`, to be exact).
**The purpose of root key wrapping is many-fold:**
1. To allow for secure storage of root key when no secure keychain is available (i.e web browsers).
2. Even in cases when a keychain is available, root key wrapping allows users to choose an arbitrary password to protect their storage with.
3. To allow for encryption of local storage.
4. To allow applications to introduce cryptographically-backed UI-level app locking.
When a root key is wrapped, no information about the wrapper is persisted locally or in memory beyond the `keyParams` for the wrapper. This includes any sort of hash for verification of the correctness of the entered local passcode. That is, when a user enters a local passcode, we know it is correct not because we compare one hash to another, but by whether it succeeds in decrypting some encrypted payload.
## Multi-Client Root Key Changes
Because account password changes (or, in general, root key changes) require all existing items keys to be re-encrypted with the new root key, it is possible that items keys eventually fall into an inconsistent state, such that some are encrypted with a newer root key, while others are encrypted with the new root key. Clients encountering an items key they cannot encrypt with the current account root key parameters would then reach a dead end, and users would see undecryptable data.
To recover the ability to decrypt an items key, clients can use the `kp` (key params) included the items key's authenticated_data payload. These parameters represent the the key params of the root key used to encrypt this items key.
For example, when the account password changes, and thus the root key changes, all items keys are re-encrypted with the new root key on client A. Another client (client B) who may have a valid API session, but an outdated root key, will be able to download these new items keys. However, when client B attempts to decrypt these keys using its root key, the decryption will fail. Client B enters a state where it can save items to the server (wherein those items are encrypted using its existing default readable items key), but cannot read new data encrypted with items keys encrypted with client A's root key.
When client B connects to the API with a valid session token, but an outdated root key, it will be able to download new items keys, but not yet decrypt them. However, since the key parameters for the root key underlying the items key is included in the encrypted payload, the client will be able to prompt the user for their new password.
**In general,**
A. When a client encounters an items key it cannot decrypt, whose created date is greater than any existing items key it has, it will:
1. Make an authenticated request to the server to retrieve the account's current key parameters (because we suspect that they may have changed, due to the above fact). Authenticated requests to the GET key_params endpoint bypasses the MFA requirement.
2. Verify that the incoming key params version is greater than or equal to the client's current key params version. For example, if the client's key params version is 004, but the incoming key params version is 003, the client will reject these parameters as insecure and abort this process.
3. Prompt the user for their account password, including in the prompt its reason. i.e _"Your account password was changed 3 days ago. Enter your new account password."_
4. Validate the account password based on its root key's ability to decrypt the aforementioned items key. If it succeeds, replace the client's current root key with this new root key.
At this point, this client is now in sync. It does not need to communicate with the server to handle updating its state after a password change.
If the aforementioned items key's key params are not exactly equal to the server's key params (not a logical outcome, but assuming arbitrary desync), and no items keys exists with the same key params as the server key params, it must fallback to performing the regular sign in flow to authenticate its root key (based on its `serverPassword` field).
B. When a client encounters an items key it cannot decrypt, regardless of its created date, and the server key parameters are equal to the ones the client has on hand, this indicates that the items key may be encrypted with an older root key (for whatever reason).
In such cases, the client will present a "key recovery wizard", which all attempt to decrypt the stale items key:
1. Retrieve the key parameters associated with the authenticated_data of the items key's payload.
2. Prompt the user for their account password as it was on the date the key parameters were created. For example, _"Enter your account password as it was on Oct 20, 2019, 6:15AM."_
3. Generate a root key from the account password using the relevant key params, and use that root key to decrypt the stale items key. If the decryption is successful, the client will then decrypt any items associated with that items key. It will then mark the key as needing sync.
4. When the key subsequently runs through normal syncing logic, it will then proceed to be encrypted by the account's current root key, and synced to the account.
The above procedure represents a "corrective" course of action in the case that the sync following a root key change, where all items keys must be re-encrypted with the new root key, fails silently and results in inconsistent data.
Note that the difference between case A and case B is that in case A, we prompt the user for their account password and **update our client's root key** with the generated root key, if it is valid. In case B, we generate a temporary root key for decryption purposes only, but discard of the root key after our decryption. This distinction is important because in case A, the server will be required to return key parameters with version greater than or equal to the user's current version, but in case B, key parameters can be arbitrarily old. However, because in this case the root key is not used for anything other than transient read operations, we can accept protocol versions no matter how outdated they are.
### Expired Sessions
When a client encounters an invalid session network response (typically status code 498), it will:
1. Retrieve the latest key parameters from the server. (Note that because GETting key parameters may require MFA authentication, clients must be prepared to handle an "mfa-required" error response.)
2. Ensure the key parameter version is greater than or equal to the version the client currently has on hand.
3. Prompt the user for their account password, indicating the reason. i.e _"Your session has expired. Please re-enter your account password to restore access to your account."_
4. Proceed with normal sign in flow.
## Storage
**There exists three types of storage:**
1. **Value storage**—values such as user preferences, session token, and other app-specific values.
2. **Payload storage**—encrypted item payloads (such as notes and tags).
3. **Root key storage**—the primary root key.
How data is stored depends on different key scenarios.
### Scenario A
_No root key and no root key wrapper (no account and no passcode)_
- **Value storage**: Plain, unencrypted
- **Payload storage**: Plain, unencrypted
- **Root key storage**: Not applicable
### Scenario B
_Root key but no root key wrapper (account but no passcode):_
- **Value storage**: Encrypted with root key
- **Payload storage:** Encrypted with root key
- **Root key storage**:
- With device keychain: Plainly in secure keychain
- With no device keychain: Plainly in device storage
### Scenario C
_Root key and root key wrapper (account and passcode):_
- **Value storage**: Encrypted with root key
- **Payload storage**: Encrypted with root key
- **Root key storage**: Encrypted in device storage
### Scenario D
_No root key but root key wrapper (no account but passcode):_
- **Value storage**: Encrypted with root key wrapper
- **Payload storage**: Encrypted with root key wrapper
- **Root key storage**: Not applicable
## 003 Migration
For the most part, SNJS does not branch off into different modes of behavior for different protocol versions (apart from the version specific operators). This means that new constructs in 004, like items keys, are also used in 003. This is accomplished via migrations that are performed when the application detects older data state.
In particular, when SNJS detects a pre-existing 003 account (before the user even has the chance to perform the protocol upgrade), a migration will be triggered that creates a default `itemsKey` using the account's current `rootKey.masterKey`:
```
itemsKey = { itemsKey: rootKey.masterKey, version: '003' }
```
This `itemsKey` is encrypted as usual using `rootKey.masterKey`, and synced to the user's account. When the user eventually performs the 004 upgrade (by entering their account password when prompted), a new `itemsKey` will be created as a default for 004. However, their previously created 003 `itemsKey` will continue to exist, so that data previously encrypted with 003 will still be decryptable.
## Cryptography Specifics
**Key Derivation:**
| Name | Value |
| ------------------ | -------- |
| Algorithm | Argon2id |
| Memory (Bytes) | 67108864 |
| Iterations | 5 |
| Parallelism | 1 |
| Salt Length (Bits) | 128 |
| Output Key (Bits) | 512 |
**Encryption:**
| Name | Value |
| ------------------- | ------------------ |
| Algorithm | XChaCha20+Poly1305 |
| Key Length (Bits) | 256 |
| Nonce Length (Bits) | 192 |
### Root Key Derivation Flow - Specifics
Given a user `identifier` (email) and `password` (user password):
1. Generate a random salt `seed`, 256 bits (`hex`).
2. Generate `salt`:
1. `hash = SHA256Hex('identifier:seed')`
2. `salt = hash.substring(0, 32)`
3. Generate `derivedKey = argon2(password, salt, ITERATIONS, MEMORY, OUTPUT_LENGTH) `
4. Generate `rootKey` as:
```
{
masterKey: derivedKey.firstHalf,
serverPassword: derivedKey.secondHalf,
version: '004'
}
```
5. For account registration, `identifier`, `seed`, `serverPassword`, and `version` must be uploaded to the server.
**Understanding the salt `seed`:**
Our threat model is intended to distrust the server as much as possible. For this reason, we do not want to blindly trust whatever salt value a server returns to us. For example, a malicious server may attempt to mass-weaken user security by sending the same salt for every user account, and observe what interesting results the clients send back. Instead, clients play a more significant role in salt generation, and use the value the user inputs into the email field for salt generation.
At this point we have `salt = generateSalt(email)`. However, we'd ideally like to make this value more unique. Emails are globally unique, but well-known in advance. We could introduce more variability by also including the protocol version in salt computation, such as `salt = generateSalt(email, version)`, but this could also be well-accounted for in advance.
The salt `seed` serves as a way to make it truly impossible to know a salt for an account ahead of time, without first interacting with the server the account is hosted on. While retrieving a `seed` for a given account is a public, non-authorized operation, users who configure two-factor authentication can proceed to lock this operation so that a proper 2FA code is required to retrieve the salt `seed`. Salts are thus computed via `salt = generateSalt(email, seed)`.
### Items Key Generation Flow
1. Generate random `hex` string `key`, 256 bits.
2. Create `itemsKey = {itemsKey: key, version: '004'}`
### Encryption - Specifics
An encrypted payload consists of:
- `items_key_id`: The UUID of the `itemsKey` used to encrypt `enc_item_key`.
- `enc_item_key`: An encrypted protocol string joined by colons `:` of the following components:
- protocol version
- encryption nonce
- ciphertext
- authenticated_data
- `content`: An encrypted protocol string joined by colons `:` of the following components:
- protocol version
- encryption nonce
- ciphertext
- authenticated_data
**Procedure to encrypt an item (such as a note):**
1. Generate a random 256-bit key `item_key` (in `hex` format).
2. Encrypt `item.content` using `item_key` to form `content`, and `{ u: item.uuid, v: '004', kp: rootKey.key_params IF item.type == ItemsKey }` as `authenticated_data`, following the instructions _"Encrypting a string using the 004 scheme"_ below.
3. Encrypt `item_key` using the the default `itemsKey.itemsKey` to form `enc_item_key`, and `{ u: item.uuid, v: '004', kp: rootKey.key_params IF item.type == ItemsKey }` as `authenticated_data`, following the instructions _"Encrypting a string using the 004 scheme"_ below.
4. Generate an encrypted payload as:
```
{
items_key_id: itemsKey.uuid,
enc_item_key: enc_item_key,
content: content,
}
```
### Encrypting a string using the 004 scheme:
Given a `string_to_encrypt`, an `encryption_key`, `authenticated_data`, and an item's `uuid`:
1. Generate a random 192-bit string called `nonce`.
2. Encode `authenticated_data` as a base64 encoded json string (`base64(json(authenticated_data))`) where the embedded data is recursively sorted by key for stringification (i.e `{v: '2', 'u': '1'}` should be stringified as `{u: '1', 'v': '2'}`), to get `encoded_authenticated_data`.
3. Encrypt `string_to_encrypt` using `XChaCha20+Poly1305:Base64`, `encryption_key`, `nonce`, and `encoded_authenticated_data`:
```
ciphertext = XChaCha20Poly1305(string_to_encrypt, encryption_key, nonce, encoded_authenticated_data)
```
4. Generate the final result by combining components into a `:` separated string:
```
result = ['004', nonce, ciphertext, encoded_authenticated_data].join(':')
```
## Next Steps
Join the [Slack group](https://standardnotes.com/slack) to discuss implementation details and ask any questions you may have.
You can also email [help@standardnotes.org](mailto:help@standardnotes.org).
Follow [@standardnotes on Twitter](https://twitter.com/standardnotes) for updates and announcements.

118
packages/docs/docs/specification/sync.md Normal file
View File

@@ -0,0 +1,118 @@
---
id: sync
title: Sync API
sidebar_label: Sync
description: Specification for the Standard Notes Sync API.
keywords:
- standard notes
- docs
- notes app
- end-to-end encryption
- sync
image: /img/logo.png
hide_title: false
hide_table_of_contents: false
---
This document outlines the client-server communication of the Standard Notes ecosystem.
## Item
An `Item` is largely the only model that both clients and server are concerned wtih. The `content` field stores an encrypted object that can be anything the client needs to operate.
Relationships between items are handled by the client, and not the server, and are stored encrypted in the `content` field as `references`.
Items have the following properties:
| name | type | description |
| ------------ | ----------------------------------- | -------------------------------------------------------------------------------------------- |
| uuid | `string` (or uuid for some databases) | The unique identifier for this item. |
| content | `text` | An encrypted string generated by the client. |
| content_type | `string` | The type of data contained in the `content` field. (i.e Note, Tag, SN|Component, etc.) |
| enc_item_key | `text` | The key used to encrypt `content`, itself also encrypted. |
| deleted | `boolean` | Whether the model has been deleted. |
| created_at_timestamp | `integer` | Timestamp representing when the item was created, with microsecond precision. |
| updated_at_timestamp | `integer` | Timestamp representing when the item was updated _by the server_, with microsecond precision. |
## Content
Data generated by the client is stored in the `content` field of the `Item`. An item's `content` is "sticky" and can have any property the client chooses, as well as the following:
| name | type | description |
| ---------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| appData | `dictionary` | A domain based dictionary whose content must be preserved between platforms. `appData[domain]` contains data the client stores in the item. |
| references | `array` | A metadata array of other `Item` uuids this item is related to and their respective `content_type`. See sample below. |
`references` array structure:
```JSON
[
{uuid: xxxx, content_type: "Tag"},
{uuid: xxxxx, content_type: "Tag"},
]
```
## REST API
General:
1. All requests after initial authentication should be authenticated with a session token in the `Authorization` header:
```
Authorization: Bearer <access token>
```
### POST items/sync
| | | |
|-------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| api | `string` | The API version to use. Must not be null, otherwise oldest API version used. Latest value is "20200115". |
| compute_integrity | `boolean` | Should be false in most cases. Pass true if you plan to read the `integrity_hash` result (integrity can be optionally checked once on app launch). |
| items | `array` | An array of Items. |
| limit | `integer` | The maximum number of items to return per request. Official clients use 150. |
| sync_token | `string` | An opaque token returned by the server from a previous sync request. This lets the server know to pull in only new changes. |
| cursor_token | `string` | An opaque pagination token returned by the server from a previous multi-page sync request. Send this value back only if the server returned this value from a previous request. |
#### Response
```JSON
{
"retrieved_items": Array<Item>,
"saved_items": Array<Item>,
"conflicts": Array<Conflict>,
"sync_token": string,
"cursor_token": string?,
"integrity_hash": string?,
}
```
| | |
|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| retrieved_items | Items that are new to the client or have been modified since last sync and should be merged with local values or created locally. |
| saved_items | Items which have been successfully saved. Clients should merge *only* the resulting `updated_at_timestamp` with local values. Do not update local value of `content` from remote `saved_items` value. |
| conflicts | Items which have not been saved by the server and returned to client. |
| sync_token | An opaque token which should be sent back to server in subsequent sync request. |
| cursor_token | An opaque token which should be sent back to server in subsequent sync request (if present). |
#### Conflicts
```typescript
type Conflict = {
server_item?: Item
unsaved_item?: Item
type: 'sync_conflict' | 'uuid_conflict'
}
```
A `sync_conflict` occurs when the item the client is attempting to save has a newer change on the server. The server is able to determine this by reading the `updated_at_timestamp` value from both the incoming payload and the server payload. If the incoming payload's `updated_at_timestamp` != to the server's `updated_at_timestamp` for that item, the save is rejected and returned to client as a `sync_conflict`. These types of conflicts are not uncommon.
To resolve a `sync_conflict`, the client must choose a winner between server_item and unsaved_item then re-sync. The winning item should have its updated_at_timestamp value set to the server's value.
A `uuid_conflict` occurs when the UUID the client has choosen for an item for X user already belongs to an item for Y user. This should mostly only occur when a user registers a new account and imports a data backup from another account.
To resolve a `uuid_conflict`, change the UUID of the item attempting to be saved to a new UUID, then re-sync.
#### Deletion
To delete an item permanently, set `deleted` to `true` on the `Item` and sync. When receiving an item that is `deleted`, remove it from the local database immediately.