Scope Transport Protocol Version 1¶

EBNF¶

The following common EBNF entries are defined:

pb_uint_short ::=  <32bit unsigned encoded as Protocol Buffer varint>
number        ::=  digit+
digit         ::=  "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
space         ::=  "\x20"
newline       ::=  "\x0a"

Unified Message Structure¶

The payload of all messages sent from scope will follow the specification Unified Message Structure. This specification is abbreviated as UMS throughout the document.

Protocols¶

There are currently several different protocols in use by scope and its related components. To avoid confusion between them and their improvements, they are summarized in the following sections.

stp          ::=  count terminator keyword terminator <payload>
terminator   ::=  space
count        ::=  number
keyword      ::=  command | service
command      ::=  "*" command_name
command_name ::=  "services" | "enable" | "disable" | "quit" | "hostquit"
service      ::=  <any source character except ",">+

Opera                           proxy                    client

*services     ---------------->
                                    ----------------->   *services
                                    <-----------------   *enable
*enable       <----------------
data          <--------------------------------------->  data
                                ....
                                    <------------------  *quit
*disable      <----------------
*quit         ---------------->
                                    ------------------>  *hostquit
                                    ------------------>  *quit

Problems¶

The current protocol is based around sending Unicode strings to and from the clients, which makes it difficult to send binary data. Also, the encoding is hardcoded to UTF-16 for the entire message (STP and payload). This represents uneccessary overhead for sending data which is often in US-ASCII only.

XML is used as primary format which is inefficient when transporting lots of data. Lightweight alternatives are needed. XML also affects the decoding process of some clients since it must first decode it to a DOM tree, and then extract the interesting parts using the DOM interface which is slow and cumbersome.

The format is predeterminded by each service and there is no way to change it dynamically. For instance, JavaScript based clients will be able to decode the responses more quickly if they are sent as JSON.

There is no standardized way to tie (<tag>) a response to a previous request. This is currently embedded in the content of the request which is specific to each service and each command in the service. For instance, if you receive an error message there is no information about what request caused this error. This is due to the error handler being outside of the service implementation, and it has no knowledge of the <tag> entry. There is also a chance of <tag> conflicts when multiple clients are in use. A better system for handling the tags is needed.

The protocol was designed to handle multiple clients with the use of the proxy. However, there are problems with multiple clients in some services (ecmascript-debugger). Multi-client will be removed and the proxy updated to only allow one client at a time.

Overview¶

The various parts of the scope communication chain are:

To get a better overview, a few examples follow which display how the various protocols communicate.

A typical developer setup with Opera Dragonfly communicating with the Proxy using Dragonkeeper:

+-------+ STP/0 +--------------+ HTTP/1.1 +-----------+
|       | 7001  |              |   8002   |   Opera   |
| Scope |<----->| Dragonkeeper +<-------->| Dragonfly |
|       |       |              |          |           |
+-------+       +--------------+          +-----------+

The common usage scenario with Opera Dragonfly connecting to Opera using the internal JavaScript methods. Internally these methods will communicate with scope using an internal socket (this will be changed):

+-------+ +----------+            +-----------+
|       | | Opera    | JavaScript |   Opera   |
| Scope | | Internal |<---------->| Dragonfly |
|       | | "Proxy"  |            |           |
+-------+ +----------+            +-----------+
   ^         ^
   |         |
   +---------+
      STP/0
   49152-65535

Another setup follows with Opera Dragonfly for remote debugging on an embedded device, in this case a mobile phone:

+-------+ +----------+            +-----------+
|       | | Opera    | JavaScript |   Opera   |
| Scope | | Internal |<---------->| Dragonfly |
|       | | "Proxy"  |            |           |
+-------+ +----------+            +-----------+
             ^
             |
             |
             | STP/0
+---------+  | 7001
|         |  |
| Phone   |<-+
| w/Scope |
|         |
+---------+

Other clients can communicate directly using STP. In the following case, the Python client is shown:

+-------+ STP/0 +-------+  STP/0   +---------+
|       | 7001  |       |   8001   |         |
| Scope |<----->| Proxy +<-------->| PyScope |
|       |       |       |          |         |
+-------+       +-------+          +---------+

Backwards compatibility¶

The new protocol will introduce a major break in compatibility between the host, proxy and clients. To ensure that future changes are less disruptive a set of compatibility rules will be defined. The various components in scope will be defined to either provide a break between each version change, or provide only incremental changes for each version.

The transport protocol is the fundemental part. Changes to it will be difficult to do incrementally, so there is only a need for breaks between versions. This means that clients must immediately disconnect if they encounter a version they do not know how to handle.

Services however, will use a combination of incremental and breaking changes. This is handled by supplying a version number with two components: the first is the major version and determines changes that will break existing clients, and the second is the minor version which will determine incremental (or additional) changes. This means that clients will not need to be updated if only the minor version increases. For this to be possible the following rules apply:

1. Events and responses will be sent using the same structure as the previous versions.
2. Events and commands can only get new optional parameters. Existing parameters cannot change or be removed.
3. If a command requires a change of behaviour (or parameter change), a new command must be made and the existing one must be kept.
4. New and optional parameters to commands can be used to trigger extended functionality or alternative behaviour. However, this must be confined to the client that requests the command.
5. The order of fields can never be changed.
6. New events can be added as long as they are optional. This also means that clients must ignore events which they do not recognize.

If the amount of work to keep backwards compatibility increases, or the code gets bloated, the major version must be increased. This will signal a major change and allows for older behaviour and code to be cleanup or removed.

The compatibility changes in each service are handled separately. This ensures that a client which is dependent on one specific service does not need to change unless that service gets a major change.

Finally, a global version for scope is defined. This will use the current core version. It allows clients with more complex service dependencies a way to determine available features on a global scale.

STP/1¶

The new transport layer is defined as:

connection ::=  services handshake messages
messages   ::=  message*

This shows that the original STP/0 service list SERVICES is the first entry to be sent. Next comes a handshake which results in the handshake response HANDSHAKE followed by the actual transport messages.

The outer layer of the transport message is defined as:

message  ::=  "STP" stp_ver stp_size stp_data
stp_ver  ::=  <single octet>
stp_size ::=  pb_uint_short
stp_data ::=  <octets equal to stp-size>

This allows for multiple versions of a message to be sent. Each message is uniquely identified by the string “STP” followed by a version number. The size of the entire message is followed by the data of the message. This allows any decoder to check the version and skip data that it does not understand. The decoding of STP-DATA depends on the version.

An STP/1 message will look like:

stp_one_message ::=  "STP" "\x01" stp_size stp_one_data

In addition, it is now possible to pass STP/0 messages over the STP/1 protocol. This is done by setting the STP-VER to 0 and then passing the STP/0 data. The fields COUNT and SEPARATOR found in STP/0 will be skipped as the size is already present in the STP/1 layer. This means we only transfer the KEYWORD and DATA. An STP/0 message wrapped in STP/1 will look like:

stp_zero_message ::=  "STP" "\x00" stp_size keyword terminator <payload>

services     ::=  count terminator "*services" terminator service_list
service_list ::=  service ["," service]+

handshake_req ::=  "*enable" terminator "stp-" version
version       ::=  "0" | "1"
handshake     ::=  "STP/" version newline

stp_one_data ::=  stp_one_type headers
stp_one_type ::=  pb_uint_short # 1 = command, 2 = response, 3 = event, 4 = error
headers      ::=  <protocol buffer message>

enum STPType
{
    COMMAND = 1;
    RESPONSE = 2;
    EVENT = 3;
    ERROR = 4;
}

message TransportMessage
{
  required string service = 1;
  required uint32 commandID = 2;
  required uint32 format = 3;
  optional uint32 status = 4;
  optional uint32 tag = 5;
  required bytes payload = 8;
}

message Command
{
  required string service = 1;
  required uint32 commandID = 2;
  required uint32 format = 3;
  required uint32 tag = 5;
  required bytes  payload = 8;
}

message Response
{
  required string service = 1;
  required uint32 commandID = 2;
  required uint32 format = 3;
  required uint32 tag = 5;
  required bytes  payload = 8;
}

message Event
{
  required string service = 1;
  required uint32 commandID = 2;
  required uint32 format = 3;
  required bytes  payload = 8;
}

message Error
{
  required string service = 1;
  required uint32 commandID = 2;
  required uint32 format = 3;
  optional uint32 status = 4;
  optional uint32 tag = 5;
  required bytes  payload = 8;
}

enum Status
{
  OK = 0;
  BAD_REQUEST = 3;
  INTERNAL_ERROR = 4;
  COMMAND_NOT_FOUND = 5;
  SERVICE_NOT_FOUND = 6;
  OUT_OF_MEMORY = 7;
  SERVICE_NOT_ENABLED = 8;
  SERVICE_ALREADY_ENABLED = 9;
}

message ErrorInfo
{
    optional string description = 1;
    optional sint32 line        = 2;
    optional sint32 column      = 3;
    optional sint32 offset      = 4;
}

enum Format
{
    PROTOCOL_BUFFER = 0;
    JSON = 1;
    XML = 2;
}

Message flow¶

Before the STP/1 message flow can start an initialization phase is needed. This phase is performed between the two connecting parts. This would mean between the host and proxy and the proxy to any clients. This phase is used to determine the basic capabilites of the host, and to choose the STP version to use for messages across all connected nodes.

When the client connects to a host or proxy it will receive a list of services. Some of these services are meta-services and is used to determine capabilities such as possible STP versions. For instance, the host might send back:

*services scope,ecmascript-debugger,window-manager,stp-1,core-2-4

This reports back on the STP version available through the service “stp-1”. It also reports the core version in use, in this case core-2.4 (“core-2-4”).

A set of examples follows of the message flow between a client, proxy, and host. The following symbols are used:

~~~~~~~~~> Handshake
~ ~ ~ ~ ~> Handshake response
---------> Command
- - - - -> Response
=========> Event

The client must then initiate the handshake which also determines the STP version to use, for instance to enable STP version 1:

Host                              client

*services     =================>
              <~~~~~~~~~~~~~~~~~  *enable stp-1
STP/1\n       ~ ~ ~ ~ ~ ~ ~ ~ ~>
              <~~~~~~~~~~~~~~~~~  scope.Connect
scope.Connect ~ ~ ~ ~ ~ ~ ~ ~ ~>

A typical message flow between a client, proxy and host looks like this:

Opera                             proxy                   client

handshake       <~~~~~~~~~~~~~~~~     ~ ~ ~ ~ ~ ~ ~ ~ ~>  handshake
                                      <-----------------  scope.Connect
scope.Connect   <----------------
                - - - - - - - - >
                                      - - - - - - - - ->  scope.Connect

messages        <-------------------  - - - - - - - - ->  messages
events          =======================================>
                                  ....
                                      <-----------------  scope.Disconnect
scope.Disconnect<----------------
                - - - - - - - - >
                                      - - - - - - - - ->  scope.Disconnect

If the client disconnects the socket without telling the host/proxy, then the proxy will disconnect all clients on the given socket connection. For instance:

scope.Disconnect <----------------  scope.Disconnect
                 - - - - - - - - >

A STP/0 client will initiate the message flow as described in Scope Transport Protocol.

meta_stp ::=  "stp-" number

meta_core    ::=  "core-" dash_version
dash_version ::=  number ("-" number)*

Extended STP/0¶

When STP/0 is in use it will still use the Unified Message Structure for the message content. The format will be restricted to XML and JSON as it will require too much encoding overhead to binary protocols like the protocol buffer into UTF-16BE.

The basics of the STP/0 transport is in sending a size, service name and a payload. Only the size and service name is interesting for any existing proxies (2.3 or lower). This means that it is possible to change what the payload actually contains and let the receiver decode it.

The extended STP/0 transport will change the payload to contain the extra fields required by an STP/1 message, but it will encoded to be compatible with UTF-16BE. That is, it will be sent as pure text. The payload will consist of two things: the STP/1 header and the real payload. The header can then be decoded before the actual payload is sent to the next layer.

Definition:

stp          ::=  count terminator "scope" terminator data
data         ::=  "STP/" version terminator header_size terminator header <payload>
version      ::=  number
header_size  ::=  number
header       ::=  "[" service_name "," stp_type "," command_id "," format ["," tag ["," status ] ] "]"
service_name ::=  <json string>
stp_type     ::=  <json int>
command-id   ::=  <json int>
format       ::=  <json int>
tag          ::=  <json int>
status       ::=  <json int>

Messages must always be sent to the “scope” service. This ensures that there is only one service that needs to be enabled in the old proxies. This means that a client must first enable the “scope” service by sending “*enable scope”, or use the appropriate (DOM) API. This call will be ignored by STP enabled hosts. After this is sent, the client must encode all outgoing STP/1 messages according to the definition and send it to the “scope” service. The host will recognize this extended format and decode as an STP/1 message.