Log specification

All log objects generated by the inputs and entering the pipeline must be immutable maps which follow this specification.

Log object

Attribute	Type	Required?	Description
request	Map	yes	A request map (see below)
response	Map	yes	A response map (see below)

Request object

Attribute	Type	Required?	Description
time	string	yes	When the web server received the request (ISO-8601).
address	string	yes	IPv4 or IPv6 address (dotted notation)
method	string	yes	HTTP Method in uppercase: "HEAD", "GET", "POST", …
url	string	yes	The relative URL (path + query string)
headers	Map	yes	A collection of HTTP headers (string -> string)
scheme	string	no	"http" or "https"
protocol	string	no	"HTTP/1.0", "HTTP/1.1", or "HTTP/2"
port	int	no	Default to 80 for "http" scheme and 443 for "https" scheme
captured_headers	array	no	List of headers the input had access to.

Note on captured_headers

Ideally, the log clients forward all the HTTP headers and captured_headers does not need to be used.

However, when not all the headers are available, like in an access.log file for example, the client should indicate the list of headers that it has access to by listing their names in captured_headers.

This is useful for the pipeline to know whether a header is missing because it was not sent by the website visitor or because it was not available in the logs.

For example:

{
  request: {
    captured_headers: ['user-agent', 'referer'],
  }
}

Note on headers

The name of the header must be lower-cased.

Response object

Attribute	Type	Required?	Description
status	integer	yes	The HTTP status code for the response