API Reference

Global Objects

twiggy.log

the magic log object

twiggy.internal_log

InternalLogger for reporting errors within Twiggy itself

twiggy.devel_log

InternalLogger for use by developers writing extensions to Twiggy

twiggy.emitters

the global emitters dictionary, tied to the log

Configuration

twiggy.add_emitters(*tuples)

Add multiple emitters

tuples should be (name_of_emitter, min_level, filter, output). The last three are passed to Emitter.

twiggy.dict_config(config)

Configure twiggy logging via a dictionary

Parameters:

config – a dictionary which configures twiggy’s outputs and emitters. See TWIGGY_CONFIG_SCHEMA for details of the format of the dict.

See also

dict_config() for a thorough explanation of the outputs and emitters concepts from the dictionary

twiggy.quick_setup(min_level=<LogLevel DEBUG>, file=None, msg_buffer=0)

Quickly set up emitters.

quick_setup() quickly sets up logging with reasonable defaults and minimal customizablity. Quick setup is limited to sending all messages to a file, sys.stdout or sys.stderr. A timestamp will be prefixed when logging to a file.

Parameters:

• min_level (LogLevel) – lowest message level to cause output

• file (string) – filename to log to, or sys.stdout, or sys.stderr. None means standard error.

• msg_buffer (int) – number of messages to buffer, see outputs.AsyncOutput.msg_buffer

Features

Optional additions of logging functionality

procinfo

Logging feature to add information about process, etc.

twiggy.features.procinfo.procinfo(self)

Adds the following fields:

Hostname:

current hostname

Pid:

current process id

Thread:

current thread name

socket

Logging feature to add information about a socket

twiggy.features.socket.socket(self, s)

Adds the following fields:

ip_addr:

numeric IP address

port:

port number

host:

peer hostname, as returned by getnameinfo()

service:

the human readable name of the service on port

Parameters:

s (socket) – the socket to extract information from

twiggy.features.socket.socket_minimal(self, s)

Like socket, but only log ip_addr and port

Filters

twiggy.filters.filter(msg: Message) → bool

A filter is any function that takes a Message and returns True if it should be emitted.

twiggy.filters.msg_filter(x) → filter

create a filter intelligently

You may pass:

None, True:

the filter will always return True

False:

the filter will always return False

string:

compiled into a regex

regex:

match() against the message text

callable:

returned as is

list:

apply msg_filter to each element, and all() the results

Return type:

filter function

twiggy.filters.names(*names) → filter

create a filter, which gives True if the messsage’s name equals any of those provided

names will be stored as an attribute on the filter.

Parameters:

names (strings) – names to match

Return type:

filter function

twiggy.filters.glob_names(*names) → filter

create a filter, which gives True if the messsage’s name globs those provided.

names will be stored as an attribute on the filter.

This is probably quite a bit slower than names().

Parameters:

names (strings) – glob patterns.

Return type:

filter function

class twiggy.filters.Emitter

Hold and manage an Output and associated filter()

min_level

only emit if greater than this LogLevel

filter

arbitrary filter() on message contents. Assigning to this attribute is intelligent.

_output

Output to emit messages to. Do not modify.

Formats

Formats are single-argument callables that take a Message and return an object appropriate for the Output they are assigned to.

class twiggy.formats.LineFormat(separator=':', traceback_prefix='\\nTRACE', conversion=line_conversion)

separator

string to separate line parts. Defaults to :.

traceback_prefix

string to prepend to traceback lines. Defaults to \nTRACE.

Set to '\\n' (double backslash n) to roll up tracebacks to a single line.

conversion

ConversionTable used to format fields. Defaults to line_conversion

format_text(msg)

format the text part of a message

format_fields(msg)

format the fields of a message

format_traceback(msg)

format the traceback part of a message

twiggy.formats.line_conversion

a default line-oriented ConversionTable. Produces a nice-looking string from fields.

Fields are separated by a colon (:). Resultant string includes:

time:

in iso8601 format (required)

level:

message level (required)

name:

logger name

Remaining fields are sorted alphabetically and formatted as key=value

twiggy.formats.line_format

a default LineFormat for output to a file. Sample output.

Fields are formatted using line_conversion and separated from the message text by a colon (:). Traceback lines are prefixed by TRACE.

twiggy.formats.shell_conversion

a default line-oriented ConversionTable for use in the shell. Returns the same string as line_conversion but drops the time field.

twiggy.formats.shell_format

a default LineFormat for use in the shell. Same as line_format but uses shell_conversion for fields.

Levels

Levels include (increasing severity): DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, DISABLED

class twiggy.levels.LogLevel(name, value)

A log level. Users should not create new instances.

Levels are opaque; they may be compared to each other, but nothing else.

class twiggy.levels.LogLevelMeta(name, bases, dct)

Metaclass that aids in making comparisons work the same in Python2 and Python3

Python3 raises TypeError when unorderable types are compared via lt, gt, le, ge. Python2 picks an order but it doesn’t always make much sense.

In Python3, we only need the rich comparison operators to get this behaviour.

In Python2, we use the __cmp__ function to raise TypeError for lt, gt, le, and ge. We define __eq__ and __ne__ on their own since those should just say that a LogLevel is never equal to a non-LogLevel.

twiggy.levels.name2level(name)

return a LogLevel from a case-insensitve string

Library

twiggy.lib.iso8601time(gmtime=None)

convert time to ISO 8601 format - it sucks less!

Parameters:

gmtime (time.struct_time) – time tuple. If None, use time.gmtime() (UTC)

XXX timezone is not supported

twiggy.lib.thread_name()

return the name of the current thread

Converter

class twiggy.lib.converter.Converter(key, convert_value, convert_item, required=False)

Holder for ConversionTable items

Variables:

• key – the key to apply the conversion to

• convert_value (function) – one-argument function to convert the value

• convert_item (function) – two-argument function converting the key and converted value

• required (bool) – is the item required to present. Items are optional by default.

twiggy.lib.converter.same_value(v)

return the value unchanged

twiggy.lib.converter.same_item(k, v)

return the item unchanged

twiggy.lib.converter.drop(k, v)

return None, indicating the item should be dropped

New in version 0.5.0: Add same_value, same_item, drop.

class twiggy.lib.converter.ConversionTable(seq)

Convert data dictionaries using Converters

For each item in the dictionary to be converted:

1. Find one or more corresponding converters c by matching key.

2. Build a list of converted items by calling c.convertItem(item_key, c.convertValue(item_value)). The list will have items in the same order as converters were supplied.

3. Dict items for which no converter was found are sorted by key and passed to generic_value / generic_item. These items are appended to the list from step 2.

4. If any required items are missing, ValueError is raised.

5. The resulting list of converted items is passed to aggregate. The value it returns is the result of the conversion.

Users may override generic_value/generic_item/aggregate by subclassing or assigning a new function on a ConversionTable instance.

Really, it’s pretty intuitive.

__init__(seq=None)

Parameters:

seq – a sequence of Converters

You may also pass 3-or-4 item arg tuples or kwarg dicts (which will be used to create Converters)

convert(d)

do the conversion

Parameters:

d (dict) – the data to convert. Keys should be strings.

generic_value(value)

convert values for which no specific Converter is supplied

generic_item(key, value)

convert items for which no specific Converter is supplied

aggregate(converteds)

aggregate list of converted items. The return value of convert

copy()

make an independent copy of this ConversionTable

get(key)

return the first converter for key

get_all(key)

return a list of all converters for key

add(*args, **kwargs)

Append a Converter.

args & kwargs will be passed through to its constructor

delete(key)

delete the all of the converters for key

Logger

Loggers should not be created directly by users; use the global log instead.

class twiggy.logger.BaseLogger(fields=None, options=None, min_level=None)

Base class for loggers

_fields

dictionary of bound fields for structured logging. By default, contains a single field time with value time.gmtime(). This function will be called for each message emitted, populating the field with the current time.struct_time.

_options

dictionary of bound options.

min_level

minimum LogLevel for which to emit. For optimization purposes only.

fields(**kwargs) → bound Logger

bind fields for structured logging. kwargs are interpreted as names/values of fields.

fields_dict(d) → bound Logger

bind fields for structured logging. Use this instead of fields if you have keys which are not valid Python identifiers.

Parameters:

d (dict) – dictionary of fields. Keys should be strings.

options(**kwargs) → bound Logger

bind options for message creation.

trace(trace='error') → bound Logger

convenience method to enable traceback logging

name(name) → bound Logger

convenvience method to bind name field

struct(**kwargs) → bound Logger

convenience method for structured logging. Calls fields() and emits at info

struct_dict(d) → bound Logger

convenience method for structured logging. Use instead of struct if you have keys which are not valid Python identifiers.

Parameters:

d (dict) – dictionary of fields. Keys should be strings.

The following methods cause messages to be emitted. format_spec is a template string into which args and kwargs will be substitued.

debug(format_spec='', *args, **kwargs)

Emit at DEBUG level

info(format_spec='', *args, **kwargs)

Emit at INFO level

notice(format_spec='', *args, **kwargs)

Emit at NOTICE level

warning(format_spec='', *args, **kwargs)

Emit at WARNING level

error(format_spec='', *args, **kwargs)

Emit at ERROR level

critical(format_spec='', *args, **kwargs)

Emit at CRITICAL level

class twiggy.logger.Logger(fields=None, options=None, min_level=None)

Logger for end-users. The type of the magic log

filter

Filter on format_spec. For optimization purposes only. Should have the following signature:

func(format_spec: string) → bool

Should the message be emitted.

classmethod addFeature(func, name=None)

add a feature to the class

Parameters:

• func – the function to add

• name (string) – the name to add it under. If None, use the function’s name.

classmethod disableFeature(name)

disable a feature.

A method will still exist by this name, but it won’t do anything.

Parameters:

name (string) – the name of the feature to disable.

classmethod delFeature(name)

delete a feature entirely

Parameters:

name (string) – the name of the feature to remove

class twiggy.logger.InternalLogger(output, fields=None, options=None, min_level=None)

Special-purpose logger for internal uses

Sends messages directly to output, bypassing emitters.

Variables:

output (Output) – an output to write to

twiggy.logger.emit(level)

a decorator that emits at level after calling the method. The method should return a Logger instance.

For convenience, decorators for the various levels are available as emit.debug, emit.info, etc..

Message

class twiggy.message.Message(level, format_spec, fields, options, args, kwargs)

A logging message. Users never create these directly.

Changed in version 0.4.1: Pass args/kwargs as list/dict instead of via */** expansion.

The constructor takes a dict of options to control message creation. In addition to suppress_newlines, the following options are recognized:

trace:

control traceback inclusion. Either a traceback tuple, or one of the strings always, error, in which case a traceback will be extracted from the current stack frame.

style:

the style of template used for format_spec. One of braces, percent, dollar. The aliases {}, % and $ are also supported.

Any callables passed in fields, args or kwargs will be called and the returned value used instead. See dynamic messages.

All attributes are read-only.

fields

dictionary of structured logging fields. Keys are string, values are arbitrary. A level item is required.

suppress_newlines

should newlines be escaped in output. Boolean.

traceback

a stringified traceback, or None.

text

the human-readable message. Constructed by substituting args/kwargs into format_spec. String.

__init__(level, format_spec, fields, options, args, kwargs)

Parameters:

• level (LogLevel) – the level of the message

• format_spec (string) – the human-readable message template. Should match the style in options.

• fields (dict) – dictionary of fields for structured logging

• args (tuple) – substitution arguments for format_spec.

• kwargs (dict) – substitution keyword arguments for format_spec.

• options (dict) – a dictionary of options to control message creation.

Outputs

class twiggy.outputs.Output(format=None, close_atexit=True)

_format

a callable taking a Message and formatting it for output. None means return the message unchanged.

use_locks

Class variable, indicating that locks should be used when running in a synchronous, multithreaded environment. Threadsafe subclasses may disable locking for higher throughput. Defaults to True.

__init__(format=None, close_atexit=True)

Parameters:

• format (format) – the format to use. If None, return the message unchanged.

• close_atexit (bool) – should close() be registered with atexit. If False, the user is responsible for closing the output.

New in version 0.4.1: Add the close_atexit parameter.

close()

Finalize the output.

The following methods should be implemented by subclasses.

_open()

Acquire any resources needed for writing (files, sockets, etc.)

_close()

Release any resources acquired in _open

_write(x)

Do the work of writing

Parameters:

x – an implementation-dependent object to be written.

class twiggy.outputs.AsyncOutput(msg_buffer=0)

An Output with support for asynchronous logging.

Inheriting from this class transparently adds support for asynchronous logging using the multiprocessing module. This is off by default, as it can cause log messages to be dropped.

Parameters:

msg_buffer (int) – number of messages to buffer in memory when using asynchronous logging. 0 turns asynchronous output off, a negative integer means an unlimited buffer, a positive integer is the size of the buffer.

class twiggy.outputs.FileOutput(name, format, mode='a', buffering=1, msg_buffer=0, close_atexit=True)

Output messages to a file

name, mode, buffering are passed to open()

class twiggy.outputs.StreamOutput(format, stream=sys.stderr)

Output to an externally-managed stream.

The stream will be written to, but otherwise left alone (i.e., it will not be closed).

class twiggy.outputs.NullOutput(format=None, close_atexit=True)

An output that just discards its messages

class twiggy.outputs.ListOutput(format=None, close_atexit=True)

an output that stuffs messages in a list

Useful for unittesting.

Variables:

messages (list) – messages that have been emitted

Changed in version 0.4.1: Replace DequeOutput with more useful ListOutput.