Python
Structured logs from your Python services, with a logging handler. Standard library only.
The lyrelog package needs Python 3.11 or later and has no dependencies.
Internal preview: this package isn't published yet. Until it is, uv add lyrelog fails: there
is no package on PyPI.
Install
uv add lyrelogStart
Once, at startup. lyrelog.init starts the default client (flushing and replacing any earlier
one) and returns it; the module-level functions use it.
import lyrelog
lyrelog.init(
api_key="lyr_srv_...", # a server key: it can append events, never read them
release="2.0.1", # goes to resource.app_version
app_id="orders-api",
)Use a server key, lyr_srv_..., from the project's API keys page (Send events from a
server). It is a secret: keep it in your server's environment, never in an app or a web page.
Its events are verified, and one address has no daily share of
the month. An ingest key, lyr_ing_..., works too, but its events aren't verified.
Events go to https://ingest.lyrelog.com. Pass endpoint only to send them somewhere else. It
is the base URL (a pasted .../v1/events:batch path is stripped) and must be https://, or
http:// for localhost.
An empty api_key gives a client that accepts events and sends nothing, so a
missing environment variable in development doesn't crash the app. A missing key or a bad
endpoint is reported once as a warning on the lyrelog logger.
lyrelog.Client(endpoint, api_key, ...) is the same without the module-level default, for a
second project or for tests: every function below is a method on it. lyrelog.get_client()
returns the default client, or None before init.
Log events
One function per level: trace, debug, info, notice, warning, error, critical, or
lyrelog.log(level, message, ...).
lyrelog.info("checkout_done", attrs={"items": 3, "latency_ms": 412})
lyrelog.error("payment_declined", category="checkout", attrs={"code": 51}, payload={"response": raw})A log call never blocks and never raises: anything odd you pass (objects, NaN, cycles) is turned
into something valid. After shutdown(), log calls do nothing.
Loggers for each area
billing = lyrelog.logger("billing")
billing.warning("refund_denied", attrs={"order_id": order.id}, user_id=order.user_id)A logger follows the default client at call time, so you can make it at import, before init.
Users
lyrelog.set_user("u_123") # attached to every following event, process-wide
lyrelog.info("checkout_done", user_id=request_user) # for this event onlyA server handles many people at once, so prefer user_id= on each call to set_user.
The logging handler
The quickest way is install_logging, which adds one LoggingHandler to the root logger (and,
with uvicorn=True, the default, to uvicorn and uvicorn.error) and lowers the root logger to
level if it is above it, so INFO records get through. Calling it again replaces the earlier
handler, and uninstall_logging() removes it; uvicorn.access is left out (record requests in a
middleware instead).
import logging
import lyrelog
lyrelog.install_logging(logging.INFO)To add the handler yourself, remember that the root logger stops at WARNING by default: a
handler set to INFO gets nothing below WARNING until the root logger lets it through.
import logging
import lyrelog
logging.basicConfig(level=logging.INFO) # or logging.getLogger().setLevel(logging.INFO)
logging.getLogger().addHandler(lyrelog.LoggingHandler(level=logging.INFO))
log = logging.getLogger("app.orders")
log.info("order %s shipped", order_id, extra={"items": 3, "user_id": uid})
log.exception("email_failed") # inside an except: the traceback goes in the payloadlogging | Lyrelog |
|---|---|
Logger name (app.matches, uvicorn.error) | category |
DEBUG ... CRITICAL | The same; below DEBUG is trace, between INFO and WARNING is notice |
| The formatted message | message; with arguments, the template goes to attrs.msg_template to group by |
extra= fields | attrs, except user_id (the event's user) and payload (kept as payload.data) |
exc_info | payload.exception and attrs.error_type, attrs.error_message |
| File, line, function | payload.source |
The handler never forwards the SDK's own logger, so it can't loop. That logger is named
lyrelog: to see what the SDK itself reports (a refused key, a wrong endpoint, dropped events),
give it a handler, for example logging.getLogger("lyrelog").setLevel(logging.DEBUG) with
basicConfig. For events you want to chart,
prefer direct calls with a stable name: a formatted message is prose.
FastAPI
import logging
import os
import time
import lyrelog
from fastapi import FastAPI, Request
lyrelog.init(
api_key="lyr_srv_...", # a server key, or read it from your own settings
platform="api", # instead of "python", to tell it apart from a worker
release=os.environ.get("GIT_SHA"),
app_id="orders-api",
)
# uvicorn.error carries the tracebacks of unhandled exceptions. Uvicorn's loggers don't
# propagate to the root logger (propagate=False), so a root handler never sees them:
# uvicorn=True attaches the handler to them too. Uvicorn replaces its loggers' handlers when
# it applies its config, so call this after that: here, at import of the app module that
# `uvicorn app:app` runs, or in the app's lifespan startup.
lyrelog.install_logging(logging.INFO, uvicorn=True)
app = FastAPI()
http = lyrelog.logger("http")
@app.middleware("http")
async def record_request(request: Request, call_next):
start = time.perf_counter()
status = 500
try:
response = await call_next(request)
status = response.status_code
return response
finally:
route = request.scope.get("route") # set by the router once it matched
http.log(
"error" if status >= 500 else "warning" if status >= 400 else "info",
"http_request_served",
attrs={
"method": request.method,
"route": getattr(route, "path", "unmatched"), # /matches/{match_id}, not the id
"status": status,
"duration_ms": round((time.perf_counter() - start) * 1000, 1),
},
)The route template keeps ids out of the attribute, so the dashboard can count by route. The
middleware doesn't capture unhandled exceptions itself: uvicorn logs each one with its traceback
on uvicorn.error, and install_logging forwards that record with error_type,
error_message and the traceback, so capturing it here too would record it twice. With gunicorn
or uvicorn --workers, each worker process gets its own sender.
Django
Start the client once, in settings.py or your AppConfig.ready(), and send logging records
through Django's LOGGING setting:
# settings.py
import lyrelog
lyrelog.init(api_key="lyr_srv_...", platform="api", app_id="shop")
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"handlers": {"lyrelog": {"class": "lyrelog.LoggingHandler", "level": "INFO"}},
"root": {"handlers": ["lyrelog"], "level": "INFO"},
"loggers": {
# Django's request errors carry the traceback of a 500.
"django.request": {"handlers": ["lyrelog"], "level": "WARNING", "propagate": False},
},
}A worker or a script
A job that runs for many people sets the user on each event, and stops the client before it ends:
import os
import time
import lyrelog
lyrelog.init(api_key=os.environ.get("LYRELOG_KEY", ""), platform="worker", app_id="digest")
jobs = lyrelog.logger("jobs")
def send_digest(user):
started = time.perf_counter()
try:
deliver(user)
except Exception:
jobs.capture_exception(attrs={"job": "digest"}, user_id=user.id)
return
jobs.info("digest_sent", attrs={"duration_ms": round((time.perf_counter() - started) * 1000)},
user_id=user.id)
if __name__ == "__main__":
for user in due_users():
send_digest(user)
lyrelog.shutdown() # sends what's left, up to 5 sServerless and short scripts
The worker sends every 5 s and once more at exit (atexit). A function that is frozen between
requests (AWS Lambda, Cloud Functions) may never reach either: call lyrelog.flush() before
returning, and check what it returns if you need to know.
def handler(event, context):
try:
return process(event)
finally:
lyrelog.flush(timeout=2)atexit doesn't run after os._exit(), or when the process gets SIGTERM without a handler.
Uvicorn and gunicorn handle SIGTERM and exit normally; in your own long-running process, stop
the client in a handler:
import signal
import sys
import lyrelog
def on_sigterm(signum, frame):
lyrelog.shutdown(timeout=2)
sys.exit(0)
signal.signal(signal.SIGTERM, on_sigterm)Uncaught exceptions
The SDK doesn't install a hook for uncaught exceptions. To record them, set one yourself, for the main thread and for other threads:
import sys
import threading
import lyrelog
def report(exc_type, exc, tb):
lyrelog.capture_exception(exc, level="critical")
lyrelog.flush(timeout=2)
sys.__excepthook__(exc_type, exc, tb)
def report_thread(args):
if args.exc_value is not None and not issubclass(args.exc_type, SystemExit):
name = args.thread.name if args.thread else ""
lyrelog.capture_exception(args.exc_value, attrs={"thread": name}, level="critical")
threading.__excepthook__(args)
sys.excepthook = report
threading.excepthook = report_threadFrameworks catch exceptions inside requests themselves: for those, use the middleware above or the framework's logger.
Exceptions
try:
sync()
except Exception:
lyrelog.capture_exception(category="jobs")With no argument it captures the exception being handled; pass one to capture another. It logs
exception at error with attrs.error_type and attrs.error_message (the same in every SDK),
and the type, message and traceback in the payload. message= and level= change the event's
name and level, and user_id= sets its user. A logger's capture_exception uses its category.
Environments and platform
environment comes from the LYRELOG_ENVIRONMENT environment variable, else production.
platform is python. Set either in init:
lyrelog.init(api_key=..., environment="staging", platform="worker")Any name works: lowercase letters, digits, - and _, up to 32. Other text is normalized for you.
Consent
Consent is for apps that run on someone's device. A server SDK sends what it is given; there is no consent option.
Redacting and filtering
A value becomes "[redacted]", in attributes and the payload, when its key name, lowercased and
without -, _, . or spaces, is or ends with one of authorization, cookie, password,
passwd, secret, token, apikey, privatekey, credential, credentials, accesskey or
signature. So X-Api-Key, csrf_token, Set-Cookie and user_token are redacted;
token_count and max_tokens aren't.
redact_keys=[...] replaces that list with yours (matched the same way), and redact_keys=[]
turns redaction off. To keep the defaults and add one, pass
redact_keys=[*lyrelog.DEFAULT_REDACT_KEYS, "card_number"].
To change or drop events yourself, pass before_send to init: a function that gets each event
and returns it, changed or not, or None to drop it.
def before_send(event):
if event["message"] == "health_check":
return None
return event
lyrelog.init(api_key="lyr_srv_...", before_send=before_send)Flush and stop
sent = lyrelog.flush() # rarely needed: it sends every 5 s and at exit
lyrelog.shutdown() # also runs at exit: one last attempt, up to 5 sflush waits until everything logged before the call has been sent, up to its timeout (5 s
by default), and returns True if the queue drained, False if events are still waiting (the
server asked to wait with Retry-After, or it couldn't be reached) or were lost. Each request to
the server times out after 15 s.
client.stats() (on the client init returns) reports queued, dropped (queue full),
delivered, rejected (lines the server refused), discarded (batches dropped after a 4xx or
3xx), filtered (dropped by before_send), failed (events that couldn't be shaped, or whose
before_send raised), reentrant (logged from inside the SDK, for example a signal handler) and
disabled (a refused key, no key or a bad endpoint).
In development and tests
Leave api_key empty: the client accepts events, sends nothing and warns once on the lyrelog
logger.
lyrelog.init(api_key=os.environ.get("LYRELOG_KEY", ""))Without init, the module-level functions and loggers do nothing.
Verified events
A verified event is one that can't have been sent by someone who copied the public ingest key out of an app or a web page. What that proves, and what it doesn't, is on Verified events.
Send with a server key, lyr_srv_.... Every event it sends is verified (verified_by is
server) and doesn't count toward the
daily share of one address. An owner
makes it on the project's API keys page, Send events from a server, after a recent sign-in
(within 10 minutes), and it is shown once. It is a secret: keep it out of the repository and read
it from the server's environment or secret store.
import os
lyrelog.init(api_key=os.environ["LYRELOG_KEY"]) # lyr_srv_..., set in the server's environmentTo check, open the explorer: in Filter, pick Verified by and Server key (the other values are App Attest and Not verified). An opened event says it under Context, in its Sender line: Verified, server key, or Not verified, public key.
Events sent with an ingest key, lyr_ing_..., or a full-access key, lyr_adm_..., arrive
unverified: anyone holding a public key could have sent the same, and they count toward the daily
share. A verified event proves who sent it, not that what it says is true.
See Verified events for who verifies what and the daily share.
Configuration
| Option | Default | Meaning |
|---|---|---|
endpoint | https://ingest.lyrelog.com | The base URL; a pasted /v1/... path is stripped |
api_key | required | The project's server key (lyr_srv_...), or an ingest key. Empty: nothing is sent |
environment | LYRELOG_ENVIRONMENT, else production | Any valid name |
platform | python | api, worker... |
release | none | resource.app_version |
app_id | none | resource.app_id: the service name |
minimum_level | "debug" | Events below it are dropped before queueing |
flush_interval | 5.0 s | How often the worker sends |
max_queue | 10_000 | Oldest dropped beyond it |
max_queue_bytes | 16 MiB | The same cap, in bytes |
redact_keys | DEFAULT_REDACT_KEYS | The key names to redact; replaces the defaults |
before_send | none | Change an event, or return None to drop it |
session_timeout | 30 min | Idle time that starts a new session. Only on lyrelog.Client, not init |
Delivery and limits
| Batches | Up to 500 events or 512 KiB, gzip. A full batch goes out at once, otherwise every flush_interval. An event over 1 MiB is dropped and counted. |
| Retries | 1 s doubling to 5 min with equal jitter, honoring Retry-After (at most an hour); the same bytes are resent. Any other 4xx drops the batch; a 3xx drops it with a warning (redirects aren't followed). A refused key (401, 403) stops sending until the process restarts and keeps the events in memory. |
| Monthly quota | When a Free organization has used its month's events, the server answers 429. The events stay in memory and go out once the server takes them again, up to max_queue. The SDK logs one warning on the lyrelog logger with the server's message, and warns again only after a batch has been accepted. |
| Caps | Past max_queue the oldest go, and a lyrelog_events_dropped warning with the count goes out with the next batch. |
| Threads and forks | Thread-safe. After os.fork the child gets a fresh queue, worker, install and session. |
| Context | os_name (platform.system()), os_version (platform.release()), device_model (platform.machine()), an install id per process, and a session id per process that changes after 30 minutes without an event. |
| Limits | Fitted before queueing: nested objects past 4 levels become a JSON string, dots in attribute names become _, a payload over 64 KiB becomes {"lyrelog_truncated": true, "bytes": N, "preview": "..."}, tracebacks keep their last 48 KiB. |