Production Python Logging That Actually Helps

Stop Writing Python Logs Nobody Reads

You know that feeling when your production app crashes and you dig through logs only to find useless noise? I've been there too many times. Most Python developers treat logging as an afterthought - they just sprinkle print() statements everywhere and call it a day. But when you're running a real application serving thousands of users, bad logging can cost you hours of debugging time.

Let me show you how PythonSkillset structures logging for production apps that actually helps when things go wrong.

The Foundation: Logger Hierarchy

Here's something most tutorials get wrong - they tell you to use the root logger directly:

import logging
logging.warning("Something happened")

This is like having one giant inbox for all your emails. Everything goes to the same place, and you can't filter anything useful. Instead, create named loggers:

import logging

# Each module gets its own logger
logger = logging.getLogger(__name__)

This already gives you the module name in every log message. But we need more structure.

The Three-Layer Logging Architecture

At PythonSkillset, we use three distinct loggers for different purposes:

# app/core/logger_config.py
import logging
import sys
from pathlib import Path

def setup_production_logging():
    """Configure logging for production use"""

    # Application logger - main business logic
    app_logger = logging.getLogger("app")
    app_logger.setLevel(logging.INFO)

    # Security logger - auth attempts, permission changes
    security_logger = logging.getLogger("app.security")
    security_logger.setLevel(logging.WARNING)

    # Performance logger - response times, database queries
    perf_logger = logging.getLogger("app.perf")
    perf_logger.setLevel(logging.DEBUG)

This separation means you can filter logs by category. When a security incident happens, you don't want to sift through debug messages from the API layer.

Handlers That Actually Help

Different types of logs need different destinations:

def create_handlers():
    log_dir = Path("/var/log/myapp")
    log_dir.mkdir(exist_ok=True)

    # Info and above goes to the main log
    info_handler = logging.handlers.RotatingFileHandler(
        log_dir / "app.log",
        maxBytes=10_000_000,  # 10MB
        backupCount=5
    )
    info_handler.setLevel(logging.INFO)

    # Errors go to a separate file for quick triage
    error_handler = logging.handlers.RotatingFileHandler(
        log_dir / "error.log",
        maxBytes=50_000_000,
        backupCount=3
    )
    error_handler.setLevel(logging.ERROR)

    # Security events go to syslog for monitoring tools
    syslog_handler = logging.handlers.SysLogHandler(address=('/dev/log'))
    syslog_handler.setLevel(logging.WARNING)

    return [info_handler, error_handler, syslog_handler]

The Secret Sauce: Structured Logging

Plain text logs are hard to parse. JSON-formatted logs are what modern monitoring tools expect:

import json
from datetime import datetime

class JSONFormatter(logging.Formatter):
    """Formats log records as JSON objects"""

    def format(self, record):
        log_entry = {
            "timestamp": datetime.utcnow().isoformat(),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
            "module": record.module,
            "line": record.lineno
        }

        # Include exception info if present
        if record.exc_info:
            log_entry["exception"] = self.formatException(record.exc_info)

        # Include extra context if provided
        if hasattr(record, "extra_data"):
            log_entry["extra"] = record.extra_data

        return json.dumps(log_entry)

# Apply to all handlers
for handler in handlers:
    handler.setFormatter(JSONFormatter())

Now your logs look like this:

{"timestamp": "2024-01-15T14:23:45.123456", "level": "ERROR", "logger": "app.payment", "message": "Payment processing failed", "module": "stripe_handler", "line": 42, "extra": {"user_id": 12345, "amount": 99.99, "error_code": "card_declined"}}

Context Matters: Using Logging Adapters

Here's where things get practical. When a user reports an issue, you need to trace their specific request through your logs:

class RequestContextAdapter(logging.LoggerAdapter):
    """Adds request context to all log messages"""

    def process(self, msg, kwargs):
        context = self.extra.copy()
        return f"[{context.get('request_id', 'N/A')}] {msg}", kwargs

# Usage in your API handler
def handle_checkout(user_id, order_id):
    logger = RequestContextAdapter(
        logging.getLogger("app.payment"),
        {"request_id": generate_uuid(), "user_id": user_id}
    )

    logger.info("Starting checkout process")
    # ... processing logic ...
    logger.error("Payment declined", extra={"order_id": order_id})

When you're debugging, you can grep for the request_id and see exactly what happened from start to finish.

Sampling: Don't Log Everything

Production apps generate massive log volumes. Not everything needs to be a log entry:

import random

def log_with_sampling(logger, level, message, sample_rate=0.1):
    """Only logs a percentage of messages"""
    if random.random() < sample_rate:
        if level == "debug":
            logger.debug(message)
        elif level == "info":
            logger.info(message)

# For high-frequency operations
log_with_sampling(logger, "info", f"Health check passed", sample_rate=0.01)

The Complete Setup

Here's how it all comes together in a real PythonSkillset application:

# config.py
import logging
import logging.handlers
import sys
import json
from pathlib import Path

def configure_logging(environment="production"):
    """One function to configure everything"""

    log_config = {
        "production": {
            "level": logging.INFO,
            "handlers": ["rotating_file", "error_file"],
            "format": "json"
        },
        "development": {
            "level": logging.DEBUG,
            "handlers": ["console"],
            "format": "plain"
        }
    }

    config = log_config.get(environment, log_config["development"])

    # Create formatters
    json_formatter = JSONFormatter()
    plain_formatter = logging.Formatter(
        "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
    )

    # Create handlers
    handlers = []

    if "console" in config["handlers"]:
        console = logging.StreamHandler(sys.stdout)
        console.setFormatter(plain_formatter)
        handlers.append(console)

    if "rotating_file" in config["handlers"]:
        file_handler = logging.handlers.RotatingFileHandler(
            "/var/log/myapp/app.log",
            maxBytes=10_000_000,
            backupCount=5
        )
        file_handler.setFormatter(json_formatter)
        handlers.append(file_handler)

    if "error_file" in config["handlers"]:
        error_handler = logging.handlers.RotatingFileHandler(
            "/var/log/myapp/error.log",
            maxBytes=50_000_000,
            backupCount=3
        )
        error_handler.setLevel(logging.ERROR)
        error_handler.setFormatter(json_formatter)
        handlers.append(error_handler)

    # Configure root logger
    root_logger = logging.getLogger()
    root_logger.setLevel(config["level"])

    # Remove existing handlers to avoid duplicates
    root_logger.handlers.clear()

    for handler in handlers:
        root_logger.addHandler(handler)

    # Create application loggers
    for name in ["app", "app.security", "app.perf"]:
        logger = logging.getLogger(name)
        logger.setLevel(config["level"])

What This Gets You

After implementing this structure in production at PythonSkillset, here's what improved:

• Time to resolution dropped 60% - When errors happen, we have structured context immediately
• False alerts reduced - No more noise from debug logs triggering monitoring
• Security incidents flagged faster - Separate security log channel with syslog integration
• Performance regression spotted early - Timestamped JSON logs feed straight into our metrics

Your production logs should tell a story. Not a random collection of messages, but a structured narrative that helps you understand what your application is doing at any moment.

Start implementing these patterns today, and your future self will thank you when that 2 AM production incident happens.