---

📖 The Config File That Took Down a Friday Deployment

Picture this: it's 5 PM on a Friday. A developer pushes a new service to production. The deployment succeeds, but five minutes later the service is dead — a cascade of AttributeError: 'NoneType' object has no attribute 'get' errors flooding the logs.

The root cause? A configuration file that wasn't present on the new server. The script that loaded it looked like this:

# This code works perfectly in development — and silently kills production
import json

def load_config():
    f = open("config.json")
    data = json.load(f)
    f.close()
    return data

config = load_config()
db_host = config["database"]["host"]   # AttributeError if config is None

Three things go wrong here. First, if config.json doesn't exist, open() raises FileNotFoundError and the program terminates with a raw traceback — no graceful message, no fallback, no logging. Second, the file is opened but close() is only called if nothing goes wrong between open() and close() — a raised exception skips it, leaving the file handle open. Third, the caller has no way to distinguish between "config file missing" and "config file has invalid JSON" — both just crash.

This is what production code without error handling looks like. And every Python developer eventually ships a version of it.

LBYL vs EAFP: Two Different Philosophies

Developers coming from Java, C#, or C tend to write defensive code in a style called LBYL — Look Before You Leap. The idea is to check for error conditions before attempting an operation:

# LBYL style — check first, then act
import os

def load_config_lbyl():
    if not os.path.exists("config.json"):
        return {}
    if not os.access("config.json", os.R_OK):
        return {}
    f = open("config.json")
    data = json.load(f)
    f.close()
    return data

This works, but it has a fundamental flaw: the gap between the check and the operation. You check that the file exists on line 4, but between that check and the open() call on line 7, another process could delete the file. This race condition is called TOCTOU — Time Of Check To Time Of Use. Your guard is already stale by the time you use it.

Python's idiomatic answer is EAFP — Easier to Ask Forgiveness than Permission. Instead of guarding every operation, you attempt it and handle the exception if something goes wrong:

# EAFP style — attempt first, handle exceptions
import json

def load_config_eafp():
    try:
        with open("config.json") as f:
            return json.load(f)
    except FileNotFoundError:
        print("Config file not found — using defaults.")
        return {}
    except json.JSONDecodeError as e:
        print(f"Config file is malformed: {e}")
        return {}

This version is shorter, avoids the TOCTOU race, handles exactly the exceptions it knows about, and gives a useful message for each failure mode. The with statement ensures the file is always closed — even if json.load() raises an exception. This is the Python way.

The rest of this post builds your complete mental model for writing production-grade file I/O and exception handling in Python.

---

🔍 Opening Files the Pythonic Way: open(), Modes, and the with Statement

The open() built-in is the gateway to all file operations in Python. It takes a file path and an optional mode string that controls how the file is opened.

File Opening Modes

The mode "r" is the default — calling open("myfile.txt") is identical to open("myfile.txt", "r").

Reading a File: Three Methods

Once a file is open, you have three ways to read its contents:

# All three reading methods demonstrated

with open("sample.txt", "r", encoding="utf-8") as f:
    full_text = f.read()        # Reads entire file into one string
    print(full_text)

with open("sample.txt", "r", encoding="utf-8") as f:
    first_line = f.readline()   # Reads exactly one line (including \n)
    second_line = f.readline()  # Reads the next line
    print(first_line, second_line)

with open("sample.txt", "r", encoding="utf-8") as f:
    all_lines = f.readlines()   # Returns a list of all lines
    for line in all_lines:
        print(line.strip())     # .strip() removes the trailing \n

For large files, the most memory-efficient approach is to iterate the file object directly — Python reads one line at a time without loading the entire file:

# Memory-efficient line-by-line reading — ideal for large log files
with open("large_logfile.txt", "r", encoding="utf-8") as f:
    for line in f:              # File object is iterable
        process(line.strip())   # Never loads whole file into memory

Writing to a File

# Writing and appending
with open("output.txt", "w", encoding="utf-8") as f:
    f.write("First line\n")
    f.write("Second line\n")
    f.writelines(["Third line\n", "Fourth line\n"])

# Append to an existing file (does not overwrite)
with open("output.txt", "a", encoding="utf-8") as f:
    f.write("This is appended\n")

Why with Is Always the Right Choice

The with statement uses Python's context manager protocol. When you enter a with block, Python calls __enter__() on the object. When the block exits — for any reason, including an exception — Python calls __exit__(), which closes the file handle automatically.

Without with, you must call f.close() yourself, and any exception between open() and close() will skip the cleanup, leaking an operating system file descriptor. Python processes have a hard limit on open file descriptors. Leak enough of them and your process will eventually fail with OSError: [Errno 24] Too many open files.

Always use with open(...) as f:. It is not optional for production code.

---

⚙️ Python's Exception System: Hierarchy, try/except/else/finally, and Advanced Patterns

Before you can handle exceptions intelligently, you need to understand where they come from and how they relate to each other.

The Exception Hierarchy You Actually Need to Know

Python exceptions form a class hierarchy. Every exception is an object that inherits from BaseException. The classes you'll interact with most are:

BaseException
├── SystemExit              ← raised by sys.exit() — do not catch unless cleaning up
├── KeyboardInterrupt       ← Ctrl+C — do not swallow silently
└── Exception               ← Parent of all "normal" program errors
    ├── ValueError          ← Bad argument value (e.g., int("abc"))
    ├── TypeError           ← Wrong type (e.g., "2" + 2)
    ├── AttributeError      ← Object has no attribute
    ├── IndexError          ← List index out of range
    ├── KeyError            ← Dict key not found
    ├── StopIteration       ← Iterator exhausted — used by for loops
    ├── RuntimeError        ← Miscellaneous runtime errors
    └── OSError             ← OS-level errors (file, network, permissions)
        ├── FileNotFoundError    ← File or directory does not exist
        ├── PermissionError      ← Insufficient permissions
        ├── IsADirectoryError    ← Tried to open a directory as a file
        └── TimeoutError         ← OS timeout

This hierarchy matters for your except clauses. Catching OSError will catch FileNotFoundError, PermissionError, and TimeoutError — because they are all subclasses. Catching Exception will catch almost everything except SystemExit and KeyboardInterrupt. Catching BaseException catches those too — almost never what you want.

Rule of thumb: Catch the most specific exception class you can name. Catching broad exceptions (except Exception:) hides bugs.

The Full try/except/else/finally Pattern

Python's exception block has four clauses:

import json

def parse_config(filepath):
    try:
        # The try block: code that might raise an exception
        with open(filepath, "r", encoding="utf-8") as f:
            data = json.load(f)

    except FileNotFoundError:
        # except: runs only if the named exception was raised
        print(f"Config file '{filepath}' not found.")
        return {}

    except json.JSONDecodeError as e:
        # You can catch multiple exception types, each in its own clause
        print(f"Config file has invalid JSON at line {e.lineno}: {e.msg}")
        return {}

    except (PermissionError, IsADirectoryError) as e:
        # Catch multiple exception types in one clause using a tuple
        print(f"Cannot read config file: {e}")
        return {}

    else:
        # else: runs ONLY if no exception was raised in the try block
        # This is the "success path" — a clean way to signal normal flow
        print(f"Config loaded successfully: {len(data)} keys")
        return data

    finally:
        # finally: runs ALWAYS — whether an exception occurred or not
        # Use it for cleanup that must happen regardless of outcome
        print("parse_config() finished.")

The else clause is one of the least-used but most expressive parts of Python's exception system. It answers the question "what do I do when everything went right?" without mixing that logic into the try block itself. If you put the success-path code inside try, you risk accidentally catching exceptions that are raised by that code — not by the operation you were guarding.

Re-raising Exceptions and Exception Chaining

Sometimes you want to catch an exception, log it or enrich it, and then re-raise it so the caller can still see it:

def load_user_data(user_id):
    filepath = f"data/user_{user_id}.json"
    try:
        with open(filepath) as f:
            return json.load(f)
    except FileNotFoundError as e:
        # Re-raise with additional context using "raise X from Y"
        # This preserves the original exception as the __cause__ attribute
        raise RuntimeError(f"User data for id={user_id} is missing") from e

The raise X from Y syntax creates an exception chain. When Python prints the traceback, it will show both the original FileNotFoundError and the new RuntimeError, connected with "The above exception was the direct cause of the following exception." This is invaluable when debugging — you see both the root cause and the higher-level context.

If you just want to re-raise the same exception without modification, use a bare raise inside an except block:

    except OSError:
        log_error("File operation failed")
        raise   # re-raises the current exception unchanged

Suppressing Exceptions Intentionally with contextlib.suppress

For cases where you genuinely want to ignore a specific exception, contextlib.suppress is cleaner than a bare try/except/pass:

from contextlib import suppress
import os

# Delete a temp file if it exists; ignore the error if it doesn't
with suppress(FileNotFoundError):
    os.remove("tmp_cache.json")

# Without suppress — noisier
try:
    os.remove("tmp_cache.json")
except FileNotFoundError:
    pass    # silence is the intent, but "pass" hides that intent

contextlib.suppress makes the intent explicit: "I am deliberately ignoring this exception class." Use it sparingly — it is appropriate only when the exception truly represents a normal, expected condition (like trying to remove a file that might already be gone).

---

📊 How Python Finds the Right Handler: Exception Propagation Through the Call Stack

Understanding what happens after an exception is raised requires a mental model of the call stack and how Python searches for a matching handler.

The diagram below traces a FileNotFoundError raised inside open() as it travels up a three-function call stack. Python searches for a matching except clause in the current frame first, then in each caller frame going upward. The finally block in every frame that has one will execute during unwinding, regardless of whether a handler is found. If Python reaches the top of the call stack without finding a handler, it prints the full traceback and terminates the program.

graph TD
    A[main calls load_config] --> B[load_config calls open]
    B --> C{File exists?}
    C -->|Yes| D[Return file handle]
    C -->|No| E[FileNotFoundError raised]
    E --> F{except clause in open frame?}
    F -->|No| G[Unwind to load_config frame]
    G --> H{except FileNotFoundError in load_config?}
    H -->|Yes| I[Handler executes - exception resolved]
    H -->|No| J[Unwind to main frame]
    J --> K{except clause in main?}
    K -->|Yes| L[Handler executes - exception resolved]
    K -->|No| M[Interpreter catches - full traceback printed]
    I --> N[finally blocks run on the way out]
    L --> N
    D --> O[else block runs - success path]

Notice that exception propagation moves upward through callers, not downward into callees. The finally blocks are always executed during this unwinding process — this is the guarantee that makes file handle cleanup reliable. The else block runs only on the successful path, when open() returns normally and no exception was raised. This clean separation of success and failure paths is what makes Python's exception model so readable once you internalize it.

---

🌍 File I/O in the Real World: CSV, JSON Configs, Log Files, and Binary Data

The open-read-close pattern extends to several common file formats Python developers encounter daily.

Processing CSV Files with the csv Module

Never parse CSV by splitting on commas manually — quoted fields with embedded commas will break it silently. Python's csv module handles all edge cases:

import csv

def read_employees(filepath):
    employees = []
    try:
        with open(filepath, "r", encoding="utf-8", newline="") as f:
            # newline="" is required for csv.reader to handle line endings correctly
            reader = csv.DictReader(f)
            for row in reader:
                # Each row is an OrderedDict keyed by the header row
                employees.append({
                    "name": row["name"],
                    "department": row["department"],
                    "salary": float(row["salary"]),
                })
    except FileNotFoundError:
        print(f"CSV file not found: {filepath}")
    except KeyError as e:
        print(f"Missing expected column: {e}")
    except ValueError as e:
        print(f"Could not parse salary as a number: {e}")
    return employees

Loading a JSON Configuration File

JSON configuration loading is one of the most common file I/O tasks in Python services:

import json
from pathlib import Path

DEFAULT_CONFIG = {
    "log_level": "INFO",
    "database": {"host": "localhost", "port": 5432},
    "max_connections": 10,
}

def load_config(config_path="config.json"):
    path = Path(config_path)
    try:
        with path.open("r", encoding="utf-8") as f:
            user_config = json.load(f)
        # Merge user config on top of defaults — user values win
        return {**DEFAULT_CONFIG, **user_config}
    except FileNotFoundError:
        print(f"No config found at {config_path!r} — using defaults.")
        return DEFAULT_CONFIG.copy()
    except json.JSONDecodeError as e:
        print(f"Invalid JSON in config: {e}")
        return DEFAULT_CONFIG.copy()

Note the use of pathlib.Path — it is the modern, object-oriented alternative to os.path string manipulation and is the preferred approach in new Python 3 code.

Appending to a Log File

Append mode ("a") is perfect for log files — it adds to the end without erasing existing content, and it creates the file if it does not yet exist:

from datetime import datetime

def append_log(message, logfile="app.log"):
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    entry = f"[{timestamp}] {message}\n"
    try:
        with open(logfile, "a", encoding="utf-8") as f:
            f.write(entry)
    except OSError as e:
        # Fallback: print to stderr if we cannot write the log
        import sys
        print(f"Log write failed: {e}", file=sys.stderr)

Reading Binary Files (Images, PDFs)

Binary mode ("rb") returns raw bytes objects rather than decoded strings — essential when handling any non-text data:

def read_file_header(filepath, num_bytes=8):
    """Read the first N bytes of a file to inspect its magic number."""
    try:
        with open(filepath, "rb") as f:
            header = f.read(num_bytes)
        return header
    except FileNotFoundError:
        return None

# Check if a file is a PNG (PNG magic bytes: 89 50 4E 47 0D 0A 1A 0A)
header = read_file_header("photo.png")
if header and header[:4] == b'\x89PNG':
    print("File is a valid PNG image")

Binary mode is also the correct choice for reading any file whose encoding is unknown — you get the raw bytes and decide what to do with them.

---

🧪 Three Worked Examples: From Fragile to Production-Ready

This section walks through three complete, runnable examples that apply everything covered above. Each one starts with context, shows the full implementation, then highlights what makes it robust.

Example 1: Robust Config Loader with Defaults and Validation

A production config loader needs to handle missing files, malformed JSON, and missing required keys — all without crashing the calling service:

import json
from pathlib import Path

REQUIRED_KEYS = {"database_url", "secret_key", "debug"}

def load_application_config(path="config.json"):
    """
    Load application config from a JSON file.
    Returns a config dict on success.
    Raises RuntimeError with a clear message on unrecoverable failure.
    """
    config_path = Path(path)

    try:
        with config_path.open("r", encoding="utf-8") as f:
            config = json.load(f)
    except FileNotFoundError:
        raise RuntimeError(
            f"Required config file not found: {config_path.resolve()}\n"
            "Create config.json before starting the service."
        ) from None
    except json.JSONDecodeError as e:
        raise RuntimeError(
            f"Config file contains invalid JSON: {e}\n"
            f"Fix line {e.lineno} in {config_path}."
        ) from e

    missing = REQUIRED_KEYS - set(config.keys())
    if missing:
        raise RuntimeError(
            f"Config is missing required keys: {sorted(missing)}"
        )

    return config

# Usage
try:
    config = load_application_config()
    print(f"Service starting with debug={config['debug']}")
except RuntimeError as e:
    print(f"Startup failed: {e}")
    raise SystemExit(1)

This example uses raise X from None to suppress the original traceback when the user-facing error message is already complete, and raise X from e to chain the original exception when the internals matter for debugging.

Example 2: Streaming Log Processor with Error Recovery

A streaming processor reads a potentially large log file line by line. Malformed lines should be counted and skipped, not crash the whole run:

import re
from datetime import datetime

LOG_PATTERN = re.compile(
    r"\[(?P<timestamp>[^\]]+)\]\s+(?P<level>INFO|WARN|ERROR)\s+(?P<message>.+)"
)

def process_log_file(filepath, target_level="ERROR"):
    """
    Stream-parse a log file and yield entries at or above target_level.
    Skips malformed lines and reports a summary at the end.
    """
    counts = {"parsed": 0, "skipped": 0, "matched": 0}
    results = []

    try:
        with open(filepath, "r", encoding="utf-8", errors="replace") as f:
            for line_number, line in enumerate(f, start=1):
                line = line.strip()
                if not line:
                    continue

                match = LOG_PATTERN.match(line)
                if not match:
                    counts["skipped"] += 1
                    continue

                counts["parsed"] += 1
                if match.group("level") == target_level:
                    counts["matched"] += 1
                    results.append({
                        "line": line_number,
                        "timestamp": match.group("timestamp"),
                        "message": match.group("message"),
                    })

    except FileNotFoundError:
        print(f"Log file not found: {filepath}")
        return [], counts
    except OSError as e:
        print(f"Could not read log file: {e}")
        return [], counts
    else:
        print(
            f"Processed {counts['parsed']} lines, "
            f"skipped {counts['skipped']} malformed, "
            f"found {counts['matched']} {target_level} entries."
        )

    return results, counts

# Usage
errors, stats = process_log_file("service.log", target_level="ERROR")
for entry in errors:
    print(f"Line {entry['line']}: {entry['message']}")

Note errors="replace" in the open() call — this substitutes the Unicode replacement character (\ufffd) for any byte sequences that cannot be decoded, instead of raising UnicodeDecodeError. For log files from systems with mixed encodings, this is often the right default.

Example 3: CSV-to-JSON Converter with Atomic Writes

This example converts a CSV file to JSON and writes the result atomically — meaning either the full file is written or nothing is written, avoiding partially written output files:

import csv
import json
import os
import tempfile
from pathlib import Path

def csv_to_json(csv_path, json_path, encoding="utf-8"):
    """
    Convert a CSV file to a JSON array file.
    Uses an atomic write: the output file is only replaced when
    the new content is fully written, preventing partial writes.
    """
    csv_path = Path(csv_path)
    json_path = Path(json_path)
    rows = []

    # Step 1: Read the CSV
    try:
        with csv_path.open("r", encoding=encoding, newline="") as f:
            reader = csv.DictReader(f)
            rows = list(reader)
    except FileNotFoundError:
        raise FileNotFoundError(f"Input CSV not found: {csv_path}") from None
    except csv.Error as e:
        raise ValueError(f"Malformed CSV: {e}") from e

    if not rows:
        raise ValueError(f"CSV file is empty: {csv_path}")

    # Step 2: Write JSON atomically via a temp file in the same directory
    tmp_fd, tmp_path = tempfile.mkstemp(dir=json_path.parent, suffix=".tmp")
    try:
        with os.fdopen(tmp_fd, "w", encoding=encoding) as tmp_file:
            json.dump(rows, tmp_file, indent=2, ensure_ascii=False)
        # Atomic rename — on POSIX this is guaranteed atomic
        os.replace(tmp_path, json_path)
    except OSError:
        # Clean up the temp file if something went wrong
        with suppress(OSError):
            os.remove(tmp_path)
        raise

    print(f"Converted {len(rows)} rows from {csv_path} to {json_path}")
    return len(rows)

from contextlib import suppress
# Usage
try:
    count = csv_to_json("employees.csv", "employees.json")
    print(f"Success: {count} records written")
except (FileNotFoundError, ValueError, OSError) as e:
    print(f"Conversion failed: {e}")

The atomic write pattern — write to a temp file, then os.replace() — is important whenever you are updating a file that other processes might be reading. On POSIX systems, os.replace() is guaranteed to be atomic at the filesystem level.

🧠 Deep Dive: How Python's Exception System Propagates Errors Across the Call Stack

When an exception is raised, Python constructs an exception object (a regular Python object inheriting from BaseException) and begins unwinding the call stack. At each frame, Python checks whether the current try block has a matching except clause. If it does, execution jumps to that handler. If not, the frame is popped and unwinding continues up the stack. If no handler is found anywhere in the call stack, Python catches the exception at the interpreter level and prints the traceback.

Internals: The __traceback__ Chain and Context Chaining

Every exception object carries a __traceback__ attribute pointing to a chain of frame objects representing the unwinding path. This is how traceback.format_exc() reconstructs the full stack trace for logging. Python 3 adds implicit exception chaining: if an exception is raised inside an except block, the new exception's __context__ is automatically set to the original exception. raise X from Y sets __cause__ explicitly (the "X was raised because of Y" form). Logging frameworks capture both to give you the full cause chain in your error reports.

Performance Analysis: try/except Is Nearly Free When Exceptions Don't Occur

CPython's exception handling uses a zero-cost approach for the happy path: when no exception is raised, a try block adds essentially zero overhead — a few bytes of bytecode to register the handler, but no runtime cost per iteration. This is why the EAFP (Easier to Ask Forgiveness than Permission) style is idiomatic Python: try: return d[key] except KeyError: return default is faster than if key in d: return d[key] because the LBYL version does two dictionary lookups on the happy path, while the EAFP version does one. Exception handling only costs CPU time when an exception actually propagates.

⚖️ Trade-offs & Failure Modes: File I/O and Exception Handling Pitfalls

The most common production failure is reading large files fully into memory with f.read(). A 2 GB log file read in one call allocates 2 GB of RAM in the Python process. For log processing, always iterate line by line (for line in f:) or use f.read(chunk_size) in a loop. Python's file objects are lazy iterators — for line in f reads one line at a time from the OS buffer without loading the whole file.

🧭 Decision Guide: Choosing the Right File and Exception Strategy

---

📚 Lessons Learned: What Five Years of File I/O Bugs Teach You

Never skip with. Every time you write f = open(...) without a with block, you are betting that no exception will occur between that line and f.close(). You will eventually lose that bet.

EAFP is not "ignore errors" — it's "handle errors at the right level." Catching FileNotFoundError and returning a default is not sloppy; it is appropriate when missing config is a recoverable situation. Catching bare Exception everywhere and returning None is sloppy.

Always specify encoding="utf-8" explicitly. Python's default encoding depends on the operating system — it is cp1252 on many Windows systems, which will silently corrupt files with accented characters when you move code between platforms. Explicit is always better than implicit.

Use exception chaining (raise X from Y). When you catch a low-level exception and raise a higher-level one, preserve the original with from e. Future-you will be grateful when debugging a production incident at 2 AM.

else in try/except is underused. Put your success-path logic there, not at the end of the try block. It makes your intent clear: "this code only runs when nothing went wrong."

Atomic writes prevent partial files. If your process is killed mid-write, a temp-file-plus-rename pattern means readers either see the old complete file or the new complete file — never a partial one.

Binary mode for binary files, text mode for text files. This seems obvious until you try to parse a PDF in text mode and get UnicodeDecodeError on the first byte.

---

📌 TLDR: File I/O and Exception Handling in One Paragraph

TLDR: Python favors EAFP over LBYL — attempt the operation, then handle specific exceptions if it fails. Always use with open(...) as f: to guarantee file handles are closed. Catch the most specific exception class available (FileNotFoundError, not Exception). Use try/except/else/finally — else for the success path, finally for cleanup that always runs. Chain exceptions with raise X from Y to preserve root-cause context. For production file writing, use an atomic temp-file-plus-rename pattern to avoid partial writes.

---

🔗 Related Posts in the Python Programming Series

• Python Basics: Variables, Types, and Control Flow — Start here if you are new to Python. Covers the label model, truthiness, f-strings, and loop patterns that underpin everything in this post.
• Python OOP: Classes, Dataclasses, and Dunder Methods — Learn how Python's __enter__ and __exit__ dunder methods enable the context manager protocol used by with open(...).
• Pythonic Code: Idioms Every Developer Should Know — Context managers, comprehensions, unpacking, and the rest of the idiomatic Python toolkit that makes production code readable.