The default file object iterator gives you one line at a time. For most text logs, that's enough. For binary streams, use f.read(chunk_size) in a loop or with the walrus operator. The point: don't call .read() with no argument on a 10GB file unless you have 10GB of RAM.
tempfile — when you need scratch space
tempfile.NamedTemporaryFile() gives you a file with a unique name, in the system's temp directory, automatically cleaned up. tempfile.TemporaryDirectory() gives you a directory and recursively cleans it up on context-exit. Use these for processing pipelines, test fixtures, anything where you need disk space without committing to a path.
mmap — file as memory
mmap maps a file into your process's virtual memory. You access it like a bytearray, but the OS handles paging — only the parts you actually touch get loaded. For random-access reads on a multi-GB file, this is dramatically faster than seek+read. For writes, mmap-modifications can be flushed efficiently.
StringIO and BytesIO — files in memory
From the io module: StringIO and BytesIO are file-like objects backed by memory. Useful for testing (give a function a fake file), for capturing output that would otherwise go to a file, and for building file content programmatically before flushing once. They support the same methods as real file objects.
War Story: cwkPippa's session logs are JSONL — one JSON object per line. The append-only design means we can stream-write events as they happen without ever buffering the whole conversation. Recovery code reads them line-by-line. The right shape for the right job.
Code
Streaming a large binary file·python
import io
# Pretend file (memory-backed for the demo)
big = io.BytesIO(b"xyz" * 1_000_000) # 3MB
CHUNK = 4096
big.seek(0)
total = 0
while chunk := big.read(CHUNK):
total += len(chunk)
print("read", total, "bytes")
# Or via the iterator pattern
big.seek(0)
import functools
for chunk in iter(functools.partial(big.read, CHUNK), b""):
pass
# iter() with a sentinel — calls big.read(CHUNK) until it returns b''
tempfile — auto-cleaned scratch space·python
import tempfile
from pathlib import Path
# Named file — auto-deleted on close
with tempfile.NamedTemporaryFile(suffix=".txt", delete=True) as tf:
print("path:", tf.name)
tf.write(b"hello world")
tf.flush()
# File is on disk here
print("size:", Path(tf.name).stat().st_size)
# File is gone now
# Directory — recursively cleaned
with tempfile.TemporaryDirectory() as td:
p = Path(td)
(p / "a.txt").write_text("a", encoding="utf-8")
(p / "b.txt").write_text("b", encoding="utf-8")
print("contents:", list(p.iterdir()))
# Directory and contents gone
mmap — file-as-memory for random access·python
import mmap
import tempfile
from pathlib import Path
# Make a sample file
p = Path("/tmp/mmap_demo.txt")
p.write_bytes(b"hello world hello pippa hello universe")
with open(p, "r+b") as f:
with mmap.mmap(f.fileno(), 0) as mm:
# Search
idx = mm.find(b"pippa")
print("found at:", idx)
# Slice — like a bytes object
print(mm[0:5]) # b'hello'
# Mutate — writes through to the file (with WRITE access)
mm[0:5] = b"HOWDY"
# Read back to confirm the change persisted
print(p.read_bytes()[:30])
p.unlink()
StringIO — fake file for testing·python
import io
import csv
# Test a function that reads CSV without writing real files
fake = io.StringIO('''name,age
alice,30
bob,25
''')
rows = list(csv.DictReader(fake))
print(rows)
# [{'name': 'alice', 'age': '30'}, {'name': 'bob', 'age': '25'}]
# Same for binary
binary = io.BytesIO(b"\xDE\xAD\xBE\xEF")
print(binary.read(2)) # b'\xde\xad'
(a) Create a tempfile.TemporaryDirectory and inside it write 3 files using pathlib. Print the directory contents. Confirm the directory is gone after the with block. (b) Create a 1MB file (using os.urandom or random bytes). Read it 3 ways: all at once, in 4KB chunks, and via mmap. Confirm all three give the same content (e.g., compute a SHA256 of the bytes).
Progress
Progress is local-only — sign in to sync across devices.