norboten · cheat sheet

Text is bytes plus an encoding, and nobody else knows which one — cheat sheet

python, linux-basics
open(path, encoding="utf-8")            # say it, every time, in and out
open(path, encoding="cp1252")           # what the old Windows tool writes
open(path, "w", encoding="utf-8", newline="")   # csv: let the module choose line endings
# errors=: strict (raise) · ignore (drop) · replace (U+FFFD) · surrogateescape (round-trip bytes)

import locale; locale.getpreferredencoding(False)   # the default open() would use
python3 -X utf8 …        # or PYTHONUTF8=1: ignore the locale, use UTF-8

# replace the output in one step
fd, tmp = tempfile.mkstemp(dir=os.path.dirname(out))
… ; os.replace(tmp, out)
file export.csv                  # a first guess at the encoding