In Python, text is str and binary data is bytes. That is the short answer behind most encoding questions: keep text as str inside the application, call encode() when crossing into a binary boundary, and call decode() when incoming bytes become text. UTF-8 is usually the encoding used for that conversion; it is not a Python string type.

The boundary appears in files, HTTP, databases, sockets, and JSON transport. Confusing the two domains causes explicit exceptions such as UnicodeDecodeError or silent defects such as João. A clear model replaces trial-and-error “accent fixes” with a repeatable diagnosis.

Unicode is not UTF-8

Unicode is a character repertoire that assigns code points. A is U+0041 and ç is U+00E7. A Python str is a sequence of Unicode characters regardless of its future storage. The Python strings guide covers slicing, formatting, and other text operations.

UTF-8 maps those code points to octets. ASCII characters occupy one byte, many Latin characters occupy two, and others take three or four. Character count therefore differs from byte length:

text = "ação"
data = text.encode("utf-8")

print(len(text))        # 4 characters
print(len(data))        # 6 bytes
print(data)             # b'a\xc3\xa7\xc3\xa3o'
print(data.decode("utf-8") == text)  # True

The b prefix marks a bytes literal. Hex escapes such as \xc3 display byte values, not damaged text. Python's official Unicode HOWTO explains code points, encodings, and normalization in depth.

Encode out, decode in

str.encode() returns bytes; bytes.decode() returns str. A robust rule is to decode once immediately after receiving a binary payload and encode once immediately before sending it. Business logic remains entirely textual.

def build_message(name: str) -> bytes:
    text = f"Hello, {name}!"
    return text.encode("utf-8")

packet = build_message("Lívia")
message = packet.decode("utf-8")
print(message)

Do not use str(data) as a decoder. It produces a representation such as "b'hello'", including the bytes marker. Decode using the encoding promised by the protocol. The official bytes and bytearray reference documents binary operations and their different mutability rules.

Files and JSON

Text-mode open performs conversion for you. State encoding="utf-8" so behavior does not depend on the machine locale:

from pathlib import Path

path = Path("customers.txt")
path.write_text("Ana\nJosé\n", encoding="utf-8")
content = path.read_text(encoding="utf-8")
print(content.splitlines())

The same applies to open(path, "w", encoding="utf-8", newline=""). See working with TXT, CSV, and JSON files for complete formats and Python pathlib for safer path handling.

JSON is textual. json.dumps() returns str, while json.loads() generally consumes str. ensure_ascii=False keeps Unicode readable; encode afterward only when a transport expects bytes:

import json

record = {"city": "São Luís", "currency": "€"}
json_text = json.dumps(record, ensure_ascii=False)
http_body = json_text.encode("utf-8")

restored = json.loads(http_body.decode("utf-8"))
assert restored == record

The Python JSON guide discusses validation and more complex serialization.

Diagnose encoding errors, do not guess

UnicodeEncodeError says the destination encoding cannot represent a character. ASCII cannot represent ã, for example. UnicodeDecodeError says a byte sequence is invalid under the selected encoding. Its attributes identify codec, offset, and reason:

latin1_data = b"Ol\xe1"

try:
    print(latin1_data.decode("utf-8"))
except UnicodeDecodeError as error:
    print(error.encoding, error.start, error.reason)

print(latin1_data.decode("latin-1"))  # Olá

The fix is to establish provenance: inspect the HTTP Content-Type, file specification, vendor contract, or database connection. Encoding detectors offer probabilities, not proof. If you own both ends, standardize on UTF-8 and test the contract.

Error handlers deserve care. errors="ignore" and errors="replace" keep execution moving by losing information. That is risky for names, identifiers, signatures, and audit records. Keep the default strict behavior. In an approved legacy import, replace inserts �, making damage countable. backslashreplace can help logs preserve a representation of an offending value. Python's codecs documentation lists the available handlers.

Mojibake and double encoding

João commonly means UTF-8 bytes were decoded as Latin-1 and the wrong text was subsequently stored. Manual substitutions are unreliable. Reprocess the original bytes with their actual codec. When only the corrupted string remains and the exact transformation is known, a controlled recovery may work:

broken = "João"
recovered = broken.encode("latin-1").decode("utf-8")
print(recovered)  # João

This is not universal. Back up data, validate samples, and record changes. Running it on already correct strings creates new corruption.

Normalization and trustworthy comparison

The visible é can be one code point or e followed by a combining accent. Both render alike but compare differently and produce different bytes. Normalize values from different sources before comparison:

import unicodedata

a = "café"
b = "cafe\u0301"
print(a == b)  # False
print(unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b))  # True

NFC is a common storage policy. NFKC also applies compatibility transformations and can change meaning, so require a deliberate use case. For caseless search, casefold() is broader than lower(). Security decisions, however, must not rely only on visual equality: confusable characters from different scripts can make identifiers look identical.

Practices, security, and observability

Declare encodings in file and service contracts. When diagnosing failures, do not log secrets or entire request bodies; record the source, declared codec, failing offset, and a short hexadecimal sample. Cap incoming size before decoding because valid text can still exhaust memory. Validate structure and semantics after decoding: valid UTF-8 is not inherently safe input.

Let mature libraries implement URL percent-encoding and HTTP charset rules. Verify that database client, schema, and columns support Unicode. Tests should include accents, combining forms, non-Latin scripts, and supplementary characters. The guide to Python try/except error handling helps place exception boundaries, while Python logging shows how to retain context without swallowing failures.

Frequently asked questions

What is the difference between str and bytes?

str stores Unicode text; bytes stores integers from 0 through 255. Encoding turns text into bytes, and decoding reverses that operation.

Should I always use UTF-8?

Use UTF-8 by default when you control the contract. When consuming external data, honor its declared encoding instead of forcing legacy bytes through UTF-8.

Can I use errors="ignore" to fix the exception?

It suppresses the exception by deleting data. Use it only when that loss is explicitly acceptable and measured; business records should usually fail visibly so the source can be fixed.

Why does my text display as b'...'?

The value is still bytes, or its representation was converted with str(). Decode the original bytes exactly once with the correct codec.

Conclusion

Unicode models characters, UTF-8 represents them as bytes, encode() prepares text for a binary boundary, and decode() interprets bytes entering the application. Keep str at the core, specify codecs at every boundary, and avoid handlers that silently erase evidence. When a failure occurs, preserve the original bytes, identify the real contract, and repair the boundary rather than its visible symptoms.