Coping with UnicodeEncodeError

Most non-UTF codecs handle only a small subset of the Unicode characters. When converting text to bytes, if a character is not defined in the target encoding, UnicodeEncodeError will be raised, unless special handling is provided by passing an errors argument to the encoding method or function. The behavior of the error handlers is shown in Example 4-5.

>>> city = 'São Paulo'
>>> city.encode('utf_8')  
b'Sxc3xa3o Paulo'
>>> city.encode('utf_16')
b'xffxfeSx00xe3x00ox00 x00Px00ax00ux00lx00ox00'
>>> city.encode('iso8859_1')  
b'Sxe3o Paulo'
>>> city.encode('cp437')  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../lib/python3.4/encodings/cp437.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character 'xe3' in
position 1: character maps to <undefined>
>>> city.encode('cp437', errors='ignore')  
b'So Paulo'
>>> city.encode('cp437', errors='replace')  
b'S?o Paulo'
>>> city.encode('cp437', errors='xmlcharrefreplace')  
b'S&#227;o Paulo'

The 'utf_?' encodings handle any str.

'iso8859_1' also works for the 'São Paulo' str.

'cp437' can’t encode the 'ã' (“a” with tilde). The default error handler—'strict'—raises UnicodeEncodeError.

The error='ignore' handler silently skips characters that cannot be encoded; this is usually a very bad idea.

When encoding, error='replace' substitutes unencodable characters with '?'; data is lost, but users will get a clue that something is amiss.

'xmlcharrefreplace' replaces unencodable characters with an XML entity.

ASCII is a common subset to all the encodings that I know about, therefore encoding should always work if the text is made exclusively of ASCII characters. Python 3.7 added a new boolean method str.isascii() to check whether your Unicode text is 100% pure ASCII. If it is, you should be able to encode it to bytes in any encoding without raising UnicodeEncodeError.

Coping with UnicodeDecodeError

Not every byte holds a valid ASCII character, and not every byte sequence is valid UTF-8 or UTF-16; therefore, when you assume one of these encodings while converting a binary sequence to text, you will get a UnicodeDecodeError if unexpected bytes are found.

On the other hand, many legacy 8-bit encodings like 'cp1252', 'iso8859_1', and 'koi8_r' are able to decode any stream of bytes, including random noise, without reporting errors. Therefore, if your program assumes the wrong 8-bit encoding, it will silently decode garbage.

Example 4-6 illustrates how using the wrong codec may produce gremlins or a UnicodeDecodeError.

>>> octets = b'Montrxe9al'  
>>> octets.decode('cp1252')  
'Montréal'
>>> octets.decode('iso8859_7')  
'Montrιal'
>>> octets.decode('koi8_r')  
'MontrИal'
>>> octets.decode('utf_8')  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 5:
invalid continuation byte
>>> octets.decode('utf_8', errors='replace')  
'Montr�al'

These bytes are the characters for “Montréal” encoded as latin1; 'xe9' is the byte for “é”.

Decoding with 'cp1252' (Windows 1252) works because it is a proper superset of latin1.

ISO-8859-7 is intended for Greek, so the 'xe9' byte is misinterpreted, and no error is issued.

KOI8-R is for Russian. Now 'xe9' stands for the Cyrillic letter “И”.

The 'utf_8' codec detects that octets is not valid UTF-8, and raises UnicodeDecodeError.

Using 'replace' error handling, the xe9 is replaced by “�” (code point U+FFFD), the official Unicode REPLACEMENT CHARACTER intended to represent unknown characters.

SyntaxError When Loading Modules with Unexpected Encoding

UTF-8 is the default source encoding for Python 3, just as ASCII was the default for Python 2 (starting with 2.5). If you load a .py module containing non-UTF-8 data and no encoding declaration, you get a message like this:

SyntaxError: Non-UTF-8 code starting with 'xe1' in file ola.py on line
  1, but no encoding declared; see http://python.org/dev/peps/pep-0263/
  for details

Because UTF-8 is widely deployed in GNU/Linux and OSX systems, a likely scenario is opening a .py file created on Windows with cp1252. Note that this error happens even in Python for Windows, because the default encoding for Python 3 source is UTF-8 across all platforms.

To fix this problem, add a magic coding comment at the top of the file, as shown in Example 4-7.

# coding: cp1252

print('Olá, Mundo!')

Suppose you have a text file, be it source code or poetry, but you don’t know its encoding. How do you detect the actual encoding? The next section answers that with a library recommendation.

How to Discover the Encoding of a Byte Sequence

How do you find the encoding of a byte sequence? Short answer: you can’t. You must be told.

Some communication protocols and file formats, like HTTP and XML, contain headers that explicitly tell us how the content is encoded. You can be sure that some byte streams are not ASCII because they contain byte values over 127, and the way UTF-8 and UTF-16 are built also limits the possible byte sequences. But even then, you can never be 100% positive that a binary file is ASCII or UTF-8 just because certain bit patterns are not there.

However, considering that human languages also have their rules and restrictions, once you assume that a stream of bytes is human plain text it may be possible to sniff out its encoding using heuristics and statistics. For example, if b'x00' bytes are common, it is probably a 16- or 32-bit encoding, and not an 8-bit scheme, because null characters in plain text are bugs; when the byte sequence b'x20x00' appears often, it is likely to be the space character (U+0020) in a UTF-16LE encoding, rather than the obscure U+2000 EN QUAD character—whatever that is.

That is how the package Chardet — The Universal Character Encoding Detector works to guess one of more than 30 supported encodings. Chardet is a Python library that you can use in your programs, but also includes a command-line utility, chardetect. Here is what it reports on the source file for this chapter:

$ chardetect 04-text-byte.asciidoc
04-text-byte.asciidoc: utf-8 with confidence 0.99

Although binary sequences of encoded text usually don’t carry explicit hints of their encoding, the UTF formats may prepend a byte order mark to the textual content. That is explained next.

BOM: A Useful Gremlin

In Example 4-4, you may have noticed a couple of extra bytes at the beginning of a UTF-16 encoded sequence. Here they are again:

>>> u16 = 'El Niño'.encode('utf_16')
>>> u16
b'xffxfeEx00lx00 x00Nx00ix00xf1x00ox00'

The bytes are b'xffxfe'. That is a BOM—byte-order mark—denoting the “little-endian” byte ordering of the Intel CPU where the encoding was performed.

On a little-endian machine, for each code point the least significant byte comes first: the letter 'E', code point U+0045 (decimal 69), is encoded in byte offsets 2 and 3 as 69 and 0:

>>> list(u16)
[255, 254, 69, 0, 108, 0, 32, 0, 78, 0, 105, 0, 241, 0, 111, 0]

On a big-endian CPU, the encoding would be reversed; 'E' would be encoded as 0 and 69.

To avoid confusion, the UTF-16 encoding prepends the text to be encoded with the special invisible character ZERO WIDTH NO-BREAK SPACE (U+FEFF). On a little-endian system, that is encoded as b'xffxfe' (decimal 255, 254). Because, by design, there is no U+FFFE character in Unicode, the byte sequence b'xffxfe' must mean the ZERO WIDTH NO-BREAK SPACE on a little-endian encoding, so the codec knows which byte ordering to use.

There is a variant of UTF-16—UTF-16LE—that is explicitly little-endian, and another one explicitly big-endian, UTF-16BE. If you use them, a BOM is not generated:

>>> u16le = 'El Niño'.encode('utf_16le')
>>> list(u16le)
[69, 0, 108, 0, 32, 0, 78, 0, 105, 0, 241, 0, 111, 0]
>>> u16be = 'El Niño'.encode('utf_16be')
>>> list(u16be)
[0, 69, 0, 108, 0, 32, 0, 78, 0, 105, 0, 241, 0, 111]

If present, the BOM is supposed to be filtered by the UTF-16 codec, so that you only get the actual text contents of the file without the leading ZERO WIDTH NO-BREAK SPACE. The Unicode standard says that if a file is UTF-16 and has no BOM, it should be assumed to be UTF-16BE (big-endian). However, the Intel x86 architecture is little-endian, so there is plenty of little-endian UTF-16 with no BOM in the wild.

This whole issue of endianness only affects encodings that use words of more than one byte, like UTF-16 and UTF-32. One big advantage of UTF-8 is that it produces the same byte sequence regardless of machine endianness, so no BOM is needed. Nevertheless, some Windows applications (notably Notepad) add the BOM to UTF-8 files anyway—and Excel depends on the BOM to detect a UTF-8 file, otherwise it assumes the content is encoded with a Windows code page. The character U+FEFF encoded in UTF-8 is the three-byte sequence b'xefxbbxbf'. So if a file starts with those three bytes, it is likely to be a UTF-8 file with a BOM. However, Python does not automatically assume a file is UTF-8 just because it starts with b'xefxbbxbf'.

We now move on to handling text files in Python 3.

Beware of Encoding Defaults

Several settings affect the encoding defaults for I/O in Python. See the default_encodings.py script in Example 4-10.

import sys, locale

expressions = """
        locale.getpreferredencoding()
        type(my_file)
        my_file.encoding
        sys.stdout.isatty()
        sys.stdout.encoding
        sys.stdin.isatty()
        sys.stdin.encoding
        sys.stderr.isatty()
        sys.stderr.encoding
        sys.getdefaultencoding()
        sys.getfilesystemencoding()
    """

my_file = open('dummy', 'w')

for expression in expressions.split():
    value = eval(expression)
    print(expression.rjust(30), '->', repr(value))

The output of Example 4-10 on GNU/Linux (Ubuntu 14.04 to 19.10) and MacOS (10.9 to 10.14) is identical, showing that UTF-8 is used everywhere in these systems:

$ python3 default_encodings.py
 locale.getpreferredencoding() -> 'UTF-8'
                 type(my_file) -> <class '_io.TextIOWrapper'>
              my_file.encoding -> 'UTF-8'
           sys.stdout.isatty() -> True
           sys.stdout.encoding -> 'utf-8'
            sys.stdin.isatty() -> True
            sys.stdin.encoding -> 'utf-8'
           sys.stderr.isatty() -> True
           sys.stderr.encoding -> 'utf-8'
      sys.getdefaultencoding() -> 'utf-8'
   sys.getfilesystemencoding() -> 'utf-8'

On Windows, however, the output is Example 4-11.

> chcp  
Active code page: 437
> python default_encodings.py  
 locale.getpreferredencoding() -> 'cp1252'  
                 type(my_file) -> <class '_io.TextIOWrapper'>
              my_file.encoding -> 'cp1252'  
           sys.stdout.isatty() -> True      
           sys.stdout.encoding -> 'utf-8'   
            sys.stdin.isatty() -> True
            sys.stdin.encoding -> 'utf-8'
           sys.stderr.isatty() -> True
           sys.stderr.encoding -> 'utf-8'
      sys.getdefaultencoding() -> 'utf-8'
   sys.getfilesystemencoding() -> 'utf-8'

chcp shows the active code page for the console: 437.

Running default_encodings.py with output to console.

locale.getpreferredencoding() is the most important setting.

Text files use locale.getpreferredencoding() by default.

The output is going to the console, so sys.stdout.isatty() is True.

Now, sys.stdout.encoding is not the same as the console code page reported by chcp!

Unicode support in Windows itself, and in Python for Windows, got better since I wrote about this in the 1^st edition of _Fluent Python. Example 4-11 used to report four different encodings in Python 3.4 on Windows 7. The encodings for stdout, stdin, and stderr used to be the same as the active code page reported by the chcp command, but now they’re all utf-8 thanks to PEP 528: Change Windows console encoding to UTF-8 implemented in Python 3.6, and Unicode support in PowerShell an cmd.exe (since Windows 1809 from October, 2018).⁷ It’s weird that chcp and sys.stdout.encoding say different things when stdout is writing to the console, but it’s great that now we can print Unicode strings without encoding errors on Windows—unless the user redirects output to a file, as we’ll soon see. That does not mean all your favorite emojis will appear in the console: that also depends on the font the console is using.

Another change was PEP 529: Change Windows filesystem encoding to UTF-8, also implemented in Python 3.6, which changed the file system encoding (used to represent names of directories and files) from Microsoft’s proprietary MBCS to UTF-8.

However, if the output of Example 4-10 is redirected to a file, like this:

Z:>python default_encodings.py > encodings.log

Then, the value of sys.stdout.isatty() becomes False, and sys.stdout.encoding is set by locale.getpreferredencoding(), 'cp1252' in that machine—but sys.stdin.encoding and sys.stderr.encoding remain utf-8.

This means that a script like Example 4-12 works when printing to the console, but may break when output is redirected to a file.

import sys
from unicodedata import name

print(sys.version)
print()
print('sys.stdout.isatty():', sys.stdout.isatty())
print('sys.stdout.encoding:', sys.stdout.encoding)
print()

test_chars = [
    'u2026',  # HORIZONTAL ELLIPSIS (in cp1252)
    'u221E',  # INFINITY (in cp437)
    'u32B7',  # CIRCLED NUMBER FORTY TWO
]

for char in test_chars:
    print(f'Trying to output {name(char)}:')
    print(char)

Example 4-12 displays the result of sys.stdout.isatty(), the value of sys.stdout.encoding, and these three characters:

• '…' HORIZONTAL ELLIPSIS (U+2026)--exists in CP 1252 but not in CP 437

• '∞' INFINITY (U+221E)--exists in CP 437 but not in CP 1252

• '㊷' CIRCLED NUMBER FORTY TWO (U+2026)--doesn’t exist in CP 1252 or CP 437

When I run stdout_check.py on PowerShell or cmd.exe, it works as captured in Figure 4-3.

Figure 4-3. Running stdout_check.py on PowerShell.

Despite chcp reporting the active code as 437, sys.stdout.encoding is UTF-8, so the HORIZONTAL ELLIPSIS and INFINITY both output correctly. The CIRCLED NUMBER FORTY TWO is replaced by a rectangle, but no error is raised. Presumably it is recognized as a valid character, but the console font doesn’t have the glyph to display it.

However, when I redirect the output of stdout_check.py to a file, I get Figure 4-4.

Figure 4-4. Running stdout_check.py on PowerShell, redirecting output.

The first problem demonstrated by Figure 4-4 is the UnicodeEncodeError mentioning character 'u221e', because sys.stdout.encoding is 'cp1252'--a code page that doesn’t have the INFINITY character.

Then, inspecting the partially-written out.txt, I get two surprises:

1. Reading out.txt with the type command—or a Windows editor like VS Code or Sublime Text—shows that instead of HORIZONTAL ELLIPSIS, I got 'à' (LATIN SMALL LETTER A WITH GRAVE). As it turns out, the byte value 0x85 in CP 1252 means '…', but in CP 437 the same byte value represents 'à'. So it seems the active code page does matter, not in a sensible or useful way, but as partial explanation of a bad Unicode experience.

2. out.txt was written with the UTF-16 LE encoding. This would be good, as UTF encodings support all Unicode characters—if it wasn’t for the unfortunate replacement of '…' with 'à'.

To wrap up this maddening issue of default encodings, let’s give a final look at the different encodings in Example 4-11:

• If you omit the encoding argument when opening a file, the default is given by locale.getpreferredencoding() ('cp1252' in Example 4-11).

• The encoding of sys.stdout|stdin|stderr used to be set by the PYTHONIOENCODING environment variable before Python 3.6—now that variable is ignored, unless PYTHONLEGACYWINDOWSSTDIO is set to a non-empty string. Otherwise, the encoding for standard I/O is UTF-8 for interactive I/O, or defined by locale.getpreferredencoding() if the output/input is redirected to/from a file.

• sys.getdefaultencoding() is used internally by Python in implicit conversions of binary data to/from str; this happens less often in Python 3, but still happens.⁸ Changing this setting is not supported.⁹

• sys.getfilesystemencoding() is used to encode/decode filenames (not file contents). It is used when open() gets a str argument for the filename; if the filename is given as a bytes argument, it is passed unchanged to the OS API. Before Python 3.6, this was MBCS on Windows, now it’s UTF-8. (On this topic, a useful answer on StackOverflow is “Difference between MBCS and UTF-8 on Windows”.)

To summarize, the most important encoding setting is that returned by locale.getpreferredencoding(): it is the default for opening text files and for sys.stdout/stdin/stderr when they are redirected to files. However, the documentation reads (in part):

locale.getpreferredencoding(do_setlocale=True)

Return the encoding used for text data, according to user preferences. User preferences are expressed differently on different systems, and might not be available programmatically on some systems, so this function only returns a guess. […]

Therefore, the best advice about encoding defaults is: do not rely on them.

You will avoid a lot of pain if you follow the advice of the Unicode sandwich and always are explicit about the encodings in your programs. Unfortunately, Unicode is painful even if you get your bytes correctly converted to str. The next two sections cover subjects that are simple in ASCII-land, but get quite complex on planet Unicode: text normalization (i.e., converting text to a uniform representation for comparisons) and sorting.

Case Folding

Case folding is essentially converting all text to lowercase, with some additional transformations. It is supported by the str.casefold() method since Python 3.3.

For any string s containing only latin1 characters, s.casefold() produces the same result as s.lower(), with only two exceptions—the micro sign 'µ' is changed to the Greek lowercase mu (which looks the same in most fonts) and the German Eszett or “sharp s” (ß) becomes “ss”:

>>> micro = 'µ'
>>> name(micro)
'MICRO SIGN'
>>> micro_cf = micro.casefold()
>>> name(micro_cf)
'GREEK SMALL LETTER MU'
>>> micro, micro_cf
('µ', 'μ')
>>> eszett = 'ß'
>>> name(eszett)
'LATIN SMALL LETTER SHARP S'
>>> eszett_cf = eszett.casefold()
>>> eszett, eszett_cf
('ß', 'ss')

There are nearly 300 code points for which str.casefold() and str.lower() return different results.

As usual with anything related to Unicode, case folding is a complicated issue with plenty of linguistic special cases, but the Python core team made an effort to provide a solution that hopefully works for most users.

In the next couple of sections, we’ll put our normalization knowledge to use developing utility functions.

Utility Functions for Normalized Text Matching

As we’ve seen, NFC and NFD are safe to use and allow sensible comparisons between Unicode strings. NFC is the best normalized form for most applications. str.casefold() is the way to go for case-insensitive comparisons.

If you work with text in many languages, a pair of functions like nfc_equal and fold_equal in Example 4-13 are useful additions to your toolbox.

"""
Utility functions for normalized Unicode string comparison.

Using Normal Form C, case sensitive:

    >>> s1 = 'café'
    >>> s2 = 'cafeu0301'
    >>> s1 == s2
    False
    >>> nfc_equal(s1, s2)
    True
    >>> nfc_equal('A', 'a')
    False

Using Normal Form C with case folding:

    >>> s3 = 'Straße'
    >>> s4 = 'strasse'
    >>> s3 == s4
    False
    >>> nfc_equal(s3, s4)
    False
    >>> fold_equal(s3, s4)
    True
    >>> fold_equal(s1, s2)
    True
    >>> fold_equal('A', 'a')
    True

"""

from unicodedata import normalize

def nfc_equal(str1, str2):
    return normalize('NFC', str1) == normalize('NFC', str2)

def fold_equal(str1, str2):
    return (normalize('NFC', str1).casefold() ==
            normalize('NFC', str2).casefold())

Beyond Unicode normalization and case folding—which are both part of the Unicode standard—sometimes it makes sense to apply deeper transformations, like changing 'café' into 'cafe'. We’ll see when and how in the next section.

Extreme “Normalization”: Taking Out Diacritics

The Google Search secret sauce involves many tricks, but one of them apparently is ignoring diacritics (e.g., accents, cedillas, etc.), at least in some contexts. Removing diacritics is not a proper form of normalization because it often changes the meaning of words and may produce false positives when searching. But it helps coping with some facts of life: people sometimes are lazy or ignorant about the correct use of diacritics, and spelling rules change over time, meaning that accents come and go in living languages.

Outside of searching, getting rid of diacritics also makes for more readable URLs, at least in Latin-based languages. Take a look at the URL for the Wikipedia article about the city of São Paulo:

http://en.wikipedia.org/wiki/S%C3%A3o_Paulo

The %C3%A3 part is the URL-escaped, UTF-8 rendering of the single letter “ã” (“a” with tilde). The following is much friendlier, even if it is not the right spelling:

http://en.wikipedia.org/wiki/Sao_Paulo

To remove all diacritics from a str, you can use a function like Example 4-14.

import unicodedata
import string


def shave_marks(txt):
    """Remove all diacritic marks"""
    norm_txt = unicodedata.normalize('NFD', txt)  
    shaved = ''.join(c for c in norm_txt
                     if not unicodedata.combining(c))  
    return unicodedata.normalize('NFC', shaved)  

Decompose all characters into base characters and combining marks.

Filter out all combining marks.

Recompose all characters.

Example 4-15 shows a couple of uses of shave_marks.

>>> order = '“Herr Voß: • ½ cup of Œtker™ caffè latte • bowl of açaí.”'
>>> shave_marks(order)
'“Herr Voß: • ½ cup of Œtker™ caffe latte • bowl of acai.”'  
>>> Greek = 'Ζέφυρος, Zéfiro'
>>> shave_marks(Greek)
'Ζεφυρος, Zefiro'  

Only the letters “è”, “ç”, and “í” were replaced.

Both “έ” and “é” were replaced.

The function shave_marks from Example 4-14 works all right, but maybe it goes too far. Often the reason to remove diacritics is to change Latin text to pure ASCII, but shave_marks also changes non-Latin characters—like Greek letters—which will never become ASCII just by losing their accents. So it makes sense to analyze each base character and to remove attached marks only if the base character is a letter from the Latin alphabet. This is what Example 4-16 does.

def shave_marks_latin(txt):
    """Remove all diacritic marks from Latin base characters"""
    norm_txt = unicodedata.normalize('NFD', txt)  
    latin_base = False
    preserve = []
    for c in norm_txt:
        if unicodedata.combining(c) and latin_base:   
            continue  # ignore diacritic on Latin base char
        preserve.append(c)                            
        # if it isn't combining char, it's a new base char
        if not unicodedata.combining(c):              
            latin_base = c in string.ascii_letters
    shaved = ''.join(preserve)
    return unicodedata.normalize('NFC', shaved)   

Decompose all characters into base characters and combining marks.

Skip over combining marks when base character is Latin.

Otherwise, keep current character.

Detect new base character and determine if it’s Latin.

Recompose all characters.

An even more radical step would be to replace common symbols in Western texts (e.g., curly quotes, em dashes, bullets, etc.) into ASCII equivalents. This is what the function asciize does in Example 4-17.

single_map = str.maketrans("""‚ƒ„ˆ‹‘’“”•–—˜›""",  
                           """'f"^<''""---~>""")

multi_map = str.maketrans({  
    '€': 'EUR',
    '…': '...',
    'Æ': 'AE',
    'æ': 'ae',
    'Œ': 'OE',
    'œ': 'oe',
    '™': '(TM)',
    '‰': '<per mille>',
    '†': '**',
    '‡': '***',
})

multi_map.update(single_map)  


def dewinize(txt):
    """Replace Win1252 symbols with ASCII chars or sequences"""
    return txt.translate(multi_map)  


def asciize(txt):
    no_marks = shave_marks_latin(dewinize(txt))     
    no_marks = no_marks.replace('ß', 'ss')          
    return unicodedata.normalize('NFKC', no_marks)  

Build mapping table for char-to-char replacement.

Build mapping table for char-to-string replacement.

Merge mapping tables.

dewinize does not affect ASCII or latin1 text, only the Microsoft additions in to latin1 in cp1252.

Apply dewinize and remove diacritical marks.

Replace the Eszett with “ss” (we are not using case fold here because we want to preserve the case).

Apply NFKC normalization to compose characters with their compatibility code points.

Example 4-18 shows asciize in use.

>>> order = '“Herr Voß: • ½ cup of Œtker™ caffè latte • bowl of açaí.”'
>>> dewinize(order)
'"Herr Voß: - ½ cup of OEtker(TM) caffè latte - bowl of açaí."'  
>>> asciize(order)
'"Herr Voss: - 1⁄2 cup of OEtker(TM) caffe latte - bowl of acai."'  

dewinize replaces curly quotes, bullets, and ™ (trademark symbol).

asciize applies dewinize, drops diacritics, and replaces the 'ß'.

To summarize, the functions in simplify.py go way beyond standard normalization and perform deep surgery on the text, with a good chance of changing its meaning. Only you can decide whether to go so far, knowing the target language, your users, and how the transformed text will be used.

This wraps up our discussion of normalizing Unicode text.

The next Unicode matter to sort out is… sorting.

Sorting with the Unicode Collation Algorithm

James Tauber, prolific Django contributor, must have felt the pain and created PyUCA, a pure-Python implementation of the Unicode Collation Algorithm (UCA). Example 4-20 shows how easy it is to use.

>>> import pyuca
>>> coll = pyuca.Collator()
>>> fruits = ['caju', 'atemoia', 'cajá', 'açaí', 'acerola']
>>> sorted_fruits = sorted(fruits, key=coll.sort_key)
>>> sorted_fruits
['açaí', 'acerola', 'atemoia', 'cajá', 'caju']

This is friendly and just works. I tested it on GNU/Linux, OSX, and Windows. Only Python 3.X is supported at this time.

PyUCA does not take the locale into account. If you need to customize the sorting, you can provide the path to a custom collation table to the Collator() constructor. Out of the box, it uses allkeys.txt, which is bundled with the project. That’s just a copy of the Default Unicode Collation Element Table from Unicode.org.

By the way, that table is one of the many that comprise the Unicode database, our next subject.

Finding characters by name

The unicodedata module has functions to retrieve character metadata, including unicodedata.name(), which returns a character’s official name in the standard. Figure 4-5 demonstrates that function.¹⁴

Figure 4-5. Exploring unicodedata.name() in the Python console

You can use the name() function to build apps that let users search for characters by name. Figure 4-6 demonstrates the cf.py command-line script that takes one or more words as arguments, and lists the characters that have those words in their official Unicode names. The full source code for cf.py is in Example 4-21.

Figure 4-6. Using cf.py to find smiling cats.

In Example 4-21, note the if statement in the find function using the .issubset() method to quickly test whether all the words in the query set appear in the list of words built from the character’s name. Thanks to Python’s rich set API, we don’t need a nested for loop and another if to implement this test.

#!/usr/bin/env python3
import sys
import unicodedata

FIRST, LAST = ord(' '), sys.maxunicode              


def find(*query_words, first=FIRST, last=LAST):     
    query = {w.upper() for w in query_words}        
    count = 0
    for code in range(first, last + 1):
        char = chr(code)                            
        name = unicodedata.name(char, None)         
        if name and query.issubset(name.split()):   
            print(f'U+{code:04X}	{char}	{name}')  
            count += 1
    print(f'({count} found)')


def main(words):
    if words:
        find(*words)
    else:
        print('Please provide words to find.')


if __name__ == '__main__':
    main(sys.argv[1:])

Set defaults for first and last code points to search.

find takes zero or more query_words, and optional keyword-only arguments to limit the range of the search, for easier testing.

Convert the query_words into a set of uppercased strings.

Get Unicode character for the code.

Get name of character, or None if the code point is unnassigned.

If there is a name, split it into a list words, then check that query is a subset of that.

Print out line with code point in U+9999 format, the character and its name.

The unicodedata module has other interesting functions. Next we’ll see a few that are related to getting information from characters that have numeric meaning.

Numeric meaning of characters

The unicodedata module includes functions to check whether a Unicode character represents a number and, if so, its numeric value for humans—as opposed to its code point number. Example 4-22 shows the use of unicodedata.name() and unicodedata.numeric() along with the .isdecimal() and .isnumeric() methods of str.

import unicodedata
import re

re_digit = re.compile(r'd')

sample = '1xbcxb2u0969u136bu216bu2466u2480u3285'

for char in sample:
    print('U+%04x' % ord(char),                       
          char.center(6),                             
          're_dig' if re_digit.match(char) else '-',  
          'isdig' if char.isdigit() else '-',         
          'isnum' if char.isnumeric() else '-',       
          format(unicodedata.numeric(char), '5.2f'),  
          unicodedata.name(char),                     
          sep='	')

Code point in U+0000 format.

Character centralized in a str of length 6.

Show re_dig if character matches the r'd' regex.

Show isdig if char.isdigit() is True.

Show isnum if char.isnumeric() is True.

Numeric value formated with width 5 and 2 decimal places.

Unicode character name.

Running Example 4-22 gives you Figure 4-7, if your terminal font has all those glyphs.

Figure 4-7. MacOS terminal showing numeric characters and metadata about them; re_dig means the character matches the regular expression r’d’

The sixth column of Figure 4-7 is the result of calling unicodedata.numeric(char) on the character. It shows that Unicode knows the numeric value of symbols that represent numbers. So if you want to create a spreadsheet application that supports Tamil digits or Roman numerals, go for it!

Figure 4-7 shows that the regular expression r'd' matches the digit “1” and the Devanagari digit 3, but not some other characters that are considered digits by the isdigit function. The re module is not as savvy about Unicode as it could be. The new regex module available in PyPI was designed to eventually replace re and provides better Unicode support.¹⁵ We’ll come back to the re module in the next section.

Throughout this chapter we’ve used several unicodedata functions, but there are many more we did not cover. See the standard library documentation for the unicodedata module.

Next we’ll take a quick look at a new trend: dual-mode APIs offering functions that accept str or bytes arguments with special handling depending on the type.

str Versus bytes in Regular Expressions

If you build a regular expression with bytes, patterns such as d and w only match ASCII characters; in contrast, if these patterns are given as str, they match Unicode digits or letters beyond ASCII. Example 4-23 and Figure 4-8 compare how letters, ASCII digits, superscripts, and Tamil digits are matched by str and bytes patterns.

import re

re_numbers_str = re.compile(r'd+')     
re_words_str = re.compile(r'w+')
re_numbers_bytes = re.compile(rb'd+')  
re_words_bytes = re.compile(rb'w+')

text_str = ("Ramanujan saw u0be7u0bedu0be8u0bef"  
            " as 1729 = 1³ + 12³ = 9³ + 10³.")        

text_bytes = text_str.encode('utf_8')  

print('Text', repr(text_str), sep='
  ')
print('Numbers')
print('  str  :', re_numbers_str.findall(text_str))      
print('  bytes:', re_numbers_bytes.findall(text_bytes))  
print('Words')
print('  str  :', re_words_str.findall(text_str))        
print('  bytes:', re_words_bytes.findall(text_bytes))    

The first two regular expressions are of the str type.

The last two are of the bytes type.

Unicode text to search, containing the Tamil digits for 1729 (the logical line continues until the right parenthesis token).

This string is joined to the previous one at compile time (see “2.4.2. String literal concatenation” in The Python Language Reference).

A bytes string is needed to search with the bytes regular expressions.

The str pattern r'd+' matches the Tamil and ASCII digits.

The bytes pattern rb'd+' matches only the ASCII bytes for digits.

The str pattern r'w+' matches the letters, superscripts, Tamil, and ASCII digits.

The bytes pattern rb'w+' matches only the ASCII bytes for letters and digits.

Figure 4-8. Screenshot of running ramanujan.py from Example 4-23

Example 4-23 is a trivial example to make one point: you can use regular expressions on str and bytes, but in the second case bytes outside of the ASCII range are treated as nondigits and nonword characters.

For str regular expressions, there is a re.ASCII flag that makes w, W, , B, d, D, s, and S perform ASCII-only matching. See the documentation of the re module for full details.

Another important dual-mode module is os.

str Versus bytes in os Functions

The GNU/Linux kernel is not Unicode savvy, so in the real world you may find filenames made of byte sequences that are not valid in any sensible encoding scheme, and cannot be decoded to str. File servers with clients using a variety of OSes are particularly prone to this problem.

In order to work around this issue, all os module functions that accept filenames or pathnames take arguments as str or bytes. If one such function is called with a str argument, the argument will be automatically converted using the codec named by sys.getfilesystemencoding(), and the OS response will be decoded with the same codec. This is almost always what you want, in keeping with the Unicode sandwich best practice.

But if you must deal with (and perhaps fix) filenames that cannot be handled in that way, you can pass bytes arguments to the os functions to get bytes return values. This feature lets you deal with any file or pathname, no matter how many gremlins you may find. See Example 4-24.

>>> os.listdir('.')  
['abc.txt', 'digits-of-π.txt']
>>> os.listdir(b'.')  
[b'abc.txt', b'digits-of-xcfx80.txt']

The second filename is “digits-of-π.txt” (with the Greek letter pi).

Given a byte argument, listdir returns filenames as bytes: b'xcfx80' is the UTF-8 encoding of the Greek letter pi).

To help with manual handling of str or bytes sequences that are file or pathnames, the os module provides special encoding and decoding functions fsencode(name_or_path) and os.fsdecode(name_or_path). Both of these functions accept an argument of type str, bytes, or—since Python 3.6—an object implementing the os.PathLike interface.

Enough suffering. Let’s wrap up our tour of str versus bytes with a fun topic: building emojis.

Country flags

Throughout history, countries split, join, mutate or simply adopt new flags. The Unicode consortium found a way to avoid keeping up with those changes and outsource the problem to the systems that claim Unicode support: its character database has no country flags. Instead there is a set of 26 “regional indicator symbols letters”, from A (U+1F1E6) to Z (U+1F1FF). When you combine two of those indicator letters to form an ISO 3166-1 country code, you get the corresponding country flag—if the UI supports it. Example 4-25 shows how.

# REGIONAL INDICATOR SYMBOLS
RIS_A = 'U0001F1E6'  # LETTER A
RIS_U = 'U0001F1FA'  # LETTER U
print(RIS_A + RIS_U)  # AU: Australia
print(RIS_U + RIS_A)  # UA: Ukraine
print(RIS_A + RIS_A)  # AA: no such country

Figure 4-9 shows the output of Example 4-25 on a MacOS 10.14 terminal.

Figure 4-9. Screenshot of running two_flags.py from Example 4-25. The AA combination is shown as two letters A inside dashed squares.

If your program outputs a combination of indicator letters that is not recognized by the app, you get the indicators displayed as letters inside dashed squares—again, depending on the UI. See the last line in Figure 4-9.

Now let’s see how emoji modifiers can be used to set the skin tone of emojis that show human faces, hands, noses etc.

Skin tones

Unicode provides a set of 5 emoji modifiers to set skin tone from pale to dark brown. They are based on the Fitzpatrick scale—developed to study the effects of ultraviolet light on human skin. Example 4-26 shows the use of those modifiers to set the skin tone of the thumbs up emoji.

from unicodedata import name

SKIN1 = 0x1F3FB  # EMOJI MODIFIER FITZPATRICK TYPE-1-2  
SKINS = [chr(i) for i in range(SKIN1, SKIN1 + 5)]       
THUMB = 'U0001F44d'  # THUMBS UP SIGN ?

examples = [THUMB]                                      
examples.extend(THUMB + skin for skin in SKINS)         

for example in examples:
    print(example, end='	')                            
    print(' + '.join(name(char) for char in example))   

EMOJI MODIFIER FITZPATRICK TYPE-1-2 is the first modifier.

Build list with all five modifiers.

Start list with the unmodified THUMBS UP SIGN.

Extend list with the same emoji followed by each of the modifiers.

Display emoji and tab.

Display names of characters combined in the emoji, joined by ' + '.

The output of Example 4-26 looks like Figure 4-10 on MacOS. As you can see, the unmodified emoji has a cartoonish yellow color, while the others have more realistic skin tones.

Figure 4-10. Screenshot of Example 4-26 in the MacOS 10.14 terminal.

Let’s now move to more complex emoji combinations using special markers.

Rainbow flag and other ZWJ sequences

Besides the special purpose indicators and modifiers we’ve seen, Unicode provides a marker that is used as glue between emojis and other characters, to produce new combinations: U+200D, ZERO WIDTH JOINER—a.k.a. ZWJ in many Unicode documents.

For example rainbow flag is built by joining the emojis WAVING WHITE FLAG and RAINBOW, as Figure 4-11 shows.

Figure 4-11. Making the rainbow flag in the Python console.

Unicode 13 supports more than 1100 ZWJ emoji sequences as RGI—“recommended for general interchange […] intended to be widely supported across multiple platforms”.¹⁶ You can find the full list of RGI ZWJ emoji sequences in emoji-zwj-sequences.txt and a small sample in Figure 4-12.

Figure 4-12. Sample ZWJ sequences generated by Example 4-27, running in a Jupyter Notebook, viewed on Firefox 72 on Ubuntu 19.10. This browser/OS combo can display all the emojis from this sample, including the newest: “people holding hands” and “transgender flag”, added in Emoji 12.0 and 13.0.

Example 4-27 is the source code that produced Figure 4-12. You can run it from your shell, but for better results I recommend pasting it inside a Jupyter Notebook to run it in a browser. Browsers often lead the way in Unicode support, and provide prettier emoji pictographs.

from unicodedata import name

zwg_sample = """
1F468 200D 1F9B0            |man: red hair                      |E11.0
1F9D1 200D 1F91D 200D 1F9D1 |people holding hands               |E12.0
1F3CA 1F3FF 200D 2640 FE0F  |woman swimming: dark skin tone     |E4.0
1F469 1F3FE 200D 2708 FE0F  |woman pilot: medium-dark skin tone |E4.0
1F468 200D 1F469 200D 1F467 |family: man, woman, girl           |E2.0
1F3F3 FE0F 200D 26A7 FE0F   |transgender flag                   |E13.0
1F469 200D 2764 FE0F 200D 1F48B 200D 1F469 |kiss: woman, woman  |E2.0
"""

markers = {'u200D': 'ZWG', # ZERO WIDTH JOINER
           'uFE0F': 'V16', # VARIATION SELECTOR-16
          }

for line in zwg_sample.strip().split('
'):
    code, descr, version = (s.strip() for s in line.split('|'))
    chars = [chr(int(c, 16)) for c in code.split()]
    print(''.join(chars), version, descr, sep='	', end='')
    while chars:
        char = chars.pop(0)
        if char in markers:
            print(' + ' + markers[char], end='')
        else:
            ucode = f'U+{ord(char):04X}'
            print(f'
	{char}	{ucode}	{name(char)}', end='')
    print()

One trend in modern Unicode is the addition of gender-neutral emojis such as SWIMMER (U+1F3CA) or ADULT (U+1F9D1), which can then be shown as they are, or with different gender in ZWJ sequences with the female sign ♀ (U+2640) or the male sign ♂ (U+2642). The Unicode Consortium is also moving towards more diversity in the supported family emojis. Figure 4-13 is a matrix of family emojis showing current support for families with different combinations of parents and children—as of January, 2020.

Figure 4-13. The table shows adult singles and couples at the top, and boys and girls on the left side. Cells have the combined emoji of a family with the parent(s) from the top and kid(s) from the left. If a combination is not supported by the browser, more than one emoji will appear inside a cell. Firefox 72 on Windows 10 is able to show all combinations.

The code I wrote to build Figure 4-13 is mostly concerned with HTML formatting, but is listed in [Link to Come] for completeness.

Unicode is a fascinating topic. However, now is time to wrap up our exploration of str and bytes.