Strings

Note: this is a modified version of an original notebook by Cliburn Chan. The original and this modified version are both released under the terms of the MIT license.

The process of cleaning data for analysis often requires working with text, for example, to correct typos, convert to standard nomenclature and resolve ambiguous labels. In some statistical fields that deal with (say) processing electronic medical records, information science or recommendations based on user feedback, text must be processed before analysis - for example, by converting to a bag of words.

We will use a whimsical example to illustrate Python tools for munging text data using string methods and regular expressions. Finally, we will see how to format text data for reporting.

Get “Through the Looking Glass”

import requests

try:
    with open('looking_glass.txt') as f:
        text = f.read()
except IOError:
    url = 'http://www.gutenberg.org/cache/epub/12/pg12.txt'
    res = requests.get(url)
    text = res.text
    with open('looking_glass.txt', 'w') as f:
        f.write(str(text))

Slice to get Jabberwocky

start = text.find('JABBERWOCKY')

text[start:start+2000]

"JABBERWOCKY\n\n     'Twas brillig, and the slithy toves\n      Did gyre and gimble in the wabe;\n     All mimsy were the borogoves,\n      And the mome raths outgrabe.\n\n     'Beware the Jabberwock, my son!\n      The jaws that bite, the claws that catch!\n     Beware the Jubjub bird, and shun\n      The frumious Bandersnatch!'\n\n     He took his vorpal sword in hand:\n      Long time the manxome foe he sought--\n     So rested he by the Tumtum tree,\n      And stood awhile in thought.\n\n     And as in uffish thought he stood,\n      The Jabberwock, with eyes of flame,\n     Came whiffling through the tulgey wood,\n      And burbled as it came!\n\n     One, two! One, two! And through and through\n      The vorpal blade went snicker-snack!\n     He left it dead, and with its head\n      He went galumphing back.\n\n     'And hast thou slain the Jabberwock?\n      Come to my arms, my beamish boy!\n     O frabjous day! Callooh! Callay!'\n      He chortled in his joy.\n\n     'Twas brillig, and the slithy toves\n      Did gyre and gimble in the wabe;\n     All mimsy were the borogoves,\n      And the mome raths outgrabe.\n\n\n'It seems very pretty,' she said when she had finished it, 'but it's\nRATHER hard to understand!' (You see she didn't like to confess, even\nto herself, that she couldn't make it out at all.) 'Somehow it seems\nto fill my head with ideas--only I don't exactly know what they are!\nHowever, SOMEBODY killed SOMETHING: that's clear, at any rate--'\n\n'But oh!' thought Alice, suddenly jumping up, 'if I don't make haste I\nshall have to go back through the Looking-glass, before I've seen what\nthe rest of the house is like! Let's have a look at the garden first!'\nShe was out of the room in a moment, and ran down stairs--or, at least,\nit wasn't exactly running, but a new invention of hers for getting down\nstairs quickly and easily, as Alice said to herself. She just kept the\ntips of her fingers on the hand-rail, and floated gently down without\neven touching the stairs with her feet; then she floate"

end = text.find('It seems very pretty', start)

poem = text[start:end]
poem

"JABBERWOCKY\n\n     'Twas brillig, and the slithy toves\n      Did gyre and gimble in the wabe;\n     All mimsy were the borogoves,\n      And the mome raths outgrabe.\n\n     'Beware the Jabberwock, my son!\n      The jaws that bite, the claws that catch!\n     Beware the Jubjub bird, and shun\n      The frumious Bandersnatch!'\n\n     He took his vorpal sword in hand:\n      Long time the manxome foe he sought--\n     So rested he by the Tumtum tree,\n      And stood awhile in thought.\n\n     And as in uffish thought he stood,\n      The Jabberwock, with eyes of flame,\n     Came whiffling through the tulgey wood,\n      And burbled as it came!\n\n     One, two! One, two! And through and through\n      The vorpal blade went snicker-snack!\n     He left it dead, and with its head\n      He went galumphing back.\n\n     'And hast thou slain the Jabberwock?\n      Come to my arms, my beamish boy!\n     O frabjous day! Callooh! Callay!'\n      He chortled in his joy.\n\n     'Twas brillig, and the slithy toves\n      Did gyre and gimble in the wabe;\n     All mimsy were the borogoves,\n      And the mome raths outgrabe.\n\n\n'"

print(poem)

JABBERWOCKY

     'Twas brillig, and the slithy toves
      Did gyre and gimble in the wabe;
     All mimsy were the borogoves,
      And the mome raths outgrabe.

     'Beware the Jabberwock, my son!
      The jaws that bite, the claws that catch!
     Beware the Jubjub bird, and shun
      The frumious Bandersnatch!'

     He took his vorpal sword in hand:
      Long time the manxome foe he sought--
     So rested he by the Tumtum tree,
      And stood awhile in thought.

     And as in uffish thought he stood,
      The Jabberwock, with eyes of flame,
     Came whiffling through the tulgey wood,
      And burbled as it came!

     One, two! One, two! And through and through
      The vorpal blade went snicker-snack!
     He left it dead, and with its head
      He went galumphing back.

     'And hast thou slain the Jabberwock?
      Come to my arms, my beamish boy!
     O frabjous day! Callooh! Callay!'
      He chortled in his joy.

     'Twas brillig, and the slithy toves
      Did gyre and gimble in the wabe;
     All mimsy were the borogoves,
      And the mome raths outgrabe.


'

print(poem.title())

Jabberwocky

     'Twas Brillig, And The Slithy Toves
      Did Gyre And Gimble In The Wabe;
     All Mimsy Were The Borogoves,
      And The Mome Raths Outgrabe.

     'Beware The Jabberwock, My Son!
      The Jaws That Bite, The Claws That Catch!
     Beware The Jubjub Bird, And Shun
      The Frumious Bandersnatch!'

     He Took His Vorpal Sword In Hand:
      Long Time The Manxome Foe He Sought--
     So Rested He By The Tumtum Tree,
      And Stood Awhile In Thought.

     And As In Uffish Thought He Stood,
      The Jabberwock, With Eyes Of Flame,
     Came Whiffling Through The Tulgey Wood,
      And Burbled As It Came!

     One, Two! One, Two! And Through And Through
      The Vorpal Blade Went Snicker-Snack!
     He Left It Dead, And With Its Head
      He Went Galumphing Back.

     'And Hast Thou Slain The Jabberwock?
      Come To My Arms, My Beamish Boy!
     O Frabjous Day! Callooh! Callay!'
      He Chortled In His Joy.

     'Twas Brillig, And The Slithy Toves
      Did Gyre And Gimble In The Wabe;
     All Mimsy Were The Borogoves,
      And The Mome Raths Outgrabe.


'

poem.count('the')

15

print(poem.replace('the', 'XXX'))

JABBERWOCKY

     'Twas brillig, and XXX slithy toves
      Did gyre and gimble in XXX wabe;
     All mimsy were XXX borogoves,
      And XXX mome raths outgrabe.

     'Beware XXX Jabberwock, my son!
      The jaws that bite, XXX claws that catch!
     Beware XXX Jubjub bird, and shun
      The frumious Bandersnatch!'

     He took his vorpal sword in hand:
      Long time XXX manxome foe he sought--
     So rested he by XXX Tumtum tree,
      And stood awhile in thought.

     And as in uffish thought he stood,
      The Jabberwock, with eyes of flame,
     Came whiffling through XXX tulgey wood,
      And burbled as it came!

     One, two! One, two! And through and through
      The vorpal blade went snicker-snack!
     He left it dead, and with its head
      He went galumphing back.

     'And hast thou slain XXX Jabberwock?
      Come to my arms, my beamish boy!
     O frabjous day! Callooh! Callay!'
      He chortled in his joy.

     'Twas brillig, and XXX slithy toves
      Did gyre and gimble in XXX wabe;
     All mimsy were XXX borogoves,
      And XXX mome raths outgrabe.


'

Find palindromic words in poem if any

poem = poem.lower()

We need to remove the punctuation characters to only leave plain text:

import string
string.punctuation

'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

poem  = poem.translate(dict.fromkeys(map(ord, string.punctuation)))
poem

'jabberwocky\n\n     twas brillig and the slithy toves\n      did gyre and gimble in the wabe\n     all mimsy were the borogoves\n      and the mome raths outgrabe\n\n     beware the jabberwock my son\n      the jaws that bite the claws that catch\n     beware the jubjub bird and shun\n      the frumious bandersnatch\n\n     he took his vorpal sword in hand\n      long time the manxome foe he sought\n     so rested he by the tumtum tree\n      and stood awhile in thought\n\n     and as in uffish thought he stood\n      the jabberwock with eyes of flame\n     came whiffling through the tulgey wood\n      and burbled as it came\n\n     one two one two and through and through\n      the vorpal blade went snickersnack\n     he left it dead and with its head\n      he went galumphing back\n\n     and hast thou slain the jabberwock\n      come to my arms my beamish boy\n     o frabjous day callooh callay\n      he chortled in his joy\n\n     twas brillig and the slithy toves\n      did gyre and gimble in the wabe\n     all mimsy were the borogoves\n      and the mome raths outgrabe\n\n\n'

words = poem.split()
words[:10]

['jabberwocky',
 'twas',
 'brillig',
 'and',
 'the',
 'slithy',
 'toves',
 'did',
 'gyre',
 'and']

def is_palindrome(word):
    return word == word[::-1]

{word for word in words if is_palindrome(word)}

{'did', 'o'}

Top 10 most frequent words

import collections

poem_counter = collections.Counter(words)

poem_counter.most_common(10)

[('the', 19),
 ('and', 14),
 ('he', 7),
 ('in', 6),
 ('jabberwock', 3),
 ('my', 3),
 ('through', 3),
 ('twas', 2),
 ('brillig', 2),
 ('slithy', 2)]

poem_counter.items()

dict_items([('jabberwocky', 1), ('twas', 2), ('brillig', 2), ('and', 14), ('the', 19), ('slithy', 2), ('toves', 2), ('did', 2), ('gyre', 2), ('gimble', 2), ('in', 6), ('wabe', 2), ('all', 2), ('mimsy', 2), ('were', 2), ('borogoves', 2), ('mome', 2), ('raths', 2), ('outgrabe', 2), ('beware', 2), ('jabberwock', 3), ('my', 3), ('son', 1), ('jaws', 1), ('that', 2), ('bite', 1), ('claws', 1), ('catch', 1), ('jubjub', 1), ('bird', 1), ('shun', 1), ('frumious', 1), ('bandersnatch', 1), ('he', 7), ('took', 1), ('his', 2), ('vorpal', 2), ('sword', 1), ('hand', 1), ('long', 1), ('time', 1), ('manxome', 1), ('foe', 1), ('sought', 1), ('so', 1), ('rested', 1), ('by', 1), ('tumtum', 1), ('tree', 1), ('stood', 2), ('awhile', 1), ('thought', 2), ('as', 2), ('uffish', 1), ('with', 2), ('eyes', 1), ('of', 1), ('flame', 1), ('came', 2), ('whiffling', 1), ('through', 3), ('tulgey', 1), ('wood', 1), ('burbled', 1), ('it', 2), ('one', 2), ('two', 2), ('blade', 1), ('went', 2), ('snickersnack', 1), ('left', 1), ('dead', 1), ('its', 1), ('head', 1), ('galumphing', 1), ('back', 1), ('hast', 1), ('thou', 1), ('slain', 1), ('come', 1), ('to', 1), ('arms', 1), ('beamish', 1), ('boy', 1), ('o', 1), ('frabjous', 1), ('day', 1), ('callooh', 1), ('callay', 1), ('chortled', 1), ('joy', 1)])

Words that appear exactly twice.

[(k, v) for (k, v) in poem_counter.items() if v==2]

[('twas', 2),
 ('brillig', 2),
 ('slithy', 2),
 ('toves', 2),
 ('did', 2),
 ('gyre', 2),
 ('gimble', 2),
 ('wabe', 2),
 ('all', 2),
 ('mimsy', 2),
 ('were', 2),
 ('borogoves', 2),
 ('mome', 2),
 ('raths', 2),
 ('outgrabe', 2),
 ('beware', 2),
 ('that', 2),
 ('his', 2),
 ('vorpal', 2),
 ('stood', 2),
 ('thought', 2),
 ('as', 2),
 ('with', 2),
 ('came', 2),
 ('it', 2),
 ('one', 2),
 ('two', 2),
 ('went', 2)]

Trigrams

All possible sequences of 3 words in the poem.

list(zip(words[:], words[1:], words[2:]))[:10]

[('jabberwocky', 'twas', 'brillig'),
 ('twas', 'brillig', 'and'),
 ('brillig', 'and', 'the'),
 ('and', 'the', 'slithy'),
 ('the', 'slithy', 'toves'),
 ('slithy', 'toves', 'did'),
 ('toves', 'did', 'gyre'),
 ('did', 'gyre', 'and'),
 ('gyre', 'and', 'gimble'),
 ('and', 'gimble', 'in')]

import itertools as it

def window(x, n):
    """Sliding window of size n from iterable x."""
    s = (it.islice(x, i, None) for i in range(n))
    return zip(*s)

list(window(words, 3))[:10]

[('jabberwocky', 'twas', 'brillig'),
 ('twas', 'brillig', 'and'),
 ('brillig', 'and', 'the'),
 ('and', 'the', 'slithy'),
 ('the', 'slithy', 'toves'),
 ('slithy', 'toves', 'did'),
 ('toves', 'did', 'gyre'),
 ('did', 'gyre', 'and'),
 ('gyre', 'and', 'gimble'),
 ('and', 'gimble', 'in')]

Using Regular Expressions

• Regular Expression HOWTO
• Test your Python regular expressions

Find all words with a sequence of two or more identical letters e.g. “look”

import re

regex = re.compile(r'(\w*(\w)\2+\w*)', re.IGNORECASE | re.MULTILINE)

for match in regex.finditer(poem):
    print(match.group(2), match.group(1))

b jabberwocky
l brillig
l all
b jabberwock
o took
e tree
o stood
f uffish
o stood
b jabberwock
f whiffling
o wood
b jabberwock
o callooh
l callay
l brillig
l all

Convert the identical sequences to uppercase

def f(match):
    word, letter = match.groups()
    return word.replace(letter, letter.upper())

print(regex.sub(f, poem))

jaBBerwocky

     twas briLLig and the slithy toves
      did gyre and gimble in the wabe
     aLL mimsy were the borogoves
      and the mome raths outgrabe

     beware the jaBBerwock my son
      the jaws that bite the claws that catch
     beware the jubjub bird and shun
      the frumious bandersnatch

     he tOOk his vorpal sword in hand
      long time the manxome foe he sought
     so rested he by the tumtum trEE
      and stOOd awhile in thought

     and as in uFFish thought he stOOd
      the jaBBerwock with eyes of flame
     came whiFFling through the tulgey wOOd
      and burbled as it came

     one two one two and through and through
      the vorpal blade went snickersnack
     he left it dead and with its head
      he went galumphing back

     and hast thou slain the jaBBerwock
      come to my arms my beamish boy
     o frabjous day callOOh caLLay
      he chortled in his joy

     twas briLLig and the slithy toves
      did gyre and gimble in the wabe
     aLL mimsy were the borogoves
      and the mome raths outgrabe

Natural language processing

If you intend to perform statistical analysis on natural language, you should probably use NLTK to pre-process the text instead of using string methods and regular expressions. For example, a simple challenge is to first parse the paragraph below into sentences, then parse each sentence into words.

Paragraph from random Pubmed abstract.

para = """When compared with the control group no significant associations were found for the NS-PEecl group after adjustment of confounding variables. For the S-PEecl group, antiβ2GP1 IgG (OR 16.91, 95% CI 3.71-77.06) was associated, as well as age, obesity, smoking and multiparity. Antiβ2GP1-domain I IgG were associated with aCL, antiβ2GP1 and aPS/PT IgG in the three groups. aPS/PT IgG were associated with aCL IgG, and aPS/PT IgM were associated with aCL and antiβ2GP1 IgM in the three groups CONCLUSION: S-PEecl is a distinct entity from NS-PEecl and is mainly associated with the presence of antiβ2GP1 IgG. Antiβ2GP1 domain I correlate with other aPL IgG tests, and aPS/PT may be promising in patients in which LA tests cannot be interpreted."""

Naive splitting of sentences as anything separated by ., ! or ?

sep = re.compile(r'[\?\!\.]')

ss = sep.split(para)

for i, s in enumerate(ss, 1):
    print(i, ':', s, end='\n\n')

1 : When compared with the control group no significant associations were found for the NS-PEecl group after adjustment of confounding variables

2 :  For the S-PEecl group, antiβ2GP1 IgG (OR 16

3 : 91, 95% CI 3

4 : 71-77

5 : 06) was associated, as well as age, obesity, smoking and multiparity

6 :  Antiβ2GP1-domain I IgG were associated with aCL, antiβ2GP1 and aPS/PT IgG in the three groups

7 :  aPS/PT IgG were associated with aCL IgG, and aPS/PT IgM were associated with aCL and antiβ2GP1 IgM in the three groups CONCLUSION: S-PEecl is a distinct entity from NS-PEecl and is mainly associated with the presence of antiβ2GP1 IgG

8 :  Antiβ2GP1 domain I correlate with other aPL IgG tests, and aPS/PT may be promising in patients in which LA tests cannot be interpreted

9 :

Using NLTK

import nltk

ss_nltk = nltk.sent_tokenize(para)

for i, s in enumerate(ss_nltk, 1):
    print(i, ':', s, end='\n\n')

1 : When compared with the control group no significant associations were found for the NS-PEecl group after adjustment of confounding variables.

2 : For the S-PEecl group, antiβ2GP1 IgG (OR 16.91, 95% CI 3.71-77.06) was associated, as well as age, obesity, smoking and multiparity.

3 : Antiβ2GP1-domain I IgG were associated with aCL, antiβ2GP1 and aPS/PT IgG in the three groups.

4 : aPS/PT IgG were associated with aCL IgG, and aPS/PT IgM were associated with aCL and antiβ2GP1 IgM in the three groups CONCLUSION: S-PEecl is a distinct entity from NS-PEecl and is mainly associated with the presence of antiβ2GP1 IgG.

5 : Antiβ2GP1 domain I correlate with other aPL IgG tests, and aPS/PT may be promising in patients in which LA tests cannot be interpreted.

Naive parsing of the second sentence into words

s = ss_nltk[1]
s

'For the S-PEecl group, antiβ2GP1 IgG (OR 16.91, 95% CI 3.71-77.06) was associated, as well as age, obesity, smoking and multiparity.'

# remove punctuation and split on whit space
table = dict.fromkeys(map(ord, string.punctuation))
s.translate(table).split()

['For',
 'the',
 'SPEecl',
 'group',
 'antiβ2GP1',
 'IgG',
 'OR',
 '1691',
 '95',
 'CI',
 '3717706',
 'was',
 'associated',
 'as',
 'well',
 'as',
 'age',
 'obesity',
 'smoking',
 'and',
 'multiparity']

Using NLTK

text = nltk.word_tokenize(s)
text

['For',
 'the',
 'S-PEecl',
 'group',
 ',',
 'antiβ2GP1',
 'IgG',
 '(',
 'OR',
 '16.91',
 ',',
 '95',
 '%',
 'CI',
 '3.71-77.06',
 ')',
 'was',
 'associated',
 ',',
 'as',
 'well',
 'as',
 'age',
 ',',
 'obesity',
 ',',
 'smoking',
 'and',
 'multiparity',
 '.']

NLTK is a rich library for natural language processing

See http://www.nltk.org for details.

Tag tokens with part-of-speech labels

tagged_text = nltk.pos_tag(text)
tagged_text

[('For', 'IN'),
 ('the', 'DT'),
 ('S-PEecl', 'NNP'),
 ('group', 'NN'),
 (',', ','),
 ('antiβ2GP1', 'RB'),
 ('IgG', 'NNP'),
 ('(', '('),
 ('OR', 'NNP'),
 ('16.91', 'CD'),
 (',', ','),
 ('95', 'CD'),
 ('%', 'NN'),
 ('CI', 'NNP'),
 ('3.71-77.06', 'CD'),
 (')', ')'),
 ('was', 'VBD'),
 ('associated', 'VBN'),
 (',', ','),
 ('as', 'RB'),
 ('well', 'RB'),
 ('as', 'IN'),
 ('age', 'NN'),
 (',', ','),
 ('obesity', 'NN'),
 (',', ','),
 ('smoking', 'NN'),
 ('and', 'CC'),
 ('multiparity', 'NN'),
 ('.', '.')]

s

'For the S-PEecl group, antiβ2GP1 IgG (OR 16.91, 95% CI 3.71-77.06) was associated, as well as age, obesity, smoking and multiparity.'

A simplistic way to pick up nouns

[w for w, t in tagged_text if t.startswith('N')]

['S-PEecl',
 'group',
 'IgG',
 'OR',
 '%',
 'CI',
 'age',
 'obesity',
 'smoking',
 'multiparity']

String formatting

Selection

import math

stuff = ('bun', 'shoe', ['bee', 'door'], 2, math.pi, 0.05)

'One: {}, Two {}'.format(*stuff)

'One: bun, Two shoe'

'One: {0}, Two {1}'.format(*stuff)

'One: bun, Two shoe'

'One: {1}, Two {1}'.format(*stuff)

'One: shoe, Two shoe'

'One: {0}, Two {2[1]}'.format(*stuff)

'One: bun, Two door'

Formatting

'One: {0:^10s}, Two {1:_>15s}'.format(*stuff)

'One:    bun    , Two ___________shoe'

'One: {3}, Two {4}'.format(*stuff)

'One: 2, Two 3.141592653589793'

'One: {3:+10d}, Two {4:.4f}'.format(*stuff)

'One:         +2, Two 3.1416'

'One: {3:04d}, Two {4:.4g}'.format(*stuff)

'One: 0002, Two 3.142'

'One: {3:.4e}, Two {4:.4e}'.format(*stuff)

'One: 2.0000e+00, Two 3.1416e+00'

'One: {5:.2%}, Two {5:f}'.format(*stuff)

'One: 5.00%, Two 0.050000'

Old style formatting

'%s, %s, %a, %d, %.4f, %.2f' % stuff

"bun, shoe, ['bee', 'door'], 2, 3.1416, 0.05"

Formatting numpy arrays

import numpy as np

x = np.arange(1, 13).reshape(3,4)
x

array([[ 1,  2,  3,  4],
       [ 5,  6,  7,  8],
       [ 9, 10, 11, 12]])

np.set_printoptions(formatter={'int': lambda x: '%8.2f' % x})

x

array([[    1.00,     2.00,     3.00,     4.00],
       [    5.00,     6.00,     7.00,     8.00],
       [    9.00,    10.00,    11.00,    12.00]])

np.set_printoptions()

x

array([[ 1,  2,  3,  4],
       [ 5,  6,  7,  8],
       [ 9, 10, 11, 12]])