Python - Multithreading

From Torben's Wiki

Preferred: Threading

import threading
import logging
import time

# Python’s built-in data structures (lists, dictionaries, etc.) are thread-safe
#  as a side-effect of having atomic byte-codes for manipulating them
# -> so locking is them is not required.
# Other data structures implemented in Python, or simpler types like integers and floats, don’t have that protection.
# from inside a thread it is possible to manipulate dic but not integers

# Logging is nicer than print, as it can automatically add the threadname
logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(levelname)s %(threadName)s: %(message)s',
                    )

# List of "work" to to
listPileOfWork = [x for x in range(7)]

# Threads can not write in integers, but in dictionaries !
# counter = 0
dic = {}
dic["Counter"] = 0

def worker():
    logging.info('Starting')
    # print(threading.currentThread().getName(), 'Starting')
    while listPileOfWork :
        x = listPileOfWork.pop(0)
        logging.info("Took" + str(x))
        dic["Counter"] += x
        time.sleep(2)
    logging.info('No more work to do')


threads = []  # List of threads

for i in range(4):
    t = threading.Thread(name='myThread'+str(i), target=worker)
    threads.append(t)
    t.start()

# Wait for all threads to complete
for t in threads:
    t.join()
logging.info('All threads done')

Alternative: multiprocessing

From [1]: The threading module uses threads, the multiprocessing module uses processes. The difference is that threads run in the same memory space, while processes have separate memory. This makes it a bit harder to share objects between processes with multiprocessing. Since threads use the same memory, precautions have to be taken or two threads will write to the same memory at the same time. This is what the global interpreter lock is for. Spawning processes is a bit slower than spawning threads. Once they are running, there is not much difference.

Example from [2]

Prob: KeyboardInterrupt (CTRL+C) does not close program completely see [3] as starting point for a solution

from multiprocessing import Pool

def escapes(cr, ci, it):
    """Does iterating z <- z^2 + c escape after it iterations?"""
    zr = 0.0
    zi = 0.0
    for i in xrange(it):
        # z <- z^2 + c
        zr,zi = zr*zr - zi*zi + cr, 2*zr*zi + ci
        if zr*zr + zi*zi > 4:
            return True
    return False

def toChar(p):
    if p: return " "
    else: return "X"

def doRow((xmin,xmax,xstep, ymin,ymax,ystep, iterations, yc)):
    """Calculate one row of the output."""
    y = yc*(ymax-ymin)/ystep + ymin
    row = []
    for xc in xrange(xstep):
        x = xc*(xmax-xmin)/xstep + xmin
        row.append( escapes(x, y, iterations) )
    return "".join([toChar(p) for p in row])

def mandel(xmin,xmax,xstep, ymin,ymax,ystep, iterations):
    """Calculate and print a Mandelbrot set."""
    pool = Pool()  # askes the os for num of cpus ;-)
    results = []
    for yc in xrange(ystep):
        res = pool.apply_async(doRow, ((xmin,xmax,xstep, ymin,ymax,ystep, iterations, yc),))
        results.append(res)
    for yc in xrange(ystep):
        print results[yc].get()

if __name__=="__main__":
    mandel(-2.0, 1.0, 80, -1.0, 1.0, 24, 20000)