Python 的多线程


Author: yifei / Created: Nov. 9, 2017, 3:30 a.m. / Modified: Nov. 9, 2017, 11:31 a.m. / Edit

高层次多线程

by instanciante a threading.Thread

import threading

def worker(i):
    """thread worker function"""
    print 'Worker {}'.format(i)

threads = []
for i in range(5):
    t = threading.Thread(name='worker', target=worker, args=(i,))
    threads.append(t)
    t.start()

by default, it does not start running

t.start start executing the thread
t.is_alive  
t.join  wait for a thread, daemon thread can't be joined

subclassing the Thead class is not recommended, which makes the code coupling

Each Thread has its own stack, so when a child thread throws a exception, the main thread will not catch it

http://stackoverflow.com/questions/2829329/catch-a-threads-exception-in-the-caller-thread-in-python

Daemon 化线程

默认情况下,主线程等待所有子线程的执行。设定为 Daemon 线程后,主线程继续执行,并不等待,但当主线程退出时会杀掉所有子线程。 使用t.set_daemon设定为 Daemon 线程 使用t.join(timeout)等待子线程完成,如果 timeout 之内没有完成,继续执行 threading.enumerate()返回所有活动的线程 threading.current_thread()

如果想要使用Ctrl-C,必须设定为daemon

Use the python-daemon package follows https://www.python.org/dev/peps/pep-3143/ Use supervisord Things to notice

toutiao ways of daemonizing

main thread put to infinite loop with sleep put threads in daemon mode, so that when you terminate the main thread, the worker thread also got terminated~ use svc to start the program use exec to avoid creating a new process

ref: https://pymotw.com/2/threading/

shell tricks use help command to view help of builtin command

New thoughts on docker

maybe we don't need docker to manage the user space daemons, we could just use the daemontools package or supervisord

https://news.ycombinator.com/item?id=9793466

关于 coroutine,首先,他们是线程,用户级线程,也就是说虽然他们的代价比较小,但是如果递归调用,很可能会创建大量的协程。在线程中,我们显然不能创建数量过大的线程,因此,也不能无限地创造过多的协程。

[1] https://medium.com/python-pandemonium/asyncio-coroutine-patterns-beyond-await-a6121486656f [2] http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/ [3] https://medium.com/@yeraydiazdiaz/asyncio-coroutine-patterns-errors-and-cancellation-3bb422e961ff

线程池

multiporcessing.Pool.map(fn, iterable) only accepts marshalable  

# 线程池的例子
 
from multiprocessing.dummy import Pool as ThreadPool  # python2
from multiprocessing.pool import ThreadPool # python3
 
def square_number(n):
    return n ** 2
 
# function to be mapped over
def calculate_parallel(numbers, threads=2):
    pool = ThreadPool(threads)
    results = pool.map(square_number, numbers)
    pool.close()  # NOTE close before join
    pool.join()
    return results
 
if __name__ == "__main__":
    numbers = [1, 2, 3, 4, 5]
    squared_numbers = calculate_parallel(numbers, 4)
    for n in squared_numbers:
        print(n)

  使用pool的一个陷阱是不太好debug, 爆出的异常往往看不清问题, 需要使用单线程调试之后再去


评论区