Networking and Concurrency
Moving away from specific web frameworks and data stores, dealing with concurrency and high-volume networking is always a challenge. When you are serving three users, you don't stress your code the same way you do when you are serving 30,000 users (all concurrently).
It is also worth pointing out that it is not just networking that is the issue; it is concurrency. Concurrency is a common problem across many programming languages, and high-traffic networking is just a natural testing ground for constructs that make it easier to handle high concurrency. Nevertheless, the principles discussed in this track are applicable to any high-throughput process.
There are eight different talks in this track:
Extreme Network Programming with Python and Linux by Rob Ludwick ( Extreme ). Traditionally, C is the preferred language for low level network programming and works well for those who have the time and patience to work with it. As it turns out, Python is very capable for prototyping low level network code, collecting data, and testing ideas quickly without getting lost in the land of C. Obscure topics such as raw sockets, multicast, network bridging, rolling your own vpn, and disruption tolerant networking will be covered. This talk will show how you can use Python to build custom protocols, debug a network, fix broken nets, implement custom logging and processing, and simulate network traffic.
Jython Concurrency by Jim Baker (Extreme). One of the persistent troubles with concurrency in Python is the GIL - the Global Interpreter Lock used to simplify the implementation of CPython... but not all Python implementations include a GIL. Jython implements the Python language, but Jython leverages the underlying Java platform to provide an opinionated alternative to CPython in its support of concurrency. Jython instead embraces threads, provides extensive support for managing their execution and coordination through standard Java platform functionality (java.util.concurrent), and works well with Jython's implementation of {ython's standard mutable collection types. The underlying JVM provides also extensive instrumentation as well as the ability to set a variety of parameters, including choice of GC. This talk will go into a detailed discussion of some of the interesting ramifications of these design points and how they can be effectively applied to write concurrent code, as illustrated through a variety of short examples.
Ten Years of Twisted by Glyph Lefkowitz. Twisted is one of the oldest event-driven architectures for Python, and it is the oldest one still being actively maintained and extended with new functionality. The maturity that Twisted brings to event-based networking is essential - they have found and fixed bugs that other asynchronous architectures may not hit for a couple years. Despite this pedigree, however, many aspects of Twisted remain misunderstood or simply unrealized. This talk will present a brief conceptual introduction to Twisted, followed by a survey of its features, their status, and how development has been proceeding over the years, with a special focus on the last two years of sponsored development.
Using Coroutines to Create Efficient, High-Concurrency Web Applications by Matt Spitz ( Extreme ). Many people don't know that the popular web-based instant messaging service Meebo is delivered using Python. At Meebo, they have settled on using gunicorn, a lightweight WSGI server, which supports gevent, a coroutine-based network library for python. Gevent monkeypatches python's system modules to make network requests asynchronous using an event loop based on libevent. This trick allows the developer to use a simple blocking CGI as a non-blocking web application that can handle many concurrent requests. In this in-depth review, Matt discusses how Meebo worked through the various approaches to building web applications, why they ended up choosing gunicorn+gevent, the challenges this new framework presents, and how they've dealt with them.
Prototyping Go's Select with stackless.py for Stackless Python by Andrew Francis (Extreme). Google’s introduction of the Go language raised eyebrows in the Stackless Python community. Although very different languages, Go and Stackless Python’s concurrency model share a common ancestor: the Bell Labs family of languages (i.e., Newsqueak, Limbo). The common feature are channels: a synchronous message passing mechanism based on Tony Hoare’s Communicating Sequential Processes (CSP).
Both Go and Stackless Python have channels - but the select language built into Go gives it the ability to wait on multiple channels simultaneously. This talk explores the prototyping potential of stackless.py, the PyPy's framework's implementation of Stackless Python. This talk will present a brain-bending "case study" involving prototyping Go's select in stackless.py before reimplementing select in C based Stackless Python.
An Introduction to Tornado by Gavin Roy. Tornado is an open source version of the scalable, non-blocking web server and tools that power FriendFeed. It is not only a web server but it is a light-weight, use only what you need, web development framework. This talk will review the current state of the Tornado project, review the features Tornado provides and give examples of how to implement asynchronous web applications in Tornado.
Advanced Network Architectures With ZeroMQ by Zed Shaw (Extreme). Zed Shaw has raised a lot of interest with his recent work on Mongrel2, a "language agnostic" webserver that uses ZeroMQ to handle all the plumbing. This talk builds on Zed's experience with Mongrel2 to demonstrate how to use ZeroMQ with Python (and others) to do really advanced or even weird network architectures. Zed will show Python talking to other languages, handling HTTP, JSON, XML, WebSockets, encoding videos, handling chat messaging, etc. This is an extreme talk, so it will be in code, not in diagrams. This talk will assume you know ZeroMQ and Python, but if you don't know ZeroMQ you can probably still keep up.
An outsider's look at co-routines by Peter Portante. This talk is designed to take an interested beginner from a hazy understanding of coroutines up to the point where they can understand the essential concepts (and the essential differences!) associated with the different implementations of coroutines for Python.
Is this the conference you want to see? Then, register for PyCon and book your room now! We have picked up a few more rooms - including a few at a lower rate a block away. You can email (pycon4-reg@cteusa.com), or phone (847-759-4277). We have very few spots left.
Links:
- Go to Part 1 of this series
- Go to Part 2 of this series
- Go to Part 3 of this series
- Go to this post on the PyCon site.
Comments