developerWorks Premium
An all-access pass to building your next great app!
developerWorks Community
In some ways the design of Python resembles the design of Java. Both utilize a virtual machine that interprets specialized pseudo-compiled bytecodes.
One area where JVMs are more advanced than Python is in optimizing the execution of bytecodes. Psyco, a Python specializing compiler, helps to even the playing field. Right now Psyco is an external module, but it could someday be included in Python itself.
With only a tiny amount of extra programming, Psyco can often be used to increase the speed of Python code by orders of magnitude. In this article, David looks at what Psyco is, and tests it in some applications.
(171 KB) |
(), Freelance writer, Gnosis Software, Inc.
David Mertz' failures as a hunter, fisherman, and shepherd have led him to
his life of critical criticism.
Tomorrow he may try something else.
his life pored over at
Suggestions and recommendations on this, past, or future columns are welcome.
Also available in&&
Python is usually fast enough for what you want it to do.
Ninety percent
of the concerns that novice programmers express about the execution speed
of an interpreted/byte-compiled language like Python are simply naive.
modern hardware, most non-optimized Python programs run as fast as they
need to, and there is really no point in spending extra programming effort
to make an application run faster.
In this installment, therefore, I'm only interested in that other ten
percent. Once in while, Python programs (or programs in other languages)
run impracticably slowly.
What is needed for a given purpose varies
shaving off milliseconds is rarely compelling, but speeding up
tasks that run for minutes, hours, days, or weeks is often worthwhile.
Moreover, it should be noted that not everything that runs slowly is
CPU-bound. If a database query takes hours to complete, for example, it
makes little difference whether the resulting dataset takes one or two
minutes to process.
This installment is also not about I/O issues.
There are a number of ways to speed up Python programs.
technique that should come to every programmer's mind is improving the
algorithms and data structures used.
Micro-optimizing the steps of an
inefficient algorithm is a fool's errand.
For example, if the complexity
order of the current technique is O(n**2), speeding up the steps by 10x is
a lot less helpful than finding an O(n)
substitute.
This moral applies
even when considering an approach as extreme as a rewrite in assembly: the
right algorithm in Python will often go faster than the wrong algorithm in
hand-tuned assembly.
The second technique you ought to consider first is to profile your Python
application, with an eye to rewriting key portions as C extension modules.
Using an extension wrapper like SWIG (see ), you can create a C extension that
executes the most time consuming elements of your program as C code.
Extending Python in this manner is relatively straightforward, but require
some time to learn (and knowledge of C).
Very often you will find that
the large majority of the time spent in executing your Python application
is spent in just a handful of functions, so considerable gains are
A third technique builds on the second.
Greg Ewing has created a language
called Pyrex, which melds Python and C.
In particular, to use Pyrex, you
write functions in a Python-like language that adds type-declarations to
selected variables.
Pyrex (the tool) processes ".pyx" files into ".c"
extensions.
Once compiled with a C compiler, these Pyrex (the language)
modules can be imported into and used in your regular Python applications.
Since Pyrex uses almost the same syntax as Python itself (including loop,
branch and exception statements, assignment forms, block indentation,
etc.) a Pyrex programmer need not learn C to write extensions.
Pyrex allows more seamless mixing of C-level variables and Python-level
variables (objects) within the same code than does an extension written
directly in C.
A final technique is the subject of this installment.
The extension
module Psyco can plug in to the guts of the Python interpreter, and
selectively substitute optimized machine code for portions of Python's
interpreted bytecode.
Unlike the other techniques described, Psyco
operates strictly at Python runtime.
That is, Python source code is
compiled by the python command to bytecode in
exactly the same manner as before (except for a couple import statements and function calls added to invoke
But while the Python interpreter is running an application, Psyco
sometimes checks to see if it can substitute some specialized machine code
for regular Python bytecode actions.
This specialized compilation is
both very similar to what Java just-in-time compilers do (broadly
speaking, at least) and is architecture-specific.
As of right now, Psyco
is only available for i386 CPU architectures.
The nice thing about Psyco
is that you can use the very same Python code you have been writing all
along (literally!), but let it run much faster.
How Psyco works
To understand Psyco completely, you probably need to have a good grasp of
both the Python interpreter's eval_frame()
function and i386 Assembly.
Unfortunately, I myself can claim neither
expertise -- but I think I can explain Psyco in outline without going too
far wrong.
In regular Python, the eval_frame() function is
the inner loop of the Python interpreter.
Basically, the eval_frame() function looks at the current bytecode
in an execution context, and switches control out to a function
appropriate for implementing that bytecode.
The specifics of what this
support function will do depend, in general, upon the states of various
Python objects held in memory.
To make it simple, adding the Python
objects "2" and "3" produces a different result than adding the objects
"5" and "6," but both operations are dispatched in a similar way.
Psyco replaces the eval_frame() function with a
compound evaluation unit.
There are several ways that Psyco is able to
improve upon what Python does.
In the first place, Psyco compiles
operations to somewhat op in itself this produces
only slight improvements, since what the machine code needs to accomplish
is the same as what Python's dispatched functions do.
Moreover, what is
"specialized" in Psyco compilation is more than the choice of Python
bytecodes, Psyco also specializes over variable values that are known in
execution contexts.
For example, in code like the below, the variable
x is knowable for the duration of the loop:
for i in range(1000):
l.append(x*i)An optimized version of this code need not multiply each i by "the content of the x variable/object" -- it is
less expensive to simply multiply each i by 5,
saving a lookup/dereference.
Aside from creating i386-specific codes for small operations, Psyco caches
this compiled machine code for later reuse.
If Psyco is able to recognize
that a particular operation is the same as something that was performed
(and "specialized")
earlier, it can rely on this cached code, rather than
need to recompile the segment.
This saves a bit more time.
The real savings in Psyco, however, arise from Psyco's categorization of
operations into three different levels.
For Psyco, there are "run-time,"
"compile-time" and "virtual-time" variables.
Psyco promotes and demotes
variables between the levels as needed.
Run-time variables are simply the
raw bytecodes and object structures that the regular Python interpreter
Compile-time variables are represented in machine registers and
directly-accessed memory locations, once the operations have been compiled
by Psyco into machine code.
The most interesting level is the virtual-time variables.
variable is, internally, a complete structure, with lots of members --
even when the object only represents an integer.
Psyco virtual-time
variables represent Python objects that could potentially be built if the
need arose, but whose details are omitted until they are.
For example,
consider an assignment like:
x = 15 * (14 + (13 - (12 / 11)))Standard Python builds and destroys a number of objects to compute this
An entire integer object is built to hold the value of (12/11); then a value is pulled out of the temporary
object's structure, and used to compute a new temporary object (13-PyInt).
Psyco skips the objects, and just
computes the values, knowing that an object can be created "if needed"
from the value.
Using Psyco
Explaining Psyco is relatively difficult, but using Psyco is far
Basically, all there is to it is telling the Psyco module which
functions/methods to "specialize." No code changes need be made to any of
your Python functions and classes themselves.
There are a couple approaches to specifying what Psyco should do.
"shotgun" approach is to enable Psyco just-in-time operation everywhere.
To do that, put the following lines at the top of your module:
psyco.jit()
from psyco.classes import *The first line tells Psyco to do its magic on all global functions.
second line (in Python 2.2 and above) tells Psyco to do the same with
class methods.
To target Psyco's behavior a bit more precisely, you can
use the commands:
psyco.bind(somefunc)
# or method, class
newname = psyco.proxy(func)The second form leaves func as a standard
Python function, but optimizes calls involving newname.
In almost all cases other than testing and
debugging, the psyco.bind() form is what you
Psyco's performance
As magic as Psyco is, using it still requires a little thought and
The main thing to understand is that Psyco is useful for
handling blocks that loop many times, and it knows how to optimize
operations involving integers and floating point numbers.
For non-looping functions, and for operations on other types of objects,
Psyco mostly just adds overhead for its analysis and internal-compilation.
Moreover, for applications with large numbers of functions and classes,
enabling Psyco application-wide adds a large burden in machine-code
compilation and memory-usage for this caching.
It is far better to
selectively bind those functions that can benefit most from Psyco's
optimizations.
I started my testing in a completely naive fashion.
I simply considered
what application I had run recently that I would not mind speeding up.
The first example that came to mind was a text-manipulation program I use
to convert drafts of my forthcoming book (Text Processing in
Python) to LaTeX format.
This application uses some string
methods, some regular expressions, and some program logic driven mostly by
regular expressions and string matches.
It is actually a terrible
candidate for Psyco, but I use it, so I started there.
On the first pass, all I did was add psyco.jit() to the top of my script.
Unfortunately, the results were (expectedly) disappointing.
Where the script initially took about 8.5 seconds to run, after Psyco's
"speedup" it ran in about 12 seconds.
Not so good!
I guessed that the
just-in-time compilation probably had some startup overhead that swamped
the running time.
So the next thing I tried was processing a much larger
input file (consisting of multiple copies of the original one).
the very limited success of reducing running time from about 120 seconds
to 110 seconds.
The speedup was consistent across several runs, but
fairly insignificant either way.
Second pass with my text processing candidate.
Instead of adding a
blanket psyco.jit() call, I added only the line
psyco.bind(main), since the main() function does loop a number of times
(but only makes minimal use of integer arithmetic).
The results here were
nominally better.
This approach shaved a few tenths of a second off the
normal running time, and a few seconds off the large-input version.
still nothing spectacular (but also no harm done).
For a more relevant test of Psyco, I dug up some neural network code that
I had written about in an earlier article (see Resources).
"code_recognizer" application can be trained to recognize the likely
distribution of different ASCII values in different programming languages.
Potentially, something like this could be useful in guessing file types
(say of lost network packets); but the code is actually completely generic
as to what it is trained on -- it could learn to recognize faces, or
sounds, or tidal patterns just as easily.
In any case, "code_recognizer"
is based on the Python library
which is also included (in modified form) as a test case with the Psyco
0.4 distribution.
The important thing to know about "code_recognizer" for
this article is that it does a lot of looping floating point math, and it
takes a long time to run.
We have got a good candidate for Psyco to work
After a little playing around, I established several details about how to
use Psyco.
For this application, with just a small number of classes and
functions, it does not make too much difference whether you use
just-in-time or targeted binding.
But the best result, by a few
percentage points, still comes about by selectively binding the best
optimizable classes.
More significantly, however, it is important to
understand the scope of Psyco binding.
The code_recognizer.py script contains lines
from bpnn import NNclass NN2(NN):
# customized output methods, math core inheritedThat is, the interesting stuff from Psyco's point of view is in the class
Adding either psyco.jit() or psyco.bind(NN2) to the code_recognizer.py script has little effect.
Psyco to do the desired optimization, you need to either add psyco.bind(NN) to code_recognizer.py or add psyco.jit() to bpnn.py.
Contrary to what you might assume, just-in-time does not happen when an
instance is created, or methods run, but rather in the scope where the
class is defined.
In addition, binding descendent classes does not
specialize their methods that are inherited from elsewhere.
Once the small details of proper Psyco binding are worked out, the
resultant speedups are rather impressive.
Using the same test cases and
training regime the referenced article presented (500 training patterns,
1000 training iterations), neural net training time was reduced from about
2000 seconds to about 600 seconds -- better than a 3x speedup.
the iterations as low as 10 showed proportional speedups (but worthless
neural net recognition), as did intermediate numbers of iterations.
I find bringing running time down from more than 1/2 hour to about 10
minutes with two lines of new code to be quite remarkable.
This speedup
is still probably less than the speed of a similar application in C, and
it is certainly less than the 100x speedup that a few isolated Psyco test
cases exhibit.
But this application is fairly "real life" and the
improvements are enough to be significant in many contexts.
Whither Psyco?
Psyco currently does not perform any sort of internal statistics or
profiling, and does only minimal optimization of generated machine code.
Potentially, a later version might know how to target those Python
operations that could actually benefit most, and discard cached machine
code for non-optimizable sections.
In addition, perhaps a future Psyco
could decide to perform more extensive (but more costly)
optimizations on
heavily-run operations. Such runtime analysis would be similar to what
Sun's HotSpot technology does for Java. The fact that Java, unlike Python,
has type-declarations is actually less significant than many people assume
(but prior work in optimization of Self, Smalltalk, Lisp, and Scheme make
this point also).
Although I suspect it will never actually happen, it would be exciting to
have Psyco-type technology integrated into some future version of Python
A few lines for imports and bindings is not much to do, but
letting Python just inherently run much faster would be even more
We shall see.
ResourcesRead the
of Charming Python.
Find more information at
at SourceForge.
is a very widely -- perhaps
predominantly -- used tool for writing C/C++ modules for Python and other
"scripting" languages.Greg Ewing has created , which is used for writing Python extension modules.
The idea behind Pyrex is to define a language that looks very close to
Python itself, and that allows a mixture of Python and C datatypes to be
combined, but which is ultimately transformed and compiled into a Python
C-extension. John Max Skaller's Vyper language was intended to be an enhanced
Python, implemented in OCaml.
One upshot that was hoped for in the
project was compilation to the same machine code OCaml generates, which is
generally comparable with the speed of C.
Unfortunately, Vyper is a dead
project, and a compiling version was never completed.
back when the
project was alive (developerWorks, October 2000).David co-authored with Andrew Blais
(developerWorks, July 2001).
article, they provided some code based on Neil Schemenauer's Python module
The current article utilizes that
neural network code to demonstrate Psyco's capabilities.
in the developerWorks Linux zone.
developerWorks: Sign in
Required fields are indicated with an asterisk (*).
Password:*
Keep me signed in.
By clicking Submit, you agree to the .
The first time you sign into developerWorks, a profile is created for you.
Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name.
You may update your IBM account at any time.
All information submitted is secure.
Choose your display name
The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name.
Your display name accompanies the content you post on developerWorks.
Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.
Required fields are indicated with an asterisk (*).
Display name:*(Must be between 3 & 31 characters.)
By clicking Submit, you agree to the .
All information submitted is secure.
Dig deeper into Linux on developerWorks
Exclusive tools to build your next great app. Learn more.
Technical resources for innovators and early adopters to experiment with.
Evaluate IBM software and solutions, and transform challenges into opportunities.
static.content.url=/developerworks/js/artrating/SITE_ID=1Zone=LinuxArticleID=11253ArticleTitle=Charming Python: Make Python run as fast as C with Psycopublish-date=Psyco首页、文档和下载 - 让 Python 运行得像 C 一样快 - 开源中国社区
当前访客身份：游客 [
当前位置：
Python 的设计在很多方面都类似于 Java 的设计。两者都利用了解释专门的伪编译字节码的虚拟机。JVM 比 Python 更高级的一个方面在于优化了字节码的执行。Psyco，一种 Python 专用编译器，帮助平衡了这一竞争。Psyco 现在是个外部模块，但是在将来的某一天它可能会包括到 Python 本身中去。只需极少量的额外编程，通常就可以使用 Psyco 将 Python 代码的速度提高好几个数量级。
Psyco最新更新资讯，共1条&&()
授权协议： MIT/X
开发语言：
操作系统：&跨平台&
收录时间： 日
红薯 发表于4年前
，最后回答(3年前)：
使用 Psyco 中的任何问题
：说的真好
：三年不更新罗
： 这个已经死了是吗,得用pypy了?
共有 108 个类似软件
Eclipse是著名的跨平台的自由集成开发环境（IDE）。最初主要用来Java语言开发，但是...
Spyder (前身是 Pydee) 是一个强大的交互式 Python 语言开发环境，提供高级的代码编...
Codimension 是一个 Python 集成开发环境，提供了包括代码编辑和基于图表的代码分析...
PTVS (Python Tools for Visual Studio) 是一个开源项目，采用Apache 2.0许可发布。...
Rodeo 是一个以数据为中心的 Python 集成开发环境，基于 Web 浏览器使用。 安装： ...
QPython是一个在Android上运行Python脚本引擎，他整合了Python解释器、Console、编...
Python-mode 是一个 vim 插件，允许你在 vim 编辑器中使用工具：pylint, rope, pyd...
Pyston 是一个 Dropbox 推出的新的基于 JIT 的 Python 2.7 的实现。Pyston 解析 Py...
PyFormat 是记录 Python 字符串格式化系统的实例的项目。python.org 的官方文档中，...
PyParallel是Trent Nelson发起的一个研究项目，其目标是以提供高性能异步支持的方式...
共有 46 人关注 PsycoPython下使用Psyco模块优化运行速度
投稿：junjie
字体：[ ] 类型：转载 时间：
这篇文章主要介绍了Python下使用Psyco模块优化运行速度,Psyco模块可以使你的Python程序运行的像C语言一样快,本文给出了多个代码示例,并讲解了Psyco的安装和使用方法,需要的朋友可以参考下
今天介绍下Psyco模块，Psyco模块可以使你的Python程序运行的像C语言一样快。
都说Python语言易用易学，但性能上跟一些编译语言（如C语言）比较要差不少，这里可以用C语言和Python语言各编写斐波纳契数列计算程序，并计算运行时间：
int fib(int n){
&& if (n & 2)
&&&& return fib(n - 1) + fib(n - 2);
int main() {
&&& fib(40);
&&& return 0;
Python写的
def fib(n):&
& if n & 2:&
&&&& return n&
&&&& return fib(n - 1) + fib(n - 2)&
$ time ./fib
$ time python fib.py
可以看到运行时间还是有点差距的，这里的差距大概是5倍左右，现在就介绍Psyco：
Psyco 是 Python 语言的一个扩展模块，可以即时对程序代码进行专业的算法优化，可以在一定程度上提高程序的执行速度，尤其是在程序中有大量循环操作时。最早被 Armin Rigo 开发，后来由 Christian Tismer 维护并继续完善。
Psyco 可以在 32位元 的 GNU/Linux、BSD、Mac OS X、Microsoft Windows 平台上运行。Psyco 使用 C语言 编写，只针对32位元平台进行了编码。目前开发工作已经停止，由 PyPy 所接替，同时 PyPy 也提供针对 64位元 系统的支持。Psyco 可以在 Python解释器 编译代码时自动优化，将其使用C实现，并针对循环操作进行一些特殊的优化。经过这些优化，程序的性能将会得到提升，在跨平台环境下尤为明显。
sudo apt-get install python-psyco
或者到官网上下载安装包，使用easy install安装即可。
使用Psyco模块
import psyco
psyco.full()
def fib(n):
& if n & 2:
&&&& return n
&&&& return fib(n - 1) + fib(n - 2)
$ time python fib.py&
改善你的代码
现在将我大部分 Python 代码加上下列脚本来利用 Psyco 提升运行速度：
&&& import psyco&
&&& psyco.full()&
except ImportError:&
&&& pass # psyco not installed so continue as usual
您可能感兴趣的文章:
大家感兴趣的内容
12345678910
最近更新的内容
常用在线小工具