Python
Status: This page is work in progress ...
Pagecode: T->2 A->SAml H->trsa[t,a,si,di]d[t,a,si,di] C->SA[cccej]
Last changed: Friday 2010-09-03 [15:25 UTC]
Abstract:
Python is a high-level programming language first released by Guido van Rossum in 1991. Python is designed around a philosophy which emphasizes readability and the importance of programmer effort over computer effort. Python core syntax and semantics are minimalist, while the standard library is large and comprehensive. Python is a multi-paradigm programming language (primarily functional, object oriented and imperative) which has a fully dynamic type system and uses automatic memory management -- it is thus similar to Perl, Ruby, Scheme, and Tcl. The language has an open, community-based development model managed by the non-profit Python Software Foundation. While various parts of the language have formal specifications and standards, the language as a whole is not formally specified. The de facto standard for the language is the so-called CPython implementation. Some of the largest projects that use Python are the Zope application server, the Mnet distributed file store, Youtube, and the original BitTorrent client. Large organizations that make use of Python include Google and NASA. Air Canada's reservation management system is written in Python. Python has also seen extensive use in the information security industry -- it is commonly used in exploit development. Also, Python has been successfully embedded in a number of software products as a scripting language. For many OSs (Operating Systems), Python is a standard component -- it ships with most Linux distributions, with FreeBSD, NetBSD, and OpenBSD, and with Mac OS X. From a developers point of view, Python has a large standard library, commonly cited as one of Python's greatest strengths, providing tools suited to many disparate tasks. This comes from a so-called "batteries included" philosophy for Python modules. The modules of the standard library can be augmented with custom modules written in either C or Python. Recently, Boost C++ Libraries includes a library, python, to enable interoperability between C++ and Python. Because of the wide variety of tools provided by the standard library combined with the ability to use a lower-level language such as C and C++, which is already capable of interfacing between other libraries, Python can be a powerful glue language between languages and tools. This page is going to cover various aspects of Python and programming with Python as seen from a developers point of view.
|
Table of Contents
|
 
Python, it just fits your brain ...
— unknown
Introduction
This sections features as a general get to know Python section in that
it touches on the most profound theoretical and practical subjects.
Main Usage Areas
So what is it that most people use Python for? Well, there are two
main usage areas:
There are many others too like for example scientific computing or
robotics but those are areas which have a considerably smaller
userbase than the two major areas mentioned above.
Philosophy
It is important for anyone involved with Python to at least understand
a few basic/core ideas about the language itself:
- Python is FLOSS (Free/Libre Open Source Software) i.e. it is
developed by many rather than one individual or company. There are
no license fees that need to be paid for using it, there is no
risk of a vendor lock-in i.e. developing in Python gives
investment security.
- Python is a high-level programming language i.e. it is a
programming language with strong abstraction from the details of
the computer. Any high-level programming languages generally hides
the details of CPU operations such as memory access models and
management of scope. In comparison to low-level programming
languages, Python has more natural human language elements and its
code is portable across many hardware platforms and operating
systems.
- From a programming paradigm point of view, Python is a
multi-paradigm programming language. A multi-paradigm programming
language is a programming language that supports more than one
programming paradigm e.g. object oriented, functional, aspect
oriented, etc. The basic idea of a multiparadigm programming
language is to provide a framework in which programmers can work
in a variety of styles, freely intermixing constructs from
different paradigms. The design goal of such languages is to allow
programmers to use the best tool for a job, admitting that no one
paradigm solves all problems in the easiest or most efficient way.
- Python is known for its well thought out and easily readable
syntax (e.g. indentation) which in turn boosts productivity and
also makes it a great language for beginners.
- Python is a dynamic language with an dynamic type system — most
dynamic languages are dynamically typed, but not all. Despite
being dynamically typed, Python is strongly typed, forbidding
operations that are not well-defined like, for example, adding a
number to a string rather than silently attempting to make sense
of them. Being a dynamic languages means name resolution is made
through dynamic binding also known as late binding i.e. name
resolution happens during runtime. In other words, Python binds
method and variable names during program execution.
- The automatic memory management of Python is based on its dynamic
type system and a combination of reference counting and
garbage collection.
- Python is fully Unicode aware. There is also excellent support for
internationalization and localization.
- From the very beginning, the overall design concept of Python has
been: Keep the core language to a minimum and provide a large
standard library and means to easily extend the core with own code
and/or third party code.
- Python's philosophy rejects the thinking of there is more than one
way to do it approach to language design in favor of there should
be one (and preferably only one) obvious way to do it.
- As we know, premature optimization is the root of all evil.
Therefore, when speed is a problem, Python programmers tend to
try to optimize bottlenecks by algorithm improvements or data
structure changes, using a JIT (just-in-time compilation)
compiler such as Psyco, rewriting the time-critical functions in
closer to the metal languages such as C, or by translating Python
code to C code using tools like Cython.
Glossary
It is strongly recommended to read the glossary!
QuickStart
This subsection is a summary of the semantics and syntax in Python
that can be read and followed along and which should not take longer
than an hour.
It is intended as a glance into Python for those who have not had
contact with Python yet, or, for those who want a quick refresh of the
cornerstones that makeup for most of Python's look and feel. Also,
without further notice, all examples shown here are assumed Python 3
rather than Python 2.
WRITEME
Tips and Tricks
WRITEME
Basics
This section introduces the reader to basic principles and knowledge
about the Python programming language and its ecosystem.
FAQs
This section gathers FAQs about Python.
What is the History behind Python?
1991 - Dutch programmer Guido van Rossum travels to Argentina for a
mysterious operation. He returns with a large cranial scar, invents
Python, is declared Dictator for Life by legions of followers, and
announces to the world that "There Is Only One Way to Do It." Poland
becomes nervous.
Those who are looking for a serious answer, go use some search engine
;-]
Zen of Python ... What is that?
sa@wks:~$ python3.1
Python 3.1.1+ (r311:74480, Oct 12 2009, 05:40:55)
[GCC 4.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one -- and preferably only one -- obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
>>>
sa@wks:~$
Everything made of Python of course should adhere to those
plus principles of their own.
Does Python have a Coding Standard?
Yes
Is there a Cheat Sheet or RefCard for Python?
Yes, the Internet has plenty of resources on the matter. Here is one
of them.
What Python Version do I run?
sa@wks:~$ python
>>> import sys
>>> sys.version[:6]
'3.1.1+'
>>> import platform
>>> platform.python_version()
'3.1.1+'
>>>
sa@wks:~$
What is the difference between a Statement and an Expression?
An expression is something e.g. 2 + 2 is (evaluates to) 4. This is
different to statements because a statement does something
e.g. print('Hello World') prints Hello World. Another well-known
statement is the import statement ...
What is a Module? What is a Package?
We can think of modules as extensions/add-on/plugins that can be
imported into Python to extend its capabilities beyond the core i.e.
the interpreter itself.
A module is usually just a file on the filesystem, containing source
code (statements, functions, classes, etc.) for a particular use case
e.g. draw.py might be a module to draw things like circles. A package
can be thought of as a directory containing one or more modules
(files) or other packages i.e. we could have a package called graphics
that would contain the modules draw.py and colorize.py
sa@sub:/tmp$ mkdir graphics; touch graphics/{draw,colorize}.py; ta graphics
graphics
|-- colorize.py
`-- draw.py
0 directories, 2 files
sa@sub:/tmp$ type ta
ta is aliased to `tree --charset ascii -a -I \.git*\|*\.\~*\|*\.pyc'
sa@sub:/tmp$
There are two ways how modules and/or packages are distributed:
- The standard library is a collection of modules and packages which
ships with almost any Python core installation. Using those
packages or modules is easy. All we have to is import them e.g.
import module or from module import function and so forth — we do
not need to explicitly install them onto our system.
- Third-party modules/packages are distributed through PyPI or other
means. Importing works the same, what is different however is that
we do need to get those packages and/or modules onto our system
first. This is straight forward in case we find what we need on
PyPI and if we use tools like PIP for example. In case we do not
use PyPI and/or PIP, more manual labour might be involved to get
packages/modules installed before we can import it.
When importing is has become good practice to import in the following
order, one import per line:
- standard library modules at the top of the file e.g.
sys, os,
getopt, re, etc.
- next third-party library modules (anything installed in Python’s
site-packages directory) e.g. mx.DateTime, ZODB, PIL.Image, etc.
- finally, our locally-developed modules
One good example for a Python package can be found in Django where
every project and the applications it contains is/are in fact Python
packages.
How do I get a list of available Modules?
Use help('modules').
Polymorphism? Encapsulation? Inheritance?
Roughly, these terms mean that we can use the same operations on
objects of different types, and they will work as if by magic
(polymorphism); we hide unimportant details of how objects work from
the outside world (encapsulation), and we can create specialized
classes of objects from general ones (inheritance).
Coercion?
Is the implicit conversion of an instance of one type to another
during an operation which involves two arguments of the same type. For
example, int(3.15) converts the floating point number to the integer
3, but in 3 + 4.5, each argument is of a different type (one int, one
float), and both must be converted to the same type before they can be
added or it will raise a TypeError.
Without coercion, all arguments of even compatible types would have to
be normalized to the same value by the programmer e.g. float(3) + 4.5
rather than just 3 + 4.5.
What is a Class?
A class is a datatype, same as a list, tuple, dictionary etc. are
datatypes. Objects derived from one particular class are said to be
instances of that class.
Subclass? Superclass?
When the objects belonging to class A form a subset of the objects
belonging to class B, class A is called a superclass of class B. Class
B is then called a subclass of class A.
For example, cat can be a superclass to a subclass tiger. Lion can
also be a subclass of the superclass cat — both, tigers and lions are
cats, sorta, bulky though. Anyhow, let us not get off-topic: shark is
not a subclass of cat since obviously a shark ain't no cat ... shark
can have fish as its superclass.
Yeah, yeah, yeah ... you smarty pants, the answer is yes! You can have
your tigershark as well ;-]
Metaclass?
Please go here.
Abstract Class? Abstract Base Class?
Please go here.
Context Manager?
An object which controls the environment seen in a with statement by
defining __enter__() and __exit__() methods.
Static Method
WRITEME
- http://docs.python.org/dev/library/functions.html#staticmethod
- statische Methoden, also solche, für deren Ausführung keine
Objekte benötigt werden. ... or if we wanted it a bit more
complicated/shallow/confusing, here is how to describe it:
- A static method is a method that does not receive an implicit
first argument.
Class Method
WRITEME
Descriptor?
It is actually a so-called protocol i.e. any object which defines the
methods __get__(), __set__(), or __delete__() is said to implement the
descriptor protocol.
When a class attribute is a descriptor, its special binding behaviour
is triggered upon attribute lookup. Normally, using a.b to get, set or
delete an attribute looks up the object named b in the class
dictionary for a, but if b is a descriptor, the respective descriptor
method gets called.
Understanding descriptors is a key to a deep understanding of Python
because they are the basis for many features including functions,
methods, properties, class methods, static methods, and reference to
super classes.
How do I tell the Datatype of some Object?
sa@wks:~$ type bp; bp
bp is aliased to `bpython'
>>> foo = 3
>>> bar = range(3)
>>> type(foo)
<type 'int'>
>>> type(bar)
<type 'list'>
>>>
sa@wks:~$
What is it about this PYTHONPATH variable?
Well, actually we are not talking about PYTHONPATH alone here but
instead we take a look at the bigger picture i.e. how does Python
find/know about code that exists on the filesystem so we can make use
of it? To answer this question, let us take a look at Python's
module search behavior and how we can influence it.
What is the Difference between Distutils, Setuptools and Distribute?
Although those tools have nothing to do with writing source code
itself, they are needed to work with the whole Python ecosystem. Go
here and here for more information.
What is a Virtual Environment?
A standard system has what is called a main Python installation also
known as global Python context/space i.e. a Python interpreter living
at /usr/bin/python and a bunch of modules/packages installed into the
module search paths.
Another way to have modules/packages installed would be to use
virtualenv. It can be used to create isolated Python contexts/spaces
i.e. those virtual environments can have their own Python interpreter
as well as their own set of modules/packages installed and therefore
have no connection with the global Python context/space whatsoever.
Note that we can not just clone the global Python context/space or
create an entirely separated Python context/space to work with, but we
can also link any directories into any virtual environment. This means
ultimate flexibility without risking to damage the existing main
Python installation also known as global Python context/space.
Why has Debian ../dist-packages Directories?
Before we actually answer that, let us have a look at the big picture
of having public and private installations of Python modules and
packages. Let us also have a glance at the difference about the main
Python installations (also known as global Python context/space) and
virtual environments:
By default Python modules/packages are searched in the current working
directory first, next in the directories listed in the PYTHONPATH
environment variable and finally all directories listed in the
sys.path Python variable are searched. That is just half the truth
actually ...
The full truth is that Python initializes sys.path from PYTHONPATH and
that it includes the current working directory — however, once
Python's machinery gets going, PYTHONPATH and the current working
directory are ignored.
That said, there are generally three ways to install Python
modules/packages — there are public ones and private ones with
regards to the systems main Python installation (also known as global
Python context/space) and then there are virtual environments which
are either clones of the global Python context/space or which are
entirely separated Python contexts/spaces on their own:
- Public modules/packages are installed in a public directory as
listed in the afore mentioned
PYTHONPATH environment variable or
in directories found in sys.path.
- Directories with private Python modules/packages must be absent
from both,
PYTHONPATH and sys.path, so to not being picked up. In
case we want/need paths providing private Python modules/packages
which cannot be seen from the global Python context/space, they
should be installed in a private directory such as
/usr/share/<package-name> or /usr/lib/<package-name> for example
(paths not listed in sys.path and/or PYTHONPATH) where they are
generally only accessible to a specific program or suite of
programs included in the same package.
- Another way to have modules/packages installed would be to use a
virtual environment.
Right now we are only looking at the global Python context/space and
leave aside virtual environments. We are also just looking at the
public modules/packages subset and not how to handle private
modules/packages within the global Python context/space.
Finally, why Debian has ../dist-packages directories:
The installation location for Python code packaged by Debian is the
system Python modules directory, /usr/lib/pythonX.Y/dist-packages for
Python 2.6 and later, and /usr/lib/pythonX.Y/site-packages for Python
2.5 and earlier. In other words, whenever we use APT (Advanced
Packaging Tool) to install Python software, things land in
/usr/lib/pythonX.Y/dist-packages.
Tools used for packaging Python source code for Debian like
python-central and python-support take care of using the correct path
automatically. As an exception to the above, modules managed by
python-support are installed in another directory which is added to
sys.path using the .pth files mechanism.
In case we are on Python 2.6 or later and do not use APT but some
other means (e.g. easy_install, pip, etc.) to install public Python
code, /usr/local/lib/pythonX.Y/dist-packages is used. In case of
Python 2.5 or earlier the path would change to
/usr/local/lib/pythonX.Y/site-packages. This however is problematic
since, for Python 2.5 and earlier, this directory is also visible to
the default installation of Python and could thus lead to clashes if
the same Python module/package was installed via APT as well as
manually using pip, easy_install etc. In order to avoid those clashes
Debian has introduced its ../dist-packages directories which helps
avoid those clashes.
When binary packages ship identical source code for multiple Python
versions, for instance /usr/lib/python3.1/dist-packages/foo.py,
/usr/lib/python2.6/dist-packages/foo.py and
/usr/lib/python2.5/site-packages/foo.py, these should point to a
common file. A common location to share, across Python versions,
arch-independent files which would otherwise go to the directory of
system public modules is /usr/share/pyshared.
Summary, assuming Python >= 2.6
- If we install some Python software which is packaged by Debian,
/usr/lib/pythonX.Y/dist-packages is where stuff goes. This
directory may also contain some .pth files which contain additional
paths which will be appended to sys.path.
- In case we install Python software manually (using
easy_install,
pip, etc.) things go to /usr/local/lib/pythonX.Y/dist-packages.
- Identical Python binaries for two or more versions of Python go to
/usr/share/pyshared.
Note the difference between /usr/lib/.. and /usr/local/lib/.. above
(some people, including myself, tend to overlook it). More information
can be found in /usr/lib/pythonX.Y/site.py —
read the code.
How does Python find Code on the Filesystem?
If we have code (modules or packages) somewhere on the filesystem that
we want Python to know about, we need to import that code using the
import statement. For import to work, Python needs to know where to
find the code on the filesystem. What a no brainer eh? ;-]
So how do we tell Python about the places where it should look for
code? The variable sys.path holds a bunch of paths also known as
module search paths. Python searches those directories for code so we
can start using it by importing it. In order for Python to find our
code on the filesystem, we have two choices:
- We put our code into one directory that is already part of
sys.path or
- We add another directory to
sys.path
Before we start, let us take a look at sys.path as it looks like in
its default setup:
sa@wks:~$ python3.1
Python 3.1.1+ (r311:74480, Oct 12 2009, 05:40:55)
[GCC 4.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> dir()
['__builtins__', '__doc__', '__name__', '__package__']
>>> import pprint, sys
>>> pprint.pprint(sys.path)
['',
'/usr/lib/python3.1',
'/usr/lib/python3.1/plat-linux2',
'/usr/lib/python3.1/lib-dynload',
'/usr/lib/python3.1/dist-packages',
'/usr/local/lib/python3.1/dist-packages']
If we decided to add our own or some third party code without adding a
new directory to sys.path, then /usr/local/lib/python3.1/dist-packages
would be the right place to put it. However, this might not work out
for the following reasons:
- we do not want to clutter the default directories with our
own/third party code
- we might not have permissions to do so e.g. no root permissions
- we simply want to keep our code somewhere else on the filesystem
If we want/have to add another directory to sys.path, then there are
two possibilities:
- we do it manually every time we start the Python interpreter or
- we automate the process so that maybe even Python code itself
could take care of it
Manually adding to sys.path
This one is straight forward as we only need to append to sys.path:
>>> import os
>>> sys.path.append('/tmp'); sys.path.append(os.path.expanduser('~/0/django'))
>>> pprint.pprint(sys.path)
['',
'/usr/lib/python3.1',
'/usr/lib/python3.1/plat-linux2',
'/usr/lib/python3.1/lib-dynload',
'/usr/lib/python3.1/dist-packages',
'/usr/local/lib/python3.1/dist-packages',
'/tmp',
'/home/sa/0/django']
>>>
Adding directories manually is quick and certainly nice while doing
development/testing but it is not what we want for some permanent
setup like for example a long-term development project or a production
site. For those, we want to add directories to sys.path automatically
which is shown below.
Automatically adding to sys.path
When a module named duck is imported, the interpreter searches for a
file named duck.py in the current working directory, and then in the
list of directories specified by the environment variable PYTHONPATH
— this environment variable has the same syntax as the shell variable
PATH, that is, a list of directory names separated by colons.
sa@wks:~$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/games:/home/sa/0/bash
sa@wks:~$
When PYTHONPATH is not set, or when duck.py is not found in the
current working directory, the search continues in an
installation-dependent default path.
Most Linux distributions include Python as a standard part of the
system, so prefix and exec-prefix are usually both /usr on Linux. If
we build Python ourselves on Linux (or any Unix-like system), the
default prefix and exec-prefix are /usr/local.
sa@wks:~$ python3.1
Python 3.1.1+ (r311:74480, Oct 12 2009, 05:40:55)
[GCC 4.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; sys.prefix
'/usr'
>>>
sa@wks:~$
So now we know how finding code on the filesystem works. This however
does not help us much since we do not want to use any of the default
paths/directories listed in sys.path. We also do not want to manually
add directories to sys.path every time the Python interpreter gets
restarted.
Although the standard method so far is to add directories to
PYTHONPATH, this is suboptimal for two reasons:
- it is only valid for one particular system user (e.g. for
production sites) respectively a normal Unix/Linux user account if
we are writing code
- using
PYTHONPATH is not really portable since everywhere we want to
run our code, we need to adapt PYTHONPATH
So what do we do? Piece of cake, we use .pth files. Those files are
simple text files containing paths to be added to sys.path, one path
per line.
All we need to do is to put our .pth files into one of the directories
the site module knows about — without further explanation, one of
those directories is /usr/local/lib/python<version>/dist-packages e.g.
/usr/local/lib/python3.1/dist-packages if we are using Python version
3.1. The way it works is really easy:
1 sa@wks:/tmp$ mkdir test; cd test; echo -e "foo\nbar" > our_path_file.pth
2 sa@wks:/tmp/test$ mkdir foo bar
3 sa@wks:/tmp/test$ echo 'print("inside foo.py")' > foo/foo.py
4 sa@wks:/tmp/test$ echo 'print("inside bar.py")' > bar/bar.py
5 sa@wks:/tmp/test$ type ta
6 ta is aliased to `tree -a -I \.git*\|*\.\~*\|*\.pyc'
7 sa@wks:/tmp/test$ ta ../test/
8 ../test/
9 |-- bar
10 | `-- bar.py
11 |-- foo
12 | `-- foo.py
13 `-- our_path_file.pth
14
15 2 directories, 3 files
16 sa@wks:/tmp/test$ cat our_path_file.pth
17 foo
18 bar
19 sa@wks:/tmp/test$ python3.1
20 Python 3.1.1+ (r311:74480, Oct 12 2009, 05:40:55)
21 [GCC 4.3.4] on linux2
22 Type "help", "copyright", "credits" or "license" for more information.
23 >>> import pprint, sys, site
24 >>> pprint.pprint(sys.path)
25 ['',
26 '/usr/lib/python3.1',
27 '/usr/lib/python3.1/plat-linux2',
28 '/usr/lib/python3.1/lib-dynload',
29 '/usr/lib/python3.1/dist-packages',
30 '/usr/local/lib/python3.1/dist-packages']
31 >>> site.addsitedir('/tmp/test')
32 >>> pprint.pprint(sys.path)
33 ['',
34 '/usr/lib/python3.1',
35 '/usr/lib/python3.1/plat-linux2',
36 '/usr/lib/python3.1/lib-dynload',
37 '/usr/lib/python3.1/dist-packages',
38 '/usr/local/lib/python3.1/dist-packages',
39 '/tmp/test',
40 '/tmp/test/foo',
41 '/tmp/test/bar']
42 >>> import foo
43 inside foo.py
44 >>> import foo
45 >>> import bar
46 inside bar.py
Python now finds our modules foo.py and bar.py thanks to
our_path_file.pth. Note that what we did in lines 3 and 4 are in place
only to show that importing works as we see with lines 42 to 46 —
modules are not meant to do things (e.g. print text) when they are
imported. Note also, that importing a module more than once does not
execute the code inside again (lines 42 to 44).
site.addsitedir from line 31 is quite a nifty thing — it adds a
directory to sys.path and processes its .pth file(s). That it worked
can be seen from lines 39 to 41.
Certainly, no one really cares to use /tmp for serious
development/deployment work if /tmp is set up the usual way
(everything in /tmp will vanish on reboot).
What I often do is to add to sys.path so that it is only added for one
particular Python version and only for my user account sa.
47 >>> site.USER_SITE
48 '/home/sa/.local/lib/python3.1/site-packages'
49 >>> import os
50 >>> dir()
51 ['__builtins__', '__doc__', '__name__', '__package__', '__warningregistry__', 'bar', 'foo', 'os', 'pprint', 'site', 'sys']
52 >>> mypth = os.path.join(site.USER_SITE, 'mypath.pth')
53 >>> print (mypth)
54 /home/sa/.local/lib/python3.1/site-packages/mypath.pth
55 >>> module_paths_to_add_to_sys_path = ["/home/sa/0/django", "/home/sa/0/bash"]
56 >>> if not os.path.isdir(site.USER_SITE):
57 ... os.makedirs(site.USER_SITE)
58 ...
59 >>> with open(mypth, "a") as f:
60 ... f.write("\n".join(module_paths_to_add_to_sys_path))
61 ... f.write("\n")
62 ...
63 33
64 1
65 >>> pprint.pprint(sys.path)
66 ['',
67 '/usr/lib/python3.1',
68 '/usr/lib/python3.1/plat-linux2',
69 '/usr/lib/python3.1/lib-dynload',
70 '/usr/lib/python3.1/dist-packages',
71 '/usr/local/lib/python3.1/dist-packages',
72 '/tmp/test',
73 '/tmp/test/foo',
74 '/tmp/test/bar']
75 >>> site.addsitedir(site.USER_SITE)
76 >>> pprint.pprint(sys.path)
77 ['',
78 '/usr/lib/python3.1',
79 '/usr/lib/python3.1/plat-linux2',
80 '/usr/lib/python3.1/lib-dynload',
81 '/usr/lib/python3.1/dist-packages',
82 '/usr/local/lib/python3.1/dist-packages',
83 '/tmp/test',
84 '/tmp/test/foo',
85 '/tmp/test/bar',
86 '/home/sa/.local/lib/python3.1/site-packages',
87 '/home/sa/0/django',
88 '/home/sa/0/bash']
89 >>>
90 sa@wks:/tmp/test$
What is a Decorator?
Please go here.
Why would I want to pass a Function to another Function?
There are two reasons why we want to do that:
- to do asynchronous computing using callbacks or
- to wrap one function with another one i.e. modify/influence the
result of the passed in function using its wrapper function and
return this result to the caller of the passed in function. This
is known as the decorator design pattern. Decorators are an
alternative to subclassing. They add/change behavior at runtime
whereas subclassing generally adds/changes behavior at compile
time.
Decorator vs Adapter
The decorator design pattern differs from the adapter design pattern
in that decorators wrap functions or methods whereas adapters wrap
classes or instances thereof.
An adapter wraps a class foo or an object/instance thereof so that it
works/behaves in a context intended for a class or an object/instance
bar.
What is a Callback?
A callback is a function provided by the consumer of an API
(Application Programming Interface) that the API can then turn around
and invoke (calling us back).
For example, if we setup a Dr.'s appointment, we can give them our
phone number, so they can call us the day before to confirm the
appointment. A callback is like that, except instead of just being a
phone number, it can be arbitrary instructions like send us an email
at this address, and also call our secretary and have her put it in
our calendar.
Callbacks are often used in situations where an action is
asynchronous. If we need to call a function, and immediately continue
working, we can not sit there wait for its return value to let us know
what happened, so we provide a callback. When the function has
finished its asynchronous work it will invoke our callback code with
some predetermined arguments (usually some we supply, and some about
the status and result of the asynchronous action we requested).
If the Dr. is out of the office, or they are still working on the
schedule, rather than having us wait on hold until he gets back, which
could be several hours, we hang up, and once the appointment has been
scheduled, they call us.
Python will invoke our callback code with any arguments we supply and
the result of its asynchronous computation, once this asynchronous
computation has finished executing.
Let us look at some example:
sa@wks:~$ python3.1
Python 3.1.1+ (r311:74480, Oct 12 2009, 05:40:55)
[GCC 4.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def callback(nums):
... """ The callback function """
... return sum(nums) * 2
...
>>> def another_callback(nums):
... """ Yet another callback function """
... return sum(nums) * 3
...
>>> def strange_sum(nums, cb):
... """
... Asynchronous computation: Returns the sum, if less than 10
... else returns the result of calling the callback function cb(),
... which must accepts one list argument
... """
... if sum(nums) > 10:
... print("no callback function used")
... else:
... return cb(nums)
...
>>> print (strange_sum([1, 10], callback))
no callback function used
None
>>> print (strange_sum([3, 2], another_callback))
15
>>> print (strange_sum([6,4,3], another_callback))
no callback function used
None
>>>
sa@wks:~$
So basically, a callback is a function that we pass as an argument (to
another function that is; functions itself are only values in Python
i.e. calling foo() is different to calling foo since the later would
only return the function itself, as a value) that may be called when
a certain condition happens.
What is a Handler?
A handler is a asynchronous callback subroutine that can be told to do
some work for us and call back when it is done (see
Dr.'s appointment example).
What is setup.py?
setup.py is Python's answer to a multi-platform installer and
make file. In other words: setup.py in combination with either
distutils/setuptools/distribute can be thought off the equivalent to
make && make install — it translates to python setup.py build &&
python setup.py install.
Some packages are pure Python and are only byte-compiled, other
packages may, for example, contain native C code which will require a
native compiler like gcc or cl and some Python interfacing module like
swig or pyrex.
Generally setup.py can be thought of being at the core of
packaging/distributing/installing Python software.
What is setup.cfg?
It is a configuration file local to some package which is used to
record configuration data for a particular package. setup.cfg is the
last one of three layers where Python looks for configuration
information.
At first it looks at the system-wide configuration file e.g.
/usr/lib/python<version>/distutils/distutils.cfg, next it looks at our
personal settings e.g. ~/.pydistutils.cfg and lastly it looks local to
a package i.e. setup.cfg.
Any of those levels overrides the former one e.g. personal overrides
system-wide, package-local overrides personal and of course,
package-local also overrides system-wide.
What is __init__.py for?
Files named __init__.py are used to mark directories on disk as
so-called Python packages — basically a sort of meta-module
containing other modules.
Every time we use import, Python goes off and looks for the stuff we
actually want to import (e.g. a package, a module, a class, a
function, etc.) in the module search path know to it at the time. More
information on the matter can be found here.
What is site.py for?
site.py is run when our interpreter starts. It loads a few things in
the builtins, and adds some paths to the module search paths like
per-user paths and the like.
What is EAFP?
EAFP (Easier to ask for Forgiveness than Permission) is a
programming principle how to approach problems/things when
programming. This clean and fast style is characterized by the
presence of many try and except statements.
It is nothing Python specific but can actually be found with many
programming languages. With Python however, because of it is nature,
adhering to this principle works quite well. In Python EAFP is
generally preferred over LBYL (Look before you Leap), which is the
contrary principle and, for example, the predominant coding style with
C.
Monkey Patching?
A monkey patch is a way to extend and/or modify the runtime code of
dynamic languages such as Smalltalk, JavaScript, Objective-C, Ruby,
Perl, Python, Groovy, etc. without altering its on-disk source code.
In Python, the term monkey patch only refers to dynamic modifications
of a class at runtime based on the intent to patch existing methods in
an external class as a workaround to a bug and/or feature which does
not act as we desire. Examples of using monkey patching are
- Replace methods/attributes/functions at runtime e.g. to stub out a
function during testing.
- Modify/extend behaviour of a third-party product without
maintaining a private copy of the source code.
- Apply a patch at runtime to the objects in memory, instead of the
on-disk source code.
- Distribute security and/or behavioural fixes that live alongside
the original on-disk source code (an example of this would be
distributing the fix as a plugin for the Ruby on Rails platform).
In general however it is fair to say that one should refrain from
monkey patching since it mostly introduces more problems than it
solves.
- Features/fixes made through monkey patching almost never get
documented, it is the oh-so-well-known quick hack/fix ...
- They can lead to upgrade problems when the patch makes assumptions
about the patched object that are no longer true e.g. if the
product we have changes with a new release it may very well break
our patch. For this reason monkey patches are often made
conditional, and only applied if appropriate.
- If two modules attempt to monkey-patch the same method, one of them
(whichever one runs last) wins and the other patch has no effect,
unless monkeypatches are written with pattern like
alias_method_chain.
- They create a discrepancy between the original on-disk source code
and the observed behaviour that can be very confusing to anyone
unaware of the patche's existence.
Even if monkey patching is not used, many people see a problem with
the availability of the feature, since the ability to use monkey
patching in a programming language is incompatible with enforcing
strong encapsulation, as required by the object-capability model,
between objects. Bottom line is, one should not use monkey patching
for the afore mentioned reasons and many more ...
What is a Method Stub?
Please go here for further information.
What is a Finder / Loader / Importer?
To use functionality that is not built-in with Python core
i.e. the interpreter itself, we need to get it from somewhere. This is
called importing — basically everything that involves the import
statement.
This process of importing is, as many other things, specified by a
so-called protocol. The Importer protocol involves two objects: a
finder and a loader.
- The finder object returns a loader object if the module was found,
or
None otherwise. A finder object defines a find_module() object.
- The loader object returns the loaded module or raises an exception,
preferably
ImportError if an existing exception is not being
propagated. A loader object defines a load_module() method.
In many cases the finder and loader are one and the same object i.e.
find_module() would just return self. The combined functionality of
the finder and loader object is called importer. See PEP 302 for more
information.
I see this __future__ thing a lot? What is it?
Basically what the name promises, it is a module that brings future
features which are not enabled with the current version of Python core
(the interpreter) by default. Simply using import __future__ will
enable such features. Go here for more information.
Garbage Collection?
Not the thing your neighbours are talking about when referring to your
car but rather the automatic memory management of Python which is
based on its dynamic type system and a combination of
reference counting and garbage collection.
In a nutshell: once the last reference to an object is removed, the
object is deallocated ... that is, left floating around in memory
until deleted/overwritten. The memory it occupied is said to be freed
and possibly immediately reused by another (new) object. Python's
memory management is smart enough to detect and break cyclic
references between objects that might otherwise occupy memory
indefinitely, which in its worst case might cause memory shortage.
Reference Count
The number of references to an object. When the reference count of an
object drops to zero, it is deallocated. Reference counting is
generally not visible to Python code, but it is a key element of the
CPython implementation.
The sys module defines a getrefcount() function that can be called
from Python code in order to return the reference count for a
particular object.
Mutable vs Immutable?
Immutable objects cannot change their value and keep their id() as
mutable ones can. In other words, if we want to alter an immutable
object, we need to create a new/different one — the new one will have
a different id(); CPython implementation detail: id() is the memory
address of the object.
Immutable objects play an important role in places where a constant
hash value is needed, for example as a key in a dictionary. To name a
few immutable objects: numbers, strings, tuples, frozensets, byte,
etc. What all those have in common is that they are Python built-in
datatypes.
It is fair to say that immutable datatypes can be thought of as the
basic building blocks used to assemble more complex datatypes e.g. if
we use a class to create ourselves a particular datatype used for our
individual web application, this class, or rather instances thereof,
would probably be mutable but some of its attributes might not be.
In a nutshell: most Python built-in datatypes are immutable (cannot
change their value without changing their id()). Most custom datatypes
are mutable but might have attributes made of immutable datatypes.
Iterable
A container object capable of returning its members one at a time
rather than all at once.
Examples of iterable objects include sequence types such as list, str,
or tuple as well as non-sequence types like dict, file and objects of
any classes we define with an __iter__() or __getitem__() method.
Iterables can be used in a for loop and in many other places where a
sequence is needed (zip(), map(), ...). When an iterable object is
passed as an argument to the built-in function iter(), it returns an
iterator for the object. This iterator is good for one pass over the
set of values.
When using iterable objects, it is usually not necessary to call
iter() or deal with iterators. The for statement does that
automatically for us, creating a temporary unnamed variable to hold
the iterator for the duration of the for loop.
Iterator
An object representing a stream of data. Repeated calls to the
iterator's __next__() (or passing it to the built-in function next())
method return successive items in the stream.
When there is no more data available from the stream, a StopIteration
exception is raised. At this point, the iterator object is exhausted
and any further calls to its next() method just raise StopIteration
again.
Iterators are required to have an __iter__() method that returns the
iterator object itself so every iterator is also iterable and may be
used in most places where other iterables are accepted.
One notable exception is code which attempts multiple iteration
passes. A container object (such as a list) produces a fresh new
iterator each time we pass it to the iter() function or use it in a
for loop. Attempting this with an iterator will just return the same
exhausted iterator object used in the previous iteration pass, making
it appear like an empty container.
Generator
A generator is a function which returns an iterator. It looks like a
normal function except that it contains yield statements for producing
a series of values usable in a for loop or that can be retrieved one
at a time with the next() function.
Each yield temporarily suspends processing, remembering the location
execution state (including local variables and pending
try-statements). When the generator resumes, it picks-up where it
left-off (in contrast to functions which start fresh on every
invocation.
Generator Expression
An expression that returns an iterator. It looks like a normal
expression followed by a for expression defining a loop variable,
range, and an optional if expression. The combined expression
generates values for an enclosing function:
>>> sum(i*i for i in range(10)) # sum of squares 0, 1, 4, ... 81
285
Global Interpreter Lock
A GIL (Global Interpreter Lock) is a mutual exclusion lock held by an
interpreter thread. Its use is to avoid sharing code that is not
thread-safe with other threads.
There is always one GIL for one interpreter process. Problem is, while
this gives better performance on single-core machines, it fails to
scale for multiprocessor machines.
Is there a better Python Shell/Interpreter?
Yes, yes, there is! There is iPython and then there is bpython which I
have come to love. It is packaged with Debian
sa@wks:~$ dpl bpy* | grep ii
ii bpython 0.9.5.2-1 fancy curses interface to the Python interactive
sa@wks:~$
There is also http://bpaste.net, a pastebin site. This for itself is
no big deal. The fact that bpython can ship off its contents (what we
typed) at the press of a button, right into bpaste.net, however is —
I often use it to sketch things in a live interpreter session and then
quickly show it to folks while we talk on IRC, maybe during debugging
some code and stuff like that.
There are a lot more goodies at our disposal like for example
Django support. Most of it can be configured in ~/.bpython/config:
sa@wks:~$ cat .bpython/config | grep -v \# | grep .
[general]
auto_display_list = True
syntax = True
arg_spec = True
hist_file = ~/.pythonhist
hist_len = 5000
tab_length = 4
color_scheme = suno
[keyboard]
pastebin = F8
save = C-s
And then there is of course a custom theme we might use
sa@wks:~$ cat .bpython/suno.theme | grep -v \# | grep .
[syntax]
keyword = y
name = W
comment = w
string = M
error = r
number = G
operator = Y
punctuation = y
token = C
[interface]
background = d
output = w
main = w
prompt = w
prompt_more = w
sa@wks:~$
The coolest thing about bpython is probably autocompletion, inline
syntax highlighting, the fact that is shows us the expected parameter
list as we type and last but not least, the possibility to rewind what
we typed not just graphically but also internally i.e. the results of
each such expression we typed. Below is a screenshot showing a few of
the just mentioned things:
Using bpython with Django
Usually, being at the root of a Django project, which we created using
django-admin startproject, we could run python manage.py shell which
actually tries to run iPython if available:
sa@wks:~/0/django/myproject$ python manage.py help shell | grep Runs
Runs a Python interactive interpreter. Tries to use IPython, if it's available.
sa@wks:~/0/django/myproject$
If however we want python manage.py shell to use bpython instead, here
is what we can do: We start with using PYTHONSTARTUP i.e. we put
export PYTHONSTARTUP=$HOME/.pythonrc into our ~/.bashrc
file. Next, we put some magic inside ~/.pythonrc to make python
manage.py shell use bpython instead of iPython.
The rationale behind PYTHONSTARTUP is simple: When we use Python
interactively, it is frequently handy to have some standard commands
executed every time the interpreter is started. We can do this by
setting the environment variable PYTHONSTARTUP to the name of a file
containing our start-up commands (~/.pythonrc for example). This is no
Python speciallity as it is the same as .profile and friends are for
any Unix shell out there ...
Note that whatever file PYTHONSTARTUP points to, it is only read in
interactive sessions, not when Python reads commands from a script,
and not when /dev/tty is given as the explicit source of commands
(which otherwise behaves like an interactive session). It is executed
in the same namespace where interactive commands are executed, so that
objects that it defines or imports can be used without qualification
in the interactive session.
Furthermore, we can also change the prompts sys.ps1 (>>>) and sys.ps2
(...) in this file — those are the primary respectively secondary
prompts of the interpreter. They are only defined if the interpreter is
in interactive mode.
Now that we have PYTHONSTARTUP in place and use it to point to
~/.pythonrc, we can use it to do all kinds of setup work like for
example setup the Django environment i.e. what we do here manually is
the same what python manage.py shell otherwises does for us
automatically. Below is the magic code to get this behavior with
bpython:
try:
from django.core.management import setup_environ
import settings
setup_environ(settings)
print 'imported django settings'
except:
pass
This way, bpython (or even just the ordinary python interpreter),
imports the django environment for us. Let us have a look — we are at
the root of the project myproject:
1 sa@wks:~/0/django/myproject$ ta -L 2
2 .
3 |-- __init__.py
4 |-- locale
5 | `-- de
6 |-- manage.py
7 |-- polls
8 | |-- __init__.py
9 | |-- admin.py
10 | |-- models.py
11 | |-- tests.py
12 | `-- views.py
13 |-- settings.py
14 |-- sqlite3.db
15 |-- templates
16 | |-- admin
17 | `-- polls
18 `-- urls.py
19
20 6 directories, 10 files
21 sa@wks:~/0/django/myproject$ cat /home/sa/.pythonrc
22 sa@wks:~/0/django/myproject$ bpython
23 >>> dir()
24 ['__builtins__', '__doc__', '__name__', 'help']
25 >>>
26
27
28 [ here we use some editor to put our code into ~/.pythonrc ...]
29
30
31
32 sa@wks:~/0/django/myproject$ cat /home/sa/.pythonrc
33 try:
34 from django.core.management import setup_environ
35 import settings
36 setup_environ(settings)
37 print 'imported django settings'
38 except:
39 pass
40 sa@wks:~/0/django/myproject$ source /home/sa/.bashrc
41 sa@wks:~/0/django/myproject$ bpython
42 imported django settings
43 >>> dir()
44 ['__builtins__', '__doc__', '__name__', 'help', 'settings', 'setup_environ']
45 >>>
Note the different output from lines 24 and 44 — this comes from
lines 33 to 39 which were not active in line 22. Line 40 is just to
source ~/.bashrc which in turn makes the code in ~/.pythonrc
available. ta from line 1 is an alias to tree in my ~/.bashrc.
So, what we have so far is great since we can, for example, access our
settings from an interpreter session without the need to load anything
explicitly i.e. we would not have to issue line 46 which I just did so
it can be seen where line 52 gets its information from (settings.py).
46 sa@wks:~/0/django/myproject$ grep -A5 TEMPLATE_DIRS settings.py | grep -v \#
47 TEMPLATE_DIRS = (
48 '/home/sa/0/django/myproject/templates'
49 )
50 sa@wks:~/0/django/myproject$ bpython
51 imported django settings
52 >>> settings.TEMPLATE_DIRS
53 '/home/sa/0/django/myproject/templates'
54 >>>
To put the cherry on top of this convenience, lets add what is
available trough using django-extensions namely the shell_plus command
extension:
55 sa@wks:~/0/django/myproject$ python manage.py help shell_plus | grep Like
56 Like the 'shell' command but autoloads the models of all installed Django apps.
We add to ~/.pythonrc, source ~/.bashrc again and start bpython:
57 sa@wks:~/0/django/myproject$ cat ~/.pythonrc
58 try:
59 from django.core.management import setup_environ
60 import settings
61 setup_environ(settings)
62 print 'imported django settings'
63 try:
64 exec_strs = ["from %s.models import *"%apps for apps in settings.INSTALLED_APPS ]
65 for x in exec_strs:
66 try:
67 exec(x)
68 except:
69 print 'Not imported for %s' %x
70 print 'imported django models'
71 except:
72 pass
73 except:
74 pass
75 sa@wks:~/0/django/myproject$ source /home/sa/.bashrc; bpython
76 imported django settings
77 imported django models
78 >>> import pprint
79 >>> pprint.pprint(dir())
80 ['ADDITION',
81 'AnonymousUser',
82 'CHANGE',
83 'Choice',
84 'ContentType',
85 'ContentTypeManager',
86 'DELETION',
87 'EmptyManager',
88 'Group',
89 'ImproperlyConfigured',
90 'LogEntry',
91 'LogEntryManager',
92 'Message',
93 'Permission',
94 'Poll',
95 'RequestSite',
96 'SITE_CACHE',
97 'Session',
98 'SessionManager',
99 'Site',
100 'SiteManager',
101 'SiteProfileNotAvailable',
102 'UNUSABLE_PASSWORD',
103 'User',
104 'UserManager',
105 '__builtins__',
106 '__doc__',
107 '__name__',
108 'apps',
109 'auth',
110 'base64',
111 'check_password',
112 'datetime',
113 'exec_strs',
114 'get_hexdigest',
115 'help',
116 'mark_safe',
117 'md5_constructor',
118 'models',
119 'pickle',
120 'pprint',
121 'quote',
122 'settings',
123 'setup_environ',
124 'sha_constructor',
125 'smart_str',
126 'smart_unicode',
127 'urllib',
128 'x']
129 >>> datetime.datetime.utcnow()
130 datetime.datetime(2009, 12, 12, 22, 10, 8, 361387)
131 >>>
132 sa@wks:~/0/django/myproject$ cat polls/models.py
133 from django.db import models
134 import datetime
135
136
137 class Poll(models.Model):
138 question = models.CharField(max_length=200)
139 pub_date = models.DateTimeField('date published')
140 def __unicode__(self):
141 return self.question
142 def was_published_today(self):
143 return self.pub_date.date() == datetime.date.today()
144 was_published_today.short_description = 'Published today?'
145
146
147 class Choice(models.Model):
148 poll = models.ForeignKey(Poll)
149 choice = models.CharField(max_length=200)
150 votes = models.IntegerField()
151 def __unicode__(self):
152 return self.choice
153 sa@wks:~/0/django/myproject$
This is great! Note that the models Poll and Choice from my polls
application (line 10 respectively lines 133 to 152) show up
automatically now (lines 83 and 94). The only thing loaded explicitly
here was pprint which I only did so I can have a nicer output for this
article i.e. pprint is not used anywhere inside myproject.
Last but not least, we can use bpython's ability to save the current
session to a file — this file is then used to load our former session
into bpython again, effectivelly allowing us to resume our work where
we left off before.
We use the C+s keys and when prompted for the filename to save our
session, we use startup.py. Next, we add from startup import * to
~/.pythonrc to make it all work:
sa@wks:~/0/django/mysite$ bpython
imported django settings
imported django models
>>> print 'funky donkey at work'
funky donkey at work
[ here I used C+s to save the current session to startup.py ... ]
>>>
sa@wks:~/0/django/mysite$ cat startup.py
# OUT: imported django settings
# OUT: imported django models
print 'funky donkey at work'
# OUT: funky donkey at work
sa@wks:~/0/django/mysite$ cat /home/sa/.pythonrc
# import saved bpython session if available
try:
from startup import *
except:
pass
# do for bpython what shell_plus from django-extensions does for iPython
try:
from django.core.management import setup_environ
import settings
setup_environ(settings)
print 'imported django settings'
try:
exec_strs = ["from %s.models import *"%apps for apps in settings.INSTALLED_APPS ]
for x in exec_strs:
try:
exec(x)
except:
print 'Not imported for %s' %x
print 'imported django models'
except:
pass
except:
pass
sa@wks:~/0/django/mysite$ source /home/sa/.bashrc; bpython
funky donkey at work
imported django settings
imported django models
>>>
sa@wks:~/0/django/mysite$
Using bpython with multiple Python versions
Currently (May 2010) we can run bpython with those Python versions:
2.4, 2.5, 2.6, 2.7 and 3. The default version is the one our system
links to:
sa@wks:~$ type ll; ll $(which python)
ll is aliased to `ls -lh'
lrwxrwxrwx 1 root root 9 Jan 23 07:56 /usr/bin/python -> python2.5
sa@wks:~$
However, we can use others as well by simply creating an alias in our
~/.bashrc:
sa@wks:~$ grep bpython .bashrc
###_ . bpython
alias bp='bpython'
alias bp26='python2.6 -m bpython.cli'
alias bp31='python3.1 -m bpython.cli'
sa@wks:~$
We can then type bp and always run bpython from our system default
Python interpreter, or, we can, for example, issue bp26 to run bpython
from Python version 2.6 etc.
Datatypes and Structures
Python is a dynamically typed language with a strong type system.
Bei vielen Algorithmen hängt der Ressourcenbedarf, also sowohl die
benötigte Laufzeit als auch der Speicherplatzbedarf, von der
Verwendung geeigneter Datenstrukturen ab.
Literals
Strings
Decimal
Containers also known as Collections
Sequences
Ordered Sequences
Unordered Sequences
Mappings
Syntax and Semantics
One of the things I like most about Python is that it is not as wordy
as Java and not as cryptic as Perl (once you know Python, you will
stay away from those two anyways) but just a great language with an
pragmatic approach to software development.
Theory
Stackless Python
Type System
Scope / Namespaces
- What are variables, really? You can think of them as names
referring to values. So, after the assignment x = 1, the name x
refers to the value 1.
- “invisible dictionary” is called a namespace or scope.
- So, how many namespaces are there? In addition to the global scope,
each function call creates a new one.
- Variables that are used inside functions like this are called local
variables (as opposed to global variables).
- shadowing; If a local variable or parameter exists with the same
name as the global variable you want to access, you can’t do it
directly. The global variable is shadowed by the local one.
- we can rebind global variables using, within the function, `global
<variable_name_of_global_variable>'
- Use global variables only when you have to. They tend to make your
code less readable and less robust. Local variables make your
program more abstract because they are “hidden” inside functions.
- nested scopes are possible since Python v2.2
- http://docs.python.org/py3k/tutorial/classes.html#id2
Polymorphism
Encapsulation
- encapsulation isn’t the same as polymorphism. Polymorphism enables
you to call the methods of an object without knowing its class
(type of object). Encapsulation enables you to use the object
without worrying about how it’s constructed.
- So how can you “encapsulate” the name within the object? No
problem. You make it an attribute. Attributes are variables that
are a part of the object, just like methods;
- The state of an object is described by its attributes (like its
name, for example).
- Python doesn’t support privacy directly (using accessor methods),
but relies on the programmer to know when it is safe to modify an
attribute from the outside. properties, a powerful alternative to
accessors.
- To make a method or attribute private (inaccessible from the
outside), simply start its name with two underscores:
- Inside a class definition, all names beginning with a double
underscore are “translated” by adding a single underscore and the
class name to the beginning. So, if you know how this works behind
the scenes, it is still possible to access private methods outside
the class, even though you’re not supposed to.
- If you don’t want the name-mangling effect, but you still want to
send a signal for other objects to stay away, you can use a single
initial underscore. This is mostly just a convention, but has some
practical effects. For example, names with an initial underscore
aren’t imported with starred imports (from module import ).
* Functional Programming
Functions in Python are so-called First-Class objects.
Introspection
Aspect Oriented Programming
Magic Methods
Metaprogramming with Python
Metaclass
A basic blueprint to build other/enhanced classes with ...
- metaclass
- http://en.wikipedia.org/wiki/Metaclass
- the built-in class type is a metaclass
- Eine Metaklasse ist in der Objektorientierung die Klasse einer
Klasse. Sie definiert Verfahren zum Erzeugen von Instanzen der
Klasse, deren Metaklasse sie ist, sowie statische Methoden, also
solche, für deren Ausführung keine Objekte benötigt werden.
Abstract (Base) Class
Its main use is to define and build interfaces ...
Reflection
Design Patterns
A design pattern is a general reusable solution to a commonly
occurring problem in software design.
— Wikipedia
This is a huge subject, ready to fill bookshelves on its own. This
subsection will look at design patterns at code-level with regards to
Python i.e. not all known design patterns exist in Python respectively
make sense using them when programming in Python (this is true for any
other language as well).
Design patterns can be divided into several categories: Creational
Patterns, Structural Patterns, Behavioral Patterns and Concurrency
Patterns. They are described using the concepts of delegation,
aggregation, and consultation.
There also exists another classification that has the notion of
architectural design patterns which may be applied at the architecture
level of the software such as the MVC (Model-View-Controller) pattern.
This high-level view on design patterns is not covered here.
Creational Pattern
Creational design patterns are design patterns that deal with object
creation mechanisms, trying to create objects in a manner suitable to
the situation. The basic form of object creation could result in
design problems or added complexity to the design. Creational design
patterns solve this problem by somehow controlling this object
creation.
Singleton
Structural Pattern
Structural design patterns are design patterns that ease the design by
identifying a simple way to realize relationships between entities.
Facade
Proxy
Adapter
Decorator
We already know the
difference between a decorator and an adapter.
What a decorator does is attach additional responsibilities to an
object dynamically, keeping the same interface. Decorators provide a
flexible alternative to subclassing for extending functionality.
Subclassing adds behaviour at compile time whereas decorating can
provide new behaviour at runtime.
Decorators vs. the Decorator Pattern
First, we need to understand that the word decorator in Python was
used with some trepidation, because there was concern that it would be
completely confused with the Decorator pattern from the Design
Patterns book. At one point other terms were considered for the
feature, but decorator seems to be the one that sticks.
Indeed, we can use Python decorators to implement the Decorator
pattern, but that is an extremely limited use of it. Python
decorators, I think, are best equated to macros.
Example:
A decorator wraps a function i.e. the two below are equivalent
This
orig_function = my_decorator(orig_function)
is equivalent to
@my_decorator
def orig_function():
print "inside orig_function"
When the compiler passes over this code, orig_function() is compiled
and the resulting function object is passed to the my_decorator code,
which does something to produce a function-like object (read callable)
that is then substituted for the original orig_function().
Behavioral Pattern
Behavioral design patterns are design patterns that identify common
communication patterns between objects and realize these patterns. By
doing so, these patterns increase flexibility in carrying out this
communication.
Delegation
Strategy
Chain of Responsibility
Observer
Visitor
Template
Concurrency Pattern
Concurrency patterns are those types of design patterns that deal with
multi-threaded programming paradigm.
Practice
This section provides miscellaneous information within regards to
Python.
Package / Distribute / Install
This section is all about packaging, distributing and installing
Python software.
History
This text is a literal copy take from Martijn Faassen blog where he
describes the current (February 2010) state of packaging/distributing
Python software.
The reason I (Markus Gattol) include it here in full length again is
that I find it utterly important for anyone to understand the pig
picture about why things are the way the are today and what happened
during the last ten or so years so that we finally ended up with a
pretty amazing toolchain and infrastructure in order to develop,
package, distribute and share Python software.
Introduction
Earlier this year I (read Martijn Faassen) was at PyCon in the US. I
had an interesting experience there: people were talking about the
problem of packaging and distributing Python libraries. People had the
impression that this was an urgent problem that had not been solved
yet. I detected a vibe asking for the Python core developers to please
come and solve our packaging problems for us.
I felt like I had stepped into a parallel universe. I have been using
powerful tools to assemble applications from Python packages
automatically for years now. Last summer at EuroPython, when this
discussion came up again, I maintained that packaging and distributing
Python libraries is a solved problem. I put the point strongly, to
make people think. I fully agree that the current solutions are
imperfect and that they can be improved in many ways. But I also
maintain that the current solutions are indeed solutions.
There is now a lot of packaging infrastructure in the Python
community, a lot of technology, and a lot of experience. I think that
for a lot of Python developers the historical background behind all
this is missing. I will try to provide one here. It is important to
realize that progress has been made, step by step, for more than a
decade now, and we have a fine infrastructure today.
I have named some important contributors to the Python packaging
story, but undoubtedly I have also not mentioned a lot of other
important names. My apologies in advance to those I missed.
The dawn of Python packaging
The Python world has been talking about solutions for packaging and
distributing Python libraries for a very long time. I remember when I
was new in the Python world about a decade ago, in the late 90s, it
was considered important and urgent that the Python community
implement something like Perl's CPAN. I am sure too that this debate
had started long before I started paying attention.
I have never used CPAN, but over the years I have seen it held up by
many as something that seriously contributes to the power of the Perl
language. With CPAN, I understand, you can search and browse Perl
packages and you can install them from the net.
So, lots of people were talking about a Python equivalent to CPAN with
some urgency. At the same time, the Python world did not seem to move
very quickly on this front...
Distutils
The Distutils SIG (special interest group) was started in late 1998.
Greg Ward in the context of this discussion group started to create
Distutils about this time. Distutils allows you to structure your
Python project so that it has a setup.py. Through this setup.py you
can issue a variety of commands, such as creating a tarball out of
your project, or installing your project. Distutils importantly also
has infrastructure to help compiling C extensions for your Python
package. Distutils was added to the Python standard library in Python
1.6, released in 2000.
Metadata
We now had a way to distribute and install Python packages, if we did
the distribution ourselves. We did not have a centralized index (or
catalog) of packages yet, however. To work on this, the Catalog SIG
was started in the year
2000.
The first step was to standardize the metadata that could be cataloged
by any index of Python packages. Andrew Kuchling drove the effort on
this, culminating in PEP 241 in 2001, later updated by PEP 314.
Distutils was modified so it could work with this standardized
metadata.
PyPI (Python Project Index)
In late 2002, Richard Jones started work on the PyPI. PyPI was
initially known as the Cheeseshop. Until around January 2010 it was
then known as Python Package Index which turned out to
not be appropriate anymore since the Python community now distributes
so-called Projects rather than Packages.
The first work on an implementation started, and PEP 301 that
describes PyPI was also created then. Distutils was extended so the
metadata and packages themselves could be uploaded to this package
index. By 2003, the Python package index was up and running.
The Python world now had a way to upload packages and metadata to a
central index. If we then manually downloaded a package we could
install it using setup.py thanks to Distutils.
Setuptools
Phillip Eby started work on Setuptools in 2004. Setuptools is a whole
range of extensions to Distutils such as from a binary installation
format (eggs), an automatic package installation tool, and the
definition and declaration of scripts for installation. Work continued
throughout 2005 and 2006, and feature after feature was added to
support a whole range of advanced usage scenarios.
By 2005, you could install packages automatically into your Python
interpreter using easy_install. Dependencies would be automatically
pulled in. If packages contained C code it would pull in the binary
egg, or if not available, it would compile one automatically.
The sheer amount of features that Setuptools brings to the table must
be stressed: namespace packages, optional dependencies, automatic
manifest building by inspecting version control systems, web scraping
to find packages in unusual places, recognition of complex version
numbering schemes, and so on, and so on. Some of these features
perhaps seem esoteric to many, but complex projects use many of them.
The Problems of Shared Packages
The problem remained that all these packages were installed into your
Python interpreter. This is icky. People's site-packages directories
became a mess of packages. You also need root access to easy_install a
package into your system Python. Sharing all packages in a direcory in
general, even locally, is not always a good idea: one version of a
library needed by one application might break another one. Solutions
for this emerged in 2006.
Virtualenv
Ian Bicking drove one line of solutions: virtual-python, which evolved
into workingenv, which evolved into virtualenv in 2007. The concept
behind this approach is to allow the developer to create as many fully
working Python environments as they like from a central system
installation of Python. When the developer activates the virtualenv,
easy_install respectively its successor pip will install all packages
into its the virtualenv's site-packages directory. This allows you to
create a virtualenv per project and thus isolate each project from
each other.
Buildout
In 2006 as well, Jim Fulton created Buildout, building on Setuptools
and easy_install. Buildout can create an isolated project environment
like virtualenv does, but is more ambitious: the goal is to create a
system for repeatable installations of potentially very complex
projects. Instead of writing an INSTALL.txt that tells others who to
install the prerequites for a package (Python or not), with Buildout
these prerequisites can be installed automatically.
The brilliance of Buildout is that it is easily extensible with new
installation recipes. These recipes themselves are also installed automatically
from PyPI. This has spawned a whole ecosystem of Buildout recipes that can do a
whole range of things, from generating documentation to installing MySQL.
Since Buildout came out of the Zope world, Buildout for a long time was seen as
something only Zope developers would use, but the technology is not
Zope-specific at all, and more and more developers are picking up on it.
In 2008, Ian Bicking created an alternative for easy_install called
pip, also building on Setuptools. Less ambitious than buildout, it
aimed to fix some of the shortcomings of easy_install. I have not used
it myself yet, so I will leave it to others to go into details.
Setuptools and the Standard Library
The many improvements that Setuptools brought to the Python packaging
story had not made it into the Python Standard Library, where
Distutils was stagnating. Attempts had been made to bring Setuptools
into the standard library at some point during its development, but
for one reason or another these efforts had foundered.
Setuptools probably got where it is so quickly because it worked
around often very slow process of adopting something into the standard
library, but that approach also helped confuse the situation for
Python developers.
Last year Tarek Ziade started looking into the topic of bringing
improvements into Distutils. There was a discussion just before PyCon
2009 about this topic between various Python developers as well, which
probably explains why the topic was in the air. I understood that some
decisions were made:
- Let the people with extensive packaging experience (such as Tarek)
drive this process.
- Free the metadata from Distutils and Setuptools so that other
packaging tools can make use of it more easily.
Distribute
By 2008, Setuptools had become a vital part of the Python development
infrastructure. Unfortunately the Setuptools development process has
some flaws. It is very centered around Phillip Eby. While he had been
extremely active before, by that time he was spending a lot less
energy on it. Because of the importance of the technology to the wider
community, various developers had started contributing improvements
and fixes, but these were piling up.
This year, after some period of trying to open up the Setuptools
project itself, some of these developers led by Tarek Ziade decided to
fork Setuptools. The fork is named Distribute. The aim is to develop
the technology with a larger community of developers. One of the first
big improvements of the Distribute project is Python 3 support.
Quite understandably this fork led to some friction between Tarek,
Phillip and others. I trust that this friction will resolve itself and
that the developers involved will continue to work with each other, as
all have something valuable contribute.
Operating System Packaging
One point that always comes up in discussions about Python packaging
tools is operating system packaging. In particular Linux distributions
have developed extremely powerful ways to distribute and install
complex libraries and application, manage versions and dependencies
and so on.
Naturally when the topic of Python packaging comes up, people think
about operating system packaging solutions like this. Let me start off
that I fully agree that Python packaging solutions can learn a lot
from operating system packaging solutions.
Why don't we just use a solution like that directly, though? Why is a
Python specific packaging solution necessary at all?
There are a number of answers to this. One is that operating packaging
solutions are not universal: if we decided to use Debian's system,
what would we do on Windows?
The most important answer however is that there are two related but
also very different use cases for packaging:
- system administration: deploying and administrating existing software.
- development: combining software to develop new software.
The Python packaging systems described above primarily try to solve
the development use case: I am a Python developer, and I am developing
multiple projects at the same time, perhaps in multiple versions, that
have different dependencies. I need to reuse packages created by other
developers, so I need an easy way to depend on such packages. These
packages are sometimes in a rather early state of development, or
perhaps I am even creating a new one. If I want to improve such a
package I depend on, I need an easy way to start hacking on it.
Operating system packaging solutions as I have seen them used are ill
suited for the development use case. They are aimed at creating a
single consistent installation that is easy to upgrade with an eye on
security. Backwards compatibility is important. Packages tend to be
relatively mature.
For all I know it might indeed be possible to use an operating system
packaging tool as a good development packaging tool. But I have heard
very little about such practices. Please enlighten me if you have.
It is also important to note that the Python world is not as good as
it should be at supporting operating system packaging solutions. The
freeing up of package metadata from the confines of the setup.py file
into a more independently reusable format as was decided at PyCon
should help here.
Conclusions
We are now in a time of consolidation and opening up. Many of the
solutions pioneered by Setuptools are going to be polished to go into
the Python Standard Library. At the same time, the community
surrounding these technologies is opening up. By making metadata used
by Distutils and Setuptools more easily available to other systems,
new tools can also more easily be created.
The Python packaging story had many contributors over the years. We
now have a powerful infrastructure. Do we have an equivalent to CPAN?
I do not know enough about CPAN to be sure. But what we have is
certainly useful and valuable. In my parallel universe, I use advanced
Python packaging tools every day, and I recommend all Python
programmers to look into this technology if they have not already.
Join me in my parallel universe!
Update: I just found out there was a huge thread on python-dev about
this in the last few days which focused around the question whether we
have the equivalent of CPAN now. One of them funny coincidences...
History Continues
At PyCon 2010 the decision was made to basically exchange distutils
with distutils2 where distutils2 is a fork of Distribute. Setuptools,
distutils and Distribute are going to die (read phased out). pip will
stay and once distutils is replaced by distutils2, it will work with
it as it does now (March 2010) with Distribute. Take a look at this
picture:
The distutils module is currently part of the standard library and
will be until Python 3.3 — it will be discontinued in Python 3.3 in
favor of distutils2 which will be backwards compatible down to Python
2.4.
Glossary
WRITEME
Examples
Just provide the below links and then show a bunch of examples for
each step i.e. package, distribute, install; thereby using the tools
(pip, distribute, etc.) listed below
WRITEME
Tools / Utils
This section provides information on what tools I use on a daily basis
when it comes to developing/deploying/administer/test/etc. Python
software.
Virtualenv / Virtualenvwrapper / Virtualenv-commands
This one is all about gaining freedom — the kind of freedom that
allows us to be creative, have fun and get things done quickly and in
a straight forward and simple manner. So, what is it that virtualenv
does in a nutshell?
By using virtualenv and possibly virtualenvwrapper and/or
virtualenv-commands on top of it, we can create sanboxes also known as
virtual environments. What this gives us is the benefit of isolated
environments i.e. so we can work without risking to mess up the rest
of our system by mistake.
By default we have symlinks going from those virtual environments out
and going into our global Python context/space e.g. to the Python
interpreter. However, we can also make those sandboxes totally
separated from the rest of the system by using the --no-site-packages
switch to virtualenv, thus allowing us to try out software, alter
software, add/remove things, etc. — all without any danger of
accidentally doing something stupid to our global Python
context/space.
In other words: virtualenv is basically a Python symmetric link
utility for cloning an existing Python installation or creating an
entirely separated one so that we can easily install/uninstall/develop
Python software at a different location than the standard one e.g.
/usr/lib/python<version>/dist-packages.
Installing and setting up Virtualenv
Installing virtualenv is easy. Debian provides a package for it
sa@wks:~$ type dpl; dpl virtualenv | grep ii
dpl is aliased to `dlocate -l'
ii python-virtualenv 1.4.9-1 Python virtual environment creator
sa@wks:~$ virtualenv --version
1.4.9
sa@wks:~$ virtualenv --help
Usage: virtualenv [OPTIONS] DEST_DIR
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-v, --verbose Increase verbosity
-q, --quiet Decrease verbosity
-p PYTHON_EXE, --python=PYTHON_EXE
The Python interpreter to use, e.g.,
--python=python2.6 will use the python2.6
interpreter to create the new environment. The
default is the interpreter that virtualenv was
installed with (/usr/bin/python)
--clear Clear out the non-root install and start from
scratch
--no-site-packages Don't give access to the global site-packages dir
to the virtual environment
--unzip-setuptools Unzip Setuptools or Distribute when installing it
--relocatable Make an EXISTING virtualenv environment
relocatable. This fixes up scripts and makes all
.pth files relative
--distribute Use Distribute instead of Setuptools. Set environ
variableVIRTUALENV_USE_DISTRIBUTE to make it the
default
sa@wks:~$
Of course, one could also use easy_install virtualenv or even better,
pip install virtualenv but then it is probably best to use Debian's
package for the global Python context/space (the opposite of a virtual
environment context/space created using virtualenv) right away — I
tend to sometimes replicate systems (e.g. when I swap the HDD in my
subnotebook for a SSD (Solid State Drive)) in which case it is easy to
automatically replicate the set of installed Debian packages so ...
Using Virtualenv
Basically, what we need to know is how to create a new virtual
environment (line 1), enter and activate it (lines 27 and 28), carry
out some commands (e.g. line 29, looking what Python interpreter is
currently active) and last but not least, switch back from the virtual
environment into the global Python context/space (line 31) and yet
again, look up the currently active Python interpreter (lines 32 and
33):
1 sa@wks:~/0/1$ virtualenv --distribute --no-site-packages my_test_virt_env
2 New python executable in my_test_virt_env/bin/python
3 Installing distribute................................done.
4 sa@wks:~/0/1$ type td; td my_test_virt_env/
5 td is aliased to `tree --charset ascii -d -I \.git*\|*\.\~*\|*\.pyc'
6 my_test_virt_env/
7 |-- bin
8 |-- include
9 | `-- python2.6 -> /usr/include/python2.6
10 `-- lib
11 `-- python2.6
12 |-- config -> /usr/lib/python2.6/config
13 |-- distutils
14 |-- encodings -> /usr/lib/python2.6/encodings
15 |-- lib-dynload -> /usr/lib/python2.6/lib-dynload
16 `-- site-packages
17 |-- distribute-0.6.10-py2.6.egg
18 | |-- EGG-INFO
19 | `-- setuptools
20 | |-- command
21 | `-- tests
22 `-- pip-0.8-py2.6.egg
23 `-- EGG-INFO
24 `-- scripts
25
26 18 directories
27 sa@wks:~/0/1$ cd my_test_virt_env/
28 sa@wks:~/0/1/my_test_virt_env$ source bin/activate
29 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env$ which python
30 /home/sa/0/1/my_test_virt_env/bin/python
31 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env$ deactivate
32 sa@wks:~/0/1/my_test_virt_env$ which python
33 /usr/bin/python
34 sa@wks:~/0/1/my_test_virt_env$ cd
The whole point of using virtualenv can be best seen from lines 30 and
33 — first we use a virtual environment and therefore our Python
interpreter lives at /home/sa/0/1/my_test_virt_env/bin/python but then
we are back in the global Python context/space where we would use
/usr/bin/python. By the way, td from line 4 is just an alias in my
~/.bashrc.
Virtualenvwrapper
Virtualenvwrapper is a set of extensions to virtualenv. The extensions
include wrappers for creating and deleting virtual environments and
otherwise managing our development workflow, making it easier to work
on more than one project at a time without introducing conflicts in
their dependencies.
Installing and activating virtualenvwrapper is easy — one might
either use pip install virtualenvwrapper or, in my opinion even
better, go straight for the aptitude install virtualenvwrapper option
which works out of the box in case we also have bash-completion
installed and enabled it in /etc/bash.bashrc (see
/usr/share/doc/virtualenvwrapper/README.Debian for more information).
Next we are going to address our ~/.bashrc file:
35 sa@wks:~$ grep -A3 -m1 'WORKON_HOME .bashrc
36 export WORKON_HOME=$HOME/0/1
37 alias cdveroot='cd $WORKON_HOME'
38 sa@wks:~$ source .bashrc; echo $WORKON_HOME
39 /home/sa/0/1
The important part here is with line 36 where we tell
virtualenvwrapper where our virtual environments are going to live on
the filesystem.
With line 37 we also add an alias which is going to save us a lot of
time down the road since it always beams us back into $WORKON_HOME no
matter where we are on the filesystem — in my case that is
/home/sa/0/1 as can be seen from line 39.
Excellent! We are done installing and setting up virtualenv and
virtualenvwrapper. More information can be found here, here and here.
Usage Examples - Commands
Below I am going to provide a few examples about how to use
virtualenvwrapper so folks can see how things work right away ;-]
40 sa@wks:~$ workon
41 my_test_virt_env
42 sa@wks:~$ workon my_test_virt_env
43 (my_test_virt_env)sa@wks:~$ cdvirtualenv
44 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env$ cdveroot
45 (my_test_virt_env)sa@wks:~/0/1$ ll
46 total 36K
47 drwxr-xr-x 6 sa sa 4.0K Dec 27 14:20 my_test_virt_env
48 -rwxr-xr-x 1 sa sa 67 Dec 27 15:27 postactivate
49 -rwxr-xr-x 1 sa sa 69 Dec 27 15:27 postdeactivate
50 -rwxr-xr-x 1 sa sa 67 Dec 27 15:27 postmkvirtualenv
51 -rwxr-xr-x 1 sa sa 61 Dec 27 15:27 postrmvirtualenv
52 -rwxr-xr-x 1 sa sa 68 Dec 27 15:27 preactivate
53 -rwxr-xr-x 1 sa sa 70 Dec 27 15:27 predeactivate
54 -rwxr-xr-x 1 sa sa 92 Dec 27 15:27 premkvirtualenv
55 -rwxr-xr-x 1 sa sa 62 Dec 27 15:27 prermvirtualenv
56 (my_test_virt_env)sa@wks:~/0/1$ cd /tmp
57 (my_test_virt_env)sa@wks:/tmp$ cdvirtualenv
58 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env$ pwd
59 /home/sa/0/1/my_test_virt_env
The command reference list all commands available. My favorite is
probably workon and cdvirtualenv — the former is used to list/switch
amongst virtual environments and the later one to beam us right back
into the root of the currently activated virtual environment no matter
where we are on the filesystem. Gosh! I love it! About lines 48 to 55,
those are hooks which I will tell more about later.
60 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env$ mkvirtualenv --no-site-packages --distribute test
61 New python executable in test/bin/python
62 Installing distribute.........................done.
63 (test)sa@wks:~/0/1/my_test_virt_env$ cdvirtualenv
64 (test)sa@wks:~/0/1/test$ workon
65 my_test_virt_env
66 test
Line 60 shows how easy it is to create a new virtual environment using
mkvirtualenv. As with virtualenv from line 1, we can supply
--no-site-packages and --distribute (we could also put
VIRTUALENV_USE_DISTRIBUTE into ~/.bashrc as shown below) to
mkvirtualenv — command line arguments to virtualenvwrapper are passed
right through to virtualenv! Also note that by creating our new
virtual environment test using mkvirtualenv, we switched right to it
as can be seen in line 63.
Now we have two virtual environments (lines 65 and 66) already which
can be listed using workon without any argument.
67 (test)sa@wks:~/0/1/test$ workon my_test_virt_env
68 (my_test_virt_env)sa@wks:~/0/1/test$ cdvirtualenv
69 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env$ rmvirtualenv my_test_virt_env
70 ERROR: You cannot remove the active environment ('my_test_virt_env').
71 Either switch to another environment, or run 'deactivate'.
72 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env$ rmvirtualenv test
73 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env$ workon
74 my_test_virt_env
Lines 67 to 74 show a few things about deleting a virtual environment.
As we can see from lines 69 to 71, deleting/removing the currently
active virtual environment does not work — this is a safety switch
provided by virtualenvwrapper. As lines 72 to 74 show, our former
created virtual environment test has been removed — basically this is
the same as using rm -r /home/sa/0/1/test but then rmvirtualenv takes
care not to wipe out the currently active virtual environment.
75 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env$ cd
76 (my_test_virt_env)sa@wks:~$ cdvirtualenv bin
77 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env/bin$ pwd
78 /home/sa/0/1/my_test_virt_env/bin
79 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env/bin$
80 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env/bin$ cdsitepackages
81 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env/lib/python2.6/site-packages$ pwd
82 /home/sa/0/1/my_test_virt_env/lib/python2.6/site-packages
Since I am such a fan of cdvirtualenv, line 76 shows us more of its
magic — appending an argument such as bin does not beam us back into
the root of the currently active virtual environment but actually
moves us down one level into /home/sa/0/1/my_test_virt_env/bin. Gosh
the 2nd! ;-]
cdvirtualenv has a friend called cdsitepackages which is no less
amazing as it beams us right into the site-packages directory of our
currently activated virtual environment. Now listing its contents
would be a simple matter of using ls.
83 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env/lib/python2.6/site-packages$ cd
84 (my_test_virt_env)sa@wks:~$ lssitepackages
85 distribute-0.6.10-py2.6.egg pip-0.8-py2.6.egg virtualenvwrapper
86 easy-install.pth setuptools.pth virtualenvwrapper-1.23-py2.6.egg-info
87 (my_test_virt_env)sa@wks:~$ lssitepackages -l
88 total 24
89 drwxr-xr-x 4 sa sa 4096 Dec 27 12:55 distribute-0.6.10-py2.6.egg
90 -rw-r--r-- 1 sa sa 236 Dec 27 12:55 easy-install.pth
91 drwxr-xr-x 3 sa sa 4096 Dec 27 12:55 pip-0.8-py2.6.egg
92 -rw-r--r-- 1 sa sa 29 Dec 27 12:55 setuptools.pth
93 drwxr-xr-x 3 sa sa 4096 Dec 27 14:30 virtualenvwrapper
94 drwxr-xr-x 2 sa sa 4096 Dec 27 14:30 virtualenvwrapper-1.23-py2.6.egg-info
95 (my_test_virt_env)sa@wks:~/0/1$ cd /tmp
However, what if we just wanted to know its contents without visiting
../site-packages/? Easy, we use lssitepackages as shown in lines 84
and 87 respectively. Line 84 lists all contents of
/home/sa/0/1/my_test_virt_env/lib/python2.6/site-packages even though
we are currently inside /home/sa. Also, again, note how the -l switch
gets passed through in line 87.
The last command we are going to take a look at is add2virtualenv. It
is used to link code into the currently active virtual environment.
Note that linking here does not determine a symmetrical link but
rather adding another path to Python's module search paths.
96 (my_test_virt_env)sa@wks:/tmp$ git clone git://github.com/pinax/pinax.git
97 Initialized empty Git repository in /tmp/pinax/.git/
98 remote: Counting objects: 26080, done.
99 remote: Compressing objects: 100% (9391/9391), done.
100 remote: Total 26080 (delta 14937), reused 25917 (delta 14828)
101 Receiving objects: 100% (26080/26080), 13.43 MiB | 120 KiB/s, done.
102 Resolving deltas: 100% (14937/14937), done.
103 (my_test_virt_env)sa@wks:/tmp$ cdvirtualenv
104 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env$ python
105 Python 2.6.4 (r254:67916, Nov 19 2009, 22:14:20)
106 [GCC 4.3.4] on linux2
107 Type "help", "copyright", "credits" or "license" for more information.
108 >>> import sys, pprint
109 >>> pprint.pprint(sys.path)
110 ['',
111 '/home/sa/0/1/my_test_virt_env/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg',
112 '/home/sa/0/1/my_test_virt_env/lib/python2.6/site-packages/pip-0.8-py2.6.egg',
113 '/home/sa/0/1/my_test_virt_env/lib/python2.6',
114 '/home/sa/0/1/my_test_virt_env/lib/python2.6/plat-linux2',
115 '/home/sa/0/1/my_test_virt_env/lib/python2.6/lib-tk',
116 '/home/sa/0/1/my_test_virt_env/lib/python2.6/lib-dynload',
117 '/usr/lib/python2.6',
118 '/usr/lib64/python2.6',
119 '/usr/lib/python2.6/plat-linux2',
120 '/usr/lib/python2.6/lib-tk',
121 '/usr/lib64/python2.6/lib-tk',
122 '/home/sa/0/1/my_test_virt_env/lib/python2.6/site-packages']
123 >>>
124 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env$ cdsitepackages
125 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env/lib/python2.6/site-packages$ type pi; pi pth
126 pi is aliased to `ls -la | grep'
127 -rw-r--r-- 1 sa sa 236 Dec 27 12:55 easy-install.pth
128 -rw-r--r-- 1 sa sa 29 Dec 27 12:55 setuptools.pth
129 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env/lib/python2.6/site-packages$ add2virtualenv /tmp/pinax/
130 Warning: Converting "/tmp/pinax/" to "/tmp/pinax"
131 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env/lib/python2.6/site-packages$ pi pth
132 -rw-r--r-- 1 sa sa 236 Dec 27 12:55 easy-install.pth
133 -rw-r--r-- 1 sa sa 29 Dec 27 12:55 setuptools.pth
134 -rw-r--r-- 1 sa sa 11 Dec 27 22:46 virtualenv_path_extensions.pth
135 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env/lib/python2.6/site-packages$ cat virtualenv_path_extensions.pth
136 /tmp/pinax
137 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env/lib/python2.6/site-packages$ python
138 Python 2.6.4 (r254:67916, Nov 19 2009, 22:14:20)
139 [GCC 4.3.4] on linux2
140 Type "help", "copyright", "credits" or "license" for more information.
141 >>> import sys, pprint
142 >>> pprint.pprint(sys.path)
143 ['',
144 '/home/sa/0/1/my_test_virt_env/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg',
145 '/home/sa/0/1/my_test_virt_env/lib/python2.6/site-packages/pip-0.8-py2.6.egg',
146 '/home/sa/0/1/my_test_virt_env/lib/python2.6',
147 '/home/sa/0/1/my_test_virt_env/lib/python2.6/plat-linux2',
148 '/home/sa/0/1/my_test_virt_env/lib/python2.6/lib-tk',
149 '/home/sa/0/1/my_test_virt_env/lib/python2.6/lib-dynload',
150 '/usr/lib/python2.6',
151 '/usr/lib64/python2.6',
152 '/usr/lib/python2.6/plat-linux2',
153 '/usr/lib/python2.6/lib-tk',
154 '/usr/lib64/python2.6/lib-tk',
155 '/home/sa/0/1/my_test_virt_env/lib/python2.6/site-packages',
156 '/tmp/pinax']
157 >>>
158 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env/lib/python2.6/site-packages$ cdvirtualenv
159 (my_test_virt_env)sa@wks:~/0/1/my_test_virt_env$ add2virtualenv
160 Usage: add2virtualenv dir [dir ...]
161
162 Existing paths:
163 /tmp/pinax
164 (my_test_virt_env) sa@wks:~/0/1/my_test_virt_env$ python -c 'import pinax; print pinax.VERSION'
165 (0, 9, 0, 'alpha', 1)
166 (my_test_virt_env) sa@wks:~/0/1/my_test_virt_env$ cd; deactivate
With this example we first clone (read download) Pinax source code
into /tmp in lines 96 to 102 — this Pinax source code is what we are
going to link into our currently active virtual environment
my_test_virt_env.
The important thing above is with line 129 which makes it so that
virtualenv_path_extensions.pth from line 134 is put into place. Line
129 puts a new module search path into that file as can be seen from
line 156. It worked as can be seen from lines 164 and 165 where we
first import Pinax and then take a look at its version number — which
would not be possible if Python would not know where to find it on the
filesystem.
Usage Examples - Hooks
Virtualenvwrapper provides hooks that can be used to carry out actions
at certain times depending on the work we do with regards to our
virtual environments.
There are two types of hooks. Global hooks (lines 48 to 55) which live
in $WORKON_HOME are the same for any of our virtual environments
therefore the actions carried out by them are the same for any of our
virtual environments.
Secondly, there are per virtual environment hooks which live in
$VIRTUAL_ENV/bin. Those are specific to any virtual environment and so
the actions they carry out are only applied to this particular virtual
environment.
While we have a bunch of global hooks, currently (August 2010) there
are only four per virtual environment hooks namely postactivate,
postdeactivate, preactivate and predeactivate.
Hooks are either sourced (allowing them to modify our shell
environment e.g. change the color of our shell prompt) or run as an
external program (e.g. cp, ls, another shell script, some Python
script, etc.) at the appropriate trigger time.
As an example, we are going to add a little color in order to make it
easier for us to distinguish whether we are using a virtual
environment or whether we are acting within the global Python
context/space of our operating system.
167 sa@wks:~/0/1$ ll
168 total 36K
169 drwxr-xr-x 6 sa sa 4.0K Dec 27 14:20 my_test_virt_env
170 -rwxr-xr-x 1 sa sa 150 Dec 27 21:11 postactivate
171 -rwxr-xr-x 1 sa sa 69 Dec 27 15:27 postdeactivate
172 -rwxr-xr-x 1 sa sa 67 Dec 27 15:27 postmkvirtualenv
173 -rwxr-xr-x 1 sa sa 61 Dec 27 15:27 postrmvirtualenv
174 -rwxr-xr-x 1 sa sa 68 Dec 27 15:27 preactivate
175 -rwxr-xr-x 1 sa sa 70 Dec 27 15:27 predeactivate
176 -rwxr-xr-x 1 sa sa 92 Dec 27 15:27 premkvirtualenv
177 -rwxr-xr-x 1 sa sa 62 Dec 27 15:27 prermvirtualenv
178 sa@wks:~/0/1$ cat postactivate
179 #!/bin/sh
180 # This hook is run after every virtualenv is activated.
181
182 sa@wks:~/0/1$ workon my_test_virt_env
183 (my_test_virt_env)sa@wks:~/0/1$ tsw
184 (my_test_virt_env)sa@wks:~/0/1$ deactivate
185 sa@wks:~/0/1$ cat postactivate
186 #!/bin/sh
187 # This hook is run after every virtualenv is activated.
188 PS1="\[\033[01;33m\](`basename \"$VIRTUAL_ENV\"`)\[\033[00m\] $_OLD_VIRTUAL_PS1"
189 sa@wks:~/0/1$ workon my_test_virt_env
190 (my_test_virt_env) sa@wks:~/0/1/$ tsw
191 (my_test_virt_env) sa@wks:~/0/1/$ deactivate; cd
192 sa@wks:~$
The first image above show things before (read default virtualenv
setting) line 188 was put into place. The second image shows things
after we put line 188 in place — the currently active virtual
environment is now yellow plus we got a blank in between the yellow
colored virtual environment and our default prompt.
The arcane tsw from lines 183 and 190 is yet another alias from my
~/.bashrc. It is used to take screenshots.
Another very handy thing I would recommend putting into postdeactivate
is cd $VIRTUAL_ENV — it automatically beams us into the virtual
environment's root when we active it i.e. no more need for using
cdvirtualenv right after workon <foo>.
Since PIP is used by pretty much anybody these days, here is how we
make virtualenv/virtualenvwrapper and PIP complement each other
nicely, thus providing for enhanced user experience:
sa@wks:~$ grep -A5 '. pip' .bashrc
###_ . pip
export PIP_VIRTUALENV_BASE=$WORKON_HOME
export PIP_RESPECT_VIRTUALENV=true
export PIP_REQUIRE_VIRTUALENV=true
export VIRTUALENV_USE_DISTRIBUTE=1
sa@wks:~$
This makes PIP detect an active virtual environment and install to it,
without having to pass it the -E switch plus it tells PIP to bail out
if we currently do not have a virtual environment activated — the
latter is some sort of precaution in order to avoid doing something
silly to our global Python context/space.
Also, VIRTUALENV_USE_DISTRIBUTE makes PIP use Distribute rather than
Setuptools — without this setting we would have to specify
--distribute on the CLI (Command Line Interface).
Django Environment
This one I love! So far we have seen how to create virtual
environments. We even use virtualenvwrapper to make it a joy to work
with those virtual environments.
We create all kinds of Python projects atop/inside those virtual
environments e.g. we can test our code with different Python versions
like for example 2.7 and 3.2 by simply switching back and forth with
workon 2.7 and workon 3.2 respectively. We can test our Django
application with different third party applications ... the things we
can do are literally endless ...
However, while creating virtual environments is now easy and fast
(mkvirtualenv), creating a Django project still is quite a bit work —
all the pip install django, django-admin startproject, pip install
django-extensions, pip install fabric, etc. stuff ... more so even the
creation of things like aliases and its dynamic enabling/disabling of
scoping based on what the current virtual environment is, symmetrical
links and all other custom things we might have in place. Gosh, we do
not want that tedious repetitive work no more ... the solution to this
problem is with django-environment!
Virtualenv-Commands
So far I am not using it but from what I have seen it is pretty cool
too. Please go here and here for more information.
PIP
Bash Completion
In order to have Bash completion with PIP, here is what we do:
- issue
pip completion --bash and
- put its output into
~/.bashrc which then looks like this
sa@wks:~$ grep -A7 "^_pip_completion()" .bashrc
_pip_completion()
{
COMPREPLY=( $( COMP_WORDS="${COMP_WORDS[*]}" \
COMP_CWORD=$COMP_CWORD \
PIP_AUTO_COMPLETE=1 $1 ) )
}
complete -o default -F _pip_completion pip
sa@wks:~$
A quick source ~/.bashrc picks up the changes/additions to
all the stuff in my ~/.bashrc and we are good to go ... TAB TAB ;-]
Editable Packages
Those are packages/applications we install using pip install -e e.g.
pip install -e git+git://github.com/pinax/pinax.git#egg=pinax
Disutils / Setuptools / Distribute
One should have read about the history of packaging before continuing
here.
python setup.py develop installs a package without moving it into
../site-packages/
Distribute
This is the one we recommend to use. It is the successor to
Setuptools, which itself has always been considered the better
Distutils.
Fabric
Debugger
Checker
- http://pychecker.sf.net
- http://www.logilab.org/projects/pylint
- In addition to the bug checking that PyChecker performs, Pylint
offers some additional features such as checking line length,
whether variable names are well-formed according to your coding
standard, whether declared interfaces are fully implemented, and
more
- pyflakes
Profiling
GNU Emacs
Since GNU Emacs is my weapon of choice for pretty much any battle
these days, I would like to honor my good fellow by explicitly telling
a bit how I made the out of the box setup which Emacs provides for
Python programming even more cosy ;-]
Miscellaneous
Security
This section is about handling security with special regards to how to
do this in the Django and Pinax ecosystem. It will also touch on pure
Python matters wherever needed and appropriate. The more general view
towards security on a lower layer (operating system, network, storage,
etc.) has its own page.
Anti-spam Measures
Authentication
Multimedia
This section is about handling multimedia with special regards to how
to do this in the Django and Pinax ecosystem. It will however touch on
pure Unix/Linux/Python matters wherever needed and appropriate. In
short, with this section we are going to have a look at how to
retrieve, process, store and serve all sorts of multimedia contents.
Audio
Dynamic Images
Encoding
sa@wks:~$ dpkg -l ffmpeg* | grep ii
ii ffmpeg2theora 0.24-2+b1 Theora video encoder using ffmpeg
sa@wks:~$
Static Images
Project and Life-Cycle Management
Project Management
Life-Cycle Management
Software Development Techniques
Test-driven Development
TDD (Test-driven Development)
|
