Using additional libraries and virtualenv

Author: Tomas Zulberti

The subjects I'm going to talk about are:

Python's standard library
What to do when something isn't on that library.
Installing additional libraries
Using virtualenv to solve the problems that show up

Python comes with around 350 usable libraries. http://docs.python.org/library/index.html

There's a bit of everything:

Stuff to work with mails
Fetching web pages
Interacting with the operating system
Working with compressed files
Parsing different types of documents (xml, json, etc...)

AND MANY MANY MORE....

But first of all, lets see some examples of things you can do with Python's built-in batteries.

Lets send an e-mail using a GMail account:

>>> import smtplib
>>> from email.MIMEText import MIMEText

>>> USERNAME = "tzulberti@gmail.com" # Put your username here
>>> PASSWORD = "not-gonna-tell" # Put your password here
>>> mailServer = smtplib.SMTP('smtp.gmail.com',587)
>>> mailServer.ehlo()
>>> mailServer.starttls()
>>> mailServer.ehlo()
>>> mailServer.login(USERNAME, PASSWORD)
>>> msg = MIMEText(" This is the e-mail's content... ")
>>> msg['From'] = USERNAME
>>> msg['To'] = USERNAME
>>> msg['Subject'] = "This is the e-mail's subject"
>>> mailServer.sendmail("tzulberti@gmail.com", "tzulberti@gmail.com", msg.as_string())

Lets read a web page's content.

>>> from urllib2 import urlopen
>>> response = urlopen("http://python.org.ar/pyar/")
>>> html_content = response.read()
>>> print html_content
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>

        <link rel="icon" href="/images/pyar.ico" type="image/ico">
            <meta http-equiv="Content-Type" content="text/html;charset=utf-8">
            <meta name="keywords" content="Python, PyAr, Python Argentina, user group,
                                           grupo de usuarios, community portal">
            <meta name="robots" content="index,follow">

            <title>Inicio - PyAr - Python Argentina</title>
            <script type="text/javascript" src="/moin_static182/common/js/common.js"></script>

Or work with compressed files

>>> import zipfile
>>> compressed_file = zipfile.ZipFile("THA_FILE.zip", "r")
>>> for name in compressed_file.namelist():
...    print name
>>> compressed_file.extract()

But python's standard library doesn't have it all:

It doesn't know how to read some file formats (yaml, mp3, video)
Web Frameworks (pylons, django, web2py)
Using APIs from Youtube, Deliciosus, etc....

We've seen that the standard library can do many things, but there are some that are easier if done with third-party libraries:

Web frameworks save us from some repetitive work.
API bindings save us the work of reading a lot of documentation on the services they wrap.
Though python knows how to read text and binary files, it's easier if it simply converts certain file formats to python structures.

Where to look for third-party libraries?

http://pypi.python.org/pypi

There are around 11.000 libraries ready to use. As we saw in Roberto Alsina's lecture (import antigravity), there are libraries to do mostly anything:

Colored console output
Alternative interactive python consoles (bpython, ipython)
Working with weird file formats
Downloading files from cuevana
Downloading videos from youtube
Parsing xml using jQuery-like syntax
AND LOTS MORE

How do we install packages from PyPi?

First, you need to install python-setuptools.

This depends on the operating system:

Ubuntu:

apt-get install python-setuptools

Windows: you have to download an .exe from here:

http://pypi.python.org/pypi/setuptools/0.6c11#windows

Python packages are generally distributed in an .egg file. These files are a kind of zip (with a different extension) that:

Have the source code
Have other common files (images, sounds, documentation, etc...)
Have an all too important metadata files: setup.py
To install .egg files you need to install first a module called setuptools.

Once setuptools has been installed, you'll have the easy_install command made available.

easy_install libraryName

For example:

easy_install django
easy_install yaml
easy_install pylons==1.0
easy_install -U rst2pdf

Something important to note is that the easy_install command also installs all the dependencies. For instance, rst2pdf depends on Pygments and reportlab. If you run:

easy_install rst2pdf

Then reportlab and Pygments will also be installed. For those using Ubuntu, easy_install works like apt-get.

There is an alternative to easy_install called pip. I'd recommend using that option.

easy_install pip

Pip can do all that easy_install did, and some more:

Searching in pypi

pip search rst2pdf

Uninstalling packages

pip uninstall rst2pdf

Listing installed libraries

pip freeze

Install all the packages in a file

pip freeze > archivoConLibrerias.txt
pip install -r archivoConLibrerias.txt

For example:

pip install django
pip install -U rst2pdf

However, there are a few problems:

You have to have admin/root privileges. It's easy when you're developing, but hardly viable in production.
You cannot install two versions of the same package.

This is very important when the package isn't backwards-compatible. Some examples are:
- SQLAlchemy
- django
For instance, applications that work wit django 0.9.7 won't work with django 1.2, and viceversa. Because of the way Python's package system works, it only allows one version of any package.

Therefore, one has to choose which version to use.

Using virtualenv

To solve these problems there is virtualenv. It takes care of creating an isolated Python environment within the system. First, you have to install it:

easy_install virtualenv

This is the only time you'll need to be root/admin to install a library. All the servers that allow hosting python projects have it preinstalled.

If you don't have root/administrator privileges, you can also use it without installing it into the system. For this, you can download the tarball from:

http://pypi.python.org/pypi/virtualenv

Once installed, the first thing to do is to create a virtual environment:

virtualenv environmentName

This will create a folder environmentName in the directory we're sitting. If you didn't install it into the system, if you downloaded the tarball, then use

python virtualenv.py environmentName

The previous command created a Python environment that is separate from the system's global configuration. Therefore, one does not have any permission problems there because one owns the tree.

To tell the operating system that we want to use the environment we just created, and not the system's, it is necessary to activate it.

Linux:

source environmentName/bin/activate

Windows:

environmentName\Scripts\activate.bat

Both in windows and in linux, when you activate an environment, you'll see the name of the activated environment between parenthesis:

(environmentName)tzulberti@myhost:~

When you have an active environment, you can see that the python you're using is not the global one, but the one inside a folder virtualenv created.

(environmentName)tzulberti@myhost:which python
/home/tzulberti/environmentName/bin/python

The same happens with easy_install and pip. Also, any packages installed will be installed within the virtual environment.

To stop using the environment we have to do the following:

Linux:

deactive

Windows:

environmentName\Scripts\deactivate.bat

How does virtualenv work?

When one creates an environment, inside it three folders are created:

bin: when one has the environment active, and executes a command like python, pip or easy_install, one of the binaries in this folder is ran, instead of the system's.
include: it's simply a link to the files installed with Python.
lib: this is another important folder. Just like include it has a folder named python, but in contrast to include, this one isn't a link to the files installed with Python. Instead, the folder has two important things:
- A link to some Python files. In this case, the .py
- A folder, site-packages, which is where packages are installed when one uses pip or easy_install

Q & A session (Part 1)

Before I commented that one of the problems we had was that we couldn't install two versions of the same library. What I never did is comment how this problem is solved.

The solution is easy. You create a virtual environment to use with django 0.97, and another for 1.2. You can create as many virtual environments as you wish, and they're isolated from each other.

For production servers, there are configurations that can be set up in apache for it to use some specific virtual environment.

Q & A session (Part 2)

I create a virtualenv, activate it, and do pip freeze, and lots of things that I never installed appear in the environment.

The default behavior of virtualenv when one creates an environment is to also create it with the packages one has installed in the system. For it not to do this, when you create the environment, you can do:

virtualenv --no-site-packages environmentName

This makes it create a completely empty environment. Lastly, another option to note is --python, which makes the virtual environment use that version of python (it is necessary to have installed that version for this to work). For instance, in my machine I have both Python 2.7 and 3.1, being 2.7 the default one.

Thus, when I create an environment, it will be created with Python 2.7. For it to use Python 3.1, I have to do the following:

virtualenv --python=python3.1 environmentName

Q & A session (Part 3)

There are some packages that have code written in C. What happens with those?

Well, in those cases pip or easy_install will try to build the C code when the package is installed. In distros like Ubuntu, I recommend installing:

build-essential
python-dev

Those two packages make it much easier to build.

However, in Windows it is not so easy. Several packages that have C code are distributed as an .exe too. However, those installers do not allow us to select where to install them. Here http://www.developerzen.com/2010/09/23/the-complete-guide-to-setting-up-python-development-environment-on-windows/ is a very good guide on how to make it build packages as it downloads it.

Q & A session (Part 4)

Do you recommend installing system packages or is it better to use easy_install?

There are various distros that come with various python packages that can be installed in the system. For example:

apt-get python-numpy python-reportlab python-pil

But the versions that can be installed with apt-get are old. Here we'll see some comparative examples:

Package name	Ubuntu 10.10 version	PyPi version
rst2pdf	0.14.2	0.16
reportlab	2.4.3	2.5
django	1.2.3	1.2.3

Thus, it depends on what one wants to do. For instance, if one wants to make a program work with Ubuntu, then one can create a virtual environment and install the same versions that apt-get has, or directly install system-supplied ones.

What I would NOT recommend is overwriting the installed version. For instance, if one does:

apt-get python-reportlab

Then you do NOT do:

easy_install -U reportlab

Not without using a virtual environment, because it could break the system.

Help PET: Donate

blog comments powered by Disqus

Last Change: Sat Jul 9 15:00:35 2011. - This magazine is under a Creative Commons license