Exercise 46. A Project Skeleton
This will be where you start learning how to set up a good project “skeleton” directory. This skeleton directory will have all the basics you need to get a new project up and running. It will have your project layout, automated tests, modules, and install scripts. When you go to make a new project, just copy this directory to a new name and edit the files to get started.
macOS/Linux Setup
Before you can begin this exercise you need to install some software for Python by using a tool called pip3.6 (or just pip) to install new modules. The pip3.6 command should be included with your python3.6 installation. You’ll want to verify this using this command:
$ pip3.6 list
pip (9.0.1)
setuptools (28.8.0)
$
You can ignore any deprecation warning if you see it. You may also see other tools installed, but the base should be pip and setuptools. Once you’ve verified this you can then install virtualenv:
$ sudo pip3.6 install virtualenv
Password:
Collecting virtualenv
Downloading virtualenv–15.1.0–py2.py3–none–any.whl (1.8MB)
100% ||||||||||||||||||||||||||||||||| 1.8MB 1.1MB/s
Installing collected packages: virtualenv
Successfully installed virtualenv–15.1.0
$
This is for Linux or macOS systems. If you’re on Linux/macOS, you’ll want to run the following command to make sure you use the correct virtualenv:
$ whereis virtualenv
/Library/Frameworks/Python.framework/Versions/3.6/bin/virtualenv
You should see something like the above on macOS, but Linux will be variable. On Linux you might have an actual virtualenv3.6 command, or you may be better off installing a package for it from your package management system.
Once you’ve installed virtualenv you can use it to create a “fake” Python installation, which makes it easier to manage versions of your packages for different projects. First, run this command, and I’ll explain what it’s doing:
$ mkdir ~/.venvs
$ virtualenv ––system–site–packages ~/.venvs/lpthw
$ . ~/.venvs/lpthw/bin/activate
(lpthw) $
Here’s what’s going on line by line:
1. You create a directory called .venvs in your HOME ~/ to store all your virtual environments.
2. You run virtualenv and tell it to include the system site packages (--system-site-packages), then instruct it to build the virtualenv in ~/.venvs/lpthw.
3. You then “source” the lpthw virtual environment by using the . operator in bash, followed by the ~/.venvs/lpthw/bin/activate script.
4. Finally, your prompt changes to include (lpthw), so you know that you’re using that virtual environment.
Now you can see where things are installed:
(lpthw) $ which python
/Users/zedshaw/.venvs/lpthw/bin/python
(lpthw) $ python
Python 3.6.0rc2 (v3.6.0rc2:800a67f7806d, Dec 16 2016, 14:12:21)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
(lpthw) $
You can see that the python that gets run is installed in the /Users/zedshaw/.venvs/lpthw/bin/ python directory instead of the original location. This also solves the problem of having to type python3.6 since it installs both:
$ which python3.6
/Users/zedshaw/.venvs/lpthw/bin/python3.6
(lpthw) $
You’ll find the same thing for virtualenv and pip commands. The final step in this setup is to install nose, a testing framework we’ll use in this exercise:
$ pip install nose
Collecting nose
Downloading nose–1.3.7–py3–none–any.whl (154kB)
100% |||||||||||||||||||||||||||||||||| 163kB 3.2MB/s
Installing collected packages: nose
Successfully installed nose–1.3.7
(lpthw) $
Windows 10 Setup
The Windows 10 install is a little simpler than on Linux or macOS, but only if you have one version of Python installed. If you have both Python 3.6 and Python 2.7 installed then you are on your own as it is much too difficult to manage multiple installations. If you have followed the book so far and have only Python 3.6, then here’s what you do. First, change to your home directory and confirm you’re running the right version of python:
> cd ~
> python
Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12)
[MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
Then you’ll want to run pip to confirm you have a basic install:
> pip list
pip (9.0.1)
setuptools (28.8.0)
You can safely ignore any deprecation warning, and it’s alright if you have other packages installed. Next, you’ll install virtualenv for setting up simple virtual environments for the rest of the book:
> pip install virtualenv
Collecting virtualenv
Using cached virtualenv–15.1.0–py2.py3–none–any.whl
Installing collected packages: virtualenv
Successfully installed virtualenv–15.1.0
Once you have virtualenv installed you’ll need to make a .venvs directory and fill it with a virtual environment:
> mkdir .venvs
> virtualenv ––system–site–packages .venvs/lpthw
Using base prefix
'c:\users\zedsh\appdata\local\programs\python\python36'
New python executable in
C:Userszedshaw.venvslpthwScriptspython.exe
Installing setuptools, pip, wheel...done.
Those two commands create a .venvs folder for storing different virtual environments and then create your first one named lpthw. A virtual environment (virtualenv) is a “fake” place to install software so that you can have different versions of different packages for each project you’re working on. Once you have the virtualenv set up you need to activate it:
> ..venvslpthwScriptsactivate
That will run the activate script for PowerShell, which configures the lpthw virtualenv for your current shell. Every time you want to use your software for the book you’ll run this command. You’ll notice in our next command that there is now a (lpthw) added to the PowerShell prompt showing you which virtualenv you’re using. Finally, you just need to install nose for running tests later:
(lpthw) > pip install nose
Collecting nose
Downloading nose–1.3.7–py3–none–any.whl (154kB)
100% |||||||||||||||||||||||||||||||||| 163kB 1.2MB/s
Installing collected packages: nose
Successfully installed nose–1.3.7
(lpthw) >
You’ll see that this installs nose, except pip will install it into your .venvslpthw virtual environment instead of the main system packages directory. This lets you install conflicting versions of Python packages for each project you work on without infecting your main system configuration.
Creating the Skeleton Project Directory
First, create the structure of your skeleton directory with these commands:
$ mkdir projects
$ cd projects/
$ mkdir skeleton
$ cd skeleton
$ mkdir bin NAME tests docs
I use a directory named projects to store all the various things I’m working on. Inside that directory I have my skeleton directory that I put the basis of my projects into. The directory NAME will be renamed to whatever you are calling your project’s main module when you use the skeleton.
Next, we need to set up some initial files. Here’s how you do that on Linux/macOS:
$ touch NAME/__init__.py
$ touch tests/__init__.py
Here’s the same thing on Windows PowerShell:
$ new–item –type file NAME/__init__.py
$ new–item –type file tests/__init__.py
That creates an empty Python module directory we can put our code in. Then we need to create a setup.py file we can use to install our project later if we want:
setup.py
1 try:
2 from setuptools import setup
3 except ImportError:
4 from distutils.core import setup
5
6 config = {
7 'description': 'My Project',
8 'author': 'My Name',
9 'url': 'URL to get it at.',
10 'download_url': 'Where to download it.',
11 'author_email': 'My email.',
12 'version': '0.1',
13 'install_requires': ['nose'],
14 'packages': ['NAME'],
15 'scripts': [],
16 'name': 'projectname'
17 }
18
19 setup(**config)
Edit this file so that it has your contact information and is ready to go for when you copy it.
Finally, you will want a simple skeleton file for tests named tests/NAME_tests.py:
NAME_tests.py
1 from nose.tools import *
2 import NAME
3
4 def setup():
5 print("SETUP!")
6
7 def teardown():
8 print("TEAR DOWN!")
9
10 def test_basic():
11 print("I RAN!")
Final Directory Structure
When you are done setting all of this up, your directory should look like mine here:
skeleton/
NAME/
__init__.py
bin/
docs/
setup.py
tests/
NAME_tests.py
__init__.py
And from now on, you should run your commands from this directory. If you can’t, do ls -R and if you don’t see this same structure, then you are in the wrong place. For example, people commonly go into the tests/ directory to try to run files there, which won’t work. To run your application’s tests, you would need to be above tests/ and this location I have above. So, if you try this:
$ cd tests/ # WRONG! WRONG! WRONG!
$ nosetests
–––––––––––––––––––––––––––––––––––––––––––––––––––––––––
Ran 0 tests in 0.000s
OK
That is wrong! You have to be above tests/, so assuming you made this mistake, you would fix it by doing this:
$ cd .. # get out of tests/
$ ls # CORRECT! you are now in the right spot
NAME bin docs setup.py tests
$ nosetests
.
––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
Ran 1 test in 0.004s
OK
Remember this because people make this mistake quite frequently.
Warning!
At the time of publication I learned that the nose project has been abandoned and might not work well. If you have strange syntax errors when you run nosetests, then look at the error output. If it references “python2.7” in the output, then chances are nosetests is trying to run the 2.7 version of Python on your computer. The solution is to run nose using python3.6 -m "nose" on Linux or macOS. On Windows you may not have this problem, but using python -m "nose" will solve it if you do.
$ nosetests
.
––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
Ran 1 test in 0.007s
OK
I’ll explain what this nosetests thing is doing in the next exercise, but for now if you do not see that, you probably got something wrong. Make sure you put __init__.py files in your NAME and tests directories, and make sure you got tests/NAME_tests.py right.
Using the Skeleton
You are now done with most of your yak shaving. Whenever you want to start a new project, just do this:
1. Make a copy of your skeleton directory. Name it after your new project.
2. Rename (move) the NAME directory to be the name of your project or whatever you want to call your root module.
3. Edit your setup.py to have all the information for your project.
4. Rename tests/NAME_tests.py to also have your module name.
5. Double check it’s all working by using nosetests again.
6. Start coding.
Required Quiz
This exercise doesn’t have Study Drills; instead, here’s quiz you should complete:
1. Read about how to use all of the things you installed.
2. Read about the setup.py file and all it has to offer. Warning: It is not a very well-written piece of software, so it will be very strange to use.
3. Make a project and start putting code into the module, then get the module working.
4. Put a script in the bin directory that you can run. Read about how you can make a Python script that’s runnable for your system.
5. Mention the bin script you created in your setup.py so that it gets installed.
6. Use your setup.py to install your own module and make sure it works, then use pip to uninstall it.
Common Student Questions
Do these instructions work on Windows? They should, but depending on the version of Windows you may need to struggle with the setup a bit to get it working. Just keep researching and trying it until you get it, or see if you can ask a more experienced Python+Windows friend to help out.
What do I put in the config dictionary in my setup.py? Make sure you read the documentation for distutils at http://docs.python.org/distutils/setupscript.html.
I can’t seem to load the NAME module and just get an ImportError. Make sure that you made the NAME/__init__.py file. If you’re on Windows, make sure you didn’t accidentally name it NAME/__init__.py.txt, which happens by default with some editors.
Why do we need a bin/ folder at all? This is just a standard place to put scripts that are run on the command line, not a place to put modules.
My nosetests run only shows one test being run. Is that right? Yes, that’s what my output shows, too.