2. Python and the Shell

By Bernd Klein. Last modified: 06 Apr 2022.

Shell

Shell is a term, which is often used and often misunderstood. Like the shell of an egg, either hen or Python snake, or a mussel, the shell in computer science is generally seen as a piece of software that provides an interface for a user to some other software or the operating system. So the shell can be an interface between the operating system and the services of the kernel of this operating system. But a web browser or a program functioning as an email client can be seen as shell as well.

Understanding this, it's obvious that a shell can be either

• a command-line interface (CLI) or
• a graphical user interface (GUI)

But in most cases the term shell is used as a synonym for a command line interface (CLI). The best known and most often used shells under Linux and Unix are the Bourne-Shell, C-Shell or Bash shell. The Bourne shell (sh) was modelled after the Multics shell, and is the first Unix shell. Most operating system shells can be used in both interactive and batch mode.

Live Python training

Enjoying this page? We offer live Python training courses covering the content of this site.

See: Live Python courses overview

Enrol here

System programming and Python

System programming (also known as systems programming) stands for the activity of programming system components or system software. System programming provides software or services to the computer hardware, while application programming produces software which provides tools or services for the user.

"System focused programming" serves as an abstraction layer between the application, i.e. the Python script or program, and the operating system, e.g. Linux or Microsoft Windows. By means of such an abstraction layer it is possible to implement platform independent applications in Python, even if they access operating specific functionalities. Python provides various modules to interact with the operating system, such as

• os
• platform
• subprocess
• shutils
• glob
• sys

Therefore Python is well suited for system programming, or even platform independent system programming. This is one of the reasons by the way why Python is wrongly considered by many as a scripting language. The proper way to say it: Python is a full-fledged programming language which can be easily used as a scripting language. The general advantages of Python are valid in system focused programming as well:

• simple and clear
• wel structured
• highly flexible

os Module

The os module is the most important module for interacting with the operating system. The os module allows platform independent programming by providing abstract methods. Nevertheless it is also possible by using the system() and the exec() function families to include system independent program parts. (Remark: The exec()-Functions are introduced in detail in our chapter "Forks and Forking in Python" The os module provides various methods, e.g. the access to the file system.

Platform independant application often need to know on which platform the program or script is running. The os module provides with os.name the perfect command for this purpose:

import os

print(os.name)

posix

The output is of course dependant on the the operating system you are running. As of Python version 3.8 the following names are registered: posix, nt, java.

These names define the following operating systems:

posix

Unix-like operating systems like Unix, Linux, BSD, Minix and others.

nt

Windows systems like "Windows 10", "Windows 8.1", "Windows 8", "Windows 7" and so on.

java

Java operating system.

Most Python scripts dealing with the operating system need to know the postion in the file system. The function os.getcwd() returns a unicode string representing the current working directory.

print(os.getcwd())

/home/bernd/Dropbox (Bodenseo)/notebooks/advanced_topics

You also need the capability to move around in your filesystem. For this purpose the module os provides the function chdir. It has a parameter path, which is specified as a string. If called it will change the current working directory to the specified path.

os.chdir("/home/bernd/dropbox/websites/python-course.eu/cities")
print(os.getcwd())

/home/bernd/Dropbox (Bodenseo)/websites/python-course.eu/cities

After you have reached the desired directory, you may want to get its content. The function listdir returns a list containing the names of the files of this directory.

print(os.listdir())

['Freiburg.png', 'Stuttgart.png', 'Hamburg.png', 'Berlin.png', 'Basel.png', 'Nürnberg.png', 'Erlangen.png', 'index.html', 'Ulm.png', 'Singen.png', 'Bremen.png', 'Kassel.png', 'Frankfurt.png', 'Saarbrücken.png', 'Zürich.png', 'Konstanz.png', 'Hannover.png']

Live Python training

Enjoying this page? We offer live Python training courses covering the content of this site.

See: Live Python courses overview

Enrol here

Executing Shell scripts with os.system()

It's not possible in Python to read a character without having to type the return key as well. On the other hand this is very easy on the Bash shell. The Bash command read -n 1 waits for a key (any key) to be typed. If you import os, it's easy to write a script providing getch() by using os.system() and the Bash shell. getch() waits just for one character to be typed without a return:

import os
def getch():
     os.system("bash -c \"read -n 1\"")
 
getch()

The script above works only under Linux. Under Windows you will have to import the module msvcrt. Pricipially we only have to import getch() from this module. So this is the Windows solution of the problem:

from msvcrt import getch

The following script implements a platform independent solution to the problem:

import os, platform
if platform.system() == "Windows":
    import msvcrt
def getch():
    if platform.system() == "Linux":
        os.system('bash -c "read -n 1"')
    else:
        msvcrt.getch()

print("Type a key!")
getch()
print("Okay")

Type a key!
Okay

The previous script harbours a problem. You can't use the getch() function, if you are interested in the key which has been typed, because os.system() doesn't return the result of the called shell commands.

We show in the following script, how we can execute shell scripts and return the output of these scripts into python by using os.popen():

import os
dir = os.popen("ls").readlines()

print(dir)

['Basel.png\n', 'Berlin.png\n', 'Bremen.png\n', 'Erlangen.png\n', 'Frankfurt.png\n', 'Freiburg.png\n', 'Hamburg.png\n', 'Hannover.png\n', 'index.html\n', 'Kassel.png\n', 'Konstanz.png\n', 'Nürnberg.png\n', 'Saarbrücken.png\n', 'Singen.png\n', 'Stuttgart.png\n', 'Ulm.png\n', 'Zürich.png\n']

The output of the shell script can be read line by line, as can be seen in the following example:

import os

command = " "
while (command != "exit"):
    command = input("Command: ")
    handle = os.popen(command)
    line = " "
    while line:
        line = handle.read()
        print(line)
    handle.close()

print("Ciao that's it!")

Basel.png
Berlin.png
Bremen.png
Erlangen.png
Frankfurt.png
Freiburg.png
Hamburg.png
Hannover.png
index.html
Kassel.png
Konstanz.png
Nürnberg.png
Saarbrücken.png
Singen.png
Stuttgart.png
Ulm.png
Zürich.png



Ciao that's it!

subprocess Module

The subprocess module is available since Python 2.4. It's possible to create spawn processes with the module subprocess, connect to their input, output, and error pipes, and obtain their return codes. The module subprocess was created to replace various other modules:

• os.system
• os.spawn*
• os.popen*
• popen2.*
• commands.*

Live Python training

Enjoying this page? We offer live Python training courses covering the content of this site.

See: Live Python courses overview

Enrol here

Working with the subprocess Module

Instead of using the system method of the os-Module

os.system('Konstanz.png')

32256

we can use the Popen() command of the subprocess Module. By using Popen() we are capable to get the output of the script:

import subprocess
x = subprocess.Popen(['touch', 'xyz'])
print(x)
print(x.poll())
print(x.returncode)

<subprocess.Popen object at 0x7f92c5319290>
None
None

The shell command cp -r xyz abc can be send to the shell from Python by using the Popen() method of the subprocess-Module in the following way:

p = subprocess.Popen(['cp','-r', "Zürich.png", "Zurich.png"])

There is no need to escape the Shell metacharacters like $, >, ... and so on. If you want to emulate the behaviour of `os.system, the optional parameter shell has to be set to True and we have to use a string instead of a list:

p = subprocess.Popen("cp -r xyz abc", shell=True)

As we have mentioned above, it is also possible to catch the output from the shell command or shell script into Python. To do this, we have to set the optional parameter stdout of Popen() to subprocess.PIPE:

process = subprocess.Popen(['ls','-l'], stdout=subprocess.PIPE)
directory_content = process.stdout.readlines()
print(directory_content)

[b'total 0\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:32 abc\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Basel.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Berlin.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Bremen.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Erlangen.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Frankfurt.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Freiburg.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Hamburg.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Hannover.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:05 index.html\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Kassel.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Konstanz.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 N\xc3\xbcrnberg.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Saarbr\xc3\xbccken.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Singen.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Stuttgart.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Ulm.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:28 xyz\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:29 Zurich.png\n', b'-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Z\xc3\xbcrich.png\n']

The b indicates that what you have is bytes, which is a binary sequence of bytes rather than a string of Unicode characters. Subprocesses output bytes, not characters. We can turn it to unicode string with the following list comprehension:

directory_content = [el.decode() for el in directory_content]
print(directory_content)

['total 0\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:32 abc\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Basel.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Berlin.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Bremen.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Erlangen.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Frankfurt.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Freiburg.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Hamburg.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Hannover.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:05 index.html\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Kassel.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Konstanz.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Nürnberg.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Saarbrücken.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Singen.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Stuttgart.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Ulm.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:28 xyz\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:29 Zurich.png\n', '-rw-r--r-- 1 bernd bernd 0 Jan 23 08:07 Zürich.png\n']

Functions to manipulate paths, files and directories

>>> os.chdir("c:\Windows")
>>> os.getcwd()
'c:\\Windows'

>>> import os
>>> os.getcwd()
'/home/homer'
>>> os.chdir("/home/lisa")
>>> os.getcwd()
'/home/lisa'
>>> 

>>> os.listdir("/home/homer")
['.gnome2', '.pulse', '.gconf', '.gconfd', '.beagle', '.gnome2_private', '.gksu.lock', 'Public', '.ICEauthority', '.bash_history', '.compiz', '.gvfs', '.update-notifier', '.cache', 'Desktop', 'Videos', '.profile', '.config', '.esd_auth', '.viminfo', '.sudo_as_admin_successful', 'mbox', '.xsession-errors', '.bashrc', 'Music', '.dbus', '.local', '.gstreamer-0.10', 'Documents', '.gtk-bookmarks', 'Downloads', 'Pictures', '.pulse-cookie', '.nautilus', 'examples.desktop', 'Templates', '.bash_logout']
>>> 

Further function and methods working on files and directories can be found in the module shutil. Amongst other possibilities it provides the possibility to copy files and directories with shutil.copyfile(src,dst).

Live Python training

Enjoying this page? We offer live Python training courses covering the content of this site.

See: Live Python courses overview

Enrol here