Running OS Commands

Let us understand how to run OS commands using Python using libraries such as os and subprocess.

  • Python provides several libraries which can be used to run OS commands. os and subprocess are most popular ones.

  • We can import the libraries such as os and subprocess to start using them.

  • There are bunch of commands to create directories, change ownership, change permission, run general system commands etc.

  • os library is extensively used to read environment variables at run time of the application. It is used to pass keys and credentials to work with databases, external applications etc.

  • Typically keys and credentials should not be part of the source code.

  • subprocess can be used to run the commands and also to process the output.

import os

  • Get current working directory.

os.getcwd()

'/home/itversity/itversity-material/mastering-python/06_basic_programming_constructs'

  • Read environment variables

os.environ.get('PATH')

'/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/anaconda3/envs/beakerx/bin'

os.environ.get('USER')

'itversity'

os.environ.get('HOME')

'/home/itversity'

%%sh

env

USER=itversity
MPLBACKEND=module://ipykernel.pylab.backend_inline
JUPYTERHUB_HOST=
JUPYTERHUB_USER=itversity
HOME=/home/itversity
JUPYTERHUB_OAUTH_CALLBACK_URL=/jupyter/user/itversity/oauth_callback
JUPYTERHUB_API_URL=http://127.0.0.1:8081/jupyter/hub/api
BEAKERX_AUTOTRANSLATION_PORT=40427
PAGER=cat
JUPYTERHUB_CLIENT_ID=jupyterhub-user-itversity
TERM=xterm-color
PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/anaconda3/envs/beakerx/bin
JUPYTERHUB_ACTIVITY_URL=http://127.0.0.1:8081/jupyter/hub/api/users/itversity/activity
LANG=C.UTF-8
SHELL=/bin/bash
JUPYTERHUB_SERVICE_PREFIX=/jupyter/user/itversity/
GIT_PAGER=cat
JUPYTERHUB_API_TOKEN=ba11bf5bb6cd4f779527e753f05574de
PWD=/home/itversity/itversity-material/mastering-python/06_basic_programming_constructs
CLICOLOR=1
JUPYTERHUB_SERVER_NAME=
JUPYTERHUB_BASE_URL=/jupyter/
JPY_PARENT_PID=3315
JPY_API_TOKEN=ba11bf5bb6cd4f779527e753f05574de
BEAKERX_AUTOTRANSLATION_PASSWORD=aHI0e0jNCHRxg6YP7pnvG9fMl4ZdkqAem30nCUOvGJi5aKMq4xFY9RZ8d1mH27CWS4gCRWxDTpLdzfaiMhBS58cMMKA4W0ZMjktRxtAn8oZNRTmku7DzZ0sSB1bmceWA

os.environ.get?

Signature: os.environ.get(key, default=None)
Docstring: D.get(k[,d]) -> D[k] if k in D, else d.  d defaults to None.
File:      /opt/anaconda3/envs/beakerx/lib/python3.6/_collections_abc.py
Type:      method

os.environ.get('PASSWORD', 'Passwords should be confidential')

'Passwords should be confidential'

  • Run ls -ltr command to get list of files in the current directory.

import subprocess

%%sh

ls -ltr

total 72
-rw-rw-r-- 1 itversity itversity  3241 Dec  4 23:34 06_comments_and_doc_strings.ipynb
-rw-rw-r-- 1 itversity itversity  1070 Dec  4 23:34 04_data_types_commonly_used.ipynb
-rw-rw-r-- 1 itversity itversity  2638 Dec  4 23:34 03_variables_and_objects.ipynb
-rw-rw-r-- 1 itversity itversity  1532 Dec  4 23:34 02_getting_help.ipynb
-rw-rw-r-- 1 itversity itversity   917 Dec  4 23:34 01_basic_programming_constructs.ipynb
-rw-rw-r-- 1 itversity itversity  8533 Dec 10 15:43 05_operators_in_python.ipynb
-rw-rw-r-- 1 itversity itversity  8050 Dec 16 04:25 07_conditionals.ipynb
-rw-rw-r-- 1 itversity itversity 12419 Dec 16 05:13 08_all_about_for_loops.ipynb
-rw-rw-r-- 1 itversity itversity  2769 Dec 16 05:14 10_exercises.ipynb
-rw-rw-r-- 1 itversity itversity  8469 Dec 16 05:20 09_running_os_commands.ipynb

output = subprocess.check_call(['ls', '-ltr'])

output

0

output = subprocess.check_output(['ls', '-ltr'])

output # output is of type binary

b'total 72\n-rw-rw-r-- 1 itversity itversity  3241 Dec  4 23:34 06_comments_and_doc_strings.ipynb\n-rw-rw-r-- 1 itversity itversity  1070 Dec  4 23:34 04_data_types_commonly_used.ipynb\n-rw-rw-r-- 1 itversity itversity  2638 Dec  4 23:34 03_variables_and_objects.ipynb\n-rw-rw-r-- 1 itversity itversity  1532 Dec  4 23:34 02_getting_help.ipynb\n-rw-rw-r-- 1 itversity itversity   917 Dec  4 23:34 01_basic_programming_constructs.ipynb\n-rw-rw-r-- 1 itversity itversity  8533 Dec 10 15:43 05_operators_in_python.ipynb\n-rw-rw-r-- 1 itversity itversity  8050 Dec 16 04:25 07_conditionals.ipynb\n-rw-rw-r-- 1 itversity itversity 12419 Dec 16 05:13 08_all_about_for_loops.ipynb\n-rw-rw-r-- 1 itversity itversity  2769 Dec 16 05:14 10_exercises.ipynb\n-rw-rw-r-- 1 itversity itversity  9956 Dec 16 05:22 09_running_os_commands.ipynb\n'

type(output)

bytes

output.decode('utf-8') # converts to string of type utf-8

'total 72\n-rw-rw-r-- 1 itversity itversity  3241 Dec  4 23:34 06_comments_and_doc_strings.ipynb\n-rw-rw-r-- 1 itversity itversity  1070 Dec  4 23:34 04_data_types_commonly_used.ipynb\n-rw-rw-r-- 1 itversity itversity  2638 Dec  4 23:34 03_variables_and_objects.ipynb\n-rw-rw-r-- 1 itversity itversity  1532 Dec  4 23:34 02_getting_help.ipynb\n-rw-rw-r-- 1 itversity itversity   917 Dec  4 23:34 01_basic_programming_constructs.ipynb\n-rw-rw-r-- 1 itversity itversity  8533 Dec 10 15:43 05_operators_in_python.ipynb\n-rw-rw-r-- 1 itversity itversity  8050 Dec 16 04:25 07_conditionals.ipynb\n-rw-rw-r-- 1 itversity itversity 12419 Dec 16 05:13 08_all_about_for_loops.ipynb\n-rw-rw-r-- 1 itversity itversity  2769 Dec 16 05:14 10_exercises.ipynb\n-rw-rw-r-- 1 itversity itversity  9956 Dec 16 05:22 09_running_os_commands.ipynb\n'

type(output.decode('utf-8'))

str

Note

Let us convert string into list of strings. Once it is broken into list of strings we can process the data as per our requirements either by using Map Reduce libraries or Pandas based libraries.

output.decode('utf-8').splitlines()

['total 72',
 '-rw-rw-r-- 1 itversity itversity  3241 Dec  4 23:34 06_comments_and_doc_strings.ipynb',
 '-rw-rw-r-- 1 itversity itversity  1070 Dec  4 23:34 04_data_types_commonly_used.ipynb',
 '-rw-rw-r-- 1 itversity itversity  2638 Dec  4 23:34 03_variables_and_objects.ipynb',
 '-rw-rw-r-- 1 itversity itversity  1532 Dec  4 23:34 02_getting_help.ipynb',
 '-rw-rw-r-- 1 itversity itversity   917 Dec  4 23:34 01_basic_programming_constructs.ipynb',
 '-rw-rw-r-- 1 itversity itversity  8533 Dec 10 15:43 05_operators_in_python.ipynb',
 '-rw-rw-r-- 1 itversity itversity  8050 Dec 16 04:25 07_conditionals.ipynb',
 '-rw-rw-r-- 1 itversity itversity 12419 Dec 16 05:13 08_all_about_for_loops.ipynb',
 '-rw-rw-r-- 1 itversity itversity  2769 Dec 16 05:14 10_exercises.ipynb',
 '-rw-rw-r-- 1 itversity itversity  9956 Dec 16 05:22 09_running_os_commands.ipynb']

type(output.decode('utf-8').splitlines())

list

# splitlines is the function available on string type
# It converts string with line breaks into list of strings
for rec in output.decode('utf-8').splitlines():
    print(rec)

total 72
-rw-rw-r-- 1 itversity itversity  3241 Dec  4 23:34 06_comments_and_doc_strings.ipynb
-rw-rw-r-- 1 itversity itversity  1070 Dec  4 23:34 04_data_types_commonly_used.ipynb
-rw-rw-r-- 1 itversity itversity  2638 Dec  4 23:34 03_variables_and_objects.ipynb
-rw-rw-r-- 1 itversity itversity  1532 Dec  4 23:34 02_getting_help.ipynb
-rw-rw-r-- 1 itversity itversity   917 Dec  4 23:34 01_basic_programming_constructs.ipynb
-rw-rw-r-- 1 itversity itversity  8533 Dec 10 15:43 05_operators_in_python.ipynb
-rw-rw-r-- 1 itversity itversity  8050 Dec 16 04:25 07_conditionals.ipynb
-rw-rw-r-- 1 itversity itversity 12419 Dec 16 05:13 08_all_about_for_loops.ipynb
-rw-rw-r-- 1 itversity itversity  2769 Dec 16 05:14 10_exercises.ipynb
-rw-rw-r-- 1 itversity itversity  9956 Dec 16 05:22 09_running_os_commands.ipynb
All about for Loops Exercises - Basic Programming Constructs