How to call an external command?
How do you call an external command (as if I'd typed it at the Unix shell or Windows command prompt) from within a Python script?
Look at the subprocess module in the standard library:
import subprocess subprocess.run(["ls", "-l"])
The advantage of
system is that it is more flexible (you can get the
stderr, the "real" status code, better error handling, etc...).
The official documentation recommends the
subprocess module over the alternative
subprocessmodule provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function [
The Replacing Older Functions with the subprocess Module section in the
subprocess documentation may have some helpful recipes.
For versions of Python before 3.5, use
import subprocess subprocess.call(["ls", "-l"])
Here's a summary of the ways to call external programs and the advantages and disadvantages of each:
os.system("some_command with args")passes the command and arguments to your system's shell. This is nice because you can actually run multiple commands at once in this manner and set up pipes and input/output redirection. For example:
os.system("some_command < input_file | another_command > output_file")
However, while this is convenient, you have to manually handle the escaping of shell characters such as spaces, etc. On the other hand, this also lets you run commands which are simply shell commands and not actually external programs. See the documentation.
stream = os.popen("some_command with args")will do the same thing as
os.systemexcept that it gives you a file-like object that you can use to access standard input/output for that process. There are 3 other variants of popen that all handle the i/o slightly differently. If you pass everything as a string, then your command is passed to the shell; if you pass them as a list then you don't need to worry about escaping anything. See the documentation.
Popenclass of the
subprocessmodule. This is intended as a replacement for
os.popenbut has the downside of being slightly more complicated by virtue of being so comprehensive. For example, you'd say:
print subprocess.Popen("echo Hello World", shell=True, stdout=subprocess.PIPE).stdout.read()
print os.popen("echo Hello World").read()
but it is nice to have all of the options there in one unified class instead of 4 different popen functions. See the documentation.
callfunction from the
subprocessmodule. This is basically just like the
Popenclass and takes all of the same arguments, but it simply waits until the command completes and gives you the return code. For example:
return_code = subprocess.call("echo Hello World", shell=True)
See the documentation.
If you're on Python 3.5 or later, you can use the new
subprocess.runfunction, which is a lot like the above but even more flexible and returns a
CompletedProcessobject when the command finishes executing.
The os module also has all of the fork/exec/spawn functions that you'd have in a C program, but I don't recommend using them directly.
subprocess module should probably be what you use.
Finally please be aware that for all methods where you pass the final command to be executed by the shell as a string and you are responsible for escaping it. There are serious security implications if any part of the string that you pass can not be fully trusted. For example, if a user is entering some/any part of the string. If you are unsure, only use these methods with constants. To give you a hint of the implications consider this code:
print subprocess.Popen("echo %s " % user_input, stdout=PIPE).stdout.read()
and imagine that the user enters something "my mama didnt love me && rm -rf /" which could erase the whole filesystem.
Read more... Read less...
import subprocess p = subprocess.Popen('ls', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) for line in p.stdout.readlines(): print line, retval = p.wait()
You are free to do what you want with the
stdout data in the pipe. In fact, you can simply omit those parameters (
stderr=) and it'll behave like
Some hints on detaching the child process from the calling one (starting the child process in background).
Suppose you want to start a long task from a CGI script. That is, the child process should live longer than the CGI script execution process.
The classical example from the subprocess module documentation is:
import subprocess import sys # Some code here pid = subprocess.Popen([sys.executable, "longtask.py"]) # Call subprocess # Some more code here
The idea here is that you do not want to wait in the line 'call subprocess' until the longtask.py is finished. But it is not clear what happens after the line 'some more code here' from the example.
My target platform was FreeBSD, but the development was on Windows, so I faced the problem on Windows first.
On Windows (Windows XP), the parent process will not finish until the longtask.py has finished its work. It is not what you want in a CGI script. The problem is not specific to Python; in the PHP community the problems are the same.
The solution is to pass DETACHED_PROCESS Process Creation Flag to the underlying CreateProcess function in Windows API. If you happen to have installed pywin32, you can import the flag from the win32process module, otherwise you should define it yourself:
DETACHED_PROCESS = 0x00000008 pid = subprocess.Popen([sys.executable, "longtask.py"], creationflags=DETACHED_PROCESS).pid
/* UPD 2015.10.27 @eryksun in a comment below notes, that the semantically correct flag is CREATE_NEW_CONSOLE (0x00000010) */
On FreeBSD we have another problem: when the parent process is finished, it finishes the child processes as well. And that is not what you want in a CGI script either. Some experiments showed that the problem seemed to be in sharing sys.stdout. And the working solution was the following:
pid = subprocess.Popen([sys.executable, "longtask.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
I have not checked the code on other platforms and do not know the reasons of the behaviour on FreeBSD. If anyone knows, please share your ideas. Googling on starting background processes in Python does not shed any light yet.
import os os.system("your command")
Note that this is dangerous, since the command isn't cleaned. I leave it up to you to google for the relevant documentation on the 'os' and 'sys' modules. There are a bunch of functions (exec* and spawn*) that will do similar things.