I have been asked to write a script that pulls the latest code from Git, makes a build, and performs some automated unit tests.
I found that there are two built-in Python modules for interacting with Git that are readily available: GitPython and libgit2.
What approach/module should I use?
An easier solution would be to use the Python subprocess module to call git. In your case, this would pull the latest code and build:
import subprocess
subprocess.call(["git", "pull"])
subprocess.call(["make"])
subprocess.call(["make", "test"])
Docs:
subprocess - Python 2.x
subprocess - Python 3.x
I agree with Ian Wetherbee. You should use subprocess to call git directly. If you need to perform some logic on the output of the commands then you would use the following subprocess call format.
import subprocess
PIPE = subprocess.PIPE
branch = 'my_branch'
process = subprocess.Popen(['git', 'pull', branch], stdout=PIPE, stderr=PIPE)
stdoutput, stderroutput = process.communicate()
if 'fatal' in stdoutput:
# Handle error case
else:
# Success!
So with Python 3.5 and later, the .call() method has been deprecated.
https://docs.python.org/3.6/library/subprocess.html#older-high-level-api
The current recommended method is to use the .run() method on subprocess.
import subprocess
subprocess.run(["git", "pull"])
subprocess.run(["make"])
subprocess.run(["make", "test"])
Adding this as when I went to read the docs, the links above contradicted the accepted answer and I had to do some research. Adding my 2 cents to hopefully save someone else a bit of time.
In EasyBuild, we rely on GitPython, and that's working out fine.
See here, for examples of how to use it.
If GitPython package doesn't work for you there are also the PyGit and Dulwich packages. These can be easily installed through pip.
But, I have personally just used the subprocess calls. Works perfect for what I needed, which was just basic git calls. For something more advanced, I'd recommend a git package.
I had to use shlex on top of the run call because my command was too complex for the subprocess alone to understand.
import subprocess
import shlex
git_command = "git <command>"
subprocess.run(shlex.split(git_command))
If you're on Linux or Mac, why use python at all for this task? Write a shell script.
#!/bin/sh
set -e
git pull
make
./your_test #change this line to actually launch the thing that does your test
Related
I'm trying to convert a video between two file types, when run, nothing happens and no file is made. I've tried doing it with subprocess and os - both have the same result - nothing.
I can do the command fine through shell. I really want to be able to use this through python.
import subprocess
command = "ffmpeg -i X:/Desktop/twd.mp4 X:/Desktop/twd.mp3"
subprocess.run(command.split(),shell=True)
nothing happens and no file is made
This is very strange. Absolutely no text printed on console? I agree #Rotem's suggestion, but at the minimum the version info should print on your console (assuming you are using one of prebuilt binaries). BTW, shell=True is not needed and not recommended.
If you wish, you can give my ffmpegio package a try, it might make your life a bit easier.
import ffmpegio
ffmpegio.transcode('X:/Desktop/twd.mp4','X:/Desktop/twd.mp3')
It should auto detect your ffmpeg binaries and run your example above.
action_publisher = subprocess.Popen(
["bash", "-c", "/opt/ros/melodic/bin/rostopic pub -r 20 /robot_operation std_msgs/String start"],
env={'ROS_MASTER_URI': 'http://10.42.0.49:11311\''})
I tried to run it shell=True and shell=False. Also calling it with bash or just running my executable and I am always getting an error:
Traceback (most recent call last):
File "/opt/ros/melodic/bin/rostopic", line 34, in <module>
import rostopic
ImportError: No module named rostopic
How can I make a call of a shell executable with open through python removing this issue? Tried all combination possible and also other stack proposed solution and still, it tries to import the executable instead of running it on a shell.
I can identify several problems with your attempt, but I'm not sure I have identified them all.
You should use subprocess.check_call or subprocess.run if you just want the subprocess to run, and your Python script to wait for that to complete. If you need to use raw subprocess.Popen(), there are several additional required steps of plumbing which you need to do yourself, which check_call or run will perform for you if you use these higher-level functions.
Your use of env will replace the variables in the environment. Copy the existing environment instead so you don't clobber useful settings like PYTHONPATH etc which may well be preventing the subprocess from finding the library it needs.
The shell invocation seems superfluous.
The stray escaped single quote at the end of 'http://10.42.0.49:11311\'' definitely looks wrong.
With that, try this code instead; but please follow up with better diagnostics if this does not yet solve your problem completely.
import subprocess
import os
# ...
env = os.environ.copy()
env['ROS_MASTER_URI'] = 'http://10.42.0.49:11311'
action_publisher = subprocess.run(
["/opt/ros/melodic/bin/rostopic", "pub", "-r", "20",
"/robot_operation", "std_msgs/String", "start"],
env=env, check=True)
If rostopic is actually a Python program, a better solution altogether might be to import it directly instead.
It sounds like the code of that file is trying to import the Python module. If you're getting that import error even when you try to execute the file in bash/from a shell, then it has nothing to do with subprocess.Popen.
From the traceback, it looks like it's a Python file itself and it's trying to import that module, which would explain why you see the issue when executing it from a shell.
Did you go through the installation correctly, specifically the environment setup? http://wiki.ros.org/melodic/Installation/Ubuntu#melodic.2FInstallation.2FDebEnvironment.Environment_setup
It sounds like you need to source a particular file to have the correct paths available where the Python module is located so it can be found when you execute the script.
As far as I can see, your .Popen command will try to execute
bash -c /opt/ros/melodic/bin/rostopic pub -r 20 /robot_operation std_msgs/String start
while bash -c has to be followed by a string. Thus, you may need to add single quotes.
I'm calling another program with a piece of code that looks like this:
import subprocess
lc="/package/bin/program --do stuff"
command_list = lc.split()
ljs=subprocess.Popen(command_list,stdout=subprocess.PIPE)
ljs.communicate()[0]
The string works fine at the UNIX command line and the code works in Python 2.7. But, in Python 3.4, I get an error like this:
File "/package/bin/program", line 2, in <module>
from package import module
ImportError: No module named package
"/package/bin/program" is calling a dependency from another file in the package here, which I think is the core issue. I have calls to other programs that are working fine in 3.4.
What's changed in 3.4 that might be causing this?
(Sorry in advance for the cryptic code - I'm calling company internal tools that I can't expose here).
The problem is that the working directory of the subproccess instance is default the directory of bash shell. To set a new working directory, set the cwd argument in your Popen to your working directory.
Here's an example:
subprocess.Popen(['random' '--command'], stdout = subprocess.PIPE, cwd='C:/Path/To/Working/Directory/')
Comments above have been helpful in exploring the issue, but at the end of the day this seems to be some permissions conflict - adding a sudo -u <user> before the command fixes the issue. Still not clear why Py3 requires this and Py2 doesn't, but perhaps I need to explore the issue more closely with other internal users.
Thanks!
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Calling an external command in Python
I want to run commands in another directory using python.
What are the various ways used for this and which is the most efficient one?
What I want to do is as follows,
cd dir1
execute some commands
return
cd dir2
execute some commands
Naturally if you only want to run a (simple) command on the shell via python, you do it via the system function of the os module. For instance:
import os
os.system('touch myfile')
If you would want something more sophisticated that allows for even greater control over the execution of the command, go ahead and use the subprocess module that others here have suggested.
For further information, follow these links:
Python official documentation on os.system()
Python official documentation on the subprocess module
If you want more control over the called shell command (i.e. access to stdin and/or stdout pipes or starting it asynchronously), you can use the subprocessmodule:
import subprocess
p = subprocess.Popen('ls -al', shell=True, stdout=subprocess.PIPE)
stdout, stderr = p.communicate()
See also subprocess module documentation.
os.system("/dir/to/executeble/COMMAND")
for example
os.system("/usr/bin/ping www.google.com")
if ping program is located in "/usr/bin"
Naturally you need to import the os module.
os.system does not wait for any output, if you want output, you should use
subprocess.call or something like that
You can use Python Subprocess ,which offers many modules to execute commands, checking outputs and receive error messages etc.
Within a python script, I want to issue a command. In perl, I could define a command, save it as a variable (here, $cmd) then type system($cmd) and then the command is executed.
How can i do that in python?
You can use os.system(), but prefer subprocess instead.
Another good choice is "commands" module: http://docs.python.org/library/commands.html.
you can use os.system(), or the newer subprocess module. Other possible alternatives (for older Python versions) include these. (eg os.spawn*,os.popen*,etc)
Lastly, try using Python's modules to do operating system stuff instead of calling external commands if possible, unless its a third party tool you are executing and Python doesn't have the api.