Develop Reference

subprocess.Popen trying to open the module instead of executing it on shell

action_publisher = subprocess.Popen(
["bash", "-c", "/opt/ros/melodic/bin/rostopic pub -r 20 /robot_operation std_msgs/String start"],
env={'ROS_MASTER_URI': 'http://10.42.0.49:11311\''})
I tried to run it shell=True and shell=False. Also calling it with bash or just running my executable and I am always getting an error:
Traceback (most recent call last):
File "/opt/ros/melodic/bin/rostopic", line 34, in <module>
import rostopic
ImportError: No module named rostopic
How can I make a call of a shell executable with open through python removing this issue? Tried all combination possible and also other stack proposed solution and still, it tries to import the executable instead of running it on a shell.

I can identify several problems with your attempt, but I'm not sure I have identified them all.
You should use subprocess.check_call or subprocess.run if you just want the subprocess to run, and your Python script to wait for that to complete. If you need to use raw subprocess.Popen(), there are several additional required steps of plumbing which you need to do yourself, which check_call or run will perform for you if you use these higher-level functions.
Your use of env will replace the variables in the environment. Copy the existing environment instead so you don't clobber useful settings like PYTHONPATH etc which may well be preventing the subprocess from finding the library it needs.
The shell invocation seems superfluous.
The stray escaped single quote at the end of 'http://10.42.0.49:11311\'' definitely looks wrong.
With that, try this code instead; but please follow up with better diagnostics if this does not yet solve your problem completely.
import subprocess
import os
# ...
env = os.environ.copy()
env['ROS_MASTER_URI'] = 'http://10.42.0.49:11311'
action_publisher = subprocess.run(
["/opt/ros/melodic/bin/rostopic", "pub", "-r", "20",
"/robot_operation", "std_msgs/String", "start"],
env=env, check=True)
If rostopic is actually a Python program, a better solution altogether might be to import it directly instead.

It sounds like the code of that file is trying to import the Python module. If you're getting that import error even when you try to execute the file in bash/from a shell, then it has nothing to do with subprocess.Popen.
From the traceback, it looks like it's a Python file itself and it's trying to import that module, which would explain why you see the issue when executing it from a shell.
Did you go through the installation correctly, specifically the environment setup? http://wiki.ros.org/melodic/Installation/Ubuntu#melodic.2FInstallation.2FDebEnvironment.Environment_setup
It sounds like you need to source a particular file to have the correct paths available where the Python module is located so it can be found when you execute the script.

As far as I can see, your .Popen command will try to execute
bash -c /opt/ros/melodic/bin/rostopic pub -r 20 /robot_operation std_msgs/String start
while bash -c has to be followed by a string. Thus, you may need to add single quotes.

How to run a git command in python? [duplicate]

I have been asked to write a script that pulls the latest code from Git, makes a build, and performs some automated unit tests.
I found that there are two built-in Python modules for interacting with Git that are readily available: GitPython and libgit2.
What approach/module should I use?

An easier solution would be to use the Python subprocess module to call git. In your case, this would pull the latest code and build:
import subprocess
subprocess.call(["git", "pull"])
subprocess.call(["make"])
subprocess.call(["make", "test"])
Docs:
subprocess - Python 2.x
subprocess - Python 3.x

I agree with Ian Wetherbee. You should use subprocess to call git directly. If you need to perform some logic on the output of the commands then you would use the following subprocess call format.
import subprocess
PIPE = subprocess.PIPE
branch = 'my_branch'
process = subprocess.Popen(['git', 'pull', branch], stdout=PIPE, stderr=PIPE)
stdoutput, stderroutput = process.communicate()
if 'fatal' in stdoutput:
# Handle error case
else:
# Success!

So with Python 3.5 and later, the .call() method has been deprecated.
https://docs.python.org/3.6/library/subprocess.html#older-high-level-api
The current recommended method is to use the .run() method on subprocess.
import subprocess
subprocess.run(["git", "pull"])
subprocess.run(["make"])
subprocess.run(["make", "test"])
Adding this as when I went to read the docs, the links above contradicted the accepted answer and I had to do some research. Adding my 2 cents to hopefully save someone else a bit of time.

In EasyBuild, we rely on GitPython, and that's working out fine.
See here, for examples of how to use it.

If GitPython package doesn't work for you there are also the PyGit and Dulwich packages. These can be easily installed through pip.
But, I have personally just used the subprocess calls. Works perfect for what I needed, which was just basic git calls. For something more advanced, I'd recommend a git package.

I had to use shlex on top of the run call because my command was too complex for the subprocess alone to understand.
import subprocess
import shlex
git_command = "git <command>"
subprocess.run(shlex.split(git_command))

If you're on Linux or Mac, why use python at all for this task? Write a shell script.
#!/bin/sh
set -e
git pull
make
./your_test #change this line to actually launch the thing that does your test

How to use Paramiko logging?

I'm using Paramiko in Python to run command on a box through SSH. How to use Paramiko logging? I mean force it to make logs (in a file or terminal) and set the log level.

Paramiko names its loggers, so simply:
import logging
import paramiko
logging.basicConfig()
logging.getLogger("paramiko").setLevel(logging.WARNING) # for example
See the logging cookbook for some more examples.
You can also use log_to_file from paramiko.util to log directly to a file.

paramiko.util.log_to_file("<log_file_path>", level = "WARN")

Logging all errors and stdout output with Python

I know this has been discussed here before, but I haven't found a solution that will work for me. I already have a python script that I wrote, and I currently have it run at boot. What I would like to do is log all outputs, which include any print statements to the console, or any error messages that would come up. I do like the Logging module, and would prefer to use that over looking at all outputs on the console. Any suggestions?

If you manage your script using supervisor it will automatically handle all logging of stdout/stderr for you.
Additionally, it can automatically restart your script if it were to crash

py.test logging control

We have recently switched to py.test for python testing (which is fantastic btw). However, I'm trying to figure out how to control the log output (i.e. the built-in python logging module). We have pytest-capturelog installed and this works as expected and when we want to see logs we can pass --nologcapture option.
However, how do you control the logging level (e.g. info, debug etc.) and also filter the logging (if you're only interested in a specific module). Is there existing plugins for py.test to achieve this or do we need to roll our own?
Thanks,
Jonny

Installing and using the pytest-capturelog plugin could satisfy most of your pytest/logging needs. If something is missing you should be able to implement it relatively easily.

As Holger said you can use pytest-capturelog:
def test_foo(caplog):
caplog.setLevel(logging.INFO)
pass
If you don't want to use pytest-capturelog you can use a stdout StreamHandler in your logging config so pytest will capture the log output. Here is an example basicConfig
logging.basicConfig(level=logging.DEBUG, stream=sys.stdout)

A bit of a late contribution, but I can recommend pytest-logging for a simple drop-in logging capture solution. After pip install pytest-logging you can control the verbosity of the your logs (displayed on screen) with
$ py.test -s -v tests/your_test.py
$ py.test -s -vv tests/your_test.py
$ py.test -s -vvvv tests/your_test.py
etc... NB - the -s flag is important, without it py.test will filter out all the sys.stderr information.

Pytest now has native support for logging control via the caplog fixture; no need for plugins.
You can specify the logging level for a particular logger or by default for the root logger:
import pytest
def test_bar(caplog):
caplog.set_level(logging.CRITICAL, logger='root.baz')
Pytest also captures log output in caplog.records so you can assert logged levels and messages. For further information see the official documentation here and here.

A bit of an even later contribution: you can try pytest-logger. Novelty of this plugin is logging to filesystem: pytest provides nodeid for each test item, which can be used to organize test session logs directory (with help of pytest tmpdir facility and it's testcase begin/end hooks).
You can configure multiple handlers (with levels) for terminal and filesystem separately and provide own cmdline options for filtering loggers/levels to make it work for your specific test environment - e.g. by default you can log all to filesystem and small fraction to terminal, which can be changed on per-session basis with --log option if needed. Plugin does nothing by default, if user defines no hooks.