I am trying to build a snakemake pipeline with custom python scripts.
Some of my scripts run into errors, leading to a shutdown of the pipeline.
However, while in the shell output I can barely see the end of the python error message that leads to the shutdown, this error is not logged anywhere. It is not logged in the snakemake.log that gets automatically created (only stating which script failed without giving the error message), and adding a "log: " with a folder to the rule that fails only creates an empty log.
Is there a way to access the error message, so I can solve the underlying issue?
Edit:
my current snakemake rule looks like this:
rule do_X:
input: "{Wildcard}_a"
output: "{wildcard}_b"
log: out = "{Wildcard}_stdout.log"
err = "{Wildcardd}_stderr.err"
shell: python script x.py {input}{output}
If the script fails, I recive empty logs,
The link provided by KeyboardCat's comment seems to work well.
Demonstration adapted from your code and the suggested solution:
Save this as script_x.py:
#script_x.py
print("test output in stdout made my script")
#import sys # WITH THIS COMMENTED OUT, YOU'LL GET TRACEBACK IN `test_stderr.err`
sys.stderr.write("This was made from std.err")
with open("test_b", "w") as out_file:
out_file.write("result\n")
Then your Snakefile is:
with open("test_a", "w") as out_file:
out_file.write("#trigger file")
rule all:
input: "test_b"
rule do_X:
input: "{wildcard}_a"
output: "{wildcard}_b"
log: out = "{wildcard}_stdout.log",
err = "{wildcard}_stderr.err"
shell: 'python script_x.py {input}{output} 2> {log.err} 1> {log.out}'
Execute snakemake so it uses the Snakefile. It will cause an issue because script_x.py has an error in it and fails.
In test_stderr.err, you'll see the traceback from when the script_x.py errored because sys wasn't imported as the script was written:
Traceback (most recent call last):
File "script_x.py", line 3, in <module>
sys.stderr.write("This was made from std.err")
NameError: name 'sys' is not defined
If you delete the files test_a and remove the # from in front of the import sys line inside script_x.py (line #3), it should run without error now and result in the text This was made from std.err showing up in the file test_stderr.err.
test_stdout.log will always end up with test output in stdout made by script in it because it gets created before the line containing the error in the script.
You say you tried the solution suggested at the link provided by KeyboardCat's comment; however, what did you actually try? Always best to include the variation you tried in the comment or add an update section to your OP.
Related
Lately I started working with Q# and python as host language. I was working on a project and everything seemed fine. Suddenly I get the error "module not found" and I seem to get it in all my previous projects too.
So I have a directory: C:\Users\Username\Q#projects
In this folder I have 2 files: HostProtocol.py, which is the main file, and BB84.qs, which is the file from which I want to import.
The HostProtocol.py file looks like this:
import qsharp
from Quantum.BB84 import Run_BB84Protocol
Run_BB84Protocol.simulate()
The BB84.qs file looks like this:
namespace Quantum.BB84 {
open Microsoft.Quantum.Arrays;
open Microsoft.Quantum.Measurement;
open Microsoft.Quantum.Canon;
open Microsoft.Quantum.Intrinsic;
open Microsoft.Quantum.Diagnostics;
open Microsoft.Quantum.Convert;
open Microsoft.Quantum.Math;
function Run_BB84Protocol() Unit{... the code from the function...}
}
When I try to run HostProtocol.py I get the following error message:
Exception has occurred: ModuleNotFoundError
No module named 'Quantum'
File "C:\Users\Username\Q#projects\HostProtocol.py", line 3, in
from Quantum.BB84 import Run_BB84Protocol
And this is for all my previous projects too. It's very frustrating and I have no clue what could have caused it because it worked fine previously. Any help is definitely welcome!
If you have any Q# compilation errors in your Q# source file, your Q# operations and functions will not be available. Check your Python output for errors. When I run the sample code you provided, it reports a syntax error in the Q# code (there's a missing colon before the return type Unit):
fail: Microsoft.Quantum.IQSharp.Workspace[0]
QS3211: Invalid return type annotation. The argument tuple needs to be followed
by a colon and the return type of the callable.
I'm studying from 'Coding the matrix' by Philip Klein. In chapter two, there's an example for plotting complex numbers
from plotting import plot
S = {2+2j, 3+2j, 1.75+1j, 2+1j, 2.25+1j, 2.5+1j, 2.75+1j, 3+1j, 3.25+1j}
plot(S, 4)
plotting module: http://resources.codingthematrix.com
When I run the code directly through python in the terminal, it works fine, but when I run it as seperate file "$ python example.py", I get this error:
gvfs-open: file:///tmp/tmpOYFVs8.html: error opening location: Error
when getting information for file '/tmp/tmpOYFVs8.html': No such file
or directory
Not sure how to resolve this. Tried to play with module code a bit, but got nowhere.
I checked the code of plotting.py and found out that there is atexit event registered at the end of the code which basically deletes the file when your programs exits.So when you invoke it as script python intrepreter exits which will intern calls atexit register to delete the file.
def remove_at_exit(path):
atexit.register(os.remove, path)
you can directly comment out the call to remove_at_exit method in plotting.py at line no 92
open plotting.py then patch this
hpath = os.getcwd() + "/something.html"
Instead or a line after this
hpath = create_temp('.html')
Is there a command line command to disable access to all files and folders in the system from a python program? It should give an error if the program tries to access any file. For example, in the command line:
$ python filename.py <some_command>
or something similar.
It should not allow functions like open('filename.txt') in the program.
Edit : sudo allows to run programs with admin access. Can we create command like sudo which will limits access to another files and folders?
Thank you.
From the list of command line options for python there doesn't seem to be this option (and sandboxing to prevent IO appears to not be too effective). You could make your own command arguments to gain this functionality, for example
import argparse
class blockIOError(Exception):
pass
parser = argparse.ArgumentParser(description='Block IO operations')
parser.add_argument('-bIO','-blockIO',
action='store_true',
help='Flag to prevent input/output (default: False)')
args = parser.parse_args()
blockIO = args.bIO
if not blockIO:
with open('filename.txt') as f:
print(f.read())
else:
raise blockIOError("Error -- input/output not allowed")
The down side is you need to wrap every open, read etc in an if statement. The advantage is you can specify exactly what you want to allow. Output then would look like:
$ python 36477901.py -bIO
Traceback (most recent call last):
File "36477901.py", line 19, in <module>
raise blockIOError("Error -- input/output not allowed")
__main__.blockIOError: Error -- input/output not allowed
I have written a small piece of code in Python, I was trying to run the code from the command line as follows
python -c MyClass.py
The contents of MyClass.py are as follows:
#!/usr/bin/python
import logging;
class MyClass():
data = None
logger = logging.getLogger(__name__)
def __init__(self, data):
self.data = data
if __name__ == "__main__":
a = MyClass("Data")
a.logger.info("This is message")
The above code fails with the following error
Traceback (most recent call last):
File "<string>", line 1, in <module>
NameError: name 'MyClass' is not defined
Mind you, when I ran this from the python tool, IDLE, I saw no issues and the code ran as expected, I am confused as to what I am missing. Also, I saw several variants of this question, but the problem I am facing is specific to me,so , please help me.
The -c flag is to run the passed string as Python code. python -c MyClass.py is equivalent to opening a fresh interpreter and entering MyClass.py. When you do that, MyClass isn't defined, so it fails. To run a script from your system's terminal, simply use python MyClass.py.
The reason your code fails is that you are using wrong command line option:
-c command
Specify the command to execute (see next section).
This terminates the option list (following options
are passed as arguments to the command).
So, what you give after -c is evaluated as a statement, like python -c "print('Hello!')". So, with -c MyClass.py you are literally trying to access an attribute named py from an object named MyClass, which is likely not what you are trying to do.
To execute source code from file MyClass.py, you should omit -c: python MyClass.py.
However, your code won't work either because of two reasons:
1. You are using non-configured logger instance
2. Your logging level is way too low to notice the error.
For example, replace logger.info with logger.warning — you will immediately get No handlers could be found for logger "__main__" error. Searching reference for this message leads us to 15.6.1.6. Configuring Logging for a Library section. Reason is that you are trying to access the logger that is not configured — so it has no idea what to do next with generated messages.
To address this, in your main project file you can configure your logger as suggested in section 15.6.1.5. Configuring Logging, using either your own handler, or reusing some library-provided shortcuts for common tasks, as per section 15.6.3. Useful Handlers.
python -c "import MyClass" will work, but why not run it with a command python MyClass.py?
When ran as "python MyClass.py" , I dont see any output printed to the console, hence I was forced to use python -c MyClass.py.
It work well, because of logging level setting,you can not see log message
.you can set log level or use high level such as logger.error("msg").
Anyway, a.logger.error("This is message") will get No handlers could be found for logger "__main__".It's clear that there is no handlers for logger "main",you need set it or using "root" logger by logging.info("message")
Good luck!
I have a script that executes the following code through a Django view:
# generate dag file
try:
commandString = [
'python',
os.path.join('/srv/nfsshare/transcode50', userFolder, directory, 'condor_execute.py')
]
subprocess.check_call(commandString,
stdout=open('/srv/nfsshare/transcode50/output2.txt', 'w'),
stderr=subprocess.STDOUT)
except Exception, e:
open('/tmp/test_exception.txt', 'w').write(str(e))
raise
What that is supposed to do is execute a Python file I have generated. If I run that exact command from the server's command prompt (i.e., python /srv/nfsshare/transcode50/cogden/algorithms/condor_execute.py), it works just fine and produces the needed files it's supposed to within that directory. However, if I run it from Django, it produces no error messages, produces the correct console output (a message that basically says "Created file blah at blah"), but produces no files whatsoever, when it normally generates two. I'm getting no permissions errors or anything and have ensured the directory is chmodded appropriately.
I suspect your process is never getting around to closing the file (as you can't do it explicitly as you have no reference to it - while run as a script - the interpreter would have stopped - but Django works different with loaded processes etc...) and it's not big enough to be flushed - try moving out your open.
with open('/srv/nfsshare/transcode50/output2.txt', 'w') as stdout:
subprocess.check_call(commandString, stdout=stdout, stderr=subprocess.STDOUT)
Also, make sure that when you make changes to make sure the Django server is restarted/similar to make sure code is reloaded and changes are taken into effect.