I can check python files or modules for error via some libraries, like: pylint, pychecker, pyflakes, etc. In most case, I must specify file or directory for checking. For example:
pylint directory/mymodule.py
It's ok, but not enough for me. I want analyze separated code block and get all detected errors and warnings. So, I must call a python code analyzer from my own module as a part of program.
import some_magic_checker
code = '''import datetime; print(datime.datime.now)'''
errors = some_magic_checker(code)
if len(errors) == 0:
exec(code) # code has no errors, I can execute its
else:
print(errors) # display info about detected errors
Does exist some python library like pylint or pyflakes, which provide functionality for python code checking without code compile? Thanks for your help.
UPD
I will try to explain what i mean in the simple example. I have one variable "codeString", which contains python source code. I must analyze this code (without any file creation and code execution, but I can compile code) and detect all warnings about incorrect blocks of a code. Let's look inside the pyflakes module and understand how it works.
There is a function "check" in module "pyflakes.api".
from pyflakes import checker
from pyflakes import reporter as modReporter
import _ast
import sys
def check(codeString, filename):
reporter = modReporter._makeDefaultReporter()
try:
tree = compile(codeString, filename, "exec", _ast.PyCF_ONLY_AST)
except SyntaxError:
value = sys.exc_info()[1]
msg = value.args[0]
(lineno, offset, text) = value.lineno, value.offset, value.text
# If there's an encoding problem with the file, the text is None.
if text is None:
# Avoid using msg, since for the only known case, it contains a
# bogus message that claims the encoding the file declared was
# unknown.
reporter.unexpectedError(filename, 'problem decoding source')
else:
reporter.syntaxError(filename, msg, lineno, offset, text)
return 1
except Exception:
reporter.unexpectedError(filename, 'problem decoding source')
return 1
# Okay, it's syntactically valid. Now check it.
w = checker.Checker(tree, filename)
w.messages.sort(key=lambda m: m.lineno)
for warning in w.messages:
reporter.flake(warning)
return len(w.messages)
How you can see, this function cannot work only with one parameter "codeString", we must also provide second parameter "filename". And this is my biggest problem, I don't have any file, just Python code in string variable.
pylint, pychecker, pyflakes and all libraries, what i know, works only with created files. So i try to find some solutions, which don't require link to Python file.
Built-in function "compile" allow to create compiled code or AST object and catch some errors without file creation and code execution. We must pass '<string>' value as a second parameter, because code wasn’t read from a file.
>>> def execute(code_string):
>>> output = list()
>>> try:
>>> tree = compile(code_string, '<string>', 'exec')
>>> except Exception as e:
>>> print(e)
>>> else:
>>> exec(tree)
>>> # Now I can check code before calling the "exec" function
>>> code_string = 'print("Hello_World!")'
>>> execute(code_string) # All is ok
Hello_World!
>>> code_string = 'ERROR! print("Hello_World!")'
>>> execute(code_string) # This code will not executed
invalid syntax (<string>, line 1)
Related
I want to test my lexer/parser in a Python Unittest. For that purpose I'd like antlr to fail with an Exception everytime anything is out of order. What is the most elegant way to achieve this?
I have read the ANTLR Mega Tutorial, there they write an ErrorListener which overrides syntaxError() to save the last offending symbol and then in the end, they check if any offending symbol was encountered.
I saw that there are different ErrorStrategy classes and BailErrorStrategy sounds like this is what I need. But then I read How can I fail on first syntax error in a python ANTLR generated parser while keeping the error message? which says this strategy doesn't alway throw an Exception.
My best idea so far is to throw an Exception in ErrorListener.syntxError():
import unittest
from antlr4 import *
from antlr.myLexer import myLexer
from antlr.myParser import myParser
from antlr4.error.ErrorListener import ErrorListener
class MyErrorListener(ErrorListener):
def syntaxError(self, recognizer, offendingSymbol, line, column, msg, e):
raise Exception("ERROR: when parsing line %d column %d: %s\n" % \
(line, column, msg))
class TestMyParser(unittest.TestCase):
def test_with_testfile(self):
error_listener = MyErrorListener()
input_stream = FileStream("testfile")
lexer = myLexer(input_stream)
lexer.removeErrorListeners()
lexer.addErrorListener(error_listener)
stream = CommonTokenStream(lexer)
parser = myParser(stream)
parser.removeErrorListeners()
parser.addErrorListener(error_listener)
tree = parser.startrule()
if __name__ == '__main__':
unittest.main()
The best option you have is the BailErrorStategy. It uses an exception which is not catched anywhere in the ANTLR4 runtime and hence that bubbles up to your own code directly.
If you use a normal error handler/default strategy it will always try to recover from syntax errors to allow continuing the parse run.
But even with the BailErrorStrategy in place you can have an error listener to get the first (and only) error that came up. For the error messages I recommend attaching your own error listener class to produce your own version of the error message like I did in this C++ code
A parser I created reads recorded chess games from a file. The API is used like this:
import chess.pgn
pgn_file = open("games.pgn")
first_game = chess.pgn.read_game(pgn_file)
second_game = chess.pgn.read_game(pgn_file)
# ...
Sometimes illegal moves (or other problems) are encountered. What is a good Pythonic way to handle them?
Raising exceptions as soon as the error is encountered. However, this makes every problem fatal, in that execution stops. Often, there is still useful data that has been parsed and could be returned. Also, you can not simply continue parsing the next data set, because we are still in the middle of some half-read data.
Accumulating exceptions and raising them at the end of the game. This makes the error fatal again, but at least you can catch it and continue parsing the next game.
Introduce an optional argument like this:
game = chess.pgn.read_game(pgn_file, parser_info)
if parser_info.error:
# This appears to be quite verbose.
# Now you can at least make the best of the sucessfully parsed parts.
# ...
Are some of these or other methods used in the wild?
The most Pythonic way is the logging module. It has been mentioned in comments but unfortunately without stressing this hard enough. There are many reasons it's preferable to warnings:
Warnings module is intended to report warnings about potential code issues, not bad user data.
First reason is actually enough. :-)
Logging module provides adjustable message severity: not only warnings, but anything from debug messages to critical errors can be reported.
You can fully control output of logging module. Messages can be filtered by their source, contents and severity, formatted in any way you wish, sent to different output targets (console, pipes, files, memory etc)...
Logging module separates actual error/warning/message reporting and output: your code can generate messages of appropriate type and doesn't have to bother how they're presented to end user.
Logging module is the de-facto standard for Python code. Everyone everywhere is using it. So if your code is using it, combining it with 3rd party code (which is likely using logging too) will be a breeze. Well, maybe something stronger than breeze, but definitely not a category 5 hurricane. :-)
A basic use case for logging module would look like:
import logging
logger = logging.getLogger(__name__) # module-level logger
# (tons of code)
logger.warning('illegal move: %s in file %s', move, file_name)
# (more tons of code)
This will print messages like:
WARNING:chess_parser:illegal move: a2-b7 in file parties.pgn
(assuming your module is named chess_parser.py)
The most important thing is that you don't need to do anything else in your parser module. You declare that you're using logging system, you're using a logger with a specific name (same as your parser module name in this example) and you're sending warning-level messages to it. Your module doesn't have to know how these messages are processed, formatted and reported to user. Or if they're reported at all. For example, you can configure logging module (usually at the very start of your program) to use a different format and dump it to file:
logging.basicConfig(filename = 'parser.log', format = '%(name)s [%(levelname)s] %(message)s')
And suddenly, without any changes to your module code, your warning messages are saved to a file with a different format instead of being printed to screen:
chess_parser [WARNING] illegal move: a2-b7 in file parties.pgn
Or you can suppress warnings if you wish:
logging.basicConfig(level = logging.ERROR)
And your module's warnings will be ignored completely, while any ERROR or higher-level messages from your module will still be processed.
Actually, those are fatal errors -- at least, as far as being able to reproduce a correct game; on the other hand, maybe the player actually did make the illegal move and nobody noticed at the time (which would make it a warning, not a fatal error).
Given the possibility of both fatal errors (file is corrupted) and warnings (an illegal move was made, but subsequent moves show consistency with that move (in other words, user error and nobody caught it at the time)) I recommend a combination of the first and second options:
raise an exception when continued parsing isn't an option
collect any errors/warnings that don't preclude further parsing until the end
If you don't encounter a fatal error then you can return the game, plus any warnings/non-fatal errors, at the end:
return game, warnings, errors
But what if you do hit a fatal error?
No problem: create a custom exception to which you can attach the usable portion of the game and any other warnings/non-fatal errors to:
raise ParsingError(
'error explanation here',
game=game,
warnings=warnings,
errors=errors,
)
then when you catch the error you can access the recoverable portion of the game, along with the warnings and errors.
The custom error might be:
class ParsingError(Exception):
def __init__(self, msg, game, warnings, errors):
super().__init__(msg)
self.game = game
self.warnings = warnings
self.errors = errors
and in use:
try:
first_game, warnings, errors = chess.pgn.read_game(pgn_file)
except chess.pgn.ParsingError as err:
first_game = err.game
warnings = err.warnings
errors = err.errors
# whatever else you want to do to handle the exception
This is similar to how the subprocess module handles errors.
For the ability to retrieve and parse subsequent games after a game fatal error I would suggest a change in your API:
have a game iterator that simply returns the raw data for each game (it only has to know how to tell when one game ends and the next begins)
have the parser take that raw game data and parse it (so it's no longer in charge of where in the file you happen to be)
This way if you have a five-game file and game two dies, you can still attempt to parse games 3, 4, and 5.
I offered the bounty because I'd like to know if this is really the best way to do it. However, I'm also writing a parser and so I need this functionality, and this is what I've come up with.
The warnings module is exactly what you want.
What turned me away from it at first was that every example warning used in the docs looks like these:
Traceback (most recent call last):
File "warnings_warn_raise.py", line 15, in <module>
warnings.warn('This is a warning message')
UserWarning: This is a warning message
...which is undesirable because I don't want it to be a UserWarning, I want my own custom warning name.
Here's the solution to that:
import warnings
class AmbiguousStatementWarning(Warning):
pass
def x():
warnings.warn("unable to parse statement syntax",
AmbiguousStatementWarning, stacklevel=3)
print("after warning")
def x_caller():
x()
x_caller()
which gives:
$ python3 warntest.py
warntest.py:12: AmbiguousStatementWarning: unable to parse statement syntax
x_caller()
after warning
I'm not sure if the solution is pythonic or not, but I use it rather often with slight modifications: a parser does its job within a generator and yields results and a status code. The receiving code makes decisions what to to with failed items:
def process_items(items)
for item in items:
try:
#process item
yield processed_item, None
except StandardError, err:
yield None, (SOME_ERROR_CODE, str(err), item)
for processed, err in process_items(items):
if err:
# process and log err, collect failed items, etc.
continue
# further process processed
A more general approach is to practice in using design patterns. A simplified version of Observer (when you register callbacks for specific errors) or a kind of Visitor (where the visitor has methods for procesing specific errors, see SAX parser for insights) might be a clear and well understood solution.
Without libraries, it is difficult to do this cleanly, but still possible.
There are different methods of handling this, depending on the situation.
Method 1:
Put all contents of while loop inside the following:
while 1:
try:
#codecodecode
except Exception as detail:
print detail
Method 2:
Same as Method 1, except having multiple try/except thingies, so it doesn't skip too much code & you know the exact location of the error.
Sorry, in a rush, hope this helps!
I have tried the following which doesn't work:
try:
f = h5py.File(filepath)
except NameError:
f = scipy.io.loadmat(filepath)
Basically, a user will pass a particular input(a filepath) to a function which is
supposed to load the data in that file. But, I don't expect the user to know whether the function is defined for that input.
I'm getting the following error:
OSError: Unable to create file (Unable to open file: name = '/users/cyrilrocke/documents/c_elegans/data/chemotaxis/n2laura_benzaldehyde_l_2013_03_17__15_39_19___2____features.mat', errno = 17, error message = 'file exists', flags = 15, o_flags = a02)
Note: basically, I want to be able to switch between the function h5py.File() and scipy.io.loadmat() depending on which one doesn't work. Given the inputs, one of these functions must work.
I guess you want something like that:
flag = True;
try:
f = h5py.File(filepath)
except:
flag = False;
try:
if not flag:
f = scipy.io.loadmat(filepath)
except:
print('Error')
The problems in your code are:
you are catching for the wrong error. Without specifying the error type you can handle all kinds of error in a single way (you can also add multiple except clauses)
you are executing a potentially failing function inside the except clause. In this way you won't be able to handle new errors. I then splitted them in two separate try/except blocks.
First: your code should know whether a function is defined! That's not something external, it is something that is entirely in your power.
If an import might fail, maybe do something like
try:
import h5py
HAVE_H5PY = True
except ImportError:
HAVE_H5PY = False
and then use that to check.
Second, you could instead do something like
if 'h5py' in globals():
but that is ugly and unnecessary (you have to catch the ImportError anyway, if that is the problem you're trying to solve).
Third, your error message has nothing to do with any of this, it is returned by one of those two commands. It is trying to create a file that already exists.
error message = 'file exists'
Your program is getting EEXIST and if you see a difference between trials with h5py and scipy.io, it's because they have different flags that they're passing to open() and perhaps differing states if you cleanup the files created between trials.
From the h5py.File docs:
Valid modes are:
...
r Readonly, file must exist
w Create file, truncate if exists
w- or x Create file, fail if exists
a Read/write if exists, create otherwise (default)
Since you're looking to read the file (only?) you should specify 'r' as the mode.
If you're experiencing this problem only rarely, then perhaps it's a race generated by a TOCTOU-style error in h5py.
Indeed, such a TOCTOU design error exists in h5py.
https://github.com/h5py/h5py/blob/c699e741de64fda8ac49081e9ee1be43f8eae822/h5py/_hl/files.py#L99
# Open in append mode (read/write).
# If that fails, create a new file only if it won't clobber an
# existing one (ACC_EXCL)
try:
fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
except IOError:
fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl)
This is the reason for the O_EXCL flag used by posix-style OSs. h5f.ACC_EXCL likely does the same thing, but if it's only used upon failure of h5f.ACC_RDWR then its atomicity is diminished.
I'm attempting to incorporate error handling into my python script.
prod.py searches the kernel config file for a module and returns it's branch.
example:
comp#able~$ python prod.py CRYPTO_DEV
# Random Number Generation
import sys
module = sys.argv[1]
def probe(module):
with open('/usr/src/linux/.config', 'r') as module_list:
modlist = module_list.read()
blocks = modlist.split('\n\n')
block = filter (lambda search:module in search, blocks)
for line in block:
head = line.split('\n')
print head[1]
break
probe(module)
I'd like to capture typos and print "Are you sure that is a module?" in return.
End Question
The errors below were thrown by shell, and not python. Apologies.
but I would also like to handle errors like;
comp#able~$ ./prod.py CRYPTO_DEV;[
# Random Number Generation
bash:[:missing `]
comp#able~$ ./prod.py CRYPTO_DEV(
# Random Number Generation
bash:syntax error near unexpected token `('
---
I attempted some variations of 'try:' and 'except', but either broke the script, returned no value, or returned a value regardless of error.
Any help is greatly appreciated.
I'm trying to save myself just a few keystrokes for a command I type fairly regularly in Python.
In my python startup script, I define a function called load which is similar to import, but adds some functionality. It takes a single string:
def load(s):
# Do some stuff
return something
In order to call this function I have to type
>>> load('something')
I would rather be able to simply type:
>>> load something
I am running Python with readline support, so I know there exists some programmability there, but I don't know if this sort of thing is possible using it.
I attempted to get around this by using the InteractivConsole and creating an instance of it in my startup file, like so:
import code, re, traceback
class LoadingInteractiveConsole(code.InteractiveConsole):
def raw_input(self, prompt = ""):
s = raw_input(prompt)
match = re.match('^load\s+(.+)', s)
if match:
module = match.group(1)
try:
load(module)
print "Loaded " + module
except ImportError:
traceback.print_exc()
return ''
else:
return s
console = LoadingInteractiveConsole()
console.interact("")
This works with the caveat that I have to hit Ctrl-D twice to exit the python interpreter: once to get out of my custom console, once to get out of the real one.
Is there a way to do this without writing a custom C program and embedding the interpreter into it?
Edit
Out of channel, I had the suggestion of appending this to the end of my startup file:
import sys
sys.exit()
It works well enough, but I'm still interested in alternative solutions.
You could try ipython - which gives a python shell which does allow many things including automatic parentheses which gives you the function call as you requested.
I think you want the cmd module.
See a tutorial here:
http://wiki.python.org/moin/CmdModule
Hate to answer my own question, but there hasn't been an answer that works for all the versions of Python I use. Aside from the solution I posted in my question edit (which is what I'm now using), here's another:
Edit .bashrc to contain the following lines:
alias python3='python3 ~/py/shellreplace.py'
alias python='python ~/py/shellreplace.py'
alias python27='python27 ~/py/shellreplace.py'
Then simply move all of the LoadingInteractiveConsole code into the file ~/py/shellreplace.py Once the script finishes executing, python will cease executing, and the improved interactive session will be seamless.