Is there a pythonic way to throw an exception to other developers to warn them about using a piece of code with bugs in it?
For example:
def times_two(x):
raise BrokenException("Attn.This code is unreliable. Only works on positive numbers")
x = abs(x) * 2
return x
I understand that I can raise a generic exception with a message, or even derive my own exception classes, but i just want to know if there is a built-in, pythonic way to do something like this.
And also, I understand that why the actual times_two function doesn't work. That was just an example function.
This is not something to validate input parameters, or even returned values.
This is simply to mark a function as potentially unreliable.
The code must be used in some areas under very specific circumstances, but when devs are writing code and run across this function should be warned of the limitations.
Your example is pretty flawed for any use case in which alerting the developers would be needed. This would need to alert the user not to input a negative number.
def times_two(x):
if x < 0:
raise BrokenException("Attn user. Don't give me negitive numbers.")
return x * 2
Although, I think if your example more accurately described an actual error needing developer attention then you should just fix that and not put it into production knowing there is an error in it.
sentry.io on the other hand can help find errors and help developers fix errors while in production. You may want to look into that if warnings isn't for you. From their README.me:
Sentry fundamentally is a service that helps you monitor and fix
crashes in realtime. The server is in Python, but it contains a full
API for sending events from any language, in any application.
Builtin Exception 'ValueError' is the one that should be used.
def times_two(x):
if x < 0:
raise ValueError('{} is not a positive number.'.format(x))
return x * 2
This seems like an XY problem. The original problem is that you have some code which is incomplete or otherwise known to not work. If it is something you are currently working on, then the correct tool to use here is your version control. With Git, you would create a new branch which only be merged into master and prepared for release to production after you have completed the work. You shouldn't release a partial implementation.
Do you want to stop execution when the function is called? If so, then some sort of exception, like the BrokenException in your example is a good way of doing this.
But if you want to warn the caller, and then continue on anyway, then you want a Warning instead of an exception. You can still create your own:
class BrokenCodeWarning(Warning)
pass
When you raise BrokenCodeWarning, execution will not be halted by default, but a warning will be printed to stderr.
The warnings filter controls whether warnings are ignored, displayed, or turned into errors (raising an exception).
https://docs.python.org/3.7/library/warnings.html#the-warnings-filter
Related
Hello this question is about python.
Is there a way to ignore all kinds off exceptions. I know i could just put the whole code in an huge try cahtch but i want it to continue even if one part fails and in result of this some other parts fail too. One way to achieve this would be to put every single line in a try except statement. But is there an other more elegant way to do this?
Well , you can
1 - Put every separate part in a try catch
try:
#something
except:
pass
2 - Put everything in a bigger try catch
try:
#do something
#so something else
#do something else
except:
pass
or you can use contextlib.suppress (https://docs.python.org/3/library/contextlib.html#contextlib.suppress) as Random Davis suggested to ignore certain types of exceptions
But ignoring all exception is a really bad idea , instead you should do
try:
#something
except:
#something else
As far as I know , there is no other "elegant" way to ignore exceptions
(This could have been a comment but I lack rep to post comments)
Besides using try, my python professor will generally comment out a function that he isn't testing so he could test a specific function. If part of your code isn't working try adding a breakpoint before debugging, then you can run your code up until the breakpoint to see if a certain line is doing what you want it to. In visual studio code you can do this by clicking in the empty space just to the left of the line numbers, and you will see a red dot on the line if done correctly.
If you are referencing that a function isn't doing what you want it to do, employing these methods will help you find the error in your ways, starting with tracking the input all the way until you get the output you want, going function by function.
If you're referencing that your code is simply too broken to function correctly, your code will always run until it returns an error, and if the error is early in that process between converting an input to an output, you can come across a multitude of reasons behind this that will stop functions that are supposed to run afterwards from working correctly. If that is the case comment out the later functions until your first one is working, and keep working chronologically to debug those errors. In the future, I highly recommend posting your exact code when asking a coding question, otherwise it can be hard for others to extract the necessary information to answer your question effectively. Good luck!
In reference to Global-Occult's answer, although you can basically try something and except extraneous information, you really don't want to be coding like that, because to develop higher level programs that extra information will no longer be extraneous, in fact it could be very important data that will allow you to develop a program much further.
Can someone explain WHEN to use and what are the BEST SUITABLE usage scenarios for assert?
My points are:
Since it just equals if not then raise
And in optimisation mode -O it will be ignored
So what are the usage scenarios for assert in source code (not unit test)?
From my very traditional experiences, assert should only exist in unit tests, really cannot get much point why it starts appearing in Python project code more and more.
A good way to think of an assert statement in library code is as a loaded comment, it's a like a little piece of documentation about how the code works (or how you thought it worked) that will blow up and make a big noise if the "comment" is ever making a claim that turns out to be wrong.
When to use assert?
When you write an assert statement into source code (not test code), you should be very confident that it will not fire, regardless of the program input. Of course you can't be 100% certain that it will never fire, but you should be sure that if it were to fire then you have made an incorrect assumption somewhere and you'll need to revisit this section of the code.
Why add asserts into library code at all? If you should be sure they won't fire, then what's the point?
Because you make mistakes and you're not perfect. Using an assertion can be a safeguard against your own mistakes. It's like when you lock the car door and then check it worked anyway by trying to lift up the door handle.
It prevents code from continuing if you have made an error somewhere. A plain old comment can't do that. If the logic is relying on some incorrect assumption, a well-written assertion of that assumption can protect you from the unsafe execution of code. This allows a controlled failure mode of the code, rather than the bug manifesting in some bizarre way later on, making it harder to find the root cause.
It's a form of defensive programming, which can be used to protect against changes from the future! This is a good trick. A future you, or another developer working on project, may add some code in a later commit which invalidates an assumption you had made a year ago. An assertion statement is like a signpost here that says "hey, if you changed that thing, you'll need to change something here too" and draws attention to some subtle piece of implementation detail that could be easy to miss otherwise.
When NOT to use assert
Don't use them to validate input! Use exceptions for that if necessary. If an assertion fires, that's a bug. If a user reports an unhandled AssertionError, it is your problem and not the user's fault. Something needs to be fixed.
Here's an example of a bad assertion:
def square(n):
assert isinstance(n, int)
...
If this fires, that was the caller's fault. Should you need it, a TypeError is more appropriate here than unhandled AssertionError.
Here's an example of an ok assertion:
s = None
while s not in {'y', 'n'}:
s = input("do the thing? [y/n] ").lower()
if s == 'y':
# do the thing
else:
assert s == 'n'
# do other stuff
It doesn't validate data, i.e. the user can't type any input that would cause an assertion fire - the "assumption" the developer is making here is that since the while loop has exited, s must be 'y' or 'n'. This is better than an elif s == 'n':, else: raise type construction, because that else: block could never be reached, so it would not receive test coverage unless you do some really intrusive mocking. Last but not least, it protects against entering the handling for the 'n' branch incorrectly should the foolish future-you add 6 more choices to the prompt but only add handling for 5 of those choices (oops!)
I'm thinking about where to write the log record around an operation. Here are two different styles. The first one, write log before the operation.
Before:
log.info("Perform operation XXX")
operation()
And here is a different style, write the log after the operation.
After:
operation()
log.info("Operation XXX is done.")
With the before-style, the logging records say what is going to do now. The pro of this style is that when something goes wrong, developer can detect it easily, because they know what is the program doing now. But the con is that you are not sure is the operation finished correctly, if something wrong is inside the operation, for example, a function call gets blocked there and never return, you can't never know it by reading the logging records. With the after-style, you are sure the operation is done.
Of course, we can mix those two style together
Both:
log.info("Perform operation XXX")
operation()
log.info("Operation XXX is done.")
But I feel that is kinda verbose, it makes double logging records. So, here is my question - what is the good logging style? I would like to know how do you think.
I'd typically use two different log levels.
The first one I put on a "debug" level, and the second one on an "info" level. That way typical production machines would only log what's being done, but I can turn on the debug logging and see what it tries to do before it errors out.
It all depends what you want to log. If you're interested in the code getting to the point where it's about to do an operation. If you want to make sure the operation succeeded, do it after. If you want both, do both.
Maybe you could use something like a try catch ? Here 's a naive python example :
try :
operation()
log.info("Operation XXX is done.")
except Exception:
log.info("Operation xxx Failed")
raise Exception() # optional : if you want to propagate failure to another try catch statement and/or crash eventually.
Operation will be launched.
If it doesn't fail (no exception raised) you get a success statement in the logs.
If it fails (by raising an exception. Like disc full or whatever you are trying to do), Exception is caught and you get a failure statement.
Log is more meaning full. You get to keep the verbosity to a oneliner and get to know if operation succeeded. Best of all choices.
Oh and you get a hook point where you can add some code to be executed in case of failure.
I hope it help.
There's another style that I've seen used in Linux boot scripts and in strace. It's got the advantages of your combined style with less verbosity, but you've got to make sure that your logging facility isn't doing any buffering. I don't know log.info, so here's a rough example with print:
print "Doing XXX... ", # Note lack of newline :)
operation()
print "Done."
(Since in most cases print uses buffering, using this example verbatim won't work properly. You won't see "Doing XXX" until you see the "Done". But you get the general idea.)
The other disadvantage of this style is that things can get mixed up if you have multiple threads writing to the same log.
I plan to use those functions in web-environment, so my concern is if those functions can be exploited and used for executing malicious software on the server.
Edit: I don't execute the result. I parse the AST tree and/or catch SyntaxError.
This is the code in question:
try:
#compile the code and check for syntax errors
compile(code_string, filename, "exec")
except SyntaxError, value:
msg = value.args[0]
(lineno, offset, text) = value.lineno, value.offset, value.text
if text is None:
return [{"line": 0, "offset": 0,
"message": u"Problem decoding source"}]
else:
line = text.splitlines()[-1]
if offset is not None:
offset = offset - (len(text) - len(line))
else:
offset = 0
return [{"line": lineno, "offset": offset, "message": msg}]
else:
#no syntax errors, check it with pyflakes
tree = compiler.parse(code_string)
w = checker.Checker(tree, filename)
w.messages.sort(lambda a, b: cmp(a.lineno, b.lineno))
checker.Checker is pyflakes class that parses the AST tree.
I think the more interesting question is what are you doing with the compiled functions? Running them is definitely unsafe.
I've tested the few exploits i could think of seeing as its just a syntax checker (can't redefine classes/functions etc) i don't think there is anyway to get python to execute arbitrary code at compile time
If the resulting code or AST object is never evaluated, I think you are only subject to DDoS attacks.
If you are evaluating user inputed code, it is the same as giving shell access as the webserver user to every user.
They are not, but it's not too hard finding a subset of Python that can be sandboxed to a point. If you want to go down that road you need to parse that subset of Python yourself and intercept all calls, attribute lookups and everything else involved. You also don't want to give users access to any language construct such as unterminating loops and more.
Still interested? Head over to jinja2.sandbox :)
compiler.parse and compile could most definitely be used for an attack if the attacker can control their input and the output is executed. In most cases, you are going to either eval or exec their output to make it run so those are still the usual suspects and compile and compiler.parse (deprecated BTW) are just adding another step between the malicious input and the execution.
EDIT: Just saw that you left a comment indicating that you are actually planning on using these on USER INPUT. Don't do that. Or at least, don't actually execute the result. That's a huge security hole for whoever ends up running that code. And if nobody's going to run it, why compile it? Since you clarified that you only want to check syntax, this should be fine. I would not store the output though as there's no reason to make anything easier for a potential attacker and being able to get arbitrary code onto your system is a first step.
If you do need to store it, I would probably favor a scheme similar to that commonly used for images where they are renamed in a non-predictable manner with the added step of making sure that it is not stored on the import path.
Yes, they can be maliciously exploited.
If you really want safe sandboxing, you could look at PyPy's sandboxing features, but be aware that sandboxing is not easy, and there may be better ways to accomplish whatever you are seeking.
Correction
Since you've updated your question to clarify that you're only parsing the untrusted input to AST, there is no need to sandbox anything: sandboxing is specifically about executing untrusted code (which most people probably assumed your goal was, by asking about sandboxing).
Using compile / compiler only for parsing this way should be safe: Python source parsing does not have any hooks into code execution. (Note that this is not necessarily true of all languages: for example, Perl cannot be (completely) parsed without code execution.)
The only other remaining risk is that someone may be able to craft some pathological Python source code that makes one of the parsers use runaway amounts of memory / processor time, but resource exhaustion attacks affect everything, so you'll just want to manage this as it becomes necessary. (For example, if your deployment is mission-critical and cannot afford a denial of service by an attacker armed with pathological source code, you can execute the parsing in a resource-limited subprocess).
Say I want to manipulate some files on a floppy drive or a USB card reader. How do I check to see if the drive in question is ready? (That is, has a disk physically inserted.)
The drive letter exists, so os.exists() will always return True in this case. Also, at this point in the process I don't yet know any file names, so checking to see if a given file exists also won't work.
Some clarification: the issue here is exception handling. Most of the win32 API calls in question just throw an exception when you try to access a drive that isn't ready. Normally, this would work fine - look for something like the free space, and then catch the raised exception and assume that means there isn't a disk present. However, even when I catch any and all exceptions, I still get an angry exception dialog box from Windows telling me the floppy / card reader isn't ready. So, I guess the real question is - how do I suppress the windows error box?
And the answer, as with so many things, turns out to be in an article about C++/Win32 programming from a decade ago.
The issue, in a nutshell, is that Windows handles floppy disk errors slightly differently than other kinds of drive errors. By default, no matter what you program does, or thinks it's doing, Windows will intercept any errors thrown by the device and present a dialog box to the user rather than letting the program handle it - the exact issue I was having.
But, as it turns out, there's a Win32 API call to solve this issue, primarily SetErrorMode()
In a nutshell (and I'm handwaving a way a lot of the details here), we can use SetErrorMode() to get Windows to stop being quite so paranoid, do our thing and let the program handle the situation, and then reset the Windows error mode back to what it was before as if we had never been there. (There's probably a Keyser Soze joke here, but I've had the wrong amount of caffeine today to be able to find it.)
Adapting the C++ sample code from the linked article, that looks about like this:
int OldMode; //a place to store the old error mode
//save the old error mode and set the new mode to let us do the work:
OldMode = SetErrorMode(SEM_FAILCRITICALERRORS);
// Do whatever we need to do that might cause an error
SetErrorMode(OldMode); //put things back the way they were
Under C++, detecting errors the right way needs the `GetLastError()' function, which we fortunately don't need to worry about here, since this is a Python question. In our case, Python's exception handling works fine. This, then, is the function I knocked together to check a drive letter for "readiness", all ready for copy-pasting if anyone else needs it:
import win32api
def testDrive( currentLetter ):
"""
Tests a given drive letter to see if the drive is question is ready for
access. This is to handle things like floppy drives and USB card readers
which have to have physical media inserted in order to be accessed.
Returns true if the drive is ready, false if not.
"""
returnValue = False
#This prevents Windows from showing an error to the user, and allows python
#to handle the exception on its own.
oldError = win32api.SetErrorMode( 1 ) #note that SEM_FAILCRITICALERRORS = 1
try:
freeSpace = win32file.GetDiskFreeSpaceEx( letter )
except:
returnValue = False
else:
returnValue = True
#restore the Windows error handling state to whatever it was before we
#started messing with it:
win32api.SetErrorMode( oldError )
return returnValue
I've been using this quite a bit the last few days, and it's been working beautifully for both floppies and USB card readers.
A few notes: pretty much any function needing disk access will work in the try block - all we're looking for in an exception due to the media not being present.
Also, while the python win32api package exposes all the functions we need, it dones't seem to have any of the flag constants. After a trip to the ancient bowels of MSDN, it turns out that SEM_FAILCRITICALERRORS is equal to 1, which makes our life awfully easy.
I hope this helps someone else with a similar problem!
You can compare len(os.listdir("path")) to zero to see if there are any files in the directory.
If you have pythonwin, does any of the information in this recipe help?
At a guess, "Availability" and "Status" may be worth looking at. Or you could test volume name, which I guess will be either 'X:' or '' if there is nothing in the drive. Or, heck, look for free space or total number of blocks.
You can use win32 functions via the excellent pywin32 (http://sourceforge.net/projects/pywin32/) for this purpose.
I suggest looking at the GetDiskFreeSpace function. You can check the free space on the target drive and continue based on that information.
As an alternative you can watch the changes of a directory or file with the ReadDirectoryChangesW function. You'll get notifications about file changes etc. But you have to check whether this works for you or not. You can look at this example foryourself:
http://timgolden.me.uk/python/downloads/watch_directory.py
Not sure about your platform, but SNMP might be the answer for you.
I hope this will solve your Problem.
import os
cmd = 'Powershell "Get-PhysicalDisk"'
drive_details = []
for each_line in os.popen(cmd).readlines():
try:
disk_info = str(each_line)
if int(disk_info[0]) >= 0:
each_line = each_line.split()
drive_details.append({
'sequence':each_line[0],
'disk_type':str(each_line[-7]),
'health_status':str(each_line[-4]),
'disk_size':str(each_line[-2]) + str(each_line[-1])
})
except:
pass
print(drive_details)