This is more of a convenience than a real problem, but the project I'm working on has a lot of separate files, and I want to basically be able to run any of those files (that all basically only contain classes) to run the main file.
Now in the middle of writing the first sentence of this question, I tried just importing main.py into each file, and that seemed to work fine and dandy, but I cant help but feeling that:
it might cause problems, and
that I had problems with circular imports before and I am somewhat surprised that nothing came up.
First let me say: this is most likely a bad idea, and it's definitely not at all standard. It will likely lead to confusion and frustration down the road.
However, if you really want to do it, you can put:
if __name__ == "__main__":
from mypackage import main
main.run()
Which, assuming mypackage.main.run() is your main entry point, will let you run any file you want as if it were the main file.
You may still hit issues issues with circular imports, and those will be completely unavoidable, unless mypackage.main doesn't import anything… Which would make it fairly useless :)
As an alternative, you may wish to use a testing framework like doctest or unittest, then configure your IDE to run the unit tests from a hotkey. This way you're automatically building the repeatable tests as you develop your code.
Related
I'm developing a personal app which envolves web scraping stuffs, and inside its folders there's some files that need multiprocessing tools to makes things go more efficient as any other web scraping software. I know that Windows lacks of some aspects available on Linux such as fork, and thus when we want to instanciate a pool process in our program we need to make sure that the process code is enclosed by if __name__ == "__main__":.
But something is bothering me. Here's the scenario (there are several of them currently on my program). In my file main I'm calling import web_scraping_x, which contains a multiprocess pool. The question is: How do I protect my web_scraping_x code without messing up everything inside of it?
Here's some pseudo-code (for the better good of comprehension):
"__main__"
from web_scraping_x import scraping_manager
instructions = ["some list or dict"]
data = scraping_manager(instructions)
and
"web_scraping_x"
def escraping_manager(instructions):
if instructions is True:
scraping_a(instructios)
def scraping_a(instructions):
"do a lot of work ur proletariat"
I'm trying not to come up with a solution as bad as the problem on Windows, even though it may not be possible, such as importing my module with a changed __name__ ... so any tips humans I really appreciate. If needed, I could show my code, but I think it's just gonna make things more difficulty to share with you.
Say, I have two files demo.py
# demo.py
from pathlib import Path
for i in range(5):
exec(Path('another_file.txt').read_text())
and another_file.txt (note the indent)
print(i)
Is it possible to make python demo.py run?
N.B. This is useful when using Page (or wxformbuilder or pyqt's designer) to generate a GUI layout where skeletons of callback functioins are automatically generated. The skeletons will have to be modified, at the same time, each iteration overwrites the skeletons -- code snippets will have to be copied back. Anyway, you know what I am talking about if you have used any of Page or wxformbuilder or pyqt's designer.
You can solve the basic problem by removing the indent:
from pathlib import Path
import textwrap
for i in range(5):
exec(textwrap.dedent(Path('another_file.txt').read_text()))
There are still two rather major problems with this:
There are serious security implications here. You're running code without including it in your project. The idea that you can "worry about security and other issues later" will cause you pain at a later point. You'll see similar advice on this site with avoiding SQL injection. That later date may never come, and even if it does, there's a very real chance you won't remember or correctly identify all of the problems. It's absolutely better to avoid the problems in the first place.
Also, with dynamic code like this, you run the very real risk of running into a syntax error with a call stack that shows you nothing about where the code is from. It's not bad for simple cases like this, but as you add more and more complexity to a project like this, you'll likely find you're adding more support to help you debug issues you run into, rather than spending your time adding features.
And, combining these two problems can be fun. It's contrived, but if you changed the for loop to a while loop like this:
i = 0
while i < 5:
exec(textwrap.dedent(Path('another_file.txt').read_text()))
i += 1
And then modified the text file to be this:
print(i)
i += 1
It's trivial to understand why it's no longer operating the 5 times you expect, but as both "sides" of this project get more complex, figuring out the complex interplay between the elements will get much harder.
In short, don't use eval. Future you will thank past you for making your life easier.
This problem involves the collision of several problems, all of which I understand only somewhat well, but I include them together because they could all be the entry point for a solution. Here is the best description I can give.
I have an app, in python. (I imagine I could theoretically solve all of these problems by learning Cocoa and ObjectiveC, but that seems like QUITE a lift, for this problem -- AND, as noted below, this problem may not actually be related to python, really, at all. I just don't know.) A CORE feature of this app is to trigger a minigame, with a hotkey -- meaning, the hotkey itself is fundamental to the desired functionality. And furthermore, I would really like to package this app, to let other people use it. (Locally, it works great! Hey!)
The problem starts with the fact that adding the hotkey -- which I am doing with
import keyboard
keyboard.add_hotkey('windows+shift+y', trigger_minigame)
-- requires root access. Due to DIRE WARNINGS in another SO post Forcing a GUI application to run as root (which, honestly, I only vaguely understand), I would like to grant that access to ONLY this part of the program. I IMAGINE, such an approach would look something like this:
# needs_root.py
import keyboard
from shouldnt_have_root import trigger_minigame
keyboard.add_hotkey('windows+shift+y', trigger_minigame)
# shouldnt_have_root.py
def minigame():
buncha pygame, GUI stuff (which is dangerous???)
def trigger_minigame():
adds event to minigame's event queue
# bash script
sudo python needs_root.py
HOWEVER -- there are several major challenges!
The biggest is that I don't even know if THAT is safe, since I don't know how security and permissions (especially with imports) works at all! And more generally, how dangerous are the imports? It appears that I may in fact have to import substantially more, to make it clear what event queue the trigger is adding an event TO -- and I don't know how to have that communication happen, while still isolating the GUI parts (or generally dangerous ones) from unnecessary and hazardous access.
There's another layer too though; packaging it through pyinstaller means that I can't target the scripts directly, because they'll have been turned into binaries, but according to THIS answer Packaging multiple scripts in PyInstaller it appears I can just target the binaries instead, i.e. have the first binary call
osascript -e 'do shell script "python needs_root_binary" with admin.'
to get the user to bless only the necessary part, but I don't know if that will put OTHER obstacles, or vulnerabilities (or inter-file communication difficulties), in the way.
LAST, I could try STARTING as root, and then switching away from it, as soon as the hotkey is set (and before anything else happens) -- but would that be safe? I'm still worried about the fact that it involves running sudo on the whole app.
In any event --
is this as big a mess as it feels?
How do I give root access to only a piece of a packaged .app, that I've written in python?
I'd advice You to:
enable the root access,
write the script,
disable the root access
as it's closer described in here.
The Pyinstaller is another chapter. When I was making software requiring usage of hotkeys, I was forced to use another than keyboard, because it wasn't working properly on PC without Python, therefore I made a hotkey with tkinter built-in function canvas.bind() (more info here).
Hopefully I helped.
You can not run a specific Python function as root, only the Python process executing your script can be run with elevated permissions.
So my answer is: your problem as described is unsolvable.
Often I write command line utilities that are only meant to be run as main. For example, I might have a file that looks like this:
#!/usr/bin/env python
if __name__ == '__main__':
import sys
# do stuff
In other words, there is nothing going on that isn't under the if statement checking that this file is being run as main. I tried importing a file like this to see what would happen, and the import was successful.
So as I expected, one is allowed to import files like this, but what is the convention surrounding this practice? Is one supposed to throw an error telling the user that there is nothing to be imported? Or if all the contents of the file are supposed to be run as main, does one need to check if the program is being run as main? Or is the conditional not necessary?
Also, if I have import statements, should they be at the top of the file, or under the conditional? If the modules are only being used under the conditional, it would seem to me that they should be imported under the conditional and not at the top of the file.
If you are writing simple utilities that you are entirely certain that you will never import as a module in another program, then you really do not need to include the if __name__ == '__main__' stuff. The fundamental point of that construct is to allow a module to be developed that can both be imported as a module for use, and run as a stand-alone program for some other purpose. For example, if you had a module and had some test vectors you wanted to run on it regularly, you would put the trigger mechanism for your test vectors in the if __name__ block.
Another example might be if you have a stand-alone program that you develop, that would also provide useful functions for others. If you have a look at the pip module, this is an excellent example of this technique.
Google has a Python tutorial, and they describe boilerplate code as "unfortunate" and provide this example:
#!/usr/bin/python
# import modules used here -- sys is a very standard one
import sys
# Gather our code in a main() function
def main():
print 'Hello there', sys.argv[1]
# Command line args are in sys.argv[1], sys.argv[2] ..
# sys.argv[0] is the script name itself and can be ignored
# Standard boilerplate to call the main() function to begin
# the program.
if __name__ == '__main__':
main()
Now, I've heard boilerplate code being described as "seemingly repetitive code that shows up again and again in order to get some result that seems like it ought to be much simpler" (example).
Anyways, in Python, the part considered "boilerplate" code of the example above was:
if __name__ == '__main__':
main()
Now, my questions are as follows:
1) Does boilerplate code in Python (like the example provided) take on the same definition as the definition I provided? If so, why?
2) Is this code even necessary? It seems to me like the code runs whether or not there's a main method. What makes using this code better? Is it even better?
3) Why do we use that code and what service does it provide?
4) Does this occur throughout Python? Are there other examples of "boilerplate code"?
Oh, and just an off topic question: do you need to import sys to use command line arguments in Python? How does it handle such arguments if its not there?
It is repetitive in the sense that it's repeated for each script that you might execute from the command line.
If you put your main code in a function like this, you can import the module without executing it. This is sometimes useful. It also keeps things organized a bit more.
Same as #2 as far as I can tell
Python is generally pretty good at avoiding boilerplate. It's flexible enough that in most situations you can write code to produce the boilerplate rather then writing boilerplate code.
Off topic question:
If you don't write code to check the arguments, they are ignored.
The reason that the if __name__ == "__main__": block is called boilerplate in this case is that it replicates a functionality that is automatic in many other languages. In Java or C++, among many others, when you run your code it will look for a main() method and run it, and even complain if it's not there. Python runs whatever code is in your file, so you need to tell it to run the main() method; a simple alternative would be to make running the main() method the default functionality.
So, if __name__ == "__main__": is a common pattern that could be shorter. There's no reason you couldn't do something different, like:
if __name__ == "__main__":
print "Hello, Stack Overflow!"
for i in range(3):
print i
exit(0)
This will work just fine; although my example is a little silly, you can see that you can put whatever you like there. The Python designers chose this behavior over automatically running the main() method (which may well not exist), presumably because Python is a "scripting" language; so you can write some commands directly into a file, run it, and your commands execute. I personally prefer it the Python way because it makes starting up in Python easier for beginners, and it's always nice to have a language where Hello World is one line.
The reason you use an "if main" check is so you can have a module that runs some part of its code at toplevel (to create the things – constants, functions, or classes – it exports), and some part only when executed as a script (e.g. unit tests for its functionality).
The reason the latter code should be wrapped in a function is because local variables of the main() block would leak into the module's scope.
Now, an alternate design could be that a file executed as a script would have to declare a function named, say, __main__(), but that would mean adding a new magic function name to the language, while the __name__ mechanism is already there. (And couldn't be removed, because every module has to have a __name__, and a module executed as a script has to have a "special" name because of how module names are assigned.) Introducing two mechanisms to do the same thing just to get rid of two lines of boilerplate – and usually two lines of boilerplate per application – just doesn't seem worth it.
You don't need to add a if __name__ == '__main__' for one off scripts that aren't intended to be a part of a larger project. See here for a great explanation. You only need it if you want to run the file by itself AND include it as a module along with other python files.
If you just want to run one file, you can have zero boilerplate:
print 1
and run it with $ python your_file.py
Adding the shebang line #!/usr/bin/python and running chmod +x print_one.py gets you the ability to run with
./print_one.py
Finally, # coding: utf-8 allows you to add unicode to your file if you want to put ❤'s all over the place.
1) main boilerplate is common, but cannot be any simpler
2) main() is not called without the boilerplate
3) the boilerplate allows module usage both as a standalone script, and as a library in other programs
4) it’s very common. doctest is another one.
Train to become a Python guru…and good luck with the thesis! ;-)
Let’s take a moment to see what happened when you called import sys:
Python looks at a list and brings in the sys module
It finds the argv function and runs it
So, what’s happening here?
A function written elsewhere is being used to perform certain operations within the scope of the current program. Programming in this fashion has a lots of benefits. It separates the logic from actual labour.
Now, as far as the boilerplate is concerned, there are two parts:
the program itself (the logic), defined under main, and
the call part that checks if main exists
You essentially write your program under main, using all the functions you defined just before defining main (or elsewhere), and let Python look for main.
I am equally confused by what the tutorial means by "boilerplate code": does it mean that this code can be avoided in a simple script? Or it is a criticism towards Python features that force the use of this syntax? Or even an invitation to use this "boilerplate" code?
I don't know, however, after many years of Python programming, I have at least clear what the different syntaxes do, even if I am probably still not sure on what is the best way of doing it.
Often you want to put at the end of the script code for tests or code that want to execute, but this has some implications/side-effects:
the code gets executed even when the script is imported, that it is rarely what is wanted.
variables and values in the code are defined and exported in the calling namespace
the code at the end of the script can be executed by calling the script (python script.py) or by running from ipython shell (%run script.py), but there is no way to run it from other scripts.
The most basic mechanism to avoid to execute following code in all conditions, is the syntax:
if __name__ == '__main__':
which makes the code run only if the script is called or run, avoiding problem 1. The other two points still hold.
The "boilerplate" code with a separate main() function, adds a further step, excluding also above points 2 and 3, so for example you can call a number of tests from different scripts, that sometimes can take another level (e.g.: a number of functions, one for each test, so they can be individually be called from outside, and a main that calls all test functions, without needs to know from outside which one they are).
I add that the main reason I find this structures often unsatisfying, apart from its complexity, is that sometimes I would like to maintain point 2 and I lose this possibility if the code is moved to a separate function.