My first question on this subject was closed for not being specific enough so I'll try to tell you what I have tried so far that hasn't worked. I'm trying to move Excel 2010 workbooks to my new Mac mini Big Sur machine using Office 365. It is my first Mac so I am very new to the operating system and the differences from the Microsoft world. It is also the first time I'm using Office 365. I am hopeful you will be kind enough to help me. In my VBA macros I run Python scripts to scrape web pages for data. Usually I have the Python script create a .csv file which I then open in the VBA macro to read and populate the necessary cells in the workbook. This is the code I use in the PC version to call the Python script:
Kill strFileOfData
Set objWsh = VBA.CreateObject("WScript.shell")
strPython = """C:\Users\Phil\AppData\Local\Programs\Python\Python37-32\python.exe"""
strPyScript = """C:\Users\Phil\Documents\PyScripts\GetMarketsStks1-0.py"""
objWsh.Run strPython & strPyScript
Err = 0
Do Until Dir(StrFileOfData) <> 0
Application.Wait (Now() + TimeValue("0:00:01"))
Loop
It may not be the best but it works reliably on the PC. I delete the data file first, run the Python script, wait for the data file to be created, then continue.
I installed the latest version of Python on the Mac and rewrote the Python script to get more data and ran the Python script on Terminal to make sure it executed properly. It ran fine and created the .csv file correctly.
I then changed the code in the VBA macro to account for the different file structures. This is the new code:
Kill strFileOfData
Set objWsh = VBA.CreateObject("WScript.shell")
strPython = """/Library/Frameworks/Python.framework/Python"""
strPyScript = """/Users/minime/MyDocuments/Finance/GetHistory1-0.py"""
objWsh.Run strPython & strPyScript
Err = 0
Do Until Dir(StrFileOfData) <> 0
Application.Wait (Now() + TimeValue("0:00:01"))
Loop
When I run this on the Mac I get:
Run-Time error '429': ActiveX component can't create object
Suspecting this was a difference in the way shells are created and used I began researching how to call Python from Excel on the Mac. After considerable dead ends I found this thread from 4 years ago:
How can I launch an external python process from Excel 365 VBA on OSX?
I tried to simply plow ahead and follow the instructions. I learned a bit about AppleScript, added the folder: "~/Library/Application Scripts/com.microsoft.Excel/", created the AppleScript named PythonCommand.scpt and placed it in that folder. Since I couldn't find the path in the example I substituted what I thought to be the correct path, assuming it was due to the difference in MacOS from 4 years ago. My AppleScript looks like this:
on PythonCommand(pythonScript)
do shell script "/Library/Frameworks/Python.framework/Python" & pythonScript
end PythonCommand
I then added this code to the VBA macro:
strPyScript = """/Users/minime/MyDocuments/Finance/GetHistory1-0.py"""
Dim result As String
result = AppleScriptTask("PythonCommand.scpt", "PythonCommand", strPyScript)
When I ran the VBA macro. I got this message:
Run-time error '13': Type mismatch
I tried it again with single quotes instead of the triple quotes and got the same result.
I tried to work backward to make sure the pieces worked. I again ran the Python script from a Terminal window with no problem so the next step I tried was running the AppleScript from IDLE. I typed in this:
AppleScriptTask("PythonCommand.scpt", "PythonCommand","/Users/minime/MyDocuments/Finance/GetHistory1-0.py")
and got this result:
Traceback (most recent call last): File "<pyshell#0>", line 1, in
AppleScriptTask("PythonCommand.scpt", "PythonCommand","/Users/minime/My Documents/Finance/GetHistory1-0.py") NameError: name 'AppleScriptTask'
is not defined
After more research I tried this AppleScript in the editor:
do shell script "/Library/Frameworks/Python.framework/Versions/3.9/Python /Users/minime/MyDocuments/Finance/GetHistory1-0.py"
I got this result:
error "sh: /Library/Frameworks/Python.framework/Versions/3.9/Python:
cannot execute binary file" number 126
Next I tried this:
do shell script "Python /Users/minime/MyDocuments/Finance/GetHistory1-0.py"
and got this:
error "Traceback (most recent call last):
File \"/Users/philipackermann/MyDocuments/Finance/GetHistory1-0.py\", line 2, in <module>
from urllib.request import urlopen
ImportError: No module named request" number 1
This might actually be some progress! That import statement is the first line of code in my Python script but I have no clue why this got further than the last attempt and why this is running differently than the execution of the Python script in Terminal.
But I just noticed that in Terminal I enter Python3 so I tried this:
do shell script "Python3 /Users/minime/MyDocuments/Finance/GetHistory1-0.py"
and got a pop up with a script error:
and this:
After more reading about Terminal I realized there are hidden files on the Mac so I tried this AppleScript:
do shell script "/usr/local/bin/Python3 /Users/minime/MyDocuments/Finance/GetHistory1-0.py"
And it worked!! The .csv file is created successfully. So now I need to figure out how to call this from Excel. I found an article here:
https://stackoverflow.com/questions/38723420/how-to-simply-run-an-applescript-task-from-mac-excel-2016
That actually works. I converted the AppleScript into an app (just changed the extension from scpt to app) and moved it to the Applications folder, then put this code into the VBA macro:
ThisWorkbook.FollowHyperlink Address:="/Applications/RunGetHistory.app", NewWindow:=True
This worked! Of course it isn't passing any arguments or receiving any results. One issue though, like my previous PC version, the code doesn't wait for the app to finish so I need a way to check for it to be done and since I changed the Python Script to open the file in the beginning and write lines to it while scraping the web pages the file is created right away so this code which I was using in the PC version isn't sufficient:
Do Until Dir(strFileNmHistory) <> ""
Application.Wait (Now() + TimeValue("0:00:01"))
Loop
I suppose I could add a separate file written at the end of the Python script just to say it's done but that's a pretty lame hack. So technically I guess I could say I solved this and CAN run a Python script from Excel but I have to believe there is a better way and I'm sure this community has folks much smarter than me who can make this better. Has anyone been able to get the solution from 4 years ago to work? That might solve the problems of arguments, results, and waiting for the Python script to end. Any suggestions you can make would be most helpful. I have researched this issue extensively and some answers said that sandboxing is preventing Excel from running Python but if we can run an app that runs the Python script I guess that gets around it. If you can comment on the specific errors I encountered above I'll try different approaches and report back.
Please help.
Phil
SOLVED! ...and I learned a lot along the way! I'll try to lay out only the steps needed to make this work but I do tend to ramble, sorry. One issue that made this harder than it needed to be was a simple one: I was missing a space in the "do shell script" statement of the AppleScript. Here is the VBA code that works for me:
Result = AppleScriptTask("PythonCommand.scpt", "PythonCommandHandler", "/Users/minime/MyDocuments/Finance/PythonScripts/GetHistory1-0.py")
Here is the AppleScript code:
on PythonCommandHandler(pythonScript)
do shell script "/usr/local/bin/Python3 " & pythonScript
return "Handler succeeded! " & pythonScript
end PythonCommandHandler
Note the space after "Python3". Create the AppleScript in a convenient folder and copy it to
"/Users/minime/Library/Application Scripts/com.microsoft.Excel"
Trying to edit it in that folder is problematic, trust me. Move it there.
The Python script can be anywhere as long as you give the path in the AppleScriptTask command in the VBA code.
Be sure all references to .csv files point to an Excel sandbox folder. I used this one but I suspect others would work too:
"/Users/minime/Library/Containers/com.microsoft.Excel/Data/Documents"
See notes below if you can't find this folder. This worked on 7/1/2021! We'll see what happens when the new OS X comes out. The call to AppleScriptTask waits for the full execution of the Python script before continuing. I haven't figured out a good way to handle errors in the Python script yet.
Here are some notes on important things I learned along the way that might be helpful for first time Mac users like myself:
CHANGE YOUR VBA EDITOR FONT. Ok, not really necessary but the default font for the VBA editor on the Mac Excel 365 version I'm using was not a proportional font so things like the "as" part of the Dim statements wouldn't line up for me. Minor maybe but a big annoyance easily solved: In the editor, click the word "Excel" in the Menu Bar (the top row of words and icons.) Click "Preferences" and a window will pop up with the title "Options". Click the "Editor Format" tab and select "Courier New" from the "Font:" drop down. I changed the "Size:" to 16. Oh and by the way, "Command, shift, i" will step you through the VBA code like F8 does on the PC.
PUT SOME OF THE FILES IN THE SANDBOX. A word about the "sandbox" which I'm sure others could explain better than I. To improve security the sandbox concept is supposed to restrict the code from executing something outside it's expected area: e.g. you wouldn't want an error in your code (or malicious code) to change some system settings that had nothing to do with what you intended. I get that. Good concept. In ancient times (mainframes) we got a SOC 1 for that type of issue. The sandbox basically is a portion of the computer in which you are allowed to play. Don't try to go outside of it (there are exceptions I don't fully understand yet). On the Mac this means that if you even try to do anything with a .csv file in your VBA code that isn't in the sandbox you will get an alert saying you need to grant permission for that to happen. The Excel workbook with the VBA code doesn't need to be in the sandbox because what matters is that the Excel executable is in the sandbox (I assume) and is allowed to reach out to your workbook (an exception to the rule?) and do all the things you asked it to including run your VBA code, with limitations. The AppleScript also needs to be in the sandbox but the Python script does not (I don't know why). We are only interested here with the Excel sandbox, there are, apparently, different ones for different apps. Now a word about aliases. For simplicity consider your Excel sandbox to be here:
"/Users/philipackermann/Library/Containers/com.microsoft.Excel"
There is an alias located here:
"/Users/philipackermann/Library/Containers/com.microsoft.Excel/Data/Library/Application Scripts/com.microsoft.Excel"
Moving a file to one will, by magic, move the file to the other one as well. By the way, the first "com.microsoft.Excel" in that alias path will show up as "Microsoft Excel" in Finder if you are trying to look for it by walking the path from the beginning. Oh by the way, "Command, shift, ." will let you see the hidden folders and files in Finder. You'll need that to get started. If I understand this correctly, the folder icons in Finder with the little arrow in the lower left corner indicate that the folder is an alias (or has an alias pointing to it? I'm not sure). The folder I used for the .csv files is not an alias or doesn't have an alias) so I had to use the really long path. So the Excel workbook and the Python script don't need to be in the sandbox but the AppleScript and the .csv files do need to be there.
DON'T EDIT IN THE SANDBOX ALIAS FOLDER! Check out the link I mentioned in the comments above with VaughanR. He helped point the way for me to understand the alias issue better. Thank you VaughanR. Trying to edit in those folders was a madding exercise in futility. Save yourself the trauma and edit your AppleScript in a local folder and copy it to the sandbox folder. I did this by opening two Finder windows (one with the local folder and one with the sandbox folder) and holding down the Command and Option keys while dragging the AppleScript file from the local folder to the sandbox folder.
TEST THE APPLESCRIPT IN IN THE SCRIPT EDITOR. As I mentioned somewhere else, that pesky Error 5: can pop up for a number of reasons. In the script editor add this line above the PythonCommandHandler on statement:
"PythonCommandHandler("GetHistory1-0.py")"
and run it in the editor to be sure it is doing what you want it to do. That's how I found the problem with the missing space and tested the different paths to get it running.
MISC. EXCEL NOTES. If you are trying to bring Excel workbooks from a Windows environment to the Mac watch out for these issues I have run into so far: SORTS. I had hardcoded sorts in my VBA code that caused Excel to crash when I tried to run them on the Mac. I recorded a sort and that works fine in the code. There are NO USERFORMS on the Mac! I have not yet found an alternative to use.
Original question: I am running a python script
python script.py #runs/run_1/parameters.txt
Is there some way to access the string runs/run_1 in my script?
Actual question: I noticed that what I really need is a little different from the above. Independently from the directory from which I run the script, I need to get the location of parameters.txt.
For example, when I'm running the script from the directory runs/run_1 itself, I still need to get the path. I could do this with os.getcwd().
But when I'm running the code from PyCharm, I pass #/runs/run_1/parameters.txt as a parameter, while the script itself lives in some other directory. Here, I would need to read from sys.argv as suggested in the comments below.
For now I will have to do differentiate these cases with an if-statement checking whether the string before parameters.txt in python script.py #.../parameters.txt is empty. Is there a better way?
I am writing a watchman command with watchman-make and I'm at a loss when trying to access exactly what was changed in the directory. I want to run my upload.py script and inside the script I would like to access filenames of newly created files in /var/spool/cups-pdf/ANONYMOUS .
so far I have
$ watchman-make -p '/var/spool/cups-pdf/ANONYMOUS' -—run 'python /home/pi/upload.py'
I'd like to add another argument to python upload.py so I can have an exact filepath to the newly created file so that I can send the new file over to my database in upload.py,
I've been looking at the docs of watchman and the closest thing I can think to use is a trigger object. Please help!
Solution with watchman-wait:
Assuming project layout like this:
/posts/_SUBDIR_WITH_POST_NAME_/index.md
/Scripts/convert.sh
And the shell script like this:
#!/bin/bash
# File: convert.sh
SrcDirPath=$(cd "$(dirname "$0")/../"; pwd)
cd "$SrcDirPath"
echo "Converting: $SrcDirPath/$1"
Then we can launch watchman-wait like this:
watchman-wait . --max-events 0 -p 'posts/**/*.md' | while read line; do ./Scripts/convert.sh $line; done
When we changing file /posts/_SUBDIR_WITH_POST_NAME_/index.md the output will be like this:
...
Converting: /Users/.../Angular/dartweb_quickstart/posts/swift-on-android-building-toolchain/index.md
Converting: /Users/.../Angular/dartweb_quickstart/posts/swift-on-android-building-toolchain/index.md
...
watchman-make is intended to be used together with tools that will perform a follow-up query of their own to discover what they want to do as a next step. For example, running the make tool will cause make to stat the various deps to bring things up to date.
That means that your upload.py script needs to know how to do this for itself if you want to use it with watchman.
You have a couple of options, depending on how sophisticated you want things to be:
Use pywatchman to issue an ad-hoc query
If you want to be able to run upload.py whenever you want and have it figure out the right thing (just like make would do) then you can have it ask watchman directly. You can have upload.py use pywatchman (the python watchman client) to do this. pywatchman will get installed if the the watchman configure script thinks you have a working python installation. You can also pip install pywatchman. Once you have it available and in your PYTHONPATH:
import pywatchman
client = pywatchman.client()
client.query('watch-project', os.getcwd())
result = client.query('query', os.getcwd(), {
"since": "n:pi_upload",
"fields": ["name"]})
print(result["files"])
This snippet uses the since generator with a named cursor to discover the list of files that changed since the last query was issued using that same named cursor. Watchman will remember the associated clock value for you, so you don't need to complicate your script with state tracking. We're using the name pi_upload for the cursor; the name needs to be unique among the watchman clients that might use named cursors, so naming it after your tool is a good idea to avoid potential conflict.
This is probably the most direct way to extract the information you need without requiring that you make more invasive changes to your upload script.
Use pywatchman to initiate a long running subscription
This approach will transform your upload.py script so that it knows how to directly subscribe to watchman, so instead of using watchman-make you'd just directly run upload.py and it would keep running and performing the uploads. This is a bit more invasive and is a bit too much code to try and paste in here. If you're interested in this approach then I'd suggest that you take the code behind watchman-wait as a starting point. You can find it here:
https://github.com/facebook/watchman/blob/master/python/bin/watchman-wait
The key piece of this that you might want to modify is this line:
https://github.com/facebook/watchman/blob/master/python/bin/watchman-wait#L169
which is where it receives the list of files.
Why not triggers?
You could use triggers for this, but we're steering folks away from triggers because they are hard to manage. A trigger will run in the background and have its output go to the watchman log file. It can be difficult to tell if it is running, or to stop it running.
The interface is closer to the unix model and allows you to feed a list of files on stdin.
Speaking of unix, what about watchman-wait?
We also have a command that emits the list of changed files as they change. You could potentially stream the output from watchman-wait in your upload.py. This would make it have some similarities with the subscription approach but do so without directly using the pywatchman client.
I don't have a lot of experience coding so I'm sorry if this has been answered before; I couldn't find anything that helped.
I just completed a project on a Raspberry Pi that runs some RGB LED strips via PWM. I have a program that runs the lights and works fine with a few different modes (rainbow shifting, strobe, solid color) but with each new mode I add the program get longer and more convoluted. I would like to have each separate mode be its own script that gets started or stopped by a sort of master script. That way I could easily add a new mode by simply writing a separate program and adding it to the list on the master script instead of mucking around inside a giant program with everything in it and hoping I don't break something. I guess what I want is a simple way to start a python script with some specific setting (Determined by variables passed from the master script) and be able to kill that script when the master script receives the command to change modes.
Keeping your code modulable is indeed a good practice ! If your code is not Objet oriented, the best way is to create another python file (let's call it util.py) in the same directory as your "main". You can simply include util.py with the following command at the beginning of your main code :
import util
And then when you want to use a function that you've defined in your util.py file, juste use :
util.myFunction(param1, param2,...)
I'm trying to run an external, separate program from Python. It wouldn't be a problem normally, but the program is a game, and has a Python interpreter built into it. When I use subprocess.Popen, it starts the separate program, but does so under the original program's Python instance, so that they share the first Python console. I can end the first program fine, but I would rather have separate consoles (mainly because I have the console start off hidden, but it gets shown when I start the program from Python with subprocess.POpen).
I would like it if I could start the second program wholly on its own, as though I just 'double-clicked on it'. Also, os.system won't work because I'm aiming for cross-platform compatibility, and that's only available on Windows.
I would like it if I could start the second program wholly on its own, as though I just 'double-clicked on it'.
As of 2.7 and 3.3, Python doesn't have a cross-platform way to do this. A new shutil.open method may be added in the future (possibly not under that name); see http://bugs.python.org/issue3177 for details. But until then, you'll have to write your own code for each platform you care about.
Fortunately, what you're trying to do is simpler and less general than what shutil.open is ultimately hoped to provide, which means it's not that hard to code:
On OS X, there's a command called open that does exactly what you want: "The open command opens a file (or a directory or URL), just as if you had double-clicked the file's icon." So, you can just popen open /Applications/MyGame.app.
On Windows, the equivalent command is start, but unfortunately, that's part of the cmd.exe shell rather than a standalone program. Fortunately, Python comes with a function os.startfile that does the same thing, so just os.startfile(r'C:\Program Files\MyGame\MyGame.exe').
On FreeDesktop-compatible *nix systems (which includes most modern linux distros, etc.), there's a very similar command called xdg-open: "xdg-open opens a file or URL in the user's preferred application." Again, just popen xdg-open /usr/local/bin/mygame.
If you expect to run on other platforms, you'll need to do a bit of research to find the best equivalent. Otherwise, for anything besides Mac and Windows, I'd just try to popen xdg-open, and throw an error if that fails.
See http://pastebin.com/XVp46f7X for an (untested) example.
Note that this will only work to run something that actually can be double-clicked to launch in Finder/Explorer/Nautilus/etc. For example, if you try to launch './script.py', depending on your settings, it may just fire up a text editor with your script in it.
Also, on OS X, you want to run the .app bundle, not the UNIX executable inside it. (In some cases, launching a UNIX executable—whether inside an .app bundle or standalone—may work, but don't count on it.)
Also, keep in mind that launching a program this way is not the same as running it from the command line—in particular, it will inherit its environment, current directory/drive, etc. from the Windows/Launch Services/GNOME/KDE/etc. session, not from your terminal session. If you need more control over the child process, you will need to look at the documentation for open, xdg-open, and os.startfile and/or come up with a different solution.
Finally, just because open/xdg-open/os.startfile succeeds doesn't actually mean that the game started up properly. For example, if it launches and then crashes before it can even create a window, it'll still look like success to you.
You may want to look around PyPI for libraries that do what you want. http://pypi.python.org/pypi/desktop looks like a possibility.
Or you could look through the patches in issue 3177, and pick the one you like best. As far as I know, they're all pure Python, and you can easily just drop the added function in your own module instead of in os or shutil.
As a quick hack, you may be able to (ab)use webbrowser.open. "Note that on some platforms, trying to open a filename using this function, may work and start the operating system’s associated program. However, this is neither supported nor portable." In particular, IIRC, it will not work on OS X 10.5+. However, I believe that making a file: URL out of the filename actually does work on OS X and Windows, and also works on linux for most, but not all, configurations. If so, it may be good enough for a quick&dirty script. Just keep in mind that it's not documented to work, it may break for some of your users, it may break in the future, and it's explicitly considered abuse by the Python developers, so I wouldn't count on it for anything more serious. And it will have the same problems launching 'script.py' or 'Foo.app/Contents/MacOS/foo', passing env variables, etc. as the more correct method above.
Almost everything else in your question is both irrelevant and wrong:
It wouldn't be a problem normally, but the program is a game, and has a Python interpreter built into it.
That doesn't matter. If the game were writing to stdout from C code, it would do the exact same thing.
When I use subprocess.Popen, it starts the separate program, but does so under the original program's Python instance
No it doesn't. It starts an entirely new process, whose embedded Python interpreter is an entirely new instance of Python. You can verify that by, e.g., running a different version of Python than the game embeds.
so that they share the first Python console.
No they don't. They may share the same tty/cmd window, but that's not the same thing.
I can end the first program fine, but I would rather have separate consoles (mainly because I have the console start off hidden, but it gets shown when I start the program from Python with subprocess.POpen).
You could always pipe the child's stdout and stderr to, e.g., a logfile, which you could then view separately from the parent process's output, if you wanted to. But I think this is going off on a tangent that has nothing to do with what you actually care about.
Also, os.system won't work because I'm aiming for cross-platform compatibility, and that's only available on Windows.
Wrong; os.system is available on "Unix, Windows"--which is probably everywhere you care about. However, it won't work because it runs the child program in a subshell of your script, using the same tty. (And it's got lots of other problems—e.g., blocking until the child finishes.)
When I use subprocess.Popen, it starts the separate program, but does so under the original program's Python instance...
Incorrect.
... so that they share the first Python console.
This is the crux of your problem. If you want it to run in another console then you must run another console and tell it to run your program instead.
... I'm aiming for cross-platform compatibility ...
Sorry, there's no cross-platform way to do it. You'll need to run the console/terminal appropriate for the platform.