I have decided to complete cs231 course and do its assignment. I happily watched the first 2 videos of the course and now I had to solve the first assignments.
I followed the guidelines step by step which was shown in the video in this link:
https://cs231n.github.io/setup-instructions/
Then, when I run the first cell, which is not the cell shown in the video but nonetheless it's in the assignments1 file which I downloaded from their site, I get a nasty error which has paralyzed me four a couple of hours. I'd be happy if anyone could respond.
IF you take a look at my picture, you'll see that files are added in the google drive, but surprisingly, it gives an error out of nowhere.
Thanks.
===========================================================================
Update:
Here is the snapshot of the video provided to guide students how to setup their google colab (in that link).
As you can see, in their vide the first chunk of code specifies their working directory but in the file that they have uploaded as their assignment1, they have not done so!
cs231n is a virtual environment according to documentation from the link u provided.
Every time you want to work on assignment you should activate that environment by source ~/cs231n/bin/activate
This happens because cs231n is not in the current path. Add these line at the beginning of the code :
import sys
sys.path.append('./cs231n')
I had the same problem. I ran the first block which imports the module 'sys' and appends the file path. After that I left and came back after 5 hours. When I log in again, I didn't run the first block. Then I came with the problem. That's the question! The new Google Drive allocated is different. Therefore, the python doesn't know where the file path is. So I tried to run the first block again and it really works. That's why you succeed by downloading the assignment1 file and running it again.
This is my opinion. Hope this can help you!
This was ridiculous!
After dealing with it for couple of hours (trying to discover the current directory using cwd command and cd) with no avail, I decided to clear ALL files from my google drive and download the assignment1 file again
Suprisingly, this time the colab page had the code in its beginning. I still wonder how that happened.
Related
My first question on this subject was closed for not being specific enough so I'll try to tell you what I have tried so far that hasn't worked. I'm trying to move Excel 2010 workbooks to my new Mac mini Big Sur machine using Office 365. It is my first Mac so I am very new to the operating system and the differences from the Microsoft world. It is also the first time I'm using Office 365. I am hopeful you will be kind enough to help me. In my VBA macros I run Python scripts to scrape web pages for data. Usually I have the Python script create a .csv file which I then open in the VBA macro to read and populate the necessary cells in the workbook. This is the code I use in the PC version to call the Python script:
Kill strFileOfData
Set objWsh = VBA.CreateObject("WScript.shell")
strPython = """C:\Users\Phil\AppData\Local\Programs\Python\Python37-32\python.exe"""
strPyScript = """C:\Users\Phil\Documents\PyScripts\GetMarketsStks1-0.py"""
objWsh.Run strPython & strPyScript
Err = 0
Do Until Dir(StrFileOfData) <> 0
Application.Wait (Now() + TimeValue("0:00:01"))
Loop
It may not be the best but it works reliably on the PC. I delete the data file first, run the Python script, wait for the data file to be created, then continue.
I installed the latest version of Python on the Mac and rewrote the Python script to get more data and ran the Python script on Terminal to make sure it executed properly. It ran fine and created the .csv file correctly.
I then changed the code in the VBA macro to account for the different file structures. This is the new code:
Kill strFileOfData
Set objWsh = VBA.CreateObject("WScript.shell")
strPython = """/Library/Frameworks/Python.framework/Python"""
strPyScript = """/Users/minime/MyDocuments/Finance/GetHistory1-0.py"""
objWsh.Run strPython & strPyScript
Err = 0
Do Until Dir(StrFileOfData) <> 0
Application.Wait (Now() + TimeValue("0:00:01"))
Loop
When I run this on the Mac I get:
Run-Time error '429': ActiveX component can't create object
Suspecting this was a difference in the way shells are created and used I began researching how to call Python from Excel on the Mac. After considerable dead ends I found this thread from 4 years ago:
How can I launch an external python process from Excel 365 VBA on OSX?
I tried to simply plow ahead and follow the instructions. I learned a bit about AppleScript, added the folder: "~/Library/Application Scripts/com.microsoft.Excel/", created the AppleScript named PythonCommand.scpt and placed it in that folder. Since I couldn't find the path in the example I substituted what I thought to be the correct path, assuming it was due to the difference in MacOS from 4 years ago. My AppleScript looks like this:
on PythonCommand(pythonScript)
do shell script "/Library/Frameworks/Python.framework/Python" & pythonScript
end PythonCommand
I then added this code to the VBA macro:
strPyScript = """/Users/minime/MyDocuments/Finance/GetHistory1-0.py"""
Dim result As String
result = AppleScriptTask("PythonCommand.scpt", "PythonCommand", strPyScript)
When I ran the VBA macro. I got this message:
Run-time error '13': Type mismatch
I tried it again with single quotes instead of the triple quotes and got the same result.
I tried to work backward to make sure the pieces worked. I again ran the Python script from a Terminal window with no problem so the next step I tried was running the AppleScript from IDLE. I typed in this:
AppleScriptTask("PythonCommand.scpt", "PythonCommand","/Users/minime/MyDocuments/Finance/GetHistory1-0.py")
and got this result:
Traceback (most recent call last): File "<pyshell#0>", line 1, in
AppleScriptTask("PythonCommand.scpt", "PythonCommand","/Users/minime/My Documents/Finance/GetHistory1-0.py") NameError: name 'AppleScriptTask'
is not defined
After more research I tried this AppleScript in the editor:
do shell script "/Library/Frameworks/Python.framework/Versions/3.9/Python /Users/minime/MyDocuments/Finance/GetHistory1-0.py"
I got this result:
error "sh: /Library/Frameworks/Python.framework/Versions/3.9/Python:
cannot execute binary file" number 126
Next I tried this:
do shell script "Python /Users/minime/MyDocuments/Finance/GetHistory1-0.py"
and got this:
error "Traceback (most recent call last):
File \"/Users/philipackermann/MyDocuments/Finance/GetHistory1-0.py\", line 2, in <module>
from urllib.request import urlopen
ImportError: No module named request" number 1
This might actually be some progress! That import statement is the first line of code in my Python script but I have no clue why this got further than the last attempt and why this is running differently than the execution of the Python script in Terminal.
But I just noticed that in Terminal I enter Python3 so I tried this:
do shell script "Python3 /Users/minime/MyDocuments/Finance/GetHistory1-0.py"
and got a pop up with a script error:
and this:
After more reading about Terminal I realized there are hidden files on the Mac so I tried this AppleScript:
do shell script "/usr/local/bin/Python3 /Users/minime/MyDocuments/Finance/GetHistory1-0.py"
And it worked!! The .csv file is created successfully. So now I need to figure out how to call this from Excel. I found an article here:
https://stackoverflow.com/questions/38723420/how-to-simply-run-an-applescript-task-from-mac-excel-2016
That actually works. I converted the AppleScript into an app (just changed the extension from scpt to app) and moved it to the Applications folder, then put this code into the VBA macro:
ThisWorkbook.FollowHyperlink Address:="/Applications/RunGetHistory.app", NewWindow:=True
This worked! Of course it isn't passing any arguments or receiving any results. One issue though, like my previous PC version, the code doesn't wait for the app to finish so I need a way to check for it to be done and since I changed the Python Script to open the file in the beginning and write lines to it while scraping the web pages the file is created right away so this code which I was using in the PC version isn't sufficient:
Do Until Dir(strFileNmHistory) <> ""
Application.Wait (Now() + TimeValue("0:00:01"))
Loop
I suppose I could add a separate file written at the end of the Python script just to say it's done but that's a pretty lame hack. So technically I guess I could say I solved this and CAN run a Python script from Excel but I have to believe there is a better way and I'm sure this community has folks much smarter than me who can make this better. Has anyone been able to get the solution from 4 years ago to work? That might solve the problems of arguments, results, and waiting for the Python script to end. Any suggestions you can make would be most helpful. I have researched this issue extensively and some answers said that sandboxing is preventing Excel from running Python but if we can run an app that runs the Python script I guess that gets around it. If you can comment on the specific errors I encountered above I'll try different approaches and report back.
Please help.
Phil
SOLVED! ...and I learned a lot along the way! I'll try to lay out only the steps needed to make this work but I do tend to ramble, sorry. One issue that made this harder than it needed to be was a simple one: I was missing a space in the "do shell script" statement of the AppleScript. Here is the VBA code that works for me:
Result = AppleScriptTask("PythonCommand.scpt", "PythonCommandHandler", "/Users/minime/MyDocuments/Finance/PythonScripts/GetHistory1-0.py")
Here is the AppleScript code:
on PythonCommandHandler(pythonScript)
do shell script "/usr/local/bin/Python3 " & pythonScript
return "Handler succeeded! " & pythonScript
end PythonCommandHandler
Note the space after "Python3". Create the AppleScript in a convenient folder and copy it to
"/Users/minime/Library/Application Scripts/com.microsoft.Excel"
Trying to edit it in that folder is problematic, trust me. Move it there.
The Python script can be anywhere as long as you give the path in the AppleScriptTask command in the VBA code.
Be sure all references to .csv files point to an Excel sandbox folder. I used this one but I suspect others would work too:
"/Users/minime/Library/Containers/com.microsoft.Excel/Data/Documents"
See notes below if you can't find this folder. This worked on 7/1/2021! We'll see what happens when the new OS X comes out. The call to AppleScriptTask waits for the full execution of the Python script before continuing. I haven't figured out a good way to handle errors in the Python script yet.
Here are some notes on important things I learned along the way that might be helpful for first time Mac users like myself:
CHANGE YOUR VBA EDITOR FONT. Ok, not really necessary but the default font for the VBA editor on the Mac Excel 365 version I'm using was not a proportional font so things like the "as" part of the Dim statements wouldn't line up for me. Minor maybe but a big annoyance easily solved: In the editor, click the word "Excel" in the Menu Bar (the top row of words and icons.) Click "Preferences" and a window will pop up with the title "Options". Click the "Editor Format" tab and select "Courier New" from the "Font:" drop down. I changed the "Size:" to 16. Oh and by the way, "Command, shift, i" will step you through the VBA code like F8 does on the PC.
PUT SOME OF THE FILES IN THE SANDBOX. A word about the "sandbox" which I'm sure others could explain better than I. To improve security the sandbox concept is supposed to restrict the code from executing something outside it's expected area: e.g. you wouldn't want an error in your code (or malicious code) to change some system settings that had nothing to do with what you intended. I get that. Good concept. In ancient times (mainframes) we got a SOC 1 for that type of issue. The sandbox basically is a portion of the computer in which you are allowed to play. Don't try to go outside of it (there are exceptions I don't fully understand yet). On the Mac this means that if you even try to do anything with a .csv file in your VBA code that isn't in the sandbox you will get an alert saying you need to grant permission for that to happen. The Excel workbook with the VBA code doesn't need to be in the sandbox because what matters is that the Excel executable is in the sandbox (I assume) and is allowed to reach out to your workbook (an exception to the rule?) and do all the things you asked it to including run your VBA code, with limitations. The AppleScript also needs to be in the sandbox but the Python script does not (I don't know why). We are only interested here with the Excel sandbox, there are, apparently, different ones for different apps. Now a word about aliases. For simplicity consider your Excel sandbox to be here:
"/Users/philipackermann/Library/Containers/com.microsoft.Excel"
There is an alias located here:
"/Users/philipackermann/Library/Containers/com.microsoft.Excel/Data/Library/Application Scripts/com.microsoft.Excel"
Moving a file to one will, by magic, move the file to the other one as well. By the way, the first "com.microsoft.Excel" in that alias path will show up as "Microsoft Excel" in Finder if you are trying to look for it by walking the path from the beginning. Oh by the way, "Command, shift, ." will let you see the hidden folders and files in Finder. You'll need that to get started. If I understand this correctly, the folder icons in Finder with the little arrow in the lower left corner indicate that the folder is an alias (or has an alias pointing to it? I'm not sure). The folder I used for the .csv files is not an alias or doesn't have an alias) so I had to use the really long path. So the Excel workbook and the Python script don't need to be in the sandbox but the AppleScript and the .csv files do need to be there.
DON'T EDIT IN THE SANDBOX ALIAS FOLDER! Check out the link I mentioned in the comments above with VaughanR. He helped point the way for me to understand the alias issue better. Thank you VaughanR. Trying to edit in those folders was a madding exercise in futility. Save yourself the trauma and edit your AppleScript in a local folder and copy it to the sandbox folder. I did this by opening two Finder windows (one with the local folder and one with the sandbox folder) and holding down the Command and Option keys while dragging the AppleScript file from the local folder to the sandbox folder.
TEST THE APPLESCRIPT IN IN THE SCRIPT EDITOR. As I mentioned somewhere else, that pesky Error 5: can pop up for a number of reasons. In the script editor add this line above the PythonCommandHandler on statement:
"PythonCommandHandler("GetHistory1-0.py")"
and run it in the editor to be sure it is doing what you want it to do. That's how I found the problem with the missing space and tested the different paths to get it running.
MISC. EXCEL NOTES. If you are trying to bring Excel workbooks from a Windows environment to the Mac watch out for these issues I have run into so far: SORTS. I had hardcoded sorts in my VBA code that caused Excel to crash when I tried to run them on the Mac. I recorded a sort and that works fine in the code. There are NO USERFORMS on the Mac! I have not yet found an alternative to use.
An intern of mine is trying to use deeptools' bamCoverage function, and it throws a '/my/dir/data.bam' file does not exist error, despite the file being there. Yes, the file does exist, we can manipulate it using bash commands just fine so there's no real reason for it to throw that error.
According to this thread, it could be an issue with pysam or python. Both are fully up to date. Do you know how I could investigate issues with pysam or python IO further?
All of this is happening on a server that our team uses. Could it be an issue with python's paths for his user session?
For reference, here is the code I'm running in my bash terminal. It's pretty basic:
my.name#server:/mnt/data1/my.name/PROJET_X/DATA/BIGWIG$ bamCoverage -p 8 -b /mnt/data1/my.name/PROJET_X/DATA/BAM_sort/G0-G00.Inputs.sorted.bam -o /mnt/data1/my.name/PROJET_X/DATA/BIGWIG/G0-G00.Inputs.RPGC.bw --normalizeUsing RPGC --effectiveGenomeSize 2913022398 -bs 10
The file '/mnt/data1/my.name/PROJET_X/DATA/BAM_sort/G0-G00.Inputs.sorted.bam' does not exist
my.name#server:/mnt/data1/my.name/PROJET_X/DATA/BIGWIG$ ll /mnt/data1/my.name/PROJET_X/DATA/BAM_sort/G0-G00.Inputs.sorted.bam
-rwxr-xr-x 1 my.name bioinfo 4171366400 juin 22 10:14 /mnt/data1/my.name/PROJET_X/DATA/BAM_sort/G0-G00.Inputs.sorted.bam
I have used this line probably 100 times this past year, but now it won't find the file.
I've also tried installing deeptools in my home directory using conda, but that gives the same errors.
EDIT: Apparently, it was just a problem with that one file. bamCoverage will work on other data files. It would be nice if deeptools would tell you that instead of just "file not found"...
It would be helpful to see what code is being run here - please,
provide a minimal reproducible example, including the full output / error, if at all possible.
Right now, I can think of 4 possible solutions to common problems:
As mentioned in the comments, providing the full path might be one solution. You can get the full path by using the pwd bash command in the folder where the file is stored and then add the file name at the end of the string.
Another thing that sometimes helps with relative (local) paths is to start the path with a ".", so the path would be something like this: './my/dir/data.bam'.
Another problem I've encountered (mostly when running things on Windows environments) was that the backslashes had to be used instead: '\my\dir\data.bam'
For the sake of completeness here, I'll mention that especially with beginning programmers, the path is often erroneously provided without commas (to make it a string), but that does not seem to be the case here.
Hope this helps!
Spyder crashed at the last minute and I had force quit the programme as it wasn't responding at all. After reopening it, I am not able to find my file that I was working on, i hadn't saved it and had quite a bit of code written in it. It will be a huge help if someone knows a way to recover it, and can guide me through it.
Thanks alot!
hist -g will show console history logs. You can recover your code from history if you had run your code earlier.
I experienced a similar situation where I had added code to existing script that later wasn't saved as I expected it to be. I was able to recover my code from the history log (copy and paste). Not a perfect solution, but much better than having to re-write the code.
I am on Spyder 3.3.4
I'll comment to help in case (someone crazy like me) clicked the discard button when Spyder asked if you wanted to restore the file.
I found the following solution:
open a new terminal
cd /home/pc/.config/spyder-py3/
code .
open the history.py file
Heart attack stopped successfully!
Anyway, you will no longer find the file you discarded in the autosave folder, however, in the history.py file are saved the last 1000 lines that you executed, in my case the last 30 were enough.
I hope it helps
Just for the people who still want to know the solution, close and open again Spyder application. It will ask you to recover the file. This worked for me today.
Go to C:\path.spyder-py3
You will have directories with the name autosave, you can find your autosaved files here. These are same ones that come up when you close and open spyder. It loads it from this autosave dir.
Another method is to check the history.py file which stores commands run in spyder console. However, there is a limit to this history. The old ones will get deleted.
I'm trying to run an automated test in Ride.py. This test works on my colleague's computer but for some reason does not work on mine. The test starts but at a certain point i get the following error:
[ ERROR ] Calling method 'start_keyword' of listener 'C:\Python27\lib\site-packages\robotide\contrib\testrunner\TestRunnerAgent.py' failed: IndexError: list index out of range
The interesting part is that this error occurs on the same spot ever time, but with a different test it happens at a different time.
I tried to google several things and nothing worked. One solution suggested there was a '#' commented somewhere and this caused the crash. I looked but I don't see a '#' commented anywhere.
Another suggestion lead me to believe my testrunneragent.py file must have been installed wrong. I went online to find the file and replaced it. This did not work either (reran the test before and after a restart of ride)
We tried to re-import the test files thinking perhaps something went wrong there. This did not help either.
Googling juts the last part (IndexError: list index out of range) gave me the suggestion it does not recognize all the lines of code in the back-end file. I would have no clue how to solve this as im not a major coder.
One difference between me and my colleague could be the versions. I downloaded python version 2.7.16 and ride 1.7.3.1. My colleague uses an older version of both python and RIDE. Perhaps the problem could be here?
https://paste.fedoraproject.org/paste/TLekH3az0m4wuUyM8C2RYw
I expect the test will run without failing (it is a happy flow) I have included some screenshots with code in the previous segment that might help
downgraded to the same version of Ride.py
This seems to have fixed the issue
I need to run the python program in the backend. To the script I have given one input file and the code is processing that file and creating new output file. Now if I change the input file content I don't want to run the code again. It should run in the back end continously and generate the output file. Please if someone knows the answer for this let me know.
thank you
Basically, you have to set up a so-called FileWatcher, i.e. some mechanism which looks out for changes in a file.
There are several techniques for watching file/directory changes in python. Have a look at this question: Monitoring contents of files/directories?. Another link is here, this is about directory changes but file changes are handled in a similar way. You could also google for "watch file changes python" in order to get a lot of answers :)
Note: If you're programming in windows, you should probably implement your program as windows service, look here for how to do that.