I'd like to run a Python script in my TRANSFORM that uses an external library, which I'm unable to install in the cluster.
For tests outside Hive, I could just copy the folder with the main files to my home directory, where the script is, and run from there, with an "import "
For Hive, I don't know how to use this library. I tried importing the files one by one with ADD FILE and also tried zipping and using ADD ARCHIVE, but I keep getting the message
FAILED: Execution Error, return code 20003 from org.apache.hadoop.hive.ql.exec.MapRedTask. An error occurred when trying to close the Operator running your custom script.
Any suggestions on how I can adapt my script to use these files?
Can you please post the log dump. That would be more helpful to troubleshoot. I came across the same problem and it resulted in the transformer library i wrote was referring to non-existent classes.
I din't had enough reputation to put this in as comment.
Related
I created an main.py
Where the code is on.system('program.exe')
And I compiled it with auto-py-to-exe as one file added the program.exe as add files when I execute it says program.exe is not recognized as internal or external error lease help thank you 😊
The problem is too complex for me please help
You are not supplying a ton of context but I am pretty sure this is because your compiled Python script cannot find program.exe. And this is probably because the "current working directory" is not the one containing program.exe. An easy solution would be to specify the complete path like C:\path\to\program.exe.
It must be very simple, but somehow I am missing something... Apologies for the long message but it's driving me crazy.
I'm reading thinkstat2, a free pdf about statistics for data science (beginner level).
It comes with exercises so I'm trying to follow the instructions and import the data. I'll describe the steps I'm following and give the pages, it's like a 2 minutes read.
Step 1a: Page 8 - "0.2 Using the code" - Get the code from author's github. I don't have a github account, so I dl the code as a zip on my hard drive. Then I extracted the content in a new folder.
Step 1b: Page 9 - Author suggests to get anaconda. From my understanding, anaconda contains many different app you can use through the "anaconda navigator". I have anaconda and will use Jupyter notebook for this.
Step 2: Page 9 - Author asks to run "nsfg.py" from his zip to make sure the reader has all the packages installed. Author says I should have a confirmation message. When I run it, there's a flash of command screen without any error or confirmation message. But if it's all about checking that the right package are installed correctly, I can just add "import x" in my code to add the missing package, so not a real issue here (or so I believed). I moved on.
Step 3: Pages 24 to 26 - "1.3 Importing the data" and "1.4 Dataframes" - This is where I'm stuck. After running "nsfg.py" (I mentioned in step 2), the author asks to "import nsfg", which is his own package created from "nsfg.py"? Maybe? So I try to import it and it's not recognized. Which probably comes from that "not a real issue" in step 2... I checked inside "nsfg.py" and it seems to be code to clean and read a .dct and .gz file.
Questions:
I don't understand what I'm supposed to do to make "nsfg.py" run properly.
I don't understand how my jupyter notebook would understand the "import nsfg" since those lies on my hard drive. The author does not add any line regarding importing anything from a local drive, or from a url to his github (not in the code of nsfg.py as well, except if I missed it).
I thought about just copy pasting the code of "nsfg.py" and running it, but I believe it would not work since the .dct and .gz file mentioned are saved locally as well. So I don't get how the code pasted from nsfg.py I run on jupyter notebook would make a link with those local .dct and .gz files. My python is not advanced enough to tinker that manually...
Help :( (any alternative solution is welcome)
Two ways to solve this, both worked for me:
Firstly, to replicate what the extracts of the book say, all assuming you are using linux or mac:
Hit git clone https://github.com/AllenDowney/ThinkStats2.git in your terminal in whatever folder you want to wok in
Then cd ThinkStats2/code.
Then python nsfg.py
Then hit python again to give you an interactive shell which is what the author of the book appears to be using - note the >>> before each command which indicates that it isn't in a Jupyter notebook.
To get this to work in a Jupyter notebook:
Hit git clone https://github.com/AllenDowney/ThinkStats2.git in your terminal in whatever folder you want to wok in
In your jupyter notebook interface, create a new notebook in the same directory as the ThinkStats2/code folder.
In the notebook hit import nsfg.
I think you are probably getting errors because you are not running your notebook from the same directory as the nsfg.py file which means that python can't find the script and so can't import it.
I am currently learning to code with python in vscode but recently I have been having problems with file seemingly deleting themselves after attempted execution (image 1). I have tried Ctrl+z and I have tried recovering them but neither have worked. The files are not in my recycling bin either. If I try to create a new file of the same name in the same folder it shows me a error message (image 2) of "unable to resolve non-existing files"
When I execute my code the file saves but then becomes deleted:
This message appears if I try to undo the delete:
Have you tried to restart vscode? It is recommended that you could try the following aspects:
Check if there is any effect caused by the related expansion. For example, LaTeX Workshop extension has the function of cleaning generated files.
Check your setting.json Files,clear the contents of the user setting.json file and the vs code will return to the default value.
If it still doesn't work, you could consider installing vscode again.
Try running the Python file through a command prompt or terminal using the command python <foo_file_name>. Doing this will either confirm or deny the theory that #jasonharper made in the comments.
If you can run the script that way then we know that #jasonharper was correct and somehow your execute button/command was tampered with.
I had a similar issue using a different programming language. Building generated a file that got deleted automatically.
In my case the Windows 10 default antivirus was the cause. Turned off real-time scanning, tried again and then my files weren't deleted.
Adding an exception for the folder fixed it.
I had the Same problem with a PHP file, and turns out it wasn´t VS CODE. It was my antivirus that was moving the file to quarentine!!
During a presentation yesterday I had a colleague run one of my scripts on a fresh installation of Python 3.8.1. It was able to create and write to a csv file in his folder (proof that the csv library was working correctly), but everything else failed due to not being able to find the needed files. To try and isolate the problem and figure out why, we tried the below simple script, which also failed.
He had this test.py script in "D:/TEST", which also contained some folders and image files. Running this script printed nothing to the console. No empty list, no error message, no newline. Maybe the print() function was also not working, but I didn't get around to testing that.
import os
print(os.listdir())
This script works fine on my computer and my other colleagues computers (all Windows 10, similar hardware). I didn't have time to look into the issue more thoroughly and don't have access to his computer anymore. What could be the problem? What other things could I have him look into in order to fix this? In case this problem appears again during a future presentation, what steps could I take to figure out the cause of it?
My colleague uninstalled Python and reinstalled it. After doing this apparently the "python" command will no longer run his scripts, but using "py" instead will. Now that he is using "py" to run his scripts, it is working as expected.
I'm making a game in Python (with Pygame) using LiClipse (based on Eclipse). My main file which executes everything is called "prueba.py". I've renamed it to "Main.py" and then trying to run the project I've got this error:
!MESSAGE Variable references non-existent resource :
${workspace_loc:MyGame/prueba.py}
Is there any file where I can change the file or something like that? Anyway, how can I solve it? I've tried many solutions from google and stackoverflow (such as this Variable references non-existent resource Build.xml) but they didn't work or didn't apply to my problem.
This is probably the Run Configuration for the project.
Look in 'Run > Run Configurations' for your project and check the variables.