Hi I got this weird problem. I am new to spark, just installed spark and java, made the environmental variables and spark seems to work properly. I can enter the spark console form cmd and write print ect. its working.
But when I try to do bin\spark-submit C:\Users\User\Desktop\Big_Data\pi.py where pi.py is just example form documentation it shows this: enter image description here. Almost as if the spark was working but the python file is not executed...
I also tried with different python files such as: enter image description here with command: C:\Users\User\spark-3.1.2-bin-hadoop2.7>bin\spark-submit C:\Users\User\Desktop\Big_Data\try.py C:\Users\User\Desktop\Big_Data\pg100.txt
And results are as follows:
enter image description here
No output, errors, nothing, maybe someone has any idea what is wrong???
Clarifying: basically there is no output or execution of python file when calling spark form cmd console
Looking at Spark's Quick Start Guide examples, can you run this command:
bin\spark-submit --master local C:\Users\User\Desktop\Big_Data\pi.py
More info regarding spark-submit can be found here.
Related
I recently started using Python on VScode and I have been having problems with my output.The code runs smoothly(no problems detected) but the result doesnt appear in my output.The problem isnt with my code because even a simple command like print("Hello world") doesnt appear.
Can someone help me?
Here are some screenshots
Also I tried running the program from another terminal but I had to save the file,for exapmple dna.py, and the system wont let me.An error message appears saying it this file is not supported by .ipynb support and the name should match this pattern .ipynd(tried it,still wont accept it)
The code that I created
The image in my ouput after I run the code
I suggest reviewing the official Visual Studio Code Python documentation and tutorial.
Also, try to run the file directly in a terminal and see if it solves your problem.
ex: python my_script.py
See this answer for more details.
I have a CodeDeploy which deploys application on Windows instances. I have a Python script which is running as part of ValidateService hooks. Below is the code I have in that script:
print("hello")
So, I have removed everything and just printing hello as part of this script. When this script is called by CodeDeploy I get below error:
My appspec.yml file:
...
ValidateService:
- location: scripts/verify_deployment.py
timeout: 900
I tried getting some help on Google but got nothing. Can someone please help me here.
Thanks
As Marcin already answered in a comment, I don't think you can simply run python scripts in CodeDeploy. At least not natively.
The error you see means that Windows does not know how to execute the script you have provided. AFAIK Windows can't run python natively (like most linux distros can).
I am not very accustomed to CodeDeploy, but given the example at https://github.com/aws-samples/aws-codedeploy-samples/tree/master/applications/SampleApp_Windows, I think you have to install python first.
After so much of investigations, I found my answer. The issue is little misleading, there is nothing to do with Code format or ENOEXEC. The issue was due to Python path. While executing my script, CodeDeploy was unable to find Python (Though I had already added python.exe in Environment variable path).
Also, I found that CodeDeploy is unable to execute .py file due to Python path issue. So, I created a PowerShell script and invoking Python script from there. Like below:
C:\Users\<username>\AppData\Local\Programs\Python\Python37-32\python.exe C:\Users\<username>\Documents\verify_deployment.py
It executed Python script successfully and gave me below output:
hello
I'm learning Python and I wanna print the list of all my blob storage in Azure. Find the sample code from here.
I have done this step:
set connection string:
After the run I received this error
Note: Like my experience, your problem solved after the restart.
Your code is correct. Just restart your VS Code and run it again. It will work. You error cause by setx command. It can't work in the current command window. You can refer to Why is setx path not working?
On a local system, variables created or modified by this tool will be
available in future command windows but not in the current CMD.exe
command window.
I try to import data frame into spark using python pyspark. fro that I used jupter notebook and I write bellow code and i Got out put like below..
After that I want to run this in CMD fro that I save my python codes in text file and save as test.py (As python file) Then I Run that python file in cmd using python test.py command below the screen shot
So my task is completed but after 3 or 4 hours later I again try to do same process..Then unfortunately I got bellow error message can some one explain why its happens? because before its correctly worked, but after that its not work and I not did any changes between these two attempt. below have full error now have facing.
It looks that you have not set the Hadoop home (winutil.exe) in the command session which you have opened later.
The spark is unable to locate HADOOP_HOME. Check if you had set something different when you ran the command 1st time.
I started using Pycharm and managed to have the python prompt running after a file is run (link) and I found how to run pieces of code in the current iPython console (link).
For data science, however, it is very convenient to have a single iPython console/Python kernel where I can run files and code, just like Spyder does, and continue using the same started python kernel. For example to load a big amount of data with a script and write another script or pieces of code for plotting in different ways, exploring the data.
Is there a way in Pycharm to do it?
I think that it would imply generating automatically and running a line like:
runfile('C:/temp/my_project/src/console/load_data.py', wdir='C:/temp/my_project/src')
If the option does not exist, is it possible to make a macro or similar to do it?
The option "Single instance only" in Edit configurations doesn't help.