Develop Reference

Vscode not interactive with python. Mac M1 chip

I am on a Mac with M1 chip and I have a problem with my VScode and python. It stays stuck on the ZSH shell even when I type the command to switch to bash (chsh -s /bin/bash). Lets say I run a simple code:
import cowsay
import sys
if len(sys.argv) == 2:
cowsay.cow("Hello, " + sys.argv[1])
I am supposed to be able to type my name in the shell after my python file name and it should print the cow saying Hello, Noah.
When I do so ((base) noahhaitas#Noahs-Mac-mini ~ % python3 itunes.py Noah Haitas), this is what I get as a message in my shell:
(base) noahhaitas#Noahs-Mac-mini ~ % python3 itunes.py Noah Haitas
/Library/Frameworks/Python.framework/Versions/3.10/bin/python3: can't open file '/Users/noahhaitas/itunes.py': [Errno 2] No such file or directory
I am trying to figure out how I can switch fully to bash and have the $ in front instead of seeing the % of ZSH.
What can be done as this is frustrating and I looked everywhere online and tried pretty much every solution.

Vscode is just an editor, and the system terminal is still applied.
Use the following command in the terminal to switch bash and restart the terminal:
chsh -s /bin/bash
At the same time, you can also set manually in the vscode terminal. Take Windows as an example:
Read the docs for more details about terminal settings.
By the way, there is an error when you run the file. It seems that the python file is not run in the correct directory.

Issues trying to install AirFlow locally

I'm new at airflow and I'm trying to install locally, following the instructions on the link below:
https://airflow.apache.org/docs/apache-airflow/stable/start/local.html
I'm running this code (as mentioned on the link):
# Airflow needs a home. `~/airflow` is the default, but you can put it
# somewhere else if you prefer (optional)
export AIRFLOW_HOME=~/airflow
# Install Airflow using the constraints file
AIRFLOW_VERSION=2.2.5
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
# For example: 3.6
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-2.2.5/constraints-3.6.txt
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
# The Standalone command will initialise the database, make a user,
# and start all components for you.
airflow standalone
# Visit localhost:8080 in the browser and use the admin account details
# shown on the terminal to login.
# Enable the example_bash_operator dag in the home page
and getting this error:
File "C:\Users\F43555~1\AppData\Local\Temp/ipykernel_12908/3415008398.py", line 3
export AIRFLOW_HOME=~/airflow
^
SyntaxError: invalid syntax
Someone knows how to deal with it?
I'm running on windows 10, vs code (jupyter notebook).
Tks!

Airflow is only supported on Linux and it looks like you're trying to run this on a windows machine.
If you want to install Airflow on Windows you'll need to use something like Windows Subsystem for Linux (WSL) or Docker. There are some examples around which show you how to do this on WSL (and loads using docker) - Here is one of them with WSL.

Backup Postgres from Python on Win10

I'm trying to backup Postgres from Python on Win10.
I'm working on Anaconda python 3.8, Win10 machine with Postgres12 local. On path environment variable I have postgres (lib and bin), no anaconda, and python 3.8 (no the anaconda one).
I'm able to correctly backup the database using on window's command shell:
pg_dump --dbname=postgresql://postgres:password#127.0.0.1:5432/test > C:\backup\dumpfile3.dump
but when I run it on anaconda:
os.system("pg_dump --dbname=postgresql://postgres:password#127.0.0.1:5432/test > C:\backup\dumpfile3.dump" )
I get as output 1 , witch is error code. It creates the file, but it's empty.
Using:
import subprocess
stk= 'pg_dump --dbname=postgresql://postgres:password#127.0.0.1:5432/test > C:\backup\dumpfile3.dump'
try:
subprocess.check_output(stk, shell=True, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
I get :
RuntimeError: command 'pg_dump --dbname=postgresql://postgres:password#127.0.0.1:5432/test > C:\backup\dumpfile3.dump' return with error (code 1): b"'pg_dump' is not recognized as an internal or external command,\r\noperable program or batch file.\r\n"
If I use: subprocess.run or subprocess.call I doesn't produce error, but the created file it's empty.
It seems that neither os.system or subprocess on the anaconda interpreter, got access to the environment variables on the command shell. How is this possible? and how I can overcome it?. It is different user invoking the shell?
Thanks in advance.

The Computer was restarted, and it solves the issue... . There was no change in the paths, I believe that from the moment things (python ,postgres, ...) were installed, the machine hasn"t been restarted.
import os
os.system("pg_dump --dbname=postgresql://postgres:password#127.0.0.1:5432/test > C:\backup\dumpfile3.dump" )
worked!, and
import subprocess
subprocess.call(r"C:\some\path\backup.bat")
also worked!. Inside backup.bat is:
pg_dump pg_dump --dbname=postgresql://postgres:password#127.0.0.1:5432/test > C:\backup\dumpfile3.dump
I imagine that the issue was that the anaconda interpreter need a system restart to get access to the environment variables (where the postgres variable was), witch make very little sense as return with error (code 1): b"'pg_dump' is not recognized as an internal or external command,\r\noperable program or batch file.\r\n" seen like a console message.
If anyone have a better explanation, is welcome.

Luigi Pipelining : No module named pwd in Windows

I am trying to execute the tutorial given in https://marcobonzanini.com/2015/10/24/building-data-pipelines-with-python-and-luigi/.
I am able to run the program on its own using local scheduler, giving me:
Scheduled 2 tasks of which:
* 2 ran successfully:
- 1 PrintNumbers(n=1000)
- 1 SquaredNumbers(n=1000)
This progress looks :) because there were no failed tasks or missing external de
pendencies
===== Luigi Execution Summary =====
However, to try the visualization on the server, when I try to run luigid --background, it throws me an error saying I dont have pwd module.
I cannot find a pwd module using pip for windows.
File "c:\users\alex\appdata\local\continuum\anaconda3\lib\site-packages
\luigi\process.py", line 79, in daemonize
import daemon
File "c:\users\alex\appdata\local\continuum\anaconda3\lib\site-packages
\daemon\__init__.py", line 42, in <module>
from .daemon import DaemonContext
File "c:\users\alex\appdata\local\continuum\anaconda3\lib\site-packages
\daemon\daemon.py", line 25, in <module>
import pwd
ModuleNotFoundError: No module named 'pwd'
I am working in Anaconda Spyder with Python 3.6

I was able to fix this by installing python-daemon==2.1.2
If you already have python-daemon, try downgrading to version 2.1.2
Do this before install luigi.
Example:
pip install python-daemon==2.1.2
then
pip install luigi.

For some reason, if you dont use the --background parameter on windows it will start just fine
just write luigid in cmd

The base problem here is that luigid --background is trying to spawn a python-daemon which is a unix-specific thing.
See section titled "The luigid server" here: http://luigi.readthedocs.io/en/stable/central_scheduler.html
Specifically:
Note that this requires python-daemon. By default, the server starts on AF_INET and AF_INET6 port 8082 (which can be changed with the --port flag) and listens on all IPs. (To use an AF_UNIX socket use the --unix-socket flag)
This existing stack overflow answer provides more detail:
How to start daemon process from python on windows?
Options I see here are:
Log a request with Luigi on github to improve their windows support to spawn Luigid as a windows process for the --background switch
Run a virtual machine with a proper Unix OS in it on Windows and run your Luigi pipelines there.
Follow Steven G's suggestion and run luigid in a separate command prompt
To reproduce the root cause of this issue, open a python prompt in windows and type:
>>import daemon
Traceback (most recent call last): File "", line 1, in
File "C:\Anaconda3\lib\site-packages\daemon__init__.py",
line 42, in
from .daemon import DaemonContext File "C:\Anaconda3\lib\site-packages\daemon\daemon.py", line 25, in
import pwd ModuleNotFoundError: No module named 'pwd'

how to execute remote python to open a web on remote machine by using a command on a local machine

I've been searching on net for a long time but I still don't get the answer.
Let me give an examle to describe my question more clearly:
machine A(local) is now conneted with machine B(remote).ALL I WANT TO DO is to :
run a command on A(local),then stop and wait ,and do nothing,and then,a web page is opened on B(remote) automatically.
P.S this python program is stored on machine B.
Here's what I've achived by now:
This is my python program named test.py,and it is stored on B under /home/pi/Documents:
import webbrowser
webbrowser.open('http://www.google.com')
On A,I used command:
ssh <username of B>#<ip of B> python /home/pi/Documents/test.py
After running the above command on A,there is no errors on A but also no action on B.
if I change the command into creating a file on B or sudo reboot,then after running this command there will be a file on B created or B is shut down successfully.
if I change the python program into printing something,like:
print("hello from B")
the content is magically printed on A's terminal.
It seems this command does not work well if I want to open a web on B or print somthing on B.
Can anyone help me with this or is there any other way to accomplish it?
helpless..
Someone has any ideas please???
Thanks in advance!

Assuming B is a Linux or Unix-like system, you have a DISPLAY problem. The webbrowser module locates a browser on the machine, and tries to open it on current display. It works when you launch it localy, but a ssh session has by default no configured display, so any attempt to launch a XWindow (GUI) application will fail.
The rationale behind it is that the -X and -Y flags of the ssh command allow to pass the client display to the server and open the window on the local screen (in your example on A). So if the permissions of the X servers on A and B are compatible, you could try:
A$ ssh -Y <username of B>#<ip of B> # open an interactive shell on B
B$ echo $DISPLAY # control the DISPLAY env variable
-> localhost:10.0 # common value for ssh transported DISPLAY
B$ python /home/pi/Documents/test.py # open the window on A
To force the opening on B, you could set the DISPLAY to localhost:0.0 (primary XWindow)
A$ ssh ssh <username of B>#<ip of B> # open an interactive shell on B
B$ DISPLAY = localhost:0.0 # sets the DISPLAY env variable
B$ export DISPLAY
B$ python /home/pi/Documents/test.py # open the window on B
You might need to tweek authorization of the XWindow servers (or use the awful xhost +) on A and/or B to make the above examples work
Once you have been able to successfully open a window on the proper screen, you will just have to set the DISPLAY environment variable to the correct value in your Python script before opening the browser window.

One of the simplest solutions is to use redirect of stdin
$ ssh pi#B python <<EOF
> print "Hello World from B"
> EOF
Hello World from B
$
However, if the script is quite big, it is better to copy py file to server B and then call ssh with the file name as #Eliethesaiyan suggested.
$ scp X.py pi#B:/home/pi/
X.py 100% 26 0.0KB/s 00:00
$ ssh pi#B python X.py
Hello World from B
$

I've tested this using a VM running Ubuntu, which OS are you running on your remote system? Here's my launch_google.py:
import os
os.environ["DISPLAY"] = ":0"
import webbrowser
webbrowser.open("https://google.com")
Launch this using:
ssh <user>#<IP Address> "python launch_google.py&"
I included the ampersand otherwise the ssh session remains open. The python process doesn't need to keep running.
It is important to set the DISPLAY environment variable before importing the webbrowser module, otherwise the browsers won't be setup correctly. You can verify this running python via SSH:
>>> import os
>>> "DISPLAY" in os.environ
False
>>> import webbrowser
>>> len(webbrowser._browsers)
0
>>> webbrowser.open("https://google.com")
False
>>> os.environ["DISPLAY"] = ":0"
>>> reload(webbrowser)
<module 'webbrowser' from '/usr/lib/python2.7/webbrowser.pyc'>
>>> len(webbrowser._browsers)
3
>>> webbrowser.open("https://google.com")
True
>>>