Develop Reference

Issues trying to install AirFlow locally

I'm new at airflow and I'm trying to install locally, following the instructions on the link below:
https://airflow.apache.org/docs/apache-airflow/stable/start/local.html
I'm running this code (as mentioned on the link):
# Airflow needs a home. `~/airflow` is the default, but you can put it
# somewhere else if you prefer (optional)
export AIRFLOW_HOME=~/airflow
# Install Airflow using the constraints file
AIRFLOW_VERSION=2.2.5
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
# For example: 3.6
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-2.2.5/constraints-3.6.txt
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
# The Standalone command will initialise the database, make a user,
# and start all components for you.
airflow standalone
# Visit localhost:8080 in the browser and use the admin account details
# shown on the terminal to login.
# Enable the example_bash_operator dag in the home page
and getting this error:
File "C:\Users\F43555~1\AppData\Local\Temp/ipykernel_12908/3415008398.py", line 3
export AIRFLOW_HOME=~/airflow
^
SyntaxError: invalid syntax
Someone knows how to deal with it?
I'm running on windows 10, vs code (jupyter notebook).
Tks!

Airflow is only supported on Linux and it looks like you're trying to run this on a windows machine.
If you want to install Airflow on Windows you'll need to use something like Windows Subsystem for Linux (WSL) or Docker. There are some examples around which show you how to do this on WSL (and loads using docker) - Here is one of them with WSL.

how to remove peripheral device from devices and printers in python

I want to remove device specific device attached to my computer when i run a script in python but am not able to do this. Have searched on the web with no solution.
link 1 and
link 2
I learnt i can achieve that with power shell but am not able to achieve that.
This code is able to open the cd rom successfully
import os
os.system('powershell $driveEject = New-Object -comObject Shell.Application; $driveEject.Namespace(17).ParseName("""F:""").InvokeVerb("""Eject""")')
I came across this question how to remove printer from devices and printer which there is no answer for it.
I kindly need assistance achieve such result.

First I prepared a powershell script (s.ps1):
$driveEject = New-Object -comObject Shell.Application
$driveEject.Namespace(17).ParseName("E:").InvokeVerb("Eject")
Then I created a Python script
import os
os.system('powershell.exe -Command ./s.ps1 >log.txt')
After that I got an error in the log file
./s.ps1 : s.ps1 cannot be loaded because
running scripts is disabled on this system. For
more information, see about_Execution_Policies at
http://go.microsoft.com/fwlink/?LinkID=135170.
At line:1 char:1
+ ./s.ps1
+ ~~~~~~~
+ CategoryInfo : SecurityError: (:) [], PSSecurityException
+ FullyQualifiedErrorId : UnauthorizedAccess
After reading the article I executed the commands in cmd running under the administrator
> powershell
Windows PowerShell
Copyright (C) 2015 Microsoft Corporation. All rights reserved.
PS > Set-ExecutionPolicy -ExecutionPolicy Unrestricted
After that, I was finally able to run the script. And I saw a pop-up in the tray with a message that I can eject the disk.

Scrapy-splash no results at "Server listening on http://0.0.0.0:8050"

I'm on Windows 10 Home Edition using Docker tool box (my BIOS configuration doesn't allow me to install Linux)
When I run
docker run -it -p 8050:8050 --rm scrapinghub/splash
I receive the log message Server listening on http://0.0.0.0:8050, as it should be in normal cases. But when I go to the address at my Chrome, the only thing that returns is ERR_ADDRESS_INVALID, not possible to access the page.
I have tried the following solutions to this problem:
Changing the SPLASH_URL on my settings.py according to my Docker IP adress;
Adding new ports to my docker run, such as 5023:5023 and 8051:8051.
I imagine that this is the kind of problem which I need to investigate its caused by myself. But I have tried everything that I could find on the Internet.

I'm gonna write the right answer, thanks to the users Selcuk, for explaning some technical terms, and David Maze, for giving me the final solution.
Since we are using the Docker Toolbox, we need to use our docker-machine ip.
Go to your terminal and type:
docker-machine ip
Now that you have the right ip, go to your settings.py and set
SPLASH_URL = http://[docker-machine ip]:8050
And you're good to go!

You don't need to change any settings.
If you launch docker terminal, it shows the correct ip address at the very first, right after the whale drawing.
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
docker is configured to use the default machine with IP 192.168.99.102
For help getting started, check out the docs at https://docs.docker.com
So in my case the ip was 192.168.99.102.
If I visit 192.168.99.102:8050 on chrome, splash works.

Old topic but it has been helpfull for me. I add another way to get the IP address, without docker-machine.
With the container ID you can get the IP address with:
sudo docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' CONTAINER ID
To get the container ID use : sudo docker ps
You can also use the container name instead of the ID

Calling Matlab scripts from Django with Python's Popen class

I'm developing a Django app which runs Matlab scripts with Python's Popen class. The python script that calls Matlab scripts lives in the main folder of my Django app (with views.py). When I call the script from command line, it runs like a charm but when I make a request from the client in order to run the corresponding python script, I receive the following warning:
"< M A T L A B (R) > Copyright 1984-2018 The MathWorks, Inc. R2018a (9.4.0.813654) 64-bit (glnxa64) February 23, 2018 To get started, type one of these: helpwin, helpdesk, or demo. For product information, visit www.mathworks.com. >> [Warning: Unable to create preferences folder in /var/www/.matlab/R2018a. Preferences folder location must be writable. Using a temporary preferences folder for this MATLAB session. See the preferences documentation for more details.] >>
My app uses a Python virtual environment and it is being deployed with Apache web server.
Here is my python script that calls Matlab scripts:
import os
import subprocess as sp
import pymat_config
def pymat_run():
pwd = pymat_config.pwd_config['pwd']
cmd1 = "-r \"Arg_in = '/path/to/my/main/folder/input.txt'; Arg_out = '/path/to/my/main/folder/file.txt'; matlab_script1\""
baseCmd1 = ['/usr/local/MATLAB/R2018a/bin/matlab', '-nodesktop', '-nosplash', '-nodisplay', 'nojvm', cmd1]
os.chdir('/path/to/matlab_script1')
sudo_cmd = sp.Popen(['echo', pwd], stdout=sp.PIPE)
exec1 = sp.Popen(['sudo', '-S'] + baseCmd1, stdin=sudo_cmd.stdout, stdout=sp.PIPE, stderr=sp.PIPE)
out, err = exec1.communicate()
return out
Any suggestions ?

Finally I managed to find the solution of that issue by myself. The problem came from the kind of user who called the Matlab's script. When I was running the above script from a Python interpreter or from the shell, it was the user (with the user password) who was running the script while when I was calling the script from the client the user was the web server's user: www-data.
So at first to avoid the above warning I gave permissions to www-data user to the /var/www folder with the following command:
sudo chown -R www-data /var/www/
After that, the "Warning" disappeared but the script still didn't run because it was asking for www-data's password internally and taking user's password from pymat_config file.
To solve this, I edited /etc/sudoers file in order for www-data to be able to call Matlab scripts without asking password. So I added the following line:
www-data ALL=(ALL) NOPASSWD: /usr/local/MATLAB/R2018a/bin/matlab
and now it runs like a charm !

Powershell not recognizing conda as cmdlet, function, or operable program

I've been having this issue on my new laptop for a couple of hours and cannot figure out what's causing it. I'm trying to install scikit-learn with conda and get the following error
conda : The term 'conda' is not recognized as the name of a cmdlet, function, script file, or operable program. Check
the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ conda install -c anaconda scikit-learn
+ ~~~~~
+ CategoryInfo : ObjectNotFound: (conda:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
Has anybody else had a similar issue on Windows 10?

Have you already activated the environment for this use case?
There is a long running thread about this on the GitHub conda discussion regarding conda failures various Windows 7 and higher, here:
https://github.com/conda/conda/issues/626
One suggestion is:
The down and dirty:
Check to see if activate works in cmd.exe.
If doesn't work or not acceptable--as #TurboTim shows:
Powershell needs the path to each env (anaconda3\envs\someenv\py33.exe. Laborious! :p
If you don't mind polluting your powershell a little, you can create a profile script which is run every time you open powershell.
The below will add the functions Invoke-CmdScript, Conda-Activate, Conda-Deactivate to your powershell. See Tim's link above for why.
PS C:> New-Item -Path $profile -ItemType File -Force
This creates a script at:
PS C:\> echo $profile
...something like C:\Users\yourUser\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1
Edit that script.
PS C:\> explorer $profile
Add this code, save, and reopen powershell (or . $profile ) :
function Invoke-CmdScript {
param(
[String] $scriptName
)
$cmdLine = """$scriptName"" $args & set"
& $Env:SystemRoot\system32\cmd.exe /c $cmdLine |
Select-String '^([^=]*)=(.*)$' | ForEach-Object {
$varName = $_.Matches[0].Groups[1].Value
$varValue = $_.Matches[0].Groups[2].Value
Set-Item Env:$varName $varValue
}
}
$condaRoot = "$Env:USERPROFILE\Anaconda3"
function Conda-Activate([string]$condaEnv) {Invoke-CmdScript $condaRoot\Scripts\activate.bat $condaEnv}
function Conda-Deactivate {Invoke-CmdScript $condaRoot\Scripts\deactivate.bat}
Usage:
C:\> Conda-Activate TFTheano
C:\> Conda-Activate root
C:\> conda info --envs
Disclaimers: Deactivate, as written, didn't do the job for me, thus I just use Conda-Activate to move around. Also,
I don't suspect there are security cautions with adding Invoke-Cmd to profile, so please chime in if 2 cents

I found this online: Unblock-File -Path .\Get-RemoteProgram.ps1
I'm going to try that command in power shell as soon as I can.

I previously had Python installed before conda and since having both together is said to be bring complications I uninstalled the original Python interpreter. This is what worked for me when the same error came up while trying to run a python script using conda.
TLDR; Simply add the path of conda.exe to the environment variables.