Develop Reference

Issues trying to install AirFlow locally

I'm new at airflow and I'm trying to install locally, following the instructions on the link below:
https://airflow.apache.org/docs/apache-airflow/stable/start/local.html
I'm running this code (as mentioned on the link):
# Airflow needs a home. `~/airflow` is the default, but you can put it
# somewhere else if you prefer (optional)
export AIRFLOW_HOME=~/airflow
# Install Airflow using the constraints file
AIRFLOW_VERSION=2.2.5
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
# For example: 3.6
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-2.2.5/constraints-3.6.txt
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
# The Standalone command will initialise the database, make a user,
# and start all components for you.
airflow standalone
# Visit localhost:8080 in the browser and use the admin account details
# shown on the terminal to login.
# Enable the example_bash_operator dag in the home page
and getting this error:
File "C:\Users\F43555~1\AppData\Local\Temp/ipykernel_12908/3415008398.py", line 3
export AIRFLOW_HOME=~/airflow
^
SyntaxError: invalid syntax
Someone knows how to deal with it?
I'm running on windows 10, vs code (jupyter notebook).
Tks!

Airflow is only supported on Linux and it looks like you're trying to run this on a windows machine.
If you want to install Airflow on Windows you'll need to use something like Windows Subsystem for Linux (WSL) or Docker. There are some examples around which show you how to do this on WSL (and loads using docker) - Here is one of them with WSL.

Facing the following error while trying to create a Docker Container using a DockerFile -> "error from sender: open .Trash: operation not permitted"

Im completely new to Docker and I'm trying to create and run a very simple example using instructions defined in a DockerFile.
DockerFile->
FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y python3 pip
COPY ./ .
RUN python3 test.py
contents of test.py ->
import pandas as pd
import numpy as np
print('test code')
command being used to create a Docker Container ->
docker build --no-cache . -t intro_to_docker -f abs/path/to/DockerFile
folder structure -> (both files are present at abs/path/to)
abs/path/to:
-DockerFile
-test.py
Error message ->
error from sender: open .Trash: operation not permitted
(using sudo su did not resolve the issue, which i believe is linked to the copy commands)
I'm using a Mac.
any help in solving this will be much appreciated!

The Dockerfile should be inside a folder. Navigate to that folder and then run docker build command. I was also facing the same issue but got resovled when moved the docker file inside a folder

Usually the error would look like:
error: failed to solve: failed to read dockerfile: error from sender: open .Trash: operation not permitted
And in my case, it's clearly saying that it is unable to find the dockerfile.
Also, in your command, I see a . after --no-cache, I think that's not required?
So better, try navigating to the specified path and then run the build command replacing the -f option with a ., which specifies the build command to consider the current folder for its build process.
In your case
cd abs/path/to/
docker build --no-cache -t intro_to_docker .

It seems the system policies are not allowing the application to execute this command. The application "Terminal" might not have approval to access the entire file system.
Enable full disk access to terminal. Change it using "System Preferences > Security & Privacy > Privacy > Full Disk Access"

I had the same error message and my Dockerfile was located in the HOME directory, I moved the Docker file to a different location and executed the docker build from that newly moved location and it successfully executed.

Run Python script using virtualenv

I have a python script named utils (note without any .py extension). Where I have some utility functions. The path is also added in PATH variable.
#!/usr/bin/env python3
import click, sys
#main.command('echo', context_settings=dict(help_option_names=['-h', '--help']))
def echo_test():
click.echo("Hello World")
sys.exit(0)
It works fine. Now I can run from anywhere utils echo.
I am trying to make the script to use virtualenv instead of the global python. I have tried
#!/path/to/venv/bin python3
import click, sys
Then it throws me error permission denied
Permissions for utils file are -rwxr-xr-x
Any idea how could I use venv with script.

If your code is pasted correctly, your problem is that you are trying to execute a directory with your she-bang, not Python, because you have a space rather than a slash as a separator:
#!/path/to/venv/bin python3
rather than:
#!/path/to/venv/bin/python3
EDIT: By the way, is there a reason you want to change the code and not just activate the virtual environment, like it's supposed to be used?
If you want to do it, you can just:
$ source path/to/venv/bin/activate<.optional_extension>
You need the optional extension if you use a shell other than Bash (probably other Bourne-like shells too).

Try changing the file permissions via this command:
chmod 755
chmod -R 755 on the /usr/lib/python/site-packages/virtualenv
or even
chmod +x
Suggest you read the man page for chmod by using this command
man chmod
if you are not sure.

ModuleNotFoundError: No module named 'hello'

I did something really bad. I don't know what I did. I created a test project with hello.py where I did some mistake when running with some command. Now, I have deleted that and back to the real project, and I got the following error:
File "/home/bhojendra/anaconda3/envs/myenv/lib/python3.9/site-packages/flask/cli.py", line 240, in locate_app
import(module_name)
ModuleNotFoundError: No module named 'hello'
I don't have even the word hello anywhere in the project.
I have removed all pycaches from the project, cleaned the conda conda clean -a, removed myenv environment and removed pip cache directory. Then, I re-created the environment and and re-installed the requirements and when launching the project with flask run, it throws that error again. Not sure what's happening.
It might be the issue with flask cache, I don't know how to overcome this issue.

In your environment, you likely left your FLASK_APP set to the file hello, so it was trying to run that, although it doesn't exist. You just have to export or set your flask app based on what machine you are using.
Unix Bash (Linux, Mac, etc.):
$ export FLASK_APP=hello
$ flask run
Windows CMD:
> set FLASK_APP=hello
> flask run
Windows PowerShell:
> $env:FLASK_APP = "hello"
> flask run
You could also unset the export:
unset FLASK_APP
And then set the flask app

Setting up crontab for my Django Application

I had an issue with setting up crontab for my Django application for a week and I have almost figured out the same. (Issue linked with Unable to make a function call using Django-cron)
My crontab -e syntax is
* * * * * /Users/ashwin/dashboard/proj_application/exec.sh >> /Users/ashwin/dashboard/proj_application/data.log 2>&1
And in my exec.sh, I have
#!/bin/bash
cd "$(dirname "$0")";
CWD="$(pwd)"
echo $CWD
python -c 'import proj_application.cron as cron; cron.test()'
And in cron.py,
from django.core.mail import send_mail
from smtplib import SMTP
from email.mime.text import MIMEText
import datetime
def test():
message = "<p>This is test mail scheduled to send every minute</p>"
my_email = MIMEText(message, "html")
my_email["From"] = "xxx#domain.com"
my_email["To"] = "yyy#domain.com"
my_email["Subject"] = "Title"
sender = "person1#domain.com"
receivers = ["person2#domain.com"]
with SMTP("localhost") as smtp:
smtp.login(sender, "yyy#1234")
smtp.sendmail(sender, receivers, my_email.as_string())
Actual Problem:
The crontab is now able to call the exec.sh file and I am able to print $CWD in echo, when the call comes to cron.py, It is unable to recognize django.core.mail and throws the following error.
from django.core.mail import send_mail
ImportError: No module named django.core.mail
I think, I need to set up virtual environment or python variable path somewhere, but since I am new to crontab, I am not sure how to do that.
Any assistance is appreciated. Thanks in advance.

The way I interpret this is that you are normally running your script in a virutal environment, and it is failing now that you added it to a cron job. If this is incorrect, just skip to the last paragraph.
To run a cron job through a virutal environment, you need to use the virtual environment's python as the python you want. E.g to run the cron.py file:
* * * * * /path/to/venv/bin/python3 /path/to/cron.py >> /Users/ashwin/dashboard/proj_application/data.log 2>&1
This is the way I would recommend doing it, as it doesn't seem like that shell script is entirely necessary (anything in there can easily be done at the top of the python script), but if it is you can do something similar where you call the virtual environment's python instead of the default python. like this:
/path/to/venv/bin/python3 -c 'import proj_application.cron as cron; cron.test()'
However, if this doesn't work, this may not be an issue with the cron job, and instead with the django setup of the mail client (e.g. making sure it is included in the installed apps, etc.)
Since it is a little unclear where your issue is (mail client setup, cron setup, venv configuration, etc), if you are sure it is an issue with running your scripts you usually run in a virtual environment, than these steps should help, otherwise I would make sure your mail client is configured correctly. Make sure your scripts are correct by running them in the console (not through a cron job), and then you can come back and update the post with more detailed info about where the problem you have lies.

yes you are correct you may need to use virtual environment(although its optional but best practice).
To create virtual environment(expects python to be installed prior)
python -m venv venv
To activate (path may differ based on OS(mine is windows))
source venv/Scripts/activate
To install dependency in virtual environment
pip install Django
Now you all set to use virtual env's Django.
which python should map to venv path.
Now you have two ways to run python script
#1 use direct python path
absolute/path/of/venv/bin/python3 -c 'import proj_application.cron as cron; cron.test()'
#2 activate virtual environment and use the bash script as same.
#!/bin/bash
cd "$(dirname "$0")";
CWD="$(pwd)"
echo $CWD
source venv/Scripts/python # this will be in windows
python -c 'import proj_application.cron as cron; cron.test()'