How to setup airflow with python for a windows development environment - python

At work we are completing a PoC of Airflow by AirBnB. Standard operating practice is to decouple the development (i.e. DAG creation and python scripts etc) from the runtime environment. I.e. we are not allowed to script directly on the servers (bad practice anyway).
As such, I have configured intelliJ to work with Python (works as expected) in Windows (we are not allowed to have unix workstations) but I cannot find a way to install airflow in Windows. I am new to Python with a Java development background.
As such, how do I setup a local development environment IDEA ultimate for AirBnb/Airflow (there is no documentation ?

I have it working and compiling with the following step:
The steps below are for using intelliJ Ultimate edition, download the python modules.
Download the python extension for InttelliJ Ultimate edition or PyCharm
Configure and install python 2.7 for Windows.
Add the SDK to IDEA
Configure the following proxy details in environment variables to install modules using PIP ->
HTTP_PROXY=http : // user:password#your-company-proxy.com:8080
HTTPS_PROXY=http : // user:password#your-company-proxy.com:8080
(remove spaces, not allowed to post multiple links)
Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat).
Get it from http://aka.ms/vcpython27
C:_downloads>msiexec /i VCForPython27.msi ALLUSERS=1
pip install airflow
Create a new project, configure it as a python module.
Download the GIT repository -> airflow
Configure your path correctly, also set a python home and AIRFLOW_HOME.

Related

How to compile a python script into executable program and can be use by others

My python script is finished and working and I want to compile and have other users enjoy/benefit from it. The users don't need to install Pycharm or Visual Studio Code, something like an executable file or run in a command prompt then execute on their local machine or is there a way to convert it on a Tampermonkey Script?
How do I achieve this? Thank you very much in advance!
Googled and Youtubed but it's not what I'm looking for.
This question is probably answered multiple times, but the PyInstaller module is a great way to generate an executable that will run on Windows, and an app that will run on macOS.
Check out PyInstaller on PyPI.org: https://pypi.org/project/pyinstaller/
Project description
PyPI PyPI - Python Version Read the Docs (version) PyPI - Downloads
PyInstaller bundles a Python application and all its dependencies into a single package. The user can run the packaged app without installing a Python interpreter or any modules.
Documentation:
https://pyinstaller.org/
Code:
https://github.com/pyinstaller/pyinstaller
PyInstaller reads a Python script written by you. It analyzes your code to discover every other module and library your script needs in order to execute. Then it collects copies of all those files – including the active Python interpreter! – and puts them with your script in a single folder, or optionally in a single executable file.
PyInstaller is tested against Windows, macOS, and GNU/Linux. However, it is not a cross-compiler: to make a Windows app you run PyInstaller in Windows; to make a GNU/Linux app you run it in GNU/Linux, etc. PyInstaller has been used successfully with AIX, Solaris, FreeBSD and OpenBSD, but is not tested against them as part of the continuous integration tests.
Main Advantages
Works out-of-the-box with any Python version 3.7-3.11.
Fully multi-platform, and uses the OS support to load the dynamic libraries, thus ensuring full compatibility.
Correctly bundles the major Python packages such as numpy, PyQt5, PySide2, PyQt6, PySide6, wxPython, matplotlib and others out-of-the-box.
Compatible with many 3rd-party packages out-of-the-box. (All the required tricks to make external packages work are already integrated.)
Works with code signing on macOS.
Bundles MS Visual C++ DLLs on Windows.
Installation
PyInstaller is available on PyPI. You can install it through pip:
pip install pyinstaller
Requirements and Tested Platforms
Python:
3.7-3.11. Note that Python 3.10.0 contains a bug making it unsupportable by PyInstaller. PyInstaller will also not work with beta releases of Python 3.12.
tinyaes 1.0+ (only if using bytecode encryption). Instead of installing tinyaes, pip install pyinstaller[encryption] instead.
Windows (32bit/64bit):
PyInstaller should work on Windows 7 or newer, but we only officially support Windows 8+.
Support for Python installed from the Windows store without using virtual environments requires PyInstaller 4.4 or later.
Note that Windows on arm64 is not yet supported. If you have such a device and want to help us add arm64 support then please let us know on our issue tracker.
Linux:
GNU libc based distributions on architectures x86_64, aarch64, i686, ppc64le, s390x.
musl libc based distributions on architectures x86_64, aarch64.
ldd: Console application to print the shared libraries required by each program or shared library. This typically can be found in the distribution-package glibc or libc-bin.
objdump: Console application to display information from object files. This typically can be found in the distribution-package binutils.
objcopy: Console application to copy and translate object files. This typically can be found in the distribution-package binutils, too.
Raspberry Pi users on armv5-armv7 should add piwheels as an extra index url then pip install pyinstaller as usual.
macOS (x86_64 or arm64):
macOS 10.15 (Catalina) or newer.
Supports building universal2 applications provided that your installation of Python and all your dependencies are also compiled universal2.
Usage
Basic usage is very simple, just run it against your main script:
pyinstaller /path/to/yourscript.py
For more details, see the manual.
Untested Platforms
The following platforms have been contributed and any feedback or enhancements on these are welcome.
FreeBSD
ldd
Solaris
ldd
objdump
AIX
AIX 6.1 or newer. PyInstaller will not work with statically linked Python libraries.
ldd
Linux on any other libc implementation/architecture combination not listed above.
Before using any contributed platform, you need to build the PyInstaller bootloader. This will happen automatically when you pip install pyinstaller provided that you have an appropriate C compiler (typically either gcc or clang) and zlib’s development headers already installed.
Support
Official debugging guide: https://pyinstaller.org/en/v5.6.2/when-things-go-wrong.html
Assorted user contributed help topics: https://github.com/pyinstaller/pyinstaller/wiki
Web based Q&A forums: https://github.com/pyinstaller/pyinstaller/discussions
Email based Q&A forums: https://groups.google.com/g/pyinstaller
Changes in this Release
You can find a detailed list of changes in this release in the Changelog section of the manual.

Is there a way to pack a python project with all dependencies but not as an executable, but with access to the code?

since I think the original post wasn't clear enough -
EDIT:
Is there a tool or an option to pack my python project and all of it's dependencies and sub dependencies - same as pyinstaller does, but pyinstaller generates an executable binary, and I'd like to have the ability to make changes in the code after it's distributed (e.g on a client's environment)
Original Post
I have this python project, with a lot of dependencies and sub-dependencies, that's currently distributed by building it using pipenv to create a virtual environment and getting the 3rd party libraries, and pyinstaller to generate an executable which is being used on a virtual machine with another OS (I'm packing an executable to the target's machine OS by building it on a docker with the same OS).
the thing is - I'm using some data and scripts from this virtual machine in my python project, so i can't run it locally, and whenever there's a bug, or an error, I have to make changes locally and rebuild (which takes some time) and only then move the executable to the vm in order to run it.
my goal is to have all of the code packed with the dependencies, but with the structure of my project, and not as executable, so I can make quick changes on the VM itself.
The VM may not have an external connection, so I can't just install the dependencies on the machine.
is there a tool that can help me do such a thing?
Note
Currently the python version on the VM is different than the python version the project uses. it's possible to install another version if necessary.

How to embed python into a kotlin/java Android app?

I have an Android app written in kotlin that needs to call python code. I would like the app to come bundled with the necessary python binaries to execute some python scripts as well as the required python libraries, ideally managed by pip in a virtual environment.
I already have some code that will copy these files from the assets folder that is bundled into the APK when building the app, but I don't know where to find recent python binaries that can run natively on Android. I'm assuming I need something like a arm64 version of python >3.7 .
A second issue I have is that I'm developing the app on Windows, so the virtual environment and some dependencies are not cross platform. I would probably need a way to either automatically convert the windows virtualenv into a valid project structure with correct dependencies, or a way to create and manage a linux virtual environment from Windows (maybe WSL can save me here).
What is the preferred method? Are there open source Android apps that have solved similar issues?

missing python language when i created azure function

I'm new for azure. i just tried to create azure function for python script, but under language list, python language is missing.
Firstly, you need create the Linux OS Function, for now the Python is not support on Windows OS. So when you create the Function you should choose the Linux OS then you will be able to choose the Python stack.
Secondly, after you create the function, for now in-portal file editing this feature is disabled, You could check this issue:In-portal editing will be disabled for Python Functions. So you have to use the Azure Functions Core Tools or VS Code extension. Then upload the local function to Azure.
You can use VS CODE for the same . Visual Studio 2017/2019 only supports C#.
You can install the package for VS CODE WITH
sudo apt-get install python3-venv

Django project won't work with Python Tools Visual Studio

I'm trying to get a Django project working with PTVS. The only reason I'm using it is because I have Azure credits and want to deploy there. In Visual Studio, I go to New Project > Python > Django. I chose "Add Virtual Environment". Here is the screenshot:
After the project is created, this is my solution explorer:
You can see django is definitely installed in the environment. But when I right-click on the app and go to Python > Sync DB, I get a bunch of errors says that
ImportError: No module named django.core.management. All the django package statements are underlined and they can't be imported. Any ideas?
When I go back to create another project and I do "Install into Python 2.7 directory," it works fine. So I don't understand why I can't use the virtual environment.
Go to your Projects folder and delete all the application/projects.
Clean the project directory. (By default this is the PATH: Documents/Visual Studio 2013/Projects
Close all the other IDE's and Programs that might be using your
python interpreter.
Set no-proxy in case you are using some proxy settings because it
will prevent Visual Studio from downloading and installing the required
packages.
You might prefer restarting your machine before performing these steps.
It worked for me. I had the same problem.
Below are two useful tutorials to walk you through the entire process.
PTVS 2.0 Beta: Creating a Django Website : A step by step guide
Django and MySQL on Azure with Python Tools 2.1 for Visual Studio

Categories

Resources