Develop Reference

Can't get airflow AWS connection to work "ModuleNotFoundError: No module named 'airflow.providers.amazon"

I have been trying to run a simple Airflow DAG to show what's in an s3 bucket but I keep getting this error: ModuleNotFoundError: No module named 'airflow.providers.amazon'
I've tried several pip installs recommended in similar questions but still have no luck. Here's the python script and below is a screenshot of my Airflow webserver showing the error message. Note I'm using Airflow version 2.5.0
import datetime
import logging
from airflow import DAG
from airflow.models import Variable
from airflow.operators.python_operator import PythonOperator
from airflow.hooks.S3_hook import S3Hook
def list_keys():
hook = S3Hook(aws_conn_id='aws_credentials_old')
bucket = Variable.get('s3_bucket')
prefix = Variable.get('s3_prefix')
logging.info(f"Listing Keys from {bucket}/{prefix}")
keys = hook.list_keys(bucket, prefix=prefix)
for key in keys:
logging.info(f"- s3://{bucket}/{key}")
dag = DAG(
'lesson1.exercise4',
start_date=datetime.datetime.now())
list_task = PythonOperator(
task_id="list_keys",
python_callable=list_keys,
dag=dag
)

You can try installing the backport-providers-amazon package because it's only available in the Airflow main branch.
pip install apache-airflow-backport-providers-amazon
Here you can find more info. https://pypi.org/project/apache-airflow-backport-providers-amazon/

You are importing from the wrong place. It should be
from airflow.providers.amazon.aws.hooks.s3 import S3Hook

Import Error: cannot import 'mdss_bias_scan' from 'aif360.sklearn.metrics'

I would like to use the mdss_bias_scan function of aif360 for detecting the combination of variables that make up the privileged group and the non-privileged group.
When I try to import the function:
from aif360.sklearn.metrics import mdss_bias_scan
I get the following error:
Import error: cannot import 'mdss_bias_scan' from 'aif360.sklearn.metrics'.
Can you help me to fix it?

Update
The function mdss_bias_scan is not available in the version of aif360 you're using (v0.4.0).
Here's the source code of the file metrics.py at tag v0.4.0.
The function mdss_bias_scan was added via this commit which has not yet been released.
From the GitHub Source, it seems that you should import it as:
from aif360.sklearn.metrics.metrics import mdss_bias_scan
Also, make sure you have aif360 package installed in your Python environment. If not, install it using:
pip install aif360

Getting exception "No module named 'airflow.providers.sftp'"

I am trying to copy files from SFTP to google cloud storage.
Composer version = 1.16.12
Airflow version = 1.10.15.
While executing getting exception No module named 'airflow.providers.sftp'.
Much appreciated if some one can give pointers.
Code snippet is :
import os
import airflow
from airflow import DAG
from airflow import models
from airflow.operators import python_operator
from airflow.providers.google.cloud.transfers.sftp_to_gcs import SFTPToGCSOperator
from airflow.utils.dates import days_ago
with models.DAG("test_ssh_to_gcs", start_date=days_ago(1), schedule_interval=None) as dag:
copy_file_from_ssh_to_gcs = SFTPToGCSOperator(
task_id="file-copy-ssh-to-gcs",
source_path="/ ",
destination_bucket='test_sftp_to_gcs',
destination_path="test/test.csv",
gcp_conn_id="google_cloud_default",
sftp_conn_id="sftp_test",
)
copy_file_from_ssh_to_gcs

First, have you tried installing the package with pip install apache-airflow-providers-sftp ?
Be also careful about the documentation version you are refering to. With Airflow 2.0, some packages have been moved.

You get the error because SFTPToGCSOperator uses airflow.providers.sftp.operators.SFTPOperator under the hood, which present in airflow >= 2.0.0.
The bad news is need to upgrade your Airflow version to use airflow.providers.google.cloud.transfers.sftp_to_gcs.SFTPToGCSOperator.
If you don't want / cannot upgrade airflow, you can create a DAG chaining two operators:
Operator
Airflow 1.x import
SFTP Operator (download a file to a local using 'operation=get')
from airflow.contrib.operators.sftp_operator import SFTPOperator
Upload File to Google cloud storage
from airflow.contrib.operators.file_to_gcs import FileToGoogleCloudStorageOperator
This should do the trick:
LOCALFILE = '/tmp/kk'
with models.DAG("test_ssh_to_gcs", start_date=days_ago(1), schedule_interval=None) as dag:
download_sftp = SFTPOperator(
task_id = 'part1_sftp_download_to_local',
ssh_conn_id="sftp_test",
local_file=LOCALFILE,
remote_file='',
operation='get')
gcp_upload = FileToGoogleCloudStorageOperator(
task_id='part2_upload_to_gcs',
bucket='test_sftp_to_gcs',
src=LOCALFILE,
dst="test/test.csv",
google_cloud_storage_conn_id="google_cloud_default" # configured in Airflow
)
sftp_download >> gcp_upload

With airflow 1.10 you can install backported packages
For your case the following need to be added to your composer cluster:
1- apache-airflow-backport-providers-google
2- apache-airflow-backport-providers-sftp
3- apache-airflow-backport-providers-ssh

Install dependency via pip.
pip install apache-airflow-providers-sftp

Error with Python SDK for IBM Watson when importing into Pycharm

The error I receive is:
DeprecationWarning: watson-developer-cloud moved to ibm-watson. To get
updates, use the new package.
service = watson_developer_cloud.AssistantV1(
I have tried updating watson-developer-cloud using pip install however I still have the same error.
The code used is below. All done in Python. Just left out the API key from the original code.
Any help is appreciated.
service = watson_developer_cloud.AssistantV1(
iam_apikey= '',
version= '2021-01-20'
import os
from pathlib import Path
import slack
import ibm_watson
import ibm_cloud_sdk_core
import watson_developer_cloud
from ibm_watson import AssistantV1
from dotenv import load_dotenv
)

See here for the instructions on that Python package for IBM Watson services. It is like stated in the warning:
watson-developer-cloud is now named ibm-watson. What you have to do is
pip install ibm-watson
or
pip install --upgrade ibm-watson
Because the packagae is named ibm-watson, you would need to use that name for import...
import ibm-watson
or
from ibm_watson import AssistantV1
See the linked repository for examples.

AWS Lambda with Zappa fails on "Unable to import module 'handler': No module named builtins"

I've deployed my application to AWS Lambda with Zappa. The deployment went fine without any issues. But when I try to reach the app via AWS API Gateway I get a 500 response.
The logs says
Unable to import module 'handler': No module named builtins
The environment runs on Python 2.7. I've added future to the Pipfile but it still won't work.
When I check the code in Lambda I can see the following code
from __future__ import unicode_literals
import base64
import boto3
import collections
import datetime
import importlib
import inspect
import json
import logging
import os
import sys
import traceback
import zipfile
from builtins import str
I can't find what can be wrong. Does anyone have a clue?

I solved the problem. I'm using pipenv to install my packages, but for some reason the packages wasn't installed inside the virtual environment, which made Zappa to not include them to AWS Lambda. I had to install them manually in PyCharm by going through Settings > Project > Project Interpreter and add the packages there. Thanks for your help.