Develop Reference

Trying to reuse code across multiple Flask servers, stuck on importing with errors "ModuleNotFoundError" or "ImportError"

My goal is to import code into three separate Flask servers. It's not going well. I am on python 3.10.4. I have read perhaps 10 different posts that say things like "put a __init__.py file in your folders" which I have done.
For context I'm not exactly new to Python but I've never learned the importing/module system properly.
I have three Flask servers that run scraping operations on different (but similar) websites. I need them to be separate for various reasons. Anyway, all three need to run the same procedure of getting an IP for a proxy from my proxy provider. For this I have some code:
# we don't need the details here so I snip it to save space
def get_proxy_ip(choice):
r = requests.get(download_list, headers={"Authorization": "Token " + token})
selected_proxy_ip = r.json()["results"][choice]["proxy_address"]
selected_proxy_port = r.json()["results"][choice]["port"]
print(selected_proxy_ip)
return selected_proxy_ip, selected_proxy_port
I want to use this function across all 3 of my Flask servers. Here are some various ways I've tried to import the code into one of the Flask servers:
scrapers/rentCanada/app.py
import requests
from flask import Flask, request, make_response
print("cats")
app = Flask(__name__)
print(__name__, __package__)
# from ..shared.ipgetter import get_proxy_ip
# from ..shared.checker import check_public_ip
# from scrapers.shared.ipgetter import get_proxy_ip
# from scrapers.shared.checker import check_public_ip
import shared.ipgetter as ipgetter
import shared.checker as checker
None of them work.
import shared.ipgetter as ipgetter yields:
cats
__main__ None
Traceback (most recent call last):
File "/home/rlm/Code/canadaAps/scrapers/rentCanada/app.py", line 10, in <module>
import shared.ipgetter as ipgetter
ModuleNotFoundError: No module named 'shared'
ModuleNotFoundError: No module named 'scrapers' yields: ModuleNotFoundError: No module named 'scrapers'
from ..shared.ipgetter import get_proxy_ip yields: ImportError: attempted relative import with no known parent package
At this point you need to see my folder structure.
/scrapers
..__init__.py
..setup.py
../rentCanada
.....__init__.py
.....app.py
../rentFaster
.....__init__.py
.....app.py
../rentSeeker
.....__init__.py
.....app.py
../shared
.....__init__.py
.....ipgetter.py
.....checker.py
I need to be able to use any of the app.py files as entry points.
I also tried setup.py with this:
from setuptools import setup, find_packages
setup(
name = 'tools',
packages = find_packages(),
)
followed by python setup.py install but that didn't make a "tools" import available in app.py like I wanted.
As a final note I suspect someone will tell me to use a blueprint. To me those look like a tool I'd use if I was adding a route. I'm not sure they're right for a simple function, but maybe I'm wrong.

My solution for now is to run Flask with python rentCanada/app.py from the /scrapers folder and use this code
import sys
from pathlib import Path
sys.path.append(str(Path(__file__).parent.parent)) # necessary so util folder is available
import requests
from flask import Flask, request, make_response
print("cats")
app = Flask(__name__)
print(__name__, __package__)
from util.ipgetter import get_proxy_ip
from util.checker import check_public_ip
So the program appends the app.py file folder's parent folder to the path. That makes the util folder (which used to be shared but had a naming conflict) available within the app.py file.

How to use Azure-Python SDK `ResourcesMoveInfo` class in python

I came across this python class ResourcesMoveInfo for moving resources(Azure Images) from one subscription to another with Azure python SDK.
But it's failing when I use it like below:
Pattern 1
reference from https://buildmedia.readthedocs.org/media/pdf/azure-sdk-for-python/v1.0.3/azure-sdk-for-python.pdf
Usage:
metadata = azure.mgmt.resource.resourcemanagement.ResourcesMoveInfo(resources=rid,target_resource_group='/subscriptions/{0}/resourceGroups/{1}'.format(self.prod_subscription_id,self.resource_group))
Error:
AttributeError: module 'azure.mgmt.resource' has no attribute 'resourcemanagement'
Pattern 2
reference from - https://learn.microsoft.com/en-us/python/api/azure-mgmt-resource/azure.mgmt.resource.resources.v2019_07_01.models.resourcesmoveinfo?view=azure-python
Usage:
metadata = azure.mgmt.resource.resources.v2020_06_01.models.ResourcesMoveInfo(resources=rid,target_resource_group='/subscriptions/{0}/resourceGroups/{1}'.format(self.prod_subscription_id,self.resource_group))
Error:
AttributeError: module 'azure.mgmt.resource.resources' has no attribute 'v2020_06_01'
Any help on this requirement/issue would be helpful. Thanks!
Adding code snippet here:
import sys
import os
import time
from azure.common.credentials import ServicePrincipalCredentials
from azure.mgmt.resource import ResourceManagementClient
import azure.mgmt.resource
#from azure.mgmt.resource.resources.v2020_06_01.models import ResourcesMoveInfo
from azure.identity import ClientSecretCredential
from cred_wrapper import CredentialWrapper
class Move():
def __init__(self):
self.nonprod_subscription_id = "abc"
self.prod_subscription_id = "def"
self.credential = ClientSecretCredential(
client_id= os.environ["ARM_CLIENT_ID"],
client_secret= os.environ["ARM_CLIENT_SECRET"],
tenant_id= os.environ["ARM_TENANT_ID"]
)
#resource client for nonprod
self.sp = CredentialWrapper(self.credential)
self.resource_client = ResourceManagementClient(self.sp,self.nonprod_subscription_id)
self.resource_group = "imgs-rg"
def getresourceids(self):
resource_ids = list(resource.id for resource in self.resource_client.resources.list_by_resource_group("{0}".format(self.resource_group)) if resource.id.find("latest")>=0)
return resource_ids
def getresourcenames(self):
resource_names = list(resource.name for resource in self.resource_client.resources.list_by_resource_group("{0}".format(self.resource_group)) if resource.id.find("latest")>=0)
return resource_names
def deleteoldimages(self,name):
#resource client id for prod
rc = ResourceManagementClient(self.sp,self.prod_subscription_id)
for resource in rc.resources.list_by_resource_group("{0}".format(self.resource_group)):
if resource.name == name:
#2019-12-01 is the latest api_version supported for deleting the resource type "image"
rc.resources.begin_delete_by_id(resource.id,"2020-06-01")
print("deleted {0}".format(resource.name))
def moveimages(self):
rnames = self.getresourcenames()
for rname in rnames:
print(rname)
#self.deleteoldimages(rname)
time.sleep(10)
rids = self.getresourceids()
rid = list(rids[0:])
print(rid)
metadata = azure.mgmt.resource.resources.v2020_06_01.models.ResourcesMoveInfo(resources=rid,target_resource_group='/subscriptions/{0}/resourceGroups/{1}'.format(self.prod_subscription_id,self.resource_group))
#moving resources in the rid from nonprod subscription to prod subscription under the resource group avrc-imgs-rg
if rid != []:
print("moving {0}".format(rid))
print(self.resource_client.resources.move_resources(source_resource_group_name="{0}".format(self.resource_group),parameters=metadata))
#self.resource_client.resources.move_resources(source_resource_group_name="{0}".format(self.resource_group),resources=rid,target_resource_group='/subscriptions/{0}/resourceGroups/{1}'.format(self.prod_subscription_id,self.resource_group))
#self.resource_client.resources.begin_move_resources(source_resource_group_name="{0}".format(self.resource_group),parameters=metadata)
if __name__ == "__main__":
Move().moveimages()

From your inputs we can see that the code looks fine. From your error messages, the problem is with importing the modules.
Basically when we import a module few submodules will get installed along with and few will not. This will depend on the version of the package, to understand which modules are involved in a specific version we need to check for version-releases in official documentation.
In your case, looks like some resource modules were missing, if you could see the entire error-trace, there will be a path with sitepackages in our local. Try to find that package and its subfolder(Modules) and compare them with Azure SDK for Python under Resource module, you can find this here.
In such situation we need to explicitly add those sub modules under our package. In your case you can simple download the packaged code from Git link which I gave and can merge in your local folder.

Unable to import scipy library using Flask & Elastic Beanstalk

I am trying to set up a Flask web app using Elastic Beanstalk on AWS. I have followed the tutorial here and that works fine. I am now looking to expand the Flask webapp, and this works fine, until I import scipy.spatial as spatial, when this is part of my import statements, running eb open just times out. I receive
>>>> HTTP ERROR 504
running the webapp locally works absolutely fine even with the scipy import, it is only when I try and deploy to beanstalk that it doesn't want to work. Below is my code
import os
import requests
from bs4 import BeautifulSoup
import pandas as pd
import scipy.spatial as spatial ##### Removing this and everything works!
from flask import Flask
from flask_cors import CORS
from flask_restful import Resource, Api
from flask_jsonpify import jsonify
# print a nice greeting.
def say_hello(username = "World"):
df = pd.DataFrame({"a":[1,2,3]})
return '<p>Hello %s!</p>\n' % username
# some bits of text for the page.
header_text = '''
<html>\n<head> <title>EB Flask Test</title> </head>\n<body>'''
instructions = '''
<p><em>Hint</em>: This is a RESTful web service! Append a username
to the URL (for example: <code>/Thelonious</code>) to say hello to
someone specific.</p>\n'''
home_link = '<p>Back</p>\n'
footer_text = '</body>\n</html>'
# EB looks for an 'application' callable by default.
application = Flask(__name__)
# add a rule for the index page.
application.add_url_rule('/', 'index', (lambda: header_text +
say_hello() + instructions + footer_text))
# add a rule when the page is accessed with a name appended to the site
# URL.
application.add_url_rule('/<username>', 'hello', (lambda username:
header_text + say_hello(username) + home_link + footer_text))
# run the app.
if __name__ == "__main__":
# Setting debug to True enables debug output. This line should be
# removed before deploying a production app.
application.debug = True
application.run()
I have tried increasing the command timeout for the environment from 600 to 900, although the timeout error occurs well before 600 seconds has elapsed.

Right, I am not sure why this is the case but I updated the version of scipy in my requirements.txt and the app is working as expected!
Originally I had
scipy==1.4.1
Now I have
scipy==1.2.3
I have no idea why this has fixed the deployment issue, especially given 1.4.1 works perfectly locally. If anyone has an idea, or if this a bug I should be reporting it would be good to know!

Flask Testing in Python -- Building one API in a repo with many to unit test it via import_module

We have an ETL data API repo. We do all etl processing inside of it, then spit the data out in API's. These API's are run one at a time from a single command passing the resource class through to the server to build an API. the resource class is in a web directory in an __init__.py.
This is a wonderful convention and quite simple to use, but the problem I am having is coming from trying to get one of the 3 API's available spun up for testing. Our directory stucture is like this (calling the project 'tomato')
tomato
- category_api
- web
- etl
- test
- here is where we are writing some tests (test_category_api.py)
- data
- article_api
- web
- etl
- test
- data
- recommendation_api
- web
- etl
- test
- data
- common
- common shit
Inside this test, I have the following test class. On the seventh line up from the bottom,
you will see a comment on where it breaks. It is the import_module method.
import unittest
import sys
import os
import sys
import json
from importlib import import_module
from flask import Flask
from flask_restful import Api, abort, wraps
from flask_restful.utils import cors
from flask.ext.testing import TestCase
#dir_above_top_level = os.path.join(os.path.abspath(__file__), '../../.. /')
#sys.path.append(os.path.abspath(dir_above_top_level))
_CATEGORY_ENDPOINT = '/v1/category/'
_PACKAGE_NAME = os.environ['RECO_API_PACKAGE_NAME']
_CORS = cors.crossdomain(origin='*',
headers=['Origin', 'X-Requested-With',
'Content-Type', 'Accept'],
methods=['GET', 'HEAD', 'OPTIONS'],
max_age=3600)
class CategoryTests(TestCase):
def __init__(self):
self.app = Flask(__name__)
self._configure_app()
for resource in self.resource_classes:
self.api.add_resource(self.resource,
self.resource.URI_TEMPLATE)
def test_status_code(self):
self.response = self.client.post(_CATEGORY_ENDPOINT,
data=json.dumps(
{'title': 'Enjoy this delicious food'}),
headers=json.dumps(
{'content-type':'application/json'}))
self.assertEqual(self.response.status_code, 200)
def test_version(self):
self.response = self.client.post(_CATEGORY_ENDPOINT,
data=json.dumps(
{"title": "eat some delicious stuff"}),
headers=json.dumps(
{'content-type':'application/json'}))
self.assertEqual(json.dumps(self.response['version']), '1')
def _configure_app(self):
self.app = Flask(__name__)
self.app.config['TESTING'] = True
self.app.debug = True
self.decorators = [_CORS]
self.app.Threaded = True
self.web_package = 'tomato.category.web'
self.package = import_module('.__init__', self.web_package) # WE BREAK HERE
self.resources = package.RESOURCE_NAMES
self.resource_classes = [ getattr(package, resource) for resource in resources ]
self.api = Api(self.app, catch_all_404s=True, decorators=self.decorators)
if __name__ == '__main__':
unittest.main()
we are given an exception when running these tests:
ImportError: No module named tomato.category.web.__init__
yet cd into the main top dir, and ls tomato/category/web gets us __init__.py and its right there with the resource class.
How do I import this class so that I can instantiate the API to run the tests in this class?
Or if I'm completely on the wrong track what should I be doing instead?

You don't need to import __init__, just like you probably wouldn't do from tomato.category.web import __init__. You should be able to import the web package directly.
self.web_package = 'tomato.category.web'
self.package = import_module(self.web_package)

The problem here lies in the directory structure. In the current path, I am not at the top level. It is a module. So What was needed was to uncomment the line two lines at the top, and change the structure to append the path like this.
dir_above_top_level = os.path.join(os.path.abspath(__file__), '../../../..')
sys.path.append(os.path.abspath(dir_above_top_level))
and now, I can import it using
self.web_package = 'tomato.category.web'
self.package = import_module('.__init__', self.web_package)
and now it will import fine and I can grab the resource class to set up the testing API

How do you set the results_directory in SST Python

I'm feeling a bit foolish asking this, as with basic selenium I have no problem saving screenshots, yet with SST I use the take_screenshot('screenshot_name.png') it tells me that the results_directory should be set. Question is how do you set the results_directory. All of the examples I find set it to "NONE", yet that doesn't satisfy my test's need.
Below is how my code is written:
import unittest
from sst.actions import *
from sst import cases, config
config.results_directory = None
class TestMyTest(cases.SSTTestCase):
def test_mytestcase_home_page(self):
go_to('http://www.mywebpage.com')
assert_title_contains('MyWebPage')
#Main page is displayed
take_screenshot(filename='C/Users/Brenda/test/SST Test Project/results/home_page.png',add_timestamp=True)

I had the following script working for me using Google. The trick was to add result directory to actual config file which is located #{dir}\Python27\Lib\site-packages\sst\config.py and add results_directory = "C:\Users\{me}\Desktop\Python-pip-SST\results"
import unittest
from sst.actions import *
from sst import cases, config
#config.results_directory = "C:\Users\{me}\Desktop\Python-pip-SST\results"
go_to('https://www.google.com/')
assert_title_contains('Google')
#Main page is displayed
take_screenshot(filename='home_page.png',add_timestamp=True)
And, You also should be able to overwrite the result path from your test. Your working code should look like something like the following
import unittest
from sst.actions import *
from sst import cases, config
#Just to be safe side try not to use any spaces in filename
config.results_directory = "C:/Users/Brenda/test/SSTTestProject/results"
class TestMyTest(cases.SSTTestCase):
def test_mytestcase_home_page(self):
go_to('http://www.mywebpage.com')
assert_title_contains('MyWebPage')
#Main page is displayed
take_screenshot(filename="home_page.png",add_timestamp=True)
I added the screenshot if that helps you somehow. Changing filepath innconfig file or from test works fine for me