Develop Reference

Trying to reuse code across multiple Flask servers, stuck on importing with errors "ModuleNotFoundError" or "ImportError"

My goal is to import code into three separate Flask servers. It's not going well. I am on python 3.10.4. I have read perhaps 10 different posts that say things like "put a __init__.py file in your folders" which I have done.
For context I'm not exactly new to Python but I've never learned the importing/module system properly.
I have three Flask servers that run scraping operations on different (but similar) websites. I need them to be separate for various reasons. Anyway, all three need to run the same procedure of getting an IP for a proxy from my proxy provider. For this I have some code:
# we don't need the details here so I snip it to save space
def get_proxy_ip(choice):
r = requests.get(download_list, headers={"Authorization": "Token " + token})
selected_proxy_ip = r.json()["results"][choice]["proxy_address"]
selected_proxy_port = r.json()["results"][choice]["port"]
print(selected_proxy_ip)
return selected_proxy_ip, selected_proxy_port
I want to use this function across all 3 of my Flask servers. Here are some various ways I've tried to import the code into one of the Flask servers:
scrapers/rentCanada/app.py
import requests
from flask import Flask, request, make_response
print("cats")
app = Flask(__name__)
print(__name__, __package__)
# from ..shared.ipgetter import get_proxy_ip
# from ..shared.checker import check_public_ip
# from scrapers.shared.ipgetter import get_proxy_ip
# from scrapers.shared.checker import check_public_ip
import shared.ipgetter as ipgetter
import shared.checker as checker
None of them work.
import shared.ipgetter as ipgetter yields:
cats
__main__ None
Traceback (most recent call last):
File "/home/rlm/Code/canadaAps/scrapers/rentCanada/app.py", line 10, in <module>
import shared.ipgetter as ipgetter
ModuleNotFoundError: No module named 'shared'
ModuleNotFoundError: No module named 'scrapers' yields: ModuleNotFoundError: No module named 'scrapers'
from ..shared.ipgetter import get_proxy_ip yields: ImportError: attempted relative import with no known parent package
At this point you need to see my folder structure.
/scrapers
..__init__.py
..setup.py
../rentCanada
.....__init__.py
.....app.py
../rentFaster
.....__init__.py
.....app.py
../rentSeeker
.....__init__.py
.....app.py
../shared
.....__init__.py
.....ipgetter.py
.....checker.py
I need to be able to use any of the app.py files as entry points.
I also tried setup.py with this:
from setuptools import setup, find_packages
setup(
name = 'tools',
packages = find_packages(),
)
followed by python setup.py install but that didn't make a "tools" import available in app.py like I wanted.
As a final note I suspect someone will tell me to use a blueprint. To me those look like a tool I'd use if I was adding a route. I'm not sure they're right for a simple function, but maybe I'm wrong.

My solution for now is to run Flask with python rentCanada/app.py from the /scrapers folder and use this code
import sys
from pathlib import Path
sys.path.append(str(Path(__file__).parent.parent)) # necessary so util folder is available
import requests
from flask import Flask, request, make_response
print("cats")
app = Flask(__name__)
print(__name__, __package__)
from util.ipgetter import get_proxy_ip
from util.checker import check_public_ip
So the program appends the app.py file folder's parent folder to the path. That makes the util folder (which used to be shared but had a naming conflict) available within the app.py file.

How Do I Make an API Call from Python v2.7.13 When the Requests Module Isn't Available

Good-day Folks,
I am writing a small Python script that will be used on a Ubiquiti EdgeRouter 12P to make an API Call, using Digest Authentication, against another router for some JSON data. This is my first attempt at writing a Python script and I have been able to do this using the Requests module, but only in a Python v3.x.y environment. As I was writing my script, I discovered that the Python version on the EdgeRouter is v2.7.13 and the Requests module isn't installed/loaded.
I have attempted to do a pip install requests but it fails with an invalid syntax error message. So with my limited knowledge, I can't figure out what my options are now. Googled around a bit and saw references to using UrlLib or UrlLib2 - but I'm struggling with figuring out how to use either to make my API Call using Digest Authentication.
I would very much like to stick to the Requests module, as it appears to be the simplest and cleanest approach. Below is a snippet of my code, any help would be really appreciated, thanks.
PS. My script is not yet finished, as I'm still learning how to parse the response data received.
HERE'S MY SCRIPT
#PYTHON MODULE IMPORTS
import sys #Import the Python Sys module - used later to detect the Python version
PythonVersion = sys.version_info.major #Needed to put this global variable up here, so I can use it early
import json #Import the JSON module - used to print the API Call response content in JSON format
if PythonVersion == 3: #Import the REQUESTS and HTTPDigestAuth modules - used to make API Calls if we detect Python version 3
import requests
from requests.auth import HTTPDigestAuth
if PythonVersion == 2: #Import the UrlLib module - used to make API Calls if we detect Python version 2
import urllib
#GLOBAL VARIABLES
NodeSerial = '1234'
NodeIP = '192.168.1.100'
BasePath = 'api/v0.1'
apiURL = f"http://{NodeIP}/{BasePath}/MagiNodes"
#Define the credentials to use
Username='admin'
Passwd='mypassword'
#MAIN ROUTINE
print("This is a test script")
if PythonVersion == 2:
print("Python Version 2 detected")
elif PythonVersion == 3:
print("Python Version 3 detected")
print(f"Now querying MagiLink-{NodeSerial}...")
Query1 = requests.get(f"{apiURL}/{NodeSerial}", auth=HTTPDigestAuth(Username, Passwd)) #Using an F string
pretty_Query1 = json.dumps(Query1.json(), indent=3) #Pretty printing the response content
print(pretty_Query1)

Can I import a list of modules from a shared file? i.e. can I import imports?

I have about a dozen python module imports that are going to be reused on many different scrapers, and I would love to just throw them into a single file (scraper_functions.py) that also contains a bunch of functions, like this:
import smtplib
import requests
import re
from urllib.request import urlopen
from bs4 import BeautifulSoup
import time
def function_name(var1)
# function code here
then in my scraper I would simply do something like:
import scraper_functions
and be done with it. But listing the imports at the top of scraper_functions.py doesn't work, and neither does putting all the imports in a function. In each case I get errors in the scraper that is doing the importing.
Traceback (most recent call last):
File "{actual-scraper-name-here}.py", line 24, in <module>
x = requests.get(main_url)
NameError: name 'requests' is not defined
In addition, in VSCode, under Problems, I get errors like
Undefined variable 'requests' pylint(undefined-variable) [24,5]
None of the modules are recognized. I have made sure that all files are in the same directory.
Is such a thing possible please?

You need to either use the scraper_functions prefix (same way you do this import name) or use the from keyword to import your things from scraper_functions with the * selector.
Using the form keyword (Recommended)
from scraper_functions import * # import everything with *
...
x = requests.get(main_url)
Using the scraper_functions prefix (Not recommended)
import scraper_functions
...
x = scraper_functions.requests.get(main_url)

Pylinting a python 2 fallback

I have a python module with a python 2 fallback set up through try/catch.
try:
from urllib.parse import urlencode
except ImportError:
from urlib import urlencode
When I pylint the file I get no name 'urlencode' in module 'urllib' and similar errors. Is there anyway to specify python 2 linting for a block, disable all linting for a block, or am I stuck hand squelching all errors?

I settled on a nicer way of doing this by disabling the linting errors at the beginning of a python 2 block, and then re enabling them at the end.
# pylint: disable=no-name-in-module, import-error
from urllib import urlencode
from urllib2 import urlopen
# pylint: enable=no-name-in-module, import-error

python urllib error

so I have this code:
def crawl(self, url):
data = urllib.request.urlopen(url)
print(data)
but then when I call the function, it returns
data = urllib.request.urlopen(url)
AttributeError: 'module' object has no attribute 'request'
what did I do wrong? I already imported urllib..
using python 3.1.3

In python3, urllib is a package with three modules request, response, and error for its respective purposes.
Whenever you had import urllib or import urllib2 in Python2.
Replace them with
import urllib.request
import urllib.response
import urllib.error
The classs and methods are same.
BTW, use 2to3 tool if you converting from python2 to python3.

urllib.request is a separate module; import it explicitly.