Develop Reference

NameError: name 'athena' is not defined , when importing athena query function from another jupyter notebook

My query_distinct_data() function executes successfully when run.
But when I try to import the query_distinct_data() using Jupyter notebook from my function page map_distinct_data on to my main page I get the following error.
NameError: name 'athena' is not defined
Below is my main page below
import pandas as pd
import requests
import xml.etree.ElementTree as ET
from datetime import date
import boto3
import time
import geopandas
import folium
from ipynb.fs.full.qld_2 import qld_data
from ipynb.fs.full.vic_2 import vic_data
from ipynb.fs.full.put_to_s3_bucket import put_to_s3_bucket
from ipynb.fs.full.map_distinct_data import query_distinct_data
from ipynb.fs.full.map_distinct_data import distinct_data_df
from ipynb.fs.full.map_distinct_data import create_distinct_data_map
aws_region = "ap-southeast-2"
schema_name = "fire_data"
table_name ='rfs_fire_data'
result_output_location = "s3://camgoo2-rfs-visualisation/query_results/"
bucket='camgoo2-rfs-visualisation'
athena = boto3.client("athena",region_name=aws_region)
qld_data()
vic_data()
put_to_s3_bucket()
execution_id = query_distinct_data()
df = distinct_data_df()
create_distinct_data_map()
Below is my function that I am wanting to import from map_distinct_data notebook. This successfully executes but am getting the error when trying to import to my main page.
def query_distinct_data():
query = "SELECT DISTINCT * from fire_data.rfs_fire_data where state in ('NSW','VIC','QLD')"
response = athena.start_query_execution(
QueryString=query,
ResultConfiguration={"OutputLocation": result_output_location})
return response["QueryExecutionId"]
I am able to run function query_distinct_data() and it executes when run seperately.
But it fails when I try to import the function.
The other functions that I import using ipynb.fs.full that do involve athena are executing okay when imported.

It is all about variables visibility scope (1, 2)
In short: map_distinct_data module knows nothing about main page's athena variable.
The good and correct way is to pass athena variable inside function as parameter:
from ipynb.fs.full.map_distinct_data import create_distinct_data_map
...
athena = boto3.client("athena",region_name=aws_region)
execution_id = create_distinct_data_map(athena)
where create_distinct_data_map should be defined as
def create_distinct_data_map(athena):
...
The second way is to set variable inside imported module:
from ipynb.fs.full.map_distinct_data import create_distinct_data_map
from ipynb.fs.full import map_distinct_data
athena = boto3.client("athena",region_name=aws_region)
map_distinct_data.athena = athena
execution_id = create_distinct_data_map()
Even if second way is working it is a really bad style.
Here is some must to know information about encapsulation in Python.

Error Importing a variable from one Python script to another

I have two Python (3.8) scripts located in the same folder.
The first lookup.py is simply:
#! /usr/bin/env python3
import os
from getimp import session0
print (session0)
The second script getimp.py identifies a cookie and sets it as a variable which is imported into the first script. I have omitted some of the code here, but hopefully have the critical parts.
#! /usr/bin/env python3
import os
import json
import base64
import sqlite3
import shutil
from datetime import datetime, timedelta
import win32crypt # pip install pypiwin32
from Crypto.Cipher import AES # pip install pycryptodome
def get_chrome_datetime(chromedate):
"""Return a `datetime.datetime` object from a chrome format datetime
....
....
# you can also search by domain, e.g thepythoncode.com
cursor.execute("""
SELECT host_key, name, value, creation_utc, last_access_utc, expires_utc, encrypted_value
FROM cookies
WHERE name like '%user_id%'""")
# get the AES key
key = get_encryption_key()
for host_key, name, value, creation_utc, last_access_utc, expires_utc, encrypted_value in cursor.fetchall():
if not value:
decrypted_value = decrypt_data(encrypted_value, key)
else:
# already decrypted
decrypted_value = value
print(f"""
{decrypted_value}
===============================================================""")
session0 = decrypted_value
if __name__ == "__main__":
main()
If I run getimp.py on its own it generates the correct result but when I run lookup.py I get an error:
File "lookup", line 4, in <module>
from getimp import session0
ImportError: cannot import name 'session0' from 'getimp' (D:\Documents\ptest\getimp.py)
Am I losing the variable once the script getimp.py finishes?

Your problem is that the variablesession0 is defined inside the scope of get_chrome_datetime and therefore can not be addressed from import.
Try importing the function and create the variable inside the scope of the active script.
Inside 'get_chrome_datetime' change session0=decrypted_value into return decrypted_value
and in lookup.py :
import os
from getimp import get_chrome_datetime
print (get_chrome_datetime(argument))

Your problem is that session0 is defined inside the function.
I would suggest the following
session0 = None
def get_chrome_datetime(chromedate):
global session0
... (your code here)
Also you should call the function outside of if __name__ == '__main__' because when you're importing a module, the __name__ wouldn't be "__main__"

ModuleNotFoundError in python code - cannot find custom class in console app

I create a python console app that includes imports of a custom class I'm using. Everytime I run my app I get the error ModuleNotFoundError: "No module named 'DataServices'.
Can you help?
Provided below is my folder structure:
ETL
Baseball
Baseball_DataImport.py
DataServices
DataService.py
ConfigServices.py
PageDataMode.py
SportType.py
Here is the import section from the Baseball_DataImport.py file. This is the file when I run I get the error:
from bs4 import BeautifulSoup
import scrapy
import requests
import BaseballEntity
import mechanize
import re
from time import sleep
import logging
import time
import datetime
from functools import wraps
import json
import DataServices.DataService - Error occurs here
Here is my DataService.py file:
import pymongo
import json
import ConfigServices
import PageDataModel
#from SportType import SportType
class DataServices(object):
AppConfig: object
def __init__(self):
AppConfig = ConfigServices.ConfigService()
#print(AppConfig)
#def GetPagingDataBySport(self,Sport:SportType):
def GetPagingDataBySport(self):
#if Sport == SportType.BASEBALL:
pagingData = []
pagingData.append(PageDataModel.PageDataModel("", 2002, 2))
pagingData.append(PageDataModel.PageDataModel("", 2003, 2))
return pagingData

It might seem that your structure is:
Baseball
Baseball_DataImport.py
Dataservices
Dataservice.py
Maybe you need to do from Dataservices.Dataservice import DataServices
Edit:
I created the folder structure, and the method I showed you works:
Here's the implementation
Dataservice.py only contains:
class DataServices():
pass

Did you try copieing the Dataservice.py into the Projectfolder with the main.py?

How to use an import to only one function?

How can I import a module in Python, but only to one function?
My imagination:
def func():
localize from myModule import helperFunc # localize means "to this scope only: "
helperFunc("do something magical") # but it doesn't exist
try:
helperFunc("do something magical")
except NameError:
print("'helperFunc' doesn't exist at this scope") # this would get run
The problem here is that localize doesn't exist. Is there something in Python to simulate that?

You can just import modules normally:
def choose5(lst):
from random import choices
# choices is imported here
return choices(lst, k=5)
print(choose5([1, 2, 3]))
# choices is not imported here

def my_now():
from datetime import datetime
return datetime.now()
try:
print("Success", datetime.now())
except NameError:
print("NameError occurred", my_now())
Running the above code will give you the output
NameError occurred 2020-11-06 21:20:19.930863
datetime was scoped to the my_now function. Trying to calling datetime.now() directly failed, so the function my_now was called and that was able to call datetime.now() successfully.

Module undefined after importing it in a function

I'm trying to use a function called start to set up my enviroment in python. The function imports os.
After I run the function and do the following
os.listdir(simdir+"main")
I get a error that says os not defined
code
>>> def setup ():
import os.path
import shutil
simdir="e:\\"
maindir="c:\\backup\\bitcois\\test exit\\"
>>> setup()
>>> os.listdir(simdir+"main")
Traceback (most recent call last):
File "<pyshell#10>", line 1, in <module>
os.listdir(simdir+"main")
NameError: name 'os' is not defined

The import statement is scoped. When importing modules they are defined for the local namespace.
From the documentation:
Import statements are executed in two steps: (1) find a module, and initialize it if necessary; (2) define a name or names in the local namespace (of the scope where the import statement occurs). [...]
So in your case the os package is only defined within function setup.

You are getting this error because you are NOT importing the whole os library but just the os.path module. In this way, the other resources at the os library are not made available for your use.
In order to be able to use the os.listdir method, you need to either import it alongside the os.path like this:
>>> def setup ():
import os.path, os.listdir
import shutil
simdir="e:\\"
maindir="c:\\backup\\bitcois\\test exit\\"
or import the full library:
>>> def setup ():
import os
import shutil
simdir="e:\\"
maindir="c:\\backup\\bitcois\\test exit\\"
You can read more here:
https://docs.python.org/2/tutorial/modules.html

try:
import os.path
import shutil
import glob
def setup ():
global simdir
simdir="e:\\"
maindir="c:\\backup\\bitcois\\test exit\\"
setup()
os.listdir(simdir+"main")

You need to return the paths and assign the returned values in the global scope. Also, import os too:
import os
def setup():
# retain existing code
return simdir, maindir
simdir, maindir = setup()

When you import os or do any sort of command within a function, the command's effect only last while that function itself is running. What you need to do is
import os
...Do your function and other code
This way, your import lasts for the whole program :).