Develop Reference

Replacing def add_video(url, user_id=None, **kwargs): with a video service's api to embed videos in Python web app. What do I need to learn?

Looking to be pointed to the right direction to understand what exactly I need to learn next since there's so much info out there but I feel as if none are helping me at the moment so I'm trying to get a gist of what I need to understand more of to get this part covered.
I learned a bit of Python from Justin Mitchel from his 30 days of Python and from his Create a Video Membership Web App from Scratch with Python, NoSQL, & FastAPI 11 Hr video and I understand more but I realized that there's a few things I need help with.
I plan on learning more about htmx to get better with pages but using the pages for specifics videos via an api service has me baffled at the moment. The 2 apis I plan on using are Streamtape & Doodstream.
https://github.com/wahyubiman/DoodStream
https://github.com/scaldings/streamtape-api-python
My current requirement.txt are
fastapi
uvicorn
cassandra-driver
python-dotenv
email-validator
argon2-cffi
pytest
jinja2
python-multipart
python-jose[cryptography]
algoliasearch
doodstream
Had to manually install streamtape.
So the current code for my model.py for videos is
import uuid
from app.config import get_settings
from app.users.exceptions import InvalidUserIDException
from app.users.models import User
from app.shortcuts import templates
from cassandra.cqlengine import columns
from cassandra.cqlengine.models import Model
from cassandra.cqlengine.query import (DoesNotExist, MultipleObjectsReturned)
settings = get_settings()
from .exceptions import (
InvalidVideoURLException,
VideoAlreadyAddedException
)
# Unlisted Video -> video_id -> lock it down
class Video(Model):
__keyspace__ = settings.keyspace
host_id = columns.Text(primary_key=True) # Streamtape, Doodstream
db_id = columns.UUID(primary_key=True, default=uuid.uuid1) # UUID1
host_service = columns.Text(default='Doodstream')
title = columns.Text()
url = columns.Text() # secure
user_id = columns.UUID()
def __str__(self):
return self.__repr__()
def __repr__(self):
return f"Video(title={self.title}, host_id={self.host_id}, host_service={self.host_service})"
def render(self):
basename = self.host_service # streamtape, doodstream
template_name = f"videos/renderers/{basename}.html"
context = {"host_id": self.host_id}
t = templates.get_template(template_name)
return t.render(context)
def as_data(self):
return {f"{self.host_service}_id": self.host_id, "path": self.path, "title": self.title}
#property
def path(self):
return f"/videos/{self.host_id}"
#staticmethod
def get_or_create(url, user_id=None, **kwargs):
host_id = extract_video_id(url)
obj = None
created = False
try:
obj = Video.objects.get(host_id=host_id)
except MultipleObjectsReturned:
q = Video.objects.allow_filtering().filter(host_id=host_id)
obj = q.first()
except DoesNotExist:
obj = Video.add_video(url, user_id=user_id, **kwargs)
created = True
except:
raise Exception("Invalid Request")
return obj, created
#staticmethod
def add_video(url, user_id=None, **kwargs):
# extract video_id from url
# video_id = host_id
# Service API - Streamtape / Doostream / etc
host_id = extract_video_id(url)
if host_id is None:
raise InvalidVideoURLException("Invalid Video URL")
user_exists = User.check_exists(user_id)
if user_exists is None:
raise InvalidUserIDException("Invalid user_id")
# user_obj = User.by_user_id(user_id)
# user_obj.display_name
q = Video.objects.allow_filtering().filter(host_id=host_id) # , user_id=user_id)
if q.count() != 0:
raise VideoAlreadyAddedException("Video already added")
return Video.create(host_id=host_id, user_id=user_id, url=url, **kwargs)
# class PrivateVideo(Video):
# pass
What I'm trying to figure out is
How can I replace the static method with my API so it can pull on search requests if that makes sense? Like if someone is searching for something specific it brings them to a search page that pulls to whatever's close to what's looked for from Streamtape/Doodstream.
Or if I want to have preset tags for them to click on it shows all the videos available to be played, in which they can choose either the Streamtape server or the Doodstream server.
For the tags, would it be wise to make a separate html page per tag, to where it pulls from the specific folder of videos I want it to pull from? Like on the navigation bar has categories, and when clicking on categories it shows different ones like Education, Travel, etc and when they click on one it shows all the videos available that matches the folder from Stream/Dood that I want it to pull from.
What do I need to learn to get anywhere close to achieving something like that?

Flask, using form inputs as global variable execute custom ETL module using form submission as input and then display the charts after completion

I currently have my flask running static where I run the ETL job independently and then use the result dataset in my Flask to display chartjs line charts.
However, I'd like to integrate the ETL piece in my web framework where my users can login and submit the input parameter(locations of the input files and an added version id) using HTML form, which will then be used by my ETL job to run and use the resultant data directly to display the charts on same page.
Current Setup:
My custom ETL module has submodules that act together to form a simple pipeline process:
globals.py - has my globals such as location of the s3 and the etc. ideally i'd like my user's form inputs to be stored here so that they can be used directly in all my submodules wherever necessary.
s3_bkt = 'abc' #change bucket here
s3_loc = 's3://'+s3_bkt+'/'
ip_loc = 'alv-input/'
#Ideally ,I'd like my users form inputs to be sitting here
# ip1 = 'alv_ip.csv'
# ip2 = 'Input_product_ICL_60K_Seg15.xlsx'
#version = 'v1'
op_loc = 'alv-output/'
---main-module.py - main function
import module3 as m3
import globals as g
def main(ip1,ip2,version):
data3,ip1,ip2,version = m3.module3(ip1,ip2,version)
----perform some actions on the data and return---
return res_data
---module3.py
import module2 as m2
def mod3(ip1,ip2,version):
data2,ip1,ip2,version = m2.mod2(ip1,ip2,version)
----perform some actions on the data and return---
return data3
---module2.py
import module1 as m1
import globals as g
def mod2(ip1,ip2,version):
data1,ip1,ip2,version = m1.mod1(ip1,ip2,version)
data_cnsts = pd.read_csv(ip2) #this is where i'll be using the user's input for ip2
----perform some actions on the datasets and write them to location with version_id to return---
data1.to_csv(g.op_loc+'data2-'+ version + '.csv', index=False)
return data2
---module1.py
import globals as g
def mod1(ip1,ip2,version):
#this is the location where the form input for the data location should be actually used
data = pd.read_csv(g.s3_loc+g.ip_loc+ip1)
----perform some actions on the data and return---
return data1
Flask setup:
import main-module as mm
app = Flask(__name__)
#this is where the user first hits and submits the form
#app.route('/form')
def form():
return render_template('form.html')
#app.route('/result/',methods=['GET', 'POST'])
def upload():
msg=''
if request.method == 'GET':
return f"The URL /data is accessed directly. Try going to '/upload' to submit form"
if request.method == 'POST':
ip1 = request.form['ip_file']
ip2 = request.form['ip_sas_file']
version = request.form['version']
data = mm.main(ip1,ip2,version)
grpby_vars = ['a','b','c']
grouped_data = data.groupby(['mob'])[grpby_vars].mean().reset_index()
#labels for the chart
a = [val for val in grouped_data['a']]
#values for the chart
b = [round(val*100,3) for val in grouped_data['b']]
c = [round(val*100,3) for val in grouped_data['c']]
d = [val for val in grouped_data['d']]
return render_template('results.html', title='Predictions',a=a,b=b,c=c,d=d)
The Flask setup works perfectly fine without using any form inputs from the user(if the ETL job and Flask is de-coupled i.e. when I run the ETL and supply the result data location to Flask directly.
Problem:
The problem after integration is that I'm not very sure how to pass these inputs from users to all my sub-module. Below is the error I get when I use the current setup.
data3,ip1,ip2,version = m3.module3(ip1,ip2,version)
TypeError: module3() missing 3 required positional arguments: 'ip1', 'ip2', and 'version'
So, definitely this is due to the issue with my param passing across my sub-modules.
So my question is how do I use the data from the form as a global variable across my sub-modules. Ideally I'd like them to be stored in my globals so that I'd not have to be passing them as a param through all modules.
Is there a standard way to achieve that? Might sound very trivial but I'm struggling hard to get to my end-state.
Thanks for reading through :)

I realized my dumb mistake, I should've have just included
data,ip1,ip2,version = mm.main(ip1,ip2,version)
I could also instead use my globals file by initiating the inputs with empty string and then import my globals in the flask file and update the values. This way I can avoid the route of passing the params through my sub-modules.

Reference file with JSON locators class

I have JSON data in my main file, lets call this main.py (this uses Selenium driver):
main.py
def get_user_data(browser):
browser.get(JSON_URL)
user_data = json.loads(browser.find_element_by_tag_name('body').text)
browser.back()
return user_data
To get a value from this, I use:
name = user_data['name']
This works just fine. This is a larger program and I would like to have these locators in another file called locators.py using a class so if a locator changes, I just change it in one place:
locators.py
class UserDataLocators:
NAME = user_data['name']
Now, the above fails because 'user_data' is not defined. I like this approach since I have other selenium locators in this file in a separate class. This may be a super simple fix, but how would I get the above to work so in my main.py file, I can import UserDataLocators and do the following:
from locators import UserDataLocators
name = UserDataLocators.NAME
Thanks!
Update 1:
#maxhaz So with what you wrote, it gave me an idea to maybe just use your UserData class to parse everything instead of just storing locators. Basically this is what I'm looking to accomplish. The user_data['name'] will get updated/changed as other modules interact with it. I'd like to store the user_data in locators.py and be able to access it and update it with the other modules, and after everytime it is update, each module can get the updated data.
For example, locators.py has the user_data as None to start since no browser is initiated. Main.py will be the first to interact with it and passes the browser to get the initial user_data['name'] value. Module2.py will also import from locators.py and get that same user_data that main.py just updated. Module2 will update the user_data['name']. Once this happens, is this new user_data['name'] now updated for main.py and locators.py?
#locators.py
user_data = None
class UserData:
def __init__(self, browser):
self.browser = browser
self.name = self.get_user_data()['name']
def get_user_data(self):
self.browser.get(JSON_URL)
user_data = json.loads(browser.find_element_by_tag_name('body').text)
browser.back()
return user_data
#main.py
import locators
locators.user_data = UserData(browser).name
#module2.py
import locators
## do something here that modifies the actual user data that gets pulled ##
locators.user_data = UserData(browser).name

You probably want to move the get_user_data function in the class:
#locators.py
class UserDataLocators:
def __init__(self, browser):
self.browser = browser
self.name = self.get_user_data()['name']
def get_user_data(self):
self.browser.get(JSON_URL)
user_data = json.loads(browser.find_element_by_tag_name('body').text)
browser.back()
return user_data
#main.py
from locators import UserDataLocators
name = UserDataLocators(browser).name

Robobrowser and flask errors

I am trying to create a script for myself to use on a few classified sites and starting with cl, I am using flask web framework and robobrowser but not going so well.
The Goal It will take my preset values and put them in the fields from that classifieds websites. Doesnt seem like a difficult concept however after 5 hours of reading online different code and trial and error I remembered the best developers are on stack...
I should inform you I am new to Python and still have alot to learn so any help would be greatly appreciated.
The error I get is:
assert isinstance(form, 'str')
TypeError: isinstance() arg 2 must be a type or tuple of types
but I dont see how to fix this and completely lost. HELP!!!
thanks in advance
# autosubmit to site
from flask import Flask
from robobrowser import RoboBrowser
app = Flask(__name__)
#app.route('/', methods = ['GET', 'POST'])
class My_RoboBrowser(RoboBrowser):
def __init__(self, auth=None, parser=None, headers=None, user_agent=None, history=True):
RoboBrowser.__init__(self, parser=None, user_agent=None, history=True)
def Open(self, vURL, vVerify=True):
response = self.session.get(vURL, verify=vVerify)
self._update_state(response)
browser = My_RoboBrowser(RoboBrowser, "html.parser");
urlL = 'https://accounts.craigslist.org/login'
browser.Open(urlL)
form = browser.get_form(id='login')
assert isinstance(form, 'str')
form['username'] = 'username'
form['password'] = 'password'
browser.submit_form(form)
urlQ = 'https://post.craigslist.org/k/qGDv7K4q5RGD0B5ZEBgXOQ/GLzgd?s=edit'
browser.open(urlQ)
#Question_Tag = browser.find_all(class_="not_answered")[0]
#ID = Question_Tag.get('data-qid')
#Get the form to fill out
Form = browser.get_form(id='formpost')
Form['PostingTitle'].value = 'Create this advertise ment in py'
Form['Postal_code'].value = ['10543']
Form['PostingBody'].value = 'TOGETHER WE INNOVATE Stress free communication with developers that are in the United States. We pride ourselves in bringing your web and app ideas to life and keeping your data secured'
browser.submit_form(Form)
if __name__ == "__main__":
app.run()

isinstance returns true if the first argument is an instance (or subclass) of the second argument otherwise false. In your assertion the form variable is of type robobrowser.forms.form.Form. You can see this with the following code:
print(type(form)) # should print <class 'robobrowser.forms.form.Form'>
Your particular assertion will pass if you update the second argument to indicate this robobrowser Form class:
# Add the following to your imports
import robobrowser
# ... existing code below imports
# The following should be true if form is valid
assert isinstance(form, robobrowser.forms.form.Form)
You can also import the Form class directly but you'll have to update the assertion accordingly:
from robobrowser.forms.form import Form
# ... existing code below imports
assert isinstance(form, Form)
Edit:
Below is the code to properly get the form from that page. There are no forms with an id of 'login' but you can select it by using its action or grabbing the first form on the page.
# ... previous code ...
form = browser.get_form(action='https://accounts.craigslist.org/login')
# you can also use: form = browser.get_form(0)
# It looks like you aren't selecting the fields by their proper names either.
form['inputEmailHandle'] = 'fill in username here'
form['inputPassword'] = 'fill in password here'
browser.submit_form(form)
See if the added code above helps.

Understanding Class inheritance to DRY up some code

I am using the cloudant python library to connect to my cloudant account.
Here is the code I have so far:
import cloudant
class WorkflowsCloudant(cloudant.Account):
def __init__(self):
super(WorkflowsCloudant, self).__init__(settings.COUCH_DB_ACCOUNT_NAME,
auth=(settings.COUCH_PUBLIC_KEY,
settings.COUCH_PRIVATE_KEY))
#blueprint.route('/<workflow_id>')
def get_single_workflow(account_id, workflow_id):
account = WorkflowsCloudant()
db = account.database(settings.COUCH_DB_NAME)
doc = db.document(workflow_id)
resp = doc.get().json()
if resp['account_id'] != account_id:
return error_helpers.forbidden('Invalid Account')
return jsonify(resp)
This Flask controller will have CRUD operations inside of it, but with the current implementation, I will have to set the account and db variables in each method before performing operations on the document I want to view/manipulate. How can I clean up (or DRY up) my code so that I only have to call to my main WorkflowsCloudant class?

I don't know cloudant, so I may be totally off base, but I believe this answers your question:
Delete the account, db, and doc lines from get_single_workflow.
Add the following lines to __init__:
db = account.database(settings.COUCH_DB_NAME)
self.doc = db.document(workflow_id)
Change the resp line in get_single_workflow to:
resp = WorkflowsCloudant().doc.get().json()