Develop Reference

How to check if 2 urls in same domain? [duplicate]

how would you extract the domain name from a URL, excluding any subdomains?
My initial simplistic attempt was:
'.'.join(urlparse.urlparse(url).netloc.split('.')[-2:])
This works for http://www.foo.com, but not http://www.foo.com.au.
Is there a way to do this properly without using special knowledge about valid TLDs (Top Level Domains) or country codes (because they change).
thanks

Here's a great python module someone wrote to solve this problem after seeing this question:
https://github.com/john-kurkowski/tldextract
The module looks up TLDs in the Public Suffix List, mantained by Mozilla volunteers
Quote:
tldextract on the other hand knows what all gTLDs [Generic Top-Level Domains]
and ccTLDs [Country Code Top-Level Domains] look like
by looking up the currently living ones according to the Public Suffix
List. So, given a URL, it knows its subdomain from its domain, and its
domain from its country code.

No, there is no "intrinsic" way of knowing that (e.g.) zap.co.it is a subdomain (because Italy's registrar DOES sell domains such as co.it) while zap.co.uk isn't (because the UK's registrar DOESN'T sell domains such as co.uk, but only like zap.co.uk).
You'll just have to use an auxiliary table (or online source) to tell you which TLD's behave peculiarly like UK's and Australia's -- there's no way of divining that from just staring at the string without such extra semantic knowledge (of course it can change eventually, but if you can find a good online source that source will also change accordingly, one hopes!-).

Using this file of effective tlds which someone else found on Mozilla's website:
from __future__ import with_statement
from urlparse import urlparse
# load tlds, ignore comments and empty lines:
with open("effective_tld_names.dat.txt") as tld_file:
tlds = [line.strip() for line in tld_file if line[0] not in "/\n"]
def get_domain(url, tlds):
url_elements = urlparse(url)[1].split('.')
# url_elements = ["abcde","co","uk"]
for i in range(-len(url_elements), 0):
last_i_elements = url_elements[i:]
# i=-3: ["abcde","co","uk"]
# i=-2: ["co","uk"]
# i=-1: ["uk"] etc
candidate = ".".join(last_i_elements) # abcde.co.uk, co.uk, uk
wildcard_candidate = ".".join(["*"] + last_i_elements[1:]) # *.co.uk, *.uk, *
exception_candidate = "!" + candidate
# match tlds:
if (exception_candidate in tlds):
return ".".join(url_elements[i:])
if (candidate in tlds or wildcard_candidate in tlds):
return ".".join(url_elements[i-1:])
# returns "abcde.co.uk"
raise ValueError("Domain not in global list of TLDs")
print get_domain("http://abcde.co.uk", tlds)
results in:
abcde.co.uk
I'd appreciate it if someone let me know which bits of the above could be rewritten in a more pythonic way. For example, there must be a better way of iterating over the last_i_elements list, but I couldn't think of one. I also don't know if ValueError is the best thing to raise. Comments?

Using python tld
https://pypi.python.org/pypi/tld
Install
pip install tld
Get the TLD name as string from the URL given
from tld import get_tld
print get_tld("http://www.google.co.uk")
co.uk
or without protocol
from tld import get_tld
get_tld("www.google.co.uk", fix_protocol=True)
co.uk
Get the TLD as an object
from tld import get_tld
res = get_tld("http://some.subdomain.google.co.uk", as_object=True)
res
# 'co.uk'
res.subdomain
# 'some.subdomain'
res.domain
# 'google'
res.tld
# 'co.uk'
res.fld
# 'google.co.uk'
res.parsed_url
# SplitResult(
# scheme='http',
# netloc='some.subdomain.google.co.uk',
# path='',
# query='',
# fragment=''
# )
Get the first level domain name as string from the URL given
from tld import get_fld
get_fld("http://www.google.co.uk")
# 'google.co.uk'

There are many, many TLD's. Here's the list:
http://data.iana.org/TLD/tlds-alpha-by-domain.txt
Here's another list
http://en.wikipedia.org/wiki/List_of_Internet_top-level_domains
Here's another list
http://www.iana.org/domains/root/db/

Until get_tld is updated for all the new ones, I pull the tld from the error. Sure it's bad code but it works.
def get_tld():
try:
return get_tld(self.content_url)
except Exception, e:
re_domain = re.compile("Domain ([^ ]+) didn't match any existing TLD name!");
matchObj = re_domain.findall(str(e))
if matchObj:
for m in matchObj:
return m
raise e

Here's how I handle it:
if not url.startswith('http'):
url = 'http://'+url
website = urlparse.urlparse(url)[1]
domain = ('.').join(website.split('.')[-2:])
match = re.search(r'((www\.)?([A-Z0-9.-]+\.[A-Z]{2,4}))', domain, re.I)
if not match:
sys.exit(2)
elif not match.group(0):
sys.exit(2)

In Python I used to use tldextract until it failed with a url like www.mybrand.sa.com parsing it as subdomain='order.mybrand', domain='sa', suffix='com'!!
So finally, I decided to write this method
IMPORTANT NOTE: this only works with urls that have a subdomain in them. This isn't meant to replace more advanced libraries like tldextract
def urlextract(url):
url_split=url.split(".")
if len(url_split) <= 2:
raise Exception("Full url required with subdomain:",url)
return {'subdomain': url_split[0], 'domain': url_split[1], 'suffix': ".".join(url_split[2:])}

Python Django url method regex for comma separated string in single quotes

Attempting to regex match a string like so:
'1,2'
Using Django rest framework url method to parse incoming string, I tried this:
url(r'^path/to/api/end_point/(?P<player_ids>"\'1,2\'")/',
PlayerResultsView.as_view(),
name='get_players)
The error I get:
django.core.urlresolvers.NoReverseMatch: Reverse for 'get_players'
with arguments '()' and keyword arguments '{'player_ids': '1,2'}' not
found. 1 pattern(s) tried:
['path/to/api/end_point/(?P<player_ids>"\'1,2\'")/']
Overall Goal: pass comma separated string ('1,2,3,4') to REST endpoint, split on the comma and continue processing.

Solution 1
What you are trying to do is match a fixed string 1,2. If you want a variable input whether it is 1,2 or 1,2,3,4 or others, then try this regex pattern [\d,]+ which will capture non-empty string containing any numbers and commas.
re_path(r'^path/to/api/end_point/(?P<player_ids>[\d,]+)/', views.some_view),
Sample URL would be:
http://127.0.0.1:8000/path/to/api/end_point/1,2,3,4/
Solution 2
Instead of such variable input in the path, you might be interested in making it query parameters instead. So something like:
path('path/to/api/end_point/', views.some_view),
Sample URL would be:
http://127.0.0.1:8000/path/to/api/end_point/?player_ids=1,2,3,4
Then, just access it as part of the request.GET or request.query_params (if using djangorestframework) in your view.
def some_view(request, **kwargs):
print(f"{request.GET.get('player_ids')=}")
print(f"{request.query_params.get('player_ids')=}")
Output
request.GET.get('player_ids')='1,2,3,4'
request.query_params.get('player_ids')='1,2,3,4'
[08/Sep/2021 01:44:05] "GET path/to/api/end_point/?player_ids=1,2,3,4 HTTP/1.1" 200 17

I decided to go with a POST, easier to pass more complex parameters.
My URL pattern:
url(r'^path/to/api/end_point/',
PlayerResultsView.as_view(),
name='get_players)
My POST data is an array of player ids:
list_of_search_ids = [player1.player_id, player2.player_id]
data = {'player_ids': list_of_search_ids}
url = reverse(
"get_players"
)
resp = self.client.post(url, json.dumps(data), content_type='application/json')

redirect while passing arguments

In flask, I can do this:
render_template("foo.html", messages={'main':'hello'})
And if foo.html contains {{ messages['main'] }}, the page will show hello. But what if there's a route that leads to foo:
#app.route("/foo")
def do_foo():
# do some logic here
return render_template("foo.html")
In this case, the only way to get to foo.html, if I want that logic to happen anyway, is through a redirect:
#app.route("/baz")
def do_baz():
if some_condition:
return render_template("baz.html")
else:
return redirect("/foo", messages={"main":"Condition failed on page baz"})
# above produces TypeError: redirect() got an unexpected keyword argument 'messages'
So, how can I get that messages variable to be passed to the foo route, so that I don't have to just rewrite the same logic code that that route computes before loading it up?

You could pass the messages as explicit URL parameter (appropriately encoded), or store the messages into session (cookie) variable before redirecting and then get the variable before rendering the template. For example:
from flask import session, url_for
def do_baz():
messages = json.dumps({"main":"Condition failed on page baz"})
session['messages'] = messages
return redirect(url_for('.do_foo', messages=messages))
#app.route('/foo')
def do_foo():
messages = request.args['messages'] # counterpart for url_for()
messages = session['messages'] # counterpart for session
return render_template("foo.html", messages=json.loads(messages))
(encoding the session variable might not be necessary, flask may be handling it for you, but can't recall the details)
Or you could probably just use Flask Message Flashing if you just need to show simple messages.

I found that none of the answers here applied to my specific use case, so I thought I would share my solution.
I was looking to redirect an unauthentciated user to public version of an app page with any possible URL params. Example:
/app/4903294/my-great-car?email=coolguy%40gmail.com to
/public/4903294/my-great-car?email=coolguy%40gmail.com
Here's the solution that worked for me.
return redirect(url_for('app.vehicle', vid=vid, year_make_model=year_make_model, **request.args))
Hope this helps someone!

I'm a little confused. "foo.html" is just the name of your template. There's no inherent relationship between the route name "foo" and the template name "foo.html".
To achieve the goal of not rewriting logic code for two different routes, I would just define a function and call that for both routes. I wouldn't use redirect because that actually redirects the client/browser which requires them to load two pages instead of one just to save you some coding time - which seems mean :-P
So maybe:
def super_cool_logic():
# execute common code here
#app.route("/foo")
def do_foo():
# do some logic here
super_cool_logic()
return render_template("foo.html")
#app.route("/baz")
def do_baz():
if some_condition:
return render_template("baz.html")
else:
super_cool_logic()
return render_template("foo.html", messages={"main":"Condition failed on page baz"})
I feel like I'm missing something though and there's a better way to achieve what you're trying to do (I'm not really sure what you're trying to do)

You can however maintain your code and simply pass the variables in it separated by a comma: if you're passing arguments, you should rather use render_template:
#app.route("/baz")
def do_baz():
if some_condition:
return render_template("baz.html")
else:
return render_template("/foo", messages={"main":"Condition failed on page baz"})

Match referer by regex

I'd like to setup a simple notification if a view has a specific base referer.
Let's say I land on http://myapp.com/page/ and I came from http://myapp.com/other/page/1. Here's an example of my pseudo code, basically if I'm coming from any page/X I want to setup a notification.
I'm thinking it might be something like ^r^myapp.com/other/page/$ but I'm not so familiar with how to use regex with python.
from django.http import HttpRequest
def someview(request):
notify = False
... # other stuff not important to question
req = HttpRequest()
test = req.META['HTTP_REFERER'] like "http://myapp.com/other/page*"
# where * denotes matching anything past that point and the test returns T/F
if test:
notify = True
return # doesn't matter here
This may be more of a "how do I use regex in this context" rather than a django question specifically.

You could go with something like this:
import re
referrer = "http://myapp.com/other/page/aaa"
m = re.match("^http://myapp.com/other/page/(.*)", referrer)
if m:
print m.group(1)

Performing text processing on flatpage content to include handling of custom tag

I'm using flatpages app in my project to manage some html content. That content will include images, so I've made a ContentImage model allowing user to upload images using admin panel.
The user should then be able to include those images in content of the flatpages. He can of course do that by manually typing image url into <img> tag, but that's not what I'm looking for.
To make including images more convenient, I'm thinking about something like this:
User edits an additional, let's say pre_content field of CustomFlatPage model (I'm using custom flatpage model already)
instead of defining <img> tags directly, he uses a custom tag, something like [img=...] where ... is name of the ContentImage instance
now the hardest part: before CustomFlatPage is saved, pre_content field is checked for all [img=...] occurences and they are processed like this:
ContentImage model is searched if there's image instance with given name and if so, [img=...] is replaced with proper <img> tag.
flatpage actual content is filled with processed pre_content and then flatpage is saved (pre_content is leaved unchanged, as edited by user)
The part that I can't cope with is text processing. Should I use regular expressions? Apparently they can be slow for large strings.
And how to organize logic? I assume it's rather algorithmic question, but I'm not familliar with text processing in Python enough, to do it myself.
Can somebody give me any clues?

I finally imlemented this using regular expressions. I decided, that spaces are not allowed inside the custom tag. Main text processing function looks like this:
import re
from django.utils.translation import ugettext as _
def process_image_tags(text, ImageModel):
'''image tag usage:
... some text ... [img=image_name:image_class(optional)] ... some text ...
'''
t1 = re.split(r'(\[img=[a-z0-9\-_\:]+\])', text)
t2 = []
for i in t1:
if i[:5] == '[img=':
attrs = i[5:-1].split(':')
name_attr = attrs[0] #name attribute
error = None
try:
image = ImageModel.objects.get(name=name_attr)
except ImageModel.DoesNotExist:
error = '<span class="image_tag_error">%s</span>' % _('Image with given name not found')
except ImageModel.MultipleObjectsReturned:
error = '<span class="image_tag_error">%s</span>' % _('More than one image found')
if not error:
p = ['<img']
p.append('src="%s"' % image.image.url)
if len(attrs) > 1:
p.append('class="%s"' % attrs[1]) #class attribute
if image.description:
p.append('title="%s"' % image.description)
p.append('alt="%s"' % image.name)
p.append('/>')
t2.append(' '.join(p))
else:
t2.append(error)
else:
t2.append(i)
return ''.join(t2)
Above function is then used in save method of CustomFlatPage model, like this:
def save(self, *args, **kwargs):
self.content = process_image_tags(self.pre_content, ContentImage)
super(CustomFlatPage, self).save(*args, **kwargs)
It seems to work, so I'll probably end up using that solution. Maybe I'll add some javascript for user to insert image tags by picking images from generated list of images, but even now it's better than manually typing urls, I think.