Develop Reference

How to fix this error during scraping using BeautifulSoup?

I am trying to do web scraping using BeautifulSoup and requests Python library. I want to filter the news titles from Hacker News website but its showing an error while implementing.
import requests
from bs4 import BeautifulSoup
res = requests.get('https://news.ycombinator.com/news')
soup = BeautifulSoup(res.text, 'html.parser')
links = soup.select('.titleline a')
subtext = soup.select('.subtext')
def create_custom_hn(links, subtext):
hn = []
for index, item in enumerate(links):
title = links[index].getText()
href = links[index].get('href', None)
votes = subtext[index].select('.score')
if len(votes):
points = int(votes[0].getText().replace(' points', ''))
print(points)
hn.append({'title': title, 'href': href})
return hn
print(create_custom_hn(links, subtext))
The error says
votes = subtext[index].select('.score')
~~~~~~~^^^^^^^
IndexError: list index out of range

Here is fixed version of the code from the question:
import requests
from bs4 import BeautifulSoup
res = requests.get("https://news.ycombinator.com/news")
soup = BeautifulSoup(res.text, "html.parser")
links = soup.select(".titleline > a")
def create_custom_hn(links):
hn = []
for link in links:
title = link.getText()
href = link.get("href", None)
votes = link.find_next(class_="score")
points = int(votes.getText().replace(" points", ""))
hn.append({"title": title, "href": href, "points": points})
return hn
print(create_custom_hn(links))
Prints:
[
{
"title": "Urllib3 in 2022",
"href": "https://sethmlarson.dev/urllib3-in-2022",
"points": 97,
},
{
"title": "First public release of Pushup: a new compiler for making web apps in Go",
"href": "https://github.com/adhocteam/pushup",
"points": 18,
},
{
"title": "Intelligence – A good collection of great OSINT Resources",
"href": "https://github.com/ARPSyndicate/awesome-intelligence",
"points": 113,
},
{
"title": "Microsoft is preparing to add ChatGPT to Bing",
"href": "https://www.bloomberg.com/news/articles/2023-01-04/microsoft-hopes-openai-s-chatbot-will-make-bing-smarter",
"points": 760,
},
...and so on.

Try to select your elements more specific, your selection of soup.select('.titleline a') includes more elements (60) as you may like to select (30):
[Urllib3 in 2022,
<span class="sitestr">sethmlarson.dev</span>,...]
I would also recommend to iterate the elements in another way, so you would become able to handle missing values.
Example
import requests
from bs4 import BeautifulSoup
res = requests.get('https://news.ycombinator.com/news')
soup = BeautifulSoup(res.text)
data = []
for e in soup.select('tr.athing'):
data.append({
'title':e.select_one('.titleline a').get_text(),
'url':e.select_one('.titleline a').get('href'),
'votes':e.find_next(class_='subtext').text.split()[0]
})
print(data)
Output
[{'title': 'Urllib3 in 2022', 'url': 'https://sethmlarson.dev/urllib3-in-2022', 'votes': '93'}, {'title': 'First public release of Pushup: a new compiler for making web apps in Go', 'url': 'https://github.com/adhocteam/pushup', 'votes': '16'}, {'title': 'Intelligence – A good collection of great OSINT Resources', 'url': 'https://github.com/ARPSyndicate/awesome-intelligence', 'votes': '109'}, {'title': 'Microsoft is preparing to add ChatGPT to Bing', 'url': 'https://www.bloomberg.com/news/articles/2023-01-04/microsoft-hopes-openai-s-chatbot-will-make-bing-smarter', 'votes': '755'}, {'title': 'Juan Tamariz, the godfather of close-up card magic', 'url': 'https://www.nytimes.com/2023/01/02/magazine/juan-tamariz-magic.html', 'votes': '31'}, {'title': 'The Expanding Dark Forest and Generative AI', 'url': 'https://maggieappleton.com/ai-dark-forest', 'votes': '223'}, {'title': 'Irreconcilable differences between local and distributed computing (1994)', 'url': 'https://scholar.harvard.edu/waldo/publications/note-distributed-computing', 'votes': '29'},...]

Trying to create family tree from dictinoary

I am trying to create "child,parent" dictionary from my dictionary. How can i achieve that?
Here is my dict:
{"Sıdıka":[{"Aziz":[{"Ahmet":[{"Kuzey":[]}],"Öznur":[{"Elif":[]},{"Yiğit":[]}],"İlknur":[{"Nurullah":[]},{"Büşra":[]}],"İlker":[{"Melih":[]}]}]}]}
Left to right ancestors are growing. Sıdıka is grand grand grand grand mother, aziz is her son. And "ahmet, öznur,ilknur,ilker" are "aziz"s children etc. etc.
I want a dictionary something like this:
{'Sıdıka':[{'Aziz':[{'Ahmet':[{'Kuzey':[]}],'Öznur':[{'Elif':[]},{'Yiğit':[]}],'İlknur':[{'Nurullah':[]},{'Büşra':[]}],{'İlker':[{'Melih':[]}]}]
I think i need a very good algorithm for this. Any help?

You can use recursion for the task:
d = {
"Sıdıka": ["Aziz"],
"Aziz": ["Ahmet", "Öznur", "İlknur", "İlker"],
"Ahmet": ["Kuzey"],
"Öznur": ["Elif", "Yiğit"],
"İlknur": ["Nurullah", "Büşra"],
"İlker": ["Melih"],
}
def get_tree(root):
return {root: [get_tree(v) for v in d.get(root, [])]}
print(get_tree("Sıdıka"))
Prints:
{
"Sıdıka": [
{
"Aziz": [
{"Ahmet": [{"Kuzey": []}]},
{"Öznur": [{"Elif": []}, {"Yiğit": []}]},
{"İlknur": [{"Nurullah": []}, {"Büşra": []}]},
{"İlker": [{"Melih": []}]},
]
}
]
}

zyDE 9.15.1: Nested dictionaries example: Music library

So, I wanted to answer this question for those who could not answer this question.
I wanted to help, so here is my answer since it was a little hard for me.
If you can think of anything better, let everyone know.
zyDE 9.15.1: Nested dictionaries example: Music library.
The following example demonstrates a program that uses 3 levels of nested dictionaries to create a simple music library.
The following program uses nested dictionaries to store a small music library. Extend the program such that a user can add artists, albums, and songs to the library. First, add a command that adds an artist name to the music dictionary. Then add commands for adding albums and songs. Take care to check that an artist exists in the dictionary before adding an album, and that an album exists before adding a song.
Answer:
music = {
'Pink Floyd': {
'The Dark Side of the Moon': {
'songs': [ 'Speak to Me', 'Breathe', 'On the Run', 'Money'],
'year': 1973,
'platinum': True
},
'The Wall': {
'songs': [ 'Another Brick in the Wall', 'Mother', 'Hey you'],
'year': 1979,
'platinum': True
}
},
'Justin Bieber': {
'My World':{
'songs': ['One Time', 'Bigger', 'Love Me'],
'year': 2010,
'platinum': True
}
}
}
prompt = ("1. Enter artist information\n"
"2. Exit\n")
command = ''
while command != '2':
command = input(prompt).lower()
if command == '1':
artist = input('Artist: ')
if artist in music.keys():
print('That artist already exists. Please try again.')
artist = input('Artist: ')
album = input('Album: ')
for albums in music.values():
if album in albums:
print('That album already exists. Please try again')
album = input('Album: ')
songs = input('Song: ').split()
music[artist] = {album: {'songs': songs}}
else:
break
print(music)

Maybe you can try this way:
prompt = ("1. Enter artist information\n"
"2. Exit\n")
command = ''
while command != '2':
command = input(prompt).lower()
if command == '1':
artist = input('Artist: ')
album = input('Album: ')
songs = input('Song: ').split()
if artist in music.keys():
print(music[artist].keys())
if album in music[artist].keys():
music[artist][album]["songs"] += songs
else:
music[artist].update({album: {}})
music[artist][album].update({"songs": songs})
else:
music.update({artist: {}})
music[artist].update({album: {}})
music[artist][album].update({"songs": songs})
print('\n', music)

Saving list data in python

I am wondering how I can save whatever I added to a list when I close a python file. For example, in this "my contact" program that I wrote below, if I add information about 'Jane Doe', what could I do so that next time I open up the same file, Jane Doe still exists.
def main():
myBook = Book([{"name": 'John Doe', "phone": '123-456-7890', "address": '1000 Constitution Ave'}])
class Book:
def __init__(self, peoples):
self.peoples = peoples
self.main_menu()
def main_menu(self):
print('Main Menu')
print('1. Display Contact Names')
print('2. Search For Contacts')
print('3. Edit Contact')
print('4. New Contact')
print('5. Remove Contact')
print('6. Exit')
self.selection = input('Enter a # form the menu: ')
if (self.selection == "1"):
self.display_names()
if (self.selection == "2"):
self.search()
if (self.selection == "3"):
self.edit()
if (self.selection == "4"):
self.new()
if (self.selection == "5"):
self.delete()
if (self.selection == "6"):
self.end()
def display_names(self):
for people in self.peoples:
print("Name: " + people["name"])
self.main_menu()
def search(self):
searchname = input('What is the name of your contact: ')
for index in range(len(self.peoples)):
if (self.peoples[index]["name"] == searchname):
print("Name: " + self.peoples[index]["name"])
print("Address: " + self.peoples[index]["address"])
print("Phone: " + self.peoples[index]["phone"])
self.main_menu()
def edit(self):
searchname = input('What is the name of the contact that you want to edit: ')
for index in range(len(self.peoples)):
if (self.peoples[index]["name"] == searchname):
self.peoples.pop(index)
name = input('What is your name: ')
address = input('What is your address: ')
phone = input('What is your phone number: ')
self.peoples.append({"name": name, "phone": phone, "address": address})
self.main_menu()
def new(self):
name = input('What is your name: ')
address = input('What is your address: ')
phone = input('What is your phone number: ')
self.peoples.append({"name": name, "phone": phone, "address": address})
self.main_menu()
def delete(self):
searchname = input('What is the name of the contact that you want to delete: ')
for index in reversed(range(len(self.peoples))):
if (self.peoples[index]["name"] == searchname):
self.peoples.pop(index)
print(searchname, 'has been removed')
self.main_menu()
def end(self):
print('Thank you for using the contact book, have a nice day')
print('Copyright Carson147 2019©, All Rights Reserved')
main()

Use a module from the Data Persistence section of the standard library, or save it as json, or as a csv file.

You just convert your list to array inside in function .
np.save('path/to/save', np.array(your_list))
to load :
arr=np.load(''path/to/save.npy').tolist()
I hope it will be helpful

There are innumerable kinds of serialization options, but a time-tested favorite is JSON. JavaScript Object Notation looks like:
[
"this",
"is",
"a",
"list",
"of",
"strings",
"with",
"a",
{
"dictionary": "of",
"values": 4,
"an": "example"
},
"can strings be single-quoted?",
false,
"can objects nest?",
{
"I": {
"Think": {
"They": "can"
}
}
}
]
JSON is widely used, and the Python stdlib has a method of converting objects to and from JSON in the json package.
>>> import json
>>> data = ['a', 'list', 'full', 'of', 'entries']
>>> json.dumps(data) # dumps will dump to string
["a", "list", "full", "of", "entries"]
You can then save your Book data to json before the program shuts down, and read from json after it starts up.
# at the top
import json
from pathlib import Path
# at the bottom of your program:
if __name__ == '__main__':
persistence = Path('book.json')
if persistence.exists():
with persistence.open() as f:
data = json.load(f)
else:
data = [{"name": 'John Doe', "phone": '123-456-7890', "address": '1000 Constitution Ave'}]
book = Book(data)
with persistence.open('w') as f:
json.dump(f, indent=4)

There is no way you can do that without any external modules, such as numpy or pickle. Using pickle, you can do this: (I am assuming you want to save the myBook variable)
import pickle
pickle.dump(myBook, open("foo.bar", "wb")) #where foo is name of file and bar is extension
#also wb is saving type, you can find documentation online
To load:
pickle.load(myBook, open("foo.bar", "rb"))
EDIT:
I was wrong in my first statement. There is a way to save without importing a module. Here is how:
myBook.save(foo.bar) #foo is file name and bar is extention
To load:
myBook=open(foo.bar)

As evinced by the many other answers, there are many ways to do this, but I thought it was helpful to have a example.
By changing the top of your file as so, you can use the shelve module.
There are a variety of other things you can fix in your code if you are curious, you could try https://codereview.stackexchange.com/ if you want more feedback.
import shelve
def main():
default = [
{'name': 'John Doe', 'phone': '123-456-7890',
'address': '1000 Constitution Ave'}
]
with Book('foo', default=default) as myBook:
myBook.main_menu()
class Book:
def __init__(self, filename, default=None):
if default is None:
default = []
self._db = shelve.open(filename)
self.people = self._db.setdefault('people', default)
def __enter__(self):
return self
def __exit__(self):
self._db['people'] = self.people
self._db.close()

Display all Dict in New windows GUI using Label tkinter Python

I am New to python and going though few online training.I couldn't get any close related to below question.
I am using tkinter GUI
from Tkinter import *
root = Tk()
trainings = {"title":"Python Training Course for Beginners",
"location":"Frankfurt",
"ID": 111,"title":"Intermediate Python Training",
"location":"Berlin",
"ID": 133,"title":"Python Text Processing Course",
"location":"Mdsgtd",
"ID": 122}
for key in trainings.keys():
x = trainings.get(key)
print x
Label(root, text = x ).pack()
mainloop()
Getting output only:122
But I am expecting result should be display in GUI Label:
{'ID': 111, 'location': 'Frankfurt', 'title': 'Python Training Course for Beginners'}
{'ID': 122, 'location': 'Mdsgtd', 'title': 'Python Text Processing Course'}
{'ID': 133, 'location': 'Berlin', 'title': 'Intermediate Python Training'}
Can I used inside the function label as in below code: which is not working:
def OnButtonClick(self):
self.top= Toplevel()
self.top.title("Read Data Service Menu Item")
self.topdata = {'parakeet': ['fly', 'bird'], 'dog': 'animal', 'cat': 'feline'}
for key in self.topdata.keys():
x = self.topdata.get(key)
self.topL2 = Label(self.top, text = key).pack()
self.top.resizable(1,0)
self.top.transient(self)
self.B1.config(state = 'normal') #disable/normal
self.topButton = Button(self.top, text = 'Close', command = self.OnChildClose)
self.topButton.pack()

You have a few issues at present, as noted in the comments. Firstly, you should change your trainings dictionary to be a list of dictionaries to let you store the relevant information for each course in turn.
Assuming you want to show a different label for the information relating to each course, the following should work:
from Tkinter import *
courses = [{"title": "Python Training Course for Beginners",
"location": "Frankfurt",
"ID": 111},
{"title": "Intermediate Python Training",
"location": "Berlin",
"ID": 133},
{"title": "Python Text Processing Course",
"location": "Mdsgtd",
"ID": 122}]
root = Tk()
for course in courses:
temp_text = '{0} ({1}) - {2}'.format(course['title'], course['ID'], course['location'])
Label(root, text=temp_text).pack()
mainloop()
We use string formatting to create a nicely-written output, of the course name followed by its ID in brackets, then the location of the course after a dash.
What is critical here is that we want to create a Label widget for every course - hence, we add the new Label within our for loop to ensure this happens.