Python - Using data from scraped web data [closed] - python

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
Improve this question
i'm new here, just started to learn Python 2 days ago, i just made my first program, half done, i need help with the rest please i've tried evrything i could but i'm struggling.
the program is about web scraping a currency (name and value) then use the data to select a specific currency, display it + value then enter a specific Alert_value, if the Alert_value is reached then i get notification.
i could scrape the data using BS4 but i couldn't use it as individual data Currency + it's value, no idea if i have to use it as list or other but i have tried list but my 2 days of experience didn't help :D
the whole Currency data is index[0] i think the problem is that i couldn't make a list out of the Currency name data, index[1] is out of range means that 150 currency name is at index[0].
Annyone can help me please how to put the data of both Currency names and value, and how to select each one individually to display it, please use simple words when explaining i'm just starting. Thank you.
Some of the Code : name is currency name and price is the currency price
for container in containers:
name_container = container.findAll("a", {"class":"tv-screener__symbol"})
name = name_container[0].text
title_container = container.findAll("td", {"class":"tv-data-table__cell tv-screener-table__cell tv-screener-table__cell--big"})
price = title_container[2].text
# print(str(name) + " : " + price)
Xt=[name,price]
print(Xt)

Hi & welcome #HannibalB!
I am a new contributor myself, so unfortunately I cannot comment yet. As far as I understand correctly, you would like to query a currency name - e.g. Nexo - and have the respective value displayed.
Python offers many different data types, all of which are useful for different purposes. In your case, I would suggest a dictionary. Check out the link, if you would like to know more.
What you could do, is adjust the code slightly (I am assuming here that it already works as intended):
Dictionary = {}
for container in containers:
name_container = container.findAll("a", {"class":"tv-screener__symbol"})
name = name_container[0].text
title_container = container.findAll("td", {"class":"tv-data-table__cell tv-screener-table__cell tv-screener-table__cell--big"})
price = title_container[2].text
# print(str(name) + " : " + price)
Dictionary[name] = price
You can now simply access the elements in the dictionary like so:
Dictionary['Nexo']
This will then display the price/value that is stored under the so called dictionary key, which in this case is Nexo. In case you prefer pandas.DataFrame, you can easily convert a dictionary into one. Check out the link.
You might have also thought about appending a DataFrame directly. Even though I am not an expert, I would advise against that. A DataFrame is quite a heavy data structure and it is usually more efficient to convert you data into a DataFrame as a final step.

Related

How can I work around a KeyError in Python? [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 3 years ago.
Improve this question
I am trying to create a Forecasting tool, which analyses historical passenger traffic at a given airport. The anylysis will be based on a linear regression with various GDPs (Gross Domestic Product) of countries related to the airport.
A person can type in the name of the independant variable, which then gets selected from the Excel file.
Once a person gets the question "Which Country's GDP would you like to set as the independant variable for the Regression Analysis?", there is the possibility of typing a country wrong. In that case I receive a KeyError.
I am trying to work around that with "try / except", but I still receive a KeyError (See lines 36-49). I would really appreciate some help!
Thank you!
If it helps, here is the GitHub Link: https://github.com/DR7777/snowflake
(See lines 36-49 of main_file.py)
Here is my code:
Ive tried with while loops, for / except, but it seems I am too new to understand.
# This part saves the first row of the Excel as a list,
# so I can give the user a list of all the countries,
# if the person types in a country, that's not on the list.
loc = ("IMF_Country_GDP_Data.xlsx")
wb = xlrd.open_workbook(loc)
sheet = wb.sheet_by_index(0)
sheet.cell_value(0, 0)
list_of_countries = sheet.row_values(0)
possible_selection = (list_of_countries[1:]) #This is the list with all the possible countries, without the Excel cell A1
#Introduction
print("Hello, welcome to the Air Traffic Forecasting Tool V0.1!")
print("Which Country's GDP would you like to set as the independant variable for the Regression Analysis?")
Country_GDP = input("Please type your answer here: ")
#here we check, if the typed Country is in the list
try:
possible_selection == Country_GDP
print("Your country is on the list.")
except KeyError:
print("Oh no! You typed a country, which is not in our database!")
print("Please select one of the countries listed below and try again")
print(possible_selection)
#now continuing with the previous code
print("Ok, I will conduct the analysis based on the GDP of " + str(Country_GDP) + "!")
print("Here are your results: ")
#here is the rest of the code
What I want to achieve is:
If a person types a name, which is on the list of countries, the program runs the regression.
If the country is not on the list, I dont want to receive a KeyError. I would like the program to say:
Oh no! You typed a country, which is not in our database!
Please select one of the countries listed below and try again
And then print the possible_selection variable, so the user can see which selection he has.
Thank you very much!
No need to get a key error at all. Just use in.
while True:
selection = input('Which country?')
if selection in list_of_countries:
print('Your country is on the list')
break
else:
print('You typed an invalid entry, lets try again')

What python data type lets me manipulate CSV data the best? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I'm trying to keep track of "banned" users in my application.
username,minutes_banned,channel_banned_from
joe,5,general
sally,10,other
bob,50,gaming
I read the file to a list and it made of list of lists.
I found it hard to work with the data in this form. For example I had trouble getting it to output only usernames.
And I'm also not sure how can I keep track of each set of data. If I take bob as a username and want to check how long hes banned.. How can I relate bob to 50?
Thank you.
If usernames are unique and you don't care about a particular order, keep them in a dict:
banned = {'joe': (5, 'general'), 'sally': (10, 'other')}
print "Bob's ban duration is", banned.get('bob', 'forever')
And you can obtain all the usernames, in indeterminate order, like this: all_usernames = banned.keys() which gives ['sally', 'joe'].
That looks like a csv formatted file and the csv module seems like a good place to start. DictReader reads rows into dictionaries, which lets you reference cells by name. You can read a table from there (a list of dicts). Then, create an index of the fields you want to use the most. The index is another dict that references the dict s in the table.
import csv
with open('banned.txt', newline='') as fp:
reader = csv.DictReader(fp)
reader.fieldnames = ['username','minutes_banned','channel_banned_from']
table = list(reader)
user_index = {entry['username']:entry for entry in table}
user = input('check banned user name: ')
if user in user_index:
print('banned', user_index[user]['minutes_banned'])
else:
print('invalid user')

Python: Writing lists to .csv file [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I’m teaching myself programming, using Python as my initial weapon of choice.
I have learnt a few basics and decided to set myself the challenge of asking the user for a list of names, adding the names to a list and then finally writing the names to a .csv file.
Below is my code. It works.
My question is what would you do differently, i.e. how could this code be improved for readability and efficiency. Would you approach the situation differently, structure it differently, call different functions? I am interested in, and would appreciate a great deal, the feedback from more experienced programmers.
In particular, I find certain parts clunky; such as having to specify to the user the required format for data entry. If I were to simply request the data (name age location) without the commas however, then each record, when written to .csv, would simply end up as one record per cell (Excel) – this is not the desired result.
#Requesting user input.
guestNames = input("Please enter the names of your guests, one at a time.\n"\
"Once you have finished entering the information, please type the word \"Done\".\n"\
"Please enter your names in the following format (Name, Age, Location). ").capitalize()
guestList.append(guestNames)
while guestNames.lower() != "done".lower() :
guestNames = input("Please enter the name of your " + guestNumber[number] + " guest: ").capitalize()
guestList.append(guestNames)
number += 1
#Sorting the list.
guestList.sort()
guestList.remove("Done")
#Creating .csv file.
guestFile = open("guestList.csv","w")
guestFile.close()
#Writing to file.
for entries in guestList :
guestFile = open("guestList.csv","a")
guestFile.write(entries)
guestFile.write("\n")
guestFile.close()
I try to write down your demands:
Parse the input string according to its structure (whatever) and save results into a list
Format the result into CSV-format string
write the string to a CSV file
First of all, I would highly recommend you to read the a Python string operation and formatting tutorial like Google Developer Tutorial. When you understand the basic operation, have a look at official documentation to see available string processing methods in Python.
Your logic to write the code is right, but there are two meaningless lines:
while guestNames.lower() != "done".lower()
It's not necessary to lower "done" since it is already lower-case.
for entries in guestList :
guestFile = open("guestList.csv","a")
Here you open and close the questList.csv every loop, which is useless and costly. You could open the file at the beginning, then save all lines with a for loop, and close it at the end.
This is a sample using the same logic and different input format:
print('some notification at the beginning')
while true:
guestNames = input("Please enter the name of your " + guestNumber[number] + " guest: ").capitalize()
if guestNames == 'Done':
# Jump out of the loop if user says done
break
else:
# Assume user input 'name age location', replace all space with commas
guestList.append(guestNames.replace(' ', ','))
number += 1
guestList.sort()
# the with keyword will close the guestFile at the end
with open("guestList.csv","w") as guestFile:
guestFile.write('your headers\n')
for entries in guestList:
guestFile.write('%s\n' % entries)
Be aware that there are many ways to fulfil your demands, with different logics and methodologies.

Method to find gender of a username - with or without dictionary [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I'm trying to find a way to do both actions:
1) Retrieving a gender from a name
2) Retrieving a gender from a username (similar to the previous one but different)
I reviewed some methods how to do 1) (including: Does anyone know of a good library for mapping a person's name to his or her gender?), and might do it, but I have a problem to do the second:
Let's say that I have a dictionary containing all names split to female or male. Now I want to take a username and try to see whether the username contains one of these names (because username can contain more than just the first name itself) - what's the best way to do it? It seems not efficient to go over all the dict keys and look for them in the username one by one... There must be an optimized method...
Hope that I explained myself clearly, any help would be very much appreciated!!
A dictionary will only help in the case that the user name (or name) is an exact match. It may be best to use a simple list:
Make certain that the names you include in girl_names and boy_names are fully gender-specific. (For instance, Ben wouldn't work, since both men and women could go by Ben.) Also make sure that the lists don't have any repeated partial names, E.g. "Al" and "Albert".
girl_names = Large set of likely female-only names, all lower-case.
boy_names = Large set of likely male-only names, all lower-case.
user_name = The user name you're looking up.
def name_gender(user_name, boy_names=None, girl_names=None):
uname = user_name.lower()
girliness = 0
for girl_name in girl_names:
if girl_name in uname:
girliness += 1
boyishness = 0
for boy_name in boy_names:
if boy_name in uname:
boyishness += 1
if boyishness > girlishness: return 'Male'
if girlishness > boyishness: return 'Female'
return 'Androgynous'
This function is only a loose proxy for real intended user name gender. Lists of common names are available from various web sites, and can also be obtained from the Census information.

Get value from 2nd list based on ID from 1st list in Python [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I'm a beginner in Python and have a file i've read in that has an Market_ID field. While looping over this file (list), I need to access the Market_name in a 2nd file (Market_ID,Market_Name).
What's the best place for me to start? I can give more detail on what the program is doing if necessary.
The files(lists) are laid out as follows:
File1: Cust_ID, Market_ID, Cust_Name, IsNew
File2: Market_ID, Market_Name
My main loop is over File1. I need to be able to access the Market_Name from File2 to be able to place it in a new file i'm creating from data in File1. Hope that helps.
If the files are not too big (a few megabytes) you could store each item in a dict() using the Market_ID as key.
This looks like data drom a database - if you are familiar with
relational databases and their use you could insert each file into a
separate table and then perform queries. Python has an interface to
sqlite in its standard library.
For solving this quickly I suggest to use dicts though.
BY -- sleeplessnerd
More about dictionaries,
More about sqlite
file2 = open("/path/to/file2", "r")
market_names = {}
for line in file2:
fields = line.split(",")
market_names[int(fields[0])] = fields[1]
file2.close()
file1 = open("/path/to/file1", "r")
for line in file1:
fields = line.split(",")
market_name = market_names[int(fields[1])]
# do what ever you want with the Market_Name associated with
# the Market_ID contained in this line
file1.close()
If the files are not too big (a few megabytes) you could store each item in a dict() using the Market_ID as key.
This looks like data drom a database - if you are familiar with relational databases and their use you could insert each file into a separate table and then perform queries. Python has an interface to sqlite in its standard library.
For solving this quickly I suggest to use dicts though.
Hmm, could you please supply more details?
When you say file(list) do you mean that you've created a list that contains all the data you need from the first file? what about the second file?
If they are both lists than something like this would work:
for market_id in flie1_list:
for pair in file2_list:
if pair[0] == market_id:
market_name = pair[1]
break
<keep doing whatever it is you need to do in first loop>
Note - I'm fairly a beginner as well, and it's my first answer on this site. Go easy ;)
I'm not up on my python, but conceptually, what you'll probably want to do is read the second file into a dictionary first and then look up the id in the dictionary to get the associated name. Alternatively, if the ids are numeric and more-or-less dense, you could just read them into a regular array.

Categories

Resources