Develop Reference

print a dictionary adding a blank space between each blocks

New to python. Here's a nested dictionary with two books each having 8 attributes.
book_collection ={17104: {'Title': 'A River', 'Author': 'Elisha Mitchell', 'Publisher': 'FPG Publishing', 'Pages': '345', 'Year': '2014', 'Copies': 2, 'Available': 2, 'ID': 17104}, 37115: {'Title': 'Aim High', 'Author': 'George Tayloe Winston', 'Publisher': 'Manning Hall Press', 'Pages': '663', 'Year': '2014', 'Copies': 5, 'Available': 5, 'ID': 37115}}
for id, book in book_collection.items():
for book_attribute, attribute_value in book.items():
print(book_attribute, ': ', attribute_value, sep='')
The output:
Title: A River
Author: Elisha Mitchell
Publisher: FPG Publishing
Pages: 345
Year: 2014
Copies: 2
Available: 2
ID: 17104
Title: Aim High
Author: George Tayloe Winston
Publisher: Manning Hall Press
Pages: 663
Year: 2014
Copies: 5
Available: 5
ID: 37115
How can I add a blank space between each book, and bring the 'ID' attribute to the first row of each book. The output is supposed to look like this:
ID: 17104
Title: A River
Author: Elisha Mitchell
Publisher: FPG Publishing
Pages: 345
Year: 2014
Copies: 2
Available: 2
ID: 37115
Title: Aim High
Author: George Tayloe Winston
Publisher: Manning Hall Press
Pages: 663
Year: 2014
Copies: 5
Available: 5
If there are 20 books, how can I just print the first 10 and ask the user for permission to continue?

Use this:
for id, book in book_collection.items():
for book_attribute, attribute_value in book.items():
print(book_attribute, ': ', attribute_value, sep='')
print()
You can just use the index() function to check if the index is 9 then ask like this
for id, book in book_collection.items():
if book_collection.index(id) == 9:
n = int(input("Press 0 to continue or else to exit"))
if n != 0:
break
for book_attribute, attribute_value in book.items():
print(book_attribute, ': ', attribute_value, sep='')
print()

A dictionary's items method method returns an iterable of tuples (immutable lists). Each tuple yielded represents a pair of key and value, with the key being in the tuple's 0 index, and the value in 1 index.
The for loop you're using - for book_attribute, attribute_value in book.items(): - is syntactic sugar for "take the two values in the tuple and assign them to these variables, then run the code in this block."
It might be easier to think of it this way:
>>> book_dict = {'Title': 'A River', 'Author': 'Elisha Mitchell', 'Publisher': 'FPG Publishing', 'Pages': '345', 'Year': '2014', 'Copies': 2, 'Available': 2, 'ID': 17104}
>>> book_dict_entries = list(book_dict.items())
>>> print(book_dict_entries)
[('Title', 'A River'), ('Author', 'Elisha Mitchell'), ('Publisher', 'FPG Publishing'), ('Pages', '345'), ('Year', '2014'), ('Copies', 2), ('Available', 2), ('ID', 17104)]
There's a few directions to go from here. One way is that - since it's just a list -you can search for the one representing the ID field and swap it with whatever happens to be the first element in that list. Or, before turning it into a list, simply print ID from the dictionary, then filter that field when enumerating through the rest of the fields.
As to your second question, if you want to print an empty line at a certain point - simply call print with no arguments. Like when you've finished printing each dictionary.

I would define the dict with ID as the first key since from Python 3.7 (not before!) dicts are ordered
Put ID as the first key, and add print() after each inner loop.
book_collection = {
17104:
{'ID': 17104, 'Title': 'A River', 'Author': 'Elisha Mitchell',
'Publisher': 'FPG Publishing', 'Pages': '345', 'Year': '2014',
'Copies': 2, 'Available': 2, },
37115:
{'ID': 37115, 'Title': 'Aim High', 'Author': 'George Tayloe Winston',
'Publisher': 'Manning Hall Press', 'Pages': '663', 'Year': '2014',
'Copies': 5, 'Available': 5}
}
for id, book in book_collection.items():
for book_attribute, attribute_value in book.items():
print(book_attribute, ': ', attribute_value, sep='')
print()

Scraping profiles with Python and the "scrape-linkedin" package

I am trying to use the scrape_linkedin package. I follow the section on the github page on how to set up the package/LinkedIn li_at key (which I paste here for clarity).
Getting LI_AT
Navigate to www.linkedin.com and log in
Open browser developer tools (Ctrl-Shift-I or right click -> inspect element)
Select the appropriate tab for your browser (Application on Chrome, Storage on Firefox)
Click the Cookies dropdown on the left-hand menu, and select the www.linkedin.com option
Find and copy the li_at value
Once I collect the li_at value from my LinkedIn, I run the following code:
from scrape_linkedin import ProfileScraper
with ProfileScraper(cookie='myVeryLong_li_at_Code_which_has_characters_like_AQEDAQNZwYQAC5_etc') as scraper:
profile = scraper.scrape(url='https://www.linkedin.com/in/justintrudeau/')
print(profile.to_dict())
I have two questions (I am originally an R user).
How can I input a list of profiles:
https://www.linkedin.com/in/justintrudeau/
https://www.linkedin.com/in/barackobama/
https://www.linkedin.com/in/williamhgates/
https://www.linkedin.com/in/wozniaksteve/
and scrape the profiles? (In R I would use the map function from the purrr package to apply the function to each of the LinkedIn profiles).
The output (from the original github page) is returned in a JSON style format. My second question is how I can convert this into a pandas data frame (i.e. it is returned similar to the following).
{'personal_info': {'name': 'Steve Wozniak', 'headline': 'Fellow at
Apple', 'company': None, 'school': None, 'location': 'San Francisco
Bay Area', 'summary': '', 'image': '', 'followers': '', 'email': None,
'phone': None, 'connected': None, 'websites': [],
'current_company_link': 'https://www.linkedin.com/company/sandisk/'},
'experiences': {'jobs': [{'title': 'Chief Scientist', 'company':
'Fusion-io', 'date_range': 'Jul 2014 – Present', 'location': 'Primary
Data', 'description': "I'm looking into future technologies applicable
to servers and storage, and helping this company, which I love, get
noticed and get a lead so that the world can discover the new amazing
technology they have developed. My role is principally a marketing one
at present but that will change over time.", 'li_company_url':
'https://www.linkedin.com/company/sandisk/'}, {'title': 'Fellow',
'company': 'Apple', 'date_range': 'Mar 1976 – Present', 'location': '1
Infinite Loop, Cupertino, CA 94015', 'description': 'Digital Design
engineer.', 'li_company_url': ''}, {'title': 'President & CTO',
'company': 'Wheels of Zeus', 'date_range': '2002 – 2005', 'location':
None, 'description': None, 'li_company_url':
'https://www.linkedin.com/company/wheels-of-zeus/'}, {'title':
'diagnostic programmer', 'company': 'TENET Inc.', 'date_range': '1970
– 1971', 'location': None, 'description': None, 'li_company_url':
''}], 'education': [{'name': 'University of California, Berkeley',
'degree': 'BS', 'grades': None, 'field_of_study': 'EE & CS',
'date_range': '1971 – 1986', 'activities': None}, {'name': 'University
of Colorado Boulder', 'degree': 'Honorary PhD.', 'grades': None,
'field_of_study': 'Electrical and Electronics Engineering',
'date_range': '1968 – 1969', 'activities': None}], 'volunteering':
[]}, 'skills': [], 'accomplishments': {'publications': [],
'certifications': [], 'patents': [], 'courses': [], 'projects': [],
'honors': [], 'test_scores': [], 'languages': [], 'organizations':
[]}, 'interests': ['Western Digital', 'University of Colorado
Boulder', 'Western Digital Data Center Solutions', 'NEW Homebrew
Computer Club', 'Wheels of Zeus', 'SanDisk®']}

Firstly, You can create a custom function to scrape data and use map function in Python to apply it over each profile link.
Secondly, to create a pandas dataframe using a dictionary, you can simply pass the dictionary to pd.DataFrame.
Thus to create a dataframe df, with dictionary dict, you can do like this:
df = pd.DataFrame(dict)

How to print this nested dictionary containing several list and dictionaries?

[[{'text': '\n ', 'category': 'cooking', 'title': {'text': 'Everyday
Italian', 'lang': 'en'}, 'author': {'text': 'Giada De Laurentiis'}, 'year':
{'text': '2005'}, 'price': {'text': '30.00'}},
{'text': '\n ', 'category': 'children', 'title': {'text': 'Harry Potter',
'lang': 'en'}, 'author': {'text': 'J K. Rowling'}, 'year': {'text':
'2005'}, 'price': {'text': '29.99'}}, {'text': '\n ', 'category':
'web', 'title': {'text': 'XQuery Kick Start', 'lang': 'en'}, 'author':
[{'text': 'James McGovern'}, {'text': 'Per Bothner'}, {'text': 'Kurt
Cagle'}, {'text': 'James Linn'}, {'text': 'Vaidyanathan Nagarajan'}],
'year': {'text': '2003'}, 'price': {'text': '49.99'}}, {'text': '\n ',
'category': 'web', 'cover': 'paperback', 'title': {'text': 'Learning XML',
'lang': 'en'}, 'author': {'text': 'Erik T. Ray'}, 'year': {'text': '2003'},
'price': {'text': '39.95'}}]]
output format:
category : cooking,
title : ['Everyday Italian', 'lang': 'en'],
author : Giada De Laurentiis,
year : '2005',
price : '30.00'
category : children,
title : ['Harry Potter', 'lang': 'en'],
author : 'J K. Rowling',
year : '2005',
price : '29.99'
category : web,
title : [ 'XQuery Kick Start''lang': 'en'],
author :[ 'James McGovern' , 'Per Bothner','Kurt Cagle','James Linn', 'Vaidyanathan Nagarajan'],
year : '2003',
price : '49.99'
category : web,
cover : paperback,
title : [ 'Learning XML','lang': 'en'],
author : 'Erik T. Ray',
year : '2003',
price : '39.95'

A simple loop like the following should get the output you require.
for entry in data[0]:
for k, v in entry.items():
print(k, ':', v)

def printBook(d):
del d['text']
for i in d:
if type(d[i]) == dict:
if len(d[i])==1:
d[i] = list(d[i].values())[0]
else:
d[i] = [('' if j=='text' else (j+':')) + d[i][j] for j in d[i]]
s = '\n'
for i,j in d.items():
s += f' {i} : {j} ,\n'
print(s)
try these it prints individual dictionary into your described format

Take a look at the pprint module which provides a nice way of printing data structures without the need for writing your own formatter.

Thanks for the coding excercise. Man, that output format was specific! :D Tricky, as the strings are not quoted if being standalone, quoted if coming from text attribute. Also tricky, that stuff must be thrown into [] if not just text. Also it is a little bit underspecified, because hey, what if there is no text at all, yet other keys?
Output format disclaimer:
I think there is a missing , after 'XQuery Kick Start'
I think Giada De Laurentiis should have been quoted as it is coming from 'text'
import copy
def transform_value(stuff):
if type(stuff) is str:
return stuff
if type(stuff) is dict:
elements = []
text = stuff.pop("text", "")
if text:
elements.append(f"'{text}'") # we quote only if there was really a text entry
if not stuff: # dict just got empty
return elements[0] # no more attributes, so no [] needed around text
elements.extend(f"'{key}': '{transform_value(value)}'" for key, value in stuff.items())
if type(stuff) is list:
elements = [transform_value(e) for e in stuff]
# this will obviously raise an exception if stuff is not one of str, dict or list
return f"[{', '.join(elements)}]"
def transform_pub(d: dict):
d = copy.deepcopy(d) # we are gonna delete keys, so we don't mess with the outer data
tail = d.pop("text", "")
result = ",\n".join(f'{key} : {transform_value(value)}' for key, value in d.items())
return result + tail
if __name__ == "__main__":
for sublist in data:
for pub in sublist:
print(transform_pub(pub))
I first wanted to use somehow the same mechanism for the publications themselves via some recursion. But then the code become too complicated as the text field is appended for publications, while it is coming first for attributes.
Once I let go of the fully structured solution, I started out with a test for the publication printer:
import pytest
from printing import transform_value
#pytest.mark.parametrize('input,output', [
("cooking", "cooking"),
({"text": "J K. Rowling"}, "'J K. Rowling'"),
({"lang": "en", "text": "Everyday Italian"},
"['Everyday Italian', 'lang': 'en']"),
([{"text": "James McGovern"},
{"text": "Per Bothner"},
{"text": "Kurt Cagle"},
{"text": "James Linn"},
{"text": "Vaidyanathan Nagarajan"}],
"['James McGovern', 'Per Bothner', 'Kurt Cagle', 'James Linn', 'Vaidyanathan Nagarajan']"
),
([{"text": "Joe"}], "['Joe']"),
({"a": "1"}, "['a': '1']"),
])
def test_transform_value(input, output):
assert transform_value(input) == output

Accessing keys/values in a paginated/nested dictionary

I know that somewhat related questions have been asked here: Accessing key, value in a nested dictionary and here: python accessing elements in a dictionary inside dictionary among other places but I can't quite seem to apply the answers' methodology to my issue.
I'm getting a KeyError trying to access the keys within response_dict, which I know is due to it being nested/paginated and me going about this the wrong way. Can anybody help and/or point me in the right direction?
import requests
import json
URL = "https://api.constantcontact.com/v2/contacts?status=ALL&limit=1&api_key=<redacted>&access_token=<redacted>"
#make my request, store it in the requests object 'r'
r = requests.get(url = URL)
#status code to prove things are working
print (r.status_code)
#print what was retrieved from the API
print (r.text)
#visual aid
print ('---------------------------')
#decode json data to a dict
response_dict = json.loads(r.text)
#show how the API response looks now
print(response_dict)
#just for confirmation
print (type(response_dict))
print('-------------------------')
# HERE LIES THE ISSUE
print(response_dict['first_name'])
And my output:
200
{"meta":{"pagination":{}},"results":[{"id":"1329683950","status":"ACTIVE","fax":"","addresses":[{"id":"4e19e250-b5d9-11e8-9849-d4ae5275509e","line1":"222 Fake St.","line2":"","line3":"","city":"Kansas City","address_type":"BUSINESS","state_code":"","state":"OK","country_code":"ve","postal_code":"19512","sub_postal_code":""}],"notes":[],"confirmed":false,"lists":[{"id":"1733488365","status":"ACTIVE"}],"source":"Site Owner","email_addresses":[{"id":"1fe198a0-b5d5-11e8-92c1-d4ae526edd6c","status":"ACTIVE","confirm_status":"NO_CONFIRMATION_REQUIRED","opt_in_source":"ACTION_BY_OWNER","opt_in_date":"2018-09-11T18:18:20.000Z","email_address":"rsmith#fake.com"}],"prefix_name":"","first_name":"Robert","middle_name":"","last_name":"Smith","job_title":"I.T.","company_name":"FBI","home_phone":"","work_phone":"5555555555","cell_phone":"","custom_fields":[],"created_date":"2018-09-11T15:12:40.000Z","modified_date":"2018-09-11T18:18:20.000Z","source_details":""}]}
---------------------------
{'meta': {'pagination': {}}, 'results': [{'id': '1329683950', 'status': 'ACTIVE', 'fax': '', 'addresses': [{'id': '4e19e250-b5d9-11e8-9849-d4ae5275509e', 'line1': '222 Fake St.', 'line2': '', 'line3': '', 'city': 'Kansas City', 'address_type': 'BUSINESS', 'state_code': '', 'state': 'OK', 'country_code': 've', 'postal_code': '19512', 'sub_postal_code': ''}], 'notes': [], 'confirmed': False, 'lists': [{'id': '1733488365', 'status': 'ACTIVE'}], 'source': 'Site Owner', 'email_addresses': [{'id': '1fe198a0-b5d5-11e8-92c1-d4ae526edd6c', 'status': 'ACTIVE', 'confirm_status': 'NO_CONFIRMATION_REQUIRED', 'opt_in_source': 'ACTION_BY_OWNER', 'opt_in_date': '2018-09-11T18:18:20.000Z', 'email_address': 'rsmith#fake.com'}], 'prefix_name': '', 'first_name': 'Robert', 'middle_name': '', 'last_name': 'Smith', 'job_title': 'I.T.', 'company_name': 'FBI', 'home_phone': '', 'work_phone': '5555555555', 'cell_phone': '', 'custom_fields': [], 'created_date': '2018-09-11T15:12:40.000Z', 'modified_date': '2018-09-11T18:18:20.000Z', 'source_details': ''}]}
<class 'dict'>
-------------------------
Traceback (most recent call last):
File "C:\Users\rkiek\Desktop\Python WIP\Chris2.py", line 20, in <module>
print(response_dict['first_name'])
KeyError: 'first_name'

first_name = response_dict["results"][0]["first_name"]
Even though I think this question would be better answered by yourself by reading some documentation, I will explain what is going on here. You see the dict-object of the man named "Robert" is within a list which is a value under the key "results". So, at first you need to access the value within results which is a python-list.
Then you can use a loop to iterate through each of the elements within the list, and treat each individual element as a regular dictionary object.
results = response_dict["results"]
results = response_dict.get("results", None)
# use any one of the two above, the first one will throw a KeyError if there is no key=="results" the other will return NULL
# this results is now a list according to the data you mentioned.
for item in results:
print(item.get("first_name", None)
# here you can loop through the list of dictionaries and treat each item as a normal dictionary

Merge a specific value from one array of dicts into another if a value matches

How do I merge a specific value from one array of dicts into another array of dicts if a single specific value matches between them?
I have an array of dicts that represent books
books = [{'writer_id': '123-456-789', 'index': None, 'title': 'Yellow Snow'}, {'writer_id': '888-888-777', 'index': None, 'title': 'Python for Dummies'}, {'writer_id': '999-121-223', 'index': 'Foo', 'title': 'Something Else'}]
and I have an array of dicts that represents authors
authors = [{'roles': ['author'], 'profile_picture': None, 'author_id': '123-456-789', 'name': 'Pat'}, {'roles': ['author'], 'profile_picture': None, 'author_id': '999-121-223', 'name': 'May'}]
I want to take the name from authors and add it to the dict in books where the books writer_id matches the authors author_id.
My end result would ideally change the book array of dicts to be (notice the first dict now has the value of 'name': 'Pat' and the second book has 'name': 'May'):
books = [{'writer_id': '123-456-789', 'index': None, 'title': 'Yellow Snow', 'name': 'Pat'}, {'writer_id': '888-888-777', 'index': None, 'title': 'Python for Dummies'}, {'writer_id': '999-121-223', 'index': 'Foo', 'title': 'Something Else', 'name': 'May'}]
My current solution is:
for book in books:
for author in authors:
if book['writer_id'] == author['author_id']:
book['author_name'] = author['name']
And this works. However, the nested statements bother me and feel unwieldy. I also have a number of other such structures so I end up with a function that has a bunch of code resembling this in it:
for book in books:
for author in authors:
if book['writer_id'] == author['author_id']:
book['author_name'] = author['name']
books_with_foo = []
for book in books:
for thing in things:
if something:
// do something
for blah in books_with_foo:
for book_foo in otherthing:
if blah['bar'] == stuff['baz']:
// etc, etc.
Alternatively, how would you aggregate data from multiple database tables into one thing... some of the data comes back as dicts, some as arrays of dicts?

Pandas is almost definitely going to help you here. Convert your dicts to DataFrames for easier manipulation, then merge them:
import pandas as pd
authors = [{'roles': ['author'], 'profile_picture': None, 'author_id': '123-456-789', 'name': 'Pat'}, {'roles': ['author'], 'profile_picture': None, 'author_id': '999-121-223', 'name': 'May'}]
books = [{'writer_id': '123-456-789', 'index': None, 'title': 'Yellow Snow'}, {'writer_id': '888-888-777', 'index': None, 'title': 'Python for Dummies'}, {'writer_id': '999-121-223', 'index': 'Foo', 'title': 'Something Else'}]
df1 = pd.DataFrame.from_dict(books)
df2 = pd.DataFrame.from_dict(authors)
df1['author_id'] = df1.writer_id
df1 = df1.set_index('author_id')
df2 = df2.set_index('author_id')
result = pd.concat([df1, df2], axis=1)
you may find this page helpful for different ways of combining (merging, concatenating, etc) separate DataFrames.