Develop Reference

Scraping data from a http & javaScript site

I currently want to scrape some data from an amazon page and I'm kind of stuck.
For example, lets take this page.
https://www.amazon.com/NIKE-Hyperfre3sh-Athletic-Sneakers-Shoes/dp/B01KWIUHAM/ref=sr_1_1_sspa?ie=UTF8&qid=1546731934&sr=8-1-spons&keywords=nike+shoes&psc=1
I wanted to scrape every variant of shoe size and color. That data can be found opening the source code and searching for 'variationValues'.
There we can see sort of a dictionary containing all the sizes and colors and, below that, in 'asinToDimentionIndexMap', every product code with numbers indicating the variant from the variationValues 'dictionary'.
For example, in asinToDimentionIndexMap we can see
"B01KWIUH5M":[0,0]
Which means that the product code B01KWIUH5M is associated with the size '8M US' (position 0 in variationValues size_name section) and the color 'Teal' (same idea as before)
I want to scrape both the variationValues and the asinToDimentionIndexMap, so i can associate the IndexMap numbers to the variationValues one.
Another person in the site (thanks for the help btw) suggested doing it this way.
script = response.xpath('//script/text()').extract_frist()
import re
# capture everything between {}
data = re.findall(script, '(\{.+?\}_')
import json
d = json.loads(data[0])
d['products'][0]
I can sort of understand the first part. We get everything that's a 'script' as a string and then get everything between {}. The issue is what happens after that. My knowledge of json is not that great and reading some stuff about it didn't help that much.
Is it there a way to get, from that data, 2 dictionaries or lists with the variationValues and asinToDimentionIndexMap? (maybe using some regular expressions in the middle to get some data out of a big string). Or explain a little bit what happens with the json part.
Thanks for the help!
EDIT: Added photo of variationValues and asinToDimensionIndexMap

I think you are close Manuel!
The following code will turn your scraped source into easy-to-select boxes:
import json
d = json.loads(data[0])
JSON is a universal format for storing object information. In other words, it's designed to interpret string data into object data, regardless of the platform you are working with.
https://www.w3schools.com/js/js_json_intro.asp
I'm assuming where you may be finding things a challenge is if there are any errors when accessing a particular "box" inside you json object.
Your code format looks correct, but your access within "each box" may look different.
Eg. If your 'asinToDimentionIndexMap' object is nested within a smaller box in the larger 'products' object, then you might access it like this (after running the code above):
d['products'][0]['asinToDimentionIndexMap']
I've hacked and slash a little bit so you can better understand the structure of your particular json file. Take a look at the link below. On the right-hand side, you will see "which boxes are within one another" - which is precisely what you need to know for accessing what you need.
JSON Object Viewer
For example, the following would yield "companyCompliancePolicies_feature_div":
import json
d = json.loads(data[0])
d['updateDivLists']['full'][0]['divToUpdate']
The person helping you before outlined a general case for you, but you'll need to go in an look at structure this way to truly find what you're looking for.

variationValues = re.findall(r'variationValues\" : ({.*?})', ' '.join(script))[0]
asinVariationValues = re.findall(r'asinVariationValues\" : ({.*?}})', ' '.join(script))[0]
dimensionValuesData = re.findall(r'dimensionValuesData\" : (\[.*\])', ' '.join(script))[0]
asinToDimensionIndexMap = re.findall(r'asinToDimensionIndexMap\" : ({.*})', ' '.join(script))[0]
dimensionValuesDisplayData = re.findall(r'dimensionValuesDisplayData\" : ({.*})', ' '.join(script))[0]
Now you can easily convert them to json as use them combine as you wish.

What data type single element extracted from entries attribute of Connection object represent in python?

I am not well versed at python at all. I was asked to review someone else's python script that uses search ldap entries. Btw - I can't reach out to original developer for some reason and before it is deployed + tested visual code checking is required. With that constraints in mind, allow me to proceed.
import ldap3
from ldap3 import Server,Connection, ALL
conn = Connection(....)
conn.search(....)
for entry in conn.entries:
if (len(entry['cn']) > 0):
....
name = entry['name']
if name:
user_name = str(name)
else:
user_name = "Bob"
First question is len(entry['cn']) > 0 I like to interpret it as checking the length of characters of returned cn value from ldap e.g. cn=bob,ou=people,ou=foocomany. I am pretty sure entry['cn'] is NOT string type but I don't know what data type it represents. Can you tell me what type it is?
My 2nd + 3rd questions are not directly related to the original question, but plz bear with me asking for them with grace.
My 2nd question is, if that assumption is correct, entry['cn'] should be converted to string type like str(entry['cn']). Then check its length?
My 3rd question is on if stmt. I like to interpret it as if name is not null or if name is not None in pythonic way. Did I interpret it correctly? If so I should replace it as if not (name is None) would work? I googled on it to get that stmt.

Given the context and code provided, it looks like this snippet is using the ldap3 library.
From the relevant documentation, conn.entries should be a list of Entry objects.
This means that entry['cn'] should be returning an Attribute. Doing a bit of source diving, this appears to just be a fancy list with writable flags. len(entry['cn']) > 0 ends up calling this method, which just returns the number of values that attribute has. It being greater than 0 just ensuring that the cn is, in fact, set.

Kivy Reference id by string

I'm building an app that has a grid of 20x20 labels which I've set as ids id: x1y1, id: x1y2, ... , id: x20y20 etc. I want to be able to reference the id by passing a string, and modify the text within, using something like;
self.ids.name("x1y1").text = "Anything"
I've had a look at dir(self.ids), dir(self.ids.items()) etc but can't seem to figure it out. Is it even possible?
I could create a list of if statements like;
if str(passedString) == "x1y1":
self.id.x1y1.text == "Anything"
if str(passedString) == "x1y2":
self.id.x1y2.text == "Anything"
Although I feel that this is incredibly bad practice, also considering I'd need 400 if statements! - Sure I could write a little helper-script to write all this code out for me, but again, it's not ideal.
EDIT:
I had a look at;
for key, val in self.ids.items():
print("key={0}, val={1}".format(key, val))
Which printed;
key=x1y2, val=<kivy.uix.label.Label object at 0x7f565a34e6d0>
key=x1y3, val=<kivy.uix.label.Label object at 0x7f565a2c1e88>
Might give you/someone an idea of where to go and/or what to do.

You just want to get the reference via a string of the id? You can use normal dictionary lookup (as ids is really a dict).
the_string = 'x1y1'
the_reference = self.ids[the_string]
the_reference.text = 'the_new_text'

Why is find_withtag wont work in python

This is part of my code, the id of image wont print even I use the withtag function. I think the function canvas.delete won't work is also the same problem, it seem the tag is inserted as "123","456". However, the tag I expected to use and get is 123 instead of '123'. And I guess it's the main problem I can't get the id I want with the findtag function.
CurrentImage=Note[NoteIndexLocal]
Temp=canvas.create_image(XShow,YShow,image=CurrentImage,tag=123)
print canvas.find_withtag(123) #This Wont Work,printed()
canvas.delete(123) #This Wont Work
print canvas.gettags(Temp) #printed '123'

From: http://effbot.org/tkinterbook/canvas.htm
Tags are symbolic names attached to items. Tags are ordinary strings,
and they can contain anything except whitespace (as long as they don’t
look like item handles).
Use str(123) instead of 123
EDIT: correct answer is in text from doc "as long as they don’t look like item handles". Number 123 looks like item handle (print Temp to see how it looks like) so it doesn't work. Use text like "a123" and it will work.

Read characters in a string only if I have found the part that I need? [Python]

I phrased the title as well as I possibly could. Basically, here's my scenario:
I have a string, not a file, that I need to read from. In this string, I want to check if there is a certain piece of text in this file. If this text is found in the string, I want to read this text up until I want to stop. More specifically, I'm getting text from a webpage and storing it in a string. In this text, there is an ID that the user is given. I want to check for the text where the ID is given. If I find this text in the string, I want to continue reading until I get the actual ID, then I need to stop reading. Yes, I could just manually read the ID myself, and to be honest it's probably a better solution, but I wanna have some fun and see how something like this could be done, if it can be done at all. Here's the code I have:
if 'Your ID is: ' in str(response):
print 'Found ID!'
# I'd probably have something here to store the ID in a variable for later use.
# I need to be able to check for the ID after I check for the text 'Your ID is: '. I check for this because I know where to start reading from. I have no way of telling what line the ID could possibly be on.
Hopefully that made sense. I just need to check if a certain piece of text is in a string, and if it is read just a little more of that text and then store that in a variable for later use. This is probably a rare case and I'll probably never use it, but it never hurts to see what's possible and how it's possible, right?
Thanks in advance!

I just need to check if a certain piece of text is in a string, and if it is read just a little more of that text and then store that in a variable for later use.
The obvious thing to do is use str.partition:
>>> response = 'Hello there. Your ID is: 6. Your name is: Number 6.'
>>> before, matched, after = response.partition('Your ID is: ')
>>> if after:
... print after
6. Your name is: Number 6.
Or, if it's not found:
>>> response = 'Hello there. You have no ID. Your name is: Mud.'
>>> before, matched, after = response.partition('Your ID is: ')
>>> if after:
... print after
So how do you get "just a little more"? Well, you have to decide what "a little more" means, but it's very easy to slice or split or partition the after variable to get the appropriate "little". Maybe you want everything up to the first '.'? Then:
>>> response = 'Hello there. Your ID is: 6. Your name is: Number 6.'
>>> before, matched, after = response.partition('Your ID is: ')
>>> if after:
... yourid, _, _ = after.partition('.')
... print yourid
6