I'm using Python \ Selenium \ Chrome driver to perform webscraping. I want to pass an INT variable (id) to a URL - how do I do this? I have tried all the below but Python errors on this line:
id = 2000
# Part 1: Customer:
#urlg = 'https://mythirteen.co.uk/customerRest/show/?id=2000' working but need to pass variable
#urlg = 'https://mywebsite.com/customerRest/show/?id=' %(id)
#urlg = 'https://mywebsite.com/customerRest/show/?id={id}'
# urlg = 'https://mywebsite.com/customerRest/show/?id='.format(id)
# urlg = 'https://mywebsite.com/customerRest/show/?id='+id
# urlg = "https://mywebsite.com/customerRest/show/?id=".format(id)
# urlg = 'https://mywebsite.com/customerRest/show/?id=' % id
driver.get(urlg)
I receive errors such as:
TypeError: not all arguments converted during string formatting
I know it its not a string though - id is INT.
Ultimately, I will need to loop through and increase the id + 1 each time, but for now I just want to be able pass the actual variable.
If you want to concatenate strings and integer, for most methods you have to cast the integer to a string with str(id).
Otherwise I really like using f-strings:
urlg = f'https://mywebsite.com/customerRest/show/?id={id}'
Edit
As #chitown88 mentioned, using id as a variable is not a good idea, as it is reserved internally. Better use something like customer_id.
The problem with the methods you are trying is you essentially aren't telling it where to put the string, or trying to concat a string and an int
so this 'https://mywebsite.com/customerRest/show/?id=' %(id) needs to be 'https://mywebsite.com/customerRest/show/?id=%s' %(id)
So all these would work:
Also, I would not use id as the variable, as it's a reserved function in python. It also could make more sense to make it more descriptive:
customerId = 2000
urlg = 'https://mythirteen.co.uk/customerRest/show/?id=2000' working but need to pass variable
urlg = 'https://mywebsite.com/customerRest/show/?id=%s' %(customerId)
urlg = 'https://mywebsite.com/customerRest/show/?id=%s' %customerId
urlg = f'https://mywebsite.com/customerRest/show/?id={customerId}'
urlg = 'https://mywebsite.com/customerRest/show/?id={}'.format(customerId)
urlg = 'https://mywebsite.com/customerRest/show/?id='+str(customerId)
Related
Hi I'm new to using html and python. But I need to use html and python interchangeably.
For example,
if python output = 30302,
then I need to put '30302' in the hyperlink.
www.google.com/< output> = www.google.com/30302
html = 'www.google.com/'
python = < output>
how would I combine those two?
The problem
I guess that you wanna create a new string from two parts, i.e. you have a string "www.google.com/" and a variable output with integer 30302 and you wanna get the "www.google.com/30302" (for the future, always provide full examples of your code).
So how can you do it?
Convert int to str and concatenate strs
result = "www.google.com/" + str(output)
str(x) will turn x into a string
Formatting
"www.google.com/{}".format(31415)" is equivalent to the "www.google.com/31415" string,
so result = "www.google.com/{}".format(output)" also will work
in python 3 we also have f-stings:
f"www.google.com/{31415}" == "www.google.com/31415"
result = f"www.google.com/{output}"
I need to create variables dynamically based on the data that is coming from my UI.
Sample JSON:
Some of the sample JSON I'll get from UI to hit the python code.
data_json = {'key1':'value1','key2':'value2','key3':['abc','def']}
data_json = {'key1':'value2','key2':'value8','key3':['abc','def','ghi','jklmn']}
data_json = {'key1':'value3','key2':'value9','key3':['abc']}
data_json = {'key1':'value4','key2':'value2','key3':['abc','def','xyz']}
data_json = {'key1':'value6','key2':'value2','key3':['abc','def']}
I have data in JSON format in which the length of the "key3" value will keep changing each time.
I have to capture those values in separate variables and have to use them later in other functions.
If I pass the first data_json first block of if condition will work and assign it to variables. And if I pass the second data_json second block will define the variables.
Python:
secret = data_json['key1']
if secret in ['value1','value6']:
first_value = data_json['key3'][0]
second_value = data_json['key3'][1]
if secret in ['value2']:
first_value = data_json['key3'][0]
second_value = data_json['key3'][1]
third_value = data_json['key3'][2]
fourth_value = data_json]'key3'][3]
if secret in ['value3']:
first_value = data_json['key3'][0]
if secret in ['value4']:
first_value = data_json['key3'][0]
second_value = data_json['key3'][1]
third_value = data_json['key3'][2]
print("Value in first:%s",first_value)
print("Value in second :%s",second_value)
print("Value in third:%s",third_value)
I'm using conditions to capture those variables.The above code is working fine. But I have to avoid using if conditions. Is there any way to define the variables dynamically on the fly and so that i can use it later in same functions?
I don't think you are approaching it the right way. For such cases - where we have unknown number of variables - we use lists! Lists in python are of dynamic size. So, you don't need to know the exact size before creating a list.
Therefore, you can store your numbers in a list and then access them using the indices like this:
all_values = data_json['key3']
print("Value in first:%s", all_-values[0])
print("Value in second :%s", all_values[1])
print("Value in third:%s", all_values[2])
Note that here you don't need conditional statements to make sure you are reading the exact number of values (not more or less) from the JSON.
What you are calling dynamic variables are not needed! Wherever you need first_value, you can use all_values[0]. For, second_value, you can use all_values[1] and so on...
The best way to solve your problem is to save the values in an array and access then via indices, rather than creating separate variables for each element in the array.
data_json = {'key1':'value2','key2':'value8','key3':['abc','def','ghi','jklmn']}
key3_vars = data_json['key3']
for var in key3_vars:
print(var)
But if you have to create separate variables, then you can use the built-in function exec.
data_json = {'key1':'value2','key2':'value8','key3':['abc','def','ghi','jklmn']}
key3_vars = data_json['key3']
for i, var in enumerate(key3_vars):
exec(f"key3_var{i} = '{var}'")
print(key3_var0)
print(key3_var1)
print(key3_var2)
print(key3_var3)
I am currently working on a script to scrape data from ClinicalTrials.gov. To do this I have written the following script:
def clinicalTrialsGov (id):
url = "https://clinicaltrials.gov/ct2/show/" + id + "?displayxml=true"
data = BeautifulSoup(requests.get(url).text, "lxml")
studyType = data.study_type.text
if studyType == 'Interventional':
allocation = data.allocation.text
interventionModel = data.intervention_model.text
primaryPurpose = data.primary_purpose.text
masking = data.masking.text
enrollment = data.enrollment.text
officialTitle = data.official_title.text
condition = data.condition.text
minAge = data.eligibility.minimum_age.text
maxAge = data.eligibility.maximum_age.text
gender = data.eligibility.gender.text
healthyVolunteers = data.eligibility.healthy_volunteers.text
armType = []
intType = []
for each in data.findAll('intervention'):
intType.append(each.intervention_type.text)
for each in data.findAll('arm_group'):
armType.append(each.arm_group_type.text)
citedPMID = tryExceptCT(data, '.results_reference.PMID')
citedPMID = data.results_reference.PMID
print(citedPMID)
return officialTitle, studyType, allocation, interventionModel, primaryPurpose, masking, enrollment, condition, minAge, maxAge, gender, healthyVolunteers, armType, intType
However, the following script won't always work as not all studies will have all items (ie. a KeyError will occur). To resolve this, I could simply wrap each statement in a try-except catch, like this:
try:
studyType = data.study_type.text
except:
studyType = ""
but it seems a bad way to implement this. What's a better/cleaner solution?
This is a good question. Before I address it, let me say that you should consider changing the second parameter to the BeautifulSoup (BS) constructor from lxml to xml. Otherwise, BS does not flag the parsed markup as XML (to verify this for yourself, access the is_xml attribute on the data variable in your code).
You can avoid generating an error when attempting to access a non-existent element by passing a list of desired element names to the find_all() method:
subset = ['results_reference','allocation','interventionModel','primaryPurpose','masking','enrollment','eligibility','official_title','arm_group','condition']
tag_matches = data.find_all(subset)
Then, if you want to get a specific element from the list of Tags without iterating through it, you can convert it to a dict using the Tag names as keys:
tag_dict = dict((tag_matches[i].name, tag_matches[i]) for i in range(0, len(tag_matches)))
I am using a new script (a) to extract information from an old script (b) to create a new file (c). I am looking for an equal sign in the old script (b) and want to modify the modification script (a) to make it automated.
The string is
lev1tolev2 'from=e119-b3331l1 mappars="simp:180" targ=enceladus.bi.def.3 km=0.6 lat=(-71.5,90) lon=(220,360)'
It is written in python 3.
The current output is fixed at
cam2map from=e119-b3331l1 to=rsmap-x map=enc.Ink.map pixres=mpp defaultrange=MAP res=300 minlat=-71.5 maxlat=90 minlon=220 maxlon=360
Currently, I have the code able to export a string of 0.6 for all of the iterations of lev1tolev2, but each one of these is going to be different.
cam2map = Call("cam2map")
cam2map.kwargs["from"] = old_lev1tolev2.kwargs["from"]
cam2map.kwargs["to"] = "rsmap-x"
cam2map.kwargs["map"] = "enc.Ink.map"
cam2map.kwargs["pixres"] = "mpp"
cam2map.kwargs["defaultrange"] = "MAP"
**cam2map.kwargs["res"] = float((old_lev1tolev2.kwargs["km"]))**
cam2map.kwargs["minlat"] = lat[0]
cam2map.kwargs["maxlat"] = lat[1]
cam2map.kwargs["minlon"] = lon[0]
cam2map.kwargs["maxlon"] = lon[1]
I have two questions, why is this not converting the string to a float? And, why is this not iterating over all of the lev1tolev2 commands as everything else in the code does?
The full code is available here.
https://codeshare.io/G6drmk
The problem occurred at a different location in the code.
def escape_kw_value(value):
if not isinstance(value, str):
return value
elif (value.startswith(('"', "'")) and value.endswith(('"', "'"))):
return value
# TODO escape the quote with \" or \'
#if value.startswith(('"', "'")) or value.endswith(('"', "'")):
# return value
if " " in value:
value = '"{}"'.format(value)
return value
it doesn't seem to clear to me, but from you syntax here :
**cam2map.kwargs["res"] = float((old_lev1tolev2.kwargs["km"]))**
I'd bet that cam2map.kwargs["res"] is a dict, and you thought that it would convert every values in the dict, using the ** syntax. The float built-in should then be called in a loop over the elements of the dict, or possible a list-comprehension as here :
cam2map.kwargs["res"] = dict()
for key, value in old_lev1tolev2.kwars["res"].items():
cam2map.kwargs["res"][key] = float(value)
Edit :
Ok so, it seems you took the string 'from=e119-b3331l1 mappars="simp:180" targ=enceladus.bi.def.3 km=0.6 lat=(-71.5,90) lon=(220,360)'
And then thought that calling youstring.kwargs would give you a dict, but it won't, you can probably parse it to a dict first, using some lib, or, you use mystring.split('=') and then work your way to a dict first, like that:
output = dict()
for one_bit in lev_1_lev2.split(' '):
key, value = one_bit.split('=')
output[key] = value
Please tell me something like this is possible in Python. I can't seem to get it to work
MY_LENGTH_CONSTRAINT = 24
myFormatStr = '{mykey:<${MY_LENGTH_CONSTRAINT}s}'
myStr = myFormatStr.format(mykey='Something')
I keep getting
KeyError: 'MY_LENGTH_CONSTRAINT'
Add mcl = MY_LENGTH_CONSTRAINT to the parameters fed to format:
MY_LENGTH_CONSTRAINT = 24
myFormatStr = '{mykey:<{mlc}s}'
myStr = myFormatStr.format(mykey='Something',
mlc = MY_LENGTH_CONSTRAINT)
print(myStr)
# Something
You can also refer to local variables in your format string, and inform format of the values by passing it **locals():
MY_LENGTH_CONSTRAINT = 24
myFormatStr = '{mykey:<{MY_LENGTH_CONSTRAINT}s}'
myStr = myFormatStr.format(mykey='Something', **locals())
print(myStr)
# Something
(or similarly, use global variables, and pass format **globals().)