Trying to build a URL using the variables I've already setup and then a known part of the URL on the back end. The code will throw the following error:
TypeError: cannot concatenate 'str' and 'NoneType' objects
block_explorer_url = "https://blockexplorer.com/api/addrs/"
#?from=0&to=50
parser = argparse.ArgumentParser(description='Collect and visualize Bitcoin transactions and any related hidden services.')
parser.add_argument("--graph",help="Output filename of the graph file. Example: bitcoin.gexf",default="bitcoingraph.gexf")
parser.add_argument("--address", help="A bitcoin address to begin the search on.",)
args = parser.parse_args()
bitcoin_address = args.address
graph_file = args.graph
#
# Retrieve all bitcoin transactions for a Bitcoin address
#
def get_all_transactions(bitcoin_address):
transactions = []
from_number = 0
to_number = 50
block_explorer_url_full = block_explorer_url + bitcoin_address + "/txs?from=%d&to=%d" % (from_number,to_number)
Logically, having the variables there and then adding on the rest of the URL as a string makes since to me. Where am I going astray?
The problem is that when bitcoin_address is None (not provided by user), your program still tries to concatenate it to a str, which definitely won't work.
To solve that, you could add some code that checks the result of parse_args and raises an error when that happens, such as this:
if args.address is None:
raise ValueError('A bitcoin address must be provided.')
Separately, while your approach to string formatting is generally correct, you should consider moving away from C-style formatting and towards the format method, for example:
At the start of your script:
base_url = 'https://blockexplorer.com/api/addrs/{address}/txs?from={from}&to={to}'
And later in the function:
full_url = base_url.format(address=bitcoin_address,
from=from_number,
to=to_number)
Related
I'm currently developing locally an Azure function that communicates with Microsoft Sentinel, in order to fetch the alert rules from it, and more specifically their respective querys :
credentials = AzureCliCredential()
alert_rules_operations = SecurityInsights(credentials, SUBSCRIPTION_ID).alert_rules
list_alert_rules = alert_rules_operations.list(resource_group_name=os.getenv('RESOURCE_GROUP_NAME'), workspace_name=os.getenv('WORKSPACE_NAME'))
The issue is that when I'm looping over list_alert_rules, and try to see each rule's query, I get an error:
Exception: AttributeError: 'FusionAlertRule' object has no attribute 'query'.
Yet, when I check their type via the type() function:
list_alert_rules = alert_rules_operations.list(resource_group_name=os.getenv(
'RESOURCE_GROUP_NAME'), workspace_name=os.getenv('WORKSPACE_NAME'))
for rule in list_alert_rules:
print(type(rule))
##console: <class 'azure.mgmt.securityinsight.models._models_py3.ScheduledAlertRule'>
The weirder issue is that this error appears only when you don't print the attribute. Let me show you:
Print:
for rule in list_alert_rules:
query = rule.query
print('query', query)
##console: query YAY I GET WHAT I WANT
No print:
for rule in list_alert_rules:
query = rule.query
...
##console: Exception: AttributeError: 'FusionAlertRule' object has no attribute 'query'.
I posted the issue on the GitHub repo, but I'm not sure whether it's a package bug or a runtime issue. Has anyone ran into this kind of problems?
BTW I'm running Python 3.10.8
TIA!
EDIT:
I've tried using a map function, same issue:
def format_list(rule):
query = rule.query
# print('query', query)
# query = query.split('\n')
# query = list(filter(lambda line: "//" not in line, query))
# query = '\n'.join(query)
return rule
def main(mytimer: func.TimerRequest) -> None:
# results = fetch_missing_data()
credentials = AzureCliCredential()
alert_rules_operations = SecurityInsights(
credentials, SUBSCRIPTION_ID).alert_rules
list_alert_rules = alert_rules_operations.list(resource_group_name=os.getenv(
'RESOURCE_GROUP_NAME'), workspace_name=os.getenv('WORKSPACE_NAME'))
list_alert_rules = list(map(format_list, list_alert_rules))
I have tried with same as you used After I changed like below; I get the valid response.
# Management Plane - Alert Rules
alertRules = mgmt_client.alert_rules.list_by_resource_group('<ResourceGroup>')
for rule in alertRules:
# Try this
test.query = rule.query //Get the result
#print(rule)
if mytimer.past_due:
logging.info('The timer is past due!')
Instead of this
for rule in list_alert_rules:
query = rule.query
Try below
for rule in list_alert_rules:
# Try this
test.query = rule.query
Sorry for the late answer as I've been under tons of work these last few days.
Python has an excellent method called hasattr that checks if the object contains a specific key.
I've used it in the following way:
for rule in rules:
if hasattr(rule, 'query'):
...
The reason behind using this is because the method returns object of different classes, however inherited from the one same mother class.
Hope this helps.
I am using an API from this site https://dev.whatismymmr.com, and I want to specifically request for the closestRank but I just get a KeyError: 'ranked.closestRank'. but I can get the entire ['ranked'] object (which contains the closestRank) but I just end up with a lot of information I don't need.
How can I end up with just the Closest rank?
My code
import requests
LeagueName = input ("Summoner name")
base = ("https://eune.whatismymmr.com/api/v1/summoner?name=")
Thething = base + LeagueName
print (Thething)
response = requests.get(Thething)
print(response.status_code)
MMR = response.json()
print (MMR['ranked.closestRank'])
The API command
<queue>.closestRank (the queue is the game mode, it can be normal or ranked)
you can use the summoner name babada27 for testing.
Hope this is what you are looking for -
change The last line to
print (MMR["ranked"]["closestRank"])
I've been working with boto3 for a while in order to gather some values from the Parameter Store SSM, this is the code I use, which is very simple:
def get_raw_parameters_group_by_namespace(namespace_path):
raw_params_response = None
try:
if not namespace_path:
raise Exception('Namespace path should be specified to get data')
raw_params_response = ssm_ps.get_parameters_by_path(Path = namespace_path)
except Exception as e:
raise Exception('An error ocurred while trying to get parameters group: ' + str(e))
return raw_params_response
I used to have around 7 to 10 parameters in SSM and that method worked fine, however, we needed the add some additional parameters these days and the number of them increased to 14, so I tried adding a property in the boto3 ssm method called "MaxResults" and set it to 50:
ssm_ps.get_parameters_by_path(Path = namespace_path, MaxResults = 50)
but I get the following:
"error": "An error ocurred while trying to get parameters group: An error occurred (ValidationException) when calling the GetParametersByPath operation: 1 validation error detected: Value '50' at 'maxResults' failed to satisfy constraint: Member must have value less than or equal to 10."
After talking with the team, increasing the quota in the account is not an option, so I wonder to know if probably using the "NextToken" property would be a good option.
I am not sure on how this can be used, I have searched for examples, but I could not find something useful. Does anyone know how to use NextToken please? Or an example on how is it supposed to work?
I tried something like:
raw_params_response = ssm_ps.get_parameters_by_path(Path = namespace_path, NextToken = 'Token')
But I am not sure on the usage of this.
Thanks in advance.
I remember running into this at some point.
You want to use a paginator - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Paginator.GetParametersByPath
This is how I used it:
import boto3
client = boto3.client('ssm',region_name='eu-central-1')
paginator = client.get_paginator('get_parameters_by_path')
response_iterator = paginator.paginate(
Path='/some/path'
)
parameters=[]
for page in response_iterator:
for entry in page['Parameters']:
parameters.append(entry)
And you would get a list like [{"Name": "/some/path/param, "Value": "something"}] in parameters with all the parameters under the path.
*edit: response would be much richer than just the Name, Value keys. check the paginator docs!
Let me suggest using this library (I'm the author): AWStanding
You can achieve this easily, without worrying about pagination:
import os
from awstanding.parameter_store import load_path
load_path('/stripe', '/spotify')
STRIPE_PRICE = os.environ.get('STRIPE_PRICE', 'fallback_value')
STRIPE_WEBHOOK = os.environ.get('STRIPE_WEBHOOK', 'fallback_value')
SPOTIFY_API_KEY = os.environ.get('SPOTIFY_API_KEY', 'fallback_value')
print(f'price: {STRIPE_PRICE}, webhook: {STRIPE_WEBHOOK}, spotify: {SPOTIFY_API_KEY}')
>>> price: price_1xxxxxxxxxxxxxxxxxxxxxxx, webhook: fallback_value, spotify: fallback_value
I'm trying to process URL's in a pyspark dataframe using a class that I've written and a udf. I'm aware of urllib and other url parsing libraries but for this case I need to use my own code.
In order to get the tld of a url I cross check it against the iana public suffix list.
Here's a simplification of my code
class Parser:
# list of available public suffixes for extracting top level domains
file = open("public_suffix_list.txt", 'r')
data = []
for line in file:
if line.startswith("//") or line == '\n':
pass
else:
data.append(line.strip('\n'))
def __init__(self, url):
self.url = url
#the code here extracts port,protocol,query etc.
#I think this bit below is causing the error
matches = [r for r in self.data if r in self.hostname]
#extra functionality in my actual class
i = matches.index(self.string)
try:
self.tld = matches[i]
# logic to find tld if no match
The class works in pure python so for example I can run
import Parser
x = Parser("www.google.com")
x.tld #returns ".com"
However when I try to do
import Parser
from pyspark.sql.functions import udf
parse = udf(lambda x: Parser(x).url)
df = sqlContext.table("tablename").select(parse("column"))
When I call an action I get
File "<stdin>", line 3, in <lambda>
File "<stdin>", line 27, in __init__
TypeError: 'in <string>' requires string as left operand
So my guess is that it's failing to interpret the data as a list of strings?
I've also tried to use
file = sc.textFile("my_file.txt")\
.filter(lambda x: not x.startswith("//") or != "")\
.collect()
data = sc.broadcast(file)
to open my file instead, but that causes
Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.
Any ideas?
Thanks in advance
EDIT: Apologies, I didn't have my code to hand so my test code didn't explain very well the problems I was having. The error I initially reported was a result of the test data I was using.
I've updated my question to be more reflective of the challenge I'm facing.
Why do you need a class in this case (the code for defining your class is incorrect, you never declared self.data before using it in the init method) the only relevant line that affects the output you want is self.string=string, so you are basically passing the identity function as udf.
The UnicodeDecodeError is due to an encoding issue in your file, it has nothing to do with your definition of the class.
The second error is in the line sc.broadcast(file) , details of which can be found here : Spark: Broadcast variables: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion
EDIT 1
I would redefine your class structure as follows. You basically need to create the instance self.data by calling self.data = data before you can use it. Also anything that you write before the init method is executed irrespective of whether you call that class or not. So moving out the file parsing part will not have any effect.
# list of available public suffixes for extracting top level domains
file = open("public_suffix_list.txt", 'r')
data = []
for line in file:
if line.startswith("//") or line == '\n':
pass
else:
data.append(line.strip('\n'))
class Parser:
def __init__(self, url):
self.url = url
self.data = data
#the code here extracts port,protocol,query etc.
#I think this bit below is causing the error
matches = [r for r in self.data if r in self.hostname]
#extra functionality in my actual class
i = matches.index(self.string)
try:
self.tld = matches[i]
# logic to find tld if no match
I am using a script i've downloaded from the web to access our service API.
I am trying to run the function, but keep getting errors no matter what I am trying to do.
from PyBambooHR import PyBambooHR
bamboo = PyBambooHR(subdomain='domain', api_key='apicode')
changes = bamboo.get_employee_changes()
When I run this, I get the following error:
ValueError: Error: since argument must be a datetime.datetime instance
Now, no matter what I set as arguments, I still getting errors. I've also tried the syntax from: https://www.bamboohr.com/api/documentation/changes.php
The function is:
def get_employee_changes(self, since=None):
"""
Returns a list of dictionaries, each with id, action, and lastChanged keys, representing
the employee records that have changed since the datetime object passed in the since= argument.
#return List of dictionaries, each with id, action, and lastChanged keys.
"""
if not isinstance(since, datetime.datetime):
raise ValueError("Error: since argument must be a datetime.datetime instance")
url = self.base_url + 'employees/changed/'
params = {'since': since.strftime('%Y-%m-%dT%H:%M:%SZ')}
r = requests.get(url, params=params, headers=self.headers, auth=(self.api_key, ''))
r.raise_for_status()
return utils.transform_change_list(r.content)
Thanks for your help
As you see in that function is a parameter since of type datetime.datetime expected.
import datetime
changes = bamboo.get_employee_changes(since=datetime.datetime.now() - datetime.timedelta(days=365))
Should give you the changes since last year
Pass a variable of type datetime.datetime while calling the function bamboo.get_employee_changes()