Related
I am a network engineer by day learning python to automate tasks, please go easy as I am a python newbie.
My goal is to iterate through a range of switchport interfaces and identify down switchport interfaces, then apply a new VLAN ID to the port.
The first stage of my script is below, which presents me with a list of down ports.
The issue I am facing is that I am wanting to over iterate over port numbers 3-6, 38-52 and that are down.
At present I am iterating through the entire list of ports identified on the switch.
import netmiko
from netmiko import ConnectHandler
from getpass4 import getpass
user = 'example_user'
password = getpass('Password: ')
net_connect = ConnectHandler(
device_type="hp_procurve",
host="10.0.0.1",
username= user,
password= password,
)
print('*** Sending command ***')
show_int_brief = net_connect.send_command("show int brief", use_textfsm=True)
net_connect.disconnect()
int_down = []
for item in show_int_brief:
if item['status'] == 'Down':
int_down.append(item['port'])
print('*** Port status known as down ***\n', int_down)
Example output prior to being added to the list int_down.
[{'port': '1', 'type': '100/1000T', 'intrusion_alert': 'No', 'enabled': 'Yes', 'status': 'Up', 'mode': '1000FDx', 'mdi_mode': 'MDI', 'flow_ctrl': 'off', 'bcast_limit': '0'},
{'port': '2', 'type': '100/1000T', 'intrusion_alert': 'No', 'enabled': 'Yes', 'status': 'Up', 'mode': '1000FDx', 'mdi_mode': 'MDIX', 'flow_ctrl': 'off', 'bcast_limit': '0'},
{'port': '3', 'type': '100/1000T', 'intrusion_alert': 'No', 'enabled': 'Yes', 'status': 'Down', 'mode': '1000FDx', 'mdi_mode': 'Auto', 'flow_ctrl': 'off', 'bcast_limit': '0'},
{'port': '4', 'type': '100/1000T', 'intrusion_alert': 'No', 'enabled': 'Yes', 'status': 'Down', 'mode': '1000FDx', 'mdi_mode': 'Auto', 'flow_ctrl': 'off', 'bcast_limit': '0'},
{'port': '5', 'type': '100/1000T', 'intrusion_alert': 'No', 'enabled': 'Yes', 'status': 'Down', 'mode': '1000FDx', 'mdi_mode': 'Auto', 'flow_ctrl': 'off', 'bcast_limit': '0'},
{'port': '6', 'type': '100/1000T', 'intrusion_alert': 'No', 'enabled': 'Yes', 'status': 'Down', 'mode': '1000FDx', 'mdi_mode': 'Auto', 'flow_ctrl': 'off', 'bcast_limit': '0'}]
And so on..
Example output after being placed in 'int_down' and printed.
Numbers identified are expected, as these are in a down state.
['3', '4', '5', '6', '7', '8', '9', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '40', '41', '42', '43', '44', '45', '47', '49', '50', '51', '52']
The idea is to then use this list in another command that will proceed with applying VLAN configuration only to these ports, though will tackle this as I get past this hurdle.
Cheers,
Luppa
I don't think it is worth to limit to a range a priory, since ensuring the port data is in the proper shape and organization would be similar or even more computational effort with respect to looping through all ports directly. Instead I suggest to filter ad-hoc:
I use the boild down list of ports-dicts:
ports = [
{'port': '1', 'status': 'Down'},
{'port': '7', 'status': 'Down'},
{'port': '2', 'status': 'Up'},
{'port': '8', 'status': 'Up'},
{'port': '4', 'status': 'Down'},
{'port': '5', 'status': 'Up'},
{'port': '9', 'status': 'Up'},
{'port': '6', 'status': 'Down'}
]
Please note that they are not in order and have missing entries to simulate real world data more closely.
Then first I implement a helper function, where I can specify the ranges of interest. If these are more than two consider to use any() and a list of ranges for readability.
def of_interest(port_num: int) -> bool:
return port_num in range(3, 7) or port_num in range(38, 53)
Now your nested for and if structure can be expressed as list comprehension:
down_ports = [e['port'] for e in ports if e['status'] == 'Down' and of_interest(int(e['port']))]
Of course, depending on what you want to do in later steps it might make sense to copy all values from the port list of that entry.
Does this help?
I am trying to construct a dictionary called author_venues in which author names are the keys and values are the list of venues where they have published.
I was given two dictionaries:
A sample author_pubs dictionary where author name is the key and a list of publication ids is the value
defaultdict(list,
{'José A. Blakeley': ['2',
'25',
'2018',
'2185',
'94602',
'145114',
'182779',
'182780',
'299422',
'299426',
'299428',
'299558',
'302125',
'511816',
'521294',
'597967',
'598123',
'598125',
'598130',
'598132',
'598134',
'598136',
'598620',
'600180',
'600221',
'642049',
'643606',
'808458',
'832249',
'938531',
'939047',
'1064640',
'1064641',
'1065929',
'1118153',
'1269074',
'2984279',
'3154713',
'3169639',
'3286099',
'3494140'],
'Yuri Breitbart': ['3',
'4',
'76914',
'113875',
'140847',
'147900',
'147901',
'150951',
'176221',
'176896',
'182963',
'200336',
'262940',
'285098',
'285564',
'299526',
'301313',
'303418',
'304160',
'400040',
'400041',
'400174',
'400175',
'402178',
'482506',
'482785',
'544757',
'545233',
'545429',
'559737',
'559761',
'559765',
'559783',
'559785',
'597889',
'598201',
'598202',
'598203',
'599325',
'599899',
'620806',
'636455',
'641884',
'642157',
'654200',
'654201',
'740600',
'740602',
'833336',
'844280',
'856032',
'856222',
'888870',
'934979',
'938228',
'941484',
'945339',
'949548',
'971592',
'971593',
'972813',
'972958',
'1064100',
'1064690',
'1064691',
'1064693',
'1064694',
'1078369',
'1078370',
'1089675',
'1095084',
'1121956',
'1122006',
'1122610',
'1127610',
'1138059',
'1138061',
'1141938',
'1227365',
'1278703',
'1319498',
'2818906',
'2876867',
'2978458',
'3015058',
'3223418'],
A sample venue_pubs dictionary where venue name is the key and a list of publication ids is the value
defaultdict(list,
{'Modern Database Systems': ['2',
'3',
'4',
'5',
'6',
'7',
'8',
'9',
'10',
'11',
'12',
'13',
'14',
'15',
'16',
'17',
'18',
'19',
'20',
'21',
'22',
'23',
'24',
'25',
'26',
'27',
'28',
'29',
'30',
'31',
'32',
'33',
'34',
'1203459',
'3000615',
'3000616',
'3000617',
'3000618',
'3000619',
'3000620',
'3000621',
'3000622',
'3000623',
'3000624',
'3000625',
'3000626'],
'Object-Oriented Concepts, Databases, and Applications': ['36',
'37',
'38',
'39',
'40',
'41',
'42',
'43',
'44',
'45',
'46',
'47',
'48',
'49',
'50',
'51',
'52',
'53',
'54',
'55',
'56',
'57',
'58',
'59'],
'The INGRES Papers': ['60',
'61',
'62',
'63',
'64',
'65',
'66',
'67',
'68',
'69'],
'Temporal Databases': ['168',
'169',
'170',
'171',
'172',
'173',
'174',
'175',
'176',
'177',
'178',
'179',
'180',
'181',
'182',
'183',
'184',
'185',
'186',
'187',
'188',
'189',
'190',
'627582',
'627584',
'627588',
'627589',
'627591',
'627592',
'627593',
'627594',
'627596',
'627600',
'627601',
'627602',
'627603',
'627604',
'627605',
'627608',
'627613',
'627615',
'627616',
'627617'],
The resulting dictionary should look like {'author':['venue1','venue2','venue3']}
author_venue = defaultdict(list)
This is code I wrote:
for k,v in author_pubs.items():
for item in v:
for x,y in venue_pubs.items():
if item in y:
venue = x
author_venue[k].append(venue)
But this loop takes forever since I have over 3million records
please help!
You can "invert" the dictionary venue_pubs to speed up the search:
from collections import defaultdict
author_pubs = {
"author1": [1, 2, 3],
"author2": [3, 4, 5],
}
venue_pubs = {
"xxx1": [1, 4, 20],
"xxx2": [4, 30, 40],
}
# "invert" dictionary `venue_pubs`:
tmp = defaultdict(list)
for k, v in venue_pubs.items():
for val in v:
tmp[val].append(k)
author_venue = defaultdict(list)
for k, v in author_pubs.items():
for item in v:
venues = tmp.get(item)
if not venues is None:
author_venue[k].extend(venues)
print(author_venue)
Prints:
defaultdict(<class 'list'>, {'author1': ['xxx1'], 'author2': ['xxx1', 'xxx2']})
EDIT: To remove duplicates:
# ...
for k in author_venue:
author_venue[k] = list(set(author_venue[k]))
print(author_venue)
I'm not a coder by trade, rather an infrastructure engineer that's learning to code for my role. I have an output that I am getting and I am struggling to think how I can get this to work.
I've utilized some of my colleagues but the data is outputted in a weird format and I am unsure how to get the outcome I want. I have tried splitting the lines but it will not work perfectly.
The current code is simple. It just pulls the output command from the switch & I then have it split the lines:
output = net_connect.send_command("show switch")
switchlines = output.splitlines()
print(output)
print(switchlines[5])
It will then output the following in this case:
Switch/Stack Mac Address : 188b.45ea.a000 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
*1 Active 188b.45ea.a000 15 V01 Ready
2 Standby 00ca.e5fc.1780 14 V06 Ready
3 Member 00ca.e5fc.5e80 13 V06 Ready
4 Member 00ca.e588.f480 12 V06 Ready
5 Member 00ca.e588.ee80 11 V06 Ready
*1 Active 188b.45ea.a000 15 V01 Ready
That table comes out as a string & essentially, I need to find a way to split that into usable chunks (I.E a 2D Array) So I can use each field individually.
You already got the lines separated in a list (switchlines), so all you have left to do is iterate over that list and split each one on spaces. Because there are many spaces separating, we also want to strip those elements. So you could do something like:
res = []
for line in switchlines[5:]:
elements = [x.strip() for x in line.split()]
res.append(elements)
And this gives on your example text:
[['*1', 'Active', '188b.45ea.a000', '15', 'V01', 'Ready'],
['2', 'Standby', '00ca.e5fc.1780', '14', 'V06', 'Ready'],
['3', 'Member', '00ca.e5fc.5e80', '13', 'V06', 'Ready'],
['4', 'Member', '00ca.e588.f480', '12', 'V06', 'Ready'],
['5', 'Member', '00ca.e588.ee80', '11', 'V06', 'Ready']]
Another option that can later help you work on the data, is collect it into a dictionary instead of a list:
for line in switchlines[5:]:
switch, role, mac, prio, ver, state, *extras = [x.strip() for x in line.split()]
res.append({'switch': switch, 'role': role, 'mac': mac,
'prio': prio, 'ver': ver, 'state': state, 'extras': extras})
And this gives on your example text:
[{'switch': '*1', 'role': 'Active', 'mac': '188b.45ea.a000', 'prio': '15', 'ver': 'V01', 'state': 'Ready', 'extras': []},
{'switch': '2', 'role': 'Standby', 'mac': '00ca.e5fc.1780', 'prio': '14', 'ver': 'V06', 'state': 'Ready', 'extras': []},
{'switch': '3', 'role': 'Member', 'mac': '00ca.e5fc.5e80', 'prio': '13', 'ver': 'V06', 'state': 'Ready', 'extras': []},
{'switch': '4', 'role': 'Member', 'mac': '00ca.e588.f480', 'prio': '12', 'ver': 'V06', 'state': 'Ready', 'extras': []},
{'switch': '5', 'role': 'Member', 'mac': '00ca.e588.ee80', 'prio': '11', 'ver': 'V06', 'state': 'Ready', 'extras': []}]
I'm trying to scrape the following website:
http://mlb.mlb.com/stats/sortable_batter_vs_pitcher.jsp#season=2018&batting_team=119&batter=571771&pitching_team=133&pitcher=641941
(this is an example URL with a certain pitcher/batter matchup)
I'm able to enter the player codes and team codes easily with this function:
def matchupURL(season, batter, batterTeam, pitcher, pitcherTeam):
return "http://mlb.mlb.com/stats/sortable_batter_vs_pitcher.jsp#season=" + str(season)+ "&batting_team="+str(teamNumDict[batterTeam])+"&batter="+str(batter)+"&pitching_team="+str(teamNumDict[pitcherTeam])+"&pitcher="+str(pitcher);
which works nicely, and the returned string works when pasted into my browser.
But when i make a request a la
newURL = matchupURL(2018,i.id,x.home_team,j.id,x.away_team)
print(i+ " vs " + j)
newSes = requests.get(newURL);
html = BeautifulSoup(newSes.text, "lxml")
mydivs = html.findAll("td",{"class":"dg-ops"})
#do something with this div
I'm unable to find the div. Infact, the entire format of the HTML returned changes. Further, adding headers didnt help, nor did using urllib instead of requests.
This page is a dynamic, i.e., the content is dynamically generated by javascript and showed in the front. That is the reason you can't detect the div tag.
But in this case you can scrape easier. With inspect tool from your browser you can detect that the data comes from a GET request to an URL. For your example, you only have to provide the players id :
import requests
url = 'http://lookup-service-prod.mlb.com/json/named.stats_batter_vs_pitcher_composed.bam'
params = {"sport_code":"'mlb'","game_type":"'R'","player_id":"571771","pitcher_id":"641941"}
resp = requests.get(url, params=params).json()
print(resp)
That prints:
{'stats_batter_vs_pitcher_composed': {'stats_batter_vs_pitcher_total': {'queryResults': {'created': '2018-04-12T22:21:47', 'totalSize': '1', 'row': {'hr': '1', 'gidp': '0', 'pitcher_first_last_html': 'Emilio Pagán', 'player': 'Hernandez, Enrique', 'np': '4', 'sac': '0', 'pitcher': 'Pagan, Emilio', 'rbi': '1', 'player_first_last_html': 'Enrique Hernández', 'tb': '4', 'bats': 'R', 'xbh': '1', 'bb': '0', 'slg': '4.000', 'avg': '1.000', 'pitcher_id': '641941', 'ops': '5.000', 'hbp': '0', 'pitcher_html': 'Pagán, Emilio', 'g': '', 'd': '0', 'so': '0', 'throws': 'R', 'sf': '0', 'tpa': '1', 'h': '1', 'cs': '0', 'obp': '1.000', 't': '0', 'ao': '0', 'r': '1', 'go_ao': '-.--', 'sb': '0', 'player_html': 'Hernández, Enrique', 'sbpct': '.---', 'player_id': '571771', 'ibb': '0', 'ab': '1', 'go': '0'}}}, 'copyRight': ' Copyright 2018 MLB Advanced Media, L.P. Use of any content on this page acknowledges agreement to the terms posted here http://gdx.mlb.com/components/copyright.txt ', 'stats_batter_vs_pitcher': {'queryResults': {'created': '2018-04-12T22:21:47', 'totalSize': '1', 'row': {'hr': '1', 'gidp': '0', 'pitcher_first_last_html': 'Emilio Pagán', 'player': 'Hernandez, Enrique', 'np': '4', 'sac': '0', 'pitcher': 'Pagan, Emilio', 'rbi': '1', 'opponent': 'Oakland Athletics', 'player_first_last_html': 'Enrique Hernández', 'tb': '4', 'xbh': '1', 'bats': 'R', 'bb': '0', 'avg': '1.000', 'slg': '4.000', 'pitcher_id': '641941', 'ops': '5.000', 'hbp': '0', 'pitcher_html': 'Pagán, Emilio', 'g': '', 'd': '0', 'so': '0', 'throws': 'R', 'sport': 'MLB', 'sf': '0', 'team': 'Los Angeles Dodgers', 'tpa': '1', 'league': 'NL', 'h': '1', 'cs': '0', 'obp': '1.000', 't': '0', 'ao': '0', 'season': '2018', 'r': '1', 'go_ao': '-.--', 'sb': '0', 'opponent_league': 'AL', 'player_html': 'Hernández, Enrique', 'sbpct': '.---', 'player_id': '571771', 'ibb': '0', 'ab': '1', 'opponent_id': '133', 'team_id': '119', 'go': '0', 'opponent_sport': 'MLB'}}}}}
I have a .csv file with the regular expression patterns that I want to match as well as the replacement patterns that I want. Some are extremely simple, such as "." -> "" or "," -> "".
When I run the following code, however, it doesn't seem to recognize the variables and the pattern is never matched.
f = open('normalize_patterns.csv', 'rU')
c = csv.DictReader(f)
for row in c:
v = re.sub(row['Pattern'],row['Replacement'],v)
Afterwards, v is never changed and I can't seem to find out why. When I run the simple case of
v = re.sub("\.", "", v)
v = re.sub(",", "", v)
however, all the periods and commas are removed. Any help on the issue would be amazing. Thank you in advance! (I am pretty sure that the .csv file is formatted correctly, I've run it with just the "." and "" case and it still does not work for a certain reason)
Edit:
Here are the outputs of printing row. (Thanks David!)
{'Pattern': "r'(?i)&'", 'ID': '1', 'Replacement': "'and'"}
{'Pattern': "r'(?i)\\bAssoc\\b\\.?'", 'ID': '2', 'Replacement': "'Association'"}
{'Pattern': "r'(?i)\\bInc\\b\\.?'", 'ID': '3', 'Replacement': "'Inc.'"}
{'Pattern': "r'(?i)\\b(L\\.?){2}P\\.?'", 'ID': '4', 'Replacement': "''"}
{'Pattern': "r'(?i)\\bUniv\\b\\.?'", 'ID': '5', 'Replacement': "'University'"}
{'Pattern': "r'(?i)\\bCorp\\b\\.?'", 'ID': '6', 'Replacement': "'Corporation'"}
{'Pattern': "r'(?i)\\bAssn\\b\\.?'", 'ID': '7', 'Replacement': "'Association'"}
{'Pattern': "r'(?i)\\bUnivesity\\b'", 'ID': '8', 'Replacement': "'University'"}
{'Pattern': "r'(?i)\\bIntl\\b\\.?'", 'ID': '9', 'Replacement': "'International'"}
{'Pattern': "r'(?i)\\bInst\\b\\.?'", 'ID': '10', 'Replacement': "'Institute'"}
{'Pattern': "r'(?i)L\\.L\\.C\\.'", 'ID': '11', 'Replacement': "'LLC'"}
{'Pattern': "r'(?i)Chtd'", 'ID': '12', 'Replacement': "'Chartered'"}
{'Pattern': "r'(?i)Mfg\\b\\.?'", 'ID': '13', 'Replacement': "'Manufacturing'"}
{'Pattern': 'r"Nat\'l"', 'ID': '14', 'Replacement': "'National'"}
{'Pattern': "r'(?i)Flordia'", 'ID': '15', 'Replacement': "'Florida'"}
{'Pattern': "r'(?i)\\bLtd\\b\\.?'", 'ID': '16', 'Replacement': "'Ltd.'"}
{'Pattern': "r'(?i)\\bCo\\b\\.?'", 'ID': '17', 'Replacement': "'Company'"}
{'Pattern': "r'(?i)\\bDept\\b\\.?i\\'", 'ID': '18', 'Replacement': "'Department'"}
{'Pattern': "r'(?i)Califronia'", 'ID': '19', 'Replacement': "'California'"}
{'Pattern': "r'(?i)\\bJohn\\bHopkins\\b'", 'ID': '20', 'Replacement': "'Johns Hopkins'"}
{'Pattern': "r'(?i)\\bOrg\\b\\.?'", 'ID': '21', 'Replacement': "'Organization'"}
{'Pattern': "r'(?i)^[T]he\\s'", 'ID': '22', 'Replacement': "''"}
{'Pattern': "r'(?i)\\bAuth\\b\\.?'", 'ID': '23', 'Replacement': "'Authority'"}
{'Pattern': "r'.'", 'ID': '24', 'Replacement': "''"}
{'Pattern': "r','", 'ID': '25', 'Replacement': "''"}
{'Pattern': "r'(?i)\\s+'", 'ID': '0', 'Replacement': "''"}
And here are a few lines of the csv file (Opened in TextMate)
0,r'(?i)\s+',''
1,r'(?i)&','and'
2,r'(?i)\bAssoc\b\.?','Association'
3,r'(?i)\bInc\b\.?','Inc.'
Your issue is that your pattern values are not actually the regex pattern you want, your regex pattern is wrapped in an additional string.
For example, in your dictionary you have the value "r'.'", which you are using as a pattern. You code will run re.sub("r'.'", "", v), which probably isn't what you want:
>>> re.sub("r'.'", "", "This . won't match")
"This . won't match"
>>> re.sub("r'.'", "", "This r'x' will match")
'This will match'
To fix this you should go back to where you are adding the regex to the dictionary and stop doing whatever is causing the string wrapping. It might be something like row['Pattern'] = repr(regex).
If you need to keep the dictionary the same for reason then be very careful with eval, if the strings are coming from an untrusted source then eval is a big security risk. Use ast.literal_eval instead.
If you remove the r'' around the pattern, it will work.
So the pattern that matches . should be as simple as '\.' instead of "r'\.'"
The problem is r in your pattern is taken as a literal r instead of it raw string meaning.
So you can also try:
v=re.sub(eval(row['Pattern']), row['Replacement'], v)