Develop Reference

How do I replace characters in lists using dictionary in python

I want to write a code that will replace certain characters in a list in an efficient way using a dictionary.
If I have:
key = {'a':'z','b':'y','c':'x'}
List = ['a','b','c']
How can I get the output
zyx
edit to clarify. The output I want is really
randomvariableorsomething = ['z', 'y', 'x']
My apologies.
Will [key[x] for x in List] work if I don't have a key for it in the dict?

Use get and join:
>>> ''.join(key.get(e,'') for e in List)
'zyx'
If by 'replace' you mean to change the list to the values of the dict in the order of the elements of the original list, you can do:
>>> List[:]=[key.get(e,'') for e in List]
>>> List
['z', 'y', 'x']

key = {'a':'z','b':'y','c':'x'}
List = ['a','b','c']
print([key.get(x,"No_key") for x in List])
#### Output ####
['z', 'y', 'x']
If your interest is only to print them as string,then:
print(*[key.get(x,"No_key") for x in List],sep="")
#### Output ####
zxy
Just in case you need the solution without join.
ss = ''
def fun_str(x):
global ss
ss = ss + x
return(ss)
print([fun_str(x) for x in List][-1])
#### Output ####
zxy

Both keys and List are words in python that can collide with existing objects or methods (dict.keys() and List objects), so I replaced them with k and lst respectively for best practice:
[k[x] for x in lst]

How to configure ruamel.yaml.dump output?

With this data structure:
d = {
(2,3,4): {
'a': [1,2],
'b': 'Hello World!',
'c': 'Voilà!'
}
}
I would like to get this YAML:
%YAML 1.2
---
[2,3,4]:
a:
- 1
- 2
b: Hello World!
c: 'Voilà!'
Unfortunately I get this format:
$ print ruamel.yaml.dump(d, default_flow_style=False, line_break=1, explicit_start=True, version=(1,2))
%YAML 1.2
---
? !!python/tuple
- 2
- 3
- 4
: a:
- 1
- 2
b: Hello World!
c: !!python/str 'Voilà!'
I cannot configure the output I want even with safe_dump. How can I do that without manual regex work on the output?
The only ugly solution I found is something like:
def rep(x):
return repr([int(y) for y in re.findall('^\??\s*-\s*(\d+)', x.group(0), re.M)]) + ":\n"
print re.sub('\?(\s*-\s*(\w+))+\s*:', rep,
ruamel.yaml.dump(d, default_flow_style=False, line_break=1, explicit_start=True, version=(1,2)))

New ruamel.yaml API
You cannot get what you want using ruamel.yaml.dump(), but with the new API, which has
a few more controls, you can come very close.
import sys
import ruamel.yaml
d = {
(2,3,4): {
'a': [1,2],
'b': 'Hello World!',
'c': 'Voilà!'
}
}
def prep(d):
if isinstance(d, dict):
needs_restocking = False
for idx, k in enumerate(d):
if isinstance(k, tuple):
needs_restocking = True
try:
if 'à' in d[k]:
d[k] = ruamel.yaml.scalarstring.SingleQuotedScalarString(d[k])
except TypeError:
pass
prep(d[k])
if not needs_restocking:
return
items = list(d.items())
for (k, v) in items:
d.pop(k)
for (k, v) in items:
if isinstance(k, tuple):
k = ruamel.yaml.comments.CommentedKeySeq(k)
d[k] = v
elif isinstance(d, list):
for item in d:
prep(item)
yaml = ruamel.yaml.YAML()
yaml.indent(mapping=2, sequence=4, offset=2)
yaml.version = (1, 2)
data = prep(d)
yaml.dump(d, sys.stdout)
which gives:
%YAML 1.2
---
[2, 3, 4]:
a:
- 1
- 2
b: Hello World!
c: 'Voilà!'
There is still no simple way to suppress the space before the sequence items, so you cannot get [2,3,4] insted of [2, 3, 4] without some major effort.
Original answer:
You cannot get exactly what you want as output using ruamel.yaml.dump() without major rework of the internals.
The output you like has indentation 2 for the values of the top-level mapping (key a, b, etc) and indentation 4 for the elements of the sequence that is the value for the a key (with the - pushed in 2 positions. That would at least require differencing between indentation levels for mapping and sequences (if not for individual collections) and that is non-trivial.
Your sequence output is compacted from the , (comma, space) what a "normal" flow style emits to just a ,. IIRC this cannot currently be influenced by any parameter, and since you have little contextual knowledge when emitting a collection, it is difficult to "not include the spaces when emitting a sequence that is a key". An additional option to dump() would require changes in several of the sources files and classes.
Less difficult issues, with indication of solution:
Your tuple has to magically convert to a sequence to get rid of the tag !!python/tuple. As you don't want to affect all tuples, this is IMO best done by making a subclass of tuple and represent this as a sequence (optionally represent such tuple as list only if actually used as a key). You can use comments.CommentedKeySeq for that (assuming ruamel.yaml>=0.12.14, it has the proper representation support when using ruamel.yaml.round_trip_dump()
Your key is, when tested before emitting, not a simple key and as such it get a '? ' (question mark, space) to indicate a complex mapping key. . You would have to change the emitter so that the SequenceStartEvent starts a simple key (if it has flow style and not block style). An additional issue is that such a SequenceStartEvent then will be "tested" to have a style attribute (which might indicate an explicit need for '?' on key). This requires changing emitter.py:Emitter.check_simple_key() and emitter.py:Emitter.expect_block_mapping_key().
Your scalar string value for c gets quotes, whereas your scalar string value for b doesn't. You only can get that kind of difference in output in ruamel.yaml by making them different types. E.g. by making it type scalarstring.SingleQuotedScalarString() (and using round_trip_dump()).
If you do:
import sys
import ruamel.yaml
from ruamel.yaml.comments import CommentedMap, CommentedKeySeq
assert ruamel.yaml.version_info >= (0, 12, 14)
data = CommentedMap()
data[CommentedKeySeq((2, 3, 4))] = cm = CommentedMap()
cm['a'] = [1, 2]
cm['b'] = 'Hello World!'
cm['c'] = ruamel.yaml.scalarstring.SingleQuotedScalarString('Voilà!')
ruamel.yaml.round_trip_dump(data, sys.stdout, explicit_start=True, version=(1, 2))
you will get:
%YAML 1.2
---
[2, 3, 4]:
a:
- 1
- 2
b: Hello World!
c: 'Voilà!'
which, apart from the now consistent indentation level of 2, the extra spaces in the flow style sequence, and the required use of the round_trip_dump, will get you as close to what you want without major rework.
Whether the above code is ugly as well or not is of course a matter of taste.
The output will, non-incidently, round-trip correctly when loaded using ruamel.yaml.round_trip_load(preserve_quotes=True).
If control over the quotes is not needed, and neither is the order of your mapping keys important, then you can also patch the normal dumper:
def my_key_repr(self, data):
if isinstance(data, tuple):
print('data', data)
return self.represent_sequence(u'tag:yaml.org,2002:seq', data,
flow_style=True)
return ruamel.yaml.representer.SafeRepresenter.represent_key(self, data)
ruamel.yaml.representer.Representer.represent_key = my_key_repr
Then you can use a normal sequence:
data = {}
data[(2, 3, 4)] = cm = {}
cm['a'] = [1, 2]
cm['b'] = 'Hello World!'
cm['c'] = 'Voilà!'
ruamel.yaml.dump(data, sys.stdout, allow_unicode=True, explicit_start=True, version=(1, 2))
will give you:
%YAML 1.2
---
[2, 3, 4]:
a: [1, 2]
b: Hello World!
c: Voilà!
please note that you need to explicitly allow unicode in your output (default with round_trip_dump()) using allow_unicode=True.
¹ Disclaimer: I am the author of ruamel.yaml.

Can I create list from regular expressions?

I'm making a crawler.
User can specify regular expression string to download data.
When user input form is:
http://xxx/abc[x-z]/image(9|10|11).png
I want to download these.
http://xxx/abcx/image9.png
http://xxx/abcy/image9.png
http://xxx/abcz/image9.png
http://xxx/abcx/image10.png
http://xxx/abcy/image10.png
http://xxx/abcz/image10.png
http://xxx/abcx/image11.png
http://xxx/abcy/image11.png
http://xxx/abcz/image11.png
Can I create the following list from the above regular expression string? Or, can I use each string in for-in block?

If you are wanting to take a user's given regex as an input and generate a list of strings you can use the library sre_yield:
However, be very aware that trying to parse every possible string of a regex can get out of hand very quickly. You'll need to be sure that your users are aware of the implications that wildcard characters and open ended or repeating groups can have on the number of possible matching strings.
As an example, your regex string: http://xxx/abc[x-z]/image(9|10|11).png does not escape the ., which is a wildcard for any character, so it will generate a lot of unexpected strings. Instead we'll need to escape it as seen in the example below:
>>> import sre_yield
>>> links = []
>>> for each in sre_yield.AllStrings(r'http://xxx/abc[x-z]/image(9|10|11)\.png'):
links.append(each)
Or more simply links = list(sre_yield.AllStrings(r'http://xxx/abc[x-z]/image(9|10|11)\.png'))
The result is:
>>> links
['http://xxx/abcx/image9.png', 'http://xxx/abcy/image9.png',
'http://xxx/abcz/image9.png', 'http://xxx/abcx/image10.png',
'http://xxx/abcy/image10.png', 'http://xxx/abcz/image10.png',
'http://xxx/abcx/image11.png', 'http://xxx/abcy/image11.png',
'http://xxx/abcz/image11.png']

You can use product() from the itertools builtin:
from itertools import product
for x, y in product(['x', 'y', 'z'], range(9, 12)):
print 'http://xxx/abc{}/image{}'.format(x, y)
To build your list you can use a comprehension:
links = ['http://xxx/abc{}/image{}'.format(x, y) for x, y in product(['x', 'y', 'z'], range(9, 12))]

Simple try may be-alternative to the previous answers
lst = ['http://xxx/abc%s/image%s.png'%(x,y) for x, y in [(j,i) for i in (9,10,11) for j in ('x', 'y', 'z')]]
Omitted range and format function for quicker performance.
Analysis- I compared my way and the way posted by Jkdc
I ran both way 100000 times but mean shows that itertools approach is faster in terms of execution time-
from itertools import product
import time
from matplotlib import pyplot as plt
import numpy as np
prodct = []
native = []
def test():
start = time.clock()
lst = ['http://xxx/abc{}/image{}'.format(x, y) for x, y in product(('x', 'y', 'z'), range(9, 11))]
end = time.clock()
print '{0:.50f}'.format(end-start)
prodct.append('{0:.50f}'.format(end-start))
start1 = time.clock()
lst = ['http://xxx/abc%s/image%s'%(x,y) for x, y in [(j,i) for i in (9,10,11) for j in ('x', 'y', 'z')]]
end1 = time.clock()
print '{0:.50f}'.format(end1-start1)
native.append('{0:.50f}'.format(end1-start1))
for i in range(1,100000):
test()
y = np.dot(np.array(native).astype(np.float),100000)
x= np.dot(np.array(prodct).astype(np.float),100000)
print np.mean(y)
print np.mean(x)
and getting result for native(no module) and itertools-product as below
for native 2.1831179834
for itertools-product 1.60410432562

Python Copying part of string

I have this line
Server:x.x.x.x # U:100 # P:100 # Pre:0810 # Tel:xxxxxxxxxx
and I want to copy the value 0810 which is after Pre: value
How i can do that ?

You could use the re module ('re' stands for regular expressions)
This solution assumes that your Pre: field will always have four numbers.
If the length of the number varies, you could replace the {4}(expect exactly 4) with + (expect 'one or more')
>>> import re
>>> x = "Server:x.x.x.x # U:100 # P:100 # Pre:0810 # Tel:xxxxxxxxxx"
>>> num = re.findall(r'Pre:(\d{4})', x)[0] # re.findall returns a list
>>> print num
'0810'
You can read about it here:
https://docs.python.org/2/howto/regex.html

As usual in these cases, the best answer depends upon how your strings will vary, and we only have one example to generalize from.
Anyway, you could use string methods like str.split to get at it directly:
>>> s = "Server:x.x.x.x # U:100 # P:100 # Pre:0810 # Tel:xxxxxxxxxx"
>>> s.split()[6].split(":")[1]
'0810'
But I tend to prefer more general solutions. For example, depending on how the format changes, something like
>>> d = dict(x.split(":") for x in s.split(" # "))
>>> d
{'Pre': '0810', 'P': '100', 'U': '100', 'Tel': 'xxxxxxxxxx', 'Server': 'x.x.x.x'}
which makes a dictionary of all the values, after which you could access any element:
>>> d["Pre"]
'0810'
>>> d["Server"]
'x.x.x.x'

How to convert a malformed string to a dictionary?

I have a string s (note that the a and b are not enclosed in quotation marks, so it can't directly be evaluated as a dict):
s = '{a:1,b:2}'
I want convert this variable to a dict like this:
{'a':1,'b':2}
How can I do this?

This will work with your example:
import ast
def elem_splitter(s):
return s.split(':',1)
s = '{a:1,b:2}'
s_no_braces = s.strip()[1:-1] #s.translate(None,'{}') is more elegant, but can fail if you can have strings with '{' or '}' enclosed.
elements = (elem_splitter(ss) for ss in s_no_braces.split(','))
d = dict((k,ast.literal_eval(v)) for k,v in elements)
Note that this will fail if you have a string formatted as:
'{s:"foo,bar",ss:2}' #comma in string is a problem for this algorithm
or:
'{s,ss:1,v:2}'
but it will pass a string like:
'{s ss:1,v:2}' #{"s ss":1, "v":2}
You may also want to modify elem_splitter slightly, depending on your needs:
def elem_splitter(s):
k,v = s.split(':',1)
return k.strip(),v # maybe `v.strip() also?`
*Somebody else might cook up a better example using more of the ast module, but I don't know it's internals very well, so I doubt I'll have time to make that answer.

As your string is malformed as both json and Python dict so you neither can use json.loads not ast.literal_eval to directly convert the data.
In this particular case, you would have to manually translate it to a Python dictionary by having knowledge of the input data
>>> foo = '{a:1,b:2}'
>>> dict(e.split(":") for e in foo.translate(None,"{}").split(","))
{'a': '1', 'b': '2'}
As Updated by Tim, and my short-sightedness I missed the fact that the values should be integer, here is an alternate implementation
>>> {k: int(v) for e in foo.translate(None,"{}").split(",")
for k, v in [e.split(":")]}
{'a': 1, 'b': 2}

import re,ast
regex = re.compile('([a-z])')
ast.literal_eval(regex.sub(r'"\1"', s))
out:
{'a': 1, 'b': 2}
EDIT:
If you happen to have something like {foo1:1,bar:2} add an additional capture group to the regex:
regex = re.compile('(\w+)(:)')
ast.literal_eval(regex.sub(r'"\1"\2', s))

You can do it simply with this:
s = "{a:1,b:2}"
content = s[s.index("{")+1:s.index("}")]
to_int = lambda x: int(x) if x.isdigit() else x
d = dict((to_int(i) for i in pair.split(":", 1)) for pair in content.split(","))
For simplicity I've omitted exception handling if the string doesn't contain a valid specification, and also this version doesn't strip whitespace, which you may want. If the interpretation you prefer is that the key is always a string and the value is always an int, then it's even easier:
s = "{a:1,b:2}"
content = s[s.index("{")+1:s.index("}")]
d = dict((int(pair[0]), pair[1].strip()) for pair in content.split(","))
As a bonus, this version also strips whitespace from the key to show how simple it is.

import simplejson
s = '{a:1,b:2}'
a = simplejson.loads(s)
print a