Develop Reference

Why doesn't this decode function working in python2?

I am debugging a python2 code:
tag_list = [convert(tag) for tag in tag_list]
print('tag_list: ', str(tag_list).decode("utf-8"))
However, the print out is like below:
u"['\\xe4\\xba\\xa4\\xe9\\x80\\x9a\\xe6\\x9c\\x8d\\xe5\\x8a\\xa1', '\\xe7\\xa4\\xbe\\xe4\\xbc\\x9a', '\\xe7\\x94\\xb5\\xe8\\xa7\\x86\\xe5\\x89\\xa7', '\\xe9\\x9f\\xb3\\xe4\\xb9\\x90']"
How to correctly print out the actual strings, instead of those x codes?

print("[" + ", ".join(tag_list) + "]")
I guess would give you the output you want ... maybe

Python - newline if variable is in print function, need fix [duplicate]

This question already has answers here:
How to print without a newline or space
(26 answers)
Closed 1 year ago.
I have one problem with print in Python, I am starting to learn Python, but when I want to print variables in print function, it looks like that after one variable it adds newline to the outcome:
print(game_name + " | " + rating)
I am making a game database with my own ratings on the games but if it prints the game and the rating there is like one empty line belpw the game_name and rating is written, how can I delete that empty newline? I am very new to this, so please don't be mean...

Welcome to Stack Overflow! The most likely culprit is that there is a newline at the end of the game_name variable. The easy fix for this is to strip it off like this:
print(game_name.strip() + " | " + rating)

Say we had two variables like this.
game_name = 'hello\n'
rating = 'there'
game_name has the newline. To get rid of that use strip().
print(game_name.strip() + " | " + rating)
output
hello | there

If you want to remove the line break after printing, you can define the optional end parameter to the print statement.
print('Hello World', end='') # No new line
If your variable has a new line added to it, and you want to remove it, use the .strip() method on the variable.
print('Hello World\n'.strip()) # No empty line
For your code, you could run it:
print(game_name + " | " + rating, end='')
Or
print(game_name + " | " + rating.strip())
If the error is that a new line is printed after game_name, you'll want to call the strip method on that instead.
print(game_name.strip() + " | " + rating)

rating or game_name most likely have a new line after the specified string.
You can fix it by doing this:
game_name = game_name.strip('\n')
rating = rating.strip('\n')
print(game_name + " | " + rating)

Python 3 save string with hex escape codes as jpg

How can I save string s got in this way:
f = open("a.jpg", "rb")
b = f.read()
s = str(b)[2:-1]
as .jpg file? In other program I have only s like form of this image, so it is: "\\xff\\xd8\\xff\\xe0...".

My recommendation would be to change the code, that creates the string in this strange way.
If not possible:
Where is this string coming from? Is this a trusted source or is it coming from a web server or a person who might want to hack / break your computer?
Is the string really created with str(b)[2:-1] or is this just an approximation of the real problem?
I am asking, as this is making things a little more complicated then necessary. (It requires adding a try / except)
Following code should work:
from ast import literal_eval
def stripped_str_b_to_bytes(s):
try:
return literal_eval("b'" + s + "'")
except SyntaxError:
return literal_eval('b"' + s + '"')
testvalues = [
b"A'B",
b'A"B',
bytes([v for v in range(256)]),
]
for b in testvalues:
print("testing with ", b)
s = b[2:-1]
print("S =", s)
c = stripped_str_b_to_bytes(s)
assert b == c
It tries to prepend b' and append ' to s and evaluate that string as a python expression.
If this doesn't work, then it tries to prepend b" and append " to s and evaluate it.

Python - How to handle space as a value of a variable without quotes?

I have a string "bitrate:8000"
I need to convert it to "-bps 8000". Note that the parameter name is changed and so is the delimiter from ':' to space.
Also the delimiters are not fixed always, sometimes I would need to change from ':' to '-' using the same program.
The change rules are supplied as a config file which I am reading through the ConfigParser module. Something like:
[params]
modify_param_name = bitrate/bps
modify_delimiter = :/' '
value = 8000
In my program:
orig_param = modify_param_name.split('/')[0]
new_param = modify_param_name.split('/')[1]
orig_delimiter = modify_delimiter.split('/')[0]
new_delimiter = modify_delimiter.split('/')[1]
new_param_string = new_param + new_delimiter + value
However, this results in the string as below:
-bps' '8000
The question is how can I handle spaces without the ' ' quotes?

The reason why you're getting the ' ' string is probably related to the way you parse your modify_delimiter value.
You're reading that as a string, so that modify_delimiter == ":/' '".
When you're doing:
new_delimiter = modify_delimiter.split('/')[1]
Essentially modify_delimiter.split('/') gives you an array of [':', "' '"].
So when you're doing new_param_string = new_param + new_delimiter + value
, you are concatenating together 'bps' + "' '" + '8000'.
If your modify_delimiter contained the string ':/ ', this would work just fine:
>>> new_param_string = new_param + new_delimiter + value
>>> new_param_string
'bps 8000'
It has been pointed out that you're using ConfigParser. Unfortunatelly, I don't see an option for ConfigParser (either in python 2 or 3) to preserve trailing whitespaces - it looks like they're always stripped.
What I can suggest in that case is that you wrap your string in quotes entirely in your config file:
[params]
modify_param_name = bitrate/bps
modify_delimiter = ":/ "
And in your code, when you initialize modify_delimiter, strip the " on your own:
modify_delimiter = config.get('params', 'modify_delimiter').strip('"')
That way the trailing space will get preserved and you should get your desired output.

Parsing Snort Logs with PyParsing

Having a problem with parsing Snort logs using the pyparsing module.
The problem is with separating the Snort log (which has multiline entries, separated by a blank line) and getting pyparsing to parse each entry as a whole chunk, rather than read in line by line and expecting the grammar to work with each line (obviously, it does not.)
I have tried converting each chunk to a temporary string, stripping out the newlines inside each chunk, but it refuses to process correctly. I may be wholly on the wrong track, but I don't think so (a similar form works perfectly for syslog-type logs, but those are one-line entries and so lend themselves to your basic file iterator / line processing)
Here's a sample of the log and the code I have so far:
[**] [1:486:4] ICMP Destination Unreachable Communication with Destination Host is Administratively Prohibited [**]
[Classification: Misc activity] [Priority: 3]
08/03-07:30:02.233350 172.143.241.86 -> 63.44.2.33
ICMP TTL:61 TOS:0xC0 ID:49461 IpLen:20 DgmLen:88
Type:3 Code:10 DESTINATION UNREACHABLE: ADMINISTRATIVELY PROHIBITED HOST FILTERED
** ORIGINAL DATAGRAM DUMP:
63.44.2.33:41235 -> 172.143.241.86:4949
TCP TTL:61 TOS:0x0 ID:36212 IpLen:20 DgmLen:60 DF
Seq: 0xF74E606
(32 more bytes of original packet)
** END OF DUMP
[**] ...more like this [**]
And the updated code:
def snort_parse(logfile):
header = Suppress("[**] [") + Combine(integer + ":" + integer + ":" + integer) + Suppress("]") + Regex(".*") + Suppress("[**]")
cls = Optional(Suppress("[Classification:") + Regex(".*") + Suppress("]"))
pri = Suppress("[Priority:") + integer + Suppress("]")
date = integer + "/" + integer + "-" + integer + ":" + integer + "." + Suppress(integer)
src_ip = ip_addr + Suppress("->")
dest_ip = ip_addr
extra = Regex(".*")
bnf = header + cls + pri + date + src_ip + dest_ip + extra
def logreader(logfile):
chunk = []
with open(logfile) as snort_logfile:
for line in snort_logfile:
if line !='\n':
line = line[:-1]
chunk.append(line)
continue
else:
print chunk
yield " ".join(chunk)
chunk = []
string_to_parse = "".join(logreader(logfile).next())
fields = bnf.parseString(string_to_parse)
print fields
Any help, pointers, RTFMs, You're Doing It Wrongs, etc., greatly appreciated.

import pyparsing as pyp
import itertools
integer = pyp.Word(pyp.nums)
ip_addr = pyp.Combine(integer+'.'+integer+'.'+integer+'.'+integer)
def snort_parse(logfile):
header = (pyp.Suppress("[**] [")
+ pyp.Combine(integer + ":" + integer + ":" + integer)
+ pyp.Suppress(pyp.SkipTo("[**]", include = True)))
cls = (
pyp.Suppress(pyp.Optional(pyp.Literal("[Classification:")))
+ pyp.Regex("[^]]*") + pyp.Suppress(']'))
pri = pyp.Suppress("[Priority:") + integer + pyp.Suppress("]")
date = pyp.Combine(
integer+"/"+integer+'-'+integer+':'+integer+':'+integer+'.'+integer)
src_ip = ip_addr + pyp.Suppress("->")
dest_ip = ip_addr
bnf = header+cls+pri+date+src_ip+dest_ip
with open(logfile) as snort_logfile:
for has_content, grp in itertools.groupby(
snort_logfile, key = lambda x: bool(x.strip())):
if has_content:
tmpStr = ''.join(grp)
fields = bnf.searchString(tmpStr)
print(fields)
snort_parse('snort_file')
yields
[['1:486:4', 'Misc activity', '3', '08/03-07:30:02.233350', '172.143.241.86', '63.44.2.33']]

You have some regex unlearning to do, but hopefully this won't be too painful. The biggest culprit in your thinking is the use of this construct:
some_stuff + Regex(".*") +
Suppress(string_representing_where_you_want_the_regex_to_stop)
Each subparser within a pyparsing parser is pretty much standalone, and works sequentially through the incoming text. So the Regex term has no way to look ahead to the next expression to see where the '*' repetition should stop. In other words, the expression Regex(".*") is going to just read until the end of the line, since that is where ".*" stops without specifying multiline.
In pyparsing, this concept is implemented using SkipTo. Here is how your header line is written:
header = Suppress("[**] [") + Combine(integer + ":" + integer + ":" + integer) +
Suppress("]") + Regex(".*") + Suppress("[**]")
Your ".*" problem gets resolved by changing it to:
header = Suppress("[**] [") + Combine(integer + ":" + integer + ":" + integer) +
Suppress("]") + SkipTo("[**]") + Suppress("[**]")
Same thing for cls.
One last bug, your definition of date is short by one ':' + integer:
date = integer + "/" + integer + "-" + integer + ":" + integer + "." +
Suppress(integer)
should be:
date = integer + "/" + integer + "-" + integer + ":" + integer + ":" +
integer + "." + Suppress(integer)
I think those changes will be sufficient to start parsing your log data.
Here are some other style suggestions:
You have a lot of repeated Suppress("]") expressions. I've started defining all my suppressable punctuation in a very compact and easy to maintain statement like this:
LBRACK,RBRACK,LBRACE,RBRACE = map(Suppress,"[]{}")
(expand to add whatever other punctuation characters you like). Now I can use these characters by their symbolic names, and I find the resulting code a little easier to read.
You start off header with header = Suppress("[**] [") + .... I never like seeing spaces embedded in literals this way, as it bypasses some of the parsing robustness pyparsing gives you with its automatic whitespace skipping. If for some reason the space between "[**]" and "[" was changed to use 2 or 3 spaces, or a tab, then your suppressed literal would fail. Combine this with the previous suggestion, and header would begin with
header = Suppress("[**]") + LBRACK + ...
I know this is generated text, so variation in this format is unlikely, but it plays better to pyparsing's strengths.
Once you have your fields parsed out, start assigning results names to different elements within your parser. This will make it a lot easier to get the data out afterward. For instance, change cls to:
cls = Optional(Suppress("[Classification:") +
SkipTo(RBRACK)("classification") + RBRACK)
Will allow you to access the classification data using fields.classification.

Well, I don't know Snort or pyparsing, so apologies in advance if I say something stupid. I'm unclear as to whether the problem is with pyparsing being unable to handle the entries, or with you being unable to send them to pyparsing in the right format. If the latter, why not do something like this?
def logreader( path_to_file ):
chunk = [ ]
with open( path_to_file ) as theFile:
for line in theFile:
if line:
chunk.append( line )
continue
else:
yield "".join( *chunk )
chunk = [ ]
Of course, if you need to modify each chunk before sending it to pyparsing, you can do so before yielding it.