Develop Reference

When writing in Python a dictionary to a YAML file, how to make sure the string in the YAML file is split based on '\n'?

I have a long string in a dictionary which I will dump to a YAML file.
As an example
d = {'test': {'long_string':'this is a long string that does not succesfully split when it sees the character '\n' which is an issue'}}
ff = open('./test.yaml', 'w+')
yaml.safe_dump(d, ff)
Which produces the following output in the YAML file
test:
long_string: "this is a long string that does not successfully split when it sees\
\ the character '\n' which is an issue"
I want the string which is inside the YAML file to only be split into a new line when it sees the "\n", also, I don't want any characters indicating that it's a newline. I want the output as follows:
test:
long_string: "this is a long string that does not successfully split when it sees the character ''
which is an issue"
What do I need to do to make the yaml.dump or yaml.safe_dump fulfill this?

There is no general solution. YAML is a format intentionally designed in a way that lets the implementation decide on the exact representation of values.
What you can do is to suggest a format. The dumper will honor this suggestion if possible. The one scalar format that breaks at literal newlines in the value and nowhere else is a literal block scalar. This code will dump your string as such if possible:
import yaml, sys
class as_block(str):
#staticmethod
def represent(dumper, data):
return dumper.represent_scalar('tag:yaml.org,2002:str', data, style='|')
yaml.SafeDumper.add_representer(as_block, as_block.represent)
d = {'test': {'long_string':as_block('this is a long string that does not succes
fully split when it sees the character\n which is an issue')}}
yaml.safe_dump(d, sys.stdout)
Output:
test:
long_string: |-
this is a long string that does not succesfully split when it sees the character
which is an issue
I use as_block for the string that should be written as block scalar.
You can theoretically use this for all strings, but be aware that long_string and test would then also be written als block scalars, which is most probably not what you want.
This will not work when there is space before the line break, because YAML ignores space at the end of a line of a block scalar, so the serializer will choose another format to not lose the space character(s).
You can also take a step back and ask yourself why this is an issue in the first place. A YAML implementation is perfectly able to load the generated YAML and reconstruct your string.

Loads json with double quotes in it

I'm trying to parse a JSON string that contains double quotes in it:
import json
x = '''{"key":"Value \"123\" "}'''
When I try to load this JSON using the following statement
y = json.loads(x)
It raises the following exception:
Expecting ',' delimiter: line 1 column 15 (char 14)
As per my understanding, it is due to the double quotes around 123 in JSON. Also, I tried replacing the \" (backslash quote) with some other stuff as well but all in vain
x.replace('\"',"'")
as it also replaced the double quotes that are present around key and value as well
'''{"key": "Value \"123\" "}''' ---Replacing--> '''{'key':'Value '123' '}''')
I can not change anything in the input string. That's coming from an API.
Any help would be highly appreciated, I'm stuck with this for quite a time now. Thanks in advance...

\" within a string is simply a double quotation mark. You need to add another backslash:
x = '''{"key":"Value \\"123\\" "}'''

Escape double quotes when converting a dict to json in Python

I need to escape double quotes when converting a dict to json in Python, but I'm struggling to figure out how.
So if I have a dict like {'foo': 'bar'}, I'd like to convert it to json and escape the double quotes - so it looks something like:
'{\"foo\":\"bar\"}'
json.dumps doesn't do this, and I have tried something like:
json.dumps({'foo': 'bar'}).replace('"', '\\"') which ends up formatting like so:
'{\\"foo\\": \\"bar\\"}'
This seems like such a simple problem to solve but I'm really struggling with it.

Your last attempt json.dumps({'foo': 'bar'}).replace('"', '\\"') is actually correct for what you think you want.
The reason you see this:
'{\\"foo\\": \\"bar\\"}'
Is because you're printing the representation of the string. The string itself will have only a single backslash for each quote. If you use print() on that result, you will see a single backslash

What you have does work. Python is showing you the literal representation of it. If you save it to a variable and print it shows you what you're looking for.
>>> a = json.dumps({'foo': 'bar'}).replace('"', '\\"')
>>> print a
{\"foo\": \"bar\"}
>>> a
'{\\"foo\\": \\"bar\\"}'

How to parse data into proper json in python and remove slashes

I am getting this as my response
b'{"userdetails":[["{\\”user_id\\":[\\”54562af66ffd\\"],\\”user_name\\":[\\"bewwrking\\"],\\”room\\":[\\"31\\”]}'
I want to convert it into proper json without any double slashes.
Is there any buildin function for that or i need to do string replace

If you have control over how it is being sent, I would recommend doing to_string on any relevant field/keys that you are sending as json. I had some weird json responses before sanitizing the input to json_dump.

remove the leading b and run replace as below.
s = '{"userdetails":[["{\\"user_id\\":[\\"54562af66ffd\\"],\\"user_name\\":[\\"bewwrking\\"],\\"room\\":[\\"31\\"]}'
s = s.replace('\','')
print(s)
{"userdetails":[["{"user_id":["54562af66ffd"],"user_name":["bewwrking"],"room":["31"]}

How can I read blackslashes from a file correctly?

The following code:
key = open("C:\Scripts\private.ppk",'rb').read()
reads the file and assigns its data to the var key.
For a reason, backslashes are multiplied in the process. How can I make sure they don't get multiplied?

You ... don't. They are escaped when they are read in so that they will process properly when they are written out / used. If you're declaring strings and don't want to double up the back slashes you can use raw strings r'c:\myfile.txt', but that doesn't really apply to the contents of a file you're reading in.
>>> s = r'c:\boot.ini'
>>> s
'c:\\boot.ini'
>>> repr(s)
"'c:\\\\boot.ini'"
>>> print s
c:\boot.ini
>>>
As you can see, the extra slashes are stored internally, but when you use the value in a print statement (write a file, test for values, etc.) they're evaluated properly.

You should read this great blog post on python and the backslash escape character.
And under some circumstances, if
Python prints information to the
console, you will see the two
backslashes rather than one. For
example, this is part of the
difference between the repr() function
and the str() function.
myFilename =
"c:\newproject\typenames.txt" print
repr(myFilename), str(myFilename)
produces
'c:\newproject\typenames.txt'
c:\newproject\typenames.txt

Backslashes are represented as escaped. You'll see two backslashes for each real one existing on the file, but that is normal behaviour.
The reason is that the backslash is used in order to create codes that represent characters that cannot be easily represented, such as new line '\n' or tab '\t'.

Are you trying to put single backslashes in a string? Strings with backslashes require and escape character, in this case "\". It will print to the screen with a single slash

In fact there is a solution - using eval, as long as the file content can be wrapped into quotes of some kind. Following worked for me (PATH contains some script that executes Matlab):
MATLAB_EXE = "C:\Program Files (x86)\MATLAB\R2012b\bin\matlab.exe"
content = open(PATH).read()
MATLAB_EXE in content # False
content = eval(f'r"""{content}"""')
MATLAB_EXE in content # True
This works by evaluating the content as python string literal, making double escapes transform into single ones. Raw string is used to prevent escapes forming special characters.